After a long and slow process, I've decided to "cut the cord". I haven't been using my cable subscription much lately, and I've done very well filling in the gap with over-the-air and online services. Today I took the first big step in making that happen. I had a TV and internet bundle from Spectrum cable. I got today a faster internet-only plan from Verizon FiOS, for about half the cost. It's mostly fine. But one of the first things I discovered is that mtr was misbehaving. Every host appeared two hops away, and only a few milliseconds away. I know this is not true.
Traceroute usually/sometimes works by sending ICMP echoes with a low TTL. Too low, so the connection fails, and the hop at that distance replies that it fails. Up the TTL by one until a successful reply, and we've found the target. Verizon is forging successful answers when only one TTL (my router, plus one) remains. I discovered this by searching around for an explanation of what was going wrong exactly, and found a forum post (on a thread from 2018, so not a very temporary issue) with an acceptable workaround. Have the router never send a packet with TTL=1, bump it up to 2 instead. This doubles the next hop, but that's the best we can get if Verizon is going to lie. Problem is, the solution was given in a proprietary (?) configuration format.
I use OpenWRT, which is "just" standard Linux. So it uses iptables. I came up with:
Phew. That's: -t mangle use the "mangle" table (manpage: "This table is used for specialized packet alteration."), -I POSTROUTING use the "POSTROUTING" chain ("for altering packets as they are about to go out"), -o eth1.2 for packets going out on the eth1.2 interface (my WAN interface), -m ttl use the "ttl" match extension module, --ttl_eq 1 the TTL value equals one, -j TTL jump to the (custom) "TTL" chain, and -ttl-inc 1 increase that TTL value by one (from one, to two).
Now Verizon never sees a TTL=1 ICMP packet coming from me, and never forges the answer, and mtr just works (except that the first hop past the router is a duplicated copy of the next hop):
All I need to do is put this in OpenWRT's /etc/firewall.user file. Google is actually only a few milliseconds away, but they're also 9 hops away, not two.
For years I've had an extremely minor leak in my kitchen sink drain. For a long time I had a try under there to catch drips, and though it clearly displayed signs of having been wet, whenever I'd check, it would also be dried out. At one point I felt like things got better and removed they tray after not seeing new signs of drips for long enough. Then things got worse. More recently I re-added a tray to catch drips, and regularly found it full of standing water. And started checking more often, and needing to empty it more and more often. Can't ignore it anymore. The drips are coming from the top joint right between the sink and the drain. So today I finally did something about it.
I also feel like I've always been just a bit unhappy that the drain was off-center with regards to the opening in the sink. And when I got it out, the discharge pipe certainly does not line up with the center of the opening either. Here is where mild hijinks ensue. It took craziness to get this far: wood blocks and clamps and lots of inappropriate hammering on the lock nut to remove the drain. Now I'm stuck, if I want to fix this. I really can't see how to line the pipe up with the drain correctly, plus I'd really like to end up with a nice on-center look. So, an extra trip to Home Depot to grab a big pipe wrench, and I've got things further disassembled. I made another extra trip to grab a replacement part: Under my sink('s drain) are two 90 degree elbows, a short pipe for the dishwasher to connect to, and a huge solid brass P trap. I tried getting a different 90 degree elbow, smaller, which would move things closer in enough to (I thought) line up.
Turns out, wrong thing. Not only would it mean going from metal to plastic pieces, it didn't have the right style of connectors on both ends (threaded, both sides while I needed threaded on just one). In figuring this out I did further disassembly, and found that the existing elbows were adjustable enough to be installed so they line up correctly!
So I laid a healthy bead of plumber's putty in the sink, and (with the discharge pipe out of the way) started installing just the drain, which went pretty well. Then, re-assembled the various parts to connect to it, which also went well. It seemed. I filled up a 2-quart bucket and dumped it down to find ... worse leaks than when I started.
Thankfully all it took was re-dis-assembly to add some teflon tape (in spots where it wasn't, before I started). Tried again: no leaks, no drips at all!
There we go. Almost perfect, except tightening the last connection turned that slot a bit and it's no longer nicely lined up...
I mostly leave my window blinds in the same spot all the time. I know the spot I like. But it's autumn and so I'm opening and closing the window, and sometimes the blinds too, to let fresh air in while it's nice outside. Recently one set of blinds started to sag on one side. I got up on a stool to check, and the hanger was loose on that side. It's been a few weeks, but I've finally repaired that, today.
I first investigated the hangers and found a typical arrangement: A little springy piece of metal in back, released by levering with a screwdriver, holds the back edge of the blinds while the front edge of the hanger clips into a slot on the front. The blinds were a little tricky to move about by myself while standing on a stool, but I got them safely out of the way and looked at the hanger. It was screwed straight into drywall, no anchors. I'm surprised it lasted as long as it did. I got some anchors and installed them: voila, back in place where it belongs.
I live on the back (away from the street) side of my apartment building. So things tend to be pretty quiet, and there's no street lights outside my window. But there is another building, with a hallway with windows and bright fluorescent lights that never turn off. It's not much, but it's enough to be annoying at night.
So a while back, I got some black curtains and a rod to hang them from (which friction fits between the support beams). This basically solves the problem. They're thin but good enough to block most of the light from outside in the "upstairs" sleeping loft where my bed is. But it does leave a section open. I always wanted to fix that. When I first set this up, I considered continuing the curtains straight across to the wall. that would have divided the hallway a bit too much though.
After quite a long time I finally came up with an idea I like. I've got a standard curtain rod, intended to bridge across the top of a window. But this spot is a concrete support beam: not easy to screw into for mounting. The solution was some tiny brackets, with enough surface area to simply hot glue to the wall. One of them ripped the paint/plaster off after a day or so, but that was a week ago and they've held fast since, after being re-glued to the bare surface beneath.
First I had to do some sewing, to fit the curtain around the support beam. Then it turns the corner and covers the remaining open area on this side. It's a minor change but a great improvement in feel.
Some time last year the server I keep in my Mom's basement (for my remote backups and a few small things for her) went kaput. I don't know how or why because it's my remote server and I don't have easy access to fix it. It was almost exactly a decade old, and not very powerful when it was new (to me, refurbished) and it just wasn't booting at all. It was time to replace it. But since it's primarily a backup server, time was not of the essence. I took the time to figure out how to rebuild it around a Supermicro server-class motherboard, with built in IPMI. A technology that allows completely hands-off remote management of every aspect of the server. Great for a remote backup machine! I slowly bought parts, assembled them all at home and then once it was working well, brought it out to my Mom's at my next visit.
Point being: it introduced me to building a server with real server-class hardware, with built in remote management. I got a taste for it.
Separately I've told the story of how I broke and rebuilt the main server I keep at home, the primary source of the backups to be sent remotely. More recently I figured out that the first-generation AMD Ryzen chip that was in it was not so hot. So much so that AMD accepted an RMA even a couple years later, known design defect. And replaced it. With a more powerful chip. Which seemed nice at first. Turns out, however, it's not. In addition to the problem that was bad enough (not really, the workaround was easy and the occurrence was predictable and rare) to warrant a return and replacement for free, there's a worse problem with these early Ryzen chips: random freezes and reboots. The popular internet theories tend to revolve around bad power management or bad early motherboard support. Or both? This problem happens, ironically enough at idle as opposed to at high load (as the bad-enough-for-return problem did). Which usually means that after a few hours of sitting around doing almost nothing (e.g. overnight!) the machine will freeze, crash, or reboot.
This wasn't obvious right at first, but within a couple weeks became a real problem, every few days I'd find my server hadn't really been running for a while. I did internet research and tried to figure it out. I updated the BIOS which actually removed the primary option that used to help. At one point crashes every few hours were common. I tried buying a brand new motherboard, with a newer chipset and theoretically better support. It was worse: it wouldn't even stay running long enough to boot regularly, much less last for a few hours after that. (So I returned it, which I generally hate to do.)
Long story slightly shorter: I could get it stable, but only by completely turning off power management ("Cool'n'Quiet") in the BIOS and running a busy-loop job just to keep the CPU slightly above idle. (Again, internet theories: it's something about waking up from a completely idle low-power state that fails, so keeping the CPU busy prevents it from going into that low power state.) Just not good enough. And that nice "real server" I had built not to long ago was nagging the back of my mind...
Story
So I had a server that at least didn't crash regularly, but I was no longer happy with it. I decided it was time to build a second "real server". I took more time and put in more planning though, this round. I was mostly set on Supermicro again: first, using the same platform would make things easier in the long haul: same tools and features on both servers. Second: they're pretty common. There's enough used stuff out there to be nice and affordable, too. I looked at a wide range of boards, spanning the X9 (like the one I built last year) to the X11 (newest, as far as I can tell) model lines. The newer models never seemed to hit a better price/performance trade-off so I started looking more closely at the X9 boards. They didn't, as I first suspected, all build around the same class of processors, they varied. Within that bunch, the LGA2011 type socket chips seemed the best, and I made a spreadsheet with a chart:
Which shows the CPU Mark rating for all the LGA2011 processors (that seemed commonly available) compatible with the X9 boards I could find charted against the price (lowest stable/believable price on eBay). There's a few excursions above and below the trend line, but not really far.
What I had to decide was how much performance do I really want? I spent around two years with a Ryzen 5 1600, CPU Mark 12455 (plenty for my needs) and a few months with a Ryzen 7 1700, CPU Mark 14566 (even more!). Very few available processors were in that range. But there are dual processor boards available!
I began trolling eBay for a good deal on either a single powerful chip or a dual processor board that could house two more inexpensive chips. Around this time I learned that there's the extremely standard ATX motherboard size, a larger E-ATX size, and a yet-larger EE-ATX size. The latter is extremely non-standard. I was looking at the X9DRi-LNF4+ board for a while, but it's that EE-ATX size which is really Supermicro-specific and only works (well) in Supermicro's own (expensive) cases. Next I found the very similar X9DRi-F &emdash; same line, slightly less feature packed, and just "E"-ATX. I looked around for a while and found a real auction (not buy-it-now) item on eBay, and I won it! First part selected!
What followed was building the machine around that: picking the CPUs to go with it: two Intel Xeon E5-2650 v2s at 10006 CPU Mark each, 64 GB of ECC memory (4 16GB sticks), a brand new power supply (with two CPU power connectors) and other miscellaneous parts.
Aside: part of the reason I tended towards the dual processor board was the memory. In last year's build I accidentally bought "registered" ECC memory, or RDIMMs. Support for registered memory is mostly limited to servers and a few other "high end" computers. It's not in as much demand, so it can end up cheaper. (That's how I picked it at first: it said ECC and everything else seemed OK and it was the cheapest.) This time I already had the motherboard and I knew it supported registered memory (I think all dual processor boards do, they tend to have more RAM slots and the benefit of the registered memory is it can consume less power (I think), and work better when there are more individual sticks of memory sharing it all.) Either way this time it didn't matter, the prices were all very close, registered or not. Oh well.
Plus a case to put it all in. With hindsight I can now say that it turns out E-ATX isn't a standard, either. At least not to the degree that ATX is. If you buy a motherboard and a case that both say they work with ATX, they're virtually guaranteed to work together. Not so with E-ATX. Not only did I pick this unusual motherboard size, I also have several (redundant) disks that I want to put in the case. And despite picking server class hardware, I want a PC-like tower. All the real (rack mount) server cases I've ever seen have loud fans in them, and this is going to be in my studio apartment, where I sleep.
I looked around for a while and selected the Antec P101 Silent. Looks great. Eight side-load disk trays &emdash; enough to hold all my disks and makes for easy install and replacement. Motherboard compatibility list says "E-ATX". Great. I ordered it. I got it. As soon as I saw it, it became clear that in order to fit an E-ATX motherboard, you need to remove six of the eight drive trays. Not great.
I had patiently and slowly gathered all my parts, but now they didn't all work (together). The seller didn't reveal this fact and neither did the manufacturer, in any part of their pages or the manual. I tried to return it and was initially quoted a $47 return shipping fee, for a $110 or so item. My heart sank. After some thinking, I got the tiniest bit creative with how I filled out the return request form and got it to agree for the seller to pay return shipping. Along the way I selected "store credit" instead of refund. I don't know if that helped, but I felt like it did.
Then UPS and Newegg conspired to hold my refund hostage for over four weeks (July 16 to August 12) before finally accepting and processing the return. By then the replacement model I selected had gone up in price by over $25. Sigh. But I'd been waiting a long time and had already locked up over $100 in store credit so I just went with it. I got a Fractal Design XL R2, which in the meantime I had reached out to confirm it will really fit my motherboard. (Mostly! See below.)
The Build
The last detail is that by now I had learned how not-quite-standard E-ATX was. I had found other hobbyists like me, building machines around similar Supermicro boards, and wanting to use normal PC cases. Some but not all of the mounting holes line up, and in one of the discussions I read somebody called out how important it is, in a vertical tower case, to support the weight of the CPU coolers well. So all this time waiting, I had done nothing, because step one was to custom mount the motherboard into the case, and I needed to wait for the case.
The board has ten mounting holes. Six of them lined up with factory holes. Four I drilled out myself: Installed the bare board with the six good mounting points, mark where the remaining four holes should go, and then simply drill a hole just big enough to allow the standoff's threads to fit through. Screw the standoff into a standalone nut, rather than the threads built into the case at the factory. Worked great, just had to spend some quality time with a file to get two of the four holes to line up with the rest. Look closely and you can see the four shiny metal nuts in the picture above, on the black background of the case. Look even more closely and you can see five of the much smaller shiny points where the standoffs fit through the factory holes. (The sixth is obscured by the cabling.)
With the motherboard now securely installed (the screw heads are black and almost disappear in the photo), next was to install the CPUs. Another tiny saga:
The processors I bought were these, as pictured. I thought that blue thing was some sort of special shipping/protective case. So I biased towards listings including them. It's not. Everything's fine now, but I had to wince as I yanked the blue plastic piece off, it was glued on. It's for some (maybe vendor-specific, like for HP?) slightly more specific variant of the CPU socket, that I don't have. Ok. CPUs installed. Good. Next is the CPU coolers.
They didn't fit.
I got a pair of CPU coolers made to fit several different CPU sockets, you're meant to pick and choose the right parts (among several available) to fit the socket you're using. There's some pieces labeled either "Intel" or "AMD". And I've got an Intel CPU so the choice seemed obvious, but it was way too big. And the AMD one had two pair of holes, either a little too small or just a little smaller than that. I found the data sheet, which clearly calls out a mounting hole spacing of 80 mm square. Mine aren't square.
Turns out there's more than one LGA2011 socket, and I've got the Narrow ILM ("Independent Loading Mechanism"). Which I learned from a page using a very similar board as an example: a smaller form factor socket so that (e.g.) two can fit on a single board!
I can solve this. The AMD piece was almost the right size. So after a little time with a drill and a Dremel, I had added my own extra holes. Each piece ends up tilted just a bit, but by keeping the offset symmetric everything still lines up well enough to work. Phew!
So I'm done, right? Plug in the memory and turn it on! And it doesn't boot. Does display a code on the screen. Which indicates bad memory. After extensive testing I found that one of the four sticks of RAM I bought is no good. At least enough to prevent booting the machine: the other three each as the only installed memory work fine. The fourth: no go. (Still waiting for that replacement, the seller has been very non-helpful, but eBay claims a replacement is on the way. Return shipping is .. unclear.)
Done
Mostly. Here's the new and old machines next to each other. It's hard to capture in one picture, but (to fit the bigger motherboard) the new one is a few inches taller and deeper (and happens to be subtly wider, too).
I'm waiting for the last of the memory to arrive (either way it's a big upgrade over what I had before). And there's an additional "Storage Controller Unit" with four more SATA connections that I'd like to use but haven't figured out yet. But otherwise it's set up and working! The story is long without the software-based issues I had switching from an AMD to Intel processor. But it's basically done!
Cost
As mentioned, this was mostly assembled from used parts. I spent (including tax and shipping in-line):
And I transplanted my existing disks in. Total: $753.27. I'm happy with this; buying a standalone remote management tool is possible, but for some reason they seem to start in the $400 range, used. I got that plus a nicer server upgraded in just about every way.
I mentioned a while back that I was experimenting with setting up a TV antenna to enable me to stop paying for cable yet still have access to some of the shows I enjoy. Since then I've been continuing to think about it. I got a HDHomeRun tuner device and set up Tvheadend on my server. This allows that computer to act as a fully featured DVR. What that means, though, is wanting a stable antenna set up that always works, and gets every channel at the same time.
As I said in my earlier post, I enjoy plenty of the programming on the Quest channel. But of course it's the one that's hardest to receive. It's especially hard to receive without losing reception on plenty of other channels. I started to think about setups to get multiple antennas and combine them with filters, so one would pick up (only) Quest and one would pick up the rest. This is possible, but complicated and expensive. As I was looking into it, I found a great article titled merging feedlines. It made me confident enough to take a shot at a very cheap and easy solution: just feed two antennas into a "splitter" (which can also function just as well as a combiner), and plug that into my tuner.
It works great! I found this one tiny particular area on the wall where Quest comes in strong (really, two or three feet in any direction ruins the signal!), and most of the other channels come in well in plenty of convenient spots, so I hung the second antenna in the corner of the window (normally covered by the blinds) with a couple of binder clips. The first antenna (except for the cord hanging down) is hidden behind a painting and mostly supported by two nails at the bottom.
At this point I've got all the major three-letter broadcast stations, a couple smaller affiliates, and Quest. With a functioning DVR setup, through the computer. I'll continue using it for a while to confirm that I get a consistent and solid signal, that watching those recordings is convenient, etc. But it's starting to feel like I should really divert funds away from the cable company.
With pandemic and lockdown forcing a work from home scenario, I'm spending a lot more time at my desk in front of my computer. Very shortly after that started, I started feeling a bunch of upper back pain. I was pretty sure it was down to the instant switch from near-100% full time standing desk at work to full days of sitting desk at home. (I've got a really nice desk, but it's a standard desk that isn't set up for standing at.)
I decided to get myself a new nicer chair. It's been a few years since I went almost standing only at the office, but I spent plenty of time working in a chair before that. And my "office" chair at home is both several years old and from the lowest quality tier, before all that wear.
I can confirm that the chair was mostly to blame, because it took a long time for the replacement to come. The old chair has fixed arms, which form part of the support structure for the back. I took them off and flipped them around, so they still held the back up, but they followed the shape from seat to back, rather than rising up to make a potential armrest. Being fixed, they were at a fixed height. One that was much too high. I was scrunching my shoulders to fit around them.
That said, this chair took a full month (ordered April 14, arrived May 15) to show up. I have no idea whether to consider that as awful as it seems, or to give a lot of slack for the current global situation. I definitely don't appreciate the completely opaque communications of the seller. First I reached out because the passed "14 days" long end of the estimate turned out to be 14 business days (which feels intentionally selected for misinterpretation), not passed. Once that date finally slipped, they gave another several-week window for possible delivery, but nothing more.
Now, it was a great deal. I paid $324 (plus $34 for carpet casters), which is well under half of the $900 for a brand new one. It was listed as "open box" but this is clearly a used/refurbished chair — plenty of light scratches, but all on the sides/bottom, so I can live with it. The price where I bought it has gone up to $374 in the intervening month. Only since then I learned about National Office Interiors and Liquidators, who are currently listing the same chair (but with the full-option arms, mine only raise/lower) for $299 (though only in grey, every other color is $399!). And more honestly listed as condition "used".
Either way, this is a quality chair, and I'm already happy with it.
I haven't used my 3D printer in quite a while. This is the time, it's tough to get things these days. I'll save the long story of many failed attempts and tweaks to get it working again, but I just have done that.
I've had an extra drive in my home server for a while, since I upgraded the main set from a three-drive single parity to four-drive double parity setup. And nowhere good to put it. It's the remote backup for my Mom's files, on the server I keep at her place which is itself the remote backup for my own files. It's been awkwardly perched, upside down, on a piece of cardboard. I've got four unused 5.25" bays in that case, so it should go there!
I found a nice, compact design for brackets to fit a 3.5" drive in a 5.25" bay online and printed a set out. My printer makes "hairy" prints, with fine strands everywhere, due (I think?) largely to its design. And at least while I'm using up the ton of free ABS filament I got, bed adhesion is tough so I tend to print with a brim. First step was quickly cleaning up all that cruft, which is shown in the second picture. They're not pretty (I picked fast rather than high quality print settings), but they'll work.
Next, I drilled out all the holes. Half large enough for a screw to fit cleanly through, so they could screw into the drive. The other half exactly the right size to grab the threads of the screws, the outside holes are for holding the whole thing into the case. I did a test fit (the third picture) and it worked well enough, after a little more adjustment with the hole position. Then it was straightforward to pop the drive in a proper location, and screwed down!
In that last picture, you can see one of my main drives, currently tucked into the floppy drive slot, so the mounting holes don't line up well. I might make a second bracket to hold that one, too.
In late 2017 I broke and replaced my server computer at home. It was in a partially working enough state that I could wait a while, so it was mid January of 2018 when I finally bit the bullet and bought parts to assemble a whole new computer (except the drives, which I kept). At the time the best bang for my buck was an AMD Ryzen 5 1600 CPU, along with a compatible motherboard and RAM, and other assorted things.
Having just been through a small nightmare to get the server to work fully again, I was very happy when that was done. There were a few small oddities. It would crash when idle, which turns out to be a strange power management thing. And more rarely it would fail under load as well. It took until just last month for me to realize that segfault under load is a known failure for (early?) Ryzen CPUs.
So I contacted AMD, on the 19th. The RMA request was approved and I shipped this, pictured, CPU back to them. The second line of text there UA 1746PGS tells me, among other things, that this was from 2017, week 46. That was a little worrying — only "early" 2017 CPUs are supposed to have this issue. According to rumors online. But I had to take everything apart to get this picture to know that fact, so I was nearly committed anyway. I delayed shipping it until the 27th. It was clear that I'd have to wait for shipping in both directions, and processing in between, with no CPU in my server — which holds all my files! I was seriously thinking about buying a new replacement no matter what, and selling whatever they shipped back to me, later.
In researching what exact replacement I would pick up, I discovered both that Microcenter is open (despite global pandemic), and that they had the best priced option. I came very close to heading there to grab one the weekend before I shipped my CPU away. But I realized: I can go visit them any time. I can be patient, and waste money on a new and extra part only if and when it becomes desperate, without much delay. So I took my server apart and shipped the CPU away.
I had the tracking number, which I watched eagerly. It was delivered on the 1st of May, according to FedEx. On the 4th I get an email that the "return processor has successfully passed the inspection and your replacement product is now approved". It's been a full week, but I guess that's good news. On the 5th I get a message that the replacement was shipped. And that's it. No tracking number.
Today, the 11th, the replacement arrived. To my surprise, it's a Ryzen 7 1700. I paid $200 for my CPU just over two years ago. If I look today, the price seems to be $250 (it might be limited old stock making pricing strange, this is an older model now). The newer/better 5 3600 costs just $130. The replacement costs $320 today (the 3700 is also over $300). It's got a slightly lower base clock speed (3.0 vs 3.2 GHz), but eight instead of six cores. And it came in a retail box with new cooler and everything. I was not expecting an upgrade, but for multiple weeks of doing without my server it's a nice perk to have ended up with.
