2018-03-24 21:33 - Making
For months, stalled by a partially broken computer, I've been working on this project. I have a server at home (and another for remote backups, at a relative's house). And I've got full disk encryption which needs a password to unlock, at boot. Which means if it ever reboots I have to physically be there to get it going again. I'd like to be able to administer my servers remotely.
I've looked into commercial IP KVM devices, but they cost hundreds of dollars. Since I'm working with Linux here, in theory all I need is a serial terminal. So I've designed a serial-to-WiFi bridge. The picture above is the first one that I've ever had working, installed. The ribbon cable hooks up to the internal serial port header, the green terminal plugs into an unused USB port header for power. Then there's snaking trails of several other pairs of wires: one each hooking to the case and to the motherboard for the power LED, the power button, and the reset button.
Most newer motherboards power their USB ports all the time, even while the computer is off, this one included. So I can remotely power up or down, restart, and then control the computer. In theory. I've just gotten far enough to test all this, and discover performance issues. I've got all the computer-side setup to manage, yet. But after working on this since August, it's great to have it finally proven to really be workable.
Git provides a great bisect tool. When something has been broken, it will automatically help you find out how and why, by bisecting (splitting into halves) the source control history, incrementally narrowing down on changes. I've sometimes got situations where I'd like to do a similar operation on something else that's not git history, but is instead numbered. A while ago I threw together a python script to help me do this:
#!/usr/bin/env python
def main(argv):
if len(argv) != 3:
print "Usage: bisect.py <good> <bad>"
return
good = int(argv[1])
bad = int(argv[2])
while True:
if bad == good + 1:
print "First bad:", bad
return
totry = ((bad - good) / 2) + good
inp = raw_input('At %s [g/b]: ' % totry)
if 'g' == inp:
good = totry
elif 'b' == inp:
bad = totry
else:
print "I don't understand."
if __name__ == '__main__':
import sys
main(sys.argv)
Save this in your path, mark it executable, and run it with two arguments: the (smaller) number which was good and the (larger) number that is now bad. It will find the midpoint, you test that number and tell it good or bad and it repeats until you've tested immediate neighbors, indicating the first number which was bad: the one that introduced the problem.
2018-02-20 14:31 - Making
I recentely learned about the tangzhong bread making method. I pickeda recipe and decided to try it.
So I made the dough first of course. Then, I didn't know where my Mom kept her bread loaf pans so I used a Bundt pan instead. And I don't know where the pastry brush is, so I didn't do the egg wash. But it still came out quite nice. Very dense, and certainly tasty. Worth trying again with a few more details "right".
2017-12-30 18:55 - Gaming
As mentioned, I converted my button tapper machine to also test the buttons being tapped. Since that I revamped it to get slightly more accurate readings, and now I have interesting data to report. Beware the wall of text, but delight in the pretty graphs in the middle.
Methodology
See the linked post for a description of the raw data format. For each switch type, it's 250 events each pressing the button down, tracking the state of the switch over time, releasing the button and tracking again. All real physical switches will bounce as their contacts strike each other and spring back, only eventually settling to really be closed (or open). Each particular (model of) switch will bounce more or less, based on how it's built. With the raw data in hand, for each switch and event (press or release), I measured a virtual point at every microsecond, counting for each of the 250 tests, was the switch opened (0) or closed (1) at that point. Then take the average closed-ness, at every microsecond, for each switch and event.
So in the graphs below, a line at the bottom (0) means the switch was always open at that microsecond, and at the top (1) it was always closed. Anything in between means it was sometimes open, sometimes closed, at that microsecond. Across 250 tries, this averages out to a nice visual representation of how quickly, on average, the switch closes or opens, and how it acts while doing so.
The Switches
| Make | Model | Nickname | Activation
Force (gf) | Cost | Notes |
|---|
| Omron | D3V-01-1C22 | 1C22 | 25 | $1.39 | I've recommended these, before doing such deep research. |
| Omron | VX-01-1C23 | 1C23 | 49 | $3.74 | |
| ZF Electronics (Cherry) | D449-R1AA | D449 | 76 | $4.20 | The ZF Electronics corporation bought Cherry in 2008. |
| E-Switch | LS ??? | LS | 100? | $1.72? | This one will take its own section to discuss, see below. |
| Honeywell | V7-1D10D8 | V71D | 150 | $3.65 | Most players feel this is too stiff. |
| Honeywell | V7-3A17D8 | V73A | 50 | $3.03 | |
| Honeywell | V7-5F17D8 | V75F | 15 | $3.79 | |
| Zippy | VM05B-00D0 | VM5 | 20 | $0.85 | These are not "Zippyy" brand like the joysticks. |
In alphabetical order, by nickname. Nickname is the short form I'll use to refer to each of these switches, going forward. With the nickname, you can look above to see exactly what I mean, but hopefully otherwise avoid bias ("I'm a fan of brand X!"). Activation force is the "gram force" required to activate an idle switch. Cost is the price each, to buy quantity ten.
The LS switches were mixed in my pile of spares, where I save even bad/removed ones (sometimes the next bad one is worse, and you've got no good spares, and you don't want to wait for shipping for a replacement!). Most are visibly used, stained (drinks/spills?), which shows easily on their white plastic body. Comparing limited known properties, I'm guessing this is a LS0851500F100C1A but I could be wrong there. I left question marks in the table, because I have to guess at the various values as well. As best I can tell, actuator 00 is the bare pin (definitely matching this switch), and that comes with a minimum 100gf activation.
Also worth mentioning is that the VM5 switches were described by everyone I can remember as being unsuitable for play, immediately upon installation while brand new. We'll see why shortly.
Button Presses
1C22
1C23
D449
LS
V71D
V73A
V75F
VM5
(Hover above links to isolate their graph.)
There's eight switches on the graph. To my eye there's an obvious cluster of four near the left (don't miss the yellow one), three in the middle, and one outlier off to the right. A theoretically ideal switch would be a vertical line straight up at microsecond zero, but we're in the real world. My testing machine takes time to charge up the solenoid, time for it to move, and time for it to overcome the force of three springs (one built into the solenoid, one in the button housing, and then the one in the switch itself). That's the twelve-ish millisecond delay before any of the lines on the graph start to climb. And then real world randomness means that each trial happens at a slightly different time. That and the randomness of the bounces determines the slope of the line. Being closer to that theoretical straight-up line towards the left of the graph indicates a "better" switch. So how close does each one get?
The most critical thing is really the moment where the switch is done bouncing and is always closed. So I'll go in the order that I see them touch the top of the graph. First is the dark blue line (the V73A) which fully activates (here, this means time from the earliest to the latest reading across all 250 samples -- worst case) within 4.5ms of the first moment that it starts. Next is the yellow line (the V71D) with a nearly identical slope, and right behind it is the green line (the 1C23) which takes about 5.5ms. The light blue line (D449) finishes out this group at 6ms.
For all practical purposes, these buttons are identical. We'll have to look at some minimally processed data to tell them apart. Three of these bounce only two or three times typically (seven or eight worst case), only the V73A is bouncier. They all typically finish bouncing in well under two (or one!) millisecond, and the worst case (individual sample) is still under three milliseconds. They're all fine buttons, with the V73A being slightly towards the bottom of this pack.
So for the group in the middle: Purple (1C22) is the first of these three, taking 7.5ms. Red (V75F) is next with orange (LS) both taking 7 to 9 ms. A second glance at the data shows however that these three bounce many more times, with worse average and maximum single sample times bouncing.
Finally there's the outlier which forced me to zoom the graph way out: the VM5. Excepting the worn-out LS, it's a full two or three milliseconds slower to start the process of switching. But it has a timing spread from there of another 35ms, with lots of bounces and a 24ms worst-single-sample time to settle. This one is completely unacceptable. A simple "wait 20ms and then see what state the button is in" approach (which is exactly the sort of thing a simple debounce system might do!) will accidentally read the not-pressed bouncing state sometimes! This lines up perfectly with the real play experience with these switches: they're just not reliable enough.
So can we use data to explain where these lines fell on the graph? I think so! The first "good" group of four have activation forces of 50, 150, 49, and 76. The other four have activation forces of 25, 15, 100 (worn out), and 20. Again excepting that one worn out model, it's clear that the higher weight springs produce a more reliable button.
Button Releases
1C22
1C23
D449
LS
V71D
V73A
V75F
VM5
(Hover above links to isolate their graph.)
Releasing a switch is much easier than pressing it. So all switches tested have significantly fewer bounces on the release, and activate more quickly. There's not really much more to say here. This isn't an important criteria.
Subjective Experience
I installed all these switches into the local arcade cabinet and asked players to let me know what they thought of all of them. They all played okay, so there was only one consistent bit of feedback: despite one player who loved it, everyone else commented negatively about the 150gf V71D. So despite a heavier spring producing better looking data, we know that we can go too far.
Results?
The one thing to say for certain is that the model of replacement switch matters! Any name brand should do fine, and 50-100gf activation seems to be the sweet spot. Optimize for cost within that range. I'm interested in trying the LS in a non-worn-out state!
2017-12-25 20:24 - Making
Just in time* for Xmas, I put together this custom garbage bag holder for my Mom. Following availability at local grocery stores she's switched to plastic bags, and been awkwardly using them on this built in garbage bin, where they're too small to fit properly. Here's a mostly plywood contraption to fit over the existing bin, but with pegs to hold the bags' handles plus some cove molding to help keep the bag open and in place. It's about the same depth but narrower than the bin beneath it to be the right size for standard plastic shopping bags. A cutout in the lower layer of plywood fits around the lip of the bin, while the top layer holds all the bits mentioned above. It's all made from found materials already around the house, which is the perfect kind of gift for my Mom. No new/extra stuff!
* I definitely had an idea to do this over the Thanksgiving visit. And earlier in this visit. And forgot, despite a coded (to not give away the surprise) note to myself. Then I thought of it again, and realized what the note meant, and built this up over (mostly) a long afternoon on Xmas Eve.
2017-12-11 12:23 - Gaming
I started this game over Thanksgiving week, and just (this weekend) finally finished it. Some boss fights were a bit frustrating, but overall I found it to be a great and enjoyable 2D Metroid game.
2017-11-16 22:19 - Making
I've never posted here about this device I made. I made it earlier this year, mostly as a joke. On the right, now mostly hidden by the fan, is a solenoid. It sits poised above a yellow arcade button, also mostly hidden by the fan. On the front of this view is the electronics and the controls are on the top, all glued to a huge 12 volt lead acid battery.
The original point was to be able to press that button with inhuman speed and accuracy. I managed to leverage that to contribute to some of my videos about Killer Queen. More recently I've been interested in figuring out how to pick good arcade hardware. The moving parts wear out and need replacing. Which ones to buy as replacements?
So I updated the device, not only can it push the button — normally the one in the real arcade, now (in the extra little bit of wood it's sitting on) a dedicated one — it can read its value. So it can be used to derive fine details about the performance of the button, when pressed. I want to do quite a bit with this. Here's one example:
ED 0
D 12272
U 12280
D 12284
U 12540
D 12832
U 12836
D 12844
EU 0
U 8364
That starts ED 0 for "event down zero". Then a bunch of lines D and U indicate the timing of "down" and "up" events observed, in microseconds since we started the "down" event. All real mechanical switches "bounce" a bit when activated (and deactivated, usually). In this case, the button first went down about 12 milliseconds after we started (it takes some time for the solenoid coil to charge up, then to physically move) and the switch contacts bounced apart and back together three times after that, taking just over half a millisecond to settle down into its final resting state. And in this case when it came back up, there were no bounces. In my limited experience so far, this looks like good performance for such switches.
This is just one bit of data but we can compare quality in the "bounciness" category pretty easily: how many bounces, and how long before the last one is done? I intend to follow this with a post containing a detailed summary of this, for several switches. I need to make a final call on exactly what I want to measure, and make sure my tool will do that if at all possible, so it will take a little time.
2017-11-11 23:10 - General
I've got a story to tell, about my broken computer. It's actually still somewhat broken, but I've just climbed up out of a valley that so deep, it feels almost like it's fixed. Here's the story, but be warned: it's probably too much detail.
So, I keep a server at home. It's multi purpose. It stores my important files on a redundant ZFS array. It plays some of those (media) files on my TV. It runs and exposes various small services. It's deeply important to me given all these things I use it for. In addition to that local disk array, I have a remote backup of the data on a similar machine at my Mom's house. Which has an extra disk for her data. And I've got an extra disk for a remote backup of that.
That last fourth disk in my server is a relatively recent addition. Around when I put it in, I noticed that the drives all stuffed next to each other get a bit warm. I decided to install an extra fan, to keep them cooler. This started its own sad story. There's an unused "case fan" header on my motherboard, yay. I have a spare compatible fan, yay. It doesn't support speed control though, and is far too loud to keep running in a studio apartment. So I found a bigger fan, which supports speed control. Got it all set up, figured out how to set the speed, and I can tell even when running slow enough to be effectively silent, it still moves plenty of air. Great! So I screw it in to place, get ready to call it a day ... and discover the cable isn't long enough. Long story short, I probably crossed a wire and zapped something while trying to extend it to reach. It doesn't go anymore. Not sure if it's busted, or the power connector. But I find yet another fan, this one uses not the fan connector but just the standard power accessory (i.e. old IDE disk) connector, and is designed to be quiet. Great!
But doing all this effort to get a working fan installed involved opening up the computer, moving things around, fiddling with them... And as I said quiet is nice when you only have one room. I noticed a little noise, it seemed to be a fan (new fan, CPU fan, power supply fan?). I decided to use my standard technique of (briefly) stopping the running fan by jamming something into it to narrow down where the noise was. Wasn't the new fan. Wasn't the CPU -- but this one ran fast enough to hurt my finger a bit so I stopped using that. Popped a screw driver into the power supply fan and WHAM. Broke one of the blades off of it. Ugh. Had to open it up a bit to move the fan guard out of the way to un-jam it. Everything still worked, but I had never completed my earlier goal of figuring out where the minor noise was coming from. Everything was still off, so I put the driver back into place, intending to block that fan from spinning while powering it back on.
ZAP! I shouldn't have used a metal screwdriver. There was a spark and a pop. And I busted a fuse. And I was scheduled to leave on a flight at ten AM the following day (this was the night of October 4th). Stomach in knots. I managed to take the power supply out, take it apart, and find the fuse, confirming it was blown. And soldered in. I have an old power supply lying around that was supposed to be for a project that never came to be. Open that one up. Its fuse is soldered down too, but compatible. Remove it. Remove the blown one, replace it. Put it all back together. Plug it all in. It turns on! Everything shakes a tiny bit, as the fan with a missing blade spins, but it turns on.
But no matter what, it doesn't show anything on the TV anymore like it used to. Unplugged a monitor from my desktop to carry it over, and it won't show anything from any of the other connectors, either. Stomach drops again. Before long, I figure out that everything but the display works. If I power it on, wait patiently until I know it's asking for a password, type it in blind and wait again: it boots. It responds on the network, and so everything I use it for still works -- except playing things on the TV.
So I leave it be, fly out for my trip and eventually come back. I'm pretty patient here, but I know I've got to do something. The first thing I do is replace the power supply. Really I just need a fan, but just the right one, which won't be easy. They're not too expensive so after $25 and a few days I have the replacement in. It doesn't have an off-balance fan, but otherwise it doesn't help, still no video. What to do? By lucky coincidence (this might have been earlier...) I have a spare identical video card, so I swap it in. Still no display. So I order a replacement motherboard, wait for it, laboriously swap all the components over. Still no display. So I return the motherboard I don't seem to need. (And take a $20 hit in return shipping/restocking fees. Blech.)
So what can be left? I know sometimes the motherboard's built in video is really controlled by the CPU, so I order a replacement CPU. While waiting for that to arrive, I make a stupid mistake. Right now, the computer is functional, but I can't see its display. If something goes wrong, I can't fix it, because I can only use it in a working state, when it boots and I can then remotely log in. Somehow I forgot all this and in an otherwise idle moment, I started an update for all three of my similar machines (this one, the backup at Mom's, and the public hosted server I run). Of course, of course, this time something goes wildly wrong on this machine. Some shared library that everything links to is hosed, and I can't run any more commands at all.
That was three days ago, the 8th. Yesterday I got the replacement CPU, but it didn't help because I can't boot that machine right now. Today I traveled into Brooklyn for the Killer Queen Coronation competition, and I had a few spare hours afterwards, and I was already close to Micro Center. I was at the point that I was ready to throw money at the problem. Just buy enough replacement parts that I'll surely end up working, and put this stressful mess behind me! Well, once I get there I realize I'll be over $300 in the hole for a new motherboard, CPU, and memory.
But I've already replaced the motherboard, that didn't help. I've replaced the video card which didn't help. What is going to help? I'm getting less confident that this expensive solution is going to be it. And it gets worse. Remember that ZFS array I mentioned at the top? It's encrypted, so I need a separate boot partition. On a separate drive. Right now, that's a Compact Flash card in an IDE adapter. Which sounds crazy, but works great. Except new motherboards don't come with IDE connectors anymore, so I'll need to buy even more something to make that work, and I'm starting to really doubt myself. And with no display how will I set up that replacement? What now?
Here starts the silver lining. I figured out a temporary fix that I was confident enough in that I gave up, left the store empty handed, and headed home to try it. I took my main desktop computer mostly apart, plugged in the server's drives instead, and booted a USB rescue environment there. And it worked, I could mount the disks. I'm happy I put my root partition on the ZFS volume, because now that I could finally boot "the server" and see its display, it was trivial to do a snapshot rollback. Took everything apart again, reassembled the original server, booted it blind, and voila! It's running, sans display, again. I can remember not to update it and patiently figure out a reasonable long term fix. Phew! I'm actually only right back where I was a month ago, but with the whole thing broken for a few days, this feels quite relaxing in comparison.
