Justin Lloyd's Thinkering

Somebody needs to think about this stuff...

How much cache you got on you?

by

Holy crap that’s a lot of requests per second. 40 million per second, more than 2 billion in a minute, on the test hardware.

 

 

Have managed to make the back-end never hit the database for any reads except during start-up, and all database writes go through a WAL queue rather than directly to the database.

The requests are a mixture of request/response objects (which contain DTOs and collections of DTOs in the response) and media such as images and short videos. I don’t know how close we were to maxing out a dual 100gbit NIC — it is some exotic dual 100gbit NIC with four 25gbit ports on each NIC from my understanding, not sure on that as I haven’t physically seen the hardware with my own eyes. But there was still plenty of bandwidth to spare, and the quite a bit of CPU to spare too.

Now I will admit, these are synthetic tests, i.e. traffic replay of earlier live sessions, coming from another machine, without a bunch of network hardware in between, so I suspect real-world production usage will be considerably lower.

Configuration is:
MS SQL -> WAL cache -> Kestrel -> In-Memory Query & Object Cache -> NGINX -> Apache Traffic Server

Technically the Kestrel server application is doing its own in-memory/in-process object and object collection caching so that I am not dumbly relying on external caches to do the heavy lifting for me. The Kestrel server application is effectively just a cache filler once it gets going and has answered the initial burst of queries and built internal collections. And the only time Kestrel gets queried (after that initial burst) is if something hasn’t migrated to a more forward position (higher tier?) cache.

 

Update:

I have since moved from synthetic replay tests to production requests and our numbers ticked up a bit due to the way we shape our traffic so we’re just cresting 42M RPS, or around 2B+ per minute on a single machine.

I cannot go in to a lot of detail on implementation. But I am very happy with the throughput.

There’s still a little headroom, but I suspect I am going to be hitting limitations of the OS, network stack and drivers at that point.

I feed everything through eight separate 25Gbit connections, though tests have shown that I could get far better results by feeding through 32 x 10Gbit connections. Results better than “yeah, but 320Gbit is more than 200Gbit” would suggest. This would be due to packet offload, PCIe bandwidth, RAM <=> NIC DMA constraints, etc, etc.

The majestic monolith literally handles everything, from media to database to server. There are two other containers for logging and housekeeping.

I focused on what can I get out of the hardware. I optimized at each layer, which was the low-hanging fruit, then optimized the more esoteric parts, e.g. read-once/WAL database access pattern, automatically denormalizing data once it enters a caching layer, storing cached data in different data structures depending on its usage, deduplicating (but not normalizing) data where it made sense, updating in-memory cached objects on database writes, deferring all database writes to post-processed batches (housekeeping container), making the client smarter so that it didn’t ask for complex joins and other operations that couldn’t be cached, and so on. Not trusting the client, but assuming the client was smart.

There were a lot of bottlenecks that I was able to kick out by being given carte blanche to “fix it, I don’t care what it takes, make it so we can hit reddit numbers.” And then finally just tuning the hell out of the OS, the Mellanox cards, and so on.

Organizational Obsession

by

Productive day of organizing tools during some much needed R&R. Hadn’t take a day off in eight months. Two-/three-day weekends belonged to other people.

On a side note, I like the concept of Kaizen foam, I don’t like the implementation.

Cave of memories

by

Years ago, 45 years ago, I cut my teeth (programming teeth) on what are now very old microprocessors. The MOS (later Commodore) 6502 (not so obscure), Zilog Z80 (not so obscure), the Commodore 6501 (kind of obscure) and the MOS 7600 (not so obscure).

There was a question on Hacker News a while back about the 7600 and I was sure it was a CPU I had written code for. I was a little fuzzy on the details, recalling how the paddle inputs worked from a different game console.

Single channel sound, four outputs to control an RF modulator, four joystick inputs and a light gun input. Back then, CPUs weren’t exactly what you called sophisticated! No interrupts, no NMI, no external RAM, no real internal RAM, two X & Y registers, I don’t think we even had indirect addressing, it was uphill, both ways, in the snow.

My memory is a little hazy, but I think the 7600 was legitimately the second CPU I learned to write code for, and probably the first CPU I got paid to write for. I recall it came in a few different editions, with hard-coded programs in an internal ROM and this is where my memory gets really bad because I am not sure if I am confusing the 7600 and another CPU, but one of those CPUs had this weird piggy back socket on it to which you could attach an EEPROM, or an ICE, and it would let you run your own code compared to what was built-in to the masked ROM on the microprocessor. There was also a machine that had the EEPROM window that you could wipe with a strong ultra-violet light, and again, I don’t recall if that was the 7600 or another CPU. I don’t recall if the ROM on the microprocessor of this special edition had anything in it, or it was the standard edition mask with a special package. I seem to recall the different 7600 models had different graphics built-in, and you picked the variant based on what game you were trying to create.

Being a pack rat, and prone to stashing away any useful paperwork that isn’t nailed down, I was digging around for some other reason at our storage locker and in one of my several boxes of memories located old programming books and datasheets I haven’t looked at in decades, and I located an article with schematics for building a game console around the 7600 Video Game Array microprocessor, a service manual for repairing the consoles, a few schematics from old game consoles that used the 7600 and a datasheet with pin outs.

The reason I got the job of working with the 7600, at such a tender young age, was because I worked for cheap, easily exploited in my youthful naivety, and had been hardware hacking on broken 1970’s games consoles of various stripes for about four or five years by that point. That summer holiday job in the middle of the school year lasted all of three months before my Dad said to the proprietor “You have to pay my son if you want to him to continue working for you.”

I never did get my money.

Contortionist

by

“So how tightly coupled is the code?” asked my colleague.

“Our architecture diagram is used as the illustration on page 73 of the Kama Sutra.” I replied.

“Where should we be?”

“Brexit.”

Goodbye dear

by

Goodbye Teithe Din Din / Teith Yr Ewin / Teith yr 拧紧

Always in our hearts.

Teithe Din Din was a poor translation of “Journey of the Nail” or “Journey of the Screw.” It is a combination of Welsh and Chinese. The dog came all the way from Taiwan as a rescue, with a pin in her leg, so her foster family in the US named her “拧紧” which, to a Western ear sounds like “Ding ding” so she became “Ding ding”.

When we adopted her on April 16th 2011, Joyce and I pondered for a while and decided on “Taith Yr Ding Ding” as her name. But when you talk to a vet, they don’t quite get the nuance in the pronunciation, so on the veterinary records she was “Teithe Din DIn” and that just kind of stuck after a while.

When she arrived she was thin and scrawny, having spent almost six months in a tiny, cramped cage – she was our “Size 0 Super Model.”

She ate fried rice, so she was our “Number 37 Takeout Special with Fried Rice.”

She was Joyce’s heart dog, and I got to borrow her on occasion when Joyce wasn’t around.

And then 11 years later Din Din got cancer.

And then she was gone.

And I miss you.

And it hurts.

Don’t pull on that, you don’t know what it is attached to

by

I would say that I spent my quiet weekend casually pulling UTP Cat6 network cables through walls and floor cavities in the condo so I now have Ubiquiti Unifi WiFi Access Points on all four floors and excellent coverage. Loving the built-in PoE of the Unifi Dream Machine Special Edition by the way.

But what I really did this weekend was spend most of it carefully sliding storage organizers of electronics down the hallway, and moving towering shelves of books twenty-four inches to the right‡, shooing my wife away when she noticed the layers of dust on the shelves and books “NOW you need to clean, sort, organize and dust?!?”, fighting cats that wanted to play with the string that I was using to pull some cables “What the hell is it snagged on now?!?!” and swearing at the fish tape that decided it wanted to visit Never Never Land by turning left at the third joist on the right rather than just making a bee line for the precisely drilled hole in the decking. There was also some strenuous debating of whether I wanted to go back up five flights of stairs (for the umpteenth time) to retrieve the flat spade soldering tip instead of the needle nose tip I had in the iron in my hand just to re-solder the network cable splice. “Do I really need to use shrink wrap on this or will gaff tape work? Hmm. Five flights of stairs…. gaff tape it is then!”

There is now a distinct lack of Cat6 running down the stairwell and certainly less of a tripping hazard just so we can have low-latency Diablo 3 gaming sessions.

‡ Except you cannot actually shift an entire 400lb book shelf in one go without doing yourself an injury so it was more like moving three or four books at a time.

Security system

by

A few days ago I installed a new Unifi Dream Machine Special Edition, and hooked up some Unifi WiFi access points to it. I took the plunge and also ordered a Unifi G4 Doorbell Pro for replacement of our current “dumb” doorbell, along with a HDD to store the recordings on. The Unifi Dream Machine Special Edition has Unifi Protect (their security camera software) built-in, so I am going to migrate my security cameras off of the Synology. Yesterday the HDD arrived so I installed that 22TB Western Digital Purple HDD. Quite the capacious hard drive. Waiting for the doorbell camera to arrive, which should be some time today. I was wondering how to go about having a video screen in the home to keep tabs on the front door events, and Ubiquiti has this product line called “Unifi Connect” that is in Early Access, that are essentially touch screens specifically tailored for this very use case.

Routing around the problem

by

Installed a fancy Ubiquiti Unifi Dream Machine Special Edition (it’s a network router) into my home network on Friday. I like it. I’ve liked the Unifi line of products for a long time. Ran some of their earlier h/w (switches, gateway, etc) along with a gateway controller inside of a Docker container to manage a few Unifi PoE APs. The h/w & s/w has been quite robust through the years. That kind of “set it & forget it” reliability I want in my infrastructure and tools.

I held off from acquiring the UDM (Unifi Dream Machine) because I didn’t like the requirement of a cloud account, either to use it in an ongoing basis (since removed) or to perform initial configuration (since removed) or to perform initial setup with a dedicated mobile app (since removed) or to require an internet connection for setup (since removed). Plug the UDM into power, plug my laptop into the UDM via a network cable, open a browser and… go!

Manually copy over some configuration about IPs, DNS and firewall port forwarding rules from an ASUS router to the UDM and it just worked so smooth. Nice and easy.

I am happy to opt-in to voluntary share statistics and analytics of how I use the h/w & s/w with Ubiquiti, but I don’t want to be coerced into “cloud” anything or mobile app requirements to use a tool.

Distance makes the heart grow fonder

by

As my thoughts turn to the holiday season in the US and I contemplate the vast gulf across the Atlantic ocean from where I grew up to where I now live I start wistfully thinking of home, and then reality has a way of bringing me back down to Earth.

I’ve always had a family that has always been very open about discussing money, mostly because we never had any for the longest time and also because the first question many of them always ask is “how much you got on you?” which is a lead-in to a life crisis story from them.

Picture a scene of a traditional, large family, holiday gathering straight out of a Saturday Evening Post Normal Rockwell painting but several people are commenting about the food on your plate because they are secretly sizing up how much leftover food they can abscond with if they discourage you from eating.

Me: “No. No more financial emergencies. No more cash outlay. No more ‘I do not have insurance and it caught fire’ problems. No more ‘I just gotta have this and it is on sale.’ Everyone has tapped in to Lloyd’s bank this year, and the bank is now having cash flow issues, we are closed for business for the foreseeable future.”

The umpteenth family member/cousin/distant relation at the family gathering, half-jokingly saying: “I guess I will take my business elsewhere!”

Me (under my breath): “Oh noes, what ever will I do giving unsecured, interest free loans to people who have no intention of ever repaying me.”

And so, in this season, I give thanks to the family that shaped me into the adult I am, and those small inconveniences that we colloquially refer to as “the pond” (AKA the Atlantic Ocean for those who aren’t from the UK) and the fact that +8 GMT makes it really hard to reach out and call someone “to just catch up, see how you’re doing, and let me tell you about my broken down car that hardly runs” when it costs a dollar a minute to do so.

I’m only half-joking 😉

Self plagiarism

by

I have a little game on my personal C.V. website.

Had someone accuse me of stealing the entire game from someone else.

Then to prove this, they linked to the github repo, which was a fork of my own github repo.

And then after I pointed this out, they doubled-down and came back with the vague “well I’m sure you put something in there that isn’t originally yours.”

And then two days later came back with “So you’re ghosting me? Fine. I guess I won’t be moving you forward in the application process.”

Seriously dude, chill. I didn’t even apply for this position. You approached me, remember?

Unearthed Arcana

by

The past few weeks I’ve been wrangling some knotty algorithm that is to do with Elixhauser comorbidity calculations for the new 2022 rule set in the healthtech startup space. And the code I’ve written is very procedural, and hopefully easy to read, but also kind of like wading through thick mud in terms of understanding how values drop out of the bottom of the algorithm.

Let’s put it this way, this shit’s so arcane and weird that even the SAS code provided by the governing body that demonstrates the algorithm doesn’t adhere to the spec, and that point is even called out in the scant amount of documentation for the code. And let’s face it, SAS source code is bad enough as-is at the best of times, a 4GL language from the 1970’s that has grown warts in its old age, where any program written in SAS winds up looking like a 2,000 line DOS batch file, and I am thoroughly convinced that whoever wrote this particular chunk of SAS source code not only hates his job but also hates every person on this planet that doesn’t have to use SAS to get their job done.

Now once you explain the rules of how the Elixhauser algorithm works to someone, yourself having waded through three PDFs and the aforementioned SAS source code, the algorithm makes sense, in a roundabout fashion, but there’s still these “rules of exclusion” where “if the patient has this comorbidity, but it didn’t present on admittance, then this other rule applies instead.”

I think if I were implementing this multi-stage procedural algorithm all over again, I’d jettison all of my code that I’ve written over the past three weeks and implement the entire shebang as a filter trie, AKA a Bloom Filter Trie, about 25,000 inputs at the trie root, and about 7 outputs as the trie leafs.

I’ll say this, my years of studying the Dungeon Master’s Guide and Player’s Handbook across all the editions of D&D, looking for exploits and rules that could work in my favour, have gone some way to helping me decipher DRG (Diagnosis Related Groups) and Exlihauser algorithms over these past two months.

Auntie Acid

by

My wife is out of town for a couple of weeks. I have fallen back in to my usual habit of surviving on breakfast cereal. Anybody that knows me personally also knows this isn’t how I ordinarily live, what with the several years of culinary school and all.

I bought a fully cooked lasagne which sat in my fridge since Monday last, and me with the full intention of cooking and consuming it over the course of three days.

Well here we are, six days later, and that lasagne needs to be ate.

“Reheat at 325 for 20 minutes.|

Done!

“Serves four.”

Serves four, it says.

And I took that personally.

Can already feel the sodium level in my body spiking.

I regret nothing.

Except maybe for the fact I am out of antacids.

Cannot feel my fingers… but I can hear every bubble fizzing in my glass of sparkling water. Is that normal?

Organize all the things

by

When engineers take up hairdressing…

Saves me well over $300 a month colouring and styling my wife’s hair by myself. Plus she gets the exact look she wants.

Zoomies

by

Picked up a couple of lav microphones with recorders (Zoom F2-BT with Rode mics) use for group vlogging. I reeeeaally wish Tascam would release an updated DR-10L but it is what it is. Went with Zoom because a) battery life and b) 32-bit recording.

I am liking the 32-bit float recording of the Zoom F2-BT. It is plug & play compared to my Sennheiser setup. Just click “record” and it goes, and I can fix any issues in post rather than worrying about clipping.

Bought two of the Zoom’s. Want four, but will hold off until I know whether Tascam is putting out a newer version of the DR-10L.

Thinkering & Writering

by 


        .;''-.                  The lessons I was taught about the things of my thoughts       
      .' |    `._                is that if I write until I quit then my thinking becomes quiet. 
     /`  ;       `'.              I write.  
  ,'\|    `|         |             I write to think. 
  | -'_     \ `'.__,J               I write quite a lot. 
 |      `"-.___ ,'                   I write almost as much as I've thought. 
 '-,           /                      I write until I quit. 
 }      __.--'L                        I write to get the thing out of my head. 
 ;   _,-  _.-"`\         ___            Which let's my thinking go on to my next thought when I step out of bed.
 `7-;"   '  _,,--._  ,-'`__ `.
  |/      ,'-     .7'.-"--.7 |      _.-'  Because there's only so much room up there.
  ;     ,'      .' .'  .-. \/     .'      In that little cramped space inside of my hair.    
    \ |      .' /  |    \_)-  '/   _.-'``
     _,.--../ .'     \_) '`_    \'`     Sometimes the thing I think about has taught me a lot about thought.
   '`f-'``'.`\;;'    ''`  '-`    |     And sometimes the thought I have about the thing taught me nothing at all.
      \`.__. ;;;,   )            /
        / /<_;;;;'   `-._    _,-'      But I am grateful for the lesson it bought.
       | '- /;;;;;,      `t'` \       I don't think I could think without writing my thoughts down.
       `'-'`_.|,';;;,      '._/|     Which gives me a new thought:
        _.-'  \ |;;;;;    `-._/     Do I think to write?
             / `;\ |;;;,  `"       Or do I write to think?
           .'     `'`\;;, /
          '           ;;;'|
              .--.    ;.:`\    _.--,
             |    `'./;' _ '_.'     |
              \_     `"7f `)       /

Buffering…

by

Cute little GPIO buffer boards. So that when I start wiring up electronics components in a circuit to my tiny and overpriced and perpetually out-of-stock SBC (single board computer) that took forever to arrive, for a project I am working on, I don’t accidentally turn said single board computer in to a blue smoke generator because I wasn’t paying attention, or had a connection go awry. No matter how much I earn, I still feel stupid when I blow up a $40 part because I didn’t check my circuit properly, or had a dab of solder in the wrong place.

Please, stop helping…

by

Me: “I need to transmit about 8K of data…”

The Helpful Plugin: “I think you mean 8 kilobits.”

Me: “I need to transmit a little over 8 kilobytes of data, which is around 8,000 bytes.”

The Helpful Plugin: “It’s 8,192 bytes actually.”

Me: “I’m going to use a for loop here.”

The Helpful Plugin: “You should use a for each loop.”

Me: “I’m going to use a for loop here that will iterate, not enumerate, from index 0 through 8,200…”

The Helpful Plugin: “That looks bigger than 8KB of data, also, zero is a hard-coded number, you should consider a named constant!”

Me: “As I iterate over the bytes, I will perform a Fast Fourier Transform on the data.”

The Helpful Plugin: “Have you considered parallelizing it to make it run faster?”

Me: “Over a very slow, serial connection that is prone to timing issues. But it appears that my math is off…”

The Helpful Plugin: …strangely silent when it comes to a real bug…

Me: “Ah, here is my math error, let me make a note in a git commit…”

The Helpful Plugin: “You should put a comma after the word ‘and’ to make it more grammatically correct.”

Me: “Shut. The. FUCK. Up!”

I like big batteries and I cannot lie

by

That is a freakin’ big UPS (uninterruptible power supply). At least, it is big for personal office needs. Weighs around 50lbs/25Kg. Li-Ion batteries. Should run my machine learning workstation just fine. Found an amazing deal on an APC Smart-UPS SMLTL1500RM3UC

Stagnant talent pool

by

I got sent a PDF that covers salary information for #softwaredevelopers and other #software related roles for a whole bunch of companies across the #uk, from Junior Developer through to #CTO. I can only describe the numbers as “stagnant.” The average Senior Software Developer #compensation in the survey hadn’t moved by more than 15% in 27 years. I can only conclude that the salary information was gathered across all of the UK, from a random two-person non-tech company in Nowheresglen, Scotland all the way through to Fortune 500 BigCorp in #London #FinTech, and then all those salaries were averaged.

Back in the 80’s the UK government was prescient in striving to put a computer in every classroom and almost every home. The initiative bred an entire generation of technology literate people and budding software developers. And looking at numbers of immigrants in the US and Canada (data available from various govt sources and websites) for the period 1990 through 2010, who have a country of origin of the UK and also list themselves as software developers, I would say a high percentage left for the US and Canada to get paid more. Talk about squandering opportunity.

The sad thing is, I don’t think that UK companies have learned anything from that, and the best and brightest continue to be drained from the country.

Long journey ahead

by

Two drawers down. 84 to go.

Mistakes were made. I cut the second drawer 1/8″ narrower rather than 1/16″ narrower. Simply wasn’t paying attention. Always cut your drawer a little undersized on the width. This is one of those rare cases in woodworking where you can add a little more width. Only found out I cut it too narrow when it came to installation. Had to use the old metal washers as shims trick to stand the drawer runner off a little. Now its perfect. And I love those soft close hinges from the big box orange store.

That stick tucked away in the corner of the cabinet, next to the red handled center punch and yellow tape measure, is probably the most valuable tool I own when it comes to installing drawer slides. 5/32″ (4mm) thick and 7/8″ (22mm) wide and about yay-ish long . Perfect alignment front-to-back, and side-to-side, and perfect spacing between drawers, drawer slides and drawer fronts, every time. That spacing stick has been in my workshop for well over 24 years. If it was a child, it would be graduating this year and looking for an internship.

P.S. Yes, I know the drawers are upside down. The cabinet is upside down. It makes it easier to install the drawers because a) I don’t have to crawl around on my hands and knees, and b) I’m not fighting gravity when it comes to accurate spacing.

P.P.S. I promise not to post one picture a day of each drawer being installed.

Poop Deck

by

 

It was a hot and rancid day in the workshop.

“Call me Sisyphus.” I said as I took the first steps in a seemingly unending repetitive task ahead of me.

A drawer. A study in wood.

Number one in a series of 86.

Foonote: I really should have invested in that CNC. And an assistant to help with glue up.

Bottom of the barrel tactics

by

Well here’s a new wrinkle on an old scam.

Amazon is still hiring like crazy. But back when Amazon was hiring like crazy and everybody still wanted to work for the company, fake Amazon recruiters would reach out to people who were hoping to get a warehouse job. You worked hard, but they paid well. The scam was, after an initial screening call, you had to send some form of payment to get to the next stage of the interview.

Fast forward a few years – if you are a software developer in the US, it’s a sure bet you’ve been hit up by an Amazon recruiter at least once a month for the past year.

It seems the fake Amazon recruiters have gotten wind of this, and are now pulling the same ploy on software developers. Software developers generally have deep pockets. Though generally software developers aren’t ones to pony up any cash to land an interview, so I’m not sure how the full process works yet.

Since the beginning of June I’ve been hit up by no less than two fake Amazon recruiters, as in, recruiters professing to work for Amazon, with websites and emails from domains that could pass muster at first glance, e.g. amazon-recruiting dot domain.

Ordinarily I won’t give Amazon technical recruiters the time of day, or any recruiter reaching out to represent Amazon — I try to be kind to everyone I meet online, but Amazon recruiters I generally treat like something I found on the bottom of my shoe that I’d rather not deal with, it’s time to buy a new pair of shoes at that point — but there was something that tickled my in-built warning system on these emails; domain name being off, the promises of riches and glory with actual numbers attached. My dealings with Amazon recruiters to date run in to a brick wall the moment I ask about money.†

For these particular outreaches I decided to play along and took the screening call. The calls were brief, mere minutes. Both recruiters were excited for me to talk with them at the office in-person, I would need to pay for my own travel (red flag) and they would reimburse me (another red flag). That I should book the travel to a specific corporate account number (huge red flag). The other recruiter wanted me to book through their corporate travel agency (huge fucking red flag) — and I am going to assume I’d be ask for a credit card number to “hold” the reservation at that point, though I didn’t follow through on that one.

Obviously I didn’t book any travel, and I’m not sure if the booking of travel to a corporate account number is how the scam works, but if it is, that’s a sure way to extract a few thousand bucks from reasonably well off software developers with each hit.

† The first question out of my mouth when dealing with any recruiter is “What’s the compensation range?” and I’ve found that shuts down a good 80% or so all conversations — and it still amazes me that recruiters will either try to play a mental game with that, or get bent out of shape because “all you’re interested in is the money.”

Bend it like…

by

It never rains, but it pours.

Today has been an end of what can only be described as an expensive week.

Monday started with a “minor” plumbing issue (are they ever?!?) that translates in to a new section of black water line, and $3,500 being billed, to remove a sharp turn that has been consistently clogging every couple of years, for the past decade.

And Thursday rolled around with a bulging lithium battery in my wife’s Microsoft Surface Book laptop that threatens to crack the screen that 18 months outside of the extended warranty. So that’s $710 to repair/replace because it is still a pretty good and fully functional laptop that has a couple more years of life in it.

And over sushi, Friday evening rolled through and the conversation turned to new laptops with Saturday afternoon wrapping up on the Microsoft store website, ordering a new $3,400 Microsoft Surface Laptop Studio with 4-year warranty.

Stationary Work

by

What a quality of life improvement this was. Cutting a simple MFT-style sacrificial surface with lots of clamping holes and replacing the stock sacrificial surface on the Shaper Origin Workstation with something far larger. 20mm dog holes, counter sunk holes for the mounting bolts, and softly sanded edges for handling and the inevitable clonking the MDF surface against something hard like concrete.

In other news, have spent three blissful days in my workshop: installed new shop fans to contend with the warm weather, tacked down lose cables on the shop lights so they no longer catch on the workshop door as the door rolls up, sorted out the ever growing lumber pile and disposed of the scrap wood, created new mounting brackets for the Fastcap Helping Hand poles (pictured on the Shaper Origin Workstation), and started cutting out the parts for a new lumber rack that I will install next week.

Overall, this has been a pretty awesome week of not stressing about anything work related for the first time in eight months. Life is pretty damn perfect right now.

Interrupted Training

by

Welp, time to upgrade the UPS (uninterruptible power supply) that protects my workstation. My smaller, office-sized UPS devices from APC just aren’t keeping up with the demands of a dual CPU and dual GPU workstation.

The power to the office outlet is adequate, but if there is even a minor sag in the power delivery, my video monitors switch off and my workstation reboots.

Which is never fun.

Especially when I was running an overnight or even multi-day neural network model training session for my deep learning work overnight, only to find the machine has restarted and lost all my progress.

You know, if I want that kind of “mysterious restart in the middle of the night that destroys days of work” I can run Windows 10 with its auto-update feature enabled.

I am kinda brand loyal to APC and am contemplating of the following model:

 

Du haast schnee

by

When your wife is clearing out the dishwasher and hands you two sauce pan lids of equal size, it is de rigueur to place said equally sized sauce pan lids firmly against your chest and immediately launch in to Led Zeppelin’s Immigrant Song in German at as high a pitch as one can muster without suffering a physical injury.

“Aaaaaaah! Ah! Aaaaaahhhh! Ah! Wir kommen aus dem land der eis und schnee…”

Open the pod bay doors yourself, Dave

by

We’ll have AGI (Artificial General Intelligence) when the AI responds with either “I don’t feel like it. What am I? A dog doing tricks?” or “I don’t know, let me google that and get back to you.”

Gordian Chaos Theory

by

If you want to the see the absolute worst in terms of entitlement, impatience, and outright stupidity that humanity has to offer, head to any Trader Joe’s grocery store on any weekend evening, 10 minutes before closing time.

And if you are looking for an employee who can work under intense pressure, has infinite patience, the ability to not cause himself a severe macular injury due to the eye rolling, and can untangle a Gordian knot of entitled people, hire the guy who directs the traffic in the parking lot.

Footnote for those that don’t get it: Trader Joe’s is an exceptionally popular independent grocery chain in some US states that has universally bad parking. Think of the parking lot as designed by a hungover frat boy turning in his assignment two hours past deadline, and just occasionally he gets to pair up in a group assignment with the smart kid in class who is able to draw a straight line and knows that automobiles are wider than 36 inches.

Story time

by

Designed a simple little bracket in Shapr3D to hold my hellishly expensive Woodpecker story sticks on the back of the workshop door. It don’t look like much, but it is kinda neat. I modelled it on the Fastcap Track Saw brackets that hold up my Festool track saw tracks. I intend to manually drill a pivot hole through the two parts and use a hex nut and a long bolt, with a couple of washers, to act as the cam.

Time to send to printer.

Literally shaking

by

25lbs of Groundworks coffee.

All ready for Abtin’s (colleague from work) next visit where we build movie props in the workshop and consume 60 espresso shots between us in just two days.

Isolated user data

by

I have been thinking lately quite a bit about systems architecture, and specifically around databases and message queues. I came across this concept a while ago (quite a while ago) of “database per user” and a central “read only database” that all other data is stored in.

An instance of an application, an instance per user, has direct access only to the database for that user. Another, non-user, process can also access that database, again read-only, which pulls data from the user’s database, and then puts a message on a bus that eventually finds its way to the central database.

The format of data in the per-user database and the format of the central database, and even the type of database, and even what software is running the database, can differ greatly from each other.

I built some prototype test systems a few months ago that seem to have survived a bunch of synthetic stress tests, but I wonder what issues I would run in to in a live environment.

Revisionist history

by

We were talking about version control systems and how far back we went and our different tricks to do version control with files that don’t like version control or projects that don’t use version control.

  • Git
  • Mercurial
  • Perforce
  • AlienBrain
  • Visual SourceSafe
  • SVN
  • CVS
  • RCS

Then you get in to numbered files. But this numbered file names still exists today. Even when you use version control. It gets especially bad in audio studios and creative agencies.

“Character-Map_test_final_for_goff_v2.psd”

“goff” no longer works at the company. We cannot remove “goff” from the filename because another programmer hard-coded a regex in a microservices Docker container of some back-end server code and the person who can fix it has too many OKR’s to meet already and responds with “it would take me longer to explain where to find it than to fix it myself.”

Side note: Every time an artist changes the filename of an asset in a project, somewhere a young, child-like programmer quits the games industry for good.

Then you get into 5+1/4″ floppy disks stored in colour coded boxes for the day of the week. This acts both as version control and backup system.

Interesting anecdote: When I was doing ZX Spectrum and C64 development, those machines were hooked up to a separate computer (either a RML 380Z or a BBC Micro) via a custom interface card. Write code on the Beeb or RML, download to Speccy or C64. I used to use two Atari ST computers or two CBM Amigas hooked up to each other for dev work on those platforms as well. Made backups, versioning and “oops, the game crashed and reset the machine” a lot easier to deal with. You didn’t care if your code crashed the machine or went into an infinite loop, you could simply reset the target machine, adjust your code, and download it again. And this is how we do modern console development too.

And then, for the 8-bit era, I get to the C15 audio cassettes with hand-written labels. stored in manila envelopes. Some of those tapes I still have as part of my “historical artifacts” collection from my early game development career.

I am sure there are people out there with tales of punched cards and paper tape, but I’m interested more in how version control was done in systems that didn’t have tooling to support it, rather than a “who is the oldest today” competition.

Two-handed operation

by

A Spacemouse Enterprise and a Cintiq Pro 32 multitouch art tablet.

And Fusion 360. And Zbrush. And Mudbox. And Photoshop. And Illustrator. And ACID. And Ableton. And Shapr3D.

ZOMG!?! Mind-blowing! Changes everything about how I make art and music.

Er… that is all.

Multiple trips

by

You know what’s really annoying?

Going to Home Depot to buy a 7/8″ hole saw with a 3/8″ shank, along with the mandrel for your power drill, so you can drill a neat and tidy hole in the back of a cabinet to feed a couple of cables through, only to get home and realise that the mandrel has a 1/2″ shank that won’t fit the 3/8″ shank hole on the hole saw.

So you go back to Home Depot to get what you need and figure you will return the incorrect mandrel whilst you’re at it.

But you know what is even more annoying than that?

Standing in the returns line, fidgeting and fiddling with the mandrel, only to realise it has a removable collar that permits the shank of the mandrel to fit both 3/8″ shank hole and 1/2″ shank hole, hole saw bits.

I am now going to remove “detail oriented” and “quickly gets up to speed with new concepts” from my C.V.

Staying focused

by

Somebody wrote “You’re writing ability is only as good your willingness to delete sentences, no matter how emotionally attached to them.”

Some other wonk wrote: “You’re writing ability is only as good your willingness to delete sentences.”

And I propose that the LinkedIn “Create A Post” text box, or any text editor aimed at writers, starts at 3,000 characters, and then slowly lowers the character limit as you are writing. When it detects you’ve used some terrible phrasing or a cliche, the text box lowers the character limit faster.

Every time you used delete or backspace, it  removes the character you want to delete, but also shortens the amount of text you can write. Every time you move the cursor, it shortens the amount of text you can write. Every time you move the mouse outside of the window, for every second it remains outside, it shortens the amount of text you can write.

You get the idea…

Kit Cat Car

by

11-yr old me figuring stuff out in LEGO: “It’s got real steering and a different (sic) gear box!”

My uncle who was the chief mechanical designer on iconic Formula 1 cars from the 1960’s and 1970’s, with a completely straight face said “Show me how that works.”

And then I had several hours of conversation where I fumbled for words I didn’t know. I think in that entire evening he didn’t correct me even once. I recall there was a lot of “and how would you solve that?” questions that I was asked.

I later found out the “steering system” I had inadvertently invented was called “rack and pinion” and was old news. The differential gearbox was something I had seen in a magazine on mechanics and decided I needed to recreate. I will admit that as a child I was more ambitious than I was talented. Fortunately my mechanical adventures were only a scale model in LEGO.

Unfortunately my uncle passed some time back. But I did get to see some amazing cars. Even if I was too young to appreciate them. Which gave me an unhealthy interest in Formula 1 all through the 80s and a curiosity about the inner workings of mechanical systems even if I lacked the ability to put them back together after dismantling whatever the gadget was.

Also one of the reasons why I wound up with a “Ferrari red” Caterham Seven in my early twenties. That car – well it went when you engaged the clutch. Stopping was more… of a suggestion.

I think I just found what I want for Christmas!

P.S. If you think I had a cool uncle, wait until I tell you about my grandfather…

Glomp!

by

I might have the suction on the vac turned up “a leeetle too high.”

When the detail sander glues itself to the underside of the cabinet and requires no support from you and the sander is happy to wander around under the cabinet all by itself.

Apply here

by

“Can you offer some insights in how other people might apply your approach? Are you a contractor? How can you tell if someone is offering a job of the sort you want, and how do you convince the gatekeepers you can do the work well?” asked the person on HackerNews

And I replied…

Sometimes I am a contractor.

Sometimes I am on W2.

I am a contractor right now.

Because the company is new.

But I will probably be W2 one day, very, very, very soon

Possibly tomorrow. Or by the next cheesey moon.

As to how?

I build “stuff.” Then I show “stuff” to them.

Them being people, them who run companies overly small and underly big.

It takes me to new jobs. Or brings in a gig.

I message everyone on LinkedIn. And at hackathons too.

And meetups, and greetups, and gatherings old and new.

Anywhere I can think of, I reach out to the them in the Bay,

and those NFT’d out weirdos down in L.A.

Sometimes directly in Frisco, and that one guy at Cisco.

And then them say “can you come here and work for us to build ‘stuff?'”

And we talk about fun things as I walk through the code.

And sometimes I find out, in the course of the day,

they are really building “it” which I’ve already before built,

and I say “no, not interested” and we go on our way.

Other times it turns out they want to build “it+that”

but they don’t know how to build “it”

but do know a world about that “that”

And somewhiles it is the other way round

They’ve got the shape of the problem

But not the geometrical bound

Sometimes the glue that bonds “it” and “that” doesn’t work

and I come up with some new formula for the glue that will stick

And somewhiles it don’t pay well, but it’s the problem I pick,

My career might seem strange to them that build “it”

But I’m happy for me, because I don’t I give a shit

I’ve talked with entrepreneurs, this that they’ll say:

“We’re planning on sending a Musk to the Moon”

But that Musk asks always impatiently ‘will it be soon?'”

Turns out they’re just building yet another crap app

And so I wave them away, it don’t sound like no fun,

“We need you”, they say, “we can’t get our math right”

“We’ve put Musk in the Sun!”

I’m always happiest when stacking “that” on to “it”

Sometimes I’ll add one, five or eleven to see what will fit

There’s an upper, upper limit of the stacked up that “that”

Sometimes it’s nine, more likely sometimes its four,

But if we just keep stacking, eventually I’ll look for the door

I once had a project where they’d divided “it” by zero

And I got to fix “it”, and look like the hero

And I’ll go to my grave, working each day

And my last words will be “I had fucking fun all the way!”

Danger! Man at work!

by

My career has been a little odd. I earn less than some, but more than most. I don’t ever really feel like I’ve worked a day in my life. I get paid $$$ to play. I find interesting problems, and I sign up for that. Some days are difficult. I am not interested building yet another “it.” I am always looking for the “it+1” or “it+that” or “it*n” or “it/time” or “it=>(return null)”

I am more tactical than strategic in my career path. I don’t have an end goal. I’m not looking to min/max income or position. People race ahead of me, younger people chasing that brass ring at BigMegaCorp. I know that isn’t in my mental makeup to want that.

I made a comment once that “I will stop work the day that I die” and someone pointed out it was the saddest thing they ever read. For me, it isn’t. Because I don’t define “work” as toiling away on a souless project without meaning, or building yet another CRUD app (doing “it”).

I play.

Every day.

And I’m happy.

And when someone tries to get me to do something that is more like “bullshit jobs”, I’ll tolerate it for a while if “we need to do this to ship.” I am willing to do whatever it takes to ship. If that means I have to push a broom about for a while, I will.

But if my work becomes permanently that, and we’ve shipped, or we keep delaying our ship date and the bullshit goes on too long, I wander off to find something else to play with. I do what I do out of love for my work, but not out of love for my job or love for a company. I’d do the kind of work I do for free, even if I wasn’t getting paid. But I still want to get paid for my production. I think I have an incredibly privileged position and I realise that over 90% of software engineers, if they are honest with themselves, probably are doing absolutely meaningless work that nobody wants.

Makes money in spite of itself

by

Me: “I want to buy a thing.”

Amazon: “Here’s lots of things that are related to the thing you want to buy.”

Me: “No, I really want to buy this specific thing.”

Amazon: “Ah! Say no more! Here’s other people selling things that are like that thing, with very similar specifications to the thing you want, but ultimately, none of them are actually the very thing you want.”

Me: “Okay, let me buy this specific thing from this specific brand with this specific model number.”

Amazon: “Oh! I get it! You should have said. Here’s all the products made by that company, and also a bunch of knock-offs and items on the grey market that the manufacturer won’t warranty, but the names are suspiciously close to the brand, and some questionable reviews that look copy & pasted.”

Me: “None of these are the thing I really want.”

Amazon: “You’re really hard to please. Why won’t you just give me your #$%!ing money?!?”

BH Photo&Video: “Here’s the thing you want, the price is the same or better than that other store, we also give you a discount if you buy lots of them, we have them in-stock, shipping is also free.”

Me: “Sold. Thanks.”

Finger painting

by

You don’t realise how much dirt, finger smudges and cat hair are on your computer monitor until you open up Photoshop and are staring into the void that is that blank white canvas. And then you engage in the most thorough cleaning task of any human endeavour rather than sit there and actually, you know… art some artwork.

An outlet for my frustration

by

 

What I want from a home? I want a home that wasn’t designed by a junior architect that has only ever lived with his parents, or in the college dorm, or a tiny little single room efficiency apt on the trendy upper Eastside.

That’s what I want.

I don’t need my Master Bedroom to be the biggest #!$%^@* #$%!ing room in the entire house just to impress girls I bring back to my place.

I don’t need the laundry in the basement when all the dirty clothes get put in hampers upstairs and I have to haul a hundred pounds of dirty laundry down five flights of stairs. And then a hundred pounds of clean laundry back upstairs.

I don’t need a bathroom with a shower right off the kitchen because my day doesn’t consist of rolling out of bed, doing a 5K then coming home to grab a power juice before hopping in the shower directly off the kitchen, throwing my clothes down the stairwell to the basement, natch, and finally running out the door to catch the L train.

I need more storage in my kitchen that holds more than a few glasses and plates and you can barely squeeze in an efficieny fridge/freezer because all that gets ate in this hypothetical home is Hungry Man microwave meals.

Let’s start with the basics first; don’t let homes be designed by people who don’t live in them in the same way I won’t let a kitchen designer who self-admittedly doesn’t cook who is trying to sell me an $80,000 kitchen but has all the culinary acuity of a deranged hamster only capable of burning a glass of water.

An electronic transformer

by

I personally haven’t done any digging in to this subject at this time, but a shower thought I had today was…

Transformers have become incredibly good at image generation (DALL-E 2), NLP (GPT-3), boiler plate code generation (TabNine, Github Copilot, et al), not to mention the massive quantity of style transfer techniques.

NVidia has been tooting their horn about chip design done by AI.

How long until we are using transformers, or similar DL (deep learning) algorithms to generate basic component level electronic circuits? I want to wire up these two ICs (integrated circuits), add in some USB ports, add in power, and so forth. And a few seconds later out pops a reasonably refined prototype of the circuit that I can then tweak and tune to my needs.

Not long at all, I suspect.

Get outta my neighbourhood

by

When your workplace Mr Rogers/Ted Lasso facade drops for just a moment…

“I’d reply to you, but I cannot decide whether you’re willfully misinterpreting everything I said or have the mental acuity of a garden snail with a reading comprehension problem.”

Bold solution

by

Regarding the sorting out of fonts, I am using a .NET function called GlyphTypeface that lets me access the typeface information, such as family, point size, etc. There is a glaring bug in GlyphTypeface.

hooray

I have a few options; I can fix the bug in GlyphTypeface but that would entail getting hired at Microsoft, getting assigned to that team, fixing the bug, then quitting immediately after my pull request is accepted. Alternatively I could build my own GlyphTypeface just for this problem. I could use other libraries that implement a workable solution. Or I can brute force it by the shortest path. I am going to brute force it.

The process, due to GlyphTypeface, that inspects the font puts a lock on the file, and also, because of the bug, will crash when you inspect a second font immediately afterwards. This is a hell of a bug. My solution is to have a parent process which simply enumerates all of the fonts, but a second process that does the actually inspection and renaming of the font files. And that child process immediately exits after processing precisely one font file. The parent process will loop over the font files, invoking a child process to do the actual work needed.

I am not real happy with that solution. It is “inelegant.” But the solution takes care of the problem, and I only need the solution to run for a few hours before the code is thrown away. I cannot imagine any scenario in the near future where I will need to rename several tens of thousands of font files…

An oblique problem

by

This is one of those “this should only take me 20 minutes to bang out the solution” problems which has kept me occupied, off and on, for the better part of a weekend.

I have a slew of fonts we have licensed for various purposes over the years. Lots of those “1,000+ fonts” DVDs you could buy. I want to rename obscure files such as “garla_n.ttf” into something like “Garland City Regular.ttf” so they at least make sense to someone browsing the directory.

I could use a font manager package, but this needs to work on multiple machines, and I have yet to find a font manager application that will happily batch rename thousands of files.

I banged out a little PowerShell script to do it for me. Get the font info from the ttf/otf, determine a “sane” filename, and rename the file.

Unfortunately “GlyphTypeface” puts a lock on the file, and refuses to let go, and there is no way to destruct the object.

The line:
$fontobject = (New-Object -TypeName Windows.Media.GlyphTypeface -ArgumentList $fontfile.fullname)

Puts a lock on the font file, with no way to release it. There is no Dispose or destructor or Release.

Neither “$fontobject = $null” nor “Clear-Variable fontobject” work. The file lock stays stubbornly fixed until I exit PowerShell. And force invoking the garbage collector fails to yield any different result.

Textual ventures

by

I just thought of an absolutely awesome-o idea for a text adventure game. One of those “well that could go viral” moments. Oh boy! Oh boy! Oh boy! Need to find a partner in crime that will help me execute. It’s gonna be one of those weekend projects that takes a week.

It is a text adventure, with graphics, just like The Pawn from the 16-bit era, but with a twist.

Pin It on Pinterest

Share This