Proud Moments

Holy crap that’s a lot of requests per second. 40 million per second, more than 2 billion in a minute, on the test hardware.

 

 

Have managed to make the back-end never hit the database for any reads except during start-up, and all database writes go through a WAL queue rather than directly to the database.

The requests are a mixture of request/response objects (which contain DTOs and collections of DTOs in the response) and media such as images and short videos. I don’t know how close we were to maxing out a dual 100gbit NIC — it is some exotic dual 100gbit NIC with four 25gbit ports on each NIC from my understanding, not sure on that as I haven’t physically seen the hardware with my own eyes. But there was still plenty of bandwidth to spare, and the quite a bit of CPU to spare too.

Now I will admit, these are synthetic tests, i.e. traffic replay of earlier live sessions, coming from another machine, without a bunch of network hardware in between, so I suspect real-world production usage will be considerably lower.

Configuration is:
MS SQL -> WAL cache -> Kestrel -> In-Memory Query & Object Cache -> NGINX -> Apache Traffic Server

Technically the Kestrel server application is doing its own in-memory/in-process object and object collection caching so that I am not dumbly relying on external caches to do the heavy lifting for me. The Kestrel server application is effectively just a cache filler once it gets going and has answered the initial burst of queries and built internal collections. And the only time Kestrel gets queried (after that initial burst) is if something hasn’t migrated to a more forward position (higher tier?) cache.

 

Update:

I have since moved from synthetic replay tests to production requests and our numbers ticked up a bit due to the way we shape our traffic so we’re just cresting 42M RPS, or around 2B+ per minute on a single machine.

I cannot go in to a lot of detail on implementation. But I am very happy with the throughput.

There’s still a little headroom, but I suspect I am going to be hitting limitations of the OS, network stack and drivers at that point.

I feed everything through eight separate 25Gbit connections, though tests have shown that I could get far better results by feeding through 32 x 10Gbit connections. Results better than “yeah, but 320Gbit is more than 200Gbit” would suggest. This would be due to packet offload, PCIe bandwidth, RAM <=> NIC DMA constraints, etc, etc.

The majestic monolith literally handles everything, from media to database to server. There are two other containers for logging and housekeeping.

I focused on what can I get out of the hardware. I optimized at each layer, which was the low-hanging fruit, then optimized the more esoteric parts, e.g. read-once/WAL database access pattern, automatically denormalizing data once it enters a caching layer, storing cached data in different data structures depending on its usage, deduplicating (but not normalizing) data where it made sense, updating in-memory cached objects on database writes, deferring all database writes to post-processed batches (housekeeping container), making the client smarter so that it didn’t ask for complex joins and other operations that couldn’t be cached, and so on. Not trusting the client, but assuming the client was smart.

There were a lot of bottlenecks that I was able to kick out by being given carte blanche to “fix it, I don’t care what it takes, make it so we can hit reddit numbers.” And then finally just tuning the hell out of the OS, the Mellanox cards, and so on.

What a quality of life improvement this was. Cutting a simple MFT-style sacrificial surface with lots of clamping holes and replacing the stock sacrificial surface on the Shaper Origin Workstation with something far larger. 20mm dog holes, counter sunk holes for the mounting bolts, and softly sanded edges for handling and the inevitable clonking the MDF surface against something hard like concrete.

In other news, have spent three blissful days in my workshop: installed new shop fans to contend with the warm weather, tacked down lose cables on the shop lights so they no longer catch on the workshop door as the door rolls up, sorted out the ever growing lumber pile and disposed of the scrap wood, created new mounting brackets for the Fastcap Helping Hand poles (pictured on the Shaper Origin Workstation), and started cutting out the parts for a new lumber rack that I will install next week.

Overall, this has been a pretty awesome week of not stressing about anything work related for the first time in eight months. Life is pretty damn perfect right now.

11-yr old me figuring stuff out in LEGO: “It’s got real steering and a different (sic) gear box!”

My uncle who was the chief mechanical designer on iconic Formula 1 cars from the 1960’s and 1970’s, with a completely straight face said “Show me how that works.”

And then I had several hours of conversation where I fumbled for words I didn’t know. I think in that entire evening he didn’t correct me even once. I recall there was a lot of “and how would you solve that?” questions that I was asked.

I later found out the “steering system” I had inadvertently invented was called “rack and pinion” and was old news. The differential gearbox was something I had seen in a magazine on mechanics and decided I needed to recreate. I will admit that as a child I was more ambitious than I was talented. Fortunately my mechanical adventures were only a scale model in LEGO.

Unfortunately my uncle passed some time back. But I did get to see some amazing cars. Even if I was too young to appreciate them. Which gave me an unhealthy interest in Formula 1 all through the 80s and a curiosity about the inner workings of mechanical systems even if I lacked the ability to put them back together after dismantling whatever the gadget was.

Also one of the reasons why I wound up with a “Ferrari red” Caterham Seven in my early twenties. That car – well it went when you engaged the clutch. Stopping was more… of a suggestion.

I think I just found what I want for Christmas!

P.S. If you think I had a cool uncle, wait until I tell you about my grandfather…

I have a small USB charging cable. Measures about two inches long. Lightning connector, Apple dock connector, mini USB, micro USB, USB-C.

I call it my “precious.”

“Why do you call it your precious?” asked the coffee shop acquaintance at the conference I am attending.

“Because it is such a small thing. And with it, the wielder can convey great power.” I said in my best Sean Bean accent.

“Ohmigod, you are *SUCH* a dork. You’re probably still single.” she responded.

A few hours later she walked up to where I was perched on a railing and chatting with another conference attendee, without formal introduction or pause she interjects “I need that charging cable for my iPhone.”

I turned slowly. “Sorry, I’m a dork. Remember?”

I am just so edgy some days.

The response, to maintain the professionalism of this website, is unprintable.

On another note, what kind of young lady goes out of her way to refer to a professionally attired businessman at a technology conference as a “dork?”

I was chatting with another freelancer at a networking meet-up the other day and we got around, naturally, to talking about our consulting/freelancing business.

“I have a course I offer that can show you how to create a six-figure freelancing business” said the contact “More than happy to give you 20% discount code too.”

“Six figures is a wide range,” I responded, “are we talking $900,000 or $100,000 here?”

“Closer to the $100K range.” replied the networking contact.

“$100K? Or closer to $200k?”

“I’ve had some people hit close to $200K.” clarified my networking contact, “Though I have to emphasize not everyone achieves that level so I don’t want to get your hopes up.”

“If you want to make a small fortune, first start with a large fortune.” I jested.

“I don’t get it.” said the networking hopeful, rather confused.

“If I sign up for your course with the generous 20% discount code, and I do really well at it, you will cleverly show me how to take a 75% pay cut.”

There was a long thoughtful pause.

“I’d like to know how you are achieving those results.” said the networking peer.

“I guess you’ll need to sign-up for my course, should I ever write it.” I smiled.

My word I am smug some days.

Sat at an outdoor coffee shop working quietly when two children walk up behind me.

Screaming kid and loudly chewing kid start randomly insulting me.

I look at them.

They flip me off.

I look at their Mother who is talking loudly on the phone with “an F this and a n****r that and oh no you didn’t and so I says to her with a mmm hmmm” as her dialect.

The Mother pointedly looks right at me and then right through me.

I go back to work.

Screaming kid stops screaming long enough to start poking at my touch screen.

I slap his hand away and he becomes screaming kid once more.

I look at Mother and she continues to look through me like I don’t exist, yammering away at a hundred miles an hour on her phone.

I open up a web browser while screaming kid tries reaching for the laptop again.

I navigate to PornHub and click on the first free video, unplug my headphones and turn the volume up.

“What is wrong with you?!” shouts the Mother, finally taking notice, “they’re just children.”

And finally she pulls loudly chewing kid and screaming kid away, stomping off to another table.

I go back to my work.

Porn is a far more effective deterrent to badly behaved children than a quiet word with the Mother ever could be.

“So what’s your greatest weakness?” asked the HR person.

“I actually expect to get paid market rate for my consulting services and have my invoices paid on time.” I replied.

“Well if all you care about is the money and not the opportunities for growth, this probably isn’t the contracting role for you.” retorted the interviewer.

“You’re right.” I nodded, got up, and left.

I have convinced my American girlfriend that “twat” is the past tense of “tweet” and that it is a “British thing.”

“How should I use it?” she asked.

“Well if you want to be polite, you say ‘My twat got seen by a thousand people today. I had lots of likes and comments on it.” But if you are talking about having sent your tweet or twat in this case, to one particular person, you would say ‘I twatted that person.'”

I cannot wait for her to start sending out some twats.

Was teaching a class this morning and going over some of the algorithms and mathematics needed for providing the simulated physics of a racing game.

One of the people in the class put up their hand, “As game developers will any of us ever need to know this?” they asked.

And I blurted out “Well you won’t, but one of the smart kids in the class probably will.”

I have, on more than one occasion, used a sick day to call in and avoid my own office birthday party.

 

No matter how busy I am, I refuse to ever eat lunch at my desk just because the project manager is pushing to hit a deadline.

No, just no.

This is the start of the slippery slope to losing all semblance of good work-life balance.

I’ve done it when I was younger and I have to admit, I was incredibly naïve and stupid.

Any project manager who is going to chastise me for not being a team player because I insisted on taking my lunch break is not a project manager I want to work for.

Never live in fear of losing your job.

Make everyone else live in fear that you will quit your job.

And if they don’t live in fear of you quitting…

You’re in the wrong job and you aren’t trying hard enough.