The art of simplicity

Posts

Our P95 spiked, now what?

Yesterday one of our Application Insights dashboard showed a P95 latency spike and we had no idea why. The telemetry told us that a request was slow. But it didn't tell us why . Was it a database call? A CPU-bound loop? Lock contention? A GC pause? The naive approach would have to been to add logging statements around the code you suspect, redeploy, wait for the issue to reproduce, and repeat. In production, that's slow and it doesn't scale - you're guessing, and every guess costs a deployment cycle. We stayed away from all that guess work and reached out to the Application Insights Profiler . Instead of reasoning from logs, you get actual flame graphs of real production requests, showing exactly where time was spent. What the profiler actually captures The profiler runs as an agent alongside your application and periodically captures traces of live requests - not synthetic load, actual production traffic. For each captured request, it builds a trace you can in...

.NET Aspire: The price of forgetting WithReference

Recently I lost way more time than I'd like to admit on an error that turned out to be one missing line of code. The symptom looked like a networking problem. The cause was a missing WithReference call in my Aspire AppHost. Here's the exception my proxy threw the moment it tried to forward a request to my API: System.Net.Http.HttpRequestException: No such host is known. (api:443) ---> System.Net.Sockets.SocketException (11001): No such host is known. at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnectionPool....

The mysterious .dev.localhost checkbox

When you create a new ASP.NET Core project in Visual Studio, there's a checkbox that's easy to click past: "Use the .dev.localhost TLD in the application URL." I always want to understand what a checkbox actually does before I tick it, so let's dig into this one. The problem it solves When you're working on more than one local web project, they all end up living at the same address: localhost . Only the port number tells them apart. Open your browser's address bar with three projects running and you'll see localhost:5001 , localhost:5215 , localhost:7099 — and you have no idea which is which until you actually look at the page. There's a second, less visible issue: because everything shares the localhost name, cookies and other domain-scoped browser storage are also shared across all your local apps. That's not something you usually want when you're testing. What .dev.localhost actually is .localhost is a reserved top-level dom...

Talking to Copilot like a caveman

I think that everyone who uses AI recognizes the following pattern; you ask an LLM a simple question and it answers like it's writing a blog post: introduction, context, three examples, a closing summary. Fine for a first read, expensive when you're chaining calls or running an agent loop all day. The trick to avoid this is called "caveman prompting". You tell the model to drop articles, pleasantries and filler, and answer in short, blunt fragments. It sounds silly. But it works up to a point. A first attempt: just say "be concise" Most people's first instinct is a one-line system prompt: Be concise. No fluff. This already gets you a good chunk of the savings. In benchmarks I've seen floating around, a plain "be concise, return structured output" instruction accounts can already give you a nice reduction. It's the cheapest fix and most people stop here, which is reasonable. The caveman approach The caveman skill takes...

Cache stampede: when our cache turned against us

While investigating some performance issues, we ran into an ASP.NET Core API that cached a fairly expensive aggregation query for 60 seconds. Under normal load, that was fine: one request rebuilds the cache, everyone else reads from it. Under peak load, dozens of requests would arrive in that same expiry window, all see a cache miss, and all fire the same expensive query in parallel. The database didn't like that. That was the moment when our caching layer stopped helping and started hurting. A burst of requests comes in at the same time, all miss the cache, and all go hammer the database or the downstream API at once. That's a cache stampede . The cache was supposed to protect our backend, and for a few hundred milliseconds it did the opposite. Why this happens IMemoryCache.GetOrCreate (and its async sibling) looks like it protects you, but it doesn't add any locking on its own. Look at the naive version: public async Task<Report> GetReportAsync(string key) ...

A complex system designed from scratch never works

A few years ago, I worked as an architect on a big mainframe rewrite. I still count it as one of my failures. Not because the technology was wrong, but because I couldn't convince the management team to simplify the approach. Years later, the organization is still struggling to get the new system up and running. I left the project at the time, because I couldn't put my name behind an approach that would take very long and cost a lot of money without a working system to show for it along the way. Gall’s Law That memory keeps coming back to me, because it's a textbook case of Gall's Law playing out in real life. Gall's Law , from John Gall's Systemantics , states it plainly: A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works, and it cannot be patched to make it work. You have to start over with a simple system that works. What does that mean in practice,...