I think that everyone who uses AI recognizes the following pattern; you ask an LLM a simple question and it answers like it's writing a blog post: introduction, context, three examples, a closing summary. Fine for a first read, expensive when you're chaining calls or running an agent loop all day. The trick to avoid this is called "caveman prompting". You tell the model to drop articles, pleasantries and filler, and answer in short, blunt fragments. It sounds silly. But it works up to a point. A first attempt: just say "be concise" Most people's first instinct is a one-line system prompt: Be concise. No fluff. This already gets you a good chunk of the savings. In benchmarks I've seen floating around, a plain "be concise, return structured output" instruction accounts can already give you a nice reduction. It's the cheapest fix and most people stop here, which is reasonable. The caveman approach The caveman skill takes...
While investigating some performance issues, we ran into an ASP.NET Core API that cached a fairly expensive aggregation query for 60 seconds. Under normal load, that was fine: one request rebuilds the cache, everyone else reads from it. Under peak load, dozens of requests would arrive in that same expiry window, all see a cache miss, and all fire the same expensive query in parallel. The database didn't like that. That was the moment when our caching layer stopped helping and started hurting. A burst of requests comes in at the same time, all miss the cache, and all go hammer the database or the downstream API at once. That's a cache stampede . The cache was supposed to protect our backend, and for a few hundred milliseconds it did the opposite. Why this happens IMemoryCache.GetOrCreate (and its async sibling) looks like it protects you, but it doesn't add any locking on its own. Look at the naive version: public async Task<Report> GetReportAsync(string key) ...