Yesterday one of our Application Insights dashboard showed a P95 latency spike and we had no idea why. The telemetry told us that a request was slow. But it didn't tell us why. Was it a database call? A CPU-bound loop? Lock contention? A GC pause?
The naive approach would have to been to add logging statements around the code you suspect, redeploy, wait for the issue to reproduce, and repeat. In production, that's slow and it doesn't scale - you're guessing, and every guess costs a deployment cycle.
We stayed away from all that guess work and reached out to the Application Insights Profiler. Instead of reasoning from logs, you get actual flame graphs of real production requests, showing exactly where time was spent.
What the profiler actually captures
The profiler runs as an agent alongside your application and periodically captures traces of live requests - not synthetic load, actual production traffic. For each captured request, it builds a trace you can inspect down to the method level, showing where time went across the call stack.
If your app runs on Azure App Service, the good news is the profiler is preinstalled as part of the App Service runtime. You don't need to add anything to get it running. You enable it, and it starts capturing.
Remark: As I’ll show in the next post, you can easily enable it also in a containerized environment (like ACA or AKS) or even on-prem if you want.
Where to find it
Once it's enabled (I'll cover the actual setup in part 2), the Profiler lives under your Application Insights resource:
- Select Performance in the left menu
- Pick an operation from the operation name list - Overall is selected by default
- Select Profiler traces
- Pick one of the captured requests from the list
From there you land on the trace itself, where you can drill into the flame graph for that specific request.
Triggering a profile on demand
Waiting for the profiler to capture something interesting isn't always practical, especially if you're actively debugging an incident. There's a Profile now option that starts an on-demand session: every profiler agent sending data to your Application Insights instance starts capturing immediately, and the session shows up in the list after 5 to 10 minutes.
Remark: to trigger a session manually, you need at least write access on your role for the Application Insights component. In most setups you'll have this automatically - if you don't, it's worth checking your access before you assume the feature is broken.
So the question becomes:
How do you actually turn this on for your app, and what does reading one of these traces look like in practice?
That's what the rest of this series covers - part 2 walks through enabling the Profiler on App Service and in code, and part 3 gets into reading a flame graph without guessing.
Stay tuned!