The art of simplicity

Yesterday one of our Application Insights dashboard showed a P95 latency spike and we had no idea why. The telemetry told us that a request was slow. But it didn't tell us why . Was it a database call? A CPU-bound loop? Lock contention? A GC pause? The naive approach would have to been to add logging statements around the code you suspect, redeploy, wait for the issue to reproduce, and repeat. In production, that's slow and it doesn't scale - you're guessing, and every guess costs a deployment cycle. We stayed away from all that guess work and reached out to the Application Insights Profiler . Instead of reasoning from logs, you get actual flame graphs of real production requests, showing exactly where time was spent. What the profiler actually captures The profiler runs as an agent alongside your application and periodically captures traces of live requests - not synthetic load, actual production traffic. For each captured request, it builds a trace you can in...

In a previous post I showed how to add the .NET runtime version as an OpenTelemetry resource attribute: ResourceBuilder.CreateEmpty() .AddService($"{_sofaSettings.ApplicationName}-{_sofaSettings.EnvironmentName}") .AddAttributes(new Dictionary<string, object> { ["deployment.environment"] = _sofaSettings.EnvironmentName, ["service.name"] = _sofaSettings.ApplicationName, ["runtime.dotnet.version"] = Environment.Version.ToString() }) .Build(); The idea was clean: attach facts about what is running directly to the resource, and let OpenTelemetry carry them along with every trace, metric, and log automatically. Unfortunately, there is a catch. The problem Azure Monitor's OpenTelemetry exporter only maps a fixed set of well-known resource attributes onto Application Insights fields. service.name and service.namespace become Cloud Role Name, service.instance.id becomes Cloud Role Insta...

At our organisation running a large fleet of .NET services, a deceptively simple question can be surprisingly hard to answer: what versions of .NET are our apps actually running in production? You'd think this would be easy. It isn't. Services get deployed, teams move on, and before long nobody is quite sure whether that one legacy service is still on .NET 6 — or even .NET Core 3.1. Spreadsheets fall out of date. README files lie. The only source of truth is what's actually running. We solved this with three lines of OpenTelemetry configuration. The problem We run dozens of .NET services across multiple teams. We are the middle of a push to .NET 10, but we have no reliable, centralised way to see the current state. We wanted to answer questions like: Which services are still on end-of-life .NET versions? Which teams still have work to do? After a migration wave, how do we confirm everything moved? The solution We already had OpenTelemetry set up ac...

After enabling Aspire for your application, you get OpenTelemetry integration for free thanks to the included ServiceDefaults code.  This will setup OpenTelemetry using a set of defaults: Unfortunately this default configuration didn’t pick up any of the custom metrics I added. I first thought that this meant that I couldn’t use the service defaults anymore and configure the OpenTelemetry integration manually. But then I discovered that you can change/extend the default configuration in a specific project by using the ConfigureOpenTelemetryTracerProvider and/or ConfigureOpenTelemetryMeterProvider methods. Add your custom OpenTelemetry configuration after calling the Aspire builder.AddServiceDefaults() : Now our custom metrics appear nicely in the Aspire dashboard: More information Welcome to Aspire | Aspire Enabling .NET Aspire for an existing solution Replacing EventCounters with the new Metrics API

If you've been using EventCounters for instrumenting your .NET applications, it's time to consider migrating to the newer System.Diagnostics.Metrics API. Based on the OpenTelemetry specification, the Metrics API offers a more modern, flexible, and standardized approach to application instrumentation. Why migrate? The Metrics API provides several advantages over EventCounters: Industry Standard : Built on OpenTelemetry, ensuring compatibility with a wide ecosystem of monitoring tools Better Performance : More efficient with lower overhead Richer Functionality : Support for histograms, exemplars, and more sophisticated metric types Improved API Design : Cleaner, more intuitive interface for defining and recording metrics Better Tooling Support : Growing ecosystem support from APM vendors and monitoring solutions Microsoft has indicated that EventCounters are in maintenance mode, with new development focused on the Metrics API. So reasons enough to m...

I’ m currently working on a new feature in one of our microservices. As this new feature could have large performance impact, we did some performance benchmarks up front through BenchMarkDotNet . The results looked promising but we were not 100% confident that these results are representative for real life usage. Therefore we decided to implement A/B testing. Yesterday I showed how to implement A/B testing in .NET using .NET Feature Management . Today I want to continue on the same topic and show you how we added telemetry. Measuring and analyzing results The most critical aspect of A/B testing is measuring results. You need to track relevant metrics for each variation to determine which performs better. The most simple way to do this is to fall back to the built-in logging in .NET Core : Although this is a good starting point, we can simplify and improve this by taking advantage of the built-in telemetry through OpenTelemetry and/or Application Insights. Using OpenTelemetry...

If you are a regular reader of my blog, you've certainly seen my earlier post on how to integrate Application Insights telemetry in your HotChocolate based GraphQL backend. IMPORTANT: Although the code below will still work, I would recommend to switch to the integrated OpenTelemetry functionality and send the information to Application Insights in this way. Today I had to upgrade an application to HotChocolate 13 and (again) I noticed that the application no longer compiled. So for people who are using the ApplicationInsightsDiagnosticsListener I shared, here is an updated version that works for HotChocolate 13: As I wanted to track the full operation I have overwritten the ExecuteRequest method, however if you only want to log any exception, I would recommend to override the RequestError and ResolverError methods: More information HotChocolate OpenTelemetry (bartwullems.blogspot.com) GraphQL HotChocolate 11 - Updated Application Insights monitoring (bartwullem...

Today I had to make some changes to an existing ASP.NET Core application still using .NET 6 and a preview version of the OpenTelemetry.Extensions.Hosting nuget package. I thought it was a good time to update to .NET 8 and to the latest OpenTelemetry version so updated both the targetframework and the used package versions. After the changes my csproj file looked like this: However when I tried to compile and run my application I got the following error message: 'IServiceCollection' does not contain a definition for 'AddOpenTelemetryTracing' and no accessible extension method 'AddOpenTelemetryTracing' accepting a first argument of type 'IServiceCollection' could be found (are you missing a using directive or an assembly reference?) Here is the code that caused the error: Turns out that the OpenTelemetry API has changed quite significantly since the preview and that AddOpenTelemetryTracing has been removed in favour of AddOpenTelemetry sinc...

HotChocolate remains my favorite GraphQL client for .NET. As I find it really important to monitor my applications, I have written multiple posts (here and here ) on how to integrate monitoring with HotChocolate. At the beginning of this year, with the release of HotChocolate 12.5 , support for OpenTelemetry was introduced. Although I updated my applications to start using it, it took me until now to write a corresponding blog post about it. Instrument your GraphQL Server with OpenTelemetry Let me walk you through the steps: We’ll start with an ASP.NET Core application with OpenTelemetry enabled. Therefore we added the following NuGet packages: OpenTelemetry.Extensions.Hosting OpenTelemetry.Instrumentation.AspNetCore OpenTelemetry.Instrumentation.Http OpenTelemetry.Instrumentation.SqlClient We also add a reference to the AzureMonitor exporter because we want to use Application Insights: Azure.Monitor.OpenTelemetry.Exporter O...

After configuring our Elastic APM server, we tried to send some OpenTelemetry data to it. Unfortunately no data seemed to arrive on the target server, so time to put on our debugging head and find out what is going wrong… The first thing we did was to check if there was indeed an application listening on the Linux server where we installed APM server. This can be done by using a Powershell cmdlet called Test-NetworkConnection . From our Windows jump server, we opened a Powershell terminal. There we typed the following command: tnc <server> –port <portnumber> If the connection succeeds, the TcpTestSucceeded value will return true .  

Elastic has their own Application Performance Monitoring solution as part of their Elastic Observability product. An important part of the solution are ‘agents’. Agent are responsible for instrumenting your application, collecting all the metrics and sending it to the APM server. Specifically for .NET the APM agent is released as a serie of NuGet packages . With the release of OpenTelemetry for .NET I was wondering if we could replace this Elastic APM specific solution with standard OpenTelemetry. In a first incarnation APM server didn’t support the OpenTelemetry standard and you had to use a seperate collector that converts the OpenTelemetry data to the Elastic APM format: Since version 7.13 of Elastic APM this is no longer necessary.The OpenTelemetry Collector exporter for Elastic was deprecated and replaced by the native support of the OpenTelemetry Line Protocol in Elastic Observability (OTLP). Now the only thing you need to do is to add some specific attributes ...