Azure Application Insights– How to keep the amount of data under control

If you have read all my previous posts about Application Insights, my hope is that you would start to use a lot more of the great features it has to offer. However this comes with one big 'disadvantage'; you'll start to collect a lot more data which of course leads to increased telemetry traffic, data costs, and storage costs.

Let's have a look at 3 ways to keep this under control:

• Limit the amount of data by changing the retention duration
• Use sampling to reduce traffic
• Setting a daily data volume cap

Changing the retention duration

Data ingested into either classic or workspace-based Application Insights is retained for 90 days without any charge. For data ingested into your Log Analytics workspace can be retained at no charge for up to first 31 days (or 90 days if Azure Sentinel is enabled on the workspace).

If you are using a classic Application Insights resource, you can change the retention duration after opening the Application Insights resource in the Azure portal:

1. Go to the Configure section in the menu and select Usage and estimated costs.
2. Select Data retention at the top of the page.
3. Set the data retention in days

If you are using a workspace based Application Insights resource, you could do the same thing at the Log Analytics workspace level:

1. Go to the General section in the menu and select Usage and estimated costs.
2. Select Data retention at the top of the page.
3. Set the data retention in days

 

Use sampling to reduce traffic

A second option is to use sampling. This offers a good trade-off where we limit the data and storage costs while preserving a statistically correct analysis of application data. When metric counts are presented in the portal, they are renormalized to take into account sampling. Doing so minimizes any effect on the statistics.

You can do sampling at the application level through adaptive or fixed-rate sampling or do sampling at the Application Insights service endpoint level through ingestion sampling.

From the documentation:

• Adaptive sampling automatically adjusts the volume of telemetry sent from the SDK in your ASP.NET/ASP.NET Core app, and from Azure Functions. This is the default sampling when you use the ASP.NET or ASP.NET Core SDK. Adaptive sampling is currently only available for ASP.NET server-side telemetry, and for Azure Functions.

• Fixed-rate sampling reduces the volume of telemetry sent from both your ASP.NET or ASP.NET Core or Java server and from your users' browsers. You set the rate. The client and server will synchronize their sampling so that, in Search, you can navigate between related page views and requests.

• Ingestion sampling happens at the Application Insights service endpoint. It discards some of the telemetry that arrives from your app, at a sampling rate that you set. It doesn't reduce telemetry traffic sent from your app, but helps you keep within your monthly quota. The main advantage of ingestion sampling is that you can set the sampling rate without redeploying your app. Ingestion sampling works uniformly for all servers and clients, but it does not apply when any other types of sampling are in operation.

Adaptive sampling

Adaptive sampling is enabled by default in ASP.NET Core but can be changed using the following code.
First we have to disable the built-in adaptive sampling:

Then we can re-enable it with our customizations in place:

 

Fixed-rate sampling

Fixed-rate sampling reduces the traffic sent from your web server and web browsers at a fixed rate decided by us.

First we have to disable the built-in adaptive sampling:

Then we can enable the fixed-rate sampling module:

Ingestion sampling

Ingestion sampling should be configured in the Azure Portal:

1. Go to the Configure section in the menu, and then Usage and estimated costs.
2. Select Data sampling at the top of the page.
3. Set the fraction of the data you want to keep(percentage).

Setting a daily data volume cap

A last option you have is to set a daily volume cap.This allows you to configure a data volume limit per day.

If you are using a classic Application Insights resource, you can set this cap in the Azure portal:

1. Go to the Configure section in the menu, and then Usage and estimated costs.
2. Select Data Cap at the top of the page.
3. Set the data volume limit in GB/day.

If you are using a workspace based Application Insights resource, you should set the daily cap at the Log Analytics workspace level.

1. Go to the General section in the menu, and then Usage and estimated costs.
2. Select Data Cap at the top of the page.
3. Set the data volume limit in GB/day.