Skip to main content

Building an end-to-end monitoring solution with Azure Arc, Log Analytics and Workbooks - Part 5: Putting it all together

Wow! We covered a lot in this series.

Part 1 - Overview & Architecture

Part 2 – Data collection with Azure Arc

Part 3 – Data persistence in Log Analytics

Part 4 -  Data visualization with Azure Workbooks

Time for a wrap up and some troubleshooting

Let's trace the data flow from start to finish to make sure everything connects:

  1. The Azure Monitor Agent runs on each Arc-enabled on-prem VM.
  2. The Data Collection Rule tells the agent what health data to gather — application pools, Windows services, and scheduled tasks.
  3. The agent collects that data on a regular interval and ships it to Azure.
  4. The DCR routes the incoming data to our custom table (OnPremHealthStatus_CL) in the Log Analytics Workspace.
  5. The Workbook queries that table and renders the dashboard.

If any link in that chain breaks, data stops flowing. The troubleshooting section below covers the most common failure points.

Troubleshooting checklist

No data appearing in the workbook: Start at the table. Run a basic OnPremHealthStatus_CL | take 10 query directly in Log Analytics. If there are no results, the issue is upstream of the Workbook — either the agent isn't sending data or the DCR isn't routing it correctly.

Query timeout: If queries are timing out, reduce the time range or add more specific filters. Log Analytics has query timeout limits (typically 3 minutes for portal queries).

Conditional formatting not applying: Make sure the status values in your data exactly match the conditions you've set (case-sensitive). A status of running (lowercase) won't match a condition checking for Running.

Data is stale: Check the TimeGenerated column. If the most recent rows are hours old, the agent may have stopped collecting or lost connectivity to Azure. Check the agent health on the VM.

Partial data (some VMs missing): Verify the DCR association. Not all VMs may have the rule associated. Check the DCR's resource associations in the portal.

Agent version too old: The custom text logs data source type requires Azure Monitor Agent version 1.10 or higher. If your Arc-enabled VMs have an older agent version, update it through Azure Policy or manually via extension management in the portal.

File permissions: The Azure Monitor Agent runs under the NT AUTHORITY\SYSTEM account on Windows. Make sure the log file your script writes to is readable by SYSTEM. If you're writing to C:\ProgramData\ or C:\MonitoringData\, this should be fine by default.

DCR not updating: If you edit a DCR after it's been associated with VMs, the agents don't pick up the changes instantly. It can take up to 5 minutes for the agent to refresh its configuration. If you need an immediate update, restart the Azure Monitor Agent service on the VM.

Firewall blocking outbound connections: The Azure Monitor Agent needs outbound HTTPS access to several Azure endpoints. If your on-prem network has restrictive egress rules, make sure the following domains are allowed:

  • *.ods.opinsights.azure.com (data ingestion)
  • *.oms.opinsights.azure.com (agent configuration)
  • *.monitoring.azure.com (Arc and agent management)

If you're using a proxy, the agent can be configured to route through it via the Arc agent's proxy settings.

Schema mismatches: If you see errors when the agent tries to write to the custom table, the data being sent doesn't match the table schema. Double-check that the column names and types in your DCR data source configuration match the table definition exactly.

How to further improve this setup?

This setup gives you a solid foundation. From here, there are a few natural directions to extend it:

Alerting: Layer Azure Monitor alerts on top of the same custom table. You can alert when a critical service enters a Stopped or Failed state, without needing a separate alerting tool.

Historical trending: Extend the Workbook with time-series queries to track how component health has changed over days or weeks. This is useful for catching intermittent failures that a point-in-time view would miss.

Automation: Once you have reliable health data flowing into Log Analytics, you can trigger Logic Apps or Azure Functions in response to specific health events — auto-restarting a failed service, for example.

Maybe I’ll cover these extensions in future posts. For now, the five parts above should get you from zero to a working health dashboard.

Feel free to adapt the schema, queries, and workbook structure to fit your environment.

More information

Azure Monitor Agent Network Configuration - Azure Monitor | Microsoft Learn

Labels:

Popular posts from this blog

Podman– Command execution failed with exit code 125

After updating WSL on one of the developer machines, Podman failed to work. When we took a look through Podman Desktop, we noticed that Podman had stopped running and returned the following error message: Error: Command execution failed with exit code 125 Here are the steps we tried to fix the issue: We started by running podman info to get some extra details on what could be wrong: >podman info OS: windows/amd64 provider: wsl version: 5.3.1 Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM Error: unable to connect to Podman socket: failed to connect: dial tcp 127.0.0.1:2655: connectex: No connection could be made because the target machine actively refused it. That makes sense as the podman VM was not running. Let’s check the VM: >podman machine list NAME         ...
Read more

Cache stampede: when our cache turned against us

While investigating some performance issues, we ran into an ASP.NET Core API that cached a fairly expensive aggregation query for 60 seconds. Under normal load, that was fine: one request rebuilds the cache, everyone else reads from it. Under peak load, dozens of requests would arrive in that same expiry window, all see a cache miss, and all fire the same expensive query in parallel. The database didn't like that. That was the moment when our caching layer stopped helping and started hurting. A burst of requests comes in at the same time, all miss the cache, and all go hammer the database or the downstream API at once. That's a cache stampede . The cache was supposed to protect our backend, and for a few hundred milliseconds it did the opposite. Why this happens IMemoryCache.GetOrCreate (and its async sibling) looks like it protects you, but it doesn't add any locking on its own. Look at the naive version: public async Task<Report> GetReportAsync(string key) ...
Read more

A complex system designed from scratch never works

A few years ago, I worked as an architect on a big mainframe rewrite. I still count it as one of my failures. Not because the technology was wrong, but because I couldn't convince the management team to simplify the approach. Years later, the organization is still struggling to get the new system up and running. I left the project at the time, because I couldn't put my name behind an approach that would take very long and cost a lot of money without a working system to show for it along the way. Gall’s Law That memory keeps coming back to me, because it's a textbook case of Gall's Law playing out in real life. Gall's Law , from John Gall's Systemantics , states it plainly: A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works, and it cannot be patched to make it work. You have to start over with a simple system that works. What does that mean in practice,...
Read more