As part of our AI adoption journey, we wanted to know which custom skills, agents and MCP servers were actually in use. So, we built a small tool that scanned our repositories for SKILL.md files, .agent.md files and MCP configuration.
It worked, but it answered the wrong question. It told us what existed, not what developers used.
The Copilot usage metrics API now closes that gap.
What changed?
In September the API got CLI customization metrics. The catch: every customer-defined skill, agent and MCP server was grouped under other, and slash commands under custom. Useful for trends, useless for “is our new skill being used?”.
Since October 22, the totals_by_skill, totals_by_custom_agent, totals_by_mcp, totals_by_slash_cmd and totals_by_plugin arrays show the configured name of your own items. Each list also holds up to 25 entries per record, up from five.
Remark: names are reported from Copilot CLI, the Copilot app, clients built on the Copilot SDK (like VS Code and JetBrains) and ACP integrations. You need Copilot CLI 1.0.97 or later, or Copilot SDK 1.0.21 or later. IDE integrations that aren’t built on the SDK don’t report names.
Getting the report
The API doesn’t return the data directly. It returns signed download links to NDJSON files. For an organization, the latest 28-day aggregated report looks like this:
gh api `
-H "Accept: application/vnd.github+json" `
-H "X-GitHub-Api-Version: 2026-03-10" `
/orgs/ORG/copilot/metrics/reports/organization-28-day/latestThe response contains download_links, report_start_day and report_end_day. Let’s parse it and download the files:
$report = gh api `
-H "X-GitHub-Api-Version: 2026-03-10" `
/orgs/ORG/copilot/metrics/reports/organization-28-day/latest | ConvertFrom-Json
$i = 0
foreach ($url in $report.download_links) {
Invoke-WebRequest -Uri $url -OutFile "report-$((++$i)).ndjson"
}Per-user reports live under users-28-day/latest, and single days under organization-1-day?day=YYYY-MM-DD.
Remark: you need the Copilot usage metrics policy enabled, and either to be an organization owner or to have a role with the “View Organization Copilot Metrics” permission.
What does the data look like?
NDJSON is one JSON object per line, so we parse it line by line. In the 28-day aggregated report, the daily records sit in a day_totals array. Let’s look at what we got:
$records = Get-Content report-*.ndjson | ForEach-Object { $_ | ConvertFrom-Json }
$records.day_totals |
Select-Object -First 3 -Property day, distinct_skill_use_count,
totals_by_skill, totals_by_mcp, totals_by_custom_agent |
ConvertTo-Json -Depth 5Every entry holds a name and an interaction_count. The name field is called after the category. A skill entry looks like this:
{
"skill": "my-skill",
"interaction_count": 7
}MCP entries use mcp in the same way.
A few things to know before you start charting:
- Skills, slash commands and agents count invocations (custom agents count starts).
- MCP servers only count connection and reconnection attempts, successful or not. Calling tools on an already connected server doesn’t increase the number.
- Plugins count only skill invocations that come from a plugin. They’re a subset of the skill totals, so don’t add the two together.
- Anything beyond 25 entries is combined into a trailing
other.
Remark: the data comes from client telemetry. Outdated clients or blocked telemetry mean undercounting. The goal is to get adoption trends, not exact accounting.
Why do I still see “other”?
When I first ran this, I got this:
"totals_by_skill": [ { "skill": "other", "interaction_count": 7 } ],
"totals_by_mcp": [ { "mcp": "other", "interaction_count": 10 } ]The changelog promised names, so what was wrong? Nothing. Names only appear for report days processed after the release on October 22. Earlier days are not reprocessed and keep other (or custom for slash commands).
The 28-day report spans four weeks, so if you add up the whole window, the old days dominate. Let’s look per day instead:
$records.day_totals | ForEach-Object {
[pscustomobject]@{
day = $_.day
skills = ($_.totals_by_skill | ForEach-Object { "$($_.skill)=$($_.interaction_count)" }) -join ', '
mcp = ($_.totals_by_mcp | ForEach-Object { "$($_.mcp)=$($_.interaction_count)" }) -join ', '
}
} | Sort-Object day | Format-Table -WrapThe recent days show names, the older ones show other. The window rolls forward by itself.
If recent days also show only other, check these in order:
- Client version: users on Copilot CLI older than 1.0.97 or an older SDK-based client still send unnamed telemetry. The per-user report shows
totals_by_cli.last_known_cli_version. - Path-like names: names that look like file paths stay grouped under
otherby design. - Telemetry: blocked or outdated clients mean missing data.
Building a canvas extension to visualize it
Reading JSON in a terminal gets old quickly. The GitHub Copilot app has canvases for this: an interface that you and the agent share, opening in the app’s right side panel.
You don’t write the canvas by hand. In an agent session, type / and select /create-canvas, then describe what you want. A good prompt covers the workflow, what you should be able to do in the interface, and the data it works with:
/create-canvas Create a Copilot customization usage canvas for our organization.
Data: use the authenticated gh CLI to call
/orgs/ORG/copilot/metrics/reports/organization-28-day/latest
(X-GitHub-Api-Version: 2026-03-10), download the NDJSON files from
download_links and read the day_totals records.
Entries in totals_by_skill use a "skill" field and entries in
totals_by_mcp use an "mcp" field, both with interaction_count.
Inspect the first record to find the name field for totals_by_custom_agent.
Show: a tab per category (skills, custom agents, MCP servers) with a
horizontal bar chart of interaction_count over the whole period, and a
daily trend for the item I select. Show the distinct_*_use_count
values as a small summary. Days processed before 22 October 2026 only
contain "other"; leave those out of the chart and mention that in a note.
Actions: refresh data, and an agent-callable action that returns the
top 10 items per category.
Read-only. No GitHub writes and no source edits.The agent generates the extension and opens it in the right panel. It’s saved as a canvas extension, and you choose where it lives:
- Project scope:
.github/extensions, shared with the repository team - User scope:
~/.copilot/extensions, personal to your machine
Remark: I have planned to package this canvas in a plugin and share it on GitHub. Once ready I'll update this post with a link.
That’s it!
We introduced a scanner for the “is it used?” question, keep it for “does it exist?”, and let a canvas do the charting.
Happy AI coding!