azurebackup: Add Copilot skills for telemetry reporting and new tool workflow#2570
Merged
shrja-ms merged 9 commits intoMay 8, 2026
Merged
Conversation
Add a Copilot skill under tools/Azure.Mcp.Tools.AzureBackup/skills/ that automates generation of weekly telemetry reports by: - Running KQL queries against the ddazureclients Kusto cluster - Classifying errors into Customer (4xx) / Azure Service / MCP Tool Bug - Correlating with merged PRs, releases, and known bug IDs - Producing an Outlook-compatible HTML report Includes: - SKILL.md with full procedure and error classification decision tree - references/kql-queries.md with all 10 production KQL queries - assets/report-template.html with Outlook-compatible HTML template
Add a Copilot skill that guides the full lifecycle of adding a new Azure Backup MCP command: - Phase 1: Implementation (options, service, command, registration) - Phase 2: Input validation checklist - Phase 3: Unit tests with required test patterns - Phase 4: Live tests with Record/Playback workflow - Phase 5: CI validation (build, format, spell check) - Phase 6: ToolDescriptionEvaluator scoring - Phase 7: Documentation updates - Phase 8: PR submission checklist
… methodology - BUG-1 (backup_status ArgumentNullException) is fixed in microsoft#2518 (code included in merged PR, though PR title only lists BUG-3,4,5,6,8) - Fix unit test count methodology: count [Fact] + [InlineData] for runnable test cases, not [Fact] + [Theory] for method count - Add Release column to bug tracker table - Add note about verifying PR diff vs title for bug coverage
shrja-ms
requested review from
KarishmaGhiya,
g2vinay,
jairmyree,
jongio,
joshfree,
msalaman and
vukelich
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds repository-local Copilot skills for the Azure Backup toolset: one to generate weekly telemetry reports and one to guide contributors through adding new Azure Backup commands. It fits into the codebase as operational/developer documentation rather than product code.
Changes:
- Adds a weekly Azure Backup telemetry-report skill plus a KQL reference file.
- Adds an Outlook-oriented HTML template for the generated report.
- Adds an Azure Backup “new tool” workflow skill covering implementation, testing, validation, and PR steps.
Several of the new instructions are currently inaccurate enough to miscount metrics, skew telemetry output, or leave contributors with incomplete validation steps.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 12 comments.
| File | Description |
|---|---|
tools/Azure.Mcp.Tools.AzureBackup/skills/azurebackup-telemetry-report/SKILL.md |
End-to-end workflow for generating weekly telemetry reports. |
tools/Azure.Mcp.Tools.AzureBackup/skills/azurebackup-telemetry-report/references/kql-queries.md |
KQL query set backing the telemetry-report skill. |
tools/Azure.Mcp.Tools.AzureBackup/skills/azurebackup-telemetry-report/assets/report-template.html |
HTML scaffold for the telemetry report email/export. |
tools/Azure.Mcp.Tools.AzureBackup/skills/azurebackup-add-tool/SKILL.md |
Contributor workflow for adding new Azure Backup commands. |
Telemetry report skill (SKILL.md + kql-queries.md): - Add case-sensitivity note for Kusto bag key access (microsoft#6) - Fix week-over-week query range to include current partial week (microsoft#7) - Add InvalidOperationException to MCP Tool Bug classification (microsoft#8) - Add missing KQL queries 11-13 for WorkloadType, DatasourceType, OperationScope custom dimensions (microsoft#9) - Fix live test count to use [Fact] only, avoiding double-count with [LiveTestOnly] on same method (microsoft#5) - Update decision tree to include InvalidOperationException Add-tool skill (SKILL.md): - Fix broken relative link to new-command.md (5 levels up, not 3) (microsoft#1) - Fix live test snippet to use [Fact] + CallToolAsync pattern, not non-existent [RecordedTest]/RunCommandAndAssert (microsoft#2, microsoft#11) - Add missing Update-AzCommandsMetadata.ps1 step after docs update (microsoft#3) - Add AOT/native build verification step (Build-Local.ps1 -BuildNative) (microsoft#4) - Fix vault/vaultType to reference AzureBackupOptionDefinitions, not OptionDefinitions.Common (microsoft#10) - Expand git add scope to include README.md and eng/vscode/README.md (microsoft#12) - Add commands metadata regeneration and AOT build to PR checklist
Contributor
Author
|
/azp run mcp - pullrequest |
|
Azure Pipelines successfully started running 1 pipeline(s). |
…cker The link verification step requires absolute paths from repo root, not relative paths. Convert all ./references/ and ../../../ links to /tools/Azure.Mcp.Tools.AzureBackup/skills/... format.
Link checker requires https://github.com/microsoft/mcp/blob/main/... format, not repo-root-relative paths.
jongio
reviewed
May 4, 2026
Contributor
jongio
left a comment
jongio
left a comment
There was a problem hiding this comment.
Skill files are well-structured and the telemetry workflow is thorough. One bug in the KQL queries that will produce incorrect report data, plus a note on internal URLs.
…BI URLs with placeholder
- Query 1 (primary per-tool health) was missing InvalidOperationException
in the McpToolBug case, causing inconsistency with Queries 2 and 4
- Replace hardcoded Power BI dashboard URLs in report template with
{{POWERBI_DASHBOARD_URL}} placeholder (public repo, internal GUIDs)
anannya03
reviewed
May 6, 2026
Contributor
|
Please also add a changelog entry. |
Contributor
|
Apart from the above highlighted points, everything else looks good to me. Thanks |
jongio
reviewed
May 7, 2026
Contributor
jongio
left a comment
jongio
left a comment
There was a problem hiding this comment.
Addresses my previous feedback. Both the KQL query inconsistency and the internal URLs are fixed.
…lemetry-skill # Conflicts: # tools/Azure.Mcp.Tools.AzureBackup/docs/new-tools.md
anannya03
approved these changes
May 8, 2026
github-project-automation
Bot
moved this from Untriaged
to In Progress
in Azure MCP Server
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds two Copilot skills under \ ools/Azure.Mcp.Tools.AzureBackup/skills/\ to automate recurring Azure Backup MCP workflows:
1. \�zurebackup-telemetry-report\ — Weekly Telemetry Report Generator
Automates the weekly telemetry report by:
Includes:
eferences/kql-queries.md\ — All 10 production KQL queries with 3-way classification
2. \�zurebackup-add-tool\ — New Command Workflow
Guides the full lifecycle of adding a new Azure Backup MCP command across 8 phases:
File Structure
\
tools/Azure.Mcp.Tools.AzureBackup/skills/
├── azurebackup-telemetry-report/
│ ├── SKILL.md
│ ├── references/kql-queries.md
│ └── assets/report-template.html
└── azurebackup-add-tool/
└── SKILL.md
\\
Testing
No code changes — skills are Copilot instruction files only. Validated by generating this week's report using the skill.
Invoking Livetests
Copilot submitted PRs are not trustworthy by default. Users with \write\ access to the repo need to validate the contents of this PR before leaving a comment with the text /azp run mcp - pullrequest - live. This will trigger the necessary livetest workflows to complete required validation.