Design: Unified Bicep extension publishing for default resource types#11892
Design: Unified Bicep extension publishing for default resource types#11892kachawla wants to merge 6 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds a design note proposing how Radius should consolidate and automate publishing of Bicep extensions for default resource types from resource-types-contrib.
Changes:
- Documents the current per-namespace Bicep extension publishing model.
- Compares separate
radiusResourcespublishing versus merging into the existingradiusextension. - Proposes Option 1 and outlines publishing automation for versioned and latest artifacts.
716e105 to
da5ffd7
Compare
da5ffd7 to
8731505
Compare
brooke-hamilton
left a comment
There was a problem hiding this comment.
How is the consolidated extension versioned when one of the types has a version update?
This is the tradeoff of bundling multiple types in one extension :/, we don't have versions tied to one type. Bumping any single type would result into publishing latest tag of the whole extension (in case of radiusResource extension, will have to see how realistic it is to publish combined latest for single radius extension), or we bump the extension version whenever next Radius release happens. |
Signed-off-by: Karishma Chawla <kachawla@microsoft.com>
8731505 to
bed122a
Compare
Signed-off-by: Karishma Chawla <kachawla@microsoft.com>
Radius functional test overviewClick here to see the test run details
Test Status⌛ Building Radius and pushing container images for functional tests... |
…11914) ## Overview Update the existing `generate` subcommand of `manifest-to-bicep` to accept multiple manifest files. When given multiple YAML manifests with the same namespace, their `Types` maps are merged into a single output (`types.json`, `index.json`, `index.md`). This supports per-type manifest files (e.g. `containers.yaml`, `routes.yaml`) as introduced by the [automated resource type registration design](#11911), where each file defines a single resource type within a namespace. Backward compatible: single-file usage works exactly as before. ## Changes - `bicep-tools/cmd/manifest-to-bicep/main.go`: Updated `generate` to accept `<manifest1> [manifest2...] <output>` (last arg is always output dir). Added `mergeManifestFiles()` that validates same namespace and rejects duplicate types. - `bicep-tools/cmd/manifest-to-bicep/main_test.go`: Added tests for single-file, multi-file merge, namespace mismatch, nonexistent file, empty manifest list, and duplicate type detection. - `bicep-tools/cmd/manifest-to-bicep/testdata/`: Added test manifest files for `Radius.Compute` (containers, routes) and `Radius.Security` (secrets). ## Usage Single file (backward compatible): ```bash go run ./bicep-tools/cmd/manifest-to-bicep generate containers.yaml /tmp/out ``` Multiple files (merge into one output): ```bash go run ./bicep-tools/cmd/manifest-to-bicep generate containers.yaml routes.yaml persistentVolumes.yaml /tmp/out ``` ## Test plan - `go test ./bicep-tools/cmd/manifest-to-bicep/` - 6 tests covering single-file generation, multi-file merge, namespace mismatch rejection, nonexistent file handling, empty input, and duplicate type detection. ## Part of Unified Bicep extension publishing (PR 1/4). See [design doc](#11892). ## Dependencies - [Automated default resource type registration](#11911) (provides `defaults.yaml` and per-type manifest files that this command consumes) --------- Signed-off-by: Karishma Chawla <kachawla@microsoft.com>
## Overview Today, resource types from `resource-types-contrib` (e.g. `Radius.Compute/containers`, `Radius.Security/secrets`) are published as separate Bicep extensions (`radiusCompute`, `radiusData`, `radiusSecurity`), requiring users to configure multiple entries in `bicepconfig.json` and use different `extension` directives depending on the namespace. This PR is part of a series that consolidates them into the existing `radius` Bicep extension so users have a single `extension radius` for all Radius-authored types. Specifically, this PR wires up the build pipeline so that contrib resource type manifests (maintained via [`make update-resource-types`](#11911)) are included in the unified `radius` extension alongside the existing TypeSpec/Swagger-generated types. ## What this PR does 1. **`generate-bicep-types-contrib` Makefile target** -- reads `deploy/manifest/defaults.yaml` to discover which contrib namespaces to include and passes all per-type manifests for each namespace to `manifest-to-bicep generate` for merging into `types.json` + `index.json` + `index.md`. Creates empty `docs/` directories so the [doc generation script](https://github.com/radius-project/docs/blob/main/.github/scripts/generate_resource_references.py) doesn't crash (contrib resource type docs tracked in [#11918](#11918)). 2. **`publish-bicep-extension` Makefile target** -- wraps `bicep publish-extension` for local testability. 3. **Shared `index-builder.ts` module** -- extracts `buildTypeIndex()` from `generate.ts` into a shared module so both the autorest pipeline and the new `rebuild-index` CLI can use it without code duplication. 4. **`rebuild-index.ts` CLI wrapper** -- standalone entry point that rebuilds the unified `index.json`/`index.md` after contrib `types.json` files are added to the generated tree. 5. **Extended `generate-bicep-types` target** -- now calls `generate-bicep-types-contrib` then `rebuild-index` after the existing autorest step, so the full pipeline produces one unified extension. 6. **Removed contrib extension references** -- `radiusCompute`, `radiusData`, `radiusSecurity` removed from `bicepconfig.json`, `install-bicep.sh`, and `pkg/cli/setup` since these namespaces are now part of the single `radius` extension. 7. **Updated functional test workflows** -- added `yq` install step to `functional-test-noncloud.yaml` and `functional-test-cloud.yaml` before the `make generate-bicep-types` step, since the new `generate-bicep-types-contrib` target requires `yq` to parse `defaults.yaml`. ## How the build pipeline works after this PR ``` make generate-bicep-types | +--> Step 1: autorest pipeline (existing, unchanged) | Generates types from Swagger/TypeSpec for Applications.Core, Applications.Dapr, etc. | Output: generated/radius/applications.core/.../types.json, etc. | +--> Step 2: generate-bicep-types-contrib (new) | Reads defaults.yaml, runs manifest-to-bicep generate per namespace | Output: generated/radius/radius.compute/.../types.json, etc. | +--> Step 3: rebuild-index (new) Walks the full generated/ tree, builds unified index.json + index.md Output: generated/index.json (covers all namespaces from both steps) ``` ## CI impact The `generate-bicep-types-contrib` step requires `yq` to parse `defaults.yaml`. Workflows that call `make generate-bicep-types` need `yq` installed beforehand. This PR adds the install step to the two affected workflows (`functional-test-noncloud.yaml` and `functional-test-cloud.yaml`), using the same pattern as `verify-resource-types.yaml` from [#11911](#11911). ## Test plan - `go test ./pkg/cli/setup/...` -- verifies updated bicepconfig template - TypeScript build: `pnpm -C hack/bicep-types-radius/src/generator run build` -- verifies the refactored generate.ts and new index-builder.ts compile cleanly ## Dependencies - [Add multi-file merge support to manifest-to-bicep generate command](#11914) (merged) - [Automated default resource type registration](#11911) (merged -- provides `defaults.yaml` and per-type manifest files) ## Changes - `build/generate.mk`: New `generate-yq-installed`, `generate-bicep-types-contrib`, and `publish-bicep-extension` targets - `hack/bicep-types-radius/src/generator/src/index-builder.ts`: New shared module exporting `buildTypeIndex()` - `hack/bicep-types-radius/src/generator/src/cmd/generate.ts`: Refactored to import `buildTypeIndex` from the shared module instead of defining it inline - `hack/bicep-types-radius/src/generator/src/cmd/rebuild-index.ts`: New thin CLI wrapper that parses args and calls the shared `buildTypeIndex()` - `hack/bicep-types-radius/src/generator/package.json`: Added `rebuild-index` script entry - `bicepconfig.json`, `build/install-bicep.sh`, `pkg/cli/setup/application.go`, `pkg/cli/setup/application_test.go`: Removed contrib extension references - `.github/workflows/functional-test-noncloud.yaml`: Added `yq` install step before `make generate-bicep-types` - `.github/workflows/functional-test-cloud.yaml`: Added `yq` install step before `make generate-bicep-types` - `hack/bicep-types-radius/generated/`: Regenerated `index.json` and added contrib type definitions for `Radius.Compute`, `Radius.Data`, and `Radius.Security` (3 new namespace directories with `types.json`, `index.json`, `index.md`, and empty `docs/` placeholder) ## Part of Unified Bicep extension publishing (PR 2/4). See [design note](#11892). --------- Signed-off-by: Karishma Chawla <kachawla@microsoft.com>
|
Update - After discussion, we decided go with the single Bicep extension approach. This keeps all core and contrib resource types in one I have updated the design doc in this PR and the implementation to reflect the updates. Implementation PRs - |
## Overview Today the `radius` Bicep extension published to `biceptypes.azurecr.io` includes core namespaces (`Applications.Core`, `Applications.Dapr`, etc.) and, with [radius#11915](radius-project/radius#11915), also includes contrib resource types (`Radius.Compute`, `Radius.Data`, `Radius.Security`). However, when resource type manifests change in this repo, there is no automation to notify the Radius repo so it can refresh its manifest copies and republish the extension. This PR adds a workflow that fires a `repository_dispatch` event to `radius-project/radius` whenever resource type manifests are updated on `main`, triggering the Radius repo's automated sync and publish pipeline. ## End-to-end flow ``` resource-types-contrib merges to main | +--> notify-radius.yaml (this PR) fires repository_dispatch | +--> contrib-update-resource-types.yaml (radius repo) receives dispatch | +--> Runs 'make update-resource-types' to refresh manifest copies +--> Opens/updates PR on bot/update-resource-types branch | +--> Human reviews and merges the PR | +--> Push to main triggers build.yaml's existing build-and-push-bicep-types job | +--> Dispatches to radius-publisher +--> Publishes radius:latest to biceptypes.azurecr.io ``` ## What this PR adds ### `notify-radius.yaml` **Trigger:** Push to `main` touching any YAML file, excluding `.github/` and `docs/`. This avoids hardcoding namespace folder names (`Compute/`, `Data/`, `Security/`) so new top-level namespace folders are automatically covered without workflow changes. **What it does:** 1. Sends a `resource-types-contrib-updated` dispatch event to `radius-project/radius` using `peter-evans/repository-dispatch@v3` 2. Includes the commit SHA in the payload for traceability (informational only -- the Radius workflow always fetches `@latest`) 3. Writes a summary to the GitHub Actions UI **What happens if a non-manifest YAML changes?** The Radius workflow runs `make update-resource-types`, finds no diff, and exits cleanly without opening a PR. A no-op CI run (~1 minute) is the only cost. **What happens if a new namespace folder is added here but not in `defaults.yaml`?** Only types listed in the Radius repo's `deploy/manifest/defaults.yaml` are included in the extension. The dispatch fires but `make update-resource-types` finds no changes for the unlisted namespace and no PR is opened. The new types only appear in the extension when someone adds them to `defaults.yaml` in the Radius repo. ## Dependencies - Radius repo: [Add workflow to sync contrib resource types and publish Bicep extensions](radius-project/radius#11916) -- the receiver of the dispatch - Required secret: `GH_RAD_CI_BOT_PAT` with `repo` scope on `radius-project/radius` ## Changes - `.github/workflows/notify-radius.yaml`: New workflow ## Part of Unified Bicep extension publishing (PR 4/4). See [design doc](radius-project/radius#11892). --------- Signed-off-by: Karishma Chawla <kachawla@microsoft.com>
…ons (#11916) ## Overview Today the `radius` Bicep extension is published to `biceptypes.azurecr.io` via the existing [`build-and-push-bicep-types`](https://github.com/radius-project/radius/blob/main/.github/workflows/build.yaml#L356) job in `build.yaml`, which dispatches to the `radius-publisher` pipeline on every push to `main` and on version tag pushes. With [#11915](#11915) updating `make generate-bicep-types` to include contrib types, the existing publish pipeline automatically produces the combined extension -- no new publish workflow is needed. However, there is no automation to pull updated resource type manifests from `resource-types-contrib` into this repo. When someone merges a schema change or a new resource type in contrib, the manifest copies committed under `deploy/manifest/built-in-providers/` must be refreshed manually via `make update-resource-types` before the next publish picks them up. This PR adds a workflow that closes that gap by automating the manifest sync. ## How it works ``` resource-types-contrib merges to main | +--> notify-radius.yaml (contrib repo, PR 4) fires repository_dispatch | +--> contrib-update-resource-types.yaml (this PR) receives dispatch | +--> Runs 'make update-resource-types' to refresh manifest copies +--> Opens/updates PR on bot/update-resource-types branch | +--> Human reviews and merges the PR | +--> Push to main triggers build.yaml's existing build-and-push-bicep-types job | +--> Dispatches to radius-publisher +--> radius-publisher runs make generate-bicep-types (now includes contrib) and publishes radius:latest to biceptypes.azurecr.io ``` ## What this PR adds ### `contrib-update-resource-types.yaml` Handles `repository_dispatch` events (type: `resource-types-contrib-updated`) from `resource-types-contrib`. **Triggers:** - `repository_dispatch` -- fired by the contrib repo's `notify-radius.yaml` workflow (PR 4) - `workflow_dispatch` -- commented out for production, can be enabled during development **Steps:** 1. Validates `contrib_ref` as a hex commit SHA (informational only -- the actual version fetched is determined by `make update-resource-types` which runs `go get ...@latest`) 2. Installs yq (required by `make update-resource-types` to parse `defaults.yaml`) 3. Runs `make update-resource-types` to bump `go.mod` to latest contrib and copy manifests 4. If changes are detected (using `git status --porcelain` to catch both modified and new untracked files), opens or updates a PR on the `bot/update-resource-types` branch 5. Merging that PR triggers the existing publish pipeline to republish `radius:latest` **Security:** - `contrib_ref` is validated against `^[a-f0-9]{7,40}$` and passed via environment variables (not inline `${{ }}` interpolation) to prevent shell and script injection - Uses `GH_RAD_CI_BOT_PAT` for checkout and PR creation so the resulting push triggers CI checks (the default `GITHUB_TOKEN` cannot trigger workflows on pushes it creates) **Note:** This workflow depends on `make update-resource-types` from [#11911](#11911). It includes a pre-flight check that fails fast with a descriptive error if the target is not yet available. ## Dependencies - [Integrate contrib types into unified Bicep extension](#11915) - [Automated default resource type registration](#11911) (provides `make update-resource-types`) - Required secret: `GH_RAD_CI_BOT_PAT` ## Changes - `.github/workflows/contrib-update-resource-types.yaml`: New workflow ## Part of Unified Bicep extension publishing (PR 3/4). See [design doc](#11892). --------- Signed-off-by: Karishma Chawla <kachawla@microsoft.com> Co-authored-by: Nicole James <101607760+nicolejms@users.noreply.github.com>
…ons (radius-project#11916) ## Overview Today the `radius` Bicep extension is published to `biceptypes.azurecr.io` via the existing [`build-and-push-bicep-types`](https://github.com/radius-project/radius/blob/main/.github/workflows/build.yaml#L356) job in `build.yaml`, which dispatches to the `radius-publisher` pipeline on every push to `main` and on version tag pushes. With [radius-project#11915](radius-project#11915) updating `make generate-bicep-types` to include contrib types, the existing publish pipeline automatically produces the combined extension -- no new publish workflow is needed. However, there is no automation to pull updated resource type manifests from `resource-types-contrib` into this repo. When someone merges a schema change or a new resource type in contrib, the manifest copies committed under `deploy/manifest/built-in-providers/` must be refreshed manually via `make update-resource-types` before the next publish picks them up. This PR adds a workflow that closes that gap by automating the manifest sync. ## How it works ``` resource-types-contrib merges to main | +--> notify-radius.yaml (contrib repo, PR 4) fires repository_dispatch | +--> contrib-update-resource-types.yaml (this PR) receives dispatch | +--> Runs 'make update-resource-types' to refresh manifest copies +--> Opens/updates PR on bot/update-resource-types branch | +--> Human reviews and merges the PR | +--> Push to main triggers build.yaml's existing build-and-push-bicep-types job | +--> Dispatches to radius-publisher +--> radius-publisher runs make generate-bicep-types (now includes contrib) and publishes radius:latest to biceptypes.azurecr.io ``` ## What this PR adds ### `contrib-update-resource-types.yaml` Handles `repository_dispatch` events (type: `resource-types-contrib-updated`) from `resource-types-contrib`. **Triggers:** - `repository_dispatch` -- fired by the contrib repo's `notify-radius.yaml` workflow (PR 4) - `workflow_dispatch` -- commented out for production, can be enabled during development **Steps:** 1. Validates `contrib_ref` as a hex commit SHA (informational only -- the actual version fetched is determined by `make update-resource-types` which runs `go get ...@latest`) 2. Installs yq (required by `make update-resource-types` to parse `defaults.yaml`) 3. Runs `make update-resource-types` to bump `go.mod` to latest contrib and copy manifests 4. If changes are detected (using `git status --porcelain` to catch both modified and new untracked files), opens or updates a PR on the `bot/update-resource-types` branch 5. Merging that PR triggers the existing publish pipeline to republish `radius:latest` **Security:** - `contrib_ref` is validated against `^[a-f0-9]{7,40}$` and passed via environment variables (not inline `${{ }}` interpolation) to prevent shell and script injection - Uses `GH_RAD_CI_BOT_PAT` for checkout and PR creation so the resulting push triggers CI checks (the default `GITHUB_TOKEN` cannot trigger workflows on pushes it creates) **Note:** This workflow depends on `make update-resource-types` from [radius-project#11911](radius-project#11911). It includes a pre-flight check that fails fast with a descriptive error if the target is not yet available. ## Dependencies - [Integrate contrib types into unified Bicep extension](radius-project#11915) - [Automated default resource type registration](radius-project#11911) (provides `make update-resource-types`) - Required secret: `GH_RAD_CI_BOT_PAT` ## Changes - `.github/workflows/contrib-update-resource-types.yaml`: New workflow ## Part of Unified Bicep extension publishing (PR 3/4). See [design doc](radius-project#11892). --------- Signed-off-by: Karishma Chawla <kachawla@microsoft.com> Co-authored-by: Nicole James <101607760+nicolejms@users.noreply.github.com> Signed-off-by: Zach Casper <zachcasper@microsoft.com>
…ons (radius-project#11916) ## Overview Today the `radius` Bicep extension is published to `biceptypes.azurecr.io` via the existing [`build-and-push-bicep-types`](https://github.com/radius-project/radius/blob/main/.github/workflows/build.yaml#L356) job in `build.yaml`, which dispatches to the `radius-publisher` pipeline on every push to `main` and on version tag pushes. With [radius-project#11915](radius-project#11915) updating `make generate-bicep-types` to include contrib types, the existing publish pipeline automatically produces the combined extension -- no new publish workflow is needed. However, there is no automation to pull updated resource type manifests from `resource-types-contrib` into this repo. When someone merges a schema change or a new resource type in contrib, the manifest copies committed under `deploy/manifest/built-in-providers/` must be refreshed manually via `make update-resource-types` before the next publish picks them up. This PR adds a workflow that closes that gap by automating the manifest sync. ## How it works ``` resource-types-contrib merges to main | +--> notify-radius.yaml (contrib repo, PR 4) fires repository_dispatch | +--> contrib-update-resource-types.yaml (this PR) receives dispatch | +--> Runs 'make update-resource-types' to refresh manifest copies +--> Opens/updates PR on bot/update-resource-types branch | +--> Human reviews and merges the PR | +--> Push to main triggers build.yaml's existing build-and-push-bicep-types job | +--> Dispatches to radius-publisher +--> radius-publisher runs make generate-bicep-types (now includes contrib) and publishes radius:latest to biceptypes.azurecr.io ``` ## What this PR adds ### `contrib-update-resource-types.yaml` Handles `repository_dispatch` events (type: `resource-types-contrib-updated`) from `resource-types-contrib`. **Triggers:** - `repository_dispatch` -- fired by the contrib repo's `notify-radius.yaml` workflow (PR 4) - `workflow_dispatch` -- commented out for production, can be enabled during development **Steps:** 1. Validates `contrib_ref` as a hex commit SHA (informational only -- the actual version fetched is determined by `make update-resource-types` which runs `go get ...@latest`) 2. Installs yq (required by `make update-resource-types` to parse `defaults.yaml`) 3. Runs `make update-resource-types` to bump `go.mod` to latest contrib and copy manifests 4. If changes are detected (using `git status --porcelain` to catch both modified and new untracked files), opens or updates a PR on the `bot/update-resource-types` branch 5. Merging that PR triggers the existing publish pipeline to republish `radius:latest` **Security:** - `contrib_ref` is validated against `^[a-f0-9]{7,40}$` and passed via environment variables (not inline `${{ }}` interpolation) to prevent shell and script injection - Uses `GH_RAD_CI_BOT_PAT` for checkout and PR creation so the resulting push triggers CI checks (the default `GITHUB_TOKEN` cannot trigger workflows on pushes it creates) **Note:** This workflow depends on `make update-resource-types` from [radius-project#11911](radius-project#11911). It includes a pre-flight check that fails fast with a descriptive error if the target is not yet available. ## Dependencies - [Integrate contrib types into unified Bicep extension](radius-project#11915) - [Automated default resource type registration](radius-project#11911) (provides `make update-resource-types`) - Required secret: `GH_RAD_CI_BOT_PAT` ## Changes - `.github/workflows/contrib-update-resource-types.yaml`: New workflow ## Part of Unified Bicep extension publishing (PR 3/4). See [design doc](radius-project#11892). --------- Signed-off-by: Karishma Chawla <kachawla@microsoft.com> Co-authored-by: Nicole James <101607760+nicolejms@users.noreply.github.com> Signed-off-by: Zach Casper <zachcasper@microsoft.com>
Description
This design proposes consolidating the per-namespace Bicep extensions for default resource types (radiusCompute, radiusData, radiusSecurity) into a single published extension, and automating the publishing process to stay in sync with the default resource type registration list (defaults.yaml).
Options
Option 1 (proposed): Separate radiusResources extension - Merges contrib namespaces into one extension at br:biceptypes.azurecr.io/radiusresources. Go-only changes, independent pipeline, straightforward automation for both versioned and latest publishing.
Option 2: Merge into existing radius extension - Single extension for everything. Best UX but couples two generation pipelines (TypeScript + Go) and complicates latest publishing from contrib CI.
Key decisions to review
Type of change
Fixes: #issue_number
Contributor checklist
Please verify that the PR meets the following requirements, where applicable:
eng/design-notes/in this repository, if new APIs are being introduced.