Document and test Git-hosted root self-configuration

.github/workflows/docker-image.yml

@@ -1486,6 +1486,9 @@ jobs:
           sudo podman load -i /tmp/fetchit.tar
           sudo podman load -i /tmp/colors.tar
 
+      - name: Verify Git-hosted root bootstrap and self-reconfiguration
+        run: sudo bash tests/integration/root-config/test-root-config.sh
+
       - name: Reconcile a commit from an isolated Git repository
         run: sudo bash tests/integration/live-update/test-live-update.sh
 

docs/index.rst

@@ -19,6 +19,7 @@ images containing those changes; an older published image may lack new options.
    samples
    purpose
    methods
+   root_configuration
    lifecycle
    volumes
    sops

docs/methods.rst

@@ -40,6 +40,9 @@ directories or change its destination layout.
 Dynamic Configuration Reload
 ----------------------------
 
+For a complete Git-hosted root that configures several methods and updates
+itself, see :doc:`root_configuration`.
+
 There are a few ways currently to trigger FetchIt to reload its targets without requiring a restart. The first is to
 pass the environment variable `$FETCHIT_CONFIG_URL` to the `podman run` command running the FetchIt image.
 The second is to include a ConfigReload. If neither of these exist, a restart of the FetchIt

docs/root_configuration.rst

@@ -0,0 +1,173 @@
+One root configuration per host
+===============================
+
+FetchIt already supports a root configuration that defines all of a host's
+workloads. This provides the entry-point pattern requested in
+`issue #417 <https://github.com/containers/fetchit/issues/417>`_: point each host
+at one FetchIt YAML file in Git, then let that file define multiple independently
+scheduled methods, possibly drawing workloads from different repositories.
+The root can update its own method configuration through ``configReload``.
+No additional FetchIt instance or Kubernetes controller is needed.
+
+This is analogous to the root entry point of an app-of-apps deployment. FetchIt
+uses its own ``targetConfigs`` and method schema. It does **not** recursively
+import child FetchIt YAML files or implement Argo CD Application resources,
+ApplicationSets, dependency ordering, or transactional workload rollout.
+Each method reconciles independently. Configuration composition is a separate
+feature; the existing root workflow does not require it.
+
+Repository layout and root file
+-------------------------------
+
+For example, keep one root per host or host group in a configuration repository:
+
+.. code-block:: text
+
+   fleet-config/
+     hosts/
+       edge-01.yaml
+       edge-02.yaml
+
+``hosts/edge-01.yaml`` can reference separate workload repositories:
+
+.. code-block:: yaml
+
+   configReload:
+     configURL: https://raw.githubusercontent.com/example/fleet-config/main/hosts/edge-01.yaml
+     schedule: "*/5 * * * *"
+   targetConfigs:
+     - name: frontend-source
+       url: https://github.com/example/frontend-workloads.git
+       branch: main
+       raw:
+         - name: frontend
+           targetPath: deploy/raw
+           schedule: "*/1 * * * *"
+     - name: monitoring-source
+       url: https://github.com/example/monitoring-workloads.git
+       branch: main
+       quadlet:
+         - name: monitoring
+           targetPath: deploy/quadlet
+           root: true
+           schedule: "*/1 * * * *"
+
+Replace the URLs with your repositories and ensure the selected paths contain
+valid workload definitions for those methods. The first target applies raw
+Podman definitions; the second applies a Quadlet bundle. Other documented
+methods can be combined in the same root. See :doc:`methods` for each method's
+schema and prerequisites. Method names must remain stable to retain their Git
+baselines and lifecycle identities.
+
+The root is downloaded using HTTP(S), such as a Git hosting service's **raw file
+URL**, rather than cloned using Git. ``configURL`` is not a Git repository URL
+or an SSH clone URL. Workload ``url`` fields, by contrast, identify repositories
+that FetchIt clones/fetches on the configured branch. A root file can reference
+several methods within one repository or methods across several repositories.
+
+Bootstrap without a local configuration
+---------------------------------------
+
+For a public root, start with an empty, writable mount directory:
+
+.. code-block:: bash
+
+   mkdir -p "$HOME/fetchit-config"
+   chmod 700 "$HOME/fetchit-config"
+   sudo podman run -d --name fetchit --security-opt label=disable \
+     -e FETCHIT_CONFIG_URL=https://raw.githubusercontent.com/example/fleet-config/main/hosts/edge-01.yaml \
+     -v fetchit-state:/opt \
+     -v "$HOME/fetchit-config:/opt/mount:Z" \
+     -v /run/podman/podman.sock:/run/podman/podman.sock \
+     quay.io/fetchit/fetchit:latest
+
+This example uses the rootful Podman socket; enable that socket before launch.
+For rootless deployment, use the socket mounts and prerequisites in
+:doc:`running`. The named volume retains repository baselines and lifecycle
+state. The mount directory must be writable because FetchIt publishes
+configuration updates with atomic rename; mounting an individual config file
+or a read-only directory cannot support that operation.
+
+An existing local ``/opt/mount/config.yaml`` takes precedence on initial startup.
+For the URL bootstrap above, ensure that file does not already exist. FetchIt
+requires a valid root response before starting its methods. Supplying an
+environment URL alone does not establish recurring reloads: keep a
+``configReload`` block in the downloaded root, pointing to its own raw URL.
+Choose a different entry-point URL for each host or host group. Automatically
+reading cloud machine metadata is not implemented; provision the URL through
+your existing service/container environment or bootstrap automation.
+
+For private roots, follow the local bootstrap and authentication instructions
+in :doc:`methods`. The initial URL request before a local configuration is
+loaded has no configured credentials. Do not assume merely setting a token
+environment variable authenticates that first request. Keep top-level
+``gitAuth`` settings in the downloaded root as needed for subsequent downloads
+and workload repositories; do not store credentials in Git or URLs.
+
+Updating the root and its workloads
+-----------------------------------
+
+Commit a new method or target to the root file. On the next successful
+``configReload`` check, FetchIt validates and publishes the new configuration,
+then reloads the engine's targets **inside the existing process**. New methods
+start and existing methods replay/reconcile their saved Git baselines. The
+FetchIt container does not need to be replaced or restarted.
+
+Changes to workload files are independently observed by the methods' Git
+schedules. A workload commit does not require changing the root YAML. Updating
+the root and workload files together does not make their deployment atomic:
+methods have separate schedules, and application failures can leave a partial
+workload rollout. Removing a method retains its workload by default. Use
+``cleanupOnRemoval`` only where supported and intentionally enabled; see
+:doc:`lifecycle` for ownership, cleanup, and rollback behavior.
+
+A failed root download or invalid root schema preserves the active configuration
+and backup. A valid root is not proof that every workload will apply: image
+availability, repository access, and method runtime requirements are checked
+later. Monitor logs and the :doc:`status` endpoint for invocation activity.
+See :doc:`running` for remote configuration validation and failure behavior.
+
+Security and trust
+------------------
+
+A root configuration controls host operations through FetchIt's Podman socket.
+Treat access to its repository as administrative authority over the managed
+host. Use a trusted repository, restricted write access, reviewed changes, and
+HTTPS with certificate verification. Do not serve the root over an untrusted
+plain HTTP network. FetchIt uses ordinary HTTP(S) certificate verification for
+these downloads; workload ``gitsignVerify`` settings do not authenticate the
+root file itself. A raw URL pinned to a reviewed commit can give an immutable
+root revision, but automatic root updates then require updating that URL.
+
+Keep secret material outside root/workload repositories. All targets use the
+root's configured Git authentication; separate per-child credentials and
+privilege isolation are not part of this pattern. Do not give untrusted teams
+root repository access merely to delegate a workload. A writable configuration
+mount is sensitive local state and should be readable only by its intended
+owner. Root download validation limits document size and rejects malformed or
+unknown fields; it does not turn an untrusted configuration into a safe one.
+
+Verification
+------------
+
+The repository includes ``tests/integration/root-config/test-root-config.sh``.
+It uses a private, loopback-only Git fixture and the production engine image to:
+
+* Start with no local configuration and download a root YAML served from Git.
+* Apply two separately configured methods from that root.
+* Commit an expanded root that adds a third method and updates an existing
+  workload, without restarting the engine container.
+* Reject an invalid root update while preserving the active config, backup
+  history, and existing workloads.
+
+Run it with rootful Podman, the socket enabled, and the engine/sample images
+available:
+
+.. code-block:: bash
+
+   sudo env FETCHIT_TEST_ENGINE_IMAGE=localhost/fetchit:test \
+     bash tests/integration/root-config/test-root-config.sh
+
+The fixture's plain HTTP transport is confined to host loopback for testing.
+The test cleans up its own containers, state volume, Git server, and temporary
+files. It does not push to GitHub or query cloud instance metadata.

tests/integration/live-update/git-server.py

@@ -21,6 +21,21 @@ def serve_git(self):
             self.send_error(503, "Git fixture offline")
             return
         request = urlsplit(self.path)
+        if self.command == "GET" and request.path == "/config.yaml":
+            result = subprocess.run(
+                ["git", "--git-dir", str(Path(self.server.repository_root, "live.git")),
+                 "show", "refs/heads/main:config.yaml"],
+                stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=10,
+            )
+            if result.returncode != 0:
+                self.send_error(404)
+                return
+            self.send_response(200)
+            self.send_header("Content-Type", "application/yaml")
+            self.send_header("Content-Length", str(len(result.stdout)))
+            self.end_headers()
+            self.wfile.write(result.stdout)
+            return
         if request.path not in ("/live.git/info/refs", "/live.git/git-upload-pack"):
             self.send_error(404)
             return

tests/integration/root-config/test-root-config.sh

@@ -0,0 +1,139 @@
+#!/usr/bin/env bash
+# Verify existing self-configuration with a root YAML served from fixture Git.
+set -euo pipefail
+engine_image=${FETCHIT_TEST_ENGINE_IMAGE:-quay.io/fetchit/fetchit-amd:latest}
+sample_image=quay.io/fetchit/fetchit-sample-app:latest
+scratch=$(mktemp -d)
+fixture_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")/../live-update" && pwd)
+engine="fetchit-root-config-$$"
+first="$engine-first"
+second="$engine-second"
+third="$engine-third"
+state="$engine-state"
+server_pid=
+cleanup() {
+  status=$?
+  podman logs "$engine" || true
+  cat "$scratch/server.log" || true
+  podman rm -f "$engine" "$first" "$second" "$third" >/dev/null 2>&1 || true
+  podman volume rm "$state" >/dev/null 2>&1 || true
+  if [[ -n "$server_pid" ]]; then kill "$server_pid" 2>/dev/null || true; wait "$server_pid" 2>/dev/null || true; fi
+  rm -rf "$scratch"
+  exit "$status"
+}
+trap cleanup EXIT
+podman image exists "$engine_image"
+podman image exists "$sample_image"
+mkdir -p "$scratch/server" "$scratch/seed/apps/first" "$scratch/seed/apps/second" "$scratch/seed/apps/third" "$scratch/mount"
+git init --bare --initial-branch=main "$scratch/server/live.git"
+git -C "$scratch/server/live.git" config http.receivepack false
+git init --initial-branch=main "$scratch/seed"
+git -C "$scratch/seed" config user.name 'FetchIt root config fixture'
+git -C "$scratch/seed" config user.email 'ci@example.invalid'
+git -C "$scratch/seed" remote add origin "$scratch/server/live.git"
+python3 "$fixture_dir/git-server.py" "$scratch/server" "$scratch/port" > "$scratch/server.log" 2>&1 &
+server_pid=$!
+for ((attempt=0;attempt<50;attempt++)); do
+  if [[ -s "$scratch/port" ]]; then break; fi
+  kill -0 "$server_pid"
+  sleep 0.1
+done
+port=$(cat "$scratch/port")
+url="http://127.0.0.1:$port/config.yaml"
+write_workload() {
+  local name=$1 color=$2 app=$3
+  cat > "$scratch/seed/apps/$app/app.json" <<JSON
+{"Image":"$sample_image","Name":"$name","Env":{"APP_COLOR":"$color"}}
+JSON
+}
+write_root() {
+  cat > "$scratch/seed/config.yaml" <<CONFIG
+configReload:
+  configURL: $url
+  schedule: "*/1 * * * *"
+targetConfigs:
+- name: first-source
+  url: http://127.0.0.1:$port/live.git
+  branch: main
+  raw:
+  - name: first
+    targetPath: apps/first
+    schedule: "*/1 * * * *"
+- name: second-source
+  url: http://127.0.0.1:$port/live.git
+  branch: main
+  raw:
+  - name: second
+    targetPath: apps/second
+    schedule: "*/1 * * * *"
+CONFIG
+  if [[ "$1" == expanded ]]; then
+    cat >> "$scratch/seed/config.yaml" <<CONFIG
+- name: third-source
+  url: http://127.0.0.1:$port/live.git
+  branch: main
+  raw:
+  - name: third
+    targetPath: apps/third
+    schedule: "*/1 * * * *"
+CONFIG
+  fi
+}
+publish() {
+  git -C "$scratch/seed" add .
+  git -C "$scratch/seed" commit -m "$1"
+  git -C "$scratch/seed" push origin main
+}
+write_workload "$first" blue first
+write_workload "$second" green second
+write_workload "$third" purple third
+write_root initial
+publish initial
+# No config exists locally: bootstrap must download config.yaml from Git.
+test ! -e "$scratch/mount/config.yaml"
+podman run -d --name "$engine" --network host --security-opt label=disable \
+  -e FETCHIT_CONFIG_URL="$url" \
+  -v "$state:/opt" -v "$scratch/mount:/opt/mount" \
+  -v /run/podman/podman.sock:/run/podman/podman.sock "$engine_image"
+wait_color() {
+  local name=$1 color=$2 deadline=$((SECONDS+180))
+  while ((SECONDS<deadline)); do
+    if [[ $(podman inspect --format '{{.State.Running}}' "$engine" 2>/dev/null) != true ]]; then return 1; fi
+    if [[ $(podman exec "$name" printenv APP_COLOR 2>/dev/null) == "$color" ]]; then return 0; fi
+    sleep 1
+  done
+  echo "Timed out waiting for $name=$color" >&2
+  return 1
+}
+wait_color "$first" blue
+wait_color "$second" green
+test -s "$scratch/mount/config.yaml"
+initial_hash=$(sha256sum "$scratch/mount/config.yaml" | awk '{print $1}')
+engine_id=$(podman inspect --format '{{.Id}}' "$engine")
+! podman container exists "$third"
+# The root adds a method and existing Git methods receive updated workload data.
+write_workload "$first" pink first
+write_root expanded
+publish expand
+wait_color "$third" purple
+wait_color "$first" pink
+wait_color "$second" green
+test "$(podman inspect --format '{{.Id}}' "$engine")" = "$engine_id"
+test "$(podman inspect --format '{{.RestartCount}}' "$engine")" = 0
+test "$(sha256sum "$scratch/mount/config-backup.yaml" | awk '{print $1}')" = "$initial_hash"
+# Invalid root data cannot replace the active root or add/remove methods.
+active_hash=$(sha256sum "$scratch/mount/config.yaml" | awk '{print $1}')
+printf 'targetConfigz: []\n' > "$scratch/seed/config.yaml"
+publish invalid-root
+# Wait beyond one scheduled config check; the fixture log proves it was read.
+for ((deadline=SECONDS+100;SECONDS<deadline;)); do
+  if podman logs "$engine" 2>&1 | awk '/invalid downloaded config/ {found=1} END {exit !found}'; then break; fi
+  sleep 1
+done
+podman logs "$engine" 2>&1 | awk '/invalid downloaded config/ {found=1} END {exit !found}'
+test "$(sha256sum "$scratch/mount/config.yaml" | awk '{print $1}')" = "$active_hash"
+wait_color "$first" pink
+wait_color "$second" green
+wait_color "$third" purple
+test "$(podman inspect --format '{{.RestartCount}}' "$engine")" = 0
+echo 'Root-config integration passed: Git bootstrap, multiple targets, root expansion, workload update and invalid-root preservation.'