diff --git a/.github/workflows/cluster-setup-tests.yml b/.github/workflows/cluster-setup-tests.yml index 87d66445..bc8e06a9 100644 --- a/.github/workflows/cluster-setup-tests.yml +++ b/.github/workflows/cluster-setup-tests.yml @@ -20,5 +20,8 @@ jobs: - name: Run cluster_setup unit tests run: bash tools/cluster_setup/test_k0s_cluster_with_stack.sh + - name: Run prerequisite installer unit tests + run: bash tools/cluster_setup/test_installer_prerequisites.sh + - name: Run Splunk-optional installer dry-run tests run: bash tools/cluster_setup/test_installer_dry_run.sh diff --git a/docs/deployment/k0s-quick-reference.md b/docs/deployment/k0s-quick-reference.md index 318b7b37..3c92c6ab 100644 --- a/docs/deployment/k0s-quick-reference.md +++ b/docs/deployment/k0s-quick-reference.md @@ -7,6 +7,18 @@ explanations, diagrams, and edge cases see All commands run from `tools/cluster_setup/` unless noted otherwise. +Clone or download the **complete repository** to the admin workstation first; +do not copy only `k0s_cluster_with_stack.sh`. The installer also needs its +adjacent library, version lock, configuration templates, and helper scripts. + +```bash +git clone https://github.com/splunk/splunk-ai-operator.git +cd splunk-ai-operator/tools/cluster_setup +``` + +If Git is unavailable, use GitHub's **Code → Download ZIP** action, extract the +archive locally, and change into its `tools/cluster_setup` directory. + ## Table of Contents 1. [Prerequisites](#1-prerequisites) diff --git a/tools/cluster_setup/DEPLOYMENT_GUIDE.md b/tools/cluster_setup/DEPLOYMENT_GUIDE.md index 848fab92..c91c9ec0 100644 --- a/tools/cluster_setup/DEPLOYMENT_GUIDE.md +++ b/tools/cluster_setup/DEPLOYMENT_GUIDE.md @@ -502,7 +502,7 @@ graph TD subgraph BIN["📁 binaries/"] K0S["k0s binary\nlatest or --k0s-version"] - YQ["yq v4.44.1\nYAML processor"] + YQ["yq v4.53.3\nYAML processor"] end subgraph IMG["📁 images/ ⭐ pre-loaded OCI image bundles"] diff --git a/tools/cluster_setup/K0S_README.md b/tools/cluster_setup/K0S_README.md index 7d8dcb3d..dbde574e 100644 --- a/tools/cluster_setup/K0S_README.md +++ b/tools/cluster_setup/K0S_README.md @@ -70,7 +70,7 @@ The script installs everything needed for the AI Platform: - **Two-phase parallel installation** — Independent components install concurrently for faster deployments - **Helm retry with exponential backoff** — Automatic retries on transient errors (timeouts, TLS handshake failures) -- **Preflight validation** — Checks tools, config, SSH connectivity, and disk space before starting +- **Prerequisite bootstrap and preflight validation** — Installs missing admin-workstation tools on supported hosts, then checks config, SSH connectivity, and disk space before starting - **Safety gate** — Refuses to wipe a cluster that has Ready nodes (prevents accidental data loss) - **Session logging** — All stdout/stderr captured to `tools/cluster_setup/logs/k0s-install-YYYY-MM-DD_HH-MM-SS.log` - **Existing cluster detection** — `useExisting` flag (auto/force/never) to skip k0s install and deploy stack only @@ -95,24 +95,61 @@ AIPlatform CR → AIService → Job/RayCluster → Pods ### Required Tools (on Admin Workstation) +The `install` command manages its local tool prerequisites automatically. It +checks the admin workstation, installs any missing tools, checks again, and +then continues with the existing configuration and cluster preflight flow. +Tools that are already available are left unchanged. + +The managed tools are `curl`, `ssh`/`scp`, `git`, `jq`, `yq`, `kubectl`, Helm, +`tar`, and the GNU `timeout` utility used by the installer. On macOS, Homebrew +`coreutils` supplies `gtimeout`; the prerequisite module exposes it as +`timeout` from the user-local tool directory. +Model-staging and storage-provider helpers have additional conditional tools +such as Python, `wget`, Git LFS, the AWS CLI, or `mc`; those helpers validate +their own requirements only when the related workflow is enabled. +Downloaded `kubectl`, Helm, `jq`, and `yq` assets are pinned and checksummed in +`prerequisites.lock`. A custom `PREREQ_KUBECTL_VERSION` must be paired with the +matching `PREREQ_KUBECTL_SHA256`. +Automatic installation is supported on these **installer hosts** (this is +separate from the operating-system requirements for cluster nodes): + +| Installer host | Installation method | +|---|---| +| Ubuntu / Debian | `apt` | +| RHEL / Rocky Linux / AlmaLinux / Fedora / Amazon Linux | `dnf` or `yum` | +| macOS | An existing Homebrew installation; the installer does not install Homebrew | + +The script itself requires **Bash 4.4 or newer**. Bash cannot be bootstrapped by +the prerequisite installer because the script must start before that installer +can run. In particular, macOS ships an older `/bin/bash`; install a current +Bash manually and use it to launch the script: + ```bash -# Install required tools on macOS -brew install kubectl helm git jq yq +# macOS only: one-time bootstrap +brew install bash +"$(brew --prefix)/bin/bash" ./k0s_cluster_with_stack.sh prereqs check +``` + +Because this repository keeps the portable `/bin/bash` shebang used by existing +Linux automation, use the Homebrew Bash prefix shown above for every installer +invocation on macOS. -# Install required tools on Ubuntu/Debian -sudo apt-get update -sudo apt-get install -y kubectl helm git jq -wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/local/bin/yq -chmod +x /usr/local/bin/yq +You can inspect or prepare a workstation before an install: -# Verify installations -kubectl version --client -helm version -git --version -jq --version -yq --version +```bash +# Report missing prerequisites without changing the host +./k0s_cluster_with_stack.sh prereqs check + +# Install missing prerequisites and verify them +./k0s_cluster_with_stack.sh prereqs install ``` +Interactive runs may prompt for local `sudo` when the package manager needs +elevated privileges. Silent/non-interactive runs use non-interactive sudo and +fail with remediation instructions if passwordless sudo is unavailable; they +never wait for a password prompt. If host packages are managed separately, use +`install --no-install-prereqs` to retain the fail-on-missing-tools behavior. + ### Hardware Requirements | Node Type | Min CPU | Min RAM | Min Disk | Notes | @@ -158,11 +195,22 @@ You must provide an external S3-compatible object storage endpoint: ### 1. Clone the Repository +Download the **complete repository** to the admin workstation before running +the installer. Do not copy only `k0s_cluster_with_stack.sh`: it uses the +adjacent prerequisite library, version lock, configuration templates, and +helper scripts from this repository. + +If Git is already available: + ```bash git clone https://github.com/splunk/splunk-ai-operator.git cd splunk-ai-operator/tools/cluster_setup ``` +Otherwise, use GitHub's **Code → Download ZIP** action, extract the archive +locally, and change into its `tools/cluster_setup` directory. Run the remaining +quick-start commands from that directory. + ### 2. Create Configuration File ```bash @@ -179,6 +227,13 @@ vi my-cluster.yaml CONFIG_FILE=./my-cluster.yaml ./k0s_cluster_with_stack.sh install ``` +This checks, installs, and rechecks local prerequisites before continuing. To +disable host changes on a centrally managed workstation: + +```bash +CONFIG_FILE=./my-cluster.yaml ./k0s_cluster_with_stack.sh install --no-install-prereqs +``` + ### 4. Verify Installation ```bash @@ -566,6 +621,15 @@ imagePullSecrets: # Install cluster and full AI Platform stack CONFIG_FILE=./my-config.yaml ./k0s_cluster_with_stack.sh install +# Check prerequisites only (does not install anything) +./k0s_cluster_with_stack.sh prereqs check + +# Install any missing prerequisites, then verify them +./k0s_cluster_with_stack.sh prereqs install + +# Install without changing workstation prerequisites +CONFIG_FILE=./my-config.yaml ./k0s_cluster_with_stack.sh install --no-install-prereqs + # Stage model artifacts only (download from HF + upload to object store) CONFIG_FILE=./my-config.yaml ./k0s_cluster_with_stack.sh stage-artifacts @@ -584,6 +648,13 @@ CONFIG_FILE=./my-config.yaml ./k0s_cluster_with_stack.sh join-workers > **Air-gap uses these same commands.** With `cluster.airgap: true` in the config, `install` and `join-workers` stage the offline artifacts first and then continue; every other subcommand runs immediately, unchanged. See [Air-Gapped Deployment](#air-gapped-deployment). +For a normal `install`, prerequisite handling is `check` → install only what is +missing → `check` again. A failure to install or verify a tool stops the run +before cluster changes begin. `validate` and other read-only commands do not +install host packages. `--silent`, `-s`, and `AUTO_APPROVE=true` keep the +prerequisite step non-interactive as well: any required `sudo` is attempted +without a password prompt and fails promptly if authorization is unavailable. + ### Environment Variables #### General @@ -1309,7 +1380,7 @@ cd tools/cluster_setup | Category | Contents | |---|---| -| Binaries | `k0s` (latest stable or `--k0s-version`), `yq v4.44.1` | +| Binaries | `k0s` (latest stable or `--k0s-version`), `yq v4.53.3` | | **Image bundles** (`images/`) | **`k0s-images.tar`** — k0s control-plane images (pause, Calico, kube-proxy, CoreDNS, metrics-server); **`addon-images.tar`** — add-on component images (cert-manager, kube-prometheus-stack, kuberay, MetalLB, OTel, NVIDIA device plugin, busybox). Both built automatically and staged to `/var/lib/k0s/images/` on every node at install time. | | Manifests | `cert-manager v1.13.0`, `local-path-provisioner v0.0.24`, `nvidia-device-plugin v0.17.3` | | Helm charts | `kube-prometheus-stack` (version captured at download time), `opentelemetry-operator` (version captured at download time), `kuberay-operator 1.2.2`, `metallb 0.14.8` | diff --git a/tools/cluster_setup/TROUBLESHOOTING.md b/tools/cluster_setup/TROUBLESHOOTING.md index 73b56621..c6c7c8d7 100644 --- a/tools/cluster_setup/TROUBLESHOOTING.md +++ b/tools/cluster_setup/TROUBLESHOOTING.md @@ -64,16 +64,82 @@ that has Ready nodes. ### "Required tool not found: \" -The install machine is missing a required binary. +The install machine is missing a required binary. A normal `install` first +checks prerequisites, attempts to install anything missing, and checks again. +This message therefore means either automatic installation was disabled with +`--no-install-prereqs` or the installation/recheck did not succeed. + +Inspect the workstation without making changes, or retry only the prerequisite +step: + +```bash +./k0s_cluster_with_stack.sh prereqs check +./k0s_cluster_with_stack.sh prereqs install +``` | Tool | Install | |---|---| | `kubectl` | `brew install kubectl` / [kubernetes.io/docs/tasks/tools](https://kubernetes.io/docs/tasks/tools/) | | `helm` | `brew install helm` / [helm.sh/docs/intro/install](https://helm.sh/docs/intro/install/) | -| `yq` | `brew install yq` / `wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/local/bin/yq && chmod +x /usr/local/bin/yq` | +| `yq` | Prefer `./k0s_cluster_with_stack.sh prereqs install` (pinned and checksummed) / `brew install yq` on a manually managed Mac | | `jq` | `brew install jq` / `apt-get install jq` / `dnf install jq` | | `ssh` | `apt-get install openssh-client` / `brew install openssh` | | `curl` | `apt-get install curl` / `brew install curl` | +| `tar` | Normally provided by the base OS; `apt-get install tar` / `dnf install tar` | +| `timeout` | `apt-get install coreutils` / `dnf install coreutils` / `brew install coreutils` | + +--- + +### "Unsupported installer host" or no supported package manager + +Automatic prerequisite installation is supported on Ubuntu/Debian (`apt`), +RHEL/Rocky Linux/AlmaLinux/Fedora/Amazon Linux (`dnf` or `yum`), and macOS with +Homebrew already installed. This support matrix applies to the admin +workstation, not the remote cluster nodes. + +On another host, install the tools listed by `prereqs check` manually and run: + +```bash +./k0s_cluster_with_stack.sh prereqs check +CONFIG_FILE=./my-cluster.yaml ./k0s_cluster_with_stack.sh install --no-install-prereqs +``` + +On macOS, install Homebrew yourself before using `prereqs install`; the script +does not bootstrap a package manager. + +--- + +### Bash is older than version 4.4 + +The installer requires Bash 4.4 or newer before prerequisite management can +start, so it cannot upgrade Bash for itself. macOS's system `/bin/bash` is too +old. Install a current Bash manually, then invoke the installer with it: + +```bash +brew install bash +"$(brew --prefix)/bin/bash" ./k0s_cluster_with_stack.sh prereqs check +"$(brew --prefix)/bin/bash" ./k0s_cluster_with_stack.sh install +``` + +--- + +### Prerequisite installation cannot obtain sudo + +Linux package installation may require local `sudo`. An interactive install +may display the normal sudo password prompt. A silent/non-interactive install +(`--silent`, `-s`, or `AUTO_APPROVE=true`) uses non-interactive sudo and never +waits for a password; it fails promptly when passwordless authorization is not +available. + +Choose one of these fixes: + +1. Run `prereqs install` interactively from a terminal and authorize sudo. +2. Have an administrator install the tools reported by `prereqs check`. +3. Configure appropriately scoped passwordless sudo for automation, then retry. + +After a manual fix, run `prereqs check` again. Use +`install --no-install-prereqs` only when you intentionally want missing tools +to fail preflight without an installation attempt. --- diff --git a/tools/cluster_setup/airgap_install.sh b/tools/cluster_setup/airgap_install.sh index 07f98583..4525cd4d 100755 --- a/tools/cluster_setup/airgap_install.sh +++ b/tools/cluster_setup/airgap_install.sh @@ -17,7 +17,7 @@ set -euo pipefail # ── Versions (keep in sync with k0s_cluster_with_stack.sh) ───────────────── -YQ_VERSION="v4.44.1" +YQ_VERSION="v4.53.3" CERT_MANAGER_VERSION="v1.13.0" LOCAL_PATH_PROVISIONER_VERSION="v0.0.24" NVIDIA_DEVICE_PLUGIN_VERSION="v0.17.3" diff --git a/tools/cluster_setup/k0s-cluster-config.yaml b/tools/cluster_setup/k0s-cluster-config.yaml index 1b5ac4f2..671df8e7 100644 --- a/tools/cluster_setup/k0s-cluster-config.yaml +++ b/tools/cluster_setup/k0s-cluster-config.yaml @@ -9,14 +9,23 @@ # 3. Replace all values marked with "CHANGE THIS" # 4. Run: CONFIG_FILE=./my-k0s-config.yaml ./k0s_cluster_with_stack.sh install # -# Prerequisites (must be satisfied BEFORE running the installer): +# Prerequisites: # # Local machine: -# ssh, kubectl, helm, git, jq, yq, docker — see the Prerequisites section of -# docs/deployment/k0s-quick-reference.md for install instructions per OS. -# macOS: brew install kubectl helm git jq yq -# RHEL 9: none of kubectl/helm/docker/yq are in the default dnf repos — -# install each via its own official method (git/jq ARE available via dnf). +# - Bash 4.4+ must be installed before this script can run. macOS's /bin/bash +# is too old; install a current Bash manually (for example, brew install bash). +# - During 'install', missing ssh/scp/curl/git/jq/yq/kubectl/helm/tar/timeout +# prerequisites are installed on supported installer hosts, then verified again. +# - Supported installer hosts: Ubuntu/Debian (apt), RHEL/Rocky/Alma/Fedora/ +# Amazon Linux (dnf/yum), and macOS with Homebrew already installed. +# - Check or install them separately with: +# ./k0s_cluster_with_stack.sh prereqs check +# ./k0s_cluster_with_stack.sh prereqs install +# - Use 'install --no-install-prereqs' when the workstation is managed manually. +# - Interactive package installs may prompt for local sudo. Silent installs use +# non-interactive sudo and fail instead of waiting for a password prompt. +# - Docker is needed only for workflows that manually mirror container images; +# see docs/deployment/k0s-quick-reference.md for OS-specific instructions. # # All remote nodes (controllers + workers): # - SSH access using the key at cluster.sshKeyPath as cluster.sshUser diff --git a/tools/cluster_setup/k0s_aws_provision.sh b/tools/cluster_setup/k0s_aws_provision.sh index 336155ad..6fbd1af2 100755 --- a/tools/cluster_setup/k0s_aws_provision.sh +++ b/tools/cluster_setup/k0s_aws_provision.sh @@ -833,33 +833,32 @@ AWSCLI log "AWS CLI ready on installer" fi - log "Installing prerequisites on installer (yq, kubectl, helm, jq)..." - ssh -i "${KEY_LOCAL}" -o StrictHostKeyChecking=no "ec2-user@${eip}" 'bash -s' <<'PREREQ' -set -e -export PATH="$PATH:/usr/local/bin" -sudo dnf install -y git jq curl unzip 2>/dev/null || sudo yum install -y git jq curl unzip -command -v yq &>/dev/null || { - sudo curl -sSL -o /usr/local/bin/yq \ - "https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64" - sudo chmod +x /usr/local/bin/yq -} -command -v kubectl &>/dev/null || { - K8S_VER=$(curl -sSL https://dl.k8s.io/release/stable.txt 2>/dev/null || echo v1.32.0) - sudo curl -sSL -o /usr/local/bin/kubectl \ - "https://dl.k8s.io/release/${K8S_VER}/bin/linux/amd64/kubectl" - sudo chmod +x /usr/local/bin/kubectl -} -command -v helm &>/dev/null || \ - curl -sSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash -echo "Prerequisites ready." -PREREQ - log "Copying k0s cluster scripts to installer..." ssh -i "${KEY_LOCAL}" -o StrictHostKeyChecking=no "ec2-user@${eip}" \ - 'mkdir -p ~/cluster_setup' + 'mkdir -p ~/cluster_setup/lib' scp -i "${KEY_LOCAL}" -o StrictHostKeyChecking=no \ "${SCRIPT_DIR}/"*.sh "${SCRIPT_DIR}/"*.yaml \ "ec2-user@${eip}:~/cluster_setup/" 2>/dev/null || true + scp -i "${KEY_LOCAL}" -o StrictHostKeyChecking=no \ + "${SCRIPT_DIR}/prerequisites.lock" \ + "ec2-user@${eip}:~/cluster_setup/" \ + || err "Failed to copy prerequisites.lock to installer" + scp -i "${KEY_LOCAL}" -o StrictHostKeyChecking=no \ + "${SCRIPT_DIR}/lib/installer_prereqs.sh" \ + "ec2-user@${eip}:~/cluster_setup/lib/" \ + || err "Failed to copy installer_prereqs.sh to installer" + + log "Installing verified prerequisites on installer..." + ssh -i "${KEY_LOCAL}" -o StrictHostKeyChecking=no "ec2-user@${eip}" \ + 'sudo bash -s' <<'PREREQ' +set -euo pipefail +export PATH="/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin" +export PREREQ_INSTALL_DIR="/usr/local/bin" +source /home/ec2-user/cluster_setup/lib/installer_prereqs.sh +prereq_ensure_profile cluster noninteractive +echo "Verified prerequisites ready." +PREREQ + # Copy artifacts_download_upload_scripts (sibling of cluster_setup) — required by model staging step local artifacts_dir="${SCRIPT_DIR}/../artifacts_download_upload_scripts" if [[ -d "${artifacts_dir}" ]]; then diff --git a/tools/cluster_setup/k0s_cluster_with_stack.sh b/tools/cluster_setup/k0s_cluster_with_stack.sh index 9ad5a660..c5065e14 100755 --- a/tools/cluster_setup/k0s_cluster_with_stack.sh +++ b/tools/cluster_setup/k0s_cluster_with_stack.sh @@ -1,4 +1,11 @@ #!/bin/bash + +if (( BASH_VERSINFO[0] < 4 || (BASH_VERSINFO[0] == 4 && BASH_VERSINFO[1] < 4) )); then + echo "ERROR: k0s_cluster_with_stack.sh requires Bash 4.4 or newer (Bash 5 is recommended)." >&2 + echo "On macOS, install Homebrew Bash and invoke this script with that binary." >&2 + exit 1 +fi + set -euo pipefail # ============================================================================= @@ -82,6 +89,40 @@ err() { exit 1 } +# ====== INSTALLER PREREQUISITES ====== +# Source-only module: defining these functions has no package, network, or PATH +# side effects. The install dispatcher opts into check/install/recheck below. +_INSTALLER_SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +_PREREQ_LIBRARY="${_INSTALLER_SCRIPT_DIR}/lib/installer_prereqs.sh" +[[ -r "${_PREREQ_LIBRARY}" ]] || err "Prerequisite module not found: ${_PREREQ_LIBRARY}" +# shellcheck source=lib/installer_prereqs.sh +# shellcheck disable=SC1090,SC1091 +source "${_PREREQ_LIBRARY}" + +ensure_installer_prerequisites() { + local profile="$1" interaction="interactive" rc=0 + [[ "${SILENT_INSTALL:-false}" == "true" || "${AUTO_APPROVE:-false}" == "true" || ! -t 0 ]] \ + && interaction="noninteractive" + + if [[ "${INSTALL_PREREQS:-true}" == "true" ]]; then + prereq_ensure_profile "${profile}" "${interaction}" || rc=$? + else + prereq_check_profile "${profile}" || rc=$? + fi + + if (( rc != 0 )); then + if [[ "${INSTALL_PREREQS:-true}" == "true" ]]; then + err "Unable to prepare installer prerequisites (${profile} profile, status ${rc}): + ${PREREQ_LAST_ERROR} +Run '$0 prereqs install' after fixing the reported host issue, then retry." + else + err "Required installer tools are unavailable and automatic installation is disabled: + ${PREREQ_LAST_ERROR} +Run '$0 prereqs install', or omit --no-install-prereqs, then retry." + fi + fi +} + # ====== TOOL CHECKER ====== # Provides install instructions instead of a bare "missing in PATH" message. need() { @@ -476,63 +517,8 @@ show_install_plan() { # ====== LOAD CONFIGURATION ====== -ensure_yq() { - command -v yq >/dev/null 2>&1 && return 0 - local os arch url - local -a _yq_fetch - # Pinned version — matches download_from_huggingface.sh; update both together. - local YQ_VERSION="v4.44.1" - os="$(uname -s)" - arch="$(uname -m)" - case "${arch}" in - x86_64|amd64) arch="amd64" ;; - aarch64|arm64) arch="arm64" ;; - *) warn "yq auto-install: unsupported arch ${arch}, skipping"; return 1 ;; - esac - case "${os}" in - Linux) - url="${YQ_DOWNLOAD_URL:-https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/yq_linux_${arch}}" - log "Installing yq ${YQ_VERSION} (linux-${arch})..." - # file:// is a URI — curl rejects a relative path in it, so copy instead. - if [[ "${url}" == file://* ]]; then - _yq_fetch=( cp -- "${url#file://}" /tmp/yq ) - else - _yq_fetch=( curl -fsSL -o /tmp/yq "${url}" ) - fi - if "${_yq_fetch[@]}"; then - chmod +x /tmp/yq - if [[ "$(id -u)" -eq 0 ]]; then - mv /tmp/yq /usr/local/bin/yq - else - sudo mv /tmp/yq /usr/local/bin/yq 2>/dev/null || { mkdir -p ~/.local/bin; mv /tmp/yq ~/.local/bin/yq; export PATH="$PATH:$HOME/.local/bin"; } - fi - else - warn "yq download failed — config parsing may be unreliable"; return 1 - fi - ;; - Darwin) - if command -v brew >/dev/null 2>&1; then - log "Installing yq via brew..." - brew install yq - else - url="https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/yq_darwin_${arch}" - log "Installing yq ${YQ_VERSION} (darwin-${arch})..." - if curl -fsSL -o /tmp/yq "${url}"; then - chmod +x /tmp/yq - sudo mv /tmp/yq /usr/local/bin/yq 2>/dev/null || { mkdir -p ~/.local/bin; mv /tmp/yq ~/.local/bin/yq; export PATH="$PATH:$HOME/.local/bin"; } - else - warn "yq download failed — config parsing may be unreliable"; return 1 - fi - fi - ;; - *) warn "yq auto-install: unsupported OS ${os}, skipping"; return 1 ;; - esac - command -v yq >/dev/null 2>&1 && log "yq installed: $(yq --version 2>/dev/null)" || warn "yq install succeeded but binary not found in PATH" -} - load_config() { - ensure_yq || true - command -v yq >/dev/null 2>&1 || err "yq is required to parse ${CONFIG_FILE}. Install it (brew install yq / snap install yq) and retry." + command -v yq >/dev/null 2>&1 || err "yq is required to parse ${CONFIG_FILE}. Run '$0 prereqs install' and retry." log "Loading configuration from: ${CONFIG_FILE}" [[ -f "${CONFIG_FILE}" ]] || err "Config file not found: ${CONFIG_FILE}" @@ -1077,12 +1063,11 @@ preflight_checks() { fi done - # yq is strongly recommended — without it, config parsing falls back to - # grep/awk which cannot handle arrays or nested structures reliably. + # yq is mandatory: load_config uses structure-aware v4 expressions. if command -v yq >/dev/null 2>&1; then pf_ok "yq found" else - pf_warn "yq not found — install it for reliable config parsing (brew install yq / snap install yq). Falling back to grep/awk which may miss complex config values." + pf_fail "yq not found in PATH" fi # python3 is used by preflight_check_registry() to parse Bearer token JSON. @@ -1444,9 +1429,11 @@ preflight_check_registry() { _www_auth=$(curl "${_basic_opts[@]}" -o /dev/null -D - "${_murl}" 2>/dev/null \ | grep -i '^Www-Authenticate:' | head -1) - _realm=$(echo "${_www_auth}" | grep -oP 'realm="[^"]+"' | cut -d'"' -f2) - _service=$(echo "${_www_auth}" | grep -oP 'service="[^"]+"' | cut -d'"' -f2) - _scope=$(echo "${_www_auth}" | grep -oP 'scope="[^"]+"' | cut -d'"' -f2) + # BSD grep on macOS has no -P; these POSIX sed expressions extract the + # quoted Bearer challenge attributes on every supported installer host. + _realm=$(printf '%s\n' "${_www_auth}" | sed -n 's/.*realm="\([^"]*\)".*/\1/p') + _service=$(printf '%s\n' "${_www_auth}" | sed -n 's/.*service="\([^"]*\)".*/\1/p') + _scope=$(printf '%s\n' "${_www_auth}" | sed -n 's/.*scope="\([^"]*\)".*/\1/p') [[ -z "${_realm}" ]] && { echo "401"; return; } @@ -5462,13 +5449,21 @@ install_ai_platform_cr() { log "Cleaning up failed jobs and ImagePullBackOff pods from previous runs..." kubectl delete jobs -n "${AI_NS}" --field-selector status.successful=0 --wait=false 2>/dev/null || true kubectl delete pods -n "${AI_NS}" --field-selector status.phase=Failed --wait=false 2>/dev/null || true - # Delete pods stuck in ImagePullBackOff or ErrImagePull (use jq to avoid bash 3.x jsonpath parsing issues) - kubectl get pods -n "${AI_NS}" -o json 2>/dev/null | \ - jq -r '.items[] | select(.status.containerStatuses[]? | .state.waiting?.reason? == "ImagePullBackOff") | .metadata.name' 2>/dev/null | \ - xargs -r -I {} kubectl delete pod {} -n "${AI_NS}" --wait=false --grace-period=0 --force 2>/dev/null || true - kubectl get pods -n "${AI_NS}" -o json 2>/dev/null | \ - jq -r '.items[] | select(.status.containerStatuses[]? | .state.waiting?.reason? == "ErrImagePull") | .metadata.name' 2>/dev/null | \ - xargs -r -I {} kubectl delete pod {} -n "${AI_NS}" --wait=false --grace-period=0 --force 2>/dev/null || true + # Delete pods stuck in ImagePullBackOff or ErrImagePull. Use a shell loop + # instead of GNU-specific `xargs -r` so this also works on macOS. + local image_pull_reason image_pull_pod + for image_pull_reason in ImagePullBackOff ErrImagePull; do + while IFS= read -r image_pull_pod; do + [[ -n "${image_pull_pod}" ]] || continue + kubectl delete pod "${image_pull_pod}" -n "${AI_NS}" \ + --wait=false --grace-period=0 --force 2>/dev/null || true + done < <( + kubectl get pods -n "${AI_NS}" -o json 2>/dev/null | \ + jq -r --arg reason "${image_pull_reason}" \ + '.items[] | select(.status.containerStatuses[]? | .state.waiting?.reason? == $reason) | .metadata.name' \ + 2>/dev/null + ) + done log "✓ Cleanup complete" # Build trustedIssuers YAML fragment from config (splunk.trustedIssuers[]). @@ -7399,6 +7394,10 @@ main_install() { show_install_plan + # The normal install path is check -> install missing -> recheck. Prepared + # hosts take the no-op path, so the deployment phases below remain unchanged. + ensure_installer_prerequisites cluster + phase_start "Preflight" step_start "Preflight checks" preflight_checks @@ -7999,14 +7998,17 @@ validate_config() { # ====== USAGE ====== usage() { cat </dev/null || true +# Automatic prerequisite installation is the install default. Parse these flags +# before air-gap delegation so the wrapper and its callback inherit the same +# silent/opt-out policy through exported environment variables. +INSTALL_PREREQS="${INSTALL_PREREQS:-true}" +if [[ "${_CMD}" == "install" ]]; then + while [[ $# -gt 0 ]]; do + case "$1" in + --silent|-s) SILENT_INSTALL=true; shift ;; + --no-install-prereqs) INSTALL_PREREQS=false; shift ;; + *) echo "Unknown install option: $1" >&2; usage >&2; exit 1 ;; + esac + done + export SILENT_INSTALL="${SILENT_INSTALL:-false}" INSTALL_PREREQS + [[ -f "${CONFIG_FILE}" ]] || err "Config file not found: ${CONFIG_FILE}" + ensure_installer_prerequisites bootstrap +fi + # ====== AIR-GAP DELEGATION ====== # One command serves both modes: with cluster.airgap: true the artifacts must be # downloaded before anything can be pushed to the sealed nodes, so hand off to @@ -8414,6 +8448,9 @@ if [[ "${AIRGAP_STAGED:-false}" != "true" ]]; then # AIRGAP_MODE=true is an equally valid trigger: it is the documented way to # request air-gap for one run without editing the config. if [[ "${_ag}" == "true" || "${AIRGAP_MODE:-false}" == "true" ]]; then + # airgap_install.sh itself needs Helm/tar/curl on the connected installer + # host, so prepare the complete local profile before handing off. + [[ "${_CMD}" == "install" ]] && ensure_installer_prerequisites cluster if [[ ! -x "${_sd}/airgap_install.sh" ]]; then echo "ERROR: cluster.airgap is true but ${_sd}/airgap_install.sh is missing." >&2 echo " That script downloads the artifacts a sealed cluster cannot fetch itself." >&2 @@ -8439,14 +8476,32 @@ fi case "${_CMD}" in install) - while [[ $# -gt 0 ]]; do - case "$1" in - --silent|-s) SILENT_INSTALL=true; shift ;; - *) echo "Unknown install option: $1" >&2; usage >&2; exit 1 ;; - esac - done main_install ;; + prereqs) + [[ $# -eq 1 ]] || { echo "Usage: $0 prereqs check|install" >&2; exit 2; } + case "$1" in + check) + _prereq_rc=0 + prereq_check_profile cluster || _prereq_rc=$? + if (( _prereq_rc != 0 )); then + echo "Prerequisite check failed: ${PREREQ_LAST_ERROR}" >&2 + fi + exit "${_prereq_rc}" + ;; + install) + _prereq_interaction="interactive" + [[ "${AUTO_APPROVE:-false}" == "true" || ! -t 0 ]] && _prereq_interaction="noninteractive" + _prereq_rc=0 + prereq_ensure_profile cluster "${_prereq_interaction}" || _prereq_rc=$? + if (( _prereq_rc != 0 )); then + echo "Prerequisite installation failed: ${PREREQ_LAST_ERROR}" >&2 + fi + exit "${_prereq_rc}" + ;; + *) echo "Usage: $0 prereqs check|install" >&2; exit 2 ;; + esac + ;; validate) validate_config ;; diff --git a/tools/cluster_setup/lib/installer_prereqs.sh b/tools/cluster_setup/lib/installer_prereqs.sh new file mode 100644 index 00000000..de7ed19f --- /dev/null +++ b/tools/cluster_setup/lib/installer_prereqs.sh @@ -0,0 +1,739 @@ +#!/usr/bin/env bash +# OS-aware prerequisite management for k0s_cluster_with_stack.sh. +# +# This file is a source-only library. It intentionally performs no work when +# sourced, never changes the caller's shell options, and never exits the caller. +# Public functions return a stable status and set PREREQ_LAST_ERROR on failure. + +PREREQ_RC_MISSING=10 +PREREQ_RC_UNSUPPORTED=20 +PREREQ_RC_PRIVILEGE=21 +PREREQ_RC_PACKAGE=22 +PREREQ_RC_DOWNLOAD=23 +PREREQ_RC_CHECKSUM=24 +PREREQ_RC_VERSION=25 + +PREREQ_OS="" +PREREQ_DISTRO="" +PREREQ_OS_FAMILY="" +PREREQ_ARCH="" +PREREQ_PACKAGE_MANAGER="" +PREREQ_LAST_ERROR="" +PREREQ_LOCK_LOADED="false" +PREREQ_APT_UPDATED="false" +PREREQ_RESOLVED_KUBECTL_VERSION="" +PREREQ_KUBECTL_PINNED_VERSION="${PREREQ_KUBECTL_PINNED_VERSION:-}" +PREREQ_HELM_VERSION="${PREREQ_HELM_VERSION:-}" +PREREQ_JQ_VERSION="${PREREQ_JQ_VERSION:-}" +PREREQ_YQ_VERSION="${PREREQ_YQ_VERSION:-}" +declare -a PREREQ_PROFILE_TOOLS=() +declare -a PREREQ_MISSING_TOOLS=() +declare -a PREREQ_LOCAL_TOOLS_TO_ACTIVATE=() + +prereq_log() { + if declare -F log >/dev/null 2>&1; then + log "Prerequisites: $*" + else + printf '[prerequisites] %s\n' "$*" >&2 + fi +} + +prereq_warn() { + if declare -F warn >/dev/null 2>&1; then + warn "Prerequisites: $*" + else + printf '[prerequisites] WARNING: %s\n' "$*" >&2 + fi +} + +prereq_fail() { + PREREQ_LAST_ERROR="$1" + return "${2:-1}" +} + +# External-boundary wrappers. Tests override these instead of mocking internal +# orchestration. Commands are always passed as argument arrays; no eval is used. +prereq_command_exists() { command -v "$1" >/dev/null 2>&1; } +prereq_command_path() { command -v "$1"; } +prereq_exec() { "$@"; } +prereq_effective_uid() { id -u; } +prereq_uname_os() { uname -s; } +prereq_uname_arch() { uname -m; } + +prereq_download() { + local url="$1" destination="$2" + if [[ "${url}" == file://* ]]; then + prereq_exec cp -- "${url#file://}" "${destination}" + else + prereq_exec curl --fail --location --silent --show-error \ + --retry 3 --connect-timeout 15 --output "${destination}" "${url}" + fi +} + +prereq_verify_checksum() { + local file="$1" expected="$2" actual="" + [[ "${expected}" =~ ^[[:xdigit:]]{64}$ ]] || return 1 + if prereq_command_exists sha256sum; then + actual="$(sha256sum "${file}" | awk '{print $1}')" + elif prereq_command_exists shasum; then + actual="$(shasum -a 256 "${file}" | awk '{print $1}')" + elif prereq_command_exists openssl; then + actual="$(openssl dgst -sha256 "${file}" | awk '{print $NF}')" + else + return 1 + fi + [[ "${actual,,}" == "${expected,,}" ]] +} + +prereq_load_lock() { + [[ "${PREREQ_LOCK_LOADED}" == "true" ]] && return 0 + + local module_dir lock_file line key value + module_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + lock_file="${PREREQ_LOCK_FILE:-${module_dir}/../prerequisites.lock}" + [[ -r "${lock_file}" ]] || prereq_fail \ + "version lock is missing or unreadable: ${lock_file}" "${PREREQ_RC_VERSION}" || return $? + + while IFS= read -r line || [[ -n "${line}" ]]; do + [[ -z "${line}" || "${line}" == \#* ]] && continue + if [[ ! "${line}" =~ ^(PREREQ_[A-Z0-9_]+)=\"([^\"]*)\"$ ]]; then + prereq_fail "invalid entry in ${lock_file}: ${line}" "${PREREQ_RC_VERSION}" + return $? + fi + key="${BASH_REMATCH[1]}" + value="${BASH_REMATCH[2]}" + printf -v "${key}" '%s' "${value}" + done < "${lock_file}" + + PREREQ_LOCK_LOADED="true" +} + +prereq_activate_local_tools() { + local install_dir="${PREREQ_INSTALL_DIR:-${XDG_BIN_HOME:-${HOME}/.local/bin}}" + local runtime_dir="${PREREQ_RUNTIME_BIN_DIR:-${install_dir}/.splunk-ai-prereq-bin}" + local tool target + local -a managed_tools=(kubectl helm jq yq timeout) + + (( $# > 0 )) || return 0 + mkdir -p "${runtime_dir}" || { + prereq_fail "cannot create prerequisite runtime directory: ${runtime_dir}" "${PREREQ_RC_PACKAGE}" + return $? + } + chmod 0700 "${runtime_dir}" || return "${PREREQ_RC_PACKAGE}" + + # This module owns only symlinks in its dot-directory. Preserve verified + # links from an earlier bootstrap phase, but remove stale links before this + # directory is placed first in PATH. + for tool in "${managed_tools[@]}"; do + target="${runtime_dir}/${tool}" + if [[ -e "${target}" && ! -L "${target}" ]]; then + prereq_fail "refusing to replace non-symlink prerequisite runtime entry: ${target}" "${PREREQ_RC_PACKAGE}" + return $? + fi + if [[ -L "${target}" ]] && ! prereq_owned_tool_is_compatible "${tool}"; then + rm -f -- "${target}" + fi + done + + for tool in "$@"; do + target="${install_dir}/${tool}" + [[ -x "${target}" ]] || { + prereq_fail "verified local prerequisite disappeared: ${target}" "${PREREQ_RC_VERSION}" + return $? + } + rm -f -- "${runtime_dir}/${tool}" + ln -s "${target}" "${runtime_dir}/${tool}" || return "${PREREQ_RC_PACKAGE}" + done + + case ":${PATH}:" in + *":${runtime_dir}:"*) ;; + *) export PATH="${runtime_dir}:${PATH}" ;; + esac + + # A long-lived Bash process may have cached an earlier command location. + # Clear that cache so subsequent tool invocations use the verified runtime + # links that were just placed first in PATH. + hash -r 2>/dev/null || true +} + +prereq_read_linux_identity() { + local release_file="${PREREQ_OS_RELEASE_FILE:-/etc/os-release}" + local line key value id="" id_like="" + [[ -r "${release_file}" ]] || return 1 + while IFS= read -r line || [[ -n "${line}" ]]; do + [[ "${line}" == *=* ]] || continue + key="${line%%=*}" + value="${line#*=}" + value="${value#\"}"; value="${value%\"}" + value="${value#\'}"; value="${value%\'}" + case "${key}" in + ID) id="${value}" ;; + ID_LIKE) id_like="${value}" ;; + esac + done < "${release_file}" + PREREQ_DISTRO="${PREREQ_DISTRO_OVERRIDE:-${id}}" + printf '%s\n' "${PREREQ_DISTRO}|${id_like}" +} + +prereq_detect_platform() { + PREREQ_LAST_ERROR="" + local raw_os raw_arch identity id_like + raw_os="${PREREQ_OS_OVERRIDE:-$(prereq_uname_os)}" + raw_arch="${PREREQ_ARCH_OVERRIDE:-$(prereq_uname_arch)}" + + case "${raw_arch}" in + x86_64|amd64) PREREQ_ARCH="amd64" ;; + arm64|aarch64) PREREQ_ARCH="arm64" ;; + *) prereq_fail "unsupported installer architecture: ${raw_arch}" "${PREREQ_RC_UNSUPPORTED}"; return $? ;; + esac + + case "${raw_os}" in + Linux|linux) + PREREQ_OS="linux" + identity="$(prereq_read_linux_identity)" || { + prereq_fail "cannot read installer OS identity from ${PREREQ_OS_RELEASE_FILE:-/etc/os-release}" "${PREREQ_RC_UNSUPPORTED}" + return $? + } + PREREQ_DISTRO="${identity%%|*}" + id_like="${identity#*|}" + if [[ -n "${PREREQ_OS_FAMILY_OVERRIDE:-}" ]]; then + PREREQ_OS_FAMILY="${PREREQ_OS_FAMILY_OVERRIDE}" + else + case "${PREREQ_DISTRO}" in + ubuntu|debian) PREREQ_OS_FAMILY="debian" ;; + rhel|centos|rocky|almalinux|fedora|amzn) PREREQ_OS_FAMILY="rhel" ;; + *) + if [[ " ${id_like} " == *" debian "* ]]; then + PREREQ_OS_FAMILY="debian" + elif [[ " ${id_like} " == *" rhel "* || " ${id_like} " == *" fedora "* ]]; then + PREREQ_OS_FAMILY="rhel" + else + prereq_fail "unsupported installer Linux distribution: ${PREREQ_DISTRO:-unknown}" "${PREREQ_RC_UNSUPPORTED}" + return $? + fi + ;; + esac + fi + case "${PREREQ_OS_FAMILY}" in + debian) + prereq_command_exists apt-get || { + prereq_fail "apt-get is required on ${PREREQ_DISTRO}" "${PREREQ_RC_UNSUPPORTED}" + return $? + } + PREREQ_PACKAGE_MANAGER="apt-get" + ;; + rhel) + if prereq_command_exists dnf; then + PREREQ_PACKAGE_MANAGER="dnf" + elif prereq_command_exists yum; then + PREREQ_PACKAGE_MANAGER="yum" + else + prereq_fail "dnf or yum is required on ${PREREQ_DISTRO}" "${PREREQ_RC_UNSUPPORTED}" + return $? + fi + ;; + *) prereq_fail "unsupported installer OS family: ${PREREQ_OS_FAMILY}" "${PREREQ_RC_UNSUPPORTED}"; return $? ;; + esac + ;; + Darwin|darwin|macos) + PREREQ_OS="darwin" + PREREQ_DISTRO="macos" + PREREQ_OS_FAMILY="macos" + if prereq_command_exists brew; then + PREREQ_PACKAGE_MANAGER="brew" + else + PREREQ_PACKAGE_MANAGER="none" + fi + ;; + *) prereq_fail "unsupported installer operating system: ${raw_os}" "${PREREQ_RC_UNSUPPORTED}"; return $? ;; + esac +} + +prereq_profile_tools() { + local profile="$1" + PREREQ_PROFILE_TOOLS=() + case "${profile}" in + bootstrap) + PREREQ_PROFILE_TOOLS=(curl yq) + ;; + cluster) + # git remains here because the existing preflight requires it. It can be + # moved to a model-staging profile when that preflight becomes conditional. + PREREQ_PROFILE_TOOLS=(ssh scp curl kubectl helm git jq yq tar timeout) + ;; + *) prereq_fail "unknown prerequisite profile: ${profile}" "${PREREQ_RC_UNSUPPORTED}"; return $? ;; + esac +} + +prereq_kubectl_client_version() { + local binary="$1" output version + output="$(prereq_exec "${binary}" version --client --output=yaml 2>/dev/null)" || return 1 + version="$(printf '%s\n' "${output}" | awk '$1 == "gitVersion:" {gsub(/"/, "", $2); print $2; exit}')" + [[ "${version}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+([+-].*)?$ ]] || return 1 + printf '%s\n' "${version}" +} + +prereq_tool_is_compatible() { + local tool="$1" output="" actual_version="" required_version="" + local actual_core required_core actual_major actual_minor required_major required_minor + prereq_command_exists "${tool}" || return 1 + + case "${tool}" in + yq) + output="$(prereq_exec yq --version 2>/dev/null)" || return 1 + [[ "${output}" == *mikefarah* && "${output}" =~ version[[:space:]]+v?4\. ]] + ;; + helm) + output="$(prereq_exec helm version --short 2>/dev/null)" || return 1 + [[ "${output}" =~ ^v3\. ]] + ;; + jq) + output="$(prereq_exec jq --version 2>/dev/null)" || return 1 + [[ "${output}" =~ ^jq-1\.([6-9]|[1-9][0-9])([.-]|$) ]] + ;; + kubectl) + actual_version="$(prereq_kubectl_client_version kubectl)" || return 1 + prereq_resolve_kubectl_version || return $? + required_version="${PREREQ_RESOLVED_KUBECTL_VERSION}" + actual_core="${actual_version#v}"; actual_core="${actual_core%%[+-]*}" + required_core="${required_version#v}"; required_core="${required_core%%[+-]*}" + IFS=. read -r actual_major actual_minor _ <<< "${actual_core}" + IFS=. read -r required_major required_minor _ <<< "${required_core}" + [[ "${actual_major}" =~ ^[0-9]+$ && "${actual_minor}" =~ ^[0-9]+$ \ + && "${required_major}" =~ ^[0-9]+$ && "${required_minor}" =~ ^[0-9]+$ ]] || return 1 + (( actual_major == required_major \ + && actual_minor >= required_minor - 1 \ + && actual_minor <= required_minor + 1 )) + ;; + *) + return 0 + ;; + esac +} + +prereq_collect_missing_tools() { + local profile="$1" tool rc=0 + prereq_profile_tools "${profile}" || return $? + PREREQ_MISSING_TOOLS=() + for tool in "${PREREQ_PROFILE_TOOLS[@]}"; do + rc=0 + prereq_tool_is_compatible "${tool}" || rc=$? + if (( rc == 1 )); then + PREREQ_MISSING_TOOLS+=("${tool}") + elif (( rc != 0 )); then + return "${rc}" + fi + done +} + +prereq_check_profile() { + local profile="${1:-cluster}" + prereq_collect_missing_tools "${profile}" || return $? + if (( ${#PREREQ_MISSING_TOOLS[@]} == 0 )); then + prereq_log "${profile} profile is satisfied" + return 0 + fi + PREREQ_LAST_ERROR="missing or incompatible ${profile} tools: ${PREREQ_MISSING_TOOLS[*]}" + prereq_warn "${PREREQ_LAST_ERROR}" + return "${PREREQ_RC_MISSING}" +} + +prereq_run_as_root() { + if [[ "$(prereq_effective_uid)" == "0" ]]; then + prereq_exec "$@" + return $? + fi + prereq_command_exists sudo || { + prereq_fail "sudo is required to install operating-system packages" "${PREREQ_RC_PRIVILEGE}" + return $? + } + if [[ "${PREREQ_NONINTERACTIVE:-false}" == "true" ]]; then + prereq_exec sudo -n true >/dev/null 2>&1 || { + prereq_fail "passwordless sudo is required for non-interactive prerequisite installation" "${PREREQ_RC_PRIVILEGE}" + return $? + } + prereq_exec sudo -n "$@" + else + prereq_exec sudo -v || { + prereq_fail "sudo authorization failed while installing prerequisites" "${PREREQ_RC_PRIVILEGE}" + return $? + } + prereq_exec sudo "$@" + fi +} + +prereq_append_unique() { + local value="$1" existing + shift + for existing in "$@"; do + [[ "${existing}" == "${value}" ]] && return 1 + done + return 0 +} + +prereq_install_debian_packages() { + local tool package rc=0 + local -a packages=() + for tool in "$@"; do + case "${tool}" in + curl) package="curl"; prereq_append_unique ca-certificates "${packages[@]}" && packages+=(ca-certificates) ;; + ssh|scp) package="openssh-client" ;; + git) package="git" ;; + tar) package="tar" ;; + timeout) package="coreutils" ;; + *) continue ;; + esac + prereq_append_unique "${package}" "${packages[@]}" && packages+=("${package}") + done + (( ${#packages[@]} > 0 )) || return 0 + if [[ "${PREREQ_APT_UPDATED}" != "true" ]]; then + rc=0 + prereq_run_as_root env DEBIAN_FRONTEND=noninteractive apt-get update || rc=$? + if (( rc != 0 )); then + (( rc == PREREQ_RC_PRIVILEGE )) && return "${rc}" + PREREQ_LAST_ERROR="apt-get update failed" + return "${PREREQ_RC_PACKAGE}" + fi + PREREQ_APT_UPDATED="true" + fi + rc=0 + prereq_run_as_root env DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends "${packages[@]}" || rc=$? + if (( rc != 0 )); then + (( rc == PREREQ_RC_PRIVILEGE )) && return "${rc}" + PREREQ_LAST_ERROR="apt-get could not install: ${packages[*]}" + return "${PREREQ_RC_PACKAGE}" + fi +} + +prereq_install_rhel_packages() { + local tool package rc=0 + local -a packages=() + for tool in "$@"; do + case "${tool}" in + curl) package="curl"; prereq_append_unique ca-certificates "${packages[@]}" && packages+=(ca-certificates) ;; + ssh|scp) package="openssh-clients" ;; + git) package="git" ;; + tar) package="tar" ;; + timeout) package="coreutils" ;; + *) continue ;; + esac + prereq_append_unique "${package}" "${packages[@]}" && packages+=("${package}") + done + (( ${#packages[@]} > 0 )) || return 0 + prereq_run_as_root "${PREREQ_PACKAGE_MANAGER}" install -y "${packages[@]}" || rc=$? + if (( rc != 0 )); then + (( rc == PREREQ_RC_PRIVILEGE )) && return "${rc}" + PREREQ_LAST_ERROR="${PREREQ_PACKAGE_MANAGER} could not install: ${packages[*]}" + return "${PREREQ_RC_PACKAGE}" + fi +} + +prereq_install_macos_packages() { + local tool package install_dir gtimeout_path + local -a packages=() + for tool in "$@"; do + case "${tool}" in + curl) package="curl" ;; + ssh|scp) package="openssh" ;; + git) package="git" ;; + timeout) package="coreutils" ;; + tar) package="gnu-tar" ;; + *) continue ;; + esac + prereq_append_unique "${package}" "${packages[@]}" && packages+=("${package}") + done + (( ${#packages[@]} > 0 )) || return 0 + [[ "${PREREQ_PACKAGE_MANAGER}" == "brew" ]] || { + prereq_fail "Homebrew must already be installed to add missing macOS prerequisites: ${packages[*]}" "${PREREQ_RC_PACKAGE}" + return $? + } + prereq_exec brew install "${packages[@]}" || { + prereq_fail "Homebrew could not install: ${packages[*]}" "${PREREQ_RC_PACKAGE}" + return $? + } + + # Homebrew coreutils intentionally prefixes GNU timeout with 'g'. Provide a + # run-local user shim without editing shell startup files. + if ! prereq_command_exists timeout && prereq_command_exists gtimeout; then + install_dir="${PREREQ_INSTALL_DIR:-${XDG_BIN_HOME:-${HOME}/.local/bin}}" + mkdir -p "${install_dir}" || { + prereq_fail "cannot create prerequisite install directory: ${install_dir}" "${PREREQ_RC_PACKAGE}" + return $? + } + gtimeout_path="$(prereq_command_path gtimeout)" || { + prereq_fail "Homebrew installed coreutils but gtimeout is not available in PATH" "${PREREQ_RC_VERSION}" + return $? + } + prereq_exec ln -sf "${gtimeout_path}" "${install_dir}/timeout" || { + prereq_fail "could not create the user-local timeout shim in ${install_dir}" "${PREREQ_RC_PACKAGE}" + return $? + } + PREREQ_LOCAL_TOOLS_TO_ACTIVATE+=(timeout) + fi +} + +prereq_install_native_packages() { + case "${PREREQ_OS_FAMILY}" in + debian) prereq_install_debian_packages "$@" ;; + rhel) prereq_install_rhel_packages "$@" ;; + macos) prereq_install_macos_packages "$@" ;; + *) prereq_fail "no native prerequisite installer for ${PREREQ_OS_FAMILY:-unknown}" "${PREREQ_RC_UNSUPPORTED}"; return $? ;; + esac +} + +prereq_checksum_for() { + local tool="$1" key + key="PREREQ_${tool^^}_${PREREQ_OS^^}_${PREREQ_ARCH^^}_SHA256" + printf '%s' "${!key:-}" +} + +prereq_resolve_kubectl_version() { + PREREQ_RESOLVED_KUBECTL_VERSION="" + prereq_load_lock || return $? + if [[ -n "${PREREQ_KUBECTL_VERSION:-}" ]]; then + PREREQ_RESOLVED_KUBECTL_VERSION="${PREREQ_KUBECTL_VERSION}" + elif [[ "${K0S_VERSION:-}" =~ ^v?([0-9]+\.[0-9]+\.[0-9]+)\+k0s ]]; then + PREREQ_RESOLVED_KUBECTL_VERSION="v${BASH_REMATCH[1]}" + else + PREREQ_RESOLVED_KUBECTL_VERSION="${PREREQ_KUBECTL_PINNED_VERSION}" + fi + [[ "${PREREQ_RESOLVED_KUBECTL_VERSION}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]] || { + prereq_fail "invalid kubectl version: ${PREREQ_RESOLVED_KUBECTL_VERSION:-empty}" "${PREREQ_RC_VERSION}" + return $? + } +} + +prereq_install_atomic() { + local source="$1" tool="$2" + local install_dir="${PREREQ_INSTALL_DIR:-${XDG_BIN_HOME:-${HOME}/.local/bin}}" + local destination="${install_dir}/${tool}" staged="${install_dir}/.${tool}.prereq.$$" + mkdir -p "${install_dir}" || { + prereq_fail "cannot create prerequisite install directory: ${install_dir}" "${PREREQ_RC_PACKAGE}" + return $? + } + if cp -- "${source}" "${staged}" \ + && chmod 0755 "${staged}" \ + && mv -f -- "${staged}" "${destination}"; then + return 0 + fi + rm -f -- "${staged}" + prereq_fail "could not install ${tool} into ${install_dir}" "${PREREQ_RC_PACKAGE}" + return $? +} + +prereq_verify_installed_binary() { + local tool="$1" path="$2" output="" actual_version="" + [[ -x "${path}" ]] || return 1 + case "${tool}" in + kubectl) + actual_version="$(prereq_kubectl_client_version "${path}")" || return 1 + [[ "${actual_version}" == "${PREREQ_RESOLVED_KUBECTL_VERSION}" ]] + ;; + helm) + output="$("${path}" version --short 2>/dev/null)" || return 1 + [[ "${output}" == "${PREREQ_HELM_VERSION}"* ]] + ;; + jq) + output="$("${path}" --version 2>/dev/null)" || return 1 + [[ "${output}" == "jq-${PREREQ_JQ_VERSION}" ]] + ;; + yq) + output="$("${path}" --version 2>/dev/null)" || return 1 + [[ "${output}" == *mikefarah* && "${output}" == *"version ${PREREQ_YQ_VERSION}"* ]] + ;; + *) return 1 ;; + esac +} + +prereq_owned_tool_is_compatible() { + local tool="$1" + local install_dir="${PREREQ_INSTALL_DIR:-${XDG_BIN_HOME:-${HOME}/.local/bin}}" + local path="${install_dir}/${tool}" output="" + [[ -x "${path}" ]] || return 1 + + case "${tool}" in + kubectl) + prereq_resolve_kubectl_version || return $? + prereq_verify_installed_binary "${tool}" "${path}" + ;; + helm|jq|yq) + prereq_load_lock || return $? + prereq_verify_installed_binary "${tool}" "${path}" + ;; + timeout) + output="$("${path}" --version 2>/dev/null)" || return 1 + [[ "${output}" == *"GNU coreutils"* ]] + ;; + *) + return 1 + ;; + esac +} + +prereq_install_verified_binary() { + local tool="$1" url="" expected="" asset_name="" archive="false" + local version="" tmp_dir="" payload="" source_binary="" install_dir="" old_umask="" install_rc=0 asset_os="" + prereq_load_lock || return $? + + case "${tool}" in + kubectl) + prereq_resolve_kubectl_version || return $? + version="${PREREQ_RESOLVED_KUBECTL_VERSION}" + asset_name="kubectl" + url="${PREREQ_KUBECTL_URL:-https://dl.k8s.io/release/${version}/bin/${PREREQ_OS}/${PREREQ_ARCH}/kubectl}" + if [[ -n "${PREREQ_KUBECTL_SHA256:-}" ]]; then + expected="${PREREQ_KUBECTL_SHA256}" + elif [[ "${version}" == "${PREREQ_KUBECTL_PINNED_VERSION}" ]]; then + expected="$(prereq_checksum_for KUBECTL)" + fi + ;; + helm) + version="${PREREQ_HELM_VERSION}" + asset_name="helm-${version}-${PREREQ_OS}-${PREREQ_ARCH}.tar.gz" + url="${PREREQ_HELM_URL:-https://get.helm.sh/${asset_name}}" + expected="${PREREQ_HELM_SHA256:-$(prereq_checksum_for HELM)}" + archive="true" + ;; + jq) + version="${PREREQ_JQ_VERSION}" + asset_os="${PREREQ_OS}" + [[ "${asset_os}" == "darwin" ]] && asset_os="macos" + asset_name="jq-${asset_os}-${PREREQ_ARCH}" + url="${PREREQ_JQ_URL:-https://github.com/jqlang/jq/releases/download/jq-${version}/${asset_name}}" + expected="${PREREQ_JQ_SHA256:-$(prereq_checksum_for JQ)}" + ;; + yq) + version="${PREREQ_YQ_VERSION}" + asset_name="yq_${PREREQ_OS}_${PREREQ_ARCH}" + url="${PREREQ_YQ_URL:-${YQ_DOWNLOAD_URL:-https://github.com/mikefarah/yq/releases/download/${version}/${asset_name}}}" + expected="${PREREQ_YQ_SHA256:-${YQ_DOWNLOAD_SHA256:-$(prereq_checksum_for YQ)}}" + ;; + *) prereq_fail "no verified binary installer for ${tool}" "${PREREQ_RC_UNSUPPORTED}"; return $? ;; + esac + + [[ -n "${expected}" ]] || { + prereq_fail "no SHA-256 is recorded for ${tool} on ${PREREQ_OS}/${PREREQ_ARCH}" "${PREREQ_RC_CHECKSUM}" + return $? + } + + old_umask="$(umask)" + umask 077 + tmp_dir="$(mktemp -d "${TMPDIR:-/tmp}/splunk-prereq-${tool}.XXXXXX")" || { + umask "${old_umask}" + prereq_fail "could not create a temporary directory for ${tool}" "${PREREQ_RC_DOWNLOAD}" + return $? + } + umask "${old_umask}" + payload="${tmp_dir}/${asset_name}" + if ! prereq_download "${url}" "${payload}"; then + rm -rf -- "${tmp_dir}" + prereq_fail "download failed for ${tool} ${version}" "${PREREQ_RC_DOWNLOAD}" + return $? + fi + + if ! prereq_verify_checksum "${payload}" "${expected}"; then + rm -rf -- "${tmp_dir}" + prereq_fail "SHA-256 verification failed for ${tool} ${version}" "${PREREQ_RC_CHECKSUM}" + return $? + fi + + if [[ "${archive}" == "true" ]]; then + if ! tar -xzf "${payload}" -C "${tmp_dir}"; then + rm -rf -- "${tmp_dir}" + prereq_fail "could not extract ${asset_name}" "${PREREQ_RC_PACKAGE}" + return $? + fi + source_binary="${tmp_dir}/${PREREQ_OS}-${PREREQ_ARCH}/helm" + else + source_binary="${payload}" + fi + + prereq_install_atomic "${source_binary}" "${tool}" || install_rc=$? + if (( install_rc != 0 )); then + rm -rf -- "${tmp_dir}" + return "${install_rc}" + fi + rm -rf -- "${tmp_dir}" + + install_dir="${PREREQ_INSTALL_DIR:-${XDG_BIN_HOME:-${HOME}/.local/bin}}" + if ! prereq_verify_installed_binary "${tool}" "${install_dir}/${tool}"; then + rm -f -- "${install_dir}/${tool}" + prereq_fail "${tool} was installed but failed its version check" "${PREREQ_RC_VERSION}" + return $? + fi + prereq_log "installed ${tool} ${version} in ${install_dir}" +} + +prereq_install_profile() { + local profile="${1:-cluster}" interaction="${2:-interactive}" tool rc=0 + local -a native_tools=() binary_tools=() + PREREQ_NONINTERACTIVE="false" + [[ "${interaction}" == "noninteractive" ]] && PREREQ_NONINTERACTIVE="true" + + prereq_collect_missing_tools "${profile}" || return $? + (( ${#PREREQ_MISSING_TOOLS[@]} > 0 )) || return 0 + prereq_detect_platform || return $? + PREREQ_LOCAL_TOOLS_TO_ACTIVATE=() + + for tool in "${PREREQ_MISSING_TOOLS[@]}"; do + case "${tool}" in + kubectl|helm|jq|yq) binary_tools+=("${tool}") ;; + *) native_tools+=("${tool}") ;; + esac + done + + if (( ${#native_tools[@]} > 0 )); then + prereq_log "installing native packages for: ${native_tools[*]}" + prereq_install_native_packages "${native_tools[@]}" || return $? + fi + + # Direct binary downloads require curl. Recheck it after the native package + # phase so a missing downloader is bootstrapped before any URL is accessed. + if (( ${#binary_tools[@]} > 0 )) && ! prereq_command_exists curl; then + prereq_fail "curl is still unavailable after native package installation" "${PREREQ_RC_PACKAGE}" + return $? + fi + + for tool in "${binary_tools[@]}"; do + rc=0 + prereq_owned_tool_is_compatible "${tool}" || rc=$? + if (( rc == 0 )); then + prereq_log "reusing verified ${tool} from ${PREREQ_INSTALL_DIR:-${XDG_BIN_HOME:-${HOME}/.local/bin}}" + elif (( rc == 1 )); then + prereq_install_verified_binary "${tool}" || return $? + else + return "${rc}" + fi + PREREQ_LOCAL_TOOLS_TO_ACTIVATE+=("${tool}") + done + + # Check mode is PATH-inert. Install mode exposes only the exact verified + # binaries selected above, never unrelated or stale files in ~/.local/bin. + if (( ${#PREREQ_LOCAL_TOOLS_TO_ACTIVATE[@]} > 0 )); then + prereq_activate_local_tools "${PREREQ_LOCAL_TOOLS_TO_ACTIVATE[@]}" || return $? + fi + + rc=0 + prereq_check_profile "${profile}" || rc=$? + if (( rc != 0 )); then + prereq_fail "prerequisite installation finished but tools are still missing: ${PREREQ_MISSING_TOOLS[*]}" "${PREREQ_RC_VERSION}" + return $? + fi +} + +prereq_ensure_profile() { + local profile="${1:-cluster}" interaction="${2:-interactive}" rc=0 + prereq_check_profile "${profile}" || rc=$? + (( rc == 0 )) && return 0 + (( rc == PREREQ_RC_MISSING )) || return "${rc}" + + prereq_log "attempting to install missing ${profile} prerequisites" + prereq_install_profile "${profile}" "${interaction}" || return $? + + rc=0 + prereq_check_profile "${profile}" || rc=$? + (( rc == 0 )) || { + prereq_fail "${profile} prerequisite recheck failed: ${PREREQ_MISSING_TOOLS[*]}" "${PREREQ_RC_VERSION}" + return $? + } +} diff --git a/tools/cluster_setup/prerequisites.lock b/tools/cluster_setup/prerequisites.lock new file mode 100644 index 00000000..cf1e2fd5 --- /dev/null +++ b/tools/cluster_setup/prerequisites.lock @@ -0,0 +1,30 @@ +# Installer prerequisite versions and SHA-256 digests. +# +# This is deliberately shell syntax: the prerequisite bootstrap cannot depend +# on yq or jq in order to read its own version metadata. Keep this file limited +# to simple variable assignments; installer_prereqs.sh validates it before use. + +PREREQ_KUBECTL_PINNED_VERSION="v1.36.1" +PREREQ_HELM_VERSION="v3.21.0" +PREREQ_JQ_VERSION="1.8.2" +PREREQ_YQ_VERSION="v4.53.3" + +PREREQ_KUBECTL_LINUX_AMD64_SHA256="629d3f410e09bf49b64ae7079f7f0bda1191efed311f7d37fdbab0ad5b0ec2b7" +PREREQ_KUBECTL_LINUX_ARM64_SHA256="59f7ee8e477fae658447607dc3c8790ac17a1b016c01c622c12070e969e2d4e7" +PREREQ_KUBECTL_DARWIN_AMD64_SHA256="b4973e90ebb00537d735b63d6f8293c1959156e6ff435f6a43c08aeaa1a2e7d7" +PREREQ_KUBECTL_DARWIN_ARM64_SHA256="9092778abaef3079449da4cd70ded0e4be112480c93efcdeace3155968d1d133" + +PREREQ_HELM_LINUX_AMD64_SHA256="0093eb572e3d2380f094df162ddb525e219249de88957afe24cfbb19632acd36" +PREREQ_HELM_LINUX_ARM64_SHA256="8de5a0c9a47431e59fd560e91e0779c8cf9316c383da7efb84128a4c339ecb2d" +PREREQ_HELM_DARWIN_AMD64_SHA256="8bc0c1f85f8738cc3cda4a2cc73047145bcdcb1f4d9cdcc29073037bfb22fa2e" +PREREQ_HELM_DARWIN_ARM64_SHA256="68bfbdc022c543a2a022597b20298216877e98abe6e4a345d3ecf114d79cae5f" + +PREREQ_JQ_LINUX_AMD64_SHA256="b1c22172dd303f3be49e935aa56aa48a8b7a46e0bc838b4997d3bb451495870f" +PREREQ_JQ_LINUX_ARM64_SHA256="8b85c817833814ddca00a144c33705546355afccf0cf39b188f3cdb48b852309" +PREREQ_JQ_DARWIN_AMD64_SHA256="e94b266e3c26690550006abe63152b782280f4e14374accdf04cbde844f00bc0" +PREREQ_JQ_DARWIN_ARM64_SHA256="2d75340ba57a4b4b4c8708a21c2dc8e958a48aaa8bba13b27f77f6e4c0eca07e" + +PREREQ_YQ_LINUX_AMD64_SHA256="fa52a4e758c63d38299163fbdd1edfb4c4963247918bf9c1c5d31d84789eded4" +PREREQ_YQ_LINUX_ARM64_SHA256="578648e463a11c1b6db6010cbf41eafed6bee79466fcffa1bb446672cf7945ea" +PREREQ_YQ_DARWIN_AMD64_SHA256="b4ba1ecce3c47f00803f4f964de38394326c7a32eb6540616e04fb2935a0f08d" +PREREQ_YQ_DARWIN_ARM64_SHA256="877de31753a4dd2401aa048937aa9a7fc4d5f6ce858cf31508c5802954297213" diff --git a/tools/cluster_setup/test_installer_prerequisites.sh b/tools/cluster_setup/test_installer_prerequisites.sh new file mode 100755 index 00000000..0eab004e --- /dev/null +++ b/tools/cluster_setup/test_installer_prerequisites.sh @@ -0,0 +1,702 @@ +#!/usr/bin/env bash +# Focused unit tests for lib/installer_prereqs.sh. +# +# Every test runs in a subshell and replaces the library's external-operation +# wrappers. The suite must never access the network, a package manager, sudo, +# or the host's real toolchain state. +# shellcheck disable=SC1090,SC1091,SC2016,SC2030,SC2031,SC2034,SC2329 +# Dynamic sourcing, injected globals, and indirectly called mock functions are +# intentional parts of this harness. + +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +LIB="${SCRIPT_DIR}/lib/installer_prereqs.sh" +MAIN_SCRIPT="${SCRIPT_DIR}/k0s_cluster_with_stack.sh" +AWS_PROVISIONER="${SCRIPT_DIR}/k0s_aws_provision.sh" + +PASS=0 +FAIL=0 +VERBOSE=0 + +if [[ "${1:-}" == "-v" ]]; then + VERBOSE=1 +fi + +fail() { + echo " $*" >&2 + return 1 +} + +assert_eq() { + local expected="$1" actual="$2" + [[ "${expected}" == "${actual}" ]] || + fail "expected $(printf '%q' "${expected}"), got $(printf '%q' "${actual}")" +} + +assert_empty_file() { + local path="$1" + [[ ! -s "${path}" ]] || fail "expected no external calls, got: $(tr '\n' ' ' < "${path}")" +} + +assert_contains() { + local haystack="$1" needle="$2" + [[ "${haystack}" == *"${needle}"* ]] || + fail "expected $(printf '%q' "${haystack}") to contain $(printf '%q' "${needle}")" +} + +assert_function() { + declare -F "$1" >/dev/null || fail "required function '$1' is not defined" +} + +source_library() { + [[ -r "${LIB}" ]] || fail "library not found: ${LIB}" + # shellcheck source=lib/installer_prereqs.sh + source "${LIB}" +} + +run_test() { + local name="$1" + shift + if "$@"; then + PASS=$((PASS + 1)) + [[ "${VERBOSE}" == "1" ]] && echo " PASS ${name}" + else + FAIL=$((FAIL + 1)) + echo " FAIL ${name}" + fi +} + +test_public_api() ( + source_library || return + + local fn + for fn in \ + prereq_detect_platform \ + prereq_collect_missing_tools \ + prereq_tool_is_compatible \ + prereq_check_profile \ + prereq_install_profile \ + prereq_ensure_profile \ + prereq_install_native_packages \ + prereq_install_debian_packages \ + prereq_install_rhel_packages \ + prereq_install_macos_packages \ + prereq_install_verified_binary \ + prereq_command_exists \ + prereq_command_path \ + prereq_exec \ + prereq_run_as_root \ + prereq_download \ + prereq_verify_checksum \ + prereq_effective_uid \ + prereq_uname_os \ + prereq_uname_arch; do + assert_function "${fn}" || return + done +) + +test_source_is_inert() ( + local test_dir calls + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + calls="${test_dir}/calls" + : > "${calls}" + + curl() { echo curl >> "${calls}"; } + sudo() { echo sudo >> "${calls}"; } + apt-get() { echo apt-get >> "${calls}"; } + dnf() { echo dnf >> "${calls}"; } + yum() { echo yum >> "${calls}"; } + brew() { echo brew >> "${calls}"; } + + source_library || return + assert_empty_file "${calls}" +) + +test_platform_detection() ( + local raw_os="$1" distro="$2" available_command="$3" + local expected_os="$4" expected_family="$5" expected_manager="$6" + local raw_arch="${7:-x86_64}" expected_arch="${8:-amd64}" + local test_dir release_file + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + release_file="${test_dir}/os-release" + printf 'ID=%s\n' "${distro}" > "${release_file}" + + source_library || return + PREREQ_OS_OVERRIDE="${raw_os}" + PREREQ_ARCH_OVERRIDE="${raw_arch}" + PREREQ_OS_RELEASE_FILE="${release_file}" + prereq_command_exists() { [[ "$1" == "${available_command}" ]]; } + + prereq_detect_platform || return + assert_eq \ + "${expected_os}|${expected_family}|${expected_manager}|${expected_arch}" \ + "${PREREQ_OS}|${PREREQ_OS_FAMILY}|${PREREQ_PACKAGE_MANAGER}|${PREREQ_ARCH}" +) + +test_os_dispatch() ( + local family="$1" expected="$2" test_dir calls + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + calls="${test_dir}/calls" + : > "${calls}" + + source_library || return + PREREQ_OS_FAMILY="${family}" + PREREQ_ARCH="amd64" + prereq_detect_platform() { :; } + prereq_install_debian_packages() { printf 'debian:%s\n' "$*" >> "${calls}"; } + prereq_install_rhel_packages() { printf 'rhel:%s\n' "$*" >> "${calls}"; } + prereq_install_macos_packages() { printf 'macos:%s\n' "$*" >> "${calls}"; } + + prereq_install_native_packages ca-certificates curl || return + assert_eq "${expected}:ca-certificates curl" "$(< "${calls}")" +) + +test_debian_package_mapping() ( + local test_dir calls + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + calls="${test_dir}/calls" + : > "${calls}" + + source_library || return + PREREQ_APT_UPDATED="false" + prereq_run_as_root() { printf '%s\n' "$*" >> "${calls}"; } + + prereq_install_debian_packages curl ssh scp git tar timeout || return + assert_eq \ + $'env DEBIAN_FRONTEND=noninteractive apt-get update\nenv DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends ca-certificates curl openssh-client git tar coreutils' \ + "$(< "${calls}")" +) + +test_rhel_package_mapping() ( + local test_dir calls + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + calls="${test_dir}/calls" + : > "${calls}" + + source_library || return + PREREQ_PACKAGE_MANAGER="dnf" + prereq_run_as_root() { printf '%s\n' "$*" >> "${calls}"; } + + prereq_install_rhel_packages curl ssh scp git tar timeout || return + assert_eq \ + 'dnf install -y ca-certificates curl openssh-clients git tar coreutils' \ + "$(< "${calls}")" +) + +test_macos_package_mapping() ( + local test_dir calls + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + calls="${test_dir}/calls" + : > "${calls}" + + source_library || return + PREREQ_PACKAGE_MANAGER="brew" + prereq_exec() { printf '%s\n' "$*" >> "${calls}"; } + prereq_command_exists() { [[ "$1" == "brew" ]]; } + + prereq_install_macos_packages curl ssh git timeout || return + assert_eq 'brew install curl openssh git coreutils' "$(< "${calls}")" +) + +test_unsupported_os_dispatch() ( + local test_dir calls rc=0 + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + calls="${test_dir}/calls" + : > "${calls}" + + source_library || return + PREREQ_OS_FAMILY="solaris" + PREREQ_ARCH="amd64" + prereq_detect_platform() { :; } + prereq_install_debian_packages() { echo debian >> "${calls}"; } + prereq_install_rhel_packages() { echo rhel >> "${calls}"; } + prereq_install_macos_packages() { echo macos >> "${calls}"; } + + prereq_install_native_packages curl >/dev/null 2>&1 || rc=$? + assert_eq "${PREREQ_RC_UNSUPPORTED}" "${rc}" || return + assert_empty_file "${calls}" +) + +test_profile_check_is_non_mutating() ( + local test_dir mutations + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + mutations="${test_dir}/mutations" + : > "${mutations}" + + source_library || return + prereq_log() { :; } + prereq_tool_is_compatible() { return 0; } + prereq_exec() { echo "exec:$*" >> "${mutations}"; return 1; } + prereq_run_as_root() { echo "root:$*" >> "${mutations}"; return 1; } + prereq_download() { echo "download:$*" >> "${mutations}"; return 1; } + + prereq_check_profile cluster >/dev/null || return + assert_empty_file "${mutations}" +) + +test_profile_check_reports_missing_without_mutation() ( + local test_dir mutations rc=0 + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + mutations="${test_dir}/mutations" + : > "${mutations}" + + source_library || return + prereq_tool_is_compatible() { [[ "$1" != "yq" ]]; } + prereq_exec() { echo "exec:$*" >> "${mutations}"; return 1; } + prereq_run_as_root() { echo "root:$*" >> "${mutations}"; return 1; } + prereq_download() { echo "download:$*" >> "${mutations}"; return 1; } + + prereq_check_profile bootstrap >/dev/null 2>&1 || rc=$? + assert_eq "${PREREQ_RC_MISSING}" "${rc}" || return + assert_empty_file "${mutations}" +) + +test_install_then_recheck_is_idempotent() ( + local test_dir installs present=0 + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + installs="${test_dir}/installs" + : > "${installs}" + + source_library || return + prereq_command_exists() { + [[ "$1" != "yq" || "${present}" == "1" ]] + } + prereq_tool_is_compatible() { + [[ "$1" != "yq" || "${present}" == "1" ]] + } + prereq_install_verified_binary() { + [[ "$1" == "yq" ]] || return 1 + echo "$1" >> "${installs}" + present=1 + } + prereq_install_native_packages() { + fail "native package installation was not expected for: $*" + } + prereq_owned_tool_is_compatible() { return 1; } + prereq_activate_local_tools() { :; } + + prereq_ensure_profile bootstrap noninteractive >/dev/null 2>&1 || return + prereq_ensure_profile bootstrap noninteractive >/dev/null 2>&1 || return + assert_eq "yq" "$(< "${installs}")" +) + +test_noninteractive_root_uses_sudo_n() ( + local test_dir calls + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + calls="${test_dir}/calls" + : > "${calls}" + + source_library || return + prereq_command_exists() { [[ "$1" == "sudo" ]]; } + prereq_effective_uid() { echo 1000; } + prereq_exec() { + printf '<%s>' "$@" >> "${calls}" + } + + PREREQ_NONINTERACTIVE="true" + prereq_run_as_root apt-get install -y curl || return + assert_contains "$(< "${calls}")" '<-n>' +) + +test_noninteractive_root_without_sudo_fails() ( + local test_dir calls rc=0 + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + calls="${test_dir}/calls" + : > "${calls}" + + source_library || return + prereq_command_exists() { return 1; } + prereq_effective_uid() { echo 1000; } + prereq_exec() { echo "$*" >> "${calls}"; } + + PREREQ_NONINTERACTIVE="true" + prereq_run_as_root apt-get install -y curl >/dev/null 2>&1 || rc=$? + assert_eq "${PREREQ_RC_PRIVILEGE}" "${rc}" || return + assert_empty_file "${calls}" +) + +test_authorized_sudo_package_failure_is_not_privilege_error() ( + local rc=0 + source_library || return + PREREQ_APT_UPDATED="false" + prereq_effective_uid() { echo 1000; } + prereq_command_exists() { [[ "$1" == "sudo" ]]; } + prereq_exec() { + [[ "$1" == "sudo" && "$2" == "-n" && "$3" == "true" ]] && return 0 + return 42 + } + + PREREQ_NONINTERACTIVE="true" + prereq_install_debian_packages curl >/dev/null 2>&1 || rc=$? + assert_eq "${PREREQ_RC_PACKAGE}" "${rc}" || return + assert_eq "apt-get update failed" "${PREREQ_LAST_ERROR}" +) + +test_unsupported_arch_fails_before_mutation() ( + local test_dir mutations rc=0 + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + mutations="${test_dir}/mutations" + : > "${mutations}" + + source_library || return + prereq_uname_os() { echo Darwin; } + prereq_uname_arch() { echo ppc64; } + prereq_exec() { echo "exec:$*" >> "${mutations}"; } + prereq_run_as_root() { echo "root:$*" >> "${mutations}"; } + prereq_download() { echo "download:$*" >> "${mutations}"; } + + prereq_detect_platform >/dev/null 2>&1 || rc=$? + assert_eq "${PREREQ_RC_UNSUPPORTED}" "${rc}" || return + assert_empty_file "${mutations}" +) + +test_checksum_failure_leaves_no_binary() ( + local test_dir install_dir download_path_file rc=0 downloaded_path + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + install_dir="${test_dir}/bin" + download_path_file="${test_dir}/download-path" + mkdir -p "${install_dir}" + + source_library || return + PREREQ_OS_FAMILY="debian" + PREREQ_OS="linux" + PREREQ_ARCH="amd64" + PREREQ_INSTALL_DIR="${install_dir}" + prereq_download() { + local destination="${2}" + echo "${destination}" > "${download_path_file}" + printf 'tampered payload' > "${destination}" + } + prereq_verify_checksum() { return 1; } + prereq_run_as_root() { fail "checksum failure must occur before privileged installation"; } + + prereq_install_verified_binary yq >/dev/null 2>&1 || rc=$? + assert_eq "${PREREQ_RC_CHECKSUM}" "${rc}" || return + [[ -r "${download_path_file}" ]] || { + fail "download wrapper was not called" + return + } + downloaded_path="$(< "${download_path_file}")" + [[ ! -e "${downloaded_path}" ]] || { + fail "unverified temporary payload was not removed" + return + } + [[ ! -e "${install_dir}/yq" ]] || fail "unverified yq was installed" +) + +test_version_failure_removes_installed_binary() ( + local test_dir install_dir rc=0 + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + install_dir="${test_dir}/bin" + + source_library || return + PREREQ_OS="linux" + PREREQ_OS_FAMILY="debian" + PREREQ_ARCH="amd64" + PREREQ_INSTALL_DIR="${install_dir}" + prereq_download() { + printf '#!/usr/bin/env bash\nexit 1\n' > "$2" + } + prereq_verify_checksum() { return 0; } + + prereq_install_verified_binary yq >/dev/null 2>&1 || rc=$? + assert_eq "${PREREQ_RC_VERSION}" "${rc}" || return + [[ ! -e "${install_dir}/yq" ]] || fail "failed yq binary was left installed" +) + +test_darwin_jq_asset_mapping() ( + local test_dir install_dir url_file + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + install_dir="${test_dir}/bin" + url_file="${test_dir}/url" + + source_library || return + PREREQ_OS="darwin" + PREREQ_OS_FAMILY="macos" + PREREQ_ARCH="arm64" + PREREQ_INSTALL_DIR="${install_dir}" + prereq_log() { :; } + prereq_download() { + printf '%s' "$1" > "${url_file}" + printf '#!/bin/sh\nprintf "jq-1.8.2\\n"\n' > "$2" + } + prereq_verify_checksum() { return 0; } + + prereq_install_verified_binary jq || return + assert_contains "$(< "${url_file}")" "/jq-1.8.2/jq-macos-arm64" || return + [[ -x "${install_dir}/jq" ]] || fail "verified jq was not installed" +) + +test_incompatible_present_binary_is_replaced() ( + local replaced=0 + source_library || return + prereq_tool_is_compatible() { + [[ "$1" != "yq" || "${replaced}" == "1" ]] + } + prereq_detect_platform() { + PREREQ_OS="linux" + PREREQ_OS_FAMILY="debian" + PREREQ_ARCH="amd64" + PREREQ_PACKAGE_MANAGER="apt-get" + } + prereq_install_native_packages() { fail "no native package was expected"; } + prereq_install_verified_binary() { + [[ "$1" == "yq" ]] || return 1 + replaced=1 + } + prereq_owned_tool_is_compatible() { return 1; } + prereq_activate_local_tools() { :; } + + prereq_ensure_profile bootstrap noninteractive >/dev/null 2>&1 || return + assert_eq "1" "${replaced}" +) + +test_versioned_tool_compatibility() ( + local test_dir fake_bin + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + fake_bin="${test_dir}/bin" + mkdir -p "${fake_bin}" + PATH="${fake_bin}:${PATH}" + + source_library || return + printf '#!/bin/sh\nprintf "yq 3.4.1\\n"\n' > "${fake_bin}/yq" + chmod +x "${fake_bin}/yq" + ! prereq_tool_is_compatible yq || fail "yq v3 was accepted" || return + printf '#!/bin/sh\nprintf "yq (https://github.com/mikefarah/yq/) version v4.53.3\\n"\n' > "${fake_bin}/yq" + prereq_tool_is_compatible yq || fail "mikefarah yq v4 was rejected" || return + + printf '#!/bin/sh\nprintf "v4.0.0\\n"\n' > "${fake_bin}/helm" + chmod +x "${fake_bin}/helm" + ! prereq_tool_is_compatible helm || fail "Helm 4 was accepted" || return + printf '#!/bin/sh\nprintf "v3.21.0\\n"\n' > "${fake_bin}/helm" + prereq_tool_is_compatible helm || fail "Helm 3 was rejected" +) + +test_pinned_kubectl_compatibility_window() ( + local test_dir fake_bin + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + fake_bin="${test_dir}/bin" + mkdir -p "${fake_bin}" + PATH="${fake_bin}:${PATH}" + + source_library || return + printf '%s\n' \ + '#!/bin/sh' \ + 'printf '\''clientVersion:\n gitVersion: %s\n'\'' "${FAKE_KUBECTL_VERSION}"' \ + > "${fake_bin}/kubectl" + chmod +x "${fake_bin}/kubectl" + + FAKE_KUBECTL_VERSION="v1.20.0"; export FAKE_KUBECTL_VERSION + ! prereq_tool_is_compatible kubectl || fail "obsolete kubectl was accepted" || return + FAKE_KUBECTL_VERSION="v1.35.9"; export FAKE_KUBECTL_VERSION + prereq_tool_is_compatible kubectl || fail "kubectl within one minor of the pin was rejected" || return + assert_eq "v1.36.1" "${PREREQ_KUBECTL_PINNED_VERSION}" || return + PREREQ_OS="darwin"; PREREQ_ARCH="arm64" + assert_eq \ + "9092778abaef3079449da4cd70ded0e4be112480c93efcdeace3155968d1d133" \ + "$(prereq_checksum_for KUBECTL)" +) + +test_check_mode_preserves_path() ( + local before_path + source_library || return + before_path="${PATH}" + PREREQ_INSTALL_DIR="/tmp/should-not-be-prepended" + prereq_tool_is_compatible() { return 0; } + + prereq_check_profile cluster >/dev/null || return + assert_eq "${before_path}" "${PATH}" +) + +test_verified_local_binary_is_reused_across_process_path() ( + local test_dir install_dir runtime_dir calls original_path version_output + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + install_dir="${test_dir}/bin" + runtime_dir="${install_dir}/.splunk-ai-prereq-bin" + calls="${test_dir}/downloads" + mkdir -p "${install_dir}" + : > "${calls}" + printf '#!/bin/sh\nprintf "yq (https://github.com/mikefarah/yq/) version v4.53.3\\n"\n' > "${install_dir}/yq" + chmod +x "${install_dir}/yq" + + source_library || return + PREREQ_INSTALL_DIR="${install_dir}" + original_path="${PATH}" + PATH="/usr/bin:/bin" + prereq_command_exists() { + # GitHub runners may provide /usr/bin/yq. This case intentionally treats + # only the managed runtime link as present so activation is deterministic. + case "$1" in + curl) return 0 ;; + yq) + [[ "$(command -v yq 2>/dev/null || true)" == "${runtime_dir}/yq" ]] + ;; + *) command -v "$1" >/dev/null 2>&1 ;; + esac + } + prereq_download() { echo "$*" >> "${calls}"; return 1; } + + prereq_ensure_profile bootstrap noninteractive >/dev/null 2>&1 || return + assert_empty_file "${calls}" || return + [[ "${PATH%%:*}" == "${runtime_dir}" ]] \ + || fail "verified prerequisite runtime directory was not activated" || return + [[ -L "${runtime_dir}/yq" && "$(readlink "${runtime_dir}/yq")" == "${install_dir}/yq" ]] \ + || fail "verified off-PATH yq runtime link was not created" || return + version_output="$(yq --version)" || fail "activated yq could not be executed" || return + assert_contains "${version_output}" "version v4.53.3" || return + PATH="${original_path}" +) + +test_runtime_activation_preserves_bootstrap_tool() ( + local test_dir install_dir runtime_dir + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + install_dir="${test_dir}/bin" + runtime_dir="${install_dir}/.splunk-ai-prereq-bin" + mkdir -p "${install_dir}" + printf '#!/bin/sh\nprintf "yq (https://github.com/mikefarah/yq/) version v4.53.3\\n"\n' > "${install_dir}/yq" + printf '#!/bin/sh\nprintf "jq-1.8.2\\n"\n' > "${install_dir}/jq" + chmod +x "${install_dir}/yq" "${install_dir}/jq" + + source_library || return + PREREQ_INSTALL_DIR="${install_dir}" + prereq_activate_local_tools yq || return + prereq_activate_local_tools jq || return + + [[ -L "${runtime_dir}/yq" ]] || fail "cluster activation removed bootstrap yq" || return + [[ -L "${runtime_dir}/jq" ]] || fail "cluster jq was not activated" +) + +test_bootstrap_then_cluster_preserves_runtime_tools() ( + local test_dir install_dir runtime_dir + test_dir="$(mktemp -d)" || return + trap 'rm -rf "${test_dir}"' EXIT + install_dir="${test_dir}/bin" + runtime_dir="${install_dir}/.splunk-ai-prereq-bin" + + source_library || return + PREREQ_INSTALL_DIR="${install_dir}" + prereq_profile_tools() { + case "$1" in + bootstrap) PREREQ_PROFILE_TOOLS=(yq) ;; + cluster) PREREQ_PROFILE_TOOLS=(yq jq) ;; + *) return 20 ;; + esac + } + prereq_command_exists() { + local found="" + case "$1" in + yq|jq) + found="$(command -v "$1" 2>/dev/null || true)" + [[ "${found}" == "${runtime_dir}/$1" ]] + ;; + *) command -v "$1" >/dev/null 2>&1 ;; + esac + } + prereq_detect_platform() { + PREREQ_OS="linux" + PREREQ_OS_FAMILY="debian" + PREREQ_ARCH="amd64" + PREREQ_PACKAGE_MANAGER="apt-get" + } + prereq_install_verified_binary() { + mkdir -p "${install_dir}" + case "$1" in + yq) printf '#!/bin/sh\nprintf "yq (https://github.com/mikefarah/yq/) version v4.53.3\\n"\n' > "${install_dir}/yq" ;; + jq) printf '#!/bin/sh\nprintf "jq-1.8.2\\n"\n' > "${install_dir}/jq" ;; + *) return 1 ;; + esac + chmod +x "${install_dir}/$1" + } + + prereq_ensure_profile bootstrap noninteractive >/dev/null 2>&1 || return + prereq_ensure_profile cluster noninteractive >/dev/null 2>&1 || return + [[ -L "${runtime_dir}/yq" ]] || fail "cluster ensure removed bootstrap yq" || return + [[ -L "${runtime_dir}/jq" ]] || fail "cluster ensure did not activate jq" +) + +test_main_integration_contract() ( + local bootstrap_line delegation_line show_line cluster_line preflight_line + bootstrap_line="$(grep -n '^ ensure_installer_prerequisites bootstrap$' "${MAIN_SCRIPT}" | cut -d: -f1)" + delegation_line="$(grep -n '^# ====== AIR-GAP DELEGATION ======$' "${MAIN_SCRIPT}" | cut -d: -f1)" + show_line="$(grep -n '^ show_install_plan$' "${MAIN_SCRIPT}" | cut -d: -f1)" + cluster_line="$(grep -n '^ ensure_installer_prerequisites cluster$' "${MAIN_SCRIPT}" | cut -d: -f1)" + preflight_line="$(grep -n '^ phase_start "Preflight"$' "${MAIN_SCRIPT}" | cut -d: -f1)" + + [[ -n "${bootstrap_line}" && -n "${delegation_line}" && bootstrap_line -lt delegation_line ]] \ + || fail "bootstrap prerequisite ensure must precede air-gap delegation" || return + [[ -n "${show_line}" && -n "${cluster_line}" && -n "${preflight_line}" \ + && show_line -lt cluster_line && cluster_line -lt preflight_line ]] \ + || fail "cluster ensure must remain between install confirmation and preflight" || return + # Literal source-code contract; expansion here would defeat the assertion. + # shellcheck disable=SC2016 + grep -Fq 'INSTALL_PREREQS="${INSTALL_PREREQS:-true}"' "${MAIN_SCRIPT}" \ + || fail "automatic prerequisite installation is not the install default" || return + grep -q -- '--no-install-prereqs' "${MAIN_SCRIPT}" \ + || fail "install opt-out flag is not wired" || return + grep -q 'prerequisites.lock' "${AWS_PROVISIONER}" \ + || fail "AWS provisioner does not copy the prerequisite lock" || return + grep -q 'lib/installer_prereqs.sh' "${AWS_PROVISIONER}" \ + || fail "AWS provisioner does not copy the prerequisite module" || return + grep -q 'prereq_ensure_profile cluster noninteractive' "${AWS_PROVISIONER}" \ + || fail "AWS provisioner does not invoke the shared prerequisite module" || return + ! grep -q 'stable.txt\|get-helm-3' "${AWS_PROVISIONER}" \ + || fail "AWS provisioner still contains a floating prerequisite installer" +) + +echo "Prerequisite installer unit tests" +run_test "library exports its documented API and mock seams" test_public_api +run_test "sourcing the library has no external side effects" test_source_is_inert +run_test "Ubuntu detection selects apt" test_platform_detection Linux ubuntu apt-get linux debian apt-get +run_test "Rocky Linux detection selects dnf" test_platform_detection Linux rocky dnf linux rhel dnf +run_test "Amazon Linux ARM detection falls back to yum" test_platform_detection Linux amzn yum linux rhel yum aarch64 arm64 +run_test "macOS detection selects Homebrew" test_platform_detection Darwin ignored brew darwin macos brew +run_test "Debian package dispatch" test_os_dispatch debian debian +run_test "RHEL package dispatch" test_os_dispatch rhel rhel +run_test "macOS package dispatch" test_os_dispatch macos macos +run_test "Debian installer maps canonical tools to apt packages once" test_debian_package_mapping +run_test "RHEL installer maps canonical tools to dnf packages once" test_rhel_package_mapping +run_test "macOS installer maps canonical tools to Homebrew packages" test_macos_package_mapping +run_test "unsupported OS fails before package mutation" test_unsupported_os_dispatch +run_test "check mode is a no-op when every tool is present" test_profile_check_is_non_mutating +run_test "check mode reports a missing tool without mutation" test_profile_check_reports_missing_without_mutation +run_test "ensure performs check-install-recheck once and is then idempotent" test_install_then_recheck_is_idempotent +run_test "noninteractive privileged commands use sudo -n" test_noninteractive_root_uses_sudo_n +run_test "noninteractive privileged commands fail when sudo is unavailable" test_noninteractive_root_without_sudo_fails +run_test "authorized sudo package failures report package errors" test_authorized_sudo_package_failure_is_not_privilege_error +run_test "unsupported architecture fails before mutation" test_unsupported_arch_fails_before_mutation +run_test "checksum failure removes the temporary payload and installs nothing" test_checksum_failure_leaves_no_binary +run_test "post-install version failure removes the invalid binary" test_version_failure_removes_installed_binary +run_test "macOS jq uses the upstream macos asset name" test_darwin_jq_asset_mapping +run_test "an incompatible present binary is replaced" test_incompatible_present_binary_is_replaced +run_test "versioned tool compatibility rejects yq v3 and Helm 4" test_versioned_tool_compatibility +run_test "kubectl compatibility follows the pinned minor and locked checksum" test_pinned_kubectl_compatibility_window +run_test "check mode does not change PATH precedence" test_check_mode_preserves_path +run_test "verified user-local tools are reused when a new process PATH omits them" test_verified_local_binary_is_reused_across_process_path +run_test "cluster activation preserves the bootstrap yq link" test_runtime_activation_preserves_bootstrap_tool +run_test "bootstrap then cluster ensure preserves all verified runtime tools" test_bootstrap_then_cluster_preserves_runtime_tools +run_test "main install preserves prerequisite ordering and packaging" test_main_integration_contract + +echo +echo "Results: ${PASS} passed, ${FAIL} failed" +[[ "${FAIL}" -eq 0 ]]