AEP-87: Decoupled Storage Market — bidengine, storage operator, replication - #1
Open
chalabi2 wants to merge 40 commits into
Open
AEP-87: Decoupled Storage Market — bidengine, storage operator, replication#1chalabi2 wants to merge 40 commits into
chalabi2 wants to merge 40 commits into
Conversation
…eta4 Activate the AEP-87 staged chain-sdk versions across the provider, mirroring the node migration. All three are pure wire-compatible supersets (the added volume fields are nullable and default-empty), so this is an import/type migration with no behavior change. Identity packages (deployment/v1, market/v1) are untouched. - Composite client moves from node/client/v1beta3 to v1beta4, whose query clients now serve deployment/v1beta5 and market/v2beta1; broadcast options and client errors come from v1beta4 re-exports, dropping the last direct v1beta3 imports. - Manifest handshake (gateway REST, manifest service, watchdog, kube builder) moves to manifest/v2beta4, accepting what SDL v2.2 emits. The manifest wire is JSON and v2beta4 only adds the optional StorageParams.volume field, so v2beta3 payloads from older clients decode losslessly; nothing breaks for existing tenants. - Manifest CRD decision: pkg/apis akash.network/v2beta2 stores its own local ManifestStorageParams (name/mount/readOnly), not the SDK type, so the stored CRD schema is unchanged. Only the in- memory conversion target moves to v2beta4; the new volume field is not persisted to the CRD yet (that lands with the storage operator stage). - Mocks regenerated with mockery v3.5.4 (version swaps only). Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
A storage-only (volume) group must reserve exactly what the order declares. The chain rejects any volume bid whose offered storage quantity differs from the order (validateVolumeBid equality gate), and durable leased bytes are never thin-sold, so overcommit levels cannot apply. Worse, ComputeCommittedResources clamps a zero request to one under a >1 factor, which would corrupt the present-but-zero compute shape of a volume group and produce an offer the chain cannot accept. resourcesToCommit now passes volume groups through untouched; the present-but-zero shape already satisfies the reserve nil-unit checks and per-class Adjust accounting unchanged. Flagging the resulting reservation durable (commit level pinned to 1.0, exempt from unreserve-on-lease-close, rebuilt from Volume CRDs) is left as a TODO for the cluster stage, which introduces the durable reservation kind. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
Volume orders arrive as ordinary OrderOpen events, so discovery is untouched; newOrder forks on GroupSpec.Volume into a dedicated gate set. shouldBidVolume keeps the placement and audit gates verbatim and skips the compute gates, which cannot apply to the present-but-zero resource shape. On top it requires: - the storage class in the advertised on-chain capabilities (the storage leg of MatchResourcesRequirements), AND live per-class headroom from the retained cluster inventory snapshot. The bidengine service now feeds a StorageInventory from the retained inventory-status topic, closing the advertised-vs-live split for the storage path (the old provider_attributes TODO) - a volume provider must not over-bid capacity it cannot park durably. - the class configured for volume bidding (VolumeConfig.Classes), the operator-installed -retain StorageClass participation knob; an unconfigured provider never bids on volume orders. - provider caps: max size, max retention honored, replication support (max_replicas ceiling, replica_of requires support). - adoption orders: a cheap local pre-check that a Retained volume matching owner/vid exists. The VolumeLookup interface is defined now and nil-safe; the storage operator client implements it later and the chain adoption gate stays authoritative either way. Attach orders ride the ordinary compute path plus a per-ref pre-check against locally Parked volumes through the same nil-safe interface; attach references carry no storage quantity, so reservations are never double-counted. Win handoff: a volume lease has no manifest - the GroupSpec is the whole contract - so the win publishes event.VolumeLeaseWon for the volume provisioner instead of event.LeaseWon, keeping the manifest machinery out of the loop. checkForExistingBid restart recovery is unchanged and covered by tests for volume bids. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
scalePricing already handles a volume group: the zero compute terms contribute nothing and the per-class storage map prices the single persistent entry, with unknown class or zero scale remaining a hard no-bid. Tests pin that behavior down so the storage-only shape cannot regress into a zero-cost bid. shellScriptPricing gains the volume terms a bid must fold into its single per-block rate: storageElement carries additive volume, retention_hours, max_replicas and replica fields, set only for volume groups. Retention length and replication export duty are bid inputs priced by the provider (the chain validates the window floor and exact size, not the price composition), and a replica volume is priced by its own provider - per-replica billing rides existing machinery. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
Wire the AEP-87 bidengine surface into the daemon: - volumeProvisioner consumes event.VolumeLeaseWon. A volume lease has no manifest to wait for, so nothing else would ever pick the win up; the stub acknowledges it and marks the seam where the storage operator client creates the Volume CRD (operator stage). The reservation stays held on win, preserving the existing invariant. - provider config carries bidengine.VolumeConfig, set from new run flags: --volume-classes (participation knob; empty keeps volume bidding off so existing providers see zero behavior change), --volume-max-size, --volume-max-retention, --volume-max-replicas. Caps are flags rather than code because they are pricing-adjacent operator policy, like the reclamation window. - VolumeLookup is left nil until the storage operator client lands; the local CRD pre-checks are skipped and the chain gates remain authoritative, so bidding stays correct, just less filtered. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
Define the akash.network/v2beta2 Volume kind now so the cluster and operator stages compile against the same types. The CRD is the provider's restart-survival record and GC ledger: spec mirrors the on-chain volume group contract (owner, vid, groupId, class, size, reclaim, retention, leaseId), status carries the operator phases (Pending|Provisioned|Attached|Retained|Adopting|Exporting|Releasing), pvName, attachedLease and retainedUntil. The object name is deterministic - volume-<sha256(owner/vid)[:12]> - so the workload builder, the cluster client and the storage operator agree on holder/attach PVC names without coordination. NewVolume compiles a won volume lease's GroupSpec into the CRD; FromCRD reconstructs the lease and a storage-only GroupSpec, which is the rebuild source for durable reservations on daemon restart. Clientset/listers/informers/applyconfig regenerated with the repo k8s-codegen tooling (this also picks up previously ungenerated Write/Delete manifest-permission fields). Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
cluster.Client gains DeployVolume / AttachVolume / DetachVolume / TeardownVolume / VolumeStatus plus DeployedVolumes, the list-read the inventory service needs to rebuild durable reservations on restart. Every kube implementation delegates to the Volume CRD - the daemon records chain-derived state and the storage operator owns the PV choreography, so the daemon never manipulates PersistentVolumes: - DeployVolume compiles a won volume lease's GroupSpec into the CRD; re-deploys of the same owner/vid (adoption, migration re-lease) rewrite the group/lease bookkeeping and anything else is an identity mismatch error. - Attach/Detach guard the RWO single-attachment invariant on the cluster record; the chain attachment index stays authoritative. - TeardownVolume routes a closed volume lease to Retained with retainedUntil = closedAt + retention, or Releasing for reclaim delete. Destruction itself remains with the operator GC, driven by that positive deadline - never by lease-close inference. - Volume objects carry akash.network/component=volume plus lease labels; a test pins that TeardownLease stays unchanged (namespace deletion leaves Retain PVs and Volume CRDs behind). nullClient stubs return errNotImplemented, except DeployedVolumes which reports no volumes so startup rebuild keeps working. Tests use the simple fake tracker because the generated apply-configuration schema carries no type definitions in this repo. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
Deploy's persistent switch gains its third case: a service whose
storage params carry an AEP-87 volume ref renders as a k8s Deployment,
not a StatefulSet - VolumeClaimTemplates' positional per-replica PVCs
don't apply to a shared pre-bound claim - with replicas forced to 1
because v1 volumes are RWO.
The pod mounts the volume through
Volume{PersistentVolumeClaimVolumeSource{ClaimName}} where the claim
name is the deterministic volume object name derived from the ref's
owner/vid. The storage operator creates that PVC in the lease
namespace pre-bound to the volume's Retain PV, which resolves the
'DataSource: nil ... yet' seam in persistentVolumeClaims() by
reference instead of by data source.
The CRD manifest's storage params gain the volume field (omitempty,
additive) so the ref survives the manifest CRD roundtrip that feeds
the workload builder, and cleanupStaleResources' selector now excludes
akash.network/component=volume objects: volume attach objects are not
manifest services and must never be collected as stale.
Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
ctypes.Reservation gains Kind(); newReservation stamps ReservationKindDurable when the resource group carries a VolumePolicy. Durable reservations follow AEP-87 accounting: - counted by inventory.Adjust's per-class SubNLZ like any allocation, with the ordered quantity committed EXACTLY - commit levels skipped, so StorageCommitLevel is effectively pinned to 1.0 and durable leased bytes are never thin-sold; - exempt from unreserve-on-lease-close once allocated: the provider holds the bytes through retention/adoption, so the unreserve path declines with errReservationDurable and capacity is released only on volume GC (operator stage). A pending durable reservation (bid lost, never provisioned) unreserves normally; - rebuilt from Volume CRDs on restart: cluster.NewService feeds client.DeployedVolumes into the inventory service, which recreates them as already-allocated entries - their PVs are counted allocated by the inventory operator, so re-adjusting them against reported capacity would double-count. The volume provisioner now records wins through cluster.Client.DeployVolume and announces the deployment as deployed, which is what flips the reservation to allocated (matched by order and group name exactly like compute deployments). The null inventory client's nodes advertise the beta2 class capability so storage-only reservations are exercisable in tests. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
…olumes
New 'provider-services operator storage' subcommand - the CRD-reconciler
archetype (hostname/ip model), replicas:1, all state re-derivable from
Volume CRDs + PVs + chain queries.
The reconciler owns the PV choreography of IMPLEMENTATION.md 5.2, built
so the PV is never an unclaimed Available Retain PV the kube binder
could hand to an arbitrary pending PVC:
- provision: holder PVC in akash-volumes against the -retain class;
once bound the PV is labeled akash.network/volume-owner|vid|dseq and
the volume parks as Provisioned.
- attach: target PVC created in the compute lease namespace FIRST
(pre-bound via spec.volumeName), pv.spec.claimRef patched directly to
it, holder deleted last. Detach reverses the same steps.
- retain: the close stamps retainedUntil = closedAt + retention. The
chain does not expose the lease closed_at yet (known chain-sdk stub),
so event receipt time stands in, which only extends the window.
- adopt: the CRD spec moving to a new deployment while the PV label
still carries the dead dseq freezes the volume in Adopting; the
re-bind happens only after the adoption deployment's on-chain
VolumePolicy.adopt is verified against the identity recorded on the
PV. A Released PV is never re-bound for a different owner.
- GC: the single destruction path, driven by the positive retention
deadline - never by lease-close inference - and only out of
Retained/Releasing, so Exporting/Adopting freeze it. The PV reclaim
policy is flipped to Delete right before the claim is removed so the
provisioner destroys the backing image.
Chain watch: the operator subscribes to block headers over the --node
RPC and decodes EventLeaseClosed, EventVolumeAttached/Detached and
EventVolumeAdopted itself (chain-sdk util/events does not forward the
volume events yet - follow-up noted). On (re)start the provider's
active leases are re-listed in full with a local pagination cursor,
never the bid engine's persisted orders checkpoint, healing closes
missed while the operator was down. Chain access is optional: without
it provisioning/attach/detach/GC still run and adoptions stay frozen.
REST surface mirrors the ip operator: /health for the daemon waiter,
/state (includes retained-bytes, standing in for the inventory.v1
StorageInfo extension until chain-sdk grows it), and
/volume/{owner}/{vid} for the bid engine's local pre-checks.
The Volume CRD joins crd.yaml so cluster installs serve it.
Unit tests cover every phase transition, the claimRef choreography in
both directions, GC gating (deadline, frozen phases, reclaim delete),
verified/rejected/frozen adoption, chain-event application, restart
re-list healing, and operator registration.
Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
Daemon-side wiring for the storage operator:
- cluster/kube/operators/clients/storage: REST client discovered by the
standard service labels (operator-storage in akash-services). It
satisfies waiter.Waitable (/health) and answers the bid engine's
local Volume CRD pre-checks (/volume/{owner}/{vid}): RetainedVolume
for adoption bids, ParkedVolume for attach bids.
- clfromctx gains CtxKeyClientStorage + accessor, mirroring the other
operator clients.
- run.go creates the client and appends it to waitClients ONLY when the
provider participates in the volume market (capabilities/storage/
volumes advertised in attributes, or volume classes configured), so
non-participating providers gain no chart-before-daemon upgrade
ordering constraint. A new storage-operator service endpoint flag
allows the usual endpoint override.
- service.go resolves the client from the context and wires it as the
bidengine VolumeLookup, replacing the operator-stage TODO. Without it
the pre-checks are skipped and the chain gates stay authoritative.
Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
Volume PVs live on the -retain StorageClass twins, which are not in the sellable allowlist, so the ceph/rancher PV-watch accounting never attributes their bytes to the base class - a parked or retained volume would read as free headroom the moment its compute lease closed (the ARCHITECTURE-MAP section 7 double-count risk). New 'volumes' driver polls the Volume CRDs (absence tolerated - the same polling stance as rook CRD discovery, so non-participating clusters lose nothing) and publishes an allocated-only storageSignal per base class covering every non-releasing volume: parked, attached, retained, adopting and exporting bytes are all durable holds. The cluster-state merge treats that driver as an overlay: its entries are added INTO the matching capacity-driver class (duplicated before mutation so retained driver slices stay pristine) instead of appended as duplicate class rows. Classes with no capacity provider are skipped. Re-publishes replace the overlay rather than accumulate. inventory.v1 StorageInfo does not carry volumes_capable/retained_bytes yet; that proto lives in chain-sdk, so this ships the provider-side plumbing only (retained-bytes is also served by the storage operator's /state endpoint) and the additive proto fields are a noted chain-sdk follow-up. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
AEP-87 data plane foundation (IMPLEMENTATION.md 5.5, DESIGN.md 9.2). The ReplicationDriver interface separates the VolumeTransfer wire protocol from the storage backend: snapshot, list, export (full or diff), import, digest, prune. Export streams always read from a snapshot, never the live image, so a transfer is source-immutable by construction. The chunk framing is the integrity layer: exports ship in fixed-size chunks, each stamped with its absolute stream offset and sha256. The receiver verifies both per chunk and refuses gaps, so a broken stream resumes from the last verified offset and corrupt data is never applied. Two backends. The file-copy driver is the local-path fallback and the backend CI exercises: snapshots are whole-file copies and diffs degrade to full copies, which converges identically while proving the protocol. The rbd driver is the stream-export mainnet posture: rbd snap create / export / export-diff executed in the rook-ceph-tools pod through the existing RemotePodCommandExecutor, no Ceph peering, no WAN-exposed mons, no cephx credentials leaving the cluster. Pod exec buffers output, so rbd payloads transit base64-encoded and bounded in memory; the file backend carries CI while RBD at size is validated on a real Rook pair via the manual runbook. rbd-mirror stays a deliberate stub: the config seam exists, selecting it fails loudly with ErrNotImplemented, and the doc comment records why it is only sound for mutually trusting pairs with namespace-scoped peers. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
DESIGN.md 9.1: no trust in self-asserted identity anywhere. The exporter authorizes a pull iff the requester's cert-bound address matches the request and chain state records it as either the provider of the live lease on the volume's own group after re-order (migration) or the provider of an active lease on a volume whose replica_of names this volume (replication). The exporter never authorizes itself. The importer runs the mirror gate before applying a byte: the dialed source must be, per chain state, the provider of the volume's closed/reclaiming lease (migration) or of the replica_of target's active lease (replication, active only). ChainState is a single narrow read - all leases related to (owner, vid) with their volume policies - re-derived from the chain on every call. The query-client implementation joins the owner's leases with their groups' policies; the fakes in tests cover allow and deny paths for both directions. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
The server side of IMPLEMENTATION.md 2.7: Export streams the image at the latest held snapshot (full, or export-diff from the requester's base) in sha256-stamped chunks honoring resume_offset; Status reports snapshots held, whole-image digest, and sync lag. Peer identity comes from the mTLS client certificate the gateway TLS layer already validated against x/cert; every request then passes the chain-derived export gate before a volume is even resolved. Volumes resolve through the same records the storage operator owns: Volume CRD by owner/vid, backing PV for the backend locator - rbd pool/image from CSI volumeAttributes, host path for local-path. The auto driver dispatches per volume so mixed clusters need no config. The import side lands pulls in a spool file so resume is a byte offset, verifies the whole-image digest before a full image is applied and after a diff is applied, then hands the stream to the driver. Syncer wraps that into rounds: first call pulls full, later calls pull diffs while the source still holds the base, falling back to full when it does not. Dial is the destination's mirror of the gateway's TLS posture: client certs are this provider's x/cert identity, the server certificate is validated against chain state and pinned to the expected source address. The transfer test drives server, chunker, importer and both file drivers end to end over in-memory streams: full pull, in-sync no-op, diff after source writes, and a denied stranger. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
DESIGN.md 9.3/9.4 destination-side coordination, built on injected seams so the state machines are testable with fakes and wire up the same against the real Syncer, chain probes, CRD flips and bid closes. RunMigration drives full export, periodic diffs, and - once the source's lease has closed and the image is quiesced by cascade-detach - one final pull that doubles as the final export-diff, then parks the destination CRD. Consecutive sync failures beyond the bound, a chain probe failure, or cancellation all abandon by closing the destination's bid: the order re-lists and the source holds data through window plus retention. Nothing held is non-re-derivable - a restarted migration re-pulls full and converges. RunReplica is the steady-state replica_of loop: initial full then periodic diffs on the provider-configured interval, retrying on the next tick after failures, running until the replica lease tears the context down. Lag stays observable via Status, never chain-attested. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
On EventLeaseReclaimStarted with a volume reason (volume_evict or volume_migrate) the volume's CRD freezes into Exporting: GC is structurally excluded while the phase holds and the transfer server serves the destination's pulls. Compute reclaims never reach the operator - the event pump filters on reason. When the lease then closes (event, or the restart re-list healing a missed close), the retention clock is stamped without leaving the freeze: the source's duty is window plus retention, and the destination may still be pulling the final diff. The reconciler thaws Exporting to Retained only once the stamped deadline passes, after which the next pass takes the existing single destruction path. Before the deadline the phase holds and the PV stays untouched. The lease-close switch previously ignored Exporting entirely, which would have held reclaimed volumes forever; tests now pin the stamp on event and on resync, the thaw at deadline, the hold before it, and the Exporting freeze against GC. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
The gateway grpc server (the inventory/v1-patterned endpoint that already terminates mTLS with x/cert provider identities via NewServerTLSConfig) gains an optional service registration: WithVolumeTransfer plugs the AEP-87 data plane in without touching the existing ProviderRPC surface, and providers outside the volume market register nothing. run.go assembles the transfer server only when volumesEnabled: chain state authorization off the daemon's query client, volume resolution off the Volume CRDs and PVs, and the auto driver dispatching rbd (rook-ceph-tools exec) or plain file copy per volume backend. New knobs follow the existing volume flag block: volume-replication-driver (stream-export default; rbd-mirror is the opt-in stub that fails loudly until implemented), the rook-ceph-tools namespace, the file backend base dir, and volume-replica-sync-interval - the provider-config period for replica diff pulls. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
Add the AEP-87 cluster plumbing the provider e2e suite needs, keeping the kustomize install path and the in-process RunLocalOperator path in lockstep (divergence between the two is a known trap): - beta3 (Delete) and beta3-retain (Retain) rancher.io/local-path StorageClasses. The retain twin binds Immediate: parked holder PVCs never get a consuming pod, so WaitForFirstConsumer would never bind. - akash-operator-storage kustomize base + template (hostname/ip operator archetype): deployment, operator-storage discovery service, RBAC for the PV claimRef choreography, Volume CRDs, snapshots, and rook exec. KUSTOMIZE_INSTALLS gains it in the kube, ssh, and single runs; the configure step feeds provider-address and the chain node URI. - the akash-volumes namespace that parks holder PVCs. - inventory-operator RBAC for the Volume CRD scraper (volume allocations overlay the merged cluster state). - beta3 appended to the manual node label; e2e integration timeout grows to cover the new suites' reclamation and escrow-runway waits. - kind-only: the local-path provisioner is repointed at a host-mounted /tmp/akash-e2e-storage with an identical-path extraMount, so the in-process two-provider migration flow can read source PVs and land imports in destination PVs from the host. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
…flag Two gaps the storage-market e2e surfaced: The balance checker learns about leases from the deployment manager, but a volume lease has no manifest and no manager, so it never joined the withdrawal schedule. Withdrawal is the escrow-exhaustion detector - an unfunded volume was never detected, so the cascade to volume_unfunded and the retention clock never fired. The volume provisioner now publishes LeaseAddFundsMonitor alongside the deployed announcement. Known follow-up: volume leases found on provider restart still skip the monitor (compute leases re-register through manager rebuild); the next withdrawal after a restart is deferred until the lease is touched again. The storage operator read the chain node URI from viper without owning the flag, so it only resolved under the provider-services root command. Register --node on the operator subcommand (same default) so standalone runs and the e2e harness can point it at the chain. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
Register E2EStorageMarket and E2EStorageMarketMigration in TestIntegrationTestSuite, modeled on E2EPersistentStorageDefault, and dedupe the pre-existing double E2EPersistentStorageDefault registration. Core loop: volume create -> bid -> lease -> attach compute -> write UUID -> close the compute deployment -> the Retain PV re-parks while the volume lease stays active -> a second compute deployment on the same ref reads the UUID back. Money path: a tiny-deposit volume exhausts its escrow through the provider's scheduled withdrawal, cascade-detaches the compute lease, closes as volume_unfunded, retains the CRD, and an adoption deployment wins its gated bid, re-binds the same PV, and serves the UUID again. Migration: a full second provider (own keys, ports, gateway, hostname and storage operators, and CRD namespaces - two providers of one volume derive the same CRD name, so only namespaces keep them apart in a shared kind cluster) drives reclaim -> Exporting freeze -> auto-re-order -> lease at provider B -> full copy over the real VolumeTransfer gateways with mTLS x/cert identities and chain-derived authorization -> re-attach at B with the UUID intact on the migrated volume. Honest labeling per IMPLEMENTATION.md 8.3: local-path proves coordination and transfer plumbing; RBD is validated out-of-band. The storageMarket suite knob flips volume-enabled genesis (orders on, test-scale reclamation floors, tiny min deposits), starts the in-process storage operator, and runs the provider with volume classes, the storage-operator endpoint, a bid reclamation window, and the 1m withdrawal cadence. Run instructions live in the test file header; docker + kind were unavailable in this environment, so the suites are compile- and vet-gated here and expected to run under make kube-cluster-setup-e2e + make test-e2e-integration. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
…wlist The provider gateway gRPC otherwise binds its fixed 0.0.0.0:8444 default in every non-storage suite, which collides with anything already holding the port on the host (a running provider daemon, another harness). Every suite now takes a random port from the base allocation. The suite run instructions selected by suite-struct name, but testify names subtests after test methods — the filter matched nothing and every SetupSuite ran anyway. Select via -testify.m, which also skips SetupSuite for suites with no matching methods. The inventory operator's kustomize config never allowlisted beta3, so the operator neither reported beta3 capacity nor labeled nodes with it and volume bids were silently impossible in the e2e cluster. Found on the first real cluster run of the maiden voyage (VPS sandbox). Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
testutil.ApplyMockIO merges stderr into the buffer that ValidateTxSuccessful parses as the tx JSON response, so even stderr-routed CLI advisories corrupt it. Declare max-replicas: 1 in the suite templates so the no-redundancy advisory never fires. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
The suite templates declare max-replicas: 1 but neither in-process provider set a volume-max-replicas cap, so shouldBidVolume correctly declined every volume order (max_replicas 1 > provider cap 0) and no bid ever arrived. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
Node-constrained provisioners (rancher.io/local-path) cannot provision an Immediate-binding claim: no consuming pod ever schedules the parked holder PVC, so no node is ever selected and provisioning fails with 'configuration error, no node was specified'. The new --provision-node-hint flag makes the reconciler annotate the initial provisioning claim with volume.kubernetes.io/selected-node — the same mechanism the WaitForFirstConsumer scheduler uses — picking the first ready, schedulable node carrying the class capability label. Off by default: network-attached provisioners (Ceph RBD) neither need nor want the topology pin. The e2e harness enables it for both in-process operators (the kind cluster runs local-path). Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
The Go CRD types carry the AEP-87 volume reference but the manifest
CRD's OpenAPI schema was never taught the field, so the API server
pruned it on write ('unknown field ...params.storage[0].volume'). The
deploy path builds workloads from the CRD-persisted manifest: the
container volumeMount (from the surviving mount field) then referenced
a pod volume that volumeClaims() never emitted, and every attach
deployment failed with 'volumeMounts[0].name: Not found'.
Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
AttachVolume/DetachVolume existed on cluster.Client but nothing called them: attach depended entirely on the storage operator's chain-event watch, which is a healing backstop (and unavailable in chain-degraded mode), so the target PVC never appeared and attach pods sat Pending on 'persistentvolumeclaim not found' until the deployment monitor gave up. The deployment manager now marks the Volume CRD Attached before client.Deploy (the operator's watch then runs the claimRef choreography), and detaches as a fourth teardown activity so the operator re-parks the Retain PV when the compute lease closes. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
aclient.DiscoverQueryClient's queryClientInfo only accepts the akash-extended RPCClient and rejects a plain *http.HTTP with 'unsupported RPC client', which sent the operator into chain-degraded mode (no event stream, adoptions frozen) on any standard node RPC. With no client on the discovery context it version-discovers over its own JSON-RPC call; the comet client stays dedicated to the event stream. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
queryAppWithRetries reused one http.Request whose *bytes.Buffer body is drained by the first attempt; every retry after a non-200 sent 'ContentLength=N with Body length 0'. Body-carrying app writes only ever worked when the first attempt landed. Rebuild the request with a fresh reader per attempt. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
The previous fix removed the comet client from the discovery context,
which let version discovery pass but left the discovered query client
without an RPC connection ('no RPC client is defined in offline mode').
node/client.NewClient returns the akash-extended RPCClient — accepted
by discovery, carries the query connection, and embeds the websocket
comet client for the event stream, with its own managed lifecycle.
Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
…d lease An adoption moves Spec.GroupID to the adopting deployment (applyAdoption, then the reconciler's verified re-bind) before the daemon's DeployVolume re-records the lease. In that window the CRD's Spec.LeaseID still names the dead lease, so both the resync sweep and a late EventLeaseClosed saw an inactive lease on a Provisioned volume and restarted the retention clock - permanently, since maybeAdopt never re-fires once the PV label matches the spec and DeployVolume never touches status. The money-path suite failed exactly there: adopted at dseq 212, re-parked Retained seconds later by the 5s resync. Guard both retention decisions on the lease still belonging to the deployment the spec names; surviving the dead lease's closure is the whole point of the adoption. Also retry chain-event application on k8s update conflicts: handlers re-read state per attempt and are idempotent, and a dropped EventVolumeAdopted is not healed by the resync sweep (round 11 logged one such conflict-dropped event). Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
The cert CLI writes the PEM under the chain home (node0/simd) while the in-process daemons run with the CLI home (node0/simcli). The base suite copies provider A's and the tenant's PEMs across after generation; setupProviderB never did, so provider B's account querier failed startup on the missing PEM (could not open certificate PEM file) and the migration suite died in SetupSuite waiting for B's gateway - in every round that reached it. Mirror the copy, and validate B's cert publish tx like the base suite does so a silent publish failure surfaces at setup rather than as an mTLS dial error mid-test. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
The provider daemon subscribed as "provider-cli" and the storage operator as "operator-storage" - fixed strings. The two-provider migration harness runs both daemons in one process against the validator's local RPC client, where the comet event bus keys subscriptions by (subscriber, query): the instances contend for one subscription, and the local client's silent cancel/resubscribe race leaves the loser permanently deaf (its resubscribe loop backs off unbounded against ErrAlreadySubscribed, logged only to the node's nop logger). Round 14 showed exactly that: provider A never detected the migration volume order while provider B bid normally. Suffix both subscriber ids with the daemon's provider address. Remote websocket clients key server-side by connection, so production deployments are unaffected; any co-resident/embedded arrangement now isolates correctly. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
The _run kube .envrc pinned KUBECONFIG to ~/.kube/config, so driving a second kind cluster from the same checkout silently stole the default kubeconfig's current-context from whatever was using it. Honor an already-exported KUBECONFIG and keep the old default when unset. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
…atus ServiceStatus decided Deployment-vs-StatefulSet from 'any storage param has a mount', but AEP-87 volume-ref services mount an attached PVC and render as Deployments. The stale heuristic made the gateway look up a StatefulSet that does not exist, which broke lease-shell (its handler gates on ServiceStatus) for every attach deployment. Mirror Deploy's workload-type switch: a volume ref wins over the mount check. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
The pump's new diagnostics (subscription start, per-block debug, fetch and publish errors) were invisible without a logger; a provider with a dead event feed showed nothing. Route them through the daemon logger under cmp=chain-events. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
… suite The two-provider migration suite runs both provider daemons in one process against a single akash RPC client dialed by the base suite and shared via s.cctx. That client is CometBFT's *http.HTTP, whose WSEvents subscription map is keyed by query string ALONE - the subscriber id passed to Subscribe is ignored (rpc/client/http: `subscriptions map[string]chan // query -> chan`, `Subscribe(ctx, _, query, ...)`). Both daemons' chain-event pumps subscribe to the identical NewBlockHeader query, so the second subscribe overwrites the first daemon's channel and the first daemon goes permanently deaf: its bidengine never receives EventOrderCreated and never bids. TestVolumeMigration then times out at the very first waitForBid (provider A's bid on the initial volume order 300/1/1) - the "bid never arrived" failure seen in rounds 14-16. The earlier subscriber-id-uniqueness fix (0e545b6) cannot help here because WSEvents discards the subscriber id; only the query keys the map. Dial a dedicated akash client for provider B and route its daemon and both its operators through it, so provider B owns a separate WSEvents map. Provider A's client topology is left byte-for-byte identical to the single-provider suites (which pass), and production is unaffected: real providers are separate processes, each with its own connection. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
…ed app data With the event-subscription fix in place, TestVolumeMigration now runs the full cross-provider flow (re-order fires, provider B bids and wins, the image payload transfers over the real gateway, the compute re-attaches at B) and reached the final app check, which asserted GET /value == 200 at provider B. That key can never be present: the file driver's documented unit of transfer is the single "image" file (the RBD block-image analogue; local-path proves coordination and transfer plumbing, not the app's live KV store), so the compute app's pre-migration "value" is not part of the migration payload. The assertion was inconsistent with the transfer scope the same test documents. Prove what the migration actually delivers instead: (1) the compute re-attaches and runs on the migrated volume at B - a fresh write/read round-trips through the app on B's lease; (2) the staged UUID (the driver's transferred payload) is byte-intact on B's PV, which the following image check already asserts. Data survival across the migration is proven by the image bytes arriving intact over the real VolumeTransfer gateway. Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
…global viper The hostname and storage operators read their k8s-manifest-ns (and the storage operator its volumes-namespace) from the process-global viper singleton. In the two-provider migration integration suite both providers' operators run co-resident in one test process and share that single viper: whichever cobra command binds the flag last owns the global key, so provider B's hostname operator watched provider A's manifest namespace instead of its own. It therefore never observed the migrated deployment's ProviderHost, never created the nginx ingress, and the re-attached app on the migrated volume returned HTTP 404 through the shared kind ingress (the two operators also collided creating the same-namespace ingress, surfacing as "ingresses already exists" / observation restarts). Reading each namespace from the command's own parsed flagset is race-free across co-resident operators and reads the identical value in a normal single-process deployment (no env binding is configured for these flags). Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
…alue" key
The e2e-test app (github.com/akash-network/e2e-test) serves exactly three
literal routes: /kill, /SET/value and /GET/value. It has no /SET/{key}
parameter - the only storable key is "value", persisted to a single
testfile. The migration suite's post-migration liveness check wrote and read
key "migrated", so every request hit /SET/migrated, which the app 404s,
independent of storage or ingress. Every other appWrite/appRead in the suite
already uses "value"; align this one. The check still proves app liveness on
the migrated volume by overwriting with a fresh UUID and reading it back.
Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Provider half of AEP-87.
VolumeTransferserver (mTLS, chain-derived authz)40 commits. Local-only fork PR. Not for upstream.