diff --git a/SUMMARY.md b/SUMMARY.md index ad06631..871d6ee 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -16,6 +16,8 @@ * [Using Sia Storage](renting/using-sia-storage.md) * [Setting up indexd](renting/setting-up-indexd/README.md) * [Docker](renting/setting-up-indexd/docker.md) + * [Backups and recovery](renting/setting-up-indexd/operations.md) + * [Connect an application](renting/setting-up-indexd/connect-application.md) * [Transferring Siacoins](renting/transferring-siacoins.md) * [Setting up renterd](renting/setting-up-renterd/README.md) * [macOS](renting/setting-up-renterd/macos.md) diff --git a/renting/about-renting.md b/renting/about-renting.md index 274b17f..8875677 100644 --- a/renting/about-renting.md +++ b/renting/about-renting.md @@ -26,7 +26,7 @@ Modern Sia storage is built from a few distinct pieces, each with a single job. * **Indexers (`indexd`) — contract management.** An indexer sits between applications and storage providers. It forms and maintains the **storage contracts** with providers, tracks where each object's shards live, monitors their health, and repairs data in the background when redundancy drops. It also enforces a simple access model so multiple apps can share one indexer safely. An indexer only ever sees encrypted metadata and the storage layout — never your plaintext data. This is the "renter" role in the new architecture. -* **SDKs — data storage and retrieval.** The [Sia Storage SDKs](https://devs.sia.storage) (available for Rust, Go, Python, and JavaScript) run inside your application. They encrypt your data and metadata, [erasure-code](https://devs.sia.storage/docs/core-concepts/erasure-coding) it into redundant shards, and upload or download those shards **directly** to and from storage providers, using the layout and access information supplied by the indexer. Encryption keys stay with you; neither the indexer nor the providers can read your data. +* **SDKs — data storage and retrieval.** The [Sia Storage SDKs](https://devs.sia.storage) (available for Rust, Go, Python, and JavaScript) run inside your application. They encrypt your data and metadata, [erasure-code](https://devs.sia.storage/docs/core-concepts/erasure-coding) it into redundant shards, and upload or download those shards **directly** to and from storage providers. Encryption keys stay with you; neither the indexer nor the providers can read your data. * **Apps — built on the SDKs.** Applications use an SDK to talk to an indexer and storage providers. The indexer deliberately stays narrow (objects, metadata, health, access), so anything higher-level — filenames, folders, search, sharing, version history — is built at the app layer on top of the SDK. @@ -68,8 +68,8 @@ npm install @siafoundation/sia-storage Everything that touches your data happens locally, inside the SDK: -* **On upload**, the SDK encrypts your file and its metadata, [erasure-codes](https://devs.sia.storage/docs/core-concepts/erasure-coding) the data into many redundant shards, and uploads those encrypted shards directly to storage providers using the contracts and layout the indexer provides. It then *pins* the resulting object to the indexer so the object becomes listable and can be repaired over time. -* **On download**, the SDK asks the indexer where an object's shards live, fetches enough of them from storage providers, verifies their integrity, and decrypts the data locally before streaming it back to your application. +* **On upload**, the SDK encrypts your file and its metadata, [erasure-codes](https://devs.sia.storage/docs/core-concepts/erasure-coding) the data into many redundant shards, and uploads those encrypted shards directly to storage providers using provider information already loaded into the SDK. The upload returns an object containing the keys and slab layout; the application then *pins* that object to the indexer so it becomes listable and can be repaired over time. +* **On download**, the application passes an object to the SDK. The SDK uses that object's encryption key and slab layout to fetch enough shards from storage providers, verify their integrity, and decrypt the data locally. If the object is stored locally and the SDK has provider connection information, the application can download it while `indexd` is offline. Data is addressed by a content-derived **object ID** rather than a file path, and objects are immutable — changing the data produces a new object. Object metadata is opaque, application-defined, and encrypted, so the indexer stores it but can't read it. Because every encryption key stays inside your app, neither the indexer nor the storage providers can ever see your data, your metadata, or even your filenames. @@ -82,12 +82,16 @@ In the Sia ecosystem, an **app** is the cryptographic identity of software actin An app's identity is built from two parts: * **App ID** — a 32-byte identifier you (the developer) choose once and ship with your software. It's the same across every install of your app, and because it's an input to key derivation, it must never change after release. -* **App Key** — a per-user signing key derived from the user's **recovery phrase** and your App ID. It's unique to each (user, app) pair, stored securely by your app, and used to sign and authorize every request to the indexer. The recovery phrase itself is never stored or transmitted by the app — only the derived key is kept. +* **App Key** — a per-user signing key derived from the user's **recovery phrase**, your App ID, and a secret issued by the indexer during approval. It's unique to each (user, app, indexer approval), stored securely by your app, and used to sign and authorize every request to the indexer. The recovery phrase itself is never stored or transmitted by the app — only the derived key is kept. -Because the App Key is derived from both the recovery phrase *and* the App ID, two apps from the same user produce different keys and **cannot see or modify each other's data**. This sandboxing is structural — enforced by cryptography, not by server-side rules. Before an app can act for a user it must be explicitly approved once, and approval can be revoked at any time, immediately cutting off that app's access. +Because the App Key is derived from the recovery phrase, App ID, and indexer-issued secret, two apps from the same user produce different keys and **cannot see or modify each other's data**. This sandboxing is structural — enforced by cryptography, not by server-side rules. Before an app can act for a user it must be explicitly approved once, and approval can be revoked at any time, immediately cutting off that app's access. Indexers deliberately stay narrow: they track objects, metadata, health, and access, and nothing more. The higher-level features your users expect — filenames, folders, search, sharing links, version history — are all built at the app layer on top of the SDK's simple, object-centric interface. +{% hint style="info" %} +**Design apps for portability and recovery.** Keep a current, usable cache of complete object records locally, and store the metadata needed to rebuild local app state on the indexer. After normal reauthorization, a surviving indexer plus the app's recovery phrase can rebuild the app; if the indexer is lost, a surviving local cache provides a last-resort way to seed a replacement without re-uploading object data. This fallback does not replace indexer backups, and losing both copies makes the object metadata unrecoverable. +{% endhint %} + {% hint style="info" %} #### Building on Sia @@ -100,11 +104,11 @@ Many services marketed as "decentralized" still place a company-operated gateway * **Your data never flows through a central server.** The SDK uploads and downloads encrypted shards **directly** to and from independent storage providers. The indexer coordinates *where* data lives and keeps it healthy, but it is never in the data path — it only ever handles encrypted metadata and the storage layout. There is no gateway that can see, meter, or gate your traffic. -* **You hold the keys, so you own the data.** Encryption and erasure coding happen on your own device before anything leaves it, and the keys are derived from your recovery phrase. Neither the indexer nor the storage providers can read your files, your metadata, or even your filenames. Even the apps you use are walled off from one another — each app's key is derived from both your recovery phrase and its own identity, so one app can never read another app's data, even though both act on your behalf. Ownership is cryptographic — not a permission a provider grants you and can revoke. +* **You hold the keys, so you own the data.** Encryption and erasure coding happen on your own device before anything leaves it. Neither the indexer nor the storage providers can read your files, your metadata, or even your filenames. Even the apps you use are walled off from one another — each app has a distinct key derived during its approval, so one app cannot read another app's data even though both act on your behalf. Ownership is cryptographic — not a permission a provider grants you and can revoke. * **No single operator can hold your data hostage.** Each object is erasure-coded into redundant shards spread across many unrelated storage providers worldwide, and only a subset is needed to reconstruct it. No one provider — and no outage, business failure, or demand against a single party — can read your data or take it offline. -* **You are never locked in.** The indexer is a swappable coordinator, not the owner of your data. You decide where your index lives — a hosted indexer like Sia Storage, or one you run yourself — and applications connect to whichever you choose. Because your data lives on contracts with storage providers and is recoverable with your recovery phrase, moving between indexers does not require downloading and reuploading your data or handing your data to a new gatekeeper. +* **You choose your indexer.** A well-designed app can move its cached metadata to another indexer while the encrypted sectors remain on their storage providers. Self-hosted operators should still maintain tested [`indexd` backups](setting-up-indexd/operations.md) for quick recovery of the indexer's operational state. The result is storage that is genuinely distributed, private, and yours: no proxy in the middle, no operator that can read your data, and no provider that owns your access to it. diff --git a/renting/setting-up-indexd/README.md b/renting/setting-up-indexd/README.md index dc033e1..f3e245e 100644 --- a/renting/setting-up-indexd/README.md +++ b/renting/setting-up-indexd/README.md @@ -41,3 +41,8 @@ The only reason to run your own `indexd` is to retain full control over the infr {% hint style="info" %} You can also run the `indexd` binary directly against your own PostgreSQL server. The binaries are available on the [official website](https://sia.tech/software/indexd), and configuration options are documented in the [indexd repository](https://github.com/SiaFoundation/indexd). {% endhint %} + +## Next steps + +* Configure [**backups and recovery**](operations.md) before storing production data. +* [**Connect an application**](connect-application.md) using a connect key. diff --git a/renting/setting-up-indexd/connect-application.md b/renting/setting-up-indexd/connect-application.md new file mode 100644 index 0000000..1224452 --- /dev/null +++ b/renting/setting-up-indexd/connect-application.md @@ -0,0 +1,29 @@ +--- +description: Connect an application to a self-hosted indexd +layout: + title: + visible: true + description: + visible: true + tableOfContents: + visible: true + outline: + visible: true + pagination: + visible: true +--- + +# Connect an application + +Applications connect to the **application API** using the advertise URL configured during setup. Before creating a connect key, confirm that this URL is reachable from the application and matches the public scheme, hostname, and port exactly. + +For access from another device, put the application API behind an HTTPS reverse proxy and use that public HTTPS URL. Do not use the admin UI URL or expose the admin API on port `9980`. See [Choose the application API advertise URL](docker.md#choose-the-application-api-advertise-url) for configuration examples, common mistakes, and a connectivity check. + +1. Sign in to the `indexd` admin UI and open **Connect keys**. +2. Select **Create key**, choose a quota, and create the key. +3. Copy the new key and keep it secret. +4. In the application, choose its custom or self-hosted indexer option and enter the advertise URL exactly as configured in `indexd`. +5. On the approval page opened by the application, paste the connect key into the **App password** field and select **Accept**. +6. Return to the application and enter the application's recovery phrase when prompted. + +The connect key, `indexd` admin password, PostgreSQL password, application recovery phrase, and `indexd` wallet recovery phrase are separate credentials. Initial registration consumes one use from the connect key's quota; later requests are signed with the application's derived App Key. diff --git a/renting/setting-up-indexd/docker.md b/renting/setting-up-indexd/docker.md index 8a35838..c1a4d74 100644 --- a/renting/setting-up-indexd/docker.md +++ b/renting/setting-up-indexd/docker.md @@ -1,5 +1,5 @@ --- -description: Run indexd and PostgreSQL together using Docker Compose +description: Run indexd, PostgreSQL, and Caddy using Docker Compose layout: title: visible: true @@ -15,7 +15,7 @@ layout: # Docker Compose -This guide will walk you through setting up `indexd` using Docker Compose. Because `indexd` requires a PostgreSQL database, this is the **recommended** way to self-host: Docker Compose runs `indexd` and PostgreSQL together, so you don't have to install or manage a database separately. +This guide will walk you through setting up `indexd` using Docker Compose. This is the **recommended** way to self-host: Compose runs `indexd`, PostgreSQL, and a Caddy HTTPS reverse proxy together. {% hint style="success" %} If you just want to store files, you don't need to run `indexd` at all — [Sia Storage](https://sia.storage) gives you 50 GB free with nothing to run. @@ -26,24 +26,39 @@ If you just want to store files, you don't need to run `indexd` at all — [Sia ## Pre-requisites * **Software Requirements:** Before installing `indexd`, you will need to install [Docker](https://www.docker.com/get-started/). - * **Hardware Requirements:** A stable setup that meets the following specifications is recommended. - A quad-core CPU - 8GB of RAM - 256 GB SSD for `indexd` and its PostgreSQL database - * **Network Access:** `indexd` interacts with the Sia network, so you need a stable internet connection and open network access to connect to the Sia blockchain. +* A domain to use for the public App API {% hint style="info" %} -PostgreSQL is provided automatically by the Compose file below, so you do **not** need to install it yourself for this method. +PostgreSQL and Caddy are provided by the Compose file below, so you do **not** need to install them separately. {% endhint %} ## Create the compose file -Create a new file named `docker-compose.yml`. You can use the following as a template. The `postgres` service stores the `indexd` index, and the `indexd-data` volume holds the consensus data and config file. +Create a new file named `docker-compose.yml`. You can use the following as a template. PostgreSQL stores the `indexd` index, and Caddy exposes the application API over HTTPS. ```yml services: + caddy: + depends_on: + - indexd + image: caddy:2 + restart: unless-stopped + environment: + INDEXD_DOMAIN: ${INDEXD_DOMAIN} + ports: + - 80:80/tcp + - 443:443/tcp + - 443:443/udp + volumes: + - ./caddy:/etc/caddy:ro + - caddy-data:/data + - caddy-config:/config + postgres: image: postgres:18 restart: unless-stopped @@ -70,32 +85,48 @@ services: ports: - 127.0.0.1:9980:9980/tcp # admin UI and API (kept local) - 9981:9981/tcp # public syncer - - 9982:9982/tcp # public application API + expose: + - 9982/tcp # application API (available only to Caddy) volumes: - indexd-data:/data volumes: + caddy-data: + caddy-config: indexd-data: postgres: ``` {% hint style="warning" %} -Be careful with port 9980 (the admin UI and API) as Docker will expose it publicly by default. It is recommended to bind it to `127.0.0.1` to prevent unauthorized access. Ports 9981 (syncer) and 9982 (application API) are meant to be reachable by the network and your applications. +Port `9980` is bound to `127.0.0.1` for the private admin UI and API. Port `9982` is not published on the host; only Caddy can reach it through the Compose network. Applications connect to Caddy on port `443`. Port `9981` should be publicly reachable by the Sia network. {% endhint %} -## Set the database password +### Create the Caddyfile + +Create a directory named `caddy` beside `docker-compose.yml`, then create `caddy/Caddyfile` with the following contents: + +```caddyfile +{$INDEXD_DOMAIN} { + reverse_proxy indexd:9982 +} +``` + +Caddy uses the domain from `.env`, obtains and renews its HTTPS certificate, and forwards application API requests to `indexd` over the private Compose network. Before starting the stack, point the domain's DNS records to this server and allow inbound TCP ports `80` and `443`. UDP port `443` enables HTTP/3. + +## Set the environment variables -Create a file named `.env` in the same directory as your `docker-compose.yml` and set a password for the PostgreSQL database: +Create a file named `.env` in the same directory as your `docker-compose.yml`. Set the PostgreSQL password and the public domain Caddy will use for the application API: ```sh POSTGRES_PASSWORD=your-secure-database-password +INDEXD_DOMAIN=indexd.example.com ``` -You will enter this same password during the `indexd` configuration step so the indexer can connect to the database. +You will enter the same database password during the `indexd` configuration step. For the application API advertise URL, enter the domain as a complete HTTPS URL, such as `https://indexd.example.com`. -## Getting the `indexd` image +## Get the container images -To get the latest `indexd` image run the following command: +Download the container images: ```console docker compose pull ``` @@ -127,7 +158,7 @@ The wizard will ask you for: * Your **wallet recovery phrase** (use the one you generated above). * An **admin password** used to unlock the `indexd` admin UI. * A **database password** — enter the same value you set in `.env`. -* An **application API advertise URL** — the public URL applications will use to reach this indexer. +* An **application API advertise URL** — the exact base URL applications will use to reach this indexer, described below. When asked whether to configure **advanced settings**, answer `yes` and set the database connection so `indexd` can reach the PostgreSQL container: @@ -140,6 +171,34 @@ When asked whether to configure **advanced settings**, answer `yes` and set the ![](../../.gitbook/assets/indexd-screenshots/install/docker/02-indexd-docker-config.png) +### Choose the application API advertise URL + +The advertise URL is not the address `indexd` listens on. It is the external base URL that an application uses to reach the application API. `indexd` puts this URL into the application-approval flow and uses its hostname when verifying signed requests, so an incorrect value can break application authentication even when `indexd` itself is running. + +For an indexer reached through an HTTPS reverse proxy, a typical configuration is: + +```yml +applicationAPI: + address: :9982 + advertiseURL: https://indexd.example.com +``` + +In this example, `indexd` listens on port `9982`, while applications connect to `https://indexd.example.com`. The reverse proxy terminates HTTPS and forwards requests to port `9982`. + +The advertise URL must: + +* Include the scheme, such as `https://`. +* Use the public hostname and port, if a non-standard port is required, that applications can actually reach. +* Use `https://` when a reverse proxy provides HTTPS, even if the proxy connects to `indexd` over HTTP. +* Be the same base URL entered in the application. +* Omit a trailing slash and API endpoint paths such as `/api` or `/auth/connect`. + +Do not use `0.0.0.0`, a Docker service name, or another internal address. Use `localhost` or `127.0.0.1` only when the application runs on the same machine as `indexd`; on a phone or another computer, `localhost` refers to that device instead of the indexer. + +{% hint style="warning" %} +The admin UI URL is not the advertise URL. Port `9980` serves the private admin API; applications connect to the application API on port `9982` or its HTTPS reverse-proxy URL. +{% endhint %} + ## Running `indexd` Now that you have `indexd` configured, start the services: @@ -158,6 +217,23 @@ Once `indexd` has started, you can access the admin UI by opening your browser a `indexd` is now set up. {% endhint %} +### Verify the advertise URL + +From the device or network where the application will run, test the advertise URL you configured: + +```console +curl -i https://indexd.example.com/auth/check +``` + +An unsigned request should reach `indexd` and return `401 Unauthorized` with a message about missing query parameters. A timeout, certificate error, or proxy error means the public URL is not ready. + +To correct the URL, run the configuration wizard again, choose to change the existing value, and restart `indexd`: + +```console +docker compose run --rm -it indexd config +docker compose restart indexd +``` + ## Fund your wallet `indexd` uses a built-in wallet to pay storage providers for the contracts it forms on your behalf, so it needs Siacoin (SC) before it can store any data. After signing in, the **Welcome to Sia** checklist in the admin UI walks you through the remaining setup, including funding your wallet. @@ -191,3 +267,7 @@ docker compose pull && docker compose up -d {% hint style="success" %} `indexd` is now updated to the latest version. {% endhint %} + +## Next steps + +Configure [backups and recovery](operations.md), then connect an [application](connect-application.md). diff --git a/renting/setting-up-indexd/operations.md b/renting/setting-up-indexd/operations.md new file mode 100644 index 0000000..04c6f7b --- /dev/null +++ b/renting/setting-up-indexd/operations.md @@ -0,0 +1,75 @@ +--- +description: Back up, move, and recover indexd +layout: + title: + visible: true + description: + visible: true + tableOfContents: + visible: true + outline: + visible: true + pagination: + visible: true +--- + +# Backing up and recovering indexd + +PostgreSQL contains the operational state of an `indexd` installation. A tested database backup is the primary way to recover or move the service without losing accounts, contracts, settings, and maintenance state. + +{% hint style="danger" %} +**The `indexd` recovery phrase is not a backup of the indexer.** It can recover the on-chain wallet balance after a blockchain rescan, but it cannot recreate the object index, contracts, or application accounts. +{% endhint %} + +## Back up PostgreSQL + +Use normal production PostgreSQL practices: + +* Run scheduled automated backups; use WAL archiving and point-in-time recovery for larger or critical deployments. +* Keep backups outside the `indexd` machine and storage volume. +* Test restores regularly and monitor backup failures, database health, disk space, and WAL retention. +* Use a dedicated database and role. Use verified TLS when connecting over an untrusted network. +* Back up `indexd.yml` and the data directory, and store the wallet recovery phrase separately. + +For Docker Compose, use PostgreSQL-aware tooling such as `pg_dump`, a managed-service snapshot, or a physical base backup. Copying a live `postgres` volume is not an application-consistent backup. + +## Do not share a database + +Only one full `indexd` daemon may use a PostgreSQL database at a time. Multiple daemons are not supported or safe: `indexd` has no database-wide leader election, and each instance runs its own consensus, wallet, contract, and maintenance services. + +Use active/passive failover and ensure the old process is stopped before starting its replacement. The `indexd remote` migration worker is different: it has no database connection and only performs work assigned by the primary node. + +## What can be recovered + +| Surviving state | What it can recover | +| --- | --- | +| PostgreSQL | Server-side state, including application accounts and connect keys, contracts, object and slab records, sector locations, settings, and wallet scan state. | +| `indexd` wallet recovery phrase | The wallet identity and on-chain balance after a rescan. It does not recover the index. | +| Application recovery phrase and a surviving indexer | After normal reauthorization, the app can rebuild its local state from the object metadata it stored on that indexer. | +| A complete application metadata cache | The app can pin its objects to a replacement indexer without moving the object payload. The cache must contain the object keys and complete slab layouts, not only object IDs. | + +If both the application cache and the indexer's PostgreSQL state are lost, neither recovery phrase can reconstruct the missing object metadata. + +## Move or restore indexd + +### Restore from a database backup + +1. Stop `indexd` before the final backup. +2. Back up PostgreSQL and copy the `indexd` data directory, including `indexd.yml` and `consensus.db`. +3. Restore both on the replacement server, configure the same wallet recovery phrase, and initially use the same `indexd` version. +4. Start exactly one `indexd` instance and wait for consensus and wallet synchronization. +5. Verify the wallet balance, contracts, application accounts, connect keys, and object health before switching applications or upgrading. + +If the public application API URL stays the same, existing applications can continue using their saved credentials after traffic moves to the replacement. + +### Recover from an application's local cache + +This is a last-resort fallback when PostgreSQL is permanently lost, not a substitute for database backups: + +1. Preserve the application's local metadata cache. Do not sign out, clear its storage, or reinstall it. +2. Set up, synchronize, and fund a replacement `indexd` so it can form contracts. +3. Create a connect key on the replacement and connect the application normally. The lost indexer is not required. +4. Have the application pin its cached object and slab metadata to the replacement. The replacement can pin the sectors already held by storage providers under its contracts; the object payload does not need to be uploaded again. +5. Wait for pinning and contract maintenance to finish, then verify the object list, health, and test downloads before deleting any recovery state. + +Moving even a large metadata catalog is far cheaper than downloading and re-uploading. Start promptly after a loss, because the old indexer is no longer renewing contracts or repairing redundancy. Only objects present in the local cache can be recovered this way.