Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/run-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,9 @@ jobs:
- name: dump environment logs
if: failure()
run: |
docker compose --env-file envs/latest ps --all || true
docker compose --env-file envs/latest logs --no-color --timestamps || true
docker compose --env-file envs/latest --profile rest-api ps --all || true
docker compose --env-file envs/latest --profile rest-api logs --no-color --timestamps || true
- name: stop environment
if: always()
run: yarn test:stop
run: yarn test:stop:clean
working-directory: tests
77 changes: 62 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,24 +10,44 @@ A Docker Compose file and auxiliary scripts for running a [Polymesh](https://pol
## Running

1. **Configuration**: Copy an environment file (e.g., `envs/8.0`) to `.env` in the project root: `cp envs/8.0 .env`. This file specifies the Docker images to use. Alternatively, provide the path directly: `docker compose --env-file=envs/8.0 up`.
2. **Start**: Run `docker compose up -d` to start the services in detached mode.
3. **Stop**: Run `docker compose down` to stop the services. `docker compose down --volumes` will also remove associated volumes (chain data, database data, vault data), this will reset the environment.
2. **Start**: Run `docker compose up -d` to start the services in detached mode. This brings up the core services only; see [Runtime Modes](#runtime-modes) for the optional `rest-api` and `evm` profiles.
3. **Stop**: Run `docker compose down` to stop the services, repeating any `--profile` flags you started with — without them the profiled containers (Blockscout, eth-rpc, Vault, the REST APIs) are left running. `docker compose down --volumes` will also remove associated volumes (chain data, database data, vault data), this will reset the environment.
4. **Restart**: To apply changes (e.g., updated `.env` file), run `docker compose down && docker compose up -d`.

The `scripts/start-env.sh` and `scripts/stop-env.sh` helpers wrap these commands and manage the profile flags for you.

Full variable reference is available in `envs/template` (also linked as `.env.example`).

This setup will launch the following services:
This setup will launch the following services by default:

- `polymesh-node`: A single Polymesh node running in development mode (`--dev`).
- `postgres`: PostgreSQL database, primarily for the Subquery indexer.
- `subquery-node`: Polymesh Subquery indexer service.
- `subquery-graphql`: GraphQL API server for querying indexed data.
- `environment-ready`: Indicates when the core services and initial setup scripts have completed successfully.

The REST API and Vault services are opt-in via the `rest-api` profile, since not
every workflow signs through them and they add five containers:

- `polymesh-rest-api-local-sm`: The Polymesh REST API service that uses local signers (Alice, Bob, Charlie, etc)
- `polymesh-rest-api-vault-sm`: The Polymesh REST API service with HashiCorp Vault as the signing manager.
- `polymesh-rest-api-vault-sm-init`: Creates test accounts and identities via the REST API.
- `vault`: HashiCorp Vault for key management.
- `vault-init`: Initializes and unseals Vault on first run, unseals on subsequent runs.
- `environment-ready`: Indicates when the core services and initial setup scripts have completed successfully.

```bash
./scripts/start-env.sh --env-file envs/8.0 --profile rest-api
# or, driving compose directly
POLYMESH_WAIT_FOR_REST_API=true docker compose --env-file=envs/8.0 --profile rest-api up -d
```

`POLYMESH_WAIT_FOR_REST_API` tells `environment-ready` to wait for the REST API
account setup rather than reporting ready as soon as the core services are
healthy. `start-env.sh` sets it to match the profiles it was given, so it is
only needed when invoking `docker compose` yourself.

The integration test suite needs the REST API and Vault, so `yarn test` and
`yarn test:start` enable the profile themselves.

## Checking Environment Status

Expand All @@ -37,7 +57,7 @@ The environment involves several services starting up and performing initial set
docker compose logs environment-ready
```

Wait for the message indicating completion:
With the `rest-api` profile enabled, wait for the message indicating completion:

```text
************************************************************************************
Expand All @@ -47,6 +67,15 @@ Wait for the message indicating completion:

If the environment was already initialized in a previous run, it will indicate readiness much faster.

Without that profile there is no account setup to wait for, so readiness is
reported as soon as the core services are healthy:

```text
************************************************************************************
*** Polymesh Environment Ready! (core services healthy) ****************************
************************************************************************************
```

## Vault VS Local Signing Manager

This environment supports usage of either the Local or Vault Signing Manager
Expand All @@ -58,22 +87,33 @@ To facilitate this choice, the environment runs two independent instances of the

## Runtime Modes

This repository now has two runtime options:
Optional services are grouped into profiles, so a run only pays for what it
uses:

| Profile | Adds | Containers |
| ---------- | -------------------------------------------------------------------------- | ---------- |
| _(none)_ | node, Postgres, Subquery indexer + GraphQL, readiness check | 5 |
| `rest-api` | both REST API instances, their init job, Vault, Vault init | +5 |
| `evm` | `polymesh-eth-rpc`, Blockscout backend/frontend, Blockscout Postgres/Redis | +5 |

1. Default mode (no EVM tooling)
2. EVM tooling mode (`--profile evm`), which starts `polymesh-eth-rpc`, Blockscout backend/frontend, and dedicated Blockscout Postgres/Redis.
Profiles combine: `--profile rest-api,evm` starts everything, which is what
earlier versions of this repository did on `--profile evm` alone. If you are
only exercising smart contracts, `--profile evm` on its own leaves out Vault and
the REST APIs entirely.

Polymesh v8 includes Revive-based smart contract support, so optional EVM tooling follows Polkadot smart-contracts guidance:

- [Get started with smart contracts](https://docs.polkadot.com/smart-contracts/get-started/)
- [JSON-RPC APIs for Ethereum developers](https://docs.polkadot.com/smart-contracts/for-eth-devs/json-rpc-apis/)

### Option 1: Default mode (without EVM tooling)
### Option 1: Core mode (node and indexer only)

```bash
./scripts/start-env.sh --env-file envs/8.0
```

Add `--profile rest-api` if you need the REST API or Vault.

`--env-file` is optional; if omitted, both `start-env.sh` and `stop-env.sh`
default to `envs/latest`. You can also override the default via the
`COMPOSE_ENV` environment variable.
Expand All @@ -95,16 +135,19 @@ file uses floating tags such as `latest`), add `--pull always`:
`build`); a bare `--pull` is shorthand for `always`. The same behaviour can be
set via the `COMPOSE_PULL_POLICY` environment variable.

By default `stop-env.sh` removes the named volumes (chain data, Vault keys,
Blockscout DB, ...) for a clean slate. To stop the containers but keep the data
for the next start, pass `--keep-volumes`:
`stop-env.sh` keeps the named volumes (chain data, Vault keys, Blockscout DB,
...) so the next start resumes from the existing state. For a clean slate, pass
`--volumes` (or `-v`):

```bash
./scripts/stop-env.sh --env-file envs/8.0 --keep-volumes
./scripts/stop-env.sh --env-file envs/8.0 --volumes
```

`stop-env.sh` always tears down the EVM tooling as well, so you do not need to
repeat `--profile evm` when stopping.
`--keep-volumes` is still accepted, as a no-op, so older invocations keep
working.

`stop-env.sh` always tears down the profiled services as well, so you do not
need to repeat `--profile evm` or `--profile rest-api` when stopping.

### Option 2: With EVM tooling

Expand Down Expand Up @@ -134,6 +177,7 @@ Explorer URLs:
Notes:

- EVM tooling remains opt-in and is not started by default.
- `--profile evm` no longer pulls in Vault and the REST APIs. Blockscout and `eth-rpc` talk to the node directly, so this is the leanest way to work on contracts. Add `--profile rest-api,evm` if you want both.
- `--allow-unprotected-txs` is enabled for local experimentation with legacy transaction flows.
- Blockscout uses dedicated Postgres and Redis services, separate from Subquery services.
- If you customize ports, keep frontend `NEXT_PUBLIC_API_PORT` aligned with `POLYMESH_BLOCKSCOUT_BACKEND_PORT`.
Expand Down Expand Up @@ -164,11 +208,14 @@ Notes:

- RPC/subscription coverage can vary by image tag. Keep `POLYMESH_ETH_RPC_IMAGE` configurable in env files and pin known-good tags.
- Some paritypr image tags can be architecture-specific. Override `POLYMESH_ETH_RPC_PLATFORM` if needed.
- `eth-rpc` surfaces some native Substrate extrinsics as pseudo-Ethereum transactions: SCALE-encoded `input`, a `modl*` pallet account as `to`, and zero gas. They are visible in Blockscout but cannot be traced — `debug_traceTransaction` returns `No Ethereum extrinsic found`. Blockscout's internal transaction fetcher is therefore disabled by default (`POLYMESH_BLOCKSCOUT_DISABLE_INTERNAL_TX_FETCHER`), since it would otherwise retry those blocks indefinitely and flood the logs.
- Receipts from `eth-rpc` report the Substrate extrinsic index as `transactionIndex`, so it can disagree with the position of the hash in the block's `transactions` array.

## Additional Notes

**Vault Automation:**

- Vault, the REST APIs and the account setup below only run with `--profile rest-api`. `scripts/get-vault-token.sh` and anything reading ports 3004/3005/8200 need that profile enabled.
- The `vault-init` service automatically initializes Vault (on the very first run with the profile) and unseals it every time the services start.
- It creates one unseal key and a root token, storing them in the `vault-root-token` named volume (accessible within the `vault-init` container at `/vault-token/` as `.unseal_key` and `.token`).
- It also automatically enables the `transit` secrets engine and creates ED25519 keys named `admin`, `signer1`, `signer2`, `signer3`, and `signer4`.
Expand Down
38 changes: 38 additions & 0 deletions compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,8 @@ services:
# https://github.com/PolymeshAssociation/polymesh-rest-api#signing-managers
polymesh-rest-api-vault-sm:
image: ${POLYMESH_REST_API_IMAGE}
profiles:
- rest-api
depends_on:
polymesh-node:
condition: service_healthy
Expand Down Expand Up @@ -169,6 +171,8 @@ services:

polymesh-rest-api-vault-sm-init:
image: alpine:3.19
profiles:
- rest-api
depends_on:
polymesh-rest-api-vault-sm:
condition: service_healthy
Expand All @@ -191,6 +195,8 @@ services:
# https://github.com/PolymeshAssociation/polymesh-rest-api#signing-managers
polymesh-rest-api-local-sm:
image: ${POLYMESH_REST_API_IMAGE}
profiles:
- rest-api
depends_on:
polymesh-node:
condition: service_healthy
Expand Down Expand Up @@ -323,6 +329,10 @@ services:
NETWORK: ${POLYMESH_BLOCKSCOUT_NETWORK:-Polymesh}
SUBNETWORK: ${POLYMESH_BLOCKSCOUT_SUBNETWORK:-Polymesh Dev}
INDEXER_DISABLE_PENDING_TRANSACTIONS_FETCHER: ${POLYMESH_BLOCKSCOUT_DISABLE_PENDING_TX_FETCHER:-true}
# Native Substrate extrinsics appear as pseudo-Ethereum transactions with
# no EVM call frames, so tracing them fails and the fetcher retries those
# blocks forever. Nothing is lost: their traces are empty either way.
INDEXER_DISABLE_INTERNAL_TRANSACTIONS_FETCHER: ${POLYMESH_BLOCKSCOUT_DISABLE_INTERNAL_TX_FETCHER:-true}
DISABLE_MARKET: 'true'
ACCOUNT_ENABLED: 'false'
DISABLE_WEBAPP: 'false'
Expand Down Expand Up @@ -365,6 +375,10 @@ services:
# Required to enable in-browser wallet interaction (contract Write tab,
# MetaMask, etc). Get a free project ID from https://cloud.reown.com
NEXT_PUBLIC_WALLET_CONNECT_PROJECT_ID: ${POLYMESH_BLOCKSCOUT_WALLET_CONNECT_PROJECT_ID:-}
# RPC endpoint the browser (wagmi/viem) uses for Contract Read/Write
# calls and the "Add to wallet" network config. Must be reachable from
# the host, not the Docker network, so use the published eth-rpc port.
NEXT_PUBLIC_NETWORK_RPC_URL: ${POLYMESH_BLOCKSCOUT_NETWORK_RPC_URL:-http://localhost:${POLYMESH_ETH_RPC_PORT:-8545}}
# The upstream frontend image enables third-party ad slots by default
# (a banner on the home page and a text ad on token pages). They are
# disabled here; set a provider name to re-enable.
Expand All @@ -379,6 +393,8 @@ services:
# non-root and its binary carries no cap_ipc_lock file capability, so mlock
# fails wherever the host's memlock rlimit is finite.
image: hashicorp/vault:${VAULT_VERSION:-latest}
profiles:
- rest-api
networks:
- polymesh
restart: unless-stopped
Expand All @@ -395,6 +411,8 @@ services:
# This will make sure vault is initialized and unsealed on the first run and unsealed on subsequent runs
vault-init:
image: hashicorp/vault:${VAULT_VERSION:-latest}
profiles:
- rest-api
user: '0:0'
networks:
- polymesh
Expand Down Expand Up @@ -427,10 +445,30 @@ services:
image: alpine:latest
volumes:
- rest-api-accounts-init:/status:ro
depends_on:
polymesh-node:
condition: service_healthy
subquery-graphql:
condition: service_healthy
environment:
# The REST API and Vault services sit behind the `rest-api` profile, so
# the account-init marker they write is only produced when that profile
# is enabled. Without it, readiness is just the core services being
# healthy, which `depends_on` above already guarantees. `start-env.sh`
# sets this whenever it enables the profile.
WAIT_FOR_REST_API: ${POLYMESH_WAIT_FOR_REST_API:-false}
command: >
sh -c '
start_time=$$(date +%s)

if [ "$$WAIT_FOR_REST_API" != "true" ]; then
echo "************************************************************************************"
echo "*** Polymesh Environment Ready! (core services healthy) ****************************"
echo "*** REST API and Vault are not running: start with --profile rest-api to add them **"
echo "************************************************************************************"
exit 0
fi

if [ -f /status/.setup-complete ]; then
echo "************************************************************************************"
echo "*** Environment previously initialized and is ready! *******************************"
Expand Down
6 changes: 6 additions & 0 deletions envs/8.0
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,19 @@ POLYMESH_SUBQUERY_GRAPHQL_PORT=3000
# POLYMESH_BLOCKSCOUT_REDIS_IMAGE=redis:alpine
# POLYMESH_BLOCKSCOUT_RPC_VARIANT=geth
# POLYMESH_BLOCKSCOUT_DISABLE_PENDING_TX_FETCHER=true
# Native Substrate extrinsics appear as pseudo-Ethereum transactions that
# cannot be traced, so the fetcher would retry those blocks forever.
# POLYMESH_BLOCKSCOUT_DISABLE_INTERNAL_TX_FETCHER=true
# POLYMESH_BLOCKSCOUT_NETWORK=Polymesh
# POLYMESH_BLOCKSCOUT_SUBNETWORK=Polymesh Dev
# POLYMESH_BLOCKSCOUT_COIN=POLYX
# POLYMESH_BLOCKSCOUT_COIN_NAME=POLYX
# Enables in-browser wallet interaction (contract Write tab, MetaMask, etc).
# Get a free project ID from https://cloud.reown.com
# POLYMESH_BLOCKSCOUT_WALLET_CONNECT_PROJECT_ID=
# RPC URL the browser uses for Contract Read/Write and "Add to wallet".
# Defaults to http://localhost:<POLYMESH_ETH_RPC_PORT> in compose.yaml.
# POLYMESH_BLOCKSCOUT_NETWORK_RPC_URL=
# Third-party ad slots are disabled by default. Set a provider name
# (e.g. slise, coinzilla) to re-enable them.
# POLYMESH_BLOCKSCOUT_AD_BANNER_PROVIDER=none
Expand Down
6 changes: 6 additions & 0 deletions envs/latest
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,19 @@ POLYMESH_SUBQUERY_GRAPHQL_PORT=3000
# POLYMESH_BLOCKSCOUT_REDIS_IMAGE=redis:alpine
# POLYMESH_BLOCKSCOUT_RPC_VARIANT=geth
# POLYMESH_BLOCKSCOUT_DISABLE_PENDING_TX_FETCHER=true
# Native Substrate extrinsics appear as pseudo-Ethereum transactions that
# cannot be traced, so the fetcher would retry those blocks forever.
# POLYMESH_BLOCKSCOUT_DISABLE_INTERNAL_TX_FETCHER=true
# POLYMESH_BLOCKSCOUT_NETWORK=Polymesh
# POLYMESH_BLOCKSCOUT_SUBNETWORK=Polymesh Dev
# POLYMESH_BLOCKSCOUT_COIN=POLYX
# POLYMESH_BLOCKSCOUT_COIN_NAME=POLYX
# Enables in-browser wallet interaction (contract Write tab, MetaMask, etc).
# Get a free project ID from https://cloud.reown.com
# POLYMESH_BLOCKSCOUT_WALLET_CONNECT_PROJECT_ID=
# RPC URL the browser uses for Contract Read/Write and "Add to wallet".
# Defaults to http://localhost:<POLYMESH_ETH_RPC_PORT> in compose.yaml.
# POLYMESH_BLOCKSCOUT_NETWORK_RPC_URL=
# Third-party ad slots are disabled by default. Set a provider name
# (e.g. slise, coinzilla) to re-enable them.
# POLYMESH_BLOCKSCOUT_AD_BANNER_PROVIDER=none
Expand Down
6 changes: 6 additions & 0 deletions envs/local
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,19 @@ POLYMESH_SUBQUERY_GRAPHQL_PORT=3000
# POLYMESH_BLOCKSCOUT_REDIS_IMAGE=redis:alpine
# POLYMESH_BLOCKSCOUT_RPC_VARIANT=geth
# POLYMESH_BLOCKSCOUT_DISABLE_PENDING_TX_FETCHER=true
# Native Substrate extrinsics appear as pseudo-Ethereum transactions that
# cannot be traced, so the fetcher would retry those blocks forever.
# POLYMESH_BLOCKSCOUT_DISABLE_INTERNAL_TX_FETCHER=true
# POLYMESH_BLOCKSCOUT_NETWORK=Polymesh
# POLYMESH_BLOCKSCOUT_SUBNETWORK=Polymesh Dev
# POLYMESH_BLOCKSCOUT_COIN=POLYX
# POLYMESH_BLOCKSCOUT_COIN_NAME=POLYX
# Enables in-browser wallet interaction (contract Write tab, MetaMask, etc).
# Get a free project ID from https://cloud.reown.com
# POLYMESH_BLOCKSCOUT_WALLET_CONNECT_PROJECT_ID=
# RPC URL the browser uses for Contract Read/Write and "Add to wallet".
# Defaults to http://localhost:<POLYMESH_ETH_RPC_PORT> in compose.yaml.
# POLYMESH_BLOCKSCOUT_NETWORK_RPC_URL=
# Third-party ad slots are disabled by default. Set a provider name
# (e.g. slise, coinzilla) to re-enable them.
# POLYMESH_BLOCKSCOUT_AD_BANNER_PROVIDER=none
Expand Down
14 changes: 14 additions & 0 deletions envs/template
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,13 @@ POLYMESH_SUBQUERY_IMAGE=polymeshassociation/polymesh-subquery:v19.6.0
POLYMESH_SUBQUERY_GRAPHQL_IMAGE=onfinality/subql-query:v2.25.0
POLYMESH_SUBQUERY_GRAPHQL_PORT=3000

# The REST API and Vault services are opt-in (`--profile rest-api`). This makes
# `environment-ready` wait for their account setup instead of reporting ready
# once the core services are healthy; `start-env.sh` sets it to match the
# profiles it was given, so it is only needed when running docker compose
# directly.
# POLYMESH_WAIT_FOR_REST_API=false

# Optional EVM tooling (v8+ only, used with --profile evm)
# POLYMESH_EVM_CHAIN_ID=1641818
# POLYMESH_ETH_RPC_IMAGE=paritypr/eth-rpc:stable2606-73b734d9
Expand All @@ -35,13 +42,20 @@ POLYMESH_SUBQUERY_GRAPHQL_PORT=3000
# POLYMESH_BLOCKSCOUT_REDIS_IMAGE=redis:alpine
# POLYMESH_BLOCKSCOUT_RPC_VARIANT=geth
# POLYMESH_BLOCKSCOUT_DISABLE_PENDING_TX_FETCHER=true
# Native Substrate extrinsics appear as pseudo-Ethereum transactions that
# cannot be traced, so the fetcher would retry those blocks forever.
# POLYMESH_BLOCKSCOUT_DISABLE_INTERNAL_TX_FETCHER=true
# POLYMESH_BLOCKSCOUT_NETWORK=Polymesh
# POLYMESH_BLOCKSCOUT_SUBNETWORK=Polymesh Dev
# POLYMESH_BLOCKSCOUT_COIN=POLYX
# POLYMESH_BLOCKSCOUT_COIN_NAME=POLYX
# Enables in-browser wallet interaction (contract Write tab, MetaMask, etc).
# Get a free project ID from https://cloud.reown.com
# POLYMESH_BLOCKSCOUT_WALLET_CONNECT_PROJECT_ID=
# RPC URL the browser uses for Contract Read/Write and "Add to wallet".
# Defaults to http://localhost:<POLYMESH_ETH_RPC_PORT>; override if the
# eth-rpc port is published elsewhere or the browser runs on another host.
# POLYMESH_BLOCKSCOUT_NETWORK_RPC_URL=

# Optional frontend overrides
# POLYMESH_BLOCKSCOUT_FRONTEND_API_HOST=localhost
Expand Down
7 changes: 7 additions & 0 deletions scripts/start-env.sh
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,13 @@ if [[ -n "$COMPOSE_PROFILES" ]]; then
done
fi

# `environment-ready` waits on a marker written by the REST API account init,
# which only runs with the `rest-api` profile. Tell it which of the two it is.
case ",$COMPOSE_PROFILES," in
*,rest-api,*) export POLYMESH_WAIT_FOR_REST_API=true ;;
*) export POLYMESH_WAIT_FOR_REST_API=false ;;
esac

UP_ARGS=(--detach)
if [[ -n "$COMPOSE_PULL_POLICY" ]]; then
UP_ARGS+=(--pull "$COMPOSE_PULL_POLICY")
Expand Down
Loading
Loading