From 29ca3df7992536fa9bc934914d8ca0a37fcbd460 Mon Sep 17 00:00:00 2001 From: Alex Z Date: Tue, 9 Jun 2026 13:36:47 +0700 Subject: [PATCH] Exporter Node section --- docs/operators/exporter/README.md | 87 ++++++ docs/operators/exporter/api-reference.md | 257 ++++++++++++++++++ .../exporter/configuration-reference.md | 164 +++++++++++ docs/operators/exporter/setup.md | 108 ++++++++ docs/operators/exporter/troubleshooting.md | 185 +++++++++++++ 5 files changed, 801 insertions(+) create mode 100644 docs/operators/exporter/README.md create mode 100644 docs/operators/exporter/api-reference.md create mode 100644 docs/operators/exporter/configuration-reference.md create mode 100644 docs/operators/exporter/setup.md create mode 100644 docs/operators/exporter/troubleshooting.md diff --git a/docs/operators/exporter/README.md b/docs/operators/exporter/README.md new file mode 100644 index 0000000..a0ed69c --- /dev/null +++ b/docs/operators/exporter/README.md @@ -0,0 +1,87 @@ +--- +title: Exporter +sidebar_label: Exporter +sidebar_position: 10 +--- + +# Exporter + +Exporter mode runs `ssv-node` as an observation and query service instead of as a validator-operating node. When exporter mode is enabled, the node connects to the SSV network, subscribes to network traffic, stores exporter data locally, and exposes exporter APIs for querying observed participation and, in archive mode, duty traces. + +Exporter node's API serves as the data source for [SSV Explorer](https://explorer.ssv.network/mainnet/overview) and [ethereum2-monitor](https://hub.docker.com/r/ssvlabs/ethereum2-monitor). + +## Supported versions + +Use an `ssv-node` build that includes Exporter v2 support. Check the release notes for the minimum supported version and any exporter-specific upgrade steps for your deployment. + +Exporter functionality is not necessarily updated in every `ssv-node` release. When planning an upgrade, check the release notes for exporter-specific changes. If exporter mode is not mentioned, exporter-specific update work can usually be skipped, although you should still follow any general `ssv-node` upgrade requirements that apply to your deployment. + +## What exporter mode does + +Exporter mode is designed for reading network activity, not for performing validator duties. + +- It does not originate validator duties or signing duties. +- It skips operator signing and key-management services used by regular operator nodes. +- It subscribes broadly to SSV network traffic so it can observe messages from the network. +- It stores exporter data locally and serves read APIs. + +The exact data available depends on the selected exporter mode. + +## Exporter modes + +Exporter mode supports two modes: `standard` and `archive`. + +### Standard mode + +Standard mode is the lighter exporter mode. It uses the standard mode data path and exposes decided participation data through the exporter API. Standard mode applies slot-based pruning through the exporter retention setting, with a default retention of `50400` slots. + +Use standard mode when you need recent decided participation data and want lower operational overhead than archive mode. + +### Archive mode + +Archive mode is the fuller tracing mode. It observes consensus and partial-signature messages, reconstructs best-effort duty traces, stores them locally, and exposes trace APIs for validator and committee duties in addition to decided data. + +Archive mode also runs exporter-specific duty scheduling and prefetching for all shares so it can build schedule and trace read models. It does not execute duties. Trace data is based on observed P2P messages, so missing data does not always mean a duty did not exist or did not happen. + +Use archive mode when you need detailed historical trace data and can allocate more CPU, memory, network, and disk resources. + +## Resource requirements + +Exporter mode generally uses more network resources than a regular single-operator workload because it subscribes broadly to SSV network traffic. Both modes use Pebble storage under the configured database path with a `-pebble` suffix when exporter mode is enabled. + +### Standard mode resources + +Standard mode is the lower-resource option. + +- **Network:** subscribes broadly to SSV network traffic. +- **Storage:** stores decided participant data and prunes old participant data by `RetainSlots`. +- **CPU and memory:** generally lower than archive mode because it does not run the archive duty tracer or archive trace APIs. + +### Archive mode resources + +Archive mode has higher resource requirements. + +- **Network:** subscribes broadly to SSV network traffic and observes consensus and partial-signature messages. +- **CPU and memory:** performs trace collection, schedule computation, and duty prefetching. It keeps a short in-memory trace window before writing traces to disk. +- **Storage:** persists validator duty traces, committee duty traces, validator-to-committee links, and schedule data. No archive trace pruning limit has been identified, so disk usage can grow over time. +- **API load:** archive API queries can scan inclusive slot ranges. Large ranges may be expensive, so keep query ranges practical for your infrastructure. + +## Choosing a mode + +Choose the mode based on the data you need and the resources you can dedicate. + +| Need | Recommended mode | +| --- | --- | +| Recent decided participation data with lower overhead | Standard mode | +| Detailed validator and committee duty traces | Archive mode | +| Long-running historical trace collection | Archive mode, with storage growth planned and monitored | +| Minimal exporter-specific maintenance | Standard mode | + +If you are not sure which mode to use, start with standard mode and move to archive mode only when you need trace endpoints or longer-term trace analysis. + +## Pages in this section + +- [Setup](setup): enable exporter mode, configure the API, start the node, and verify that the exporter is running. +- [Configuration Reference](configuration-reference): look up exporter configuration keys, environment variables, defaults, and mode-specific settings. +- [API Reference](api-reference): review exporter endpoints, request fields, responses, and API behavior. +- [Troubleshooting](troubleshooting): diagnose common startup, API, storage, and data-availability issues. diff --git a/docs/operators/exporter/api-reference.md b/docs/operators/exporter/api-reference.md new file mode 100644 index 0000000..cbf1f38 --- /dev/null +++ b/docs/operators/exporter/api-reference.md @@ -0,0 +1,257 @@ +--- +title: API Reference +sidebar_label: API Reference +sidebar_position: 4 +--- + +# API Reference + +The exporter HTTP API exposes observed SSV Network participation data from an exporter-enabled `ssv-node`. Standard mode exposes decided-participant queries. Archive mode adds trace queries and uses a trace-derived decided backend. + +This page is a human-readable reference for the exporter endpoints. It describes the request fields, response behavior, and mode-specific caveats that API clients should handle. + +## Base URL + +Exporter endpoints are served by the `ssv-node` HTTP API server, not by the WebSocket API. The base URL is determined by the node's HTTP API listen address and port: + +```text +http://: +``` + +For a local exporter configured with `SSVAPIAddress: 127.0.0.1` and `SSVAPIPort: 16000`, the base URL is: + +```text +http://127.0.0.1:16000 +``` + +If the HTTP API port is disabled or unavailable from your client, these endpoints are not reachable. + +## Availability by mode + +| Endpoint | Standard mode | Archive mode | +| --- | --- | --- | +| `GET /v1/exporter/decideds` | Supported; uses the standard-mode decided backend | Supported; uses trace-derived archive data | +| `POST /v1/exporter/decideds` | Supported; uses the standard-mode decided backend | Supported; uses trace-derived archive data | +| `GET /v1/exporter/traces/validator` | Not available | Supported | +| `POST /v1/exporter/traces/validator` | Not available | Supported | +| `GET /v1/exporter/traces/committee` | Not available | Supported | +| `POST /v1/exporter/traces/committee` | Not available | Supported | + +The `/v1/exporter/decideds` route is intentionally listed for both modes, but its backend behavior differs: + +- **Standard mode:** reads from the standard-mode decided backend. +- **Archive mode:** reads decided data from archive trace storage. + +If a client can query both standard and archive exporters, do not assume that `/v1/exporter/decideds` has identical semantics in both modes. + +## Request formats + +Each endpoint supports both `GET` and `POST`: + +- Use `GET` with query parameters for simple queries. +- Use `POST` with a JSON body for structured clients or larger filter sets. + +The same logical filters are accepted in both formats, but field casing is not always identical between `GET` query parameters and `POST` JSON bodies. Slot ranges are inclusive. + +### GET query parameter casing + +For `GET` requests, query parameter names are bound from the Go request field name unless the field has an explicit `form` tag. In the current committee trace request model, `CommitteeIDs` has `json:"committeeIDs"` but no `form` tag, so the query parameter name is lowercased to `committeeids`. + +- Use `committeeids` in `GET /v1/exporter/traces/committee` URLs. +- Use `committeeIDs` in `POST /v1/exporter/traces/committee` JSON bodies. + +Do not use `committeeIDs` in a `GET` URL unless the source code changes to add a matching `form` tag. In the current implementation, `committeeIDs` is not bound as a query filter, so it will not trigger committee ID length validation and can result in a broader or unfiltered committee trace query. + +### GET query example + +```bash +curl -G http://127.0.0.1:16000/v1/exporter/decideds \ + --data-urlencode 'from=' \ + --data-urlencode 'to=' \ + --data-urlencode 'roles=PROPOSER' +``` + +### POST body example + +```bash +curl -X POST http://127.0.0.1:16000/v1/exporter/decideds \ + -H 'Content-Type: application/json' \ + -d '{ + "from": , + "to": , + "roles": ["PROPOSER"] + }' +``` + +Use narrow slot ranges first. Exporter endpoints loop over the requested range, and no maximum range validation has been identified. + +## Common request fields + +| Field | Type | Used by | Notes | +| --- | --- | --- | --- | +| `from` | integer slot | All exporter endpoints | Start of the inclusive slot range. Must be less than or equal to `to`. | +| `to` | integer slot | All exporter endpoints | End of the inclusive slot range. | +| `roles` | role string or list of role strings | Validator traces and decideds | Required for validator traces and decideds. Not supported by committee trace requests. | +| `pubkeys` | validator public key hex string or list | Validator traces and decideds | Optional validator filter. Each validator public key is 48 bytes, represented as 96 hex characters. | +| `indices` | validator index or list of indices | Validator traces and archive-mode decideds | Optional validator index filter. For `/v1/exporter/decideds`, this filter applies only in archive mode; standard mode ignores it. Can be combined with `pubkeys`; resolved indices are deduplicated where supported. | +| `committeeids` / `committeeIDs` | committee ID hex string or list | Committee traces | Optional committee filter. Use `committeeids` for `GET` query parameters and `committeeIDs` for `POST` JSON bodies. Each committee ID is 32 bytes, represented as 64 hex characters. | + +Supported role values are: + +- `ATTESTER` +- `AGGREGATOR` +- `PROPOSER` +- `SYNC_COMMITTEE` +- `SYNC_COMMITTEE_CONTRIBUTION` + +For `GET` requests, role query values are parsed from comma-separated strings. For `POST` requests, send roles as a JSON string array. + +### Hex field length validation + +Exporter handlers validate hex identifier lengths before running the query: + +- Validator public keys must decode to 48 bytes, or 96 hex characters. +- Committee IDs must decode to 32 bytes, or 64 hex characters. + +Invalid lengths return `400 Bad Request`. + +For committee trace requests, length validation applies to bound committee ID values. POST JSON `committeeIDs` values are validated, and GET `committeeids` values are validated. GET `committeeIDs` is not bound in the current implementation and therefore does not trigger committee ID length validation. + +## Endpoints + +### `GET|POST /v1/exporter/traces/validator` + +Returns archive-mode validator duty traces for the requested slot range, roles, and optional validator filters. + +**Availability:** archive mode only. + +**Request fields** + +| Field | Notes | +| --- | --- | +| `from` | Inclusive start slot. | +| `to` | Inclusive end slot. | +| `roles` | Required. One or more supported roles. | +| `pubkeys` | Optional validator public key filters. Each key must be 96 hex characters. | +| `indices` | Optional validator index filters. | + +When querying committee-duty roles through the validator trace endpoint, such as `ATTESTER` or `SYNC_COMMITTEE`, provide `pubkeys` or `indices`. If you need committee-wide trace data without validator filters, use `/v1/exporter/traces/committee`. + +**Response** + +The response includes: + +- `data`: validator trace entries, including slot, role, validator index, optional committee ID, consensus rounds, decided messages, pre-consensus partial signatures, post-consensus partial signatures, and optional proposal data. +- `schedule`: best-effort schedule entries for the requested validators and roles. +- `errors`: optional partial errors returned with successful partial responses. + +Trace data is based on messages observed by the exporter. Schedule data is a separate best-effort read model of expected duties. + +### `GET|POST /v1/exporter/traces/committee` + +Returns archive-mode committee duty traces for the requested slot range and optional committee ID filters. + +**Availability:** archive mode only. + +**Request fields** + +| Field | Notes | +| --- | --- | +| `from` | Inclusive start slot. | +| `to` | Inclusive end slot. | +| `committeeids` / `committeeIDs` | Optional committee ID filters. Use `committeeids` for `GET` query parameters and `committeeIDs` for `POST` JSON bodies. Each ID must be 64 hex characters. | + +Committee trace requests do **not** support a `roles` filter. If a `roles` field appears in older examples or client code, do not rely on it for committee trace filtering. + +**GET example** + +```bash +curl -G http://127.0.0.1:16000/v1/exporter/traces/committee \ + --data-urlencode 'from=' \ + --data-urlencode 'to=' \ + --data-urlencode 'committeeids=<64-char-committee-id-hex>' +``` + +**POST example** + +```bash +curl -X POST http://127.0.0.1:16000/v1/exporter/traces/committee \ + -H 'Content-Type: application/json' \ + -d '{ + "from": , + "to": , + "committeeIDs": ["<64-char-committee-id-hex>"] + }' +``` + +**Response** + +The response includes: + +- `data`: committee trace entries, including slot, committee ID, consensus rounds, decided messages, sync committee signer data, attester signer data, and optional proposal data. +- `schedule`: best-effort committee schedule entries grouped by role and validator index. +- `errors`: optional partial errors returned with successful partial responses. + +Committee trace data depends on the exporter observing and classifying committee messages. Missing committee traces do not prove that no committee duty existed. + +### `GET|POST /v1/exporter/decideds` + +Returns decided participation data for the requested slot range, roles, and optional validator filters. + +**Availability:** standard mode and archive mode. + +**Request fields** + +| Field | Notes | +| --- | --- | +| `from` | Inclusive start slot. | +| `to` | Inclusive end slot. | +| `roles` | Required. One or more supported roles. | +| `pubkeys` | Optional validator public key filters. Each key must be 96 hex characters. | +| `indices` | Optional validator index filters in archive mode only. Ignored by the standard-mode decided backend. | + +**Runtime caveat** + +This route uses different implementations depending on exporter mode: + +- In **archive mode**, the endpoint is trace-derived, applies `indices` and `pubkeys` filters, and can return partial `200 OK` responses with `errors[]`. +- In **standard mode**, the endpoint reads the standard-mode decided backend. It filters by `roles` and `pubkeys`; it does not read `indices`. If no `pubkeys` are supplied, it can return all participants in the slot range for the selected roles. The standard-mode response type has the same top-level fields, but current conversion does not populate `errors[]` in the same way as archive trace queries. + +**Response** + +The response includes: + +- `data`: decided participant entries with role, slot, validator public key, and signer information. +- `errors`: optional partial errors where supported by the active backend. + +Archive decided results are derived from trace storage. Standard decided results are derived from the standard-mode decided backend. + +## Errors + +Exporter endpoints return JSON responses. Error handling differs slightly by endpoint and mode, but clients should handle these common cases: + +| Status | Meaning | +| --- | --- | +| `200 OK` | Query completed. The response may still include an `errors` array when partial data was returned. | +| `400 Bad Request` | Request binding or validation failed, such as invalid JSON, `from` greater than `to`, missing required roles, or invalid hex identifier length on a bound field. | +| `500 Internal Server Error` | The query produced no data and encountered a meaningful backend error, or a non-validation backend error could not be returned as a partial success. | + +Not-found results are suppressed by archive trace queries: if the exporter does not find data for a requested slot, role, validator, or committee, that not-found condition is not returned in `errors[]` and is not treated as a server error by itself. A query that only misses data can return `200 OK` with empty `data`. + +When partial data is available and non-not-found errors also occur, archive trace endpoints can return `200 OK` with the successful data and an `errors` array. Treat `errors[]` as part of the result, especially for wider slot ranges or mixed filters. + +Empty decided entries with no signers may be omitted from archive decided responses without an API error. + +## Schema and source caveats + +This page intentionally does not reproduce the full generated OpenAPI schema. The exporter models include nested trace objects, consensus rounds, decided messages, proposal traces, and signer data, and some runtime behavior is not fully represented by the generated schema. + +Known caveats: + +- The generated OpenAPI documents all exporter paths, but it does not document that `/v1/exporter/decideds` switches implementation between standard mode and archive mode. +- Committee trace requests have no `roles` field in the handler model or generated OpenAPI. +- The committee ID field is `committeeIDs` in POST JSON, but the current GET query parameter is `committeeids`. +- Some numeric Ethereum consensus types may appear differently between generated schema annotations and actual JSON encoding. +- Trace and schedule data are best-effort. Empty responses can mean the exporter did not observe, classify, persist, or match the requested data. + +For practical client behavior, rely on the endpoint descriptions above, handle partial `200 OK` responses with `errors[]`, and keep queries narrowly scoped. diff --git a/docs/operators/exporter/configuration-reference.md b/docs/operators/exporter/configuration-reference.md new file mode 100644 index 0000000..a255e58 --- /dev/null +++ b/docs/operators/exporter/configuration-reference.md @@ -0,0 +1,164 @@ +--- +title: Configuration Reference +sidebar_label: Configuration Reference +sidebar_position: 3 +--- + +# Configuration Reference + +One of the most important parameters for an Exporter node is MaxPeers. The more peers Exporter node connects to, the more network information it can have. + +Recommended values are `p2p.DynamicMaxPeersLimit: 1200` and `p2p.MaxPeers: 1000` + +### Exporter Variables + +Exporter mode is configured under the `exporter` YAML block or through matching environment variables. + +| YAML key | Environment variable | Supported values | Default | Description | +| --- | --- | --- | --- | --- | +| `exporter.Enabled` | `EXPORTER` | `true`, `false` | `false` | Enables exporter mode. When enabled, the node runs as an observation and query service and skips operator signing and key-manager services. | +| `exporter.Mode` | `EXPORTER_MODE` | `standard`, `archive` | `standard` | Selects the exporter mode. Invalid values are fatal at startup. | +| `exporter.RetainSlots` | `EXPORTER_RETAIN_SLOTS` | unsigned integer slot count | `50400` | Slot retention setting used by the standard-mode decided data pruning path. See [Retention behavior](#retention-behavior). | + +### Standard mode + +Standard mode is selected by enabling exporter mode and using `standard` as the exporter mode. Because the default for `exporter.Mode` is `standard`, a node with `exporter.Enabled: true` and no explicit mode uses standard mode. + + +```yaml +p2p: + MaxPeers: 1000 + DynamicMaxPeersLimit: 1200 + +SSVAPIAddress: 127.0.0.1 +SSVAPIPort: 16000 + +exporter: + Enabled: true + Mode: standard + RetainSlots: 50400 +``` + +Environment-variable equivalent: + +```dotenv +P2P_MAX_PEERS=1000 +P2P_DYNAMIC_MAX_PEERS_LIMIT=1200 +SSV_API_ADDRESS=127.0.0.1 +SSV_API_PORT=16000 +EXPORTER=true +EXPORTER_MODE=standard +EXPORTER_RETAIN_SLOTS=50400 +``` + +### Archive mode + +Archive mode is selected by enabling exporter mode and setting the mode to `archive`. + +```yaml +p2p: + MaxPeers: 1000 + DynamicMaxPeersLimit: 1200 + +SSVAPIAddress: 127.0.0.1 +SSVAPIPort: 16000 + +exporter: + Enabled: true + Mode: archive +``` + +Environment-variable equivalent: + +```dotenv +P2P_MAX_PEERS=1000 +P2P_DYNAMIC_MAX_PEERS_LIMIT=1200 +SSV_API_ADDRESS=127.0.0.1 +SSV_API_PORT=16000 +EXPORTER=true +EXPORTER_MODE=archive +``` + +`exporter.RetainSlots` is still a valid configuration key in archive mode because it is part of the exporter options, but no archive duty-trace pruning path that uses it has been identified. Plan archive storage accordingly. + +## API configuration + +The exporter HTTP endpoints are served through the node HTTP API. The WebSocket API is configured separately. + +| YAML key | Environment variable | Default | Description | +| --- | --- | --- | --- | +| `SSVAPIAddress` | `SSV_API_ADDRESS` | No struct default identified. Empty address listens on all interfaces when `SSVAPIPort` is greater than `0`. | Listen address for the HTTP API. Use `127.0.0.1` for local-only access. | +| `SSVAPIPort` | `SSV_API_PORT` | No struct default identified. `0` disables the HTTP API. | Listen port for the HTTP API, including exporter HTTP endpoints. | +| `WebSocketAPIPort` | `WS_API_PORT` | No struct default identified. `0` disables the WebSocket API. | Listen port for the WebSocket API. It has no built-in authentication, binds to all interfaces when enabled, and accepts cross-origin WebSocket connections (`CheckOrigin` returns true). | +| `WithPing` | `WITH_PING` | No default identified. | Enables WebSocket ping messages. | + +Recommended local HTTP API binding: + +```yaml +SSVAPIAddress: 127.0.0.1 +SSVAPIPort: 16000 +WebSocketAPIPort: 0 +``` + +If `SSVAPIAddress` is omitted while `SSVAPIPort` is configured, the HTTP API listens on all interfaces for backward compatibility. The HTTP server setup does not include built-in authentication or CORS middleware. Keep APIs local-only unless remote access is intentional and protected by network controls such as a private network, firewall, or authenticated reverse proxy. + +## Database configuration + +Exporter-enabled nodes use Pebble storage. Non-exporter nodes use Badger storage. + +| YAML key | Environment variable | Default | Description | +| --- | --- | --- | --- | +| `db.Path` | `DB_PATH` | `./data/db` | Base database path. Exporter-enabled nodes use Pebble at `DB_PATH + "-pebble"`. | +| `db.Reporting` | `DB_REPORTING` | `false` | Database reporting setting. | +| `db.GCInterval` | `DB_GC_INTERVAL` | `6m` | Database garbage-collection interval setting. | + +When exporter mode is enabled, the startup path calls the Pebble setup path and derives the Pebble location by appending `-pebble` to `db.Path`. + +For example, with the default base path: + +```yaml +db: + Path: ./data/db +``` + +the exporter Pebble path is: + +```text +./data/db-pebble +``` + +If you change `db.Path`, update disk-capacity planning for the corresponding `-pebble` directory. + +## Retention behavior + +Retention behavior depends on the selected exporter mode. + +### Standard mode retention + +Standard mode uses the standard decided data stores. Standard mode initializes pruning only when exporter mode is enabled and `exporter.Mode` is `standard`. + +The pruning path: + +- uses `exporter.RetainSlots` / `EXPORTER_RETAIN_SLOTS`; +- computes an initial threshold from the current slot minus `RetainSlots`; +- starts continuous pruning for each standard-mode decided store as slots advance. + +The default retention value is `50400` slots. Data older than the retained slot window may no longer be available from that exporter node. + +### Archive mode retention + +Archive mode persists duty-trace data through the duty tracer and exporter store. Persisted archive objects include: + +- validator duty traces; +- committee duty traces; +- validator-to-committee links; +- scheduled duty data. + +`exporter.RetainSlots` exists in archive mode because it is part of the exporter options, but no archive duty-trace pruning path tied to `RetainSlots` has been identified. Do not assume archive trace data is pruned automatically by `RetainSlots`; monitor disk growth and add external retention or cleanup if your deployment requires it. + +Archive trace collection also uses a short in-memory window before data is written to the database. The in-memory duty-tracer TTL is `4` slots. On slot ticks, traces around `currentSlot - 4` are evicted from memory and dumped to the database. Late-arriving messages may still update persisted traces, but late processing is best effort. + +## Next Steps + +- Use [API Reference](api-reference) for endpoint fields and response behavior. +- Use [Troubleshooting](troubleshooting) if the node starts but exporter data is missing or API checks fail. diff --git a/docs/operators/exporter/setup.md b/docs/operators/exporter/setup.md new file mode 100644 index 0000000..de19924 --- /dev/null +++ b/docs/operators/exporter/setup.md @@ -0,0 +1,108 @@ +--- +title: Setup +sidebar_label: Setup +sidebar_position: 2 +--- + +# Setup + +## Prerequisites + +- An installed `ssv-node` version that supports exporter v2 (node version `v2.4.x` or newer) +- A working node configuration for the target SSV P2P network. +- Execution and Beacon nodes to run alongside with SSV node. +- A writable database path with enough disk space for the selected exporter mode. + +## Choose a setup path + +Exporter mode has two setup paths: + +- [**Standard mode:**](#configure-standard-mode) use this path for recent decided participation data with lower operational overhead. +- [**Archive mode:**](#configure-archive-mode) use this path for validator and committee duty traces, schedules, and historical trace analysis. Archive mode uses more CPU, memory, network, and disk resources. + +## Configure standard mode + +Standard mode enables exporter mode with the standard exporter data path. It exposes decided participation data through the exporter API and applies slot-based pruning through the exporter retention setting. + +At a high level, configure the node to: + +1. Enable exporter mode. +2. Select `standard` as the exporter mode. +3. Configure the HTTP API listen address and port. +4. Set retention for standard exporter data if the default does not fit your needs. + +For exact configuration keys, environment variables, defaults, and accepted values, see [Configuration Reference](configuration-reference). + +## Configure archive mode + +Archive mode enables exporter mode with the archive trace data path. It observes consensus and partial-signature messages, reconstructs best-effort duty traces, persists them locally, and exposes archive trace APIs. + +At a high level, configure the node to: + +1. Enable exporter mode. +2. Select `archive` as the exporter mode. +3. Configure the HTTP API listen address and port. +4. Plan storage for long-running trace persistence. +5. Plan for higher P2P, CPU, memory, and disk usage than standard mode. + +Archive mode subscribes broadly to SSV network traffic and runs exporter-specific scheduling and prefetching for all shares so it can build schedule and trace read models. Trace data is best-effort observation data. Missing trace data does not always mean that a duty did not exist or did not happen. + +For exact configuration keys, environment variables, defaults, and accepted values, see [Configuration Reference](configuration-reference). + +## Start the node + +Start `ssv-node` with your [normal node start command](/operators/operator-node/node-setup/manual-setup#start-the-node) and the exporter configuration you prepared. + +The exact command depends on how you install and run `ssv-node`. For example, use the same service, container, or binary invocation pattern you already use for the node, but point it at the exporter configuration. + +During startup, check the logs for exporter-related messages, including: + +- exporter mode being enabled; +- Pebble database usage for exporter-enabled nodes; +- skipped operator signing and key-manager services; +- invalid exporter mode errors, if the selected mode is not accepted; +- HTTP API startup on the address and port you configured. + +## Verify the exporter + +Run verification checks from a host that can reach the configured HTTP API. The examples below assume the API is bound locally on port `16000`; adjust the host and port for your configuration. + +Check that the node API responds: + +```bash +curl http://127.0.0.1:16000/v1/node/health +``` + +Check the decided participation endpoint. Use a small slot range where you expect data: + +```bash +curl -G http://127.0.0.1:16000/v1/exporter/decideds \ + --data-urlencode 'from=' \ + --data-urlencode 'to=' \ + --data-urlencode 'roles=PROPOSER' +``` + +In standard mode, `/v1/exporter/decideds` uses the standard decided-participant data path. In archive mode, the same path is trace-derived. + +For archive mode, also check one or both trace endpoints with a small slot range: + +```bash +curl -G http://127.0.0.1:16000/v1/exporter/traces/validator \ + --data-urlencode 'from=' \ + --data-urlencode 'to=' \ + --data-urlencode 'roles=PROPOSER' +``` + +```bash +curl -G http://127.0.0.1:16000/v1/exporter/traces/committee \ + --data-urlencode 'from=' \ + --data-urlencode 'to=' +``` + +An empty response does not always mean the exporter is misconfigured. Exporter data depends on what the node has observed, the selected mode, the slot range, and whether data has already been written to the relevant store. Start with recent, narrow ranges and confirm that the node is connected to peers before increasing query scope. + +## Next steps + +- Use [Configuration Reference](configuration-reference) for config examples, environment variables, and defaults. +- Use [API Reference](api-reference) for endpoint fields and response behavior. +- Use [Troubleshooting](troubleshooting) if the node starts but exporter data is missing or API checks fail. diff --git a/docs/operators/exporter/troubleshooting.md b/docs/operators/exporter/troubleshooting.md new file mode 100644 index 0000000..916f7c7 --- /dev/null +++ b/docs/operators/exporter/troubleshooting.md @@ -0,0 +1,185 @@ +--- +title: Troubleshooting +sidebar_label: Troubleshooting +sidebar_position: 5 +--- + +# Troubleshooting + +Use this guide when exporter endpoints are unavailable, responses are empty or partial, traces are missing, or archive storage is growing faster than expected. + +Exporter data is local to the node and depends on the selected mode, observed P2P messages, local retention, and query filters. Start with the smallest possible query and confirm the active mode before assuming data is missing. + +## Known limitations + +Archive mode data is useful for tracing and analysis, but it has important limitations: + +- Traces are best-effort observations from P2P messages. Missing trace data is not proof that a duty did not happen. +- Schedule data is a best-effort read model of expected duties. Missing schedule data is not proof that no duty was scheduled. +- Schedule computation is asynchronous. Jobs can be dropped under backpressure, and schedule errors may be logged without appearing in API responses. +- The schedule writer populates attester, proposer, and sync committee schedule data. Aggregator and sync committee contribution bits are supported by storage/API models but were not found populated by the current schedule writer. +- Committee partial-signature classification depends on proposal data and signing roots. If the exporter cannot classify pending committee signatures, those signatures may be dropped when the in-memory slot window is evicted. +- Archive API queries loop over inclusive slot ranges and no maximum range validation has been identified. Large ranges can be resource-intensive. +- No archive trace pruning path tied to the standard retention setting has been found. Plan and monitor disk usage accordingly. +- `/v1/exporter/decideds` uses different backend behavior in archive mode than in standard mode. +- Some data can be returned with partial errors. Treat response errors as part of the result, especially when querying wider ranges. + +## Quick checks + +Before debugging a specific endpoint, verify the basics: + +1. **Confirm exporter mode is enabled.** The node must run with exporter mode enabled. Standard mode is selected by `standard`; archive mode is selected by `archive`. Invalid exporter mode values are fatal at startup. +2. **Confirm the endpoint is available in the selected mode.** Standard mode exposes only `/v1/exporter/decideds`. Archive mode also exposes validator and committee trace endpoints. See [API Reference](./api-reference.md). +3. **Confirm the HTTP API is enabled.** Exporter HTTP endpoints are served through `SSVAPIPort`. If the port is `0` or unavailable from your client, the HTTP exporter endpoints are not reachable. +4. **Use a narrow slot range first.** Exporter queries scan inclusive slot ranges. Large ranges can be slow, fail, or add significant load. +5. **Check the query filters.** Make sure `from` is less than or equal to `to`, required `roles` are present for decided and validator trace queries, public keys and committee IDs have the expected hex lengths, and the queried role is supported. For `/v1/exporter/decideds`, remember that `indices` filtering applies only in archive mode; standard mode ignores `indices` and filters by `roles` and `pubkeys`. +6. **Allow for persistence timing.** Archive trace collection keeps a short in-memory window before dumping traces to the database. Very recent slots may behave differently from older slots that have already been persisted. + +## Exporter endpoints are unavailable + +### Symptoms + +- The exporter URL cannot be reached. +- The node API responds, but an exporter route returns not found or does not behave as expected. +- Trace endpoints are unavailable while `/v1/exporter/decideds` is available. + +### Likely causes and actions + +| Cause | What to check | Action | +| --- | --- | --- | +| HTTP API is disabled | `SSVAPIPort` is unset, set to `0`, or the service/container is not publishing the port. | Enable the HTTP API port in the node configuration. | +| Client is using the wrong host or port | The request target does not match `SSVAPIAddress` and `SSVAPIPort`. | Query the configured HTTP API address and port. | +| Wrong exporter mode | Trace endpoints exist only in archive mode. | Use [Archive Mode](./archive-mode.md) for trace endpoints, or query `/v1/exporter/decideds` in standard mode. | +| Exporter mode is not enabled | The node is running as a regular operator node instead of an exporter-enabled node. | Enable exporter mode and restart using the intended exporter configuration. | +| Invalid exporter mode value | The node fails during startup when the mode is not accepted. | Use `standard` or `archive`. See [Configuration Reference](./configuration-reference.md). | + +## Responses are empty + +An empty `data` array is not always an error. Archive trace queries suppress not-found results, so a query that only misses data can return `200 OK` with empty `data` and no `errors[]`. + +### Common causes + +- The slot range is wrong, too old, too new, or outside the data retained by that exporter node. +- The selected exporter mode does not collect the requested data type. +- Standard mode retention has pruned older standard decided data. +- Archive mode has not observed, classified, or persisted matching trace data. +- The node did not observe the relevant P2P messages because of connectivity, peer, or subnet coverage issues. +- The request filters do not match stored data, such as the wrong role, validator public key, validator index, or committee ID. For standard-mode `/v1/exporter/decideds`, an `indices`-only filter is ignored, so the query can still return all participants in the slot range for the selected roles. +- Archive decided entries with no signers can be omitted without an API error. + +### What to do + +1. Retry with a narrow slot range where you expect data. +2. Confirm that the endpoint is supported in the active mode. +3. For standard mode, check whether the requested slots are inside the configured `RetainSlots` window. +4. For archive mode, compare trace data with `schedule` data when available, but do not treat missing schedule data as proof that no duty existed. +5. Confirm that the node has stable P2P connectivity and enough peers for the traffic you expect to observe. +6. Remove optional filters and add them back one at a time to identify mismatches. + +## Traces are missing in archive mode + +Archive mode reconstructs traces from observed network messages. Missing traces do not prove that a duty did not occur. + +### Validator trace caveats + +- Validator trace queries require `roles`. +- When querying committee-duty roles through the validator trace endpoint, such as `ATTESTER` or `SYNC_COMMITTEE`, provide `pubkeys` or `indices`. Without validator filters, use the committee trace endpoint instead. +- The archive decided path can include trace-derived data that differs from the standard-mode decided backend. +- `indices` filters are valid for validator trace queries and archive-mode decided queries. They are ignored by standard-mode `/v1/exporter/decideds`. + +### Committee trace caveats + +- Committee trace queries do not support a `roles` filter. A `roles` field in older examples or client code should not be relied on for committee trace filtering. +- Committee partial-signature classification depends on proposal data and signing roots. If the exporter cannot classify pending committee signatures, those signatures may be dropped when the in-memory slot window is evicted. +- Schedule grouping depends on best-effort schedule computation and committee links. Missing schedule entries or links can make schedule data incomplete even when some trace data exists. + +### What to do + +- Query a short range around the expected slot before widening the range. +- Verify the role and identifier filters. +- For committee-wide data, use `/v1/exporter/traces/committee` rather than validator traces without validator filters. +- Check peer connectivity and whether the exporter has been running long enough to observe the relevant traffic. +- Remember that recent slots may still be in memory, while older slots may have been persisted after the archive mode in-memory window. + +## Responses contain `errors[]` + +Archive trace endpoints can return `200 OK` with partial data and an `errors` array. Treat `errors[]` as part of the result, not as a separate logging detail. + +### How to interpret partial responses + +- `200 OK` with `data` and `errors[]` means the exporter returned what it could and also encountered one or more non-not-found errors. +- Not-found conditions are intentionally suppressed and normally do not appear in `errors[]`. +- If the query returns no data and encounters a meaningful backend error, the endpoint can return `500 Internal Server Error` instead of a partial `200 OK`. +- Standard-mode `/v1/exporter/decideds` uses the standard-mode decided backend and does not populate partial errors in the same way as archive trace queries. + +### What to do + +1. Reduce the slot range. +2. Query one role or one identifier at a time. +3. Check whether the response is still useful despite partial errors. +4. If errors repeat for narrow queries, collect the request fields, endpoint, mode, status code, and response body for investigation. + +## Large queries are slow or fail + +Exporter APIs loop over inclusive slot ranges. No maximum range validation has been identified. Large archive queries can be expensive for CPU, memory, and local database I/O, and may time out or fail under load. + +### What to do + +- Use small slot windows for dashboards, alerting, and manual investigation. +- Split historical analysis into batches instead of one large request. +- Avoid broad unfiltered archive queries during periods of high node load. +- Prefer specific roles, public keys, committee IDs, and, where supported, validator indices. For `/v1/exporter/decideds`, use `indices` only with archive mode; use `pubkeys` for validator filtering in standard mode. + +Large query behavior is especially important in archive mode because archive endpoints can scan detailed trace storage. Standard mode is lighter, but large decided queries can still add load. + +## Archive database is growing + +Archive mode persists validator duty traces, committee duty traces, validator-to-committee links, and scheduled duty data locally. No archive duty-trace pruning path tied to `RetainSlots` has been identified. + +### Standard vs archive retention + +| Mode | Data path | Retention behavior | +| --- | --- | --- | +| Standard | Standard decided data | Uses `RetainSlots` for slot-based pruning. | +| Archive | Duty-trace persistence and schedule data | No archive trace pruning path tied to `RetainSlots` has been identified. | + +### What to do + +- Monitor the exporter Pebble database path derived from `DB_PATH + "-pebble"`. +- Plan capacity for long-running trace collection before enabling archive mode in production. +- Add external retention, cleanup, or storage lifecycle controls if your environment requires bounded archive storage. +- Keep query consumers efficient; heavy API load can add database pressure even if it is not the root cause of storage growth. + +Do not assume that lowering `RetainSlots` will prune archive trace data. That setting applies to the standard-mode decided data pruning path. + +## Known API and schema gotchas + +Account for these behavior details when writing clients or diagnosing unexpected responses: + +- `/v1/exporter/decideds` exists in both modes, but its backend differs. Standard mode reads the standard-mode decided backend; archive mode reads trace-derived data. +- Standard-mode `/v1/exporter/decideds` ignores `indices`. If a standard-mode query is filtered only by `indices`, the index filter is not applied; with no `pubkeys`, the endpoint can return all participants in the slot range for the selected roles. +- Validator trace and decided requests require `roles`. +- Committee trace requests do not support a `roles` field. +- The committee ID field is `committeeIDs` in POST JSON, but the current GET query parameter is `committeeids`. +- Validator public keys must decode to 48 bytes, represented as 96 hex characters. +- Committee IDs must decode to 32 bytes, represented as 64 hex characters. +- `from` and `to` are inclusive, and `from` must be less than or equal to `to`. +- The generated schema may not fully describe mode-dependent runtime behavior, especially for `/v1/exporter/decideds`. +- Schedule data is best-effort and asynchronous. Missing `schedule` entries do not prove that no duty was scheduled. +- Trace data is best-effort observation data. Missing `data` entries do not prove that no duty happened. + +For endpoint fields and mode-specific response behavior, see [API Reference](./api-reference.md). + +## Information to collect before escalating + +When a problem persists after the checks above, collect: + +- exporter mode: `standard` or `archive`; +- the HTTP API bind address and port configuration; +- whether the WebSocket API is enabled; +- endpoint, method, request fields, status code, and response body; +- the slot range, roles, public keys, committee IDs, and any indices used, including whether the indices were sent to an archive-mode decided query or to a validator trace endpoint; +- whether the query is expected to hit standard retention or archive trace storage; +- recent exporter-related startup and runtime logs; +- disk usage for the exporter database path; +- whether the issue reproduces with a narrow slot range and minimal filters.