Automated data ingestion from scientific instruments: documentation

Sources of the documentation site for the activity Automation of Data Acquisition and Metadata Collection in distributed infrastructure. The site describes how research teams, labs and core facilities can automate ingestion of data from large scientific instruments, and documents three solutions:

• DAREG (CEITEC MU): dataset registry with the Lab Client, built on Onedata
• DDRapp (TUL): Airflow-based collection into S3
• Onedata (e-INFRA CZ): the national data management service, used directly

The site is built with Fumadocs using the CERIT-SC framework image (cerit.io/docs/fuma). This repository contains only content and small overrides. The framework image provides the application.

Repository layout

content/docs/            MDX pages and meta.json navigation
  overview/              front page and general guidance (root folder)
  dareg/                 DAREG: overview, architecture, users/, admin/
  ddrapp/                DDRapp: overview, architecture, users/, admin/
  onedata/               Onedata: overview, architecture, users/, admin/, troubleshooting/
public/img/              images (referenced as /img/...)
components/footer.tsx    footer override copied over the framework's component
next.config.mjs, app/[lang]/layout.tsx, app/[lang]/docs/layout.tsx, lib/shared.ts
                         overrides of framework files adding BASE_PATH (path-prefix) support
helm/ingest-docs/        Helm chart for Kubernetes (CERIT-SC)
Dockerfile               optional prebuilt image (chart mode=image)
Makefile                 preview, build and deployment commands

Each top-level folder in content/docs is a fumadocs root folder ("root": true in its meta.json). The sidebar shows a switcher between them (overview, DAREG, DDRapp, Onedata) with icon and description, and each has its own navigation tree.

Writing content

• Pages are MDX with frontmatter title and description. Add each new page to the pages list in its folder's meta.json.
• Available without imports: Callout, Card, Cards, and Mermaid diagrams in fenced code blocks with the language mermaid.
• With imports: Steps/Step from fumadocs-ui/components/steps, Tabs/Tab from fumadocs-ui/components/tabs, and icons from lucide-react.
• Link pages absolutely: /docs/dareg/users/lab-client.
• Do not use bare {, } or < in prose; MDX treats them as code. Put such text in backticks.

Local preview

Requires Docker (or set CONTAINER_BIN=podman).

make dev        # live preview with hot reload: http://localhost:3000/en/docs/overview
make build      # full production build; fails on broken MDX
make serve      # production build + server, as in the cluster

Deployment (CERIT-SC Kubernetes)

The chart in helm/ingest-docs supports two modes:

Mode	How it works	When
git (default)	The framework image clones this repository at a given commit, builds and serves it. No registry needed.	Simple, same model as other CERIT-SC docs
image	Serves an image built from the Dockerfile with the site prebuilt	Fast restarts, no dependency on GitHub at runtime

Production: https://dareg.nrp.eosc.cz/docs

Runs on kubh-cluster, namespace nrp-ka53-docs-ns, with the settings in helm/ingest-docs/values-kubh-nrp.yaml: path prefix /docs, an nginx Ingress, and a letsencrypt-prod certificate. dareg.nrp.eosc.cz is a CNAME of kubh-pub.cerit-sc.cz. Put this in an untracked local.mk:

KUBE_CONTEXT = kubh-cluster
NAMESPACE    = nrp-ka53-docs-ns
VALUES       = helm/ingest-docs/values-kubh-nrp.yaml

then:

git push                 # the cluster builds from git, so push first
make deploy              # helm upgrade --install with content.revision=<current commit>
make logs                # follow the in-pod build
make status

The site is exposed through a Gateway API HTTPRoute on the shared traefik-gateway. This gives automatic DNS and TLS for <name>.dyn.cloud.e-infra.cz. For a custom domain, disable httpRoute, enable ingress (class nginx-traefik, issuer letsencrypt-gateway) and set host. See helm/ingest-docs/values.yaml.

Serving under a path prefix

The CERIT framework has no base-path support, so this repository overrides four framework files (listed above). Each starts with a comment naming the framework version it was copied from. With BASE_PATH unset, they behave exactly like the originals. After upgrading FUMA_IMAGE, diff them against the new framework, then check the result with make serve BASE_PATH=/docs. Under a prefix, the "Ask AI" chat is hidden, because the framework's chat client calls /api/chat at the domain root. It is also hidden whenever no LLM endpoint (LOCAL_URL) is configured.

Prebuilt image instead:

make image push deploy-image IMAGE=cerit.io/<harbor-project>/ingest-docs

Run make help for all targets.

Funding