diff --git "a/fluxon_doc_cn/user_doc/\347\224\250\346\210\267 - 2 - \346\234\215\345\212\241\345\271\263\351\235\242.md" "b/fluxon_doc_cn/user_doc/\347\224\250\346\210\267 - 2 - \346\234\215\345\212\241\345\271\263\351\235\242.md"
index 2ffa6e5..4c570ee 100644
--- "a/fluxon_doc_cn/user_doc/\347\224\250\346\210\267 - 2 - \346\234\215\345\212\241\345\271\263\351\235\242.md"
+++ "b/fluxon_doc_cn/user_doc/\347\224\250\346\210\267 - 2 - \346\234\215\345\212\241\345\271\263\351\235\242.md"
@@ -1,80 +1,114 @@
# 用户 - 2 - 服务平面
-
+
## 服务平面
-如果你要接入 Fluxon 的 KV，除了业务进程本身，还需要先理解一组固定的服务平面对象。它们负责控制面元数据、共享内存池，以及 KV 进程的启动编排。
+本页说明 Fluxon KV 的服务进程如何启动，以及首次运行时需要修改哪些配置。最小 KV 链路只需要 `Greptime`、`etcd`、`Master` 和 `Owner`；TiKV 只用于 FS 目录传输、预扫描等需要持久任务状态的功能。
-从用户视角，最常直接接触的对象有三类：
+### 先按使用场景选择
-- 外部依赖：`etcd`、`greptime`、`TiKV`
-- Fluxon 自有角色：`master`、`owner`
-- 启动入口：`etcd / greptime / TiKV` 原始运行时、`fluxon_py.runtime`、以及你自己的 supervisor / 启动脚本
+- **本机最小 KV**：本机依次启动 `Greptime`、`etcd`、`Master` 和 `Owner`。首次体验优先按这条路径操作。
+- **接入已有 Master**：集群中的 `Greptime`、`etcd` 和 `Master` 已经运行时，本机只启动新的 `Owner`。
+- **启用 FS 目录传输或预扫描**：完成 KV 服务平面后，再额外启动 PD 和 TiKV。只使用 KV、RPC 或 MQ 时可以跳过 TiKV。
-如果你写的是业务进程，这一页回答的问题是：哪些进程必须先起来、它们之间是什么关系、哪些对象可以由 runtime 拉起，哪些不可以。具体 API 见 [用户 - 3 - KV 和 RPC 接口](./用户%20-%203%20-%20KV-RPC接口.md)。
+### 组件关系与启动顺序
-### 角色关系
+推荐的本机启动顺序是：
-服务平面可以先收束成 5 个稳定对象：
+```text
+Greptime → etcd → Master → Owner → 业务进程 new_store(...)
+```
+
+各组件的作用如下：
-- 外部依赖：`etcd`
-- 外部依赖：`greptime`
-- 外部依赖：`TiKV`
-- Fluxon 自有角色：`master`
-- Fluxon 自有角色：`owner`
+- `Greptime` 接收 Master 上报的监控数据。
+- `etcd` 保存成员关系、路由和租约等控制面元数据。
+- `Master` 管理 KV 集群。
+- `Owner` 在当前机器提供共享内存池，并生成业务进程连接时需要的 `shared.json`。
+- PD 和 TiKV 保存 FS 目录传输、预扫描等功能的持久任务状态，不影响最小 KV 读写链路。
-它们的部署位置如下：
+部署位置可参考下图：
-其中：
+`Greptime`、`etcd`、PD 和 TiKV 是外部依赖，需要单独启动。`fluxon_py.runtime` 只启动 Fluxon 自己的 `Master`、`Owner` 等角色。
-- `etcd` 负责控制面元数据存储
-- `greptime` 负责标准监控链路
-- `TiKV` 负责需要持久任务状态的扩展能力；当前最直接的用户可见场景是 FS 目录传输和预扫描使用的 `transfer_state_store`
-- `master` 负责成员关系、路由、租约、监控广播和 master 侧日志
-- `owner` 负责在本机提供共享内存池和 `shared.json`
+### 本机最短启动路径
-### 最小启动顺序
+先按 [用户 - 0 - 安装](./用户%20-%200%20-%20安装.md) 准备运行时包，并确认最小 KV 路径需要的文件已经存在：
-KV 复用同一套底座，最小启动链路如下：
+- `ext_images/greptime/greptime`
+- `ext_images/greptime/start.sh`
+- `ext_images/etcd/etcd`
+- `ext_images/etcd/etcdctl`
+- `ext_images/etcd/start.sh`
-- KV：`greptime -> etcd -> fluxonkv master -> owner -> 业务进程 new_store(...)`
+以下命令均在 Fluxon 仓库根目录执行。每条启动命令都会持续运行，建议分别放在独立终端或 `tmux` 窗口中；确认当前组件启动成功后，再执行下一条。
-如果要启用目录传输、预扫描这类需要持久任务状态的能力，还要额外准备 TiKV 这条扩展链路：
+#### 先理解 `--config` 和 `--workdir`
-- Transfer / Pre-Scan：`TiKV PD -> TiKV -> fs master transfer_state_store`
+外部依赖的启动脚本都接收两个参数：
-这里有一个硬边界：
+- `--config/-c` 决定组件使用哪些启动参数。配置文件必须定义对应的 Bash 数组，例如 `GREPTIME_ARGS` 或 `ETCD_ARGS`。
+- `--workdir/-w` 决定当前实例的数据和日志放在哪里。每个实例应使用独立、可写的工作目录。
-- `etcd`、`greptime`、`TiKV` 是外部依赖
-- `master`、`owner` 是 Fluxon 自己的角色
+配置文件中的 `WORKDIR` 就是 `--workdir` 传入的目录。例如：
-如果控制面没起来，`master` 不可用；如果 `owner` 没起来，业务侧 `FluxonKvClientConfig({...}) -> new_store(...)` 也附着不到共享内存池。TiKV 不影响最小 KV 读写链路，但会影响依赖 `transfer_state_store` 的功能。当前文档仍以 `etcd` 为基础控制面权威对象；后续如果控制面统一迁移到 TiKV，这一页再整体收束。
+```text
+--workdir /tmp/fluxon_service_plane_demo/etcd
+--data-dir "$WORKDIR/etcd-data"
+```
-### 启动入口
+最终的数据目录是 `/tmp/fluxon_service_plane_demo/etcd/etcd-data`。首次本机运行时，只需确认工作目录可写、示例端口没有被占用；单节点的成员配置通常可以保留示例值。
-#### 启动 `etcd`、`greptime` 与 `TiKV`
+#### 1. 启动 Greptime
-先按 [用户 - 0 - 安装](./用户%20-%200%20-%20安装.md) 准备 `etcd / greptime / TiKV` 运行时包，并确认已经解压出：
+**作用**：为 Master 提供监控数据的查询和写入入口。
-- `ext_images/etcd/etcd`
-- `ext_images/etcd/etcdctl`
-- `ext_images/etcd/start.sh`
-- `ext_images/greptime/greptime`
-- `ext_images/greptime/start.sh`
-- `ext_images/tikv/pd-server`
-- `ext_images/tikv/tikv-server`
-- `ext_images/tikv/start_pd.sh`
-- `ext_images/tikv/start_tikv.sh`
+**首次确认项**：
+
+- `--http-addr 0.0.0.0:34030` 使用的 `34030` 端口没有被占用。
+- `--data-home "$WORKDIR/greptimedb"` 对应的工作目录可写。
+
+配置与启动命令：
+
+```bash
+cat > /tmp/greptime.config.sh <<'EOF'
+GREPTIME_ARGS=(
+ standalone start
+ --data-home "$WORKDIR/greptimedb"
+ --http-addr 0.0.0.0:34030
+)
+EOF
-这些启动脚本都使用同一条契约：
+bash ./ext_images/greptime/start.sh \
+ --config /tmp/greptime.config.sh \
+ --workdir /tmp/fluxon_service_plane_demo/greptime
+```
-- `--config/-c`：Shell 配置文件
-- `--workdir/-w`：本地工作目录
+**成功判据**：启动进程保持运行，终端中没有启动错误，并且本机 `34030` 端口可以连接。后面的 Master 示例默认使用这个端口。
-`etcd` 的配置文件需要定义 `ETCD_ARGS` 这个 bash array，例如：
+
+🌐 多机部署时再看｜--http-addr
+
+`0.0.0.0` 表示监听本机所有网卡，不能作为其他机器访问 Greptime 的目标地址。Master 配置中的 `GREPTIME_BASE_URL` 应填写 Master 所在机器能够访问的真实主机名或 IP。
+
+
+
+#### 2. 启动 etcd
+
+**作用**：保存 Master 和 Owner 使用的控制面元数据。
+
+**首次确认项**：
+
+- `--advertise-client-urls "http://127.0.0.1:2379"` 对应后续脚本中的 `ETCD_ENDPOINT = "127.0.0.1:2379"`。
+- `2379` 和 `2380` 端口没有被占用。
+- `--data-dir "$WORKDIR/etcd-data"` 对应的工作目录可写。
+
+`--name`、peer 地址和 `--initial-cluster` 用于 etcd 成员关系。本机单节点运行时可以保留下面的示例值。
+
+配置与启动命令：
```bash
cat > /tmp/etcd.config.sh <<'EOF'
@@ -97,89 +131,68 @@ bash ./ext_images/etcd/start.sh \
--workdir /tmp/fluxon_service_plane_demo/etcd
```
-`greptime` 的配置文件需要定义 `GREPTIME_ARGS` 这个 bash array，例如：
+**成功判据**：访问健康检查接口时返回 `"health":"true"`。
```bash
-cat > /tmp/greptime.config.sh <<'EOF'
-GREPTIME_ARGS=(
- standalone start
- --data-home "$WORKDIR/greptimedb"
- --http-addr 0.0.0.0:34030
-)
-EOF
-
-bash ./ext_images/greptime/start.sh \
- --config /tmp/greptime.config.sh \
- --workdir /tmp/fluxon_service_plane_demo/greptime
+curl -sS http://127.0.0.1:2379/health
```
-TiKV 运行时分成两个进程：PD 和 TiKV。启动脚本分别是 `ext_images/tikv/start_pd.sh` 与 `ext_images/tikv/start_tikv.sh`。它们同样使用 `--config/-c` 和 `--workdir/-w`。
+
+🌐 多机部署时再看｜etcd 的监听地址与成员地址
-PD 配置文件需要定义 `PD_ARGS` 这个 bash array，例如：
+- `--listen-client-urls` 和 `--listen-peer-urls` 决定 etcd 在本机监听哪些网卡。
+- `--advertise-client-urls` 是 Master、Owner 等客户端实际连接的地址。
+- `--initial-advertise-peer-urls` 是其他 etcd 成员连接当前成员的地址。
+- `--name` 必须和 `--initial-cluster` 中的成员名对应，成员地址必须能在机器之间访问。
-```bash
-cat > /tmp/pd.config.sh <<'EOF'
-PD_ARGS=(
- --name pd0
- --data-dir "$WORKDIR/pd-data"
- --client-urls "http://127.0.0.1:12379"
- --advertise-client-urls "http://127.0.0.1:12379"
- --peer-urls "http://127.0.0.1:12380"
- --advertise-peer-urls "http://127.0.0.1:12380"
- --initial-cluster "pd0=http://127.0.0.1:12380"
- --log-file "$WORKDIR/pd.log"
-)
-EOF
+
-bash ./ext_images/tikv/start_pd.sh \
- --config /tmp/pd.config.sh \
- --workdir /tmp/fluxon_service_plane_demo/tikv_pd
-```
+#### 3. 启动 Master 和 Owner
-TiKV 配置文件需要定义 `TIKV_ARGS` 这个 bash array，例如：
+`fluxon_py.runtime` 中最常用的两个入口是：
-```bash
-cat > /tmp/tikv.config.sh <<'EOF'
-TIKV_ARGS=(
- --pd-endpoints "127.0.0.1:12379"
- --addr "127.0.0.1:20160"
- --advertise-addr "127.0.0.1:20160"
- --status-addr "127.0.0.1:20180"
- --data-dir "$WORKDIR/tikv-data"
- --log-file "$WORKDIR/tikv.log"
-)
-EOF
+- `start_kv_master_process(config=...)`
+- `start_owner_kvclient_process(config=...)`
-bash ./ext_images/tikv/start_tikv.sh \
- --config /tmp/tikv.config.sh \
- --workdir /tmp/fluxon_service_plane_demo/tikv
-```
+示例脚本 `examples/start_master_owner.py` 默认同时启动本机 `Master` 和 `Owner`。首次运行只需先理解脚本顶部的这些字段：
-`etcd`、`greptime` 和 `TiKV` 都属于外部依赖。它们不由 `fluxon_py.runtime` 拉起。最小 KV 基础链路要求 `etcd` / `greptime` 在 `master` 启动前 ready；如果要启用目录传输或预扫描，还要求对应的 TiKV PD / TiKV 也先 ready。
+- `ETCD_ENDPOINT`：Master 和 Owner 连接的 etcd 地址。这里填写不带 `http://` 的 `host:port`；按前面的示例启动时保持 `127.0.0.1:2379`。
+- `CLUSTER_NAME`：逻辑集群名。Master、Owner 和后续业务进程必须使用同一个值。
+- `SHARE_MEM_PATH`：本机共享内存目录。同一台机器上的 Owner 和后续业务进程必须使用同一路径。
+- `MASTER_INSTANCE_KEY`：Master 的实例标识，在同一个集群中必须唯一。
+- `OWNER_INSTANCE_KEY`：Owner 的实例标识，在同一个集群中必须唯一。
+- `OWNER_DRAM_BYTES`：Owner 向集群贡献的 DRAM 字节数。默认值是 `1073741824`，即 1 GiB；该值必须大于 0 并满足容量对齐要求。
-#### `fluxon_py.runtime`
+按前面的默认端口启动 Greptime 和 etcd，并且本机 `31000` 端口未被占用时，`GREPTIME_HTTP_PORT`、`MASTER_PORT` 和各目录可以先保留示例值。
-`fluxon_py.runtime` 只负责 Fluxon 自己的角色，不负责替代 `etcd`、`greptime` 或 `TiKV`。
+脚本支持两种启动方式：
-最常用的运行时入口有：
+- **启动本机 Master 和 Owner**：
-- `start_kv_master_process(config=...)`
-- `start_owner_kvclient_process(config=...)`
+ ```bash
+ python3 examples/start_master_owner.py
+ ```
-这些入口还有一个可选参数：
+- **只启动 Owner 并接入已有 Master**：
-- `log_path=...`
+ ```bash
+ python3 examples/start_master_owner.py --without-master
+ ```
-它控制的是 Python wrapper 子进程的 `stdout/stderr` 落点，不是服务自身配置文件里的业务日志目录。
+ 此时 `ETCD_ENDPOINT` 和 `CLUSTER_NAME` 必须与已有集群一致，`OWNER_INSTANCE_KEY` 必须是新值，`SHARE_MEM_PATH` 使用当前机器的本地路径。
-如果你是安装好的 `wheel` 用户，优先直接使用这些 Python 入口，并直接传 Python dict；不要依赖 `examples/` 目录里的脚本路径。
+**成功判据**：脚本保持运行，终端打印 `waiting for Ctrl-C`，并且 `SHARE_MEM_PATH` 下生成 `shared.json`。默认配置对应的检查命令如下：
-对应示例脚本：`examples/start_master_owner.py`
+```bash
+ls -l /dev/shm/fluxon_kv_demo/shared.json
+```
-这个脚本支持两种启动方式：
+Master 和 Owner 的标准输出分别写入 `/tmp/fluxon_kv_demo/runtime/log/master.log` 和 `/tmp/fluxon_kv_demo/runtime/log/owner.log`，可在启动失败时查看。
-- 默认方式：启动 `master + owner`
-- `--without-master`：只启动 `owner`，接入已经存在的 KV 集群 `master`
+下面保留完整脚本，便于从源码仓库直接运行：
+
+
+📄 查看完整脚本（点击展开）｜examples/start_master_owner.py
```python
#!/usr/bin/env python3
@@ -317,19 +330,148 @@ if __name__ == "__main__":
main()
```
-启动命令：
+
+
+从源码仓库运行时可以直接使用这个脚本。通过 `wheel` 安装 Fluxon 后，建议在自己的启动程序中调用上面的两个 `fluxon_py.runtime` 入口，并直接传入 Python dict，不依赖 `examples/` 目录。
+
+示例使用 `wait_subproc_or_ctrlc(...)` 统一等待子进程。按 Ctrl-C 后，已经启动的 Master 和 Owner 会一起停止。
+
+### 可选：为 FS 目录传输启动 PD 和 TiKV
+
+仅使用 KV、RPC 或 MQ 时可以跳过本节。FS 目录传输和预扫描需要 `transfer_state_store` 时，启动顺序为：
+
+```text
+PD → TiKV → FS Master
+```
+
+先确认以下文件已经由运行时包提供：
+
+- `ext_images/tikv/pd-server`
+- `ext_images/tikv/tikv-server`
+- `ext_images/tikv/start_pd.sh`
+- `ext_images/tikv/start_tikv.sh`
+
+PD 和 TiKV 的脚本同样使用 `--config/-c` 与 `--workdir/-w`。
+
+#### 1. 启动 PD
+
+**作用**：管理 TiKV 集群，并向 TiKV 和 FS `transfer_state_store` 提供集群入口。
+
+**首次确认项**：
+
+- `12379` 是客户端端口，`12380` 是 PD 成员通信端口，两者都不能被占用。
+- `--data-dir` 和 `--log-file` 使用的工作目录可写。
+- 配置文件中的数组名必须是 `PD_ARGS`。
+
+本机单节点运行时，成员名、peer 地址和 `--initial-cluster` 可以保留示例值。
+
+配置与启动命令：
+
+```bash
+cat > /tmp/pd.config.sh <<'EOF'
+PD_ARGS=(
+ --name pd0
+ --data-dir "$WORKDIR/pd-data"
+ --client-urls "http://127.0.0.1:12379"
+ --advertise-client-urls "http://127.0.0.1:12379"
+ --peer-urls "http://127.0.0.1:12380"
+ --advertise-peer-urls "http://127.0.0.1:12380"
+ --initial-cluster "pd0=http://127.0.0.1:12380"
+ --log-file "$WORKDIR/pd.log"
+)
+EOF
+
+bash ./ext_images/tikv/start_pd.sh \
+ --config /tmp/pd.config.sh \
+ --workdir /tmp/fluxon_service_plane_demo/tikv_pd
+```
+
+**成功判据**：PD 进程保持运行，并且成员接口返回成功。
```bash
-python3 examples/start_master_owner.py
-python3 examples/start_master_owner.py --without-master
+curl -sS http://127.0.0.1:12379/pd/api/v1/members
```
-默认命令会启动本机 `master + owner`。`--without-master` 只启动本机 `owner`，要求同一个 `cluster_name` 对应的 `master` 已经在别处运行。
+#### 2. 启动 TiKV
+
+**作用**：保存 FS 目录传输和预扫描的持久任务状态。
+
+**首次确认项**：
+
+- PD 已经启动，`--pd-endpoints "127.0.0.1:12379"` 与前面的 PD 客户端地址一致。
+- `20160` 是 TiKV 服务端口，`20180` 是状态与 metrics 端口，两者都不能被占用。
+- `--data-dir` 和 `--log-file` 使用的工作目录可写。
+- 配置文件中的数组名必须是 `TIKV_ARGS`。
+
+配置与启动命令：
+
+```bash
+cat > /tmp/tikv.config.sh <<'EOF'
+TIKV_ARGS=(
+ --pd-endpoints "127.0.0.1:12379"
+ --addr "127.0.0.1:20160"
+ --advertise-addr "127.0.0.1:20160"
+ --status-addr "127.0.0.1:20180"
+ --data-dir "$WORKDIR/tikv-data"
+ --log-file "$WORKDIR/tikv.log"
+)
+EOF
+
+bash ./ext_images/tikv/start_tikv.sh \
+ --config /tmp/tikv.config.sh \
+ --workdir /tmp/fluxon_service_plane_demo/tikv
+```
+
+**成功判据**：TiKV 进程保持运行，日志中没有连接 PD 或初始化存储的错误，并且本机 `20160` 端口可以连接。
+
+
+🌐 多机部署时再看｜PD 与 TiKV 的公开地址
+
+- PD 的 `--advertise-client-urls` 必须能被 TiKV 和 FS Master 访问；`transfer_state_store.pd_endpoints` 使用这个客户端地址。
+- PD 的 `--advertise-peer-urls` 和 `--initial-cluster` 使用 PD 成员之间可访问的地址。
+- TiKV 的 `--pd-endpoints` 必须指向已启动的 PD，`--advertise-addr` 必须能被其他机器访问。
+- `--client-urls`、`--peer-urls`、`--addr` 和 `--status-addr` 控制进程监听位置；不能把仅用于绑定的 `0.0.0.0` 当作远端连接地址。
+
+
+
+### 进阶配置
+
+本机首次运行可以保留以下配置。只有端口、目录或部署机器发生变化时，才需要继续调整。
+
+#### 自定义监控地址和 Master 端口
+
+- `GREPTIME_HTTP_PORT` 是 Greptime 的 HTTP 端口，必须与 Greptime 的 `--http-addr` 一致。
+- `GREPTIME_BASE_URL` 是 Master 能够访问的 Greptime 地址，格式为 `http://host:port`。
+- 示例从 `GREPTIME_BASE_URL` 派生三个接口：Prometheus 查询使用 `/v1/prometheus`，remote write 使用 `/v1/prometheus/write`，OTLP 日志使用 `/v1/otlp/v1/logs`。修改主机或端口后，这三个路径保持不变。
+- `MASTER_PORT` 是本机 Master 的监听端口。只启动 Owner 时不会启动本机 Master；启动本机 Master 时，该端口不能和其他进程冲突。
+
+#### 日志与大文件目录
+
+> **注意**：`log_path` 保存 Python 启动脚本捕获的子进程标准输出和标准错误；`log_dir` 是 Master 自己的业务日志和 profile 目录；`large_file_paths` 是 Owner 保存后台日志、缓存和大文件数据的目录。三个字段含义不同。
+
+示例中的路径关系如下：
+
+- `WORKDIR/log/master.log` 和 `WORKDIR/log/owner.log`：传给 `log_path`，用于排查子进程启动问题。
+- `WORKDIR/master_logs`：传给 Master 配置的 `log_dir`。
+- `WORKDIR/large/owner`：传给 Owner 配置的 `fluxonkv_spec.large_file_paths`；Owner 模式必须提供至少一个可写目录。
+
+每组进程应使用独立、可写的 `WORKDIR`。同一个启动程序管理多个角色时，每个角色还应使用不同的 `log_path`。
+
+#### 多机部署的地址、路径和端口
+
+跨机器部署时，按下面的顺序检查：
-如果用同一个 supervisor / example 脚本拉起多个角色，要求给每个角色分配独立 `log_path`，并在主进程里用 `wait_subproc_or_ctrlc(...)` 统一等待和收束。这样 Ctrl-C 时会按已拉起的子进程组一起停止，不会把服务平面子进程继续留在终端外面。
+1. 所有 Master、Owner 和业务进程使用相同的 `CLUSTER_NAME`。
+2. 每个 Master 和 Owner 使用唯一的 `instance_key`。
+3. 每台机器使用自己的 `WORKDIR`。不同机器的 `SHARE_MEM_PATH` 可以不同，但同一台机器上的 Owner 与业务进程必须一致。
+4. 每台机器上的端口都不能与本机其他进程冲突。
+5. `listen` 或不带 `advertise` 的字段决定进程在本机如何监听；`advertise` 字段决定其他机器实际连接哪个地址。
+6. 所有提供给其他机器的 etcd、Greptime、PD 和 TiKV 地址都必须真实可达，不能使用远端无法访问的 `127.0.0.1` 或仅用于绑定的 `0.0.0.0`。
-完整配置对象和字段语义，见 [用户 - 3 - KV 和 RPC 接口](./用户%20-%203%20-%20KV-RPC接口.md)。
+完整的 KV 配置对象和字段语义，见 [用户 - 3 - KV 和 RPC 接口](./用户%20-%203%20-%20KV-RPC接口.md)。
-### 什么时候看哪一页
+### 后续文档导航
-- 你在写 Python KV API 或节点间 RPC：看 [用户 - 3 - KV 和 RPC 接口](./用户%20-%203%20-%20KV-RPC接口.md)
+- 使用 Python KV API 或进行节点间 RPC 调用时，参见 [用户 - 3 - KV 和 RPC 接口](./用户%20-%203%20-%20KV-RPC接口.md)。
+- 使用 MQ `producer` / `consumer` 收发消息时，参见 [用户 - 4 - MQ 接口](./用户%20-%204%20-%20MQ接口.md)。
+- 启动 `fs_master` / `fs_agent`、注册 export 或挂载远端目录时，参见 [用户 - 5 - FS 接口](./用户%20-%205%20-%20FS接口.md)。
diff --git "a/fluxon_doc_cn/user_doc/\347\224\250\346\210\267 - 3 - KV-RPC\346\216\245\345\217\243.md" "b/fluxon_doc_cn/user_doc/\347\224\250\346\210\267 - 3 - KV-RPC\346\216\245\345\217\243.md"
index 9a8c8e1..59113b6 100644
--- "a/fluxon_doc_cn/user_doc/\347\224\250\346\210\267 - 3 - KV-RPC\346\216\245\345\217\243.md"
+++ "b/fluxon_doc_cn/user_doc/\347\224\250\346\210\267 - 3 - KV-RPC\346\216\245\345\217\243.md"
@@ -2,258 +2,108 @@
## KV 和 RPC 接口
-本页描述 Fluxon 的 Python KV API 和节点间 RPC 调用。两者由同一个 `KvClient` 实例提供，共享生命周期。
+本页介绍 Fluxon 的 Python KV API 和节点间 RPC。两类接口都由同一个 `KvClient` 提供，使用相同的配置、数据模型和关闭流程。
-`cluster_name`、`instance_key`、`etcd` 等前置概念见 [架构和概念](用户%20-%201%20-%20架构和概念.md)。
+开始前应先按 [用户 - 2 - 服务平面](./用户%20-%202%20-%20服务平面.md) 启动 Greptime、etcd、Master 和本机 Owner。本页只关注业务进程如何连接已经运行的 KV 集群。
-Python 业务代码优先直接在代码里写 Python dict，并传给 `FluxonKvClientConfig(...)`；YAML 更适合独立进程启动、supervisor、部署和示例环境。
+业务代码建议直接把 Python dict 传给 `FluxonKvClientConfig(...)`。YAML 文件和角色配置差异放在本页的进阶说明中。
-### 服务平面
+### 开始前检查
-在写 `put_blocking/get_blocking/rpc_call` 这些 Python 业务代码之前，需要先把 KV 依赖的服务平面拉起来。共性的角色关系、启动顺序和 runtime 边界，统一见 [用户 - 2 - 服务平面](./用户%20-%202%20-%20服务平面.md)。
+运行本页示例前，确认以下条件：
-
+- Master 和本机 Owner 已经启动，本机共享内存目录下已经生成 `shared.json`。
+- 当前 Python 环境已经安装 `fluxon-*.whl` 和 `fluxon_pyo3-*.whl`；安装方式见 [用户 - 0 - 安装](./用户%20-%200%20-%20安装.md)。
+- 业务进程的 `cluster_name` 与目标集群一致。
+- 业务进程的 `share_mem_path` 与同一台机器上的 Owner 一致。
+- 每个业务进程使用不同的 `instance_key`。
-直接接触的服务平面对象主要有：
-
-- `greptime`：用于标准监控链路。安装与启动见 [用户 - 2 - 服务平面](./用户%20-%202%20-%20服务平面.md)
-- `etcd`：KV 控制面元数据存储。安装与启动见 [用户 - 2 - 服务平面](./用户%20-%202%20-%20服务平面.md)
-- `start_kv_master_process(...)`：启动 `fluxonkv master`
-- `start_owner_kvclient_process(...)`：启动 `owner`
-
-最小可运行示例脚本如下。这个脚本只启动 Fluxon 自己的角色；`etcd` / `greptime` 仍按服务平面文档单独启动，并且默认假设：
-
-- `etcd` 在 `127.0.0.1:2379`
-- `greptime` HTTP 在 `127.0.0.1:34030`
-- 当前 `python3` 所在环境已经安装 `fluxon-*.whl` 和 `fluxon_pyo3-*.whl`；安装方式见 [用户 - 0 - 安装](./用户%20-%200%20-%20安装.md)
-
-对应示例脚本：`examples/start_master_owner.py`
-
-这个脚本支持两种启动方式：
-
-- 默认方式：启动 `master + owner`
-- `--without-master`：只启动 `owner`，接入已经存在的 KV 集群 `master`
+按用户 2 的默认配置启动时，本页示例可以直接保留：
```python
-#!/usr/bin/env python3
-
-import argparse
-
-from pathlib import Path
+CLUSTER_NAME = "demo-kv-cluster"
+SHARE_MEM_PATH = "/dev/shm/fluxon_kv_demo"
+```
-from fluxon_py.runtime import (
- start_kv_master_process,
- start_owner_kvclient_process,
- wait_subproc_or_ctrlc,
-)
-from fluxon_py.runtime.process_runner import ManagedSubprocess
+本页示例创建的是只使用本机 Owner 内存池的 External Client。它不向集群贡献内存，因此配置中不应加入 Owner 专用的 `contribute_to_cluster_pool_size`、`etcd_addresses`、`sub_cluster` 或 `large_file_paths`。
-ETCD_ENDPOINT = "127.0.0.1:2379"
-GREPTIME_HTTP_PORT = 34030
-GREPTIME_BASE_URL = f"http://127.0.0.1:{GREPTIME_HTTP_PORT}"
-CLUSTER_NAME = "demo-kv-cluster"
-SHARE_MEM_PATH = Path("/dev/shm/fluxon_kv_demo").resolve()
-WORKDIR = Path("/tmp/fluxon_kv_demo/runtime").resolve()
-MASTER_PORT = 31000
-MASTER_UI_PORT = 18080
-MASTER_INSTANCE_KEY = "demo_kv_master"
-OWNER_INSTANCE_KEY = "demo_kv_owner"
-OWNER_DRAM_BYTES = 1073741824
+### 共同的调用流程
+KV 和 RPC 都从同一个 `KvClient` 开始：
-def main() -> None:
- args = parse_args()
- log_dir = (WORKDIR / "log").resolve()
-
- if args.with_master:
- master_log_dir = (WORKDIR / "master_logs").resolve()
- master_log_dir.mkdir(parents=True, exist_ok=True)
- master_stdout_log = log_dir / "master.log"
- master_proc = start_kv_master_process(
- config=build_master_config(log_dir=master_log_dir),
- log_path=master_stdout_log,
- )
- else:
- master_stdout_log = None
- master_proc = None
-
- owner_stdout_log = log_dir / "owner.log"
- owner_proc = start_owner_kvclient_process(
- config=build_owner_config(),
- log_path=owner_stdout_log,
- )
- children = []
- if master_proc is not None:
- children.append(
- ManagedSubprocess(
- label="master",
- proc=master_proc,
- )
- )
- children.append(
- ManagedSubprocess(
- label="owner",
- proc=owner_proc,
- )
- )
+```text
+FluxonKvClientConfig
+ ↓
+new_store(config).unwrap(...) → KvClient
+ ↓
+调用 KV 或 RPC 接口
+ ↓
+store.close().unwrap(...)
+```
- print(f"[fluxon_kv] share_mem_path: {SHARE_MEM_PATH}")
- print(f"[fluxon_kv] etcd endpoint: {ETCD_ENDPOINT}")
- print(f"[fluxon_kv] greptime base url: {GREPTIME_BASE_URL}")
- print(f"[fluxon_kv] start master in this script: {args.with_master}")
- if master_stdout_log is not None:
- print(f"[fluxon_kv] master stdout log: {master_stdout_log}")
- print(
- "[fluxon_kv] kv web ui: "
- f"http://:{MASTER_UI_PORT}/view?cluster_name={CLUSTER_NAME}&member_kind=kv"
- )
- else:
- print("[fluxon_kv] master stdout log: disabled by --without-master")
- print(f"[fluxon_kv] owner stdout log: {owner_stdout_log}")
- stack_label = "master and owner" if args.with_master else "owner"
- print(f"[fluxon_kv] waiting for Ctrl-C to stop {stack_label}")
- wait_subproc_or_ctrlc(
- children,
- on_ctrlc=lambda: print(f"[fluxon_kv] caught Ctrl-C, stopping {stack_label}"),
- )
+- `FluxonKvClientConfig` 保存当前 Python 进程的连接配置。
+- `new_store(...)` 创建 `KvClient`。
+- `KvClient` 同时提供 KV 和 RPC 接口。
+- 大多数公开接口返回 `Result`。示例使用 `unwrap("错误说明")` 取得成功值；失败时会带着这段说明抛出错误。
+- 不再使用客户端时，应调用 `close()` 并消费它返回的 `Result`。
+### FlatDict 数据模型
-def parse_args() -> argparse.Namespace:
- parser = argparse.ArgumentParser(description="Start KV demo owner, optionally with a local master")
- group = parser.add_mutually_exclusive_group()
- group.add_argument(
- "--with-master",
- dest="with_master",
- action="store_true",
- help="Start a local kv master in this script (default)",
- )
- group.add_argument(
- "--without-master",
- dest="with_master",
- action="store_false",
- help="Do not start a local kv master; only start owner and attach to an existing cluster master",
- )
- parser.set_defaults(with_master=True)
- return parser.parse_args()
-
-
-def build_master_config(*, log_dir: Path) -> dict:
- return {
- "instance_key": MASTER_INSTANCE_KEY,
- "cluster_name": CLUSTER_NAME,
- "port": MASTER_PORT,
- "etcd_endpoints": [ETCD_ENDPOINT],
- "log_dir": str(log_dir),
- "monitoring": {
- "prometheus_base_url": f"{GREPTIME_BASE_URL}/v1/prometheus",
- "prom_remote_write_url": [f"{GREPTIME_BASE_URL}/v1/prometheus/write"],
- "otlp_log_api": {
- "otlp_endpoint": f"{GREPTIME_BASE_URL}/v1/otlp/v1/logs",
- },
- },
- "master_ui": {
- "http_listen_addr": f"0.0.0.0:{MASTER_UI_PORT}",
- },
- }
+KV value、RPC 请求和 RPC 响应都使用一层 Python dict，例如：
+```python
+value = {
+ "payload": b"hello",
+ "count": 1,
+ "source": "demo",
+}
+```
-def build_owner_config() -> dict:
- return {
- "instance_key": OWNER_INSTANCE_KEY,
- "contribute_to_cluster_pool_size": {
- "dram": OWNER_DRAM_BYTES,
- "vram": {},
- },
- "fluxonkv_spec": {
- "etcd_addresses": [ETCD_ENDPOINT],
- "cluster_name": CLUSTER_NAME,
- "share_mem_path": str(SHARE_MEM_PATH),
- "sub_cluster": "default",
- "large_file_paths": [str((WORKDIR / "large" / "owner").resolve())],
- },
- }
+字段名必须是字符串，字段值可以是 `int`、`float`、`bool`、`str`、`bytes` 或 DLPack 数据。不要在 value 中继续嵌套 dict 或 list。
+对应类型为：
-if __name__ == "__main__":
- main()
+```python
+FlatDict = Dict[str, Union[int, float, bool, str, bytes, DLPacked]]
```
-启动命令：
+### KV 最小示例
-```bash
-python3 examples/start_master_owner.py
-python3 examples/start_master_owner.py --without-master
-```
+`examples/external_put_get_del.py` 完成一次写入、读取、删除和存在性检查。
-默认命令会启动本机 `master + owner`。`--without-master` 只启动本机 `owner`，要求同一个 `cluster_name` 对应的 `master` 已经在别处运行。
+运行前只需确认三个值：
-上面的 `build_master_config(...)` 里，`master_ui` 是可缺省配置块。配置后，`start_kv_master_process(...)` 会让 KV Web UI 直接作为 `master` 内的 HTTP 服务一起启动：
+- `INSTANCE_KEY`：当前 Python 进程的唯一标识。同一集群内不能与其他进程重复。
+- `CLUSTER_NAME`：必须与 Master 和本机 Owner 一致。
+- `SHARE_MEM_PATH`：必须与本机 Owner 一致。
-```yaml
-master_ui:
- http_listen_addr: 0.0.0.0:18080
-```
+示例中的 `test_spec_config.disable_observability = True` 只用于关闭示例进程的观测后台任务，让最小示例专注于 KV 调用；它不是连接集群所需的核心字段。
-这个 UI 的实际宿主是 `fluxon_cli` 的 KV monitor web。URL 形状固定为：
+运行命令：
-```text
-http://:18080/view?cluster_name=demo-kv-cluster&member_kind=kv
+```bash
+python3 examples/external_put_get_del.py
```
-`owner` 把共享内存池和 `shared.json` 准备好之后，再运行下面的业务最小示例。
-
-### 生命周期与调用流程（Call Flow）
+成功时会看到：
```text
-User-visible processes:
-- etcd (control-plane metadata store)
-- Fluxon cluster node process(es) (the remote peers you address via instance_key/node_id)
-- Your Python process (business code using fluxon_py)
-
-FluxonKvClientConfig (prefer Python dict; YAML also supported)
- |
- v
-new_store(cfg) -> KvClient (one instance in your Python process)
- | |
- | +-- KV: put_blocking/get_blocking/remove/... -> Result[...]
- | | get_blocking() -> Result[MemHolder, ApiError]
- | | -> access() -> Result[FlatDict, ApiError]
- | |
- | +-- Node call(server): rpc_register(path, handler) -> Result[OkNone, ApiError]
- | | (handler lives in your Python process; keep it running)
- | |
- | +-- Node call(client): rpc_call(node_id, path, payload, timeout_ms) -> Result[响应句柄, ApiError]
- | -> wait() -> Result[FlatDict, ApiError]
- |
- v
-api.close() -> Result[OkNone, ApiError]
+OK put key=hello
+world
+OK del key=hello
+OK is_exist after remove -> False
```
-### 核心对象（Python）
-
-- `FluxonKvClientConfig`：配置对象，优先直接从 Python dict 创建，也支持从 YAML 文件加载。
-- `new_store(config: FluxonKvClientConfig) -> Result[KvClient, ApiError]`：创建 KV client 实例。
-- `KvClient`：统一入口，同时提供 KV 读写与节点间调用。
-- `KvClient.third_party_logs_dir() -> Result[str, ApiError]`：返回 Fluxon 分配给第三方 Python 组件的日志根目录。组件应在这个根目录下继续派生自己的子目录，例如 `mq/`。
-- `MemHolder`：`get_blocking(...)` 成功后的读取结果持有者，`access()` 取得 `FlatDict`。
-- `PutOptionalArgs`：`put_blocking(...)` 的可选参数对象，当前常用字段是 `lease_id`。
-- `test_spec_config.disable_observability`：最小 external client 示例里显式设为 `True`，避免把 OTLP / observe 后台任务引入“只验证 KV/RPC 基本链路”的示例生命周期。
-
-注意：
-
-- `MemHolder` 没有 `bytes()`；需要 `access()` 后从 dict 里取 bytes 字段（常用字段名 `payload`）。
-- `store.close()` 会等待当前 client 暴露出去的 `MemHolder` 全部释放；示例里在 `close()` 前显式删掉 `mem` / `flat`，就是为了满足这个关闭约束。
-- `Result` 必须显式消费：调用 `unwrap()` 或 `unwrap_error()`。
-
-### FlatDict（数据模型）
+脚本依次执行：
-KV value 和节点间调用的 payload 统一为 flat dict：
+1. 使用 `new_store(...)` 连接本机 Owner。
+2. 使用 `put_blocking(...)` 写入 `{"payload": b"world"}`。
+3. 使用 `get_blocking(...)` 取得 `MemHolder`，再调用 `access()` 得到 `FlatDict`。
+4. 使用 `remove(...)` 删除 key，并用 `is_exist(...)` 确认 key 已不存在。
+5. 释放 `MemHolder` 相关引用，再关闭 `KvClient`。
-- `FlatDict = Dict[str, Union[int, float, bool, str, bytes, dlpack]]`
-
-### KV 接口最小示例
-
-对应示例脚本：`examples/external_put_get_del.py`
+
+📄 查看完整脚本（点击展开）｜examples/external_put_get_del.py
```python
#!/usr/bin/env python3
@@ -315,88 +165,87 @@ if __name__ == "__main__":
main()
```
-### 常用接口（KV）
-
-上面的 KV 最小示例如果要打开更详细的用户进程日志，直接在启动 Python 进程前设置：
-
-```bash
-FLUXON_LOG=DEBUG python3 examples/external_put_get_del.py
-```
-
-日志相关对象如下：
+
-- `FLUXON_LOG`：控制当前 Python 业务进程 console logger 的输出门限
-- Fluxon Python 侧 logger 会读取 `FLUXON_LOG`；合法值是 `DEBUG`、`INFO`、`WARNING`、`ERROR`、`CRITICAL`，默认 `INFO`
-- `log_dir`：`master` 本地日志 authority
-- `share_mem_path`：KV 共享 bundle 根目录，只承载 `mmap.file`、`shared.json` 和 peer metadata
-- `large_file_paths`：owner 侧大文件根目录，日志、profile、cache 等运行时资产都从这里派生
-- `store.third_party_logs_dir().unwrap(...)`：返回 `{large_file_paths[0]}/{cluster_name}_cluster_third_party_logs`。第三方 Python 组件应只在这个根目录下派生自己的子目录，这样目录使用更收束，Fluxon 观测平面也能统一感知和采集这些文件日志。
+### 常用 KV 接口
-如果服务平面的 `master.monitoring.otlp_log_api` 已经配置，后台服务日志还会继续采集到 Greptime 的 `fluxon_logs` 表。
+#### 写入
-`put_blocking(key: str, value: FlatDict, opts: Optional[PutOptionalArgs] = None) -> Result[OkNone, ApiError]`
+```python
+put_blocking(
+ key: str,
+ value: FlatDict,
+ opts: Optional[PutOptionalArgs] = None,
+) -> Result[OkNone, ApiError]
+```
-- 作用：写入或覆盖一个 KV。
-- `key`：要写入的 KV key。
-- `value`：要写入的 flat dict payload。
-- `opts`：可选写参数；普通写入通常传 `None`，需要额外写控制时再传 `PutOptionalArgs(...)`。
-- 返回链路：调用返回成功后，这次写入已经完成，不需要再额外 `wait()`。
+- `key` 是要写入的 KV key。
+- `value` 是一层 `FlatDict`。
+- 普通写入不需要传 `opts`。
+- 返回成功时写入已经完成，不需要再调用 `wait()`。
-`PutOptionalArgs(lease_id: Optional[int] = None)`
+#### 读取
-- 作用：`put_blocking(...)` 的可选参数对象。
-- `lease_id`：提交写入时，把这个 key 绑定到指定 lease。
-- 常用方式：普通业务写入一般不用传；只有需要 lease 生命周期控制时才显式构造它。
+```python
+get_blocking(key: str) -> Result[MemHolder, ApiError]
+MemHolder.access() -> Result[FlatDict, ApiError]
+```
-`PutOptionalArgs.support_mooncake() -> Tuple[bool, List[str]]`
+`get_blocking(...)` 返回 `MemHolder`，需要再调用 `access()` 才能取得业务 dict。`MemHolder` 本身没有 `bytes()`；bytes 字段应从 `access()` 返回的 `FlatDict` 中读取。
-- 作用：检查当前这组写参数是否兼容 mooncake 写入路径。
-- 返回值：第一个返回值表示是否兼容，第二个返回值列出不兼容字段名。
+`store.close()` 会等待当前客户端交给业务代码的 `MemHolder` 全部释放。读取结束后不要长期持有 `MemHolder`，关闭前应像示例一样释放相关引用。
-`get_blocking(key: str) -> Result[MemHolder, ApiError]`
+#### 删除与检查
-- 作用：读取一个 KV。
-- `key`：要读取的 KV key。
-- 返回链路：接口成功后直接拿到 `MemHolder`，再调用 `access()` 取得 `FlatDict`。
+```python
+remove(key: str) -> Result[OkNone, ApiError]
+is_exist(key: str) -> Result[bool, ApiError]
+```
-`MemHolder`
+`remove(...)` 用于删除 key，`is_exist(...)` 用于检查 key 是否存在。删除后立即调用 `get_blocking(...)` 不保证马上返回 `KeyNotFoundError`，因为读取路径仍可能受到 Owner 和 Master 元数据缓存清理时序的影响；验证删除请求时优先使用 `is_exist(...)`。
-- 作用：`get_blocking(...)` 成功后的读取结果持有者。
-- 理解方式：它不是最终的业务 dict，也不是原始 bytes；还要继续 `access()`。
+### 节点间 RPC 最小示例
-`MemHolder.access() -> Result[FlatDict, ApiError]`
+节点间 RPC 允许一个业务进程注册处理函数，另一个业务进程通过目标 `instance_key` 调用它。两个进程都需要先创建自己的 `KvClient`。
-- 作用：把 `MemHolder` 中的数据展开成 `FlatDict`。
-- 常用用法：`flat = mem.access().unwrap(...)`，然后再从 `flat["payload"]` 之类的字段里取业务值。
-- 注意：`MemHolder` 本身没有 `bytes()`；如果 value 里有 bytes 字段，要先 `access()` 再取。
+运行前确认：
-`get_size(key: str) -> Result[int, ApiError]`
+- `RPC_SERVER_INSTANCE_KEY` 和 `RPC_CLIENT_INSTANCE_KEY` 不同，并且在集群内都唯一。
+- Server 和 Client 的 `CLUSTER_NAME` 与目标集群一致。
+- Server 和 Client 的 `SHARE_MEM_PATH` 分别与各自所在机器的 Owner 一致；本机示例使用相同路径。
+- `--target-instance-key` 等于 Server 实际使用的 `instance_key`。
+- Client 调用的路径与 Server 注册的路径一致；本例都是 `/count`。
-- 作用：只查询 value 大小，不把 payload 整体取回。
-- `key`：要查询的 KV key。
-- 适合场景：先判断对象大小，再决定是否继续 `get(...)`。
+先在终端一启动 Server：
-`is_exist(key: str) -> Result[bool, ApiError]`
+```bash
+python3 examples/rpc_call.py serve
+```
-- 作用：判断某个 KV key 当前是否存在。
-- `key`：要检查的 KV key。
+看到下面的输出后保持 Server 运行：
-`remove(key: str) -> Result[OkNone, ApiError]`
+```text
+[rpc] handler ready instance_key=demo_rpc_server
+[rpc] waiting for Ctrl-C
+```
-- 作用：删除一个 KV。
-- `key`：要删除的 KV key。
+再在终端二启动 Client：
-`is_exist(key: str) -> Result[bool, ApiError]`
+```bash
+python3 examples/rpc_call.py call --target-instance-key demo_rpc_server
+```
-- 作用：查询当前 key 是否还存在。
-- 最小示例里，`remove(...)` 之后优先用它验证“删除请求已经生效”。
-- 注意：`remove(...)` 之后立刻 `get_blocking(...)` 不保证马上返回 `KeyNotFoundError`；删除后的读路径还会受 owner / master 元数据 cache 清理时序影响。如果你要验证“删除传播后不可读”，需要给删除传播留出观察时间。
+首次调用成功时，Client 输出 `1`；Server 同时打印调用方、payload 和累计调用次数。
-### 节点间 RPC 调用最小示例
+调用过程如下：
-这里的 RPC 指节点间 RPC 调用，目标节点通常用目标实例的 `instance_key` 来标识。
+```text
+Server：new_store → rpc_register("/count", handler) → 持续运行
+Client：new_store → rpc_call(...) → wait() → FlatDict 响应 → close
+```
-对应示例脚本：`examples/rpc_call.py`
+
+📄 查看完整脚本（点击展开）｜examples/rpc_call.py
```python
#!/usr/bin/env python3
@@ -495,46 +344,74 @@ if __name__ == "__main__":
main()
```
-### 常用接口（节点间 RPC）
+
-- `rpc_register(path: str, handler: Callable[[from_node_id: str, payload: FlatDict], FlatDict]) -> Result[OkNone, ApiError]`
-- `rpc_call(node_id: str, path: str, payload: FlatDict, timeout_ms: int = 10000) -> Result[响应句柄, ApiError]`
+### 常用 RPC 接口
-使用约束如下：
+#### 注册处理函数
-- `node_id` 通常对应目标节点的 `instance_key`（见：[架构和概念](用户%20-%201%20-%20架构和概念.md)）。
-- `timeout_ms` 默认是 `10000`；如果调用方显式指定，必须满足 `timeout_ms >= 10000`。
+```python
+rpc_register(
+ path: str,
+ handler: Callable[[from_node_id: str, payload: FlatDict], FlatDict],
+) -> Result[OkNone, ApiError]
+```
-### 配置对象与配置文件
+- `path` 是 RPC 路径，必须与 Client 调用时使用的路径一致。
+- `handler` 接收调用方的 `instance_key` 和 `FlatDict` payload，并返回一个 `FlatDict`。
+- 注册成功后，Server 进程必须保持运行。
-KV 环境至少会直接接触两类配置对象：
+#### 发起调用
-- master 配置：启动控制面进程，负责 etcd、成员路由、监控和 master 日志目录
-- client / external 配置：创建 `FluxonKvClientConfig`，供 `new_store(...)` 附着到同机 owner 并发起 KV / RPC
+```python
+rpc_call(
+ node_id: str,
+ path: str,
+ payload: FlatDict,
+ timeout_ms: int = 10000,
+) -> Result[KvFuture, ApiError]
+```
-业务代码直接编辑的通常是第二类配置对象，并且通常直接编辑 Python dict；需要先把 KV 集群拉起来时，两类配置对象都要准备，此时再把配置落成 YAML 给 CLI / runtime 进程使用。
+- `node_id` 通常是目标进程的 `instance_key`。
+- `timeout_ms` 默认是 `10000`；显式指定时不能小于 `10000`。
+- `rpc_call(...).unwrap(...)` 先取得响应句柄。
+- 响应句柄的 `wait().unwrap(...)` 等待远端完成并取得 `FlatDict` 响应。
-#### 1) master 配置
+### 进阶说明
-最小 master 配置示例：
+#### 异步接口与写入选项
-```yaml
-instance_key: my-master-1
-cluster_name: demo-kv-cluster
-port: 31000
-etcd_endpoints:
- - 127.0.0.1:2379
-log_dir: /var/lib/fluxon/master_logs
+`put_blocking(...)` 和 `get_blocking(...)` 会等待操作完成后再返回。需要先提交操作、稍后再等待结果时，可以使用：
+
+```python
+put(...) -> Result[KvFuture, ApiError]
+get(...) -> Result[KvFuture, ApiError]
```
-理解方式：
+两者都先返回 `KvFuture`，再通过 `wait()` 取得最终结果。
+
+其他接口：
+
+- `get_size(key) -> Result[int, ApiError]`：只查询 value 大小，不读取完整 payload。
+- `PutOptionalArgs(lease_id=None, reject_if_inflight_same_key=False)`：控制 lease 绑定，或在同一个 key 已有写入进行时立即拒绝新的写入。
+- `PutOptionalArgs.support_mooncake() -> Tuple[bool, List[str]]`：检查当前写入选项是否兼容 Mooncake，并返回不兼容字段名。
+
+#### 日志与观测
-- `etcd_endpoints`：master 控制面连接的 etcd 地址
-- `log_dir`：master 自己的日志 / profile authority；运行时会在这个目录下继续派生 cluster 级日志子目录
+- `FLUXON_LOG` 控制当前 Python 进程的控制台日志级别，可选值为 `DEBUG`、`INFO`、`WARNING`、`ERROR`、`CRITICAL`，默认是 `INFO`。
+- `store.third_party_logs_dir().unwrap(...)` 返回 Fluxon 为第三方 Python 组件分配的文件日志根目录。组件应继续在该目录下创建自己的子目录，例如 `mq/`。
+- `test_spec_config.disable_observability` 是测试和最小示例使用的开关。普通业务配置不应为了连接集群而依赖这个字段。
+- Master 配置了 `monitoring.otlp_log_api` 后，后台服务日志会继续写入 Greptime 的 `fluxon_logs` 表。
-#### 2) client / external 配置
+需要查看更详细的示例进程日志时，可以运行：
-业务代码里直接把 Python dict 传给 `FluxonKvClientConfig(...)` 的构造方式如下：
+```bash
+FLUXON_LOG=DEBUG python3 examples/external_put_get_del.py
+```
+
+#### Python dict 与 YAML
+
+业务代码优先直接构造 `FluxonKvClientConfig`：
```python
from fluxon_py import FluxonKvClientConfig
@@ -544,68 +421,70 @@ cfg = FluxonKvClientConfig(
"instance_key": "my-kv-client-1",
"fluxonkv_spec": {
"cluster_name": "demo-kv-cluster",
- "share_mem_path": "/dev/shm/fluxon",
+ "share_mem_path": "/dev/shm/fluxon_kv_demo",
},
}
)
```
-如果配置已经落成 YAML 文件，也可以直接从文件构造：
+配置已经保存为 YAML 时，也可以从文件加载：
```python
-from fluxon_py import FluxonKvClientConfig
-
cfg = FluxonKvClientConfig.from_file("./kv_external.yaml")
```
-这两种方式最终得到的是同一个配置对象，后续都传给 `new_store(cfg)`。
-
-最小 external-client 配置示例：
+External Client 的 YAML 与上面的 Python dict 等价：
```yaml
-# 当前 Python 进程 / external client 的唯一实例标识
instance_key: my-kv-client-1
-
fluxonkv_spec:
- # 目标集群名；必须和 master / owner 保持一致
cluster_name: demo-kv-cluster
- # 共享 bundle 根目录；运行时会在其下拼接 cluster_name
- share_mem_path: /dev/shm/fluxon
- # 可选：覆盖当前 client 的 P2P 监听端口
- p2p_listen_port: 31001
+ share_mem_path: /dev/shm/fluxon_kv_demo
```
-Owner 节点需要额外配置内存贡献和 etcd 地址：
+Owner 会向集群贡献内存，因此需要额外提供容量、etcd、子集群和大文件目录：
```yaml
-# owner 实例标识；同样要求全局唯一
instance_key: my-owner-1
-
-# owner 向集群贡献的内存池大小
contribute_to_cluster_pool_size:
- # DRAM 贡献，单位字节
- dram: 1677721600
- # VRAM 贡献；这里为空表示不贡献显存
+ dram: 1073741824
vram: {}
-
fluxonkv_spec:
- # owner 连接 etcd 的地址列表
etcd_addresses:
- 127.0.0.1:2379
- # 目标集群名；必须和 master / external 保持一致
cluster_name: demo-kv-cluster
- # 共享 bundle 根目录；运行时会在其下拼接 cluster_name
- share_mem_path: /dev/shm/fluxon
- # owner 自己的 P2P 监听端口
- p2p_listen_port: 31000
- # owner 所属子集群标签
+ share_mem_path: /dev/shm/fluxon_kv_demo
sub_cluster: default
+ large_file_paths:
+ - /tmp/fluxon_kv_demo/runtime/large/owner
+```
+
+Master 使用独立的启动配置，不传给 `FluxonKvClientConfig`：
+
+```yaml
+instance_key: my-master-1
+cluster_name: demo-kv-cluster
+port: 31000
+etcd_endpoints:
+ - 127.0.0.1:2379
+log_dir: /tmp/fluxon_kv_demo/runtime/master_logs
```
-这里需要把共享 bundle 和大文件根目录分清楚：
+这些角色的启动方式、日志目录和多机约束见 [用户 - 2 - 服务平面](./用户%20-%202%20-%20服务平面.md)。
+
+#### 可选：KV Web UI
-- `share_mem_path`：共享 bundle 根目录；运行时拼接 `cluster_name` 后，同时承载 `mmap.file`、`shared.json` 和 peer metadata
-- `large_file_paths`：owner 独占的大文件 authority，日志、profile、cache 等运行时资产都从这里派生
-- `FLUXON_LOG`：用户 Python 进程 console log 的门限，不写时默认 `INFO`
+需要查看 KV 集群状态时，可以在 Master 配置中加入：
+
+```yaml
+master_ui:
+ http_listen_addr: 0.0.0.0:18080
+```
+
+`master_ui` 依赖 Master 的 `monitoring` 配置。Master 启动后，浏览器访问：
+
+```text
+http://:18080/view?cluster_name=demo-kv-cluster&member_kind=kv
+```
-zero-contribution external 模式下有一个硬约束：`fluxonkv_spec.etcd_addresses`、`fluxonkv_spec.sub_cluster`、`fluxonkv_spec.large_file_paths`、`fluxonkv_spec.redis_compat` 这类 owner 侧字段不应出现。
+`0.0.0.0` 只表示监听本机所有网卡；浏览器地址应使用实际可访问的 Master 主机名或 IP。
diff --git "a/fluxon_doc_cn/user_doc/\347\224\250\346\210\267 - 4 - MQ\346\216\245\345\217\243.md" "b/fluxon_doc_cn/user_doc/\347\224\250\346\210\267 - 4 - MQ\346\216\245\345\217\243.md"
index a788b74..579b7ab 100644
--- "a/fluxon_doc_cn/user_doc/\347\224\250\346\210\267 - 4 - MQ\346\216\245\345\217\243.md"
+++ "b/fluxon_doc_cn/user_doc/\347\224\250\346\210\267 - 4 - MQ\346\216\245\345\217\243.md"
@@ -1,280 +1,131 @@
-# 用户 - 4 - MQ接口
+# 用户 - 4 - MQ 接口
-
+## MQ 接口
-## 总体介绍
+Fluxon MQ 复用 KV 服务平面和本机 Owner 的内存池，在 `KvClient` 上提供 Producer 和 Consumer。Producer 使用 `put_data(...)` 发送消息，Consumer 使用 `get_data(...)` 接收消息。
-Fluxon 提供的是建立在 KV 底座之上的消息队列能力。它不是一套独立于 KV 的新服务，而是复用同一套服务平面、同一套本机共享内存池，以及同一个 Python client 接入内核，在这个基础上提供 producer / consumer 语义。
+最重要的连接规则是：Producer 和 Consumer 使用相同的 `CHANNEL_KEY`、Channel 类型和 Channel 配置，才能进入同一个 Channel。
-如果先按层级架构看，MQ 相关对象可以分成三层：
+### 开始前检查
-- 服务平面：`etcd`、`greptime`、`fluxonkv master`。这一层负责元数据、成员关系、路由、租约以及标准监控链路。
-- 本机数据面常驻实例：`owner`。这一层在本机长期驻留，提供共享内存池和本机数据面资源，角色上更接近一个本机 daemon。
-- 业务进程接入层：`FluxonKvClientConfig`、`new_store(...)` 返回的 `KvClient(store)`、以及继续绑定出来的 `producer` / `consumer`。这一层负责让业务进程以 external client 身份附着到同机 `owner`，再继续使用 MQ API 收发消息。
+运行 MQ 示例前，确认以下条件：
-因此，使用时可以先按下面这条生命周期依赖关系来理解：
+- 已按 [用户 - 2 - 服务平面](./用户%20-%202%20-%20服务平面.md) 启动 Greptime、etcd、Master 和本机 Owner。
+- 本机共享内存目录下已经生成 `shared.json`。
+- 当前 Python 环境已经安装 `fluxon-*.whl` 和 `fluxon_pyo3-*.whl`；安装方式见 [用户 - 0 - 安装](./用户%20-%200%20-%20安装.md)。
+- Producer 和 Consumer 的 `cluster_name` 与目标集群一致。
+- 每个进程的 `share_mem_path` 与同一台机器上的 Owner 一致。
+- 每个 Producer 和 Consumer 进程使用不同的 `instance_key`。
-```text
-etcd + greptime + fluxonkv master
- |
- v
- kvclient owner
- |
- v
-+--------------------------------------------------------------+
-| kvclient external |
-| FluxonKvClientConfig -> new_store(...) -> KvClient(store) |
-+--------------------------------------------------------------+
- |
- +-> new_or_bind_with_unique_key(...)
- |
- +-> producer
- +-> consumer
+按 [用户 - 2 - 服务平面](./用户%20-%202%20-%20服务平面.md) 的默认配置启动时，本页示例可以直接保留：
+
+```python
+CLUSTER_NAME = "demo-kv-cluster"
+SHARE_MEM_PATH = "/dev/shm/fluxon_kv_demo"
```
-`owner`、`external client`、`shared memory` 这些前置概念见 [架构和概念](./用户%20-%201%20-%20架构和概念.md)；`new_store(...) -> KvClient` 的配置和基础语义见 [KV 和 RPC 接口](./用户%20-%203%20-%20KV-RPC接口.md)。
+Producer 和 Consumer 都是 External Client，只使用 Owner 已经提供的容量。它们的配置中不应加入 Owner 专用的 `contribute_to_cluster_pool_size`、`etcd_addresses`、`sub_cluster` 或 `large_file_paths`。
-MQ 用户侧有一个固定角色约束：producer / consumer 必须以 `external_client` 模式运行，也就是 zero-contribution 模式。原因很直接：producer / consumer 会动态加入和离开，这些业务侧进程不应改变集群容量；容量提供者应当始终是常驻的 `owner`。
+`KvClient`、`FlatDict` 和 `Result` 的基础用法见 [用户 - 3 - KV 和 RPC 接口](./用户%20-%203%20-%20KV-RPC接口.md)。
-## 服务平面
+### 对象关系
-在进入 producer / consumer 代码之前，需要先把 MQ 依赖的服务平面拉起来。共性的角色关系、启动顺序和 runtime 边界，统一见 [用户 - 2 - 服务平面](./用户%20-%202%20-%20服务平面.md)。
+MQ 的最小调用关系如下：
-
+```text
+FluxonKvClientConfig
+ ↓
+new_store(...) → KvClient
+ ↓
+使用相同 CHANNEL_KEY 绑定 Channel
+ ├─ Producer → put_data(...)
+ └─ Consumer → get_data(...)
+```
-MQ 直接复用 KV 服务平面，没有自己独立的一套底座。直接接触的服务平面对象主要有：
+`new_or_bind_with_unique_key(...)` 会根据 `CHANNEL_KEY` 查找 Channel：
-- `greptime`：用于标准监控链路。安装与启动见 [用户 - 2 - 服务平面](./用户%20-%202%20-%20服务平面.md)
-- `etcd`：MQ / KV 服务平面元数据存储。安装与启动见 [用户 - 2 - 服务平面](./用户%20-%202%20-%20服务平面.md)
-- `start_kv_master_process(...)`：启动 `fluxonkv master`
-- `start_owner_kvclient_process(...)`：启动 `owner`
+- 第一个使用该 key 的进程创建 Channel。
+- 后续进程使用相同 key 绑定已有 Channel。
+- 不同逻辑 Channel 应使用不同的 key。
-MQ 用户进程的最小前置链路如下：
+每个进程都先创建自己的 `KvClient`，再在其上创建 Producer 或 Consumer handle。退出时顺序固定为：
-1. 先起 `greptime`
-2. 再起 `etcd`
-3. 再起 `fluxonkv master`
-4. 再起 `owner`
-5. 最后再运行 producer / consumer
+```text
+先关闭 Producer / Consumer handle → 再关闭 KvClient
+```
-最小可运行示例脚本如下。MQ 复用的就是这条 KV 角色启动链路；这个脚本只启动 Fluxon 自己的角色，`etcd` / `greptime` 仍按服务平面文档单独启动：
+### MQ 最小示例
-- 当前 `python3` 所在环境已经安装 `fluxon-*.whl` 和 `fluxon_pyo3-*.whl`；安装方式见 [用户 - 0 - 安装](./用户%20-%200%20-%20安装.md)
+`examples/start_mpmc_demo.py` 使用 `ChanType.MPMC`，允许多个 Producer 和多个 Consumer。首次运行只启动一个 Consumer 和一个 Producer。
-对应示例脚本：`examples/start_master_owner.py`
+#### 首次只需确认这些值
-这个脚本支持两种启动方式：
+- `--role`：当前进程运行 Producer 还是 Consumer。
+- `CLUSTER_NAME`：必须与 Master 和本机 Owner 一致。
+- `SHARE_MEM_PATH`：必须与本机 Owner 一致。
+- `CHANNEL_KEY`：两个进程必须完全一致；示例使用 `demo_mq_channel_doc`。
+- `instance_key`：脚本根据角色生成 `demo_mq_producer` 或 `demo_mq_consumer`，因此两个进程的标识不同。
-- 默认方式：启动 `master + owner`
-- `--without-master`：只启动 `owner`，接入已经存在的 KV 集群 `master`
+示例中的其他值可以先保留：
-```python
-#!/usr/bin/env python3
+- `CHANNEL_CAPACITY = 128`
+- `CHANNEL_TTL_SECONDS = 300`
+- `PRODUCER_INTERVAL_SECONDS = 1.0`
+- `CONSUMER_BATCH_SIZE = 1`
-import argparse
-
-from pathlib import Path
+这些配置的含义和约束见后面的进阶说明。如果需要同时启动两个相同角色的进程，必须为每个进程分配不同的 `instance_key`，不能继续直接复用示例根据角色生成的固定值。
-from fluxon_py.runtime import (
- start_kv_master_process,
- start_owner_kvclient_process,
- wait_subproc_or_ctrlc,
-)
-from fluxon_py.runtime.process_runner import ManagedSubprocess
+#### 消息内容
-ETCD_ENDPOINT = "127.0.0.1:2379"
-GREPTIME_HTTP_PORT = 34030
-GREPTIME_BASE_URL = f"http://127.0.0.1:{GREPTIME_HTTP_PORT}"
-CLUSTER_NAME = "demo-kv-cluster"
-SHARE_MEM_PATH = Path("/dev/shm/fluxon_kv_demo").resolve()
-WORKDIR = Path("/tmp/fluxon_kv_demo/runtime").resolve()
-MASTER_PORT = 31000
-MASTER_INSTANCE_KEY = "demo_kv_master"
-OWNER_INSTANCE_KEY = "demo_kv_owner"
-OWNER_DRAM_BYTES = 1073741824
+Producer 发送的每条消息都是 `FlatDict`：
+```python
+{
+ "seq": 1,
+ "payload": b"hello mq #1",
+}
+```
-def main() -> None:
- args = parse_args()
- log_dir = (WORKDIR / "log").resolve()
-
- if args.with_master:
- master_log_dir = (WORKDIR / "master_logs").resolve()
- master_log_dir.mkdir(parents=True, exist_ok=True)
- master_stdout_log = log_dir / "master.log"
- master_proc = start_kv_master_process(
- config=build_master_config(log_dir=master_log_dir),
- log_path=master_stdout_log,
- )
- else:
- master_stdout_log = None
- master_proc = None
+- `seq` 是示例自定义的进程内消息序号。
+- `payload` 是示例自定义的数据字段。
+- `b"..."` 表示 Python bytes。
- owner_stdout_log = log_dir / "owner.log"
- owner_proc = start_owner_kvclient_process(
- config=build_owner_config(),
- log_path=owner_stdout_log,
- )
- children = []
- if master_proc is not None:
- children.append(
- ManagedSubprocess(
- label="master",
- proc=master_proc,
- )
- )
- children.append(
- ManagedSubprocess(
- label="owner",
- proc=owner_proc,
- )
- )
+这些字段名不是 MQ 的固定字段，可以替换为业务需要的其他 `FlatDict` 字段。
- print(f"[fluxon_kv] share_mem_path: {SHARE_MEM_PATH}")
- print(f"[fluxon_kv] etcd endpoint: {ETCD_ENDPOINT}")
- print(f"[fluxon_kv] greptime base url: {GREPTIME_BASE_URL}")
- print(f"[fluxon_kv] start master in this script: {args.with_master}")
- if master_stdout_log is not None:
- print(f"[fluxon_kv] master stdout log: {master_stdout_log}")
- else:
- print("[fluxon_kv] master stdout log: disabled by --without-master")
- print(f"[fluxon_kv] owner stdout log: {owner_stdout_log}")
- stack_label = "master and owner" if args.with_master else "owner"
- print(f"[fluxon_kv] waiting for Ctrl-C to stop {stack_label}")
- wait_subproc_or_ctrlc(
- children,
- on_ctrlc=lambda: print(f"[fluxon_kv] caught Ctrl-C, stopping {stack_label}"),
- )
+#### 启动 Consumer 和 Producer
+先在终端一启动 Consumer：
-def parse_args() -> argparse.Namespace:
- parser = argparse.ArgumentParser(description="Start KV demo owner, optionally with a local master")
- group = parser.add_mutually_exclusive_group()
- group.add_argument(
- "--with-master",
- dest="with_master",
- action="store_true",
- help="Start a local kv master in this script (default)",
- )
- group.add_argument(
- "--without-master",
- dest="with_master",
- action="store_false",
- help="Do not start a local kv master; only start owner and attach to an existing cluster master",
- )
- parser.set_defaults(with_master=True)
- return parser.parse_args()
-
-
-def build_master_config(*, log_dir: Path) -> dict:
- return {
- "instance_key": MASTER_INSTANCE_KEY,
- "cluster_name": CLUSTER_NAME,
- "port": MASTER_PORT,
- "etcd_endpoints": [ETCD_ENDPOINT],
- "log_dir": str(log_dir),
- "monitoring": {
- "prometheus_base_url": f"{GREPTIME_BASE_URL}/v1/prometheus",
- "prom_remote_write_url": [f"{GREPTIME_BASE_URL}/v1/prometheus/write"],
- "otlp_log_api": {
- "otlp_endpoint": f"{GREPTIME_BASE_URL}/v1/otlp/v1/logs",
- },
- },
- }
-
-
-def build_owner_config() -> dict:
- return {
- "instance_key": OWNER_INSTANCE_KEY,
- "contribute_to_cluster_pool_size": {
- "dram": OWNER_DRAM_BYTES,
- "vram": {},
- },
- "fluxonkv_spec": {
- "etcd_addresses": [ETCD_ENDPOINT],
- "cluster_name": CLUSTER_NAME,
- "share_mem_path": str(SHARE_MEM_PATH),
- "sub_cluster": "default",
- "large_file_paths": [str((WORKDIR / "large" / "owner").resolve())],
- },
- }
+```bash
+python3 examples/start_mpmc_demo.py --role consumer
+```
+Consumer 绑定成功后会打印包含下面内容的日志：
-if __name__ == "__main__":
- main()
+```text
+[consumer] ready: channel_key=demo_mq_channel_doc
```
-启动命令：
+再在终端二启动 Producer：
```bash
-python3 examples/start_master_owner.py
-python3 examples/start_master_owner.py --without-master
+python3 examples/start_mpmc_demo.py --role producer
```
-默认命令会启动本机 `master + owner`。`--without-master` 只启动本机 `owner`，要求同一个 `cluster_name` 对应的 `master` 已经在别处运行。
-
-`owner` 把共享内存池和 `shared.json` 准备好之后，再运行下面的 MQ 最小示例。默认模式下，这个服务平面脚本会把子进程终端输出写到 `WORKDIR/log/master.log` 和 `WORKDIR/log/owner.log`，终端只保留摘要信息；`--without-master` 模式下只会生成 `WORKDIR/log/owner.log`。
-
-`cluster`、`owner_client`、`external_client`、`shared memory` 这些前置概念见 [架构和概念](./用户%20-%201%20-%20架构和概念.md)；`new_store(...) -> KvClient` 的配置和基础语义见 [KV 和 RPC 接口](./用户%20-%203%20-%20KV-RPC接口.md)。
-
-## 对象关系
-
-`new_store(...)` 和 MQ handle 的关系如下：
+成功运行后，两端会持续出现类似日志：
```text
-FluxonKvClientConfig
- |
- v
-new_store(cfg) -> KvClient (store)
- |
- +-- new_or_bind_with_unique_key(
- | store,
- | chan_config,
- | unique_id,
- | chan_type,
- | chan_role,
- | )
- |
- +-- producer handle
- | put_data(...)
- | close()
- |
- +-- consumer handle
- | get_data(...)
- | close()
- |
- v
-store.close()
+[producer] ready: channel_key=demo_mq_channel_doc
+[producer] sent: seq=1 payload=hello mq #1
+[consumer] got: seq=1 payload=hello mq #1
```
-关键关系如下：
-
-- `new_store(...)` 构造的是 `KvClient`；本页里的 `store` 只是示例变量名
-- `KvClient` 的基础语义、配置字段、`external_client` 约束都在 [KV 和 RPC 接口](./用户%20-%203%20-%20KV-RPC接口.md) 里定义
-- `new_or_bind_with_unique_key(...)` 不是独立入口，它必须运行在 `store` 之上
-- 退出顺序固定是：先 `producer.close()` / `consumer.close()`，再 `store.close()`
-
-## MQ接口最小示例
+看到 Producer 持续发送、Consumer 持续收到相同序号和 payload，即表示最小链路已经跑通。分别在两个终端按 Ctrl-C 可以停止进程。
-跑通一对 producer / consumer 的最小对象如下：
+`seq` 只保存在 Producer 当前进程中。重启 Producer 后，它会从 `1` 重新开始，不会跨进程持久化。
-- `examples/start_mpmc_demo.py`
-
-它只保留一个命令参数：
-
-- `--role producer`
-- `--role consumer`
-
-先把服务平面拉起来，然后分别运行：
-
-```bash
-python3 examples/start_mpmc_demo.py --role producer
-python3 examples/start_mpmc_demo.py --role consumer
-```
-
-这个最小示例里，`producer` 每成功发送一条消息，就把本进程内的 `seq` 加 `1`；如果你重启 `producer` 进程，计数会从 `1` 重新开始，而不是跨进程持久延续。
-
-对应的真实脚本内容如下：
+
+📄 查看完整脚本（点击展开）｜examples/start_mpmc_demo.py
```python
#!/usr/bin/env python3
@@ -475,7 +326,6 @@ def main() -> None:
parser.add_argument("--role", choices=["producer", "consumer"], required=True)
args = parser.parse_args()
- # The minimal example keeps share_mem_path explicit and local.
# init_logger() reads FLUXON_LOG and sets the user-process console log level.
logger = init_logger(f"mpmc_demo_{args.role}")
shutdown_requested = threading.Event()
@@ -499,70 +349,128 @@ if __name__ == "__main__":
main()
```
-这个最小示例对应的前置条件只有一条：
+
-1. 先把 `greptime + etcd + master + owner` 这套服务平面拉起来
+完整脚本还处理了 Ctrl-C 和关闭期间的并发情况。首次使用只需记住：先关闭 Producer 或 Consumer handle，再关闭底层 `KvClient`。
-代码块里的注释已经把这份最小示例最容易混淆的局部因果直接写在对应位置，包括：
+### 常用接口
-- 每个全局常量控制什么
-- `new_store(...)` 和 MQ handle 的先后关系
-- `Ctrl-C` 回调为什么只做“设置退出标志位 + 调一次 close()”
-- `ProducerClosedError` / `ChannelClosedError` 为什么是正常关闭路径
-- 为什么一定是先关 handle，再关 `store`
+#### 创建或绑定 Channel
-常见启动方式：
+```python
+new_or_bind_with_unique_key(
+ api: KvClient,
+ chan_config: Dict[str, int],
+ unique_id: str,
+ chan_type: ChanType,
+ chan_role: ChanRole,
+) -> Result[
+ Union[MPSCChanProducer, MPSCChanConsumer, MPMCChanProducer, MPMCChanConsumer],
+ ApiError,
+]
+```
-```bash
-FLUXON_LOG=INFO python3 examples/start_mpmc_demo.py --role producer
-FLUXON_LOG=DEBUG python3 examples/start_mpmc_demo.py --role consumer
+- `api` 是 `new_store(...)` 创建的 `KvClient`。
+- `chan_config` 是 Channel 配置。
+- `unique_id` 是稳定的 Channel key。相同 key 指向同一个逻辑 Channel。
+- `chan_type` 选择 `ChanType.MPMC` 或 `ChanType.MPSC`。
+- `chan_role` 选择 `ChanRole.PRODUCER` 或 `ChanRole.CONSUMER`。
+
+创建或绑定失败时，应先打印 `unwrap_error()`，再检查 `cluster_name`、`share_mem_path`、`unique_id`、Channel 类型和 Channel 配置是否一致。Producer 和 Consumer 的 `chan_role` 本来就不同，不需要设置成相同角色。
+
+#### 发送消息
+
+```python
+producer.put_data(value: FlatDict) -> Result[bool, ApiError]
+```
+
+`value` 是一层 `FlatDict`。返回成功表示消息已经交给 Channel；返回错误时应消费 `unwrap_error()`。
+
+#### 接收消息
+
+```python
+consumer.get_data(
+ batch_size: int = 1,
+ try_time: Optional[int] = None,
+ prefetch_num: int = 0,
+) -> Result[List[Any], ApiError]
```
-## 常用接口
+- `batch_size`：本次最多请求的消息数，必须是正整数。
+- `try_time`：阻塞路径的最长等待秒数；设为 `0` 时用于非阻塞尝试。
+- `prefetch_num`：额外预取窗口大小，入门示例保留 `0`。
+
+成功结果是消息列表；本页 MPMC 示例中的每条消息都是 `FlatDict`。没有消息时可能返回空列表，业务循环应继续等待下一次读取。
+
+#### 查看身份和关闭
+
+- `producer.get_chan_id()` / `consumer.get_chan_id()`：返回当前 handle 绑定的 Channel ID。
+- `producer.get_producer_id()`：返回 Producer member ID。
+- `consumer.get_consumer_id()`：返回 Consumer member ID。
+- `close() -> Result[OkNone, ApiError]`：关闭当前 MQ handle。
+
+### 进阶说明
+
+#### Channel 类型和配置
-- `new_or_bind_with_unique_key(api, chan_config, unique_id, chan_type, chan_role)`：存在则 bind，不存在则创建
-- `producer.put_data(value: FlatDict) -> Result[bool, ApiError]`：写一条消息
-- `consumer.get_data(batch_size: int = 1, try_time: Optional[int] = None, prefetch_num: int = 0) -> Result[List[Any], ApiError]`：按批拉消息
-- `producer.get_chan_id()` / `consumer.get_chan_id()`：查看当前 handle 绑定的 `chan_id`
-- `producer.get_producer_id()` / `consumer.get_consumer_id()`：查看当前 member id
-- `close() -> Result[OkNone, ApiError]`：关闭当前 MQ handle
+- `ChanType.MPMC` 支持多个 Producer 和多个 Consumer；`capacity` 与 `ttl_seconds` 都是必填项。
+- `ChanType.MPSC` 支持多个 Producer 和一个 Consumer；`ttl_seconds` 必填，`capacity` 可选。
+- `capacity` 必须是正整数，用于限制在途消息数量。达到限制后，Producer 会等待 Channel 出现可用空间。
+- `ttl_seconds` 必须是整数且不小于 `90`，用于 Channel 和成员租约。
+- 绑定同一个 `CHANNEL_KEY` 的进程应使用相同的 Channel 类型、`capacity` 和 `ttl_seconds`。
-参数约束如下：
+示例中的两个本地参数不属于 Channel 配置：
-- `chan_type` 当前最常用的是 `ChanType.MPMC`，也支持 `ChanType.MPSC`
-- `chan_role` 只能是 `ChanRole.PRODUCER` 或 `ChanRole.CONSUMER`
-- `try_time` 是秒级等待上限；如果需要阻塞拉取窗口控制，再看 `prefetch_num`
+- `PRODUCER_INTERVAL_SECONDS` 只控制 Producer 两次发送之间的等待时间，必须为非负值。
+- `CONSUMER_BATCH_SIZE` 控制每次 `get_data(...)` 请求的消息数，必须是正整数。
-### 关键接口常见错误处理
+#### 生命周期和错误处理
-- `new_or_bind_with_unique_key(...)` 失败：直接把 `unwrap_error()` 打出来，先检查 `cluster_name`、`share_mem_path`、`unique_id`、`chan_role` 是否和对端一致
-- `producer.put_data(...)` 返回 `ProducerClosedError`：按正常关闭路径处理，直接退出主循环
-- `consumer.get_data(...)` 返回 `ChannelClosedError`：按正常关闭路径处理，直接退出主循环
+Ctrl-C 关闭期间，发送或接收循环可能观察到：
-### 日志路径
+- `ProducerClosedError`：Producer handle 已关闭，正常退出发送循环。
+- `ChannelClosedError`：Consumer handle 已关闭，正常退出接收循环。
-日志路径如下：
+这两种错误在主动关闭路径中不代表新的数据面故障。其他错误应打印 `unwrap_error()` 并停止当前循环。
-- MQ Python 部分：由 `init_logger(...)` 初始化，直接输出到当前终端，不默认落盘，门限由 `FLUXON_LOG` 控制
-- MQ Rust / KV 后台部分：和 KV 一起走服务平面的后台日志链路；`master` 本地日志目录由 `master_cfg["log_dir"]` 指定
-- `share_mem_path`：KV 共享 bundle 根目录，只承载 `mmap.file`、`shared.json` 和 peer metadata；后端日志、profile、cache 从 owner 的 `large_file_paths` 派生
+关闭顺序始终是：
-如果服务平面的 `master.monitoring.otlp_log_api` 已经配置，MQ Rust / KV 后台部分的日志还会继续采集到 Greptime 的 `fluxon_logs` 表。
+1. 请求业务循环停止。
+2. 关闭 Producer 或 Consumer handle。
+3. 等待循环退出。
+4. 最后关闭 `KvClient`。
-## 网页监控
+#### 日志
-网页监控页面里直接可用的对象有两个：
+- MQ Python 日志由 `init_logger(...)` 初始化，默认输出到当前终端。
+- `FLUXON_LOG` 控制日志级别，可选值为 `DEBUG`、`INFO`、`WARNING`、`ERROR`、`CRITICAL`，默认是 `INFO`。
+- MQ Rust 和 KV 后台日志使用 [用户 - 2 - 服务平面](./用户%20-%202%20-%20服务平面.md) 中说明的服务平面日志目录。
+- Master 配置了 `monitoring.otlp_log_api` 后，后台日志还会写入 Greptime 的 `fluxon_logs` 表。
+
+需要增加日志时，可以运行：
+
+```bash
+FLUXON_LOG=INFO python3 examples/start_mpmc_demo.py --role producer
+FLUXON_LOG=DEBUG python3 examples/start_mpmc_demo.py --role consumer
+```
-- `Channels` 表：看 channel 级汇总
-- `Members` 表：看单个 producer / consumer 明细
+### 网页监控
-### 查看 Channel 汇总
+先按 [用户 - 3 - KV 和 RPC 接口](./用户%20-%203%20-%20KV-RPC接口.md) 中的“可选：KV Web UI”启用 Master Web UI。进入对应集群页面后，MQ 主要查看：
-`Channels` 表适合先看每个 channel 的整体状态，尤其是积压量和各个 producer 的写入进度。
+- `Channels`：Channel 汇总。
+- `Members`：单个 Producer 和 Consumer 明细。
-重点字段：
+#### 1. 先看是否存在积压
-`producer_offsets` 是 channel 下每个 producer 的 offset 明细，格式是：
+在 `Channels` 表中，先按 `current_inflight` 降序排列：
+
+- 接近 `0`：当前 Channel 基本消费完毕。
+- 持续升高：消息产生速度高于消费速度，需要继续查看具体 Producer。
+
+#### 2. 再看具体 Producer
+
+`producer_offsets` 的格式为：
```text
producer_idx: produce_offset/consume_offset
@@ -574,70 +482,43 @@ producer_idx: produce_offset/consume_offset
producer_1: 101/88, producer_2: 57/57
```
-这里的两个 offset 都表示“下一条 offset”：
+两个 offset 都表示下一条位置：
-- `produce_offset`：这个 producer 下一条将要写入的 offset
-- `consume_offset`：consumer 下一条将要提交的 offset
+- `produce_offset`：Producer 下一条要写入的位置。
+- `consume_offset`：Consumer 下一条要提交的位置。
-所以单个 producer 的当前未消费量就是：
+单个 Producer 的未消费量为：
```text
max(produce_offset - consume_offset, 0)
```
-`current_inflight` 就是这个 channel 下所有 producer 上面这项的求和。
-
-常见查看方式：
-
-- `current_inflight` 持续升高，同时某一个 producer 在 `producer_offsets` 里差值越来越大
- - 说明这个 producer 的消息在持续堆积
-- `current_inflight` 接近 `0`
- - 说明当前 channel 基本被消费干净
-- `producer_offsets` 里多个 producer 都有明显差值
- - 说明积压不是单个 producer 偏斜，而是整个 channel 消费跟不上
-
-### 查看 Producer / Consumer 明细
-
-`Members` 表适合继续下钻到单个 producer / consumer。
-
-重点字段：
-
-- `channel_unique_keys`：显示这个 member 所属 channel 的 `unique_id` 绑定 key
-- `produce_offset` / `consume_offset`：查看单个成员当前的写入位置和消费提交位置
-- `chan_id`、`owner_id`、`external_client_id`：继续定位这个成员属于哪个 channel、owner 和 external client
-
-从 Python 侧用 `new_or_bind_with_unique_key(...)` 接入 MQ 时，定位方式如下：
-
-- 先在 `Members` 表搜 `channel_unique_keys`
-- 找到对应 row 之后，再看它的 `chan_id`、`owner_id`、`external_client_id`、offset 和消费延迟字段
+`current_inflight` 是同一个 Channel 中所有 Producer 未消费量的总和。某一个 Producer 的差值持续增大时，说明积压主要来自该 Producer；多个 Producer 都有明显差值时，说明整个 Channel 的消费速度不足。
-这里要注意一个边界：
+#### 3. 最后定位成员
-- 页面里显示的 `channel_unique_keys` 仍然是 channel 级 key
-- 当前监控页面没有单独暴露“某个 producer / consumer 实例自己的 unique member id”，因为现有 snapshot 权威数据里没有这个对象
+在 `Members` 表中：
-### 筛选与排序
+1. 使用 `channel_unique_keys` 搜索 Python 侧传入的 `CHANNEL_KEY`。
+2. 查看对应行的 `chan_id`、`owner_id` 和 `external_client_id`。
+3. 继续检查 offset 和消费延迟字段。
-网页监控里的 `Channels` / `Members` 两张表都支持和 KV 页面同一套字段排序：
+`channel_unique_keys` 是 Channel 级 key，不是单个 Producer 或 Consumer 的 member ID。需要查看 handle ID 时，使用 Python 接口的 `get_producer_id()` 或 `get_consumer_id()`。
-- 可以在表头上方的 `Sort #1` 到 `Sort #4` 里选字段排序
-- `producer_offsets` 和 `current_inflight` 都支持这套排序
-- `channel_unique_keys` 也支持过滤和排序，适合直接按 Python 侧传入的 `unique_id` 查 channel
-- `producer_offsets` 适合做文本过滤，快速定位某个 producer
-- `current_inflight` 适合做排序，快速找到当前积压最大的 channel
+`Channels` 和 `Members` 都支持字段筛选和多级排序。`current_inflight` 适合寻找积压最大的 Channel，`channel_unique_keys` 适合定位业务代码使用的 Channel。
-## 延迟排查
+### 延迟排查
-MQ 每 30s 在日志中打印一次消费延迟统计。如果遇到消费慢的问题，按以下关键词搜索日志：
+MQ 大约每 30 秒打印一次消费延迟统计。基础链路已经运行但消费较慢时，再搜索以下关键词：
-| 日志关键词 | 观测层 | 说明 |
+| 日志关键词 | 观测位置 | 含义 |
|---|---|---|
-| `py-get latency` | Python 调用侧 | 用户调用 `get_data()` 的总耗时 |
-| `get_one breakdown` | PyO3 层 | 跨语言桥接等待时间拆分 |
-| `MpscConsumer prefetch` | Rust MQ 层 | 预取队列和单条任务耗时 |
+| `py-get latency` | Python 调用侧 | `get_data()` 的总耗时 |
+| `get_one breakdown` | Python/Rust 边界 | 跨语言等待时间拆分 |
+| `MpscConsumer prefetch` | Rust MQ | 预取队列和单条任务耗时 |
-快速定位：
+快速判断：
-- `py-get` 总耗时高 → 先看 PyO3 层的 `avg_wait_rx_ms` 是否大
-- Rust 层 `avg_get_handle_ms` 高 → 预取队列为空，可能是生产侧无数据或窗口过小
-- Rust 层 `avg_handle_await_ms` 高 → 单条任务本身慢，例如 `kv_get` 或 etcd 提交慢
+- `py-get` 总耗时高时，先看 `avg_wait_rx_ms` 是否较高。
+- `avg_get_handle_ms` 高时，预取队列可能为空，常见原因是 Producer 暂时没有数据或预取窗口过小。
+- `avg_handle_await_ms` 高时，单条任务本身较慢，例如 KV 读取或 etcd 提交较慢。
diff --git "a/fluxon_doc_cn/user_doc/\347\224\250\346\210\267 - 5 - FS\346\216\245\345\217\243.md" "b/fluxon_doc_cn/user_doc/\347\224\250\346\210\267 - 5 - FS\346\216\245\345\217\243.md"
index cf9ab79..128d53a 100644
--- "a/fluxon_doc_cn/user_doc/\347\224\250\346\210\267 - 5 - FS\346\216\245\345\217\243.md"
+++ "b/fluxon_doc_cn/user_doc/\347\224\250\346\210\267 - 5 - FS\346\216\245\345\217\243.md"
@@ -1,94 +1,91 @@
-# 用户 - 5 - FS接口
+# 用户 - 5 - FS 接口
-
+## FS 接口
-## 总体介绍
+Fluxon FS 可以把远端机器上的目录挂载到当前 Python 进程中。挂载完成后，业务代码继续使用普通的 `open()`、`read()` 和 `write()` 访问文件。
-Fluxon FS 提供的是“把远端 export 挂到当前 Python 进程里，然后继续用 `open()` / `read()` / `write()` 访问”的能力。
+### 先理解四个名称
-用户直接接触的核心对象有三层：
+- `export`：FS Agent 对外暴露的一份目录。
+- `EXPORT_NAME`：这份 export 的逻辑名称。
+- `REMOTE_ROOT_DIR`：该目录在 FS Agent 机器上的实际路径。
+- `mount_dir_abs`：Reader 进程中的本地挂载目录。
-- KV 服务平面对象：`etcd`、`greptime`、`master`、`owner`
-- FS 角色对象：`fs_master`、`fs_agent`
-- 当前 Python 进程内的 FS 挂载对象：`FluxonKvClientConfig`、`new_store(...)`、`FluxonFsPatcher`、`mount_remote_dir(...)`
-
-它们之间的关系如下：
+它们的关系如下：
```text
-etcd + greptime + fluxonkv master + owner
- |
- v
- fluxon_fs master
- |
- v
- fluxon_fs agent
- |
- v
-FluxonKvClientConfig -> new_store(...) -> KvClient(store)
- |
- v
-FluxonFsPatcher(store)
- |
- +-- set_master_config_yaml(...)
- +-- set_cache_config_yaml(...)
- +-- set_request_identity(...)
- +-- install()
- +-- mount_remote_dir(...)
- |
- v
-open() / read() / write() / close()
+FS Agent 暴露 REMOTE_ROOT_DIR
+ ↓
+FS Master 发布 EXPORT_NAME
+ ↓
+Reader 把 export 挂载到 mount_dir_abs
+ ↓
+ open() / read() / write()
```
-`owner`、`external client`、`shared memory` 这些前置概念见 [架构和概念](./用户%20-%201%20-%20架构和概念.md)。`FluxonKvClientConfig` 和 `new_store(...)` 的基础语义见 [KV 和 RPC 接口](./用户%20-%203%20-%20KV-RPC接口.md)。
+FS Master 负责 export 配置、访问控制和管理页面；FS Agent 负责提供远端目录访问。`FluxonFsPatcher` 安装在 Reader 进程中，把对挂载目录的文件操作转发到对应的 FS Agent。
+
+### 开始前检查
-## 服务平面
+运行本页示例前，确认以下条件：
-在进入挂载代码之前，需要先把 FS 依赖的 KV 服务平面拉起来。共性的角色关系、启动顺序和 runtime 边界，统一见 [用户 - 2 - 服务平面](./用户%20-%202%20-%20服务平面.md)。
+- 已按 [用户 - 2 - 服务平面](./用户%20-%202%20-%20服务平面.md) 启动 Greptime 和 etcd。
+- 当前 Python 环境已经安装 `fluxon-*.whl` 和 `fluxon_pyo3-*.whl`；安装方式见 [用户 - 0 - 安装](./用户%20-%200%20-%20安装.md)。
+- 本机完整示例使用的端口没有被占用，`WORKDIR` 和 `REMOTE_ROOT_DIR` 可写。
+- `KvClient` 的基础用法已经明确；相关说明见 [用户 - 3 - KV 和 RPC 接口](./用户%20-%203%20-%20KV-RPC接口.md)。
-FS 直接复用 KV / MQ 的这条服务平面链路，直接接触的对象主要有：
+普通挂载和文件读写不使用目录传输状态。只有启用 `/ui/transfers/` 或预扫描时，才需要额外启动 PD 和 TiKV。
-- `greptime`：标准监控链路
-- `etcd`：KV / FS 控制面元数据存储
-- `start_kv_master_process(...)`：启动 `fluxonkv master`
-- `start_owner_kvclient_process(...)`：启动 `owner`
+### 先按使用场景选择
-FS 用户进程的共享前置链路如下：
+- **本机完整示例**：启动本机 KV Master、Owner、FS Master 和 FS Agent。首次运行优先使用这条路径。
+- **接入已有集群**：本机只启动新的 Owner 和 FS Agent，连接已有 KV Master 与 FS Master。
+- **目录传输或预扫描**：完成 FS 基础服务后，再准备 PD 和 TiKV。该流程放在本页末尾。
-1. 先起 `greptime`
-2. 再起 `etcd`
-3. 再起 `fluxonkv master`
-4. 再起 `owner`
-5. 再起 `fs master`
-6. 再起 `fs agent`
-7. 最后再运行挂载验证脚本
+### 启动本机 FS 服务
-这条顺序对应的是本页默认的本地完整示例，也就是由当前脚本同时拉起 `kv master + owner + fs master + fs_agent`。
+`examples/start_kv_and_fs_svc.py` 默认在本机启动：
-`examples/start_kv_and_fs_svc.py` 只启动 Fluxon 自己的角色。`etcd` / `greptime` 仍按服务平面文档单独启动；如果需要 `/ui/transfers/` 和预扫描，还要先启动 `transfer_state_store` 对应的 `pd` / `tikv`：
+```text
+KV Master → Owner → FS Master → FS Agent
+```
-- 当前 `python3` 所在环境已经安装 `fluxon-*.whl` 和 `fluxon_pyo3-*.whl`；安装方式见 [用户 - 0 - 安装](./用户%20-%200%20-%20安装.md)
+#### 首次只需确认这些值
-这个脚本支持两种启动方式：
+- `CLUSTER_NAME`：KV 与 FS 共用的集群名。
+- `SHARE_MEM_PATH`：本机 Owner 和所有本机 FS 进程共用的共享内存目录。
+- `FS_MASTER_INSTANCE_KEY`：FS Master 的实例标识，也是 FS Agent 和 Reader 获取配置时使用的目标。
+- `EXPORT_NAME`：FS Agent 发布、Reader 挂载的 export 名。
+- `REMOTE_ROOT_DIR`：本机 FS Agent 暴露的绝对目录。
+- `ADMIN_USERNAME` / `ADMIN_PASSWORD`：首次创建 access DB 时使用的管理员凭据。
-- 默认方式：启动 `kv master + owner + fs master + fs agent`
-- `--without-master`：只启动 `owner + fs_agent`，接入已经存在的 `kv master + fs master`
+首次本机运行时，其他实例标识、端口、`WORKDIR` 和 cache 大小可以先保留示例值。`admin/admin` 只适合本机演示，实际环境必须修改。
-## FS master 与 fs_agent
+#### 启动命令与成功判据
-KV 服务平面起来以后，FS 在这条链路上再加两个角色：
+在独立终端中运行：
+
+```bash
+python3 examples/start_kv_and_fs_svc.py
+```
-- `fs_master`：复用 KV external client 接入 KV 平面，并承载 panel / export 快照分发
-- `fs_agent`：向 `fs_master` 注册 export，并对外提供远端目录访问
+启动成功后：
-对应示例脚本：`examples/start_kv_and_fs_svc.py`
+- 终端打印 `cluster name`、`remote root dir`、`export name` 和四个角色的日志路径。
+- 终端最后打印 `waiting for Ctrl-C to stop fs demo stack`，进程保持运行。
+- `/tmp/fluxon_fs_demo/remote_root` 已经创建。
+- FS Panel 可以通过 `http://127.0.0.1:34180` 访问。
+- `/tmp/fluxon_fs_demo/runtime/log/` 下的四个日志中没有启动错误。
完整脚本如下：
+
+📄 查看完整脚本（点击展开）｜examples/start_kv_and_fs_svc.py
+
```python
#!/usr/bin/env python3
import argparse
-
from pathlib import Path
from fluxon_py.runtime import (
@@ -125,10 +122,15 @@ TRANSFER_STATE_STORE_KEY_PREFIX = f"/fluxon_fs_transfer/{CLUSTER_NAME}/"
FS_MASTER_ACCESS_DB_PATH = (WORKDIR / "fs_master" / "access.db").resolve()
+def build_owner_large_file_paths() -> list[str]:
+ return [str((WORKDIR / "large" / "owner").resolve())]
+
+
def main() -> None:
args = parse_args()
WORKDIR.mkdir(parents=True, exist_ok=True)
REMOTE_ROOT_DIR.mkdir(parents=True, exist_ok=True)
+
log_dir = (WORKDIR / "log").resolve()
log_dir.mkdir(parents=True, exist_ok=True)
@@ -168,7 +170,7 @@ def main() -> None:
config=build_fs_agent_config(),
log_path=fs_agent_stdout_log,
)
- children = []
+ children: list[ManagedSubprocess] = []
if kv_master_proc is not None:
children.append(
ManagedSubprocess(
@@ -281,7 +283,7 @@ def build_owner_config() -> dict:
"cluster_name": CLUSTER_NAME,
"share_mem_path": str(SHARE_MEM_PATH),
"sub_cluster": "default",
- "large_file_paths": [str((WORKDIR / "large" / "owner").resolve())],
+ "large_file_paths": build_owner_large_file_paths(),
},
}
@@ -377,162 +379,205 @@ if __name__ == "__main__":
main()
```
-启动命令：
+
+
+#### 接入已有集群
+
+远端 Agent 机器使用：
```bash
-python3 examples/start_kv_and_fs_svc.py
python3 examples/start_kv_and_fs_svc.py --without-master
```
-默认命令会启动本机 `kv master + owner + fs master + fs agent`。`--without-master` 只启动本机 `owner + fs_agent`，要求同一个 `cluster_name` 对应的 `kv master` 和 `fs master` 已经在别处运行。
+运行前必须调整：
-远端 agent 机器最关键的约束如下：
+- `ETCD_ENDPOINT` 和 `CLUSTER_NAME`：指向已有集群。
+- `FS_MASTER_INSTANCE_KEY`：与已有 FS Master 完全一致。
+- `SHARE_MEM_PATH`：与当前机器的新 Owner 一致。
+- `OWNER_INSTANCE_KEY` 和 `FS_AGENT_INSTANCE_KEY`：在集群内使用新值。
+- `EXPORT_NAME`：为当前远端目录分配不重复的名称。
+- `REMOTE_ROOT_DIR`：使用当前 Agent 机器上的绝对路径。
-- `ETCD_ENDPOINT` 必须改成现有集群实际使用的 `etcd` 地址
-- `FS_MASTER_INSTANCE_KEY` 必须和现有 `fs master` 的实例 key 一致
-- `OWNER_INSTANCE_KEY`、`FS_AGENT_INSTANCE_KEY`、`EXPORT_NAME`、`REMOTE_ROOT_DIR` 必须在每台 agent 机器上都唯一；重复使用这些值会让 UI 里的 agents 或 runtime exports 折叠到同一个成员上
-- `FS_PANEL_PUBLIC_BASE_URL` 控制 UI 页面里的对外链接；外部访问 `fs master` 时，要把它改成实际可访问地址，`FS_PANEL_LISTEN_ADDR` 只控制绑定地址
+该模式只管理本机 Owner 和 FS Agent，不会启动 KV Master、FS Master 或 Panel。
-脚本会持续运行，并打印：
+### 远程挂载读写验证
-- `cluster name`
-- `share_mem_path`
-- `remote root dir`
-- `export name`
-- `owner instance key`
-- `fs master instance key`
-- `fs agent instance key`
-- `start masters in this script`
-- `WORKDIR/log/owner.log`
-- `WORKDIR/log/fs_agent.log`
+验证流程使用三个脚本：
-默认模式下，还会额外打印：
+- `start_kv_and_fs_svc.py`：启动服务角色。
+- `start_fluxon_fs_writer.py`：注册 export，并持续写远端文件和本地对照文件。
+- `start_fluxon_fs_reader.py`：挂载 export，并持续读取远端文件和本地对照文件。
-- `panel listen addr`
-- `panel public base url`
-- `transfer state store pd_endpoints`
-- `transfer state store key_prefix`
-- 默认管理员用户名密码
-- `WORKDIR/log/kv_master.log`
-- `WORKDIR/log/fs_master.log`
+#### 运行前的配置对应关系
+
+Writer 和 Reader 都要求：
+
+- `-c/--config` 指向已经准备好的环境 YAML。
+- `-w/--workdir` 指向当前进程独占的可写目录。
+- `kvclient.instance_key` 在集群内唯一。
+- `cluster_name` 与 FS 服务一致。
+- `share_mem_path` 与各自机器上的 Owner 一致。
+
+此外：
+
+- Writer 配置中的 export 名和远端根目录必须与目标 FS Agent 对应。
+- Reader 的 `fluxon_fs.master.instance_key` 必须等于目标 `FS_MASTER_INSTANCE_KEY`。
+- Reader 的 `export_name` 必须等于 Writer 注册、FS Master 发布的 export 名。
+- Writer 和 Reader 使用相同的远端相对路径。
+
+本页不内嵌完整 Writer/Reader YAML；`-c` 应使用当前部署环境已经确认过的配置文件。
+
+#### Reader 挂载目录要求
+
+`mount_dir_abs` 必须满足：
-默认模式下，这个脚本把四个子进程的 `stdout/stderr` 都收进 `WORKDIR/log`，终端只保留摘要信息；按 `Ctrl-C` 时，主进程会统一停止这四个角色。`--without-master` 模式下，脚本只管理本机 `owner + fs_agent` 两个角色。
+- 使用绝对路径。
+- 不能是 `/`。
+- 目录不存在时可以由 Fluxon 创建。
+- 目录已经存在时必须为空。
+- 不能与当前进程中的其他挂载目录重叠。
-## 远程挂载读写验证
+挂载目录不要求放在 `/fluxon_fs/` 下，例如 `/tmp/fluxon_fs_demo/mount_demo` 也可以。
-根目录 `examples/` 里的公开 FS 验证脚本收束成三类对象：
+#### 启动 Writer
-- `examples/start_kv_and_fs_svc.py`
- - 本机直接拉起 `kv master + owner + fs master + fs agent`
-- `examples/start_fluxon_fs_writer.py`
- - 注册 export，并持续写远端 export 文件和本地 cache 规则文件
-- `examples/start_fluxon_fs_reader.py`
- - 通过 `install_patcher_from_master(...)` 安装 patcher，挂载 export，并持续交替读取远端文件和本地文件
+保持 FS 服务运行，在第二个终端执行：
-最小成功路径如下：
+```bash
+python3 examples/start_fluxon_fs_writer.py \
+ -c \
+ -w
+```
-1. 运行 `python3 examples/start_kv_and_fs_svc.py`
-2. 保持它持续运行
-3. 准备 writer 配置，运行 `python3 examples/start_fluxon_fs_writer.py -c -w `
-4. 准备 reader 配置，运行 `python3 examples/start_fluxon_fs_reader.py -c -w `
+Writer 成功后会持续打印：
-这条最小成功路径默认对应本页的本地完整示例，也就是不带 `--without-master` 的启动方式。`--without-master` 用于把当前机器接到已经存在的 KV / FS 集群；如果继续运行 `start_fluxon_fs_writer.py` / `start_fluxon_fs_reader.py`，配置里的这些对象必须和现有集群一致：
+```text
+[writer] op=write_remote ...
+[writer] op=write_local ...
+```
-- `cluster_name`
-- `share_mem_path`
-- `fluxon_fs.master.instance_key`
-- `export_name`
-- `remote_root_dir_abs`
+#### 启动 Reader
-`start_fluxon_fs_writer.py` 固定负责两件事：
+在第三个终端执行：
-- 向当前 `fs master` 注册当前 export
-- 持续写入远端 export 文件和本地 cache 规则文件
+```bash
+python3 examples/start_fluxon_fs_reader.py \
+ -c \
+ -w
+```
-`start_fluxon_fs_reader.py` 固定负责三件事：
+Reader 完成挂载并找到文件后，会持续打印：
-- 用 external client 接到本机 owner
-- 通过 `install_patcher_from_master(...)` 安装 patcher，并从 `fs master` 拉取配置
-- 把指定 export 挂到本地 mount dir，持续交替读取远端文件和本地文件
+```text
+[reader] op=read_remote ...
+[reader] op=read_local ...
+```
-当 `reader` 开始稳定打印 `op=read_remote` / `op=read_local` 时，说明远端挂载链路和本地 cache 规则都已经打通。
+这表示远端 export 挂载和本地 cache 规则都已经生效。
-如果你要直接在自己的 Python 进程里调用 `FluxonFsPatcher` 和 `mount_remote_dir(...)`，继续看下面的 API 规则；根目录 `examples/` 不再保留单独的进程内挂载演示脚本。
+> **当前示例限制**：`start_fluxon_fs_reader.py` 尚未调用 `set_request_identity(...)`，也不会从 YAML 读取用户名和密码。在启用了访问控制的 FS Master 上，需要在应用代码中显式设置请求身份，否则远端文件操作会因缺少认证 token 失败。
-## 目录传输与预扫描
+### `FluxonFsPatcher` 调用顺序
-有时候需要做一次很大的目录搬迁，例如跨机群搬迁，或者跨共享存储搬迁。目录里可能有多层子目录、很多文件，整个任务会持续很久。这个时候，用户真正关心的通常不是某一个文件，而是这次目录任务有没有开始、当前扫到哪里了、是不是已经开始写入、现在带宽怎么样。
+Reader 的推荐流程是：
-目录传输与预扫描就是为这种场景准备的。它把这种长时间的大目录搬运收成一个持续运行的任务：目录可以直接在网页里发起，也可以先做预扫描，等目录规模、目标位置和并发策略确定后，再导入成正式任务。
+```text
+new_store(...)
+→ install_patcher_from_master(...)
+→ wait_cache_config_loaded()
+→ set_request_identity(...)
+→ mount_remote_dir(...)
+→ open() / read() / write()
+→ patcher.uninstall()
+→ store.close()
+```
-### 在网页里直接发起目录传输
+关键规则：
-最常见的入口是双 pane 浏览页面里的跨 export 文件夹拖拽。这里说的是网页里的文件夹从一个 pane 拖到另一个 pane，不是把本地文件夹拖进浏览器。
+- `FluxonFsPatcher` 依赖 `new_store(...)` 返回的 `KvClient`。
+- `install_patcher_from_master(...)` 会从 FS Master 拉取 export 配置并安装 Patcher。
+- `set_request_identity(username, password)` 为后续 FS 请求设置身份。
+- 必须先调用 `patcher.uninstall()`，再关闭 `store`。
-操作顺序如下：
+需要手动控制配置加载时，可以使用：
-1. 打开两个 pane
-2. 左侧定位源文件夹
-3. 右侧定位目标 export 和目标目录
-4. 把左侧文件夹拖到右侧
-5. 在弹窗里填写 `desired_worker_count` 和 `batch_ready_bytes`
-6. 提交后，到 `/ui/transfers/` 查看任务
+- `load_cache_config_from_master_config_file(config_path)`：读取文件中的 FS Master 实例信息，并阻塞到配置拉取成功。
+- `start_cache_config_fetch_from_master_config_file(config_path)`：在后台持续从 FS Master 拉取配置；`install_patcher_from_master(...)` 已经调用了这个接口。
+- `set_cache_config_yaml(...)`：不从 FS Master 拉取时，直接注入 export 和 cache 配置。
-提交之后，这个目录任务会出现在 `/ui/transfers/` 的 `FluxonFS Transfer Jobs` 里。
+普通 Reader 优先使用 `install_patcher_from_master(...)`，不需要自行组合这些接口。
-### 页面上能看到什么
+### 进阶配置
-`/ui/transfers/` 页面里，和这组功能直接相关的有两个区域：
+#### Panel 与访问控制
-- `Pre-Scans`
-- `FluxonFS Transfer Jobs`
+- `FS_PANEL_LISTEN_ADDR` 决定 FS Master 在本机监听的网卡和端口。
+- `FS_PANEL_PUBLIC_BASE_URL` 是浏览器和页面链接使用的公开地址。跨机器访问时，应改成实际可达的主机名或 IP。
+- `FS_MASTER_ACCESS_DB_PATH` 保存用户、密码和权限，应使用稳定、可写的绝对路径。
+- `bootstrap_access_model` 只在 access DB 没有用户时写入首批账号。
+- 数据库已有用户后，修改 `ADMIN_USERNAME` 或 `ADMIN_PASSWORD` 不会覆盖现有账号。
+- `can_manage_users: true` 的管理员在运行时可以访问所有当前 export。
-`FluxonFS Transfer Jobs` 用来查看已经进入正式传输链路的目录任务。页面上会持续显示：
+#### 日志
-- 扫描进度
-- 已经拆出的 batch 数量
-- 当前 running batches 数量
-- 当前 `live bandwidth`
-- worker 明细
+服务脚本把子进程标准输出写入：
-这些信息通常足够判断：
+- `WORKDIR/log/kv_master.log`
+- `WORKDIR/log/owner.log`
+- `WORKDIR/log/fs_master.log`
+- `WORKDIR/log/fs_agent.log`
-- 任务现在还在扫描，还是已经开始写入
-- 当前带宽是否正常
-- 当前并发是否达到预期
+Writer 和 Reader 的 Python 日志默认输出到当前终端。需要增加日志时，可以设置：
-### 在页面上导入预扫描
+```bash
+FLUXON_LOG=DEBUG python3 examples/start_fluxon_fs_reader.py \
+ -c \
+ -w
+```
-当 `Pre-Scans` 里已经有一条预扫描记录后，可以直接在网页里把它导入成正式目录任务。
+#### 常见错误
-操作顺序如下：
+- **`new_store failed`**：先检查服务脚本是否仍在运行，以及 `cluster_name`、`share_mem_path` 是否与本机 Owner 一致。
+- **`unknown export_name`**：检查 Reader 的 `export_name` 是否已经由 FS Agent 注册并由 FS Master 发布。
+- **挂载目录错误**：检查路径是否为绝对路径、是否为空，以及是否与已有挂载重叠。
+- **`fluxon_fs cache config is not loaded yet`**：检查 FS Master 实例标识是否正确，并确认配置拉取已经完成。
+- **`permission denied` / `PermissionError`**：检查是否调用了 `set_request_identity(...)`，以及账号密码和 access DB 权限。
-1. 打开 `/ui/transfers/`
-2. 在 `Pre-Scans` 里找到对应任务
-3. 点击 `Import`
-4. 在弹窗里选择 `source export`
-5. 选择 `target export`
-6. 填写 `target prefix`
-7. 填写 `desired_worker_count`
-8. 提交后，这个任务会进入 `FluxonFS Transfer Jobs`
+### 可选：目录传输与预扫描
-如果同一个源目录同时匹配多个 `source export`，页面会把这些候选都列出来，由用户自己选择。
+目录传输适合跨机器或跨共享存储的大目录搬迁。它会持续记录扫描进度、batch 数量、运行中的 worker 和实时带宽。
-### 启动时的 TiKV 配置
+该功能依赖 TiKV `transfer_state_store`。开始前先按 [用户 - 2 - 服务平面](./用户%20-%202%20-%20服务平面.md) 启动 PD 和 TiKV。
-目录传输和预扫描都依赖 `transfer_state_store`。`fs master` 页面和独立发起预扫描的进程，必须共用同一份 TiKV 命名空间；否则脚本里发起的预扫描，页面上看不到。
+#### 在页面发起目录传输
-启动时最关键的是下面两项要一致：
+在双 pane 页面中：
-- `pd_endpoints`
-- `key_prefix`
+1. 左侧定位源文件夹。
+2. 右侧定位目标 export 和目录。
+3. 把左侧文件夹拖到右侧。
+4. 设置 `desired_worker_count` 和 `batch_ready_bytes`。
+5. 提交后到 `/ui/transfers/` 查看任务。
+
+`FluxonFS Transfer Jobs` 会显示扫描进度、batch 数量、运行中的 worker 和实时带宽。
+
+#### 导入预扫描
+
+`/ui/transfers/` 的 `Pre-Scans` 区域可以把已有预扫描导入为正式任务：
-本页这个 `start_kv_and_fs_svc.py` 示例已经把它们写成：
+1. 找到预扫描记录并点击 `Import`。
+2. 选择 source export 和 target export。
+3. 填写 target prefix 和 `desired_worker_count`。
+4. 提交后在 `FluxonFS Transfer Jobs` 中查看。
-- `TRANSFER_STATE_STORE_PD_ENDPOINTS = ["127.0.0.1:12379"]`
-- `TRANSFER_STATE_STORE_KEY_PREFIX = "/fluxon_fs_transfer/demo-fs-cluster/"`
+#### TiKV 命名空间
-`fs master` 的 `master_panel` 配置里需要带上这一段：
+FS Master 与独立预扫描进程必须使用相同的：
+
+- `pd_endpoints`
+- `key_prefix`
+
+本页示例统一使用：
```yaml
transfer_state_store:
@@ -543,34 +588,12 @@ transfer_state_store:
key_prefix: "/fluxon_fs_transfer/demo-fs-cluster/"
```
-独立预扫描脚本里用的 `FluxonFsTransferStateStoreTiKvConfig(...)`，也要使用同样的 `pd_endpoints` 和 `key_prefix`。
-
-### 页面截图（待补）
-
-- 图：`/ui/transfers/` 页面总览
- 需要标出 `Pre-Scans` 和 `FluxonFS Transfer Jobs`
-- 图：`Pre-Scans` 的 `Import` 弹窗
- 需要标出 `source export`、`target export`、`target prefix`、`desired_worker_count`
-- 图：双 pane 页面里跨 export 拖拽文件夹后的目录传输表单
- 需要标出 `desired_worker_count` 和 `batch_ready_bytes`
+#### 独立预扫描示例
-### 独立进程发起预扫描示例
+独立预扫描只依赖 PD 和 TiKV，不要求 KV Master、Owner、FS Master 或 FS Agent 已经启动。后续 FS Master 使用相同 `pd_endpoints` 和 `key_prefix` 启动后，页面即可看到预扫描结果。
-独立发起预扫描时，需要先启动的是 TiKV `transfer_state_store` 对应的 PD 和 TiKV。
-
-这一段不依赖下面这些组件先启动：
-
-- `etcd`
-- `fluxonkv master`
-- `owner`
-- `fs master`
-- `fs agent`
-
-也就是说，只要 TiKV `transfer_state_store` 可用，预扫描脚本就可以先单独跑起来。等后面 `fs master` 页面使用同一份 `pd_endpoints` 和 `key_prefix` 启动后，网页里就能看到这次预扫描。
-
-下面这个示例会扫描 `/data/demo_src`，并把结果写到和页面共用的 `transfer_state_store` 里。脚本跑完后，打开 `/ui/transfers/`，就可以在 `Pre-Scans` 里看到这次预扫描。
-
-最小示例如下：
+
+📄 查看完整预扫描示例（点击展开）
```python
#!/usr/bin/env python3
@@ -588,7 +611,7 @@ STORE = FluxonFsTransferStateStoreConfig(
kind=FluxonFsTransferStateStoreKind.TIKV,
tikv=FluxonFsTransferStateStoreTiKvConfig(
pd_endpoints=["127.0.0.1:12379"],
- key_prefix="/fluxon_fs_transfer/demo_prescan/",
+ key_prefix="/fluxon_fs_transfer/demo-fs-cluster/",
),
)
@@ -613,137 +636,6 @@ summary = transfer_check_local_blocking(
print(summary)
```
-`summary` 里最有用的是 `job_id`、`scan_epoch` 和 `batch_count`。网页上的 `Pre-Scans` 会按同一个 `job_id` 展示这次预扫描结果。
-
-## 关键对象与规则
-
-### `FluxonFsPatcher`
-
-`FluxonFsPatcher` 不是独立入口，它必须依附在 `new_store(...)` 返回的 `store` 上。
-
-固定顺序如下：
-
-1. `store = new_store(cfg)...`
-2. `patcher = FluxonFsPatcher(store)`
-3. `patcher.set_master_config_yaml(...)`
-4. `patcher.set_cache_config_yaml(...)`
-4. `patcher.set_request_identity(...)`
-5. `patcher.install()`
-6. `patcher.mount_remote_dir(...)`
-7. `open()` / `read()` / `write()`
-8. `patcher.uninstall()`
-9. `store.close()`
-
-这里不能把 `store.close()` 放到 `patcher.uninstall()` 前面，因为 patcher 还在工作时，后续文件操作仍然可能走到底层 client。
-
-### `set_master_config_yaml(...)` 与 `set_cache_config_yaml(...)`
-
-这两个接口一起构成当前示例里的最小 authority：
-
-- `set_master_config_yaml(...)` 负责注入 `fluxon_fs.master.instance_key`，让挂载后的 mount-registry RPC 能回报给正确的 `fs master`
-- `set_cache_config_yaml(...)` 负责注入当前 export 快照，让 patcher 知道有哪些 export 可以挂载
-
-业务代码可以直接在 Python 里构造这两段 YAML 文本；当前公开 runtime 示例 `start_fluxon_fs_reader.py` 则通过 `install_patcher_from_master(...)` 从配置文件和 `fs master` 拉取这两层信息。
-
-### `set_request_identity(...)`
-
-`set_request_identity(username, password)` 负责把当前 Python 进程里的后续 FS 请求绑定到这组身份。
-
-如果不设置身份：
-
-- 当 `access_db` 为空、系统还没有启用真实用户模型时，请求可能处于未鉴权路径
-- 一旦 master 已经有用户和 `scope_access`，请求会按这个身份做权限判断
-
-用户示例里应该显式设置身份，不要把鉴权省略掉。
-
-### `bootstrap_access_model`
-
-`bootstrap_access_model` 是 `fs master` 启动配置里的必填项，用来给一个空的 `access_db` 写入第一批账号和目录权限。
-
-配置位置：
-
-```yaml
- fluxon_fs:
- master_panel:
- access_db_path: /path/to/access.db
- bootstrap_access_model:
- users:
- - username: admin
- password: admin
- can_manage_users: true
- scope_access: []
-```
-
-规则如下：
-
-- `access_db_path` 是长期 authority
-- `bootstrap_access_model` 必须在启动配置里显式提供
-- 只有当 `access_db` 还没有用户时，`bootstrap_access_model` 才会写入数据库
-- 数据库一旦已有用户，后续重启以数据库为准
-- `can_manage_users: true` 的用户在运行时可以访问所有当前 export，不依赖在数据库里预先展开 root `scope_access`
-- 页面不再提供首次管理员创建入口；首个管理员只能通过启动配置初始化
-
-## 挂载目录规则
-
-`mount_remote_dir(local_mount_dir_abs=..., export_name=...)` 对本地挂载目录的要求如下：
-
-- 必须是绝对路径
-- 不能是 `/`
-- 如果目录不存在，Fluxon 会创建它
-- 如果目录已存在，它必须是空目录
-- 不能和当前进程里已有挂载目录互相重叠
-
-因此，挂载目录并不要求必须放在 `/fluxon_fs/...` 下。`/tmp/fluxon_fs_demo/mount_demo` 这种绝对路径也可以。
-
-## 日志
-
-如果需要更详细的 Python 侧日志，可以在启动用户进程前设置：
-
-```bash
-FLUXON_LOG=DEBUG python3 examples/start_fluxon_fs_reader.py -c -w
-```
-
-常用值有：
-
-- `DEBUG`
-- `INFO`
-- `WARNING`
-- `ERROR`
-- `CRITICAL`
-
-## 常见错误
-
-### `new_store failed`
-
-通常表示当前 external client 没有接上本机 owner。先检查：
-
-- `start_kv_and_fs_svc.py` 是否还在运行
-- `CLUSTER_NAME`
-- `SHARE_MEM_PATH`
-
-### `fluxon_fs cache config is not loaded yet`
-
-通常表示 `set_cache_config_yaml(...)` 没有成功完成，或者脚本里的 cache 配置和服务端当前 export 配置不一致。先检查：
-
-- `FS_MASTER_INSTANCE_KEY` 是否一致
-- `EXPORT_NAME`
-- `REMOTE_ROOT_DIR`
-
-### `unknown export_name`
-
-表示客户端要挂载的 `EXPORT_NAME` 没有出现在当前 `fs master` 的 export 快照里。先检查：
-
-- `start_fluxon_fs_writer.py` 和 `start_fluxon_fs_reader.py` 的 `export_name` 是否一致
-- `REMOTE_ROOT_DIR` 是否和启动脚本里的 export 配置对应
-
-### `permission denied` 或 `PermissionError`
-
-这表示路径存在，但当前身份没有访问权限。先检查：
-
-- `ADMIN_USERNAME`
-- `ADMIN_PASSWORD`
-- 当前 `access_db` 里是否已经被新的用户数据覆盖
-
-如果你已经通过页面或数据库改过管理员密码，那么旧的 `bootstrap_access_model` 密码不会再生效。
+
-管理员对所有 export 的访问权限也是运行时按 `can_manage_users` 判断，不需要额外往 `scope_access` 里补根路径。
+运行前将 `src_root_dir` 改成实际存在的源目录。`summary` 中最常用的是 `job_id`、`scan_epoch` 和 `batch_count`。
diff --git a/fluxon_doc_en/user_doc/User - 2 - Service Plane.md b/fluxon_doc_en/user_doc/User - 2 - Service Plane.md
index fce4a30..5b88f77 100644
--- a/fluxon_doc_en/user_doc/User - 2 - Service Plane.md
+++ b/fluxon_doc_en/user_doc/User - 2 - Service Plane.md
@@ -1,82 +1,114 @@
# User - 2 - Service Plane
-## Service Plane
+
-To use Fluxon KV you need to understand the fixed service-plane objects that come before your business process. They own control-plane metadata, the shared memory pool, and the startup choreography for KV roles.
+## Service Plane
-From a user point of view the most common objects are:
+This page explains how to start the Fluxon KV service processes and which settings matter on the first run. The minimum KV path needs only `Greptime`, `etcd`, `Master`, and `Owner Client`. TiKV is needed only by features that persist task state, such as FS directory transfer and pre-scan.
-- External dependencies: `etcd`, `Greptime`, `TiKV`
-- Fluxon-native roles: `Master`, `Owner Client`
-- Startup entrypoints: raw `etcd / Greptime / TiKV` runtimes, `fluxon_py.runtime`, and your own supervisor or scripts
+### Choose the Startup Scenario First
-If you are writing business code, this page answers three questions:
+- **Minimum local KV**: Start `Greptime`, `etcd`, `Master`, and `Owner Client` on the local machine. This is the recommended path for a first run.
+- **Attach to an existing Master**: If `Greptime`, `etcd`, and `Master` are already running in the cluster, start only a new local `Owner Client`.
+- **Enable FS directory transfer or pre-scan**: Start PD and TiKV after the KV service plane is running. TiKV can be skipped when using only KV, RPC, or MQ.
-- Which processes must be started first
-- How those processes relate to one another
-- Which objects may be started by `fluxon_py.runtime` and which may not
+### Components and Startup Order
-For the user-facing API, continue to [User - 3 - KV and RPC Interface](<./User - 3 - KV and RPC Interface.md>).
+The recommended local startup order is:
-### Role Model
+```text
+Greptime → etcd → Master → Owner Client → application process new_store(...)
+```
-The service plane can be reduced to five stable objects:
+Each component has one clear responsibility:
-- External dependency: `etcd`
-- External dependency: `Greptime`
-- External dependency: `TiKV`
-- Fluxon-native role: `Master`
-- Fluxon-native role: `Owner Client`
+- `Greptime` receives monitoring data reported by Master.
+- `etcd` stores control-plane metadata such as membership, routes, and leases.
+- `Master` manages the KV cluster.
+- `Owner Client` provides the shared-memory pool on the current machine and creates the `shared.json` used by application processes.
+- PD and TiKV persist task state for features such as FS directory transfer and pre-scan. They are not part of the minimum KV read/write path.
-Deployment layout:
+The deployment layout is shown below:
-Responsibilities:
+`Greptime`, `etcd`, PD, and TiKV are external dependencies and must be started separately. `fluxon_py.runtime` starts only Fluxon-native roles such as `Master` and `Owner Client`.
-- `etcd`: control-plane metadata
-- `Greptime`: standard observability path
-- `TiKV`: persistent task-state storage for extended features such as FS transfer-state persistence
-- `Master`: membership, routing, leases, monitoring broadcast, master-side logs
-- `Owner Client`: local shared-memory pool and `shared.json`
+### Shortest Local Startup Path
-### Minimum Startup Order
+Prepare the runtime package described in [User - 0 - Installation](<./User - 0 - Installation.md>) and confirm that the files required by the minimum KV path exist:
-The minimum chain for KV is:
+- `ext_images/greptime/greptime`
+- `ext_images/greptime/start.sh`
+- `ext_images/etcd/etcd`
+- `ext_images/etcd/etcdctl`
+- `ext_images/etcd/start.sh`
-- KV: `Greptime -> etcd -> Fluxon KV Master -> Owner Client -> business process new_store(...)`
+Run the following commands from the Fluxon repository root. Each startup command is long-running, so use a separate terminal or `tmux` window for each component and confirm that it starts successfully before continuing.
-If you also need transfer-state-backed features such as directory transfer or pre-scan, add:
+#### Understand `--config` and `--workdir` First
-- Transfer / Pre-Scan: `TiKV PD -> TiKV -> FS Master transfer_state_store`
+Each external-service startup script accepts two arguments:
-Hard boundary:
+- `--config/-c` determines how the component starts. The file must define the component's Bash array, such as `GREPTIME_ARGS` or `ETCD_ARGS`.
+- `--workdir/-w` determines where the component stores its local data and logs. Each instance should have its own writable work directory.
-- `etcd`, `Greptime`, and `TiKV` are external dependencies
-- `Master` and `Owner Client` are Fluxon-native roles
+Inside the config file, `WORKDIR` is the directory passed through `--workdir`. For example:
-If the control plane is missing, `Master` is unavailable. If `Owner Client` is missing, `FluxonKvClientConfig({...}) -> new_store(...)` cannot attach to the shared-memory pool. TiKV is not needed for the minimum KV read/write path, but it is required for features that depend on `transfer_state_store`.
+```text
+--workdir /tmp/fluxon_service_plane_demo/etcd
+--data-dir "$WORKDIR/etcd-data"
+```
-### Start `etcd`, `Greptime`, and `TiKV`
+The resulting data directory is `/tmp/fluxon_service_plane_demo/etcd/etcd-data`. For a first local run, confirm only that the work directories are writable and the example ports are free. The single-node membership settings can normally remain unchanged.
-First prepare the runtime package described in [User - 0 - Installation](<./User - 0 - Installation.md>) and confirm these files exist:
+#### 1. Start Greptime
-- `ext_images/etcd/etcd`
-- `ext_images/etcd/etcdctl`
-- `ext_images/etcd/start.sh`
-- `ext_images/greptime/greptime`
-- `ext_images/greptime/start.sh`
-- `ext_images/tikv/pd-server`
-- `ext_images/tikv/tikv-server`
-- `ext_images/tikv/start_pd.sh`
-- `ext_images/tikv/start_tikv.sh`
+**Purpose**: Provide monitoring query and write endpoints for Master.
+
+**First-run checks**:
+
+- Port `34030` in `--http-addr 0.0.0.0:34030` is free.
+- The work directory used by `--data-home "$WORKDIR/greptimedb"` is writable.
+
+Configuration and startup command:
+
+```bash
+cat > /tmp/greptime.config.sh <<'EOF'
+GREPTIME_ARGS=(
+ standalone start
+ --data-home "$WORKDIR/greptimedb"
+ --http-addr 0.0.0.0:34030
+)
+EOF
+
+bash ./ext_images/greptime/start.sh \
+ --config /tmp/greptime.config.sh \
+ --workdir /tmp/fluxon_service_plane_demo/greptime
+```
+
+**Success condition**: The process remains running, the terminal shows no startup error, and local port `34030` accepts connections. The later Master example uses this port by default.
+
+
+🌐 Multi-machine deployment | --http-addr
-All startup scripts use the same contract:
+`0.0.0.0` means that Greptime listens on every local network interface. It is not an address that another machine should use as a destination. Set `GREPTIME_BASE_URL` in the Master config to a real hostname or IP address reachable from the Master host.
-- `--config/-c`: shell config file
-- `--workdir/-w`: local work directory
+
-Example `etcd` config:
+#### 2. Start etcd
+
+**Purpose**: Store the control-plane metadata used by Master and Owner Client.
+
+**First-run checks**:
+
+- `--advertise-client-urls "http://127.0.0.1:2379"` corresponds to `ETCD_ENDPOINT = "127.0.0.1:2379"` in the later Python script.
+- Ports `2379` and `2380` are free.
+- The work directory used by `--data-dir "$WORKDIR/etcd-data"` is writable.
+
+`--name`, the peer addresses, and `--initial-cluster` define etcd membership. The values below can remain unchanged for a local single-node deployment.
+
+Configuration and startup command:
```bash
cat > /tmp/etcd.config.sh <<'EOF'
@@ -99,23 +131,241 @@ bash ./ext_images/etcd/start.sh \
--workdir /tmp/fluxon_service_plane_demo/etcd
```
-Example `greptime` config:
+**Success condition**: The health endpoint returns `"health":"true"`.
```bash
-cat > /tmp/greptime.config.sh <<'EOF'
-GREPTIME_ARGS=(
- standalone start
- --data-home "$WORKDIR/greptimedb"
- --http-addr 0.0.0.0:34030
+curl -sS http://127.0.0.1:2379/health
+```
+
+
+🌐 Multi-machine deployment | etcd listen and member addresses
+
+- `--listen-client-urls` and `--listen-peer-urls` determine which local interfaces etcd listens on.
+- `--advertise-client-urls` is the address used by clients such as Master and Owner Client.
+- `--initial-advertise-peer-urls` is the address used by other etcd members.
+- `--name` must match its member name in `--initial-cluster`, and every member address must be reachable between hosts.
+
+
+
+#### 3. Start Master and Owner Client
+
+The two common `fluxon_py.runtime` entrypoints are:
+
+- `start_kv_master_process(config=...)`
+- `start_owner_kvclient_process(config=...)`
+
+By default, `examples/start_master_owner.py` starts a local Master and Owner Client together. Only the following fields need to be understood before the first run:
+
+- `ETCD_ENDPOINT`: The etcd endpoint used by Master and Owner Client. Use `host:port` without `http://`. Keep `127.0.0.1:2379` when following the local etcd example above.
+- `CLUSTER_NAME`: The logical cluster name. Master, Owner Client, and later application processes must use the same value.
+- `SHARE_MEM_PATH`: The local shared-memory directory. Owner Client and later application processes on the same machine must use the same path.
+- `MASTER_INSTANCE_KEY`: The Master instance identity. It must be unique within the cluster.
+- `OWNER_INSTANCE_KEY`: The Owner Client instance identity. It must be unique within the cluster.
+- `OWNER_DRAM_BYTES`: The number of DRAM bytes contributed by Owner Client. The default is `1073741824`, or 1 GiB. The value must be greater than zero and satisfy the capacity-alignment requirement.
+
+When Greptime and etcd use the default ports above and local port `31000` is free, `GREPTIME_HTTP_PORT`, `MASTER_PORT`, and the example directories can remain unchanged.
+
+The script supports two startup modes:
+
+- **Start a local Master and Owner Client**:
+
+ ```bash
+ python3 examples/start_master_owner.py
+ ```
+
+- **Start only Owner Client and attach to an existing Master**:
+
+ ```bash
+ python3 examples/start_master_owner.py --without-master
+ ```
+
+ In this mode, `ETCD_ENDPOINT` and `CLUSTER_NAME` must match the existing cluster. `OWNER_INSTANCE_KEY` must be a new value, and `SHARE_MEM_PATH` must be a local path on the current machine.
+
+**Success condition**: The script remains running, the terminal prints `waiting for Ctrl-C`, and `shared.json` appears under `SHARE_MEM_PATH`. With the default config, check it with:
+
+```bash
+ls -l /dev/shm/fluxon_kv_demo/shared.json
+```
+
+Master and Owner Client standard output is written to `/tmp/fluxon_kv_demo/runtime/log/master.log` and `/tmp/fluxon_kv_demo/runtime/log/owner.log`, respectively. Check these files if startup fails.
+
+The full script is included below for running directly from a source checkout:
+
+
+📄 View full script (click to expand) | examples/start_master_owner.py
+
+```python
+#!/usr/bin/env python3
+
+import argparse
+
+from pathlib import Path
+
+from fluxon_py.runtime import (
+ start_kv_master_process,
+ start_owner_kvclient_process,
+ wait_subproc_or_ctrlc,
)
-EOF
+from fluxon_py.runtime.process_runner import ManagedSubprocess
+
+ETCD_ENDPOINT = "127.0.0.1:2379"
+GREPTIME_HTTP_PORT = 34030
+GREPTIME_BASE_URL = f"http://127.0.0.1:{GREPTIME_HTTP_PORT}"
+CLUSTER_NAME = "demo-kv-cluster"
+SHARE_MEM_PATH = Path("/dev/shm/fluxon_kv_demo").resolve()
+WORKDIR = Path("/tmp/fluxon_kv_demo/runtime").resolve()
+MASTER_PORT = 31000
+MASTER_INSTANCE_KEY = "demo_kv_master"
+OWNER_INSTANCE_KEY = "demo_kv_owner"
+OWNER_DRAM_BYTES = 1073741824
+
+
+def main() -> None:
+ args = parse_args()
+ log_dir = (WORKDIR / "log").resolve()
+
+ if args.with_master:
+ master_log_dir = (WORKDIR / "master_logs").resolve()
+ master_log_dir.mkdir(parents=True, exist_ok=True)
+ master_stdout_log = log_dir / "master.log"
+ master_proc = start_kv_master_process(
+ config=build_master_config(log_dir=master_log_dir),
+ log_path=master_stdout_log,
+ )
+ else:
+ master_stdout_log = None
+ master_proc = None
+
+ owner_stdout_log = log_dir / "owner.log"
+ owner_proc = start_owner_kvclient_process(
+ config=build_owner_config(),
+ log_path=owner_stdout_log,
+ )
+ children = []
+ if master_proc is not None:
+ children.append(
+ ManagedSubprocess(
+ label="master",
+ proc=master_proc,
+ )
+ )
+ children.append(
+ ManagedSubprocess(
+ label="owner",
+ proc=owner_proc,
+ )
+ )
+
+ print(f"[fluxon_kv] share_mem_path: {SHARE_MEM_PATH}")
+ print(f"[fluxon_kv] etcd endpoint: {ETCD_ENDPOINT}")
+ print(f"[fluxon_kv] greptime base url: {GREPTIME_BASE_URL}")
+ print(f"[fluxon_kv] start master in this script: {args.with_master}")
+ if master_stdout_log is not None:
+ print(f"[fluxon_kv] master stdout log: {master_stdout_log}")
+ else:
+ print("[fluxon_kv] master stdout log: disabled by --without-master")
+ print(f"[fluxon_kv] owner stdout log: {owner_stdout_log}")
+ stack_label = "master and owner" if args.with_master else "owner"
+ print(f"[fluxon_kv] waiting for Ctrl-C to stop {stack_label}")
+ wait_subproc_or_ctrlc(
+ children,
+ on_ctrlc=lambda: print(f"[fluxon_kv] caught Ctrl-C, stopping {stack_label}"),
+ )
+
+
+def parse_args() -> argparse.Namespace:
+ parser = argparse.ArgumentParser(description="Start KV demo owner, optionally with a local master")
+ group = parser.add_mutually_exclusive_group()
+ group.add_argument(
+ "--with-master",
+ dest="with_master",
+ action="store_true",
+ help="Start a local kv master in this script (default)",
+ )
+ group.add_argument(
+ "--without-master",
+ dest="with_master",
+ action="store_false",
+ help="Do not start a local kv master; only start owner and attach to an existing cluster master",
+ )
+ parser.set_defaults(with_master=True)
+ return parser.parse_args()
+
+
+def build_master_config(*, log_dir: Path) -> dict:
+ return {
+ "instance_key": MASTER_INSTANCE_KEY,
+ "cluster_name": CLUSTER_NAME,
+ "port": MASTER_PORT,
+ "etcd_endpoints": [ETCD_ENDPOINT],
+ "log_dir": str(log_dir),
+ "monitoring": {
+ "prometheus_base_url": f"{GREPTIME_BASE_URL}/v1/prometheus",
+ "prom_remote_write_url": [f"{GREPTIME_BASE_URL}/v1/prometheus/write"],
+ "otlp_log_api": {
+ "otlp_endpoint": f"{GREPTIME_BASE_URL}/v1/otlp/v1/logs",
+ },
+ },
+ }
+
+
+def build_owner_config() -> dict:
+ return {
+ "instance_key": OWNER_INSTANCE_KEY,
+ "contribute_to_cluster_pool_size": {
+ "dram": OWNER_DRAM_BYTES,
+ "vram": {},
+ },
+ "fluxonkv_spec": {
+ "etcd_addresses": [ETCD_ENDPOINT],
+ "cluster_name": CLUSTER_NAME,
+ "share_mem_path": str(SHARE_MEM_PATH),
+ "sub_cluster": "default",
+ "large_file_paths": [str((WORKDIR / "large" / "owner").resolve())],
+ },
+ }
+
+
+if __name__ == "__main__":
+ main()
+```
-bash ./ext_images/greptime/start.sh \
- --config /tmp/greptime.config.sh \
- --workdir /tmp/fluxon_service_plane_demo/greptime
+
+
+When running from a source checkout, this script can be used directly. With a `wheel` installation, call the two `fluxon_py.runtime` entrypoints from the application startup program and pass Python dicts directly instead of depending on the `examples/` directory.
+
+The example uses `wait_subproc_or_ctrlc(...)` to wait for all child processes. Pressing Ctrl-C stops the Master and Owner Client processes that it started.
+
+### Optional: Start PD and TiKV for FS Directory Transfer
+
+Skip this section when using only KV, RPC, or MQ. When FS directory transfer or pre-scan needs `transfer_state_store`, use this startup order:
+
+```text
+PD → TiKV → FS Master
```
-Example TiKV PD config:
+Confirm that the runtime package provides these files:
+
+- `ext_images/tikv/pd-server`
+- `ext_images/tikv/tikv-server`
+- `ext_images/tikv/start_pd.sh`
+- `ext_images/tikv/start_tikv.sh`
+
+The PD and TiKV scripts use the same `--config/-c` and `--workdir/-w` arguments.
+
+#### 1. Start PD
+
+**Purpose**: Manage the TiKV cluster and provide the cluster entrypoint used by TiKV and FS `transfer_state_store`.
+
+**First-run checks**:
+
+- `12379` is the client port and `12380` is the PD member port. Both ports must be free.
+- The work directory used by `--data-dir` and `--log-file` is writable.
+- The config array is named `PD_ARGS`.
+
+The member name, peer addresses, and `--initial-cluster` can remain unchanged for a local single-node deployment.
+
+Configuration and startup command:
```bash
cat > /tmp/pd.config.sh <<'EOF'
@@ -136,7 +386,24 @@ bash ./ext_images/tikv/start_pd.sh \
--workdir /tmp/fluxon_service_plane_demo/tikv_pd
```
-Example TiKV config:
+**Success condition**: The PD process remains running and the members endpoint responds successfully.
+
+```bash
+curl -sS http://127.0.0.1:12379/pd/api/v1/members
+```
+
+#### 2. Start TiKV
+
+**Purpose**: Persist task state for FS directory transfer and pre-scan.
+
+**First-run checks**:
+
+- PD is already running, and `--pd-endpoints "127.0.0.1:12379"` matches the PD client address above.
+- `20160` is the TiKV service port and `20180` is the status and metrics port. Both ports must be free.
+- The work directory used by `--data-dir` and `--log-file` is writable.
+- The config array is named `TIKV_ARGS`.
+
+Configuration and startup command:
```bash
cat > /tmp/tikv.config.sh <<'EOF'
@@ -155,28 +422,56 @@ bash ./ext_images/tikv/start_tikv.sh \
--workdir /tmp/fluxon_service_plane_demo/tikv
```
-These external services are not started by `fluxon_py.runtime`.
+**Success condition**: The TiKV process remains running, its log has no PD connection or storage-initialization errors, and local port `20160` accepts connections.
-### `fluxon_py.runtime`
+
+🌐 Multi-machine deployment | published PD and TiKV addresses
-`fluxon_py.runtime` only manages Fluxon-native roles. It does not replace `etcd`, `Greptime`, or `TiKV`.
+- PD `--advertise-client-urls` must be reachable from TiKV and FS Master. `transfer_state_store.pd_endpoints` uses this client address.
+- PD `--advertise-peer-urls` and `--initial-cluster` use addresses reachable between PD members.
+- TiKV `--pd-endpoints` must point to a running PD, and `--advertise-addr` must be reachable from other machines.
+- `--client-urls`, `--peer-urls`, `--addr`, and `--status-addr` determine where each process listens. Do not use bind-only `0.0.0.0` as a remote destination.
-Common entrypoints:
+
-- `start_kv_master_process(config=...)`
-- `start_owner_kvclient_process(config=...)`
+### Advanced Configuration
+
+The following settings can remain unchanged for a first local run. Adjust them only when ports, directories, or deployment hosts change.
+
+#### Custom Monitoring Address and Master Port
+
+- `GREPTIME_HTTP_PORT` is the Greptime HTTP port and must match Greptime `--http-addr`.
+- `GREPTIME_BASE_URL` is the Greptime address reachable from Master, in `http://host:port` form.
+- The example derives three endpoints from `GREPTIME_BASE_URL`: Prometheus queries use `/v1/prometheus`, remote write uses `/v1/prometheus/write`, and OTLP logs use `/v1/otlp/v1/logs`. These paths stay unchanged when the host or port changes.
+- `MASTER_PORT` is the local Master listen port. It is unused when the script starts only Owner Client. When a local Master is started, the port must not conflict with another process.
+
+#### Logs and Large-File Directories
+
+> **Note**: `log_path` stores child-process standard output and standard error captured by the Python startup script. `log_dir` is Master's own business-log and profile directory. `large_file_paths` contains the directories where Owner Client stores backend logs, caches, and large-file data. These fields have different purposes.
+
+The example derives these paths:
+
+- `WORKDIR/log/master.log` and `WORKDIR/log/owner.log`: Passed as `log_path` for startup troubleshooting.
+- `WORKDIR/master_logs`: Passed as `log_dir` in the Master config.
+- `WORKDIR/large/owner`: Passed as Owner Client `fluxonkv_spec.large_file_paths`. Owner Client mode requires at least one writable directory.
-Optional wrapper log argument:
+Each process group should use its own writable `WORKDIR`. When one startup program manages several roles, assign a different `log_path` to each role.
-- `log_path=...`
+#### Multi-Machine Address, Path, and Port Planning
-This controls the Python wrapper subprocess `stdout/stderr` destination, not the service's own business-log directory.
+For a multi-machine deployment, check the following items in order:
-If you are an installed-wheel user, prefer these Python entrypoints directly and pass Python dicts instead of depending on `examples/` path layout.
+1. Every Master, Owner Client, and application process uses the same `CLUSTER_NAME`.
+2. Every Master and Owner Client has a unique `instance_key`.
+3. Each machine has its own `WORKDIR`. `SHARE_MEM_PATH` may differ between machines, but Owner Client and application processes on the same machine must use the same value.
+4. Ports must not conflict with other processes on the same host.
+5. `listen` fields, or fields without `advertise`, determine where a process listens locally. `advertise` fields determine the address used by other machines.
+6. Every etcd, Greptime, PD, and TiKV address supplied to another machine must be reachable. Do not use remote-inaccessible `127.0.0.1` or bind-only `0.0.0.0`.
-See `examples/start_master_owner.py` for the common local pattern:
+For the full KV configuration objects and field semantics, see [User - 3 - KV and RPC Interface](<./User - 3 - KV and RPC Interface.md>).
-- Default: start `Master + Owner Client`
-- `--without-master`: start only `Owner Client` and attach to an existing `Master`
+### Where to Go Next
-The same role chain is reused by MQ and FS. MQ-specific behavior belongs to [User - 4 - MQ Interface](<./User - 4 - MQ Interface.md>) and FS-specific behavior belongs to [User - 5 - FS Interface](<./User - 5 - FS Interface.md>).
+- When writing Python KV APIs or making node-to-node RPC calls, see [User - 3 - KV and RPC Interface](<./User - 3 - KV and RPC Interface.md>).
+- When using Producer / Consumer messaging, see [User - 4 - MQ Interface](<./User - 4 - MQ Interface.md>).
+- When starting FS Master / Agent, registering exports, or mounting remote directories, see [User - 5 - FS Interface](<./User - 5 - FS Interface.md>).
diff --git a/fluxon_doc_en/user_doc/User - 3 - KV and RPC Interface.md b/fluxon_doc_en/user_doc/User - 3 - KV and RPC Interface.md
index 1e05e5f..12fb9ac 100644
--- a/fluxon_doc_en/user_doc/User - 3 - KV and RPC Interface.md
+++ b/fluxon_doc_en/user_doc/User - 3 - KV and RPC Interface.md
@@ -2,206 +2,108 @@
## KV and RPC Interface
-This page describes Fluxon's Python KV API and node-to-node RPC calls. Both are exposed from the same `KvClient` instance and share one lifecycle.
+This page introduces the Fluxon Python KV API and node-to-node RPC. Both are provided by the same `KvClient` and share one configuration model, data model, and shutdown flow.
-See [Architecture and Concepts](<./User - 1 - Architecture and Concepts.md>) for `cluster_name`, `instance_key`, `etcd`, and the other base concepts.
+Before continuing, follow [User - 2 - Service Plane](<./User - 2 - Service Plane.md>) to start Greptime, etcd, Master, and the local Owner Client. This page focuses only on how an application process connects to an already running KV cluster.
-For business code, prefer passing a Python dict directly into `FluxonKvClientConfig(...)`. YAML is better suited to standalone processes, supervisors, deployment, and example environments.
+Application code should normally pass a Python dict directly to `FluxonKvClientConfig(...)`. YAML files and role-specific configuration are covered in the advanced section.
-### Service Plane
+### Checks Before Starting
-Before writing `put_blocking`, `get_blocking`, or `rpc_call`, start the KV service plane first. The shared role model, startup order, and runtime boundary are described in [User - 2 - Service Plane](<./User - 2 - Service Plane.md>).
+Confirm the following before running the examples:
-
+- Master and the local Owner Client are running, and `shared.json` exists under the local shared-memory directory.
+- The current Python environment has installed `fluxon-*.whl` and `fluxon_pyo3-*.whl`; see [User - 0 - Installation](<./User - 0 - Installation.md>).
+- The application process uses the same `cluster_name` as the target cluster.
+- Its `share_mem_path` matches the Owner Client on the same machine.
+- Every application process has a different `instance_key`.
-The most common objects are:
+When the default configuration from User 2 is used, the examples on this page can keep:
-- `Greptime`: standard observability path
-- `etcd`: KV control-plane metadata
-- `start_kv_master_process(...)`: starts `Fluxon KV Master`
-- `start_owner_kvclient_process(...)`: starts `Owner Client`
+```python
+CLUSTER_NAME = "demo-kv-cluster"
+SHARE_MEM_PATH = "/dev/shm/fluxon_kv_demo"
+```
-The minimal local startup example is `examples/start_master_owner.py`. It only starts Fluxon-native roles and assumes:
+The examples create External Clients that use the local Owner Client's memory pool without contributing capacity. Their configs therefore must not contain Owner-only fields such as `contribute_to_cluster_pool_size`, `etcd_addresses`, `sub_cluster`, or `large_file_paths`.
-- `etcd` at `127.0.0.1:2379`
-- `Greptime` HTTP at `127.0.0.1:34030`
-- the current Python environment already installed `fluxon-*.whl` and `fluxon_pyo3-*.whl`
+### Shared Call Flow
-### Minimal Role Startup Example
+Both KV and RPC begin with the same `KvClient`:
-```python
-#!/usr/bin/env python3
+```text
+FluxonKvClientConfig
+ ↓
+new_store(config).unwrap(...) → KvClient
+ ↓
+call KV or RPC APIs
+ ↓
+store.close().unwrap(...)
+```
-import argparse
+- `FluxonKvClientConfig` holds the connection settings for the current Python process.
+- `new_store(...)` creates a `KvClient`.
+- `KvClient` provides both KV and RPC APIs.
+- Most public APIs return `Result`. The examples use `unwrap("error context")` to obtain a successful value; failures are raised with that context.
+- Call `close()` and consume its `Result` when the client is no longer needed.
-from pathlib import Path
+### FlatDict Data Model
-from fluxon_py.runtime import (
- start_kv_master_process,
- start_owner_kvclient_process,
- wait_subproc_or_ctrlc,
-)
-from fluxon_py.runtime.process_runner import ManagedSubprocess
+KV values, RPC requests, and RPC responses all use a single-level Python dict:
-ETCD_ENDPOINT = "127.0.0.1:2379"
-GREPTIME_HTTP_PORT = 34030
-GREPTIME_BASE_URL = f"http://127.0.0.1:{GREPTIME_HTTP_PORT}"
-CLUSTER_NAME = "demo-kv-cluster"
-SHARE_MEM_PATH = Path("/dev/shm/fluxon_kv_demo").resolve()
-WORKDIR = Path("/tmp/fluxon_kv_demo/runtime").resolve()
-MASTER_PORT = 31000
-MASTER_UI_PORT = 18080
-MASTER_INSTANCE_KEY = "demo_kv_master"
-OWNER_INSTANCE_KEY = "demo_kv_owner"
-OWNER_DRAM_BYTES = 1073741824
+```python
+value = {
+ "payload": b"hello",
+ "count": 1,
+ "source": "demo",
+}
+```
+Keys must be strings. Values may be `int`, `float`, `bool`, `str`, `bytes`, or DLPack data. Do not nest another dict or list inside a value.
-def main() -> None:
- args = parse_args()
- log_dir = (WORKDIR / "log").resolve()
-
- if args.with_master:
- master_log_dir = (WORKDIR / "master_logs").resolve()
- master_log_dir.mkdir(parents=True, exist_ok=True)
- master_stdout_log = log_dir / "master.log"
- master_proc = start_kv_master_process(
- config=build_master_config(log_dir=master_log_dir),
- log_path=master_stdout_log,
- )
- else:
- master_stdout_log = None
- master_proc = None
-
- owner_stdout_log = log_dir / "owner.log"
- owner_proc = start_owner_kvclient_process(
- config=build_owner_config(),
- log_path=owner_stdout_log,
- )
- children = []
- if master_proc is not None:
- children.append(ManagedSubprocess(label="master", proc=master_proc))
- children.append(ManagedSubprocess(label="owner", proc=owner_proc))
-
- print(f"[fluxon_kv] share_mem_path: {SHARE_MEM_PATH}")
- print(f"[fluxon_kv] etcd endpoint: {ETCD_ENDPOINT}")
- print(f"[fluxon_kv] greptime base url: {GREPTIME_BASE_URL}")
- print(f"[fluxon_kv] start master in this script: {args.with_master}")
- if master_stdout_log is not None:
- print(f"[fluxon_kv] master stdout log: {master_stdout_log}")
- print(
- "[fluxon_kv] kv web ui: "
- f"http://:{MASTER_UI_PORT}/view?cluster_name={CLUSTER_NAME}&member_kind=kv"
- )
- else:
- print("[fluxon_kv] master stdout log: disabled by --without-master")
- print(f"[fluxon_kv] owner stdout log: {owner_stdout_log}")
- stack_label = "master and owner" if args.with_master else "owner"
- print(f"[fluxon_kv] waiting for Ctrl-C to stop {stack_label}")
- wait_subproc_or_ctrlc(
- children,
- on_ctrlc=lambda: print(f"[fluxon_kv] caught Ctrl-C, stopping {stack_label}"),
- )
+The corresponding type is:
+```python
+FlatDict = Dict[str, Union[int, float, bool, str, bytes, DLPacked]]
+```
-def parse_args() -> argparse.Namespace:
- parser = argparse.ArgumentParser(description="Start KV demo owner, optionally with a local master")
- group = parser.add_mutually_exclusive_group()
- group.add_argument("--with-master", dest="with_master", action="store_true")
- group.add_argument("--without-master", dest="with_master", action="store_false")
- parser.set_defaults(with_master=True)
- return parser.parse_args()
-
-
-def build_master_config(*, log_dir: Path) -> dict:
- return {
- "instance_key": MASTER_INSTANCE_KEY,
- "cluster_name": CLUSTER_NAME,
- "port": MASTER_PORT,
- "etcd_endpoints": [ETCD_ENDPOINT],
- "log_dir": str(log_dir),
- "monitoring": {
- "prometheus_base_url": f"{GREPTIME_BASE_URL}/v1/prometheus",
- "prom_remote_write_url": [f"{GREPTIME_BASE_URL}/v1/prometheus/write"],
- "otlp_log_api": {
- "otlp_endpoint": f"{GREPTIME_BASE_URL}/v1/otlp/v1/logs",
- },
- },
- "master_ui": {
- "http_listen_addr": f"0.0.0.0:{MASTER_UI_PORT}",
- },
- }
+### Minimal KV Example
+`examples/external_put_get_del.py` performs one write, read, delete, and existence check.
-def build_owner_config() -> dict:
- return {
- "instance_key": OWNER_INSTANCE_KEY,
- "contribute_to_cluster_pool_size": {
- "dram": OWNER_DRAM_BYTES,
- "vram": {},
- },
- "fluxonkv_spec": {
- "etcd_addresses": [ETCD_ENDPOINT],
- "cluster_name": CLUSTER_NAME,
- "share_mem_path": str(SHARE_MEM_PATH),
- "sub_cluster": "default",
- "large_file_paths": [str((WORKDIR / "large" / "owner").resolve())],
- },
- }
+Only three values need to be checked before running it:
+- `INSTANCE_KEY`: Unique identity of the current Python process. It must not duplicate another process in the cluster.
+- `CLUSTER_NAME`: Must match Master and the local Owner Client.
+- `SHARE_MEM_PATH`: Must match the local Owner Client.
-if __name__ == "__main__":
- main()
-```
+The example also sets `test_spec_config.disable_observability = True` to keep observability background tasks out of this minimal process. It is not a core field required to attach to the cluster.
-Start it with:
+Run:
```bash
-python3 examples/start_master_owner.py
-python3 examples/start_master_owner.py --without-master
+python3 examples/external_put_get_del.py
```
-### Lifecycle and Call Flow
+A successful run prints:
```text
-FluxonKvClientConfig
- |
- v
-new_store(cfg) -> KvClient
- | |
- | +-- KV: put_blocking / get_blocking / remove / is_exist / ...
- | |
- | +-- RPC server: rpc_register(path, handler)
- | |
- | +-- RPC client: rpc_call(node_id, path, payload, timeout_ms)
- |
- v
-close()
+OK put key=hello
+world
+OK del key=hello
+OK is_exist after remove -> False
```
-### Core Python Objects
-
-- `FluxonKvClientConfig`: config object, usually built from a Python dict
-- `new_store(config: FluxonKvClientConfig) -> Result[KvClient, ApiError]`: create one KV client
-- `KvClient`: single entrypoint for both KV and RPC
-- `KvClient.third_party_logs_dir() -> Result[str, ApiError]`: return the Fluxon-assigned log root for third-party Python components. Components should derive their own subdirectories under this root, for example `mq/`.
-- `MemHolder`: successful result holder from `get_blocking(...)`
-- `PutOptionalArgs`: optional write controls, most commonly `lease_id`
+The script performs these steps:
-Notes:
+1. Attach to the local Owner Client with `new_store(...)`.
+2. Write `{"payload": b"world"}` with `put_blocking(...)`.
+3. Obtain a `MemHolder` with `get_blocking(...)`, then call `access()` to get the `FlatDict`.
+4. Delete the key with `remove(...)` and confirm its absence with `is_exist(...)`.
+5. Release references associated with the `MemHolder`, then close the `KvClient`.
-- `MemHolder` does not expose `bytes()` directly; call `access()` and read the bytes field from the returned flat dict
-- `store.close()` waits until all user-visible `MemHolder` references are dropped
-- `Result` values must be consumed explicitly with `unwrap()` or `unwrap_error()`
-
-### Data Model
-
-Both KV values and RPC payloads use one flat-dict contract:
-
-- `FlatDict = Dict[str, Union[int, float, bool, str, bytes, dlpack]]`
-
-### Minimal KV Example
-
-`examples/external_put_get_del.py`:
+
+📄 View full script (click to expand) | examples/external_put_get_del.py
```python
#!/usr/bin/env python3
@@ -233,15 +135,25 @@ def main() -> None:
try:
store.put_blocking(key, {"payload": value}).unwrap("put_blocking failed")
+ print(f"OK put key={key}")
+
mem = store.get_blocking(key).unwrap("get_blocking failed")
flat = mem.access().unwrap("mem.access failed")
payload = flat["payload"]
+ if not isinstance(payload, (bytes, bytearray)):
+ raise RuntimeError(f"payload is not bytes: {type(payload)}")
print(bytes(payload).decode("utf-8"))
+
store.remove(key).unwrap("remove failed")
+ print(f"OK del key={key}")
+
exists = store.is_exist(key).unwrap("is_exist failed")
if exists:
- raise RuntimeError("expected deleted key to be absent")
+ raise RuntimeError(f"expected is_exist({key!r}) to be False after remove")
+ print("OK is_exist after remove -> False")
finally:
+ # Release MemHolder-related references before close(); client shutdown waits
+ # until all user-visible holders are dropped.
if "flat" in locals():
del flat
if "mem" in locals():
@@ -253,27 +165,87 @@ if __name__ == "__main__":
main()
```
-Useful calls:
+
-- `put_blocking(key, value, opts=None)`: write or overwrite one KV object
-- `get_blocking(key)`: return `MemHolder`
-- `MemHolder.access()`: expand to `FlatDict`
-- `get_size(key)`: query payload size without reading the whole object
-- `is_exist(key)`: existence check
-- `remove(key)`: delete a key
-- `third_party_logs_dir()`: return `{large_file_paths[0]}/{cluster_name}_cluster_third_party_logs` as a `Result[str, ApiError]`
+### Common KV APIs
-To increase user-process logs:
+#### Write
-```bash
-FLUXON_LOG=DEBUG python3 examples/external_put_get_del.py
+```python
+put_blocking(
+ key: str,
+ value: FlatDict,
+ opts: Optional[PutOptionalArgs] = None,
+) -> Result[OkNone, ApiError]
+```
+
+- `key` is the KV key to write.
+- `value` is a single-level `FlatDict`.
+- Ordinary writes do not need `opts`.
+- A successful return means that the write is complete; no additional `wait()` is needed.
+
+#### Read
+
+```python
+get_blocking(key: str) -> Result[MemHolder, ApiError]
+MemHolder.access() -> Result[FlatDict, ApiError]
+```
+
+`get_blocking(...)` returns a `MemHolder`. Call `access()` to obtain the application dict. `MemHolder` does not expose `bytes()` directly; read bytes fields from the `FlatDict` returned by `access()`.
+
+`store.close()` waits for all `MemHolder` objects exposed to application code to be released. Do not retain them longer than needed, and release related references before closing as shown in the example.
+
+#### Delete and Check
+
+```python
+remove(key: str) -> Result[OkNone, ApiError]
+is_exist(key: str) -> Result[bool, ApiError]
```
-Third-party Python components should place file logs under `store.third_party_logs_dir().unwrap(...)` and then append a component subdirectory such as `mq/`. This keeps log directory usage bounded and lets the Fluxon observability plane discover and collect those file logs through one `Owner Client`-derived root.
+Use `remove(...)` to delete a key and `is_exist(...)` to check whether it exists. An immediate `get_blocking(...)` after deletion is not guaranteed to return `KeyNotFoundError` because Owner Client and Master metadata caches may still be converging. Prefer `is_exist(...)` when verifying the delete request.
### Minimal Node-to-Node RPC Example
-`examples/rpc_call.py`:
+Node-to-node RPC lets one application process register a handler and another call it through the target `instance_key`. Both processes first create their own `KvClient`.
+
+Before running the example, confirm:
+
+- `RPC_SERVER_INSTANCE_KEY` and `RPC_CLIENT_INSTANCE_KEY` are different and both unique within the cluster.
+- Server and Client use the target cluster's `CLUSTER_NAME`.
+- Each process uses the `SHARE_MEM_PATH` of the Owner Client on its own machine. The local example uses one shared path.
+- `--target-instance-key` matches the `instance_key` actually used by Server.
+- The Client call path matches the path registered by Server. This example uses `/count`.
+
+Start Server in terminal one:
+
+```bash
+python3 examples/rpc_call.py serve
+```
+
+Keep Server running after it prints:
+
+```text
+[rpc] handler ready instance_key=demo_rpc_server
+[rpc] waiting for Ctrl-C
+```
+
+Start Client in terminal two:
+
+```bash
+python3 examples/rpc_call.py call --target-instance-key demo_rpc_server
+```
+
+On the first successful call, Client prints `1`. Server also prints the caller, payload, and cumulative call count.
+
+The call flow is:
+
+```text
+Server: new_store → rpc_register("/count", handler) → keep running
+Client: new_store → rpc_call(...) → wait() → FlatDict response → close
+```
+
+
+📄 View full script (click to expand) | examples/rpc_call.py
```python
#!/usr/bin/env python3
@@ -289,6 +261,31 @@ CLUSTER_NAME = "demo-kv-cluster"
SHARE_MEM_PATH = "/dev/shm/fluxon_kv_demo"
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Minimal node-to-node RPC example")
+ subparsers = parser.add_subparsers(dest="command", required=True)
+
+ serve_parser = subparsers.add_parser("serve", help="Start one RPC handler process")
+ serve_parser.add_argument("--instance-key", default=RPC_SERVER_INSTANCE_KEY, help="RPC handler instance key")
+
+ call_parser = subparsers.add_parser("call", help="Call one RPC handler and print the counter")
+ call_parser.add_argument("--instance-key", default=RPC_CLIENT_INSTANCE_KEY, help="RPC caller instance key")
+ call_parser.add_argument(
+ "--target-instance-key",
+ default=RPC_SERVER_INSTANCE_KEY,
+ help="Target RPC handler instance key",
+ )
+
+ args = parser.parse_args()
+ if args.command == "serve":
+ run_server(instance_key=args.instance_key)
+ return
+ if args.command == "call":
+ run_client(instance_key=args.instance_key, target_instance_key=args.target_instance_key)
+ return
+ raise AssertionError("unreachable")
+
+
def _build_config(*, instance_key: str) -> FluxonKvClientConfig:
return FluxonKvClientConfig(
{
@@ -319,68 +316,175 @@ def run_server(*, instance_key: str) -> None:
try:
store.rpc_register("/count", count_handler).unwrap("rpc_register failed")
+ print(f"[rpc] handler ready instance_key={instance_key}")
+ print("[rpc] waiting for Ctrl-C")
signal.pause()
+ except KeyboardInterrupt:
+ print("[rpc] caught Ctrl-C, stopping handler")
+ raise SystemExit(130)
+ finally:
+ store.close().unwrap("close failed")
+
+
+def run_client(*, instance_key: str, target_instance_key: str) -> None:
+ store = new_store(_build_config(instance_key=instance_key)).unwrap("new_store failed")
+ try:
+ resp = (
+ store.rpc_call(target_instance_key, "/count", {"payload": b"hi"})
+ .unwrap("rpc_call failed")
+ .wait()
+ .unwrap("rpc wait failed")
+ )
+ print(resp["count"])
finally:
store.close().unwrap("close failed")
+
+
+if __name__ == "__main__":
+ main()
```
-Important constraints:
+
-- `node_id` usually matches the target node's `instance_key`
-- `timeout_ms` defaults to `10000`
-- Keep one primary public pattern: `rpc_call(...).wait()` on the response handle
+### Common RPC APIs
-### Config Objects
+#### Register a Handler
-You usually touch two config layers:
+```python
+rpc_register(
+ path: str,
+ handler: Callable[[from_node_id: str, payload: FlatDict], FlatDict],
+) -> Result[OkNone, ApiError]
+```
-- `Master` config: starts the control-plane process
-- external-client config: attaches business code to the local `Owner Client` and drives KV / RPC
+- `path` is the RPC path and must match the Client call path.
+- `handler` receives the caller's `instance_key` and a `FlatDict` payload, then returns a `FlatDict`.
+- Server must remain running after registration succeeds.
-Minimal master YAML:
+#### Make a Call
-```yaml
-instance_key: my-master-1
-cluster_name: demo-kv-cluster
-port: 31000
-etcd_endpoints:
- - 127.0.0.1:2379
-log_dir: /var/lib/fluxon/master_logs
+```python
+rpc_call(
+ node_id: str,
+ path: str,
+ payload: FlatDict,
+ timeout_ms: int = 10000,
+) -> Result[KvFuture, ApiError]
```
-Minimal external-client YAML:
+- `node_id` normally identifies the target process by its `instance_key`.
+- `timeout_ms` defaults to `10000`; an explicit value must not be lower than `10000`.
+- `rpc_call(...).unwrap(...)` first obtains the response handle.
+- Calling `wait().unwrap(...)` on that handle waits for remote completion and returns the response `FlatDict`.
+
+### Advanced Notes
+
+#### Async APIs and Write Options
+
+`put_blocking(...)` and `get_blocking(...)` return only after the operation completes. To submit an operation first and wait for its result later, use:
+
+```python
+put(...) -> Result[KvFuture, ApiError]
+get(...) -> Result[KvFuture, ApiError]
+```
+
+Both return a `KvFuture` first. Call `wait()` to obtain the final result.
+
+Other APIs:
+
+- `get_size(key) -> Result[int, ApiError]`: Query the value size without reading the full payload.
+- `PutOptionalArgs(lease_id=None, reject_if_inflight_same_key=False)`: Bind a write to a lease or reject it immediately when another write for the same key is already in flight.
+- `PutOptionalArgs.support_mooncake() -> Tuple[bool, List[str]]`: Check whether the selected write options are compatible with Mooncake and return the incompatible field names.
+
+#### Logging and Observability
+
+- `FLUXON_LOG` controls the console log level of the current Python process. Valid values are `DEBUG`, `INFO`, `WARNING`, `ERROR`, and `CRITICAL`; the default is `INFO`.
+- `store.third_party_logs_dir().unwrap(...)` returns the file-log root allocated by Fluxon for third-party Python components. Each component should create its own subdirectory below that root, such as `mq/`.
+- `test_spec_config.disable_observability` is a testing and minimal-example switch. Ordinary application configs should not depend on it merely to attach to the cluster.
+- When Master configures `monitoring.otlp_log_api`, background service logs are also written to the Greptime `fluxon_logs` table.
+
+To enable more detailed logs for the example process:
+
+```bash
+FLUXON_LOG=DEBUG python3 examples/external_put_get_del.py
+```
+
+#### Python Dicts and YAML
+
+Application code should normally construct `FluxonKvClientConfig` directly:
+
+```python
+from fluxon_py import FluxonKvClientConfig
+
+cfg = FluxonKvClientConfig(
+ {
+ "instance_key": "my-kv-client-1",
+ "fluxonkv_spec": {
+ "cluster_name": "demo-kv-cluster",
+ "share_mem_path": "/dev/shm/fluxon_kv_demo",
+ },
+ }
+)
+```
+
+When configuration is already stored in YAML, load it from a file:
+
+```python
+cfg = FluxonKvClientConfig.from_file("./kv_external.yaml")
+```
+
+The External Client YAML below is equivalent to the Python dict above:
```yaml
instance_key: my-kv-client-1
-
fluxonkv_spec:
cluster_name: demo-kv-cluster
- share_mem_path: /dev/shm/fluxon
- p2p_listen_port: 31001
+ share_mem_path: /dev/shm/fluxon_kv_demo
```
-`Owner Client` config adds memory contribution and `etcd` addresses:
+Owner Client contributes memory to the cluster and therefore also needs capacity, etcd, sub-cluster, and large-file settings:
```yaml
instance_key: my-owner-1
-
contribute_to_cluster_pool_size:
- dram: 1677721600
+ dram: 1073741824
vram: {}
-
fluxonkv_spec:
etcd_addresses:
- 127.0.0.1:2379
cluster_name: demo-kv-cluster
- share_mem_path: /dev/shm/fluxon
- p2p_listen_port: 31000
+ share_mem_path: /dev/shm/fluxon_kv_demo
sub_cluster: default
+ large_file_paths:
+ - /tmp/fluxon_kv_demo/runtime/large/owner
```
-Keep these roots separate:
+Master uses a separate startup config that is not passed to `FluxonKvClientConfig`:
+
+```yaml
+instance_key: my-master-1
+cluster_name: demo-kv-cluster
+port: 31000
+etcd_endpoints:
+ - 127.0.0.1:2379
+log_dir: /tmp/fluxon_kv_demo/runtime/master_logs
+```
-- `share_mem_path`: shared bundle root. Runtime appends `cluster_name`, and that directory holds `mmap.file`, `shared.json`, and peer metadata.
-- `large_file_paths`: `Owner Client`-only large-file authority for logs, profiles, caches, and other derived runtime assets
-- `FLUXON_LOG`: console log threshold for the user process
+See [User - 2 - Service Plane](<./User - 2 - Service Plane.md>) for startup commands, log directories, and multi-machine constraints for these roles.
+
+#### Optional: KV Web UI
+
+To inspect KV cluster state, add the following block to the Master config:
+
+```yaml
+master_ui:
+ http_listen_addr: 0.0.0.0:18080
+```
+
+`master_ui` depends on the Master `monitoring` config. After Master starts, open:
+
+```text
+http://:18080/view?cluster_name=demo-kv-cluster&member_kind=kv
+```
-In zero-contribution external mode, `Owner Client`-only fields such as `fluxonkv_spec.etcd_addresses`, `fluxonkv_spec.sub_cluster`, `fluxonkv_spec.large_file_paths`, and `fluxonkv_spec.redis_compat` should not appear.
+`0.0.0.0` only means "listen on all local interfaces." Use a reachable Master hostname or IP address in the browser URL.
diff --git a/fluxon_doc_en/user_doc/User - 4 - MQ Interface.md b/fluxon_doc_en/user_doc/User - 4 - MQ Interface.md
index f461829..92b92f4 100644
--- a/fluxon_doc_en/user_doc/User - 4 - MQ Interface.md
+++ b/fluxon_doc_en/user_doc/User - 4 - MQ Interface.md
@@ -1,193 +1,524 @@
# User - 4 - MQ Interface
-## Overview
+## MQ Interface
-Fluxon MQ is the queue layer built on top of the KV substrate. It is not a separate service stack. It reuses the same service plane, the same local shared-memory pool, and the same Python client attachment path, then adds `Producer` / `Consumer` semantics.
+Fluxon MQ reuses the KV service plane and the local Owner Client's memory pool. It provides Producer and Consumer handles on top of `KvClient`. Producer sends messages with `put_data(...)`, and Consumer receives them with `get_data(...)`.
-MQ objects can be understood in three layers:
+The most important connection rule is that Producer and Consumer must use the same `CHANNEL_KEY`, channel type, and channel config to join the same channel.
-- Service plane: `etcd`, `Greptime`, `Fluxon KV Master`
-- Local resident data-plane instance: `Owner Client`
-- Business-process attachment layer: `FluxonKvClientConfig`, `new_store(...) -> KvClient`, plus the bound `Producer` / `Consumer` handles
+### Checks Before Starting
-```text
-etcd + Greptime + Fluxon KV Master
- |
- v
- Owner Client
- |
- v
-+--------------------------------------------------------------+
-| External Client |
-| FluxonKvClientConfig -> new_store(...) -> KvClient(store) |
-+--------------------------------------------------------------+
- |
- +-> new_or_bind_with_unique_key(...)
- |
- +-> Producer
- +-> Consumer
-```
+Confirm the following before running the MQ example:
-See [Architecture and Concepts](<./User - 1 - Architecture and Concepts.md>) for `Owner Client`, `External Client`, and shared-memory terminology, and [User - 3 - KV and RPC Interface](<./User - 3 - KV and RPC Interface.md>) for `new_store(...) -> KvClient`.
+- Greptime, etcd, Master, and the local Owner Client have been started as described in [User - 2 - Service Plane](<./User - 2 - Service Plane.md>).
+- `shared.json` exists under the local shared-memory directory.
+- The current Python environment has installed `fluxon-*.whl` and `fluxon_pyo3-*.whl`; see [User - 0 - Installation](<./User - 0 - Installation.md>).
+- Producer and Consumer use the same `cluster_name` as the target cluster.
+- Each process uses the `share_mem_path` of the Owner Client on the same machine.
+- Every Producer and Consumer process has a different `instance_key`.
-MQ user processes have one fixed role constraint: `Producer` / `Consumer` processes must run as zero-contribution `external_client` attachments. Their lifecycles are expected to be dynamic, so they must not change cluster capacity; the long-lived capacity provider remains the local `Owner Client`.
+When the default configuration from User 2 is used, the example can keep:
-## Service Plane
+```python
+CLUSTER_NAME = "demo-kv-cluster"
+SHARE_MEM_PATH = "/dev/shm/fluxon_kv_demo"
+```
-MQ reuses the KV service plane directly. Start the shared chain first:
+Producer and Consumer are External Clients that use capacity already provided by Owner Client. Their configs must not contain Owner-only fields such as `contribute_to_cluster_pool_size`, `etcd_addresses`, `sub_cluster`, or `large_file_paths`.
-1. `Greptime`
-2. `etcd`
-3. `Fluxon KV Master`
-4. `Owner Client`
-5. your `Producer` / `Consumer` process
+See [User - 3 - KV and RPC Interface](<./User - 3 - KV and RPC Interface.md>) for the basic `KvClient`, `FlatDict`, and `Result` contracts.
-The common startup pattern is still `examples/start_master_owner.py`.
+### Object Relationship
-## Object Relationship
+The minimum MQ call relationship is:
```text
FluxonKvClientConfig
- |
- v
-new_store(cfg) -> KvClient (store)
- |
- +-- new_or_bind_with_unique_key(...)
- |
- +-- producer handle
- | put_data(...)
- | close()
- |
- +-- consumer handle
- | get_data(...)
- | close()
- |
- v
-store.close()
+ ↓
+new_store(...) → KvClient
+ ↓
+bind a channel with the same CHANNEL_KEY
+ ├─ Producer → put_data(...)
+ └─ Consumer → get_data(...)
+```
+
+`new_or_bind_with_unique_key(...)` looks up a channel by `CHANNEL_KEY`:
+
+- The first process using that key creates the channel.
+- Later processes using the same key bind to the existing channel.
+- Different logical channels should use different keys.
+
+Each process creates its own `KvClient` and then creates a Producer or Consumer handle on top of it. The shutdown order is fixed:
+
+```text
+close the Producer / Consumer handle first → close KvClient second
```
-Key rules:
+### Minimal MQ Example
+
+`examples/start_mpmc_demo.py` uses `ChanType.MPMC`, which supports multiple Producers and multiple Consumers. The first run needs only one Consumer and one Producer.
+
+#### Values to Check on the First Run
+
+- `--role`: Run the current process as Producer or Consumer.
+- `CLUSTER_NAME`: Must match Master and the local Owner Client.
+- `SHARE_MEM_PATH`: Must match the local Owner Client.
+- `CHANNEL_KEY`: Must match exactly in both processes. The example uses `demo_mq_channel_doc`.
+- `instance_key`: The script derives `demo_mq_producer` or `demo_mq_consumer` from the role, so the two process identities are different.
+
+The other example values can remain unchanged:
+
+- `CHANNEL_CAPACITY = 128`
+- `CHANNEL_TTL_SECONDS = 300`
+- `PRODUCER_INTERVAL_SECONDS = 1.0`
+- `CONSUMER_BATCH_SIZE = 1`
+
+Their meanings and constraints are covered in the advanced section. To start two processes with the same role, assign each process a different `instance_key` instead of reusing the example's fixed role-derived value.
+
+#### Message Content
-- `new_store(...)` creates `KvClient`; `store` is just the example variable name
-- `new_or_bind_with_unique_key(...)` is not a standalone public entrypoint; it must run on top of a store
-- Shutdown order is fixed: close the MQ handle first, then close `store`
+Each message sent by Producer is a `FlatDict`:
-## Minimal MQ Example
+```python
+{
+ "seq": 1,
+ "payload": b"hello mq #1",
+}
+```
+
+- `seq` is a process-local sequence field defined by the example.
+- `payload` is an application-data field defined by the example.
+- `b"..."` is a Python bytes literal.
-The public minimal example is `examples/start_mpmc_demo.py`.
+These field names are not fixed by MQ and may be replaced with other application-specific `FlatDict` fields.
-Run one `Producer` and one `Consumer` after the service plane is ready:
+#### Start Consumer and Producer
+
+Start Consumer in terminal one:
```bash
-python3 examples/start_mpmc_demo.py --role producer
python3 examples/start_mpmc_demo.py --role consumer
```
-This example keeps one process-local `seq` counter. Restarting the `Producer` resets that counter; it is not a cross-process persistent sequence.
+After Consumer binds successfully, its log contains:
-The most important part of the example is the ownership chain:
-
-- one `External Client`-side `KvClient`
-- one bound `Producer` or `Consumer` handle on top of that store
-- `Ctrl-C` only requests shutdown and closes the handle once
-- `ProducerClosedError` and `ChannelClosedError` are normal close-path signals
+```text
+[consumer] ready: channel_key=demo_mq_channel_doc
+```
-Common startup form:
+Start Producer in terminal two:
```bash
-FLUXON_LOG=INFO python3 examples/start_mpmc_demo.py --role producer
-FLUXON_LOG=DEBUG python3 examples/start_mpmc_demo.py --role consumer
+python3 examples/start_mpmc_demo.py --role producer
+```
+
+After both processes are running, their logs contain messages like:
+
+```text
+[producer] ready: channel_key=demo_mq_channel_doc
+[producer] sent: seq=1 payload=hello mq #1
+[consumer] got: seq=1 payload=hello mq #1
+```
+
+The minimum path is working when Producer continues to send and Consumer receives the matching sequence and payload. Press Ctrl-C in each terminal to stop both processes.
+
+`seq` is stored only in the current Producer process. Restarting Producer resets it to `1`; it is not persisted across processes.
+
+
+📄 View full script (click to expand) | examples/start_mpmc_demo.py
+
+```python
+#!/usr/bin/env python3
+
+import argparse
+import threading
+from pathlib import Path
+
+from fluxon_py.api_ext_chan import ( # type: ignore
+ ChanRole,
+ ChanType,
+ new_or_bind_with_unique_key,
+)
+from fluxon_py.api_error import ChannelClosedError, ProducerClosedError # type: ignore
+from fluxon_py.config import FluxonKvClientConfig # type: ignore
+from fluxon_py.kvclient import new_store # type: ignore
+from fluxon_py.logging import init_logger # type: ignore
+from fluxon_py.runtime import register_ctrlc_callback
+
+# These constants are the only user-facing knobs in the minimal example.
+CLUSTER_NAME = "demo-kv-cluster"
+SHARE_MEM_PATH = "/dev/shm/fluxon_kv_demo"
+CHANNEL_KEY = "demo_mq_channel_doc"
+CHANNEL_CAPACITY = 128
+CHANNEL_TTL_SECONDS = 300
+PRODUCER_INTERVAL_SECONDS = 1.0
+CONSUMER_BATCH_SIZE = 1
+
+
+def _must_ok(res, msg: str):
+ if not res.is_ok():
+ raise SystemExit(f"{msg}: {res.unwrap_error()}")
+ return res.unwrap()
+
+
+def _best_effort_close_result(obj, logger, role: str) -> None:
+ try:
+ close_res = obj.close()
+ except Exception as e: # noqa: BLE001
+ logger.warning(f"[{role}] close raised (ignored): {e}")
+ return
+
+ if close_res.is_ok():
+ _ = close_res.unwrap()
+ else:
+ logger.warning(f"[{role}] close error (ignored): {close_res.unwrap_error()}")
+
+
+def _build_store_config(*, role: str) -> FluxonKvClientConfig:
+ # MQ first attaches to the local owner via one external KvClient,
+ # then binds a producer or consumer handle on top of that store.
+ return FluxonKvClientConfig(
+ {
+ "instance_key": f"demo_mq_{role}",
+ "fluxonkv_spec": {
+ "cluster_name": CLUSTER_NAME,
+ "share_mem_path": SHARE_MEM_PATH,
+ },
+ }
+ )
+
+
+def _run_producer(store, logger, shutdown_requested: threading.Event) -> None:
+ interrupted = False
+ closed = False
+ producer = None
+ restore_signal_listener = lambda: None
+ seq = 1
+ try:
+ # Producer and consumer must bind the same channel key so they land on
+ # the same channel id.
+ producer = _must_ok(
+ new_or_bind_with_unique_key(
+ store,
+ {"capacity": CHANNEL_CAPACITY, "ttl_seconds": CHANNEL_TTL_SECONDS},
+ unique_id=CHANNEL_KEY,
+ chan_type=ChanType.MPMC,
+ chan_role=ChanRole.PRODUCER,
+ ),
+ "bind producer failed",
+ )
+
+ def _on_ctrlc(reason: str) -> None:
+ nonlocal interrupted, closed
+ # The signal callback only requests shutdown and closes the handle once.
+ # The main loop still exits through its normal close-observation path.
+ interrupted = True
+ shutdown_requested.set()
+ if closed:
+ return
+ closed = True
+ logger.info(f"[producer] caught {reason}, calling close...")
+ _best_effort_close_result(producer, logger, "producer")
+
+ restore_signal_listener = register_ctrlc_callback(
+ _on_ctrlc,
+ thread_name="mpmc-demo-producer-signal",
+ )
+ logger.info(f"[producer] ready: channel_key={CHANNEL_KEY}")
+ while not shutdown_requested.is_set():
+ payload_text = f"hello mq #{seq}"
+ payload = payload_text.encode("utf-8")
+ put_res = producer.put_data(
+ {
+ "seq": seq,
+ "payload": payload,
+ }
+ )
+ if put_res.is_ok():
+ _ = put_res.unwrap()
+ logger.info(f"[producer] sent: seq={seq} payload={payload_text}")
+ seq += 1
+ else:
+ err = put_res.unwrap_error()
+ # ProducerClosedError is the expected signal that close() already
+ # propagated into the handle, not an unexpected data-path failure.
+ if isinstance(err, ProducerClosedError):
+ logger.info("[producer] close observed, exit loop")
+ break
+ raise SystemExit(f"put_data failed: {err}")
+ if shutdown_requested.wait(PRODUCER_INTERVAL_SECONDS):
+ break
+ finally:
+ restore_signal_listener()
+ # Handle lifetime must end before store lifetime.
+ if producer is not None and not closed:
+ _best_effort_close_result(producer, logger, "producer")
+ if interrupted:
+ raise SystemExit(130)
+
+
+def _run_consumer(store, logger, shutdown_requested: threading.Event) -> None:
+ interrupted = False
+ closed = False
+ consumer = None
+ restore_signal_listener = lambda: None
+ try:
+ # Consumer binds the same channel key as producer and only changes role.
+ consumer = _must_ok(
+ new_or_bind_with_unique_key(
+ store,
+ {"capacity": CHANNEL_CAPACITY, "ttl_seconds": CHANNEL_TTL_SECONDS},
+ unique_id=CHANNEL_KEY,
+ chan_type=ChanType.MPMC,
+ chan_role=ChanRole.CONSUMER,
+ ),
+ "bind consumer failed",
+ )
+
+ def _on_ctrlc(reason: str) -> None:
+ nonlocal interrupted, closed
+ # Keep the callback minimal: request shutdown, close the MQ handle once,
+ # and let the main loop observe ChannelClosedError.
+ interrupted = True
+ shutdown_requested.set()
+ if closed:
+ return
+ closed = True
+ logger.info(f"[consumer] caught {reason}, calling close...")
+ _best_effort_close_result(consumer, logger, "consumer")
+
+ restore_signal_listener = register_ctrlc_callback(
+ _on_ctrlc,
+ thread_name="mpmc-demo-consumer-signal",
+ )
+ logger.info(f"[consumer] ready: channel_key={CHANNEL_KEY}")
+ while not shutdown_requested.is_set():
+ get_res = consumer.get_data(batch_size=CONSUMER_BATCH_SIZE)
+ if not get_res.is_ok():
+ err = get_res.unwrap_error()
+ # ChannelClosedError is the normal close path after Ctrl-C/SIGTERM.
+ if isinstance(err, ChannelClosedError):
+ logger.info("[consumer] close observed, exit loop")
+ break
+ raise SystemExit(f"get_data failed: {err}")
+ for item in get_res.unwrap() or []:
+ payload = item.get("payload", b"") if isinstance(item, dict) else item
+ seq = item.get("seq") if isinstance(item, dict) else None
+ if isinstance(payload, (bytes, bytearray, memoryview)):
+ logger.info(
+ f"[consumer] got: seq={seq} payload={bytes(payload).decode('utf-8', 'ignore')}"
+ )
+ else:
+ logger.info(f"[consumer] got: seq={seq} payload={payload}")
+ if shutdown_requested.wait(0.2):
+ break
+ finally:
+ restore_signal_listener()
+ # Always close the consumer before main() closes the backing store.
+ if consumer is not None and not closed:
+ _best_effort_close_result(consumer, logger, "consumer")
+ if interrupted:
+ raise SystemExit(130)
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Start MQ minimal demo")
+ parser.add_argument("--role", choices=["producer", "consumer"], required=True)
+ args = parser.parse_args()
+
+ # init_logger() reads FLUXON_LOG and sets the user-process console log level.
+ logger = init_logger(f"mpmc_demo_{args.role}")
+ shutdown_requested = threading.Event()
+ store = None
+ try:
+ store = _must_ok(new_store(_build_store_config(role=args.role)), "new_store failed")
+ if args.role == "producer":
+ _run_producer(store, logger, shutdown_requested)
+ else:
+ _run_consumer(store, logger, shutdown_requested)
+ finally:
+ store_to_close = store
+ store = None
+ # Store is closed last because MQ handles are already closed inside _run_*.
+ if store_to_close is not None:
+ _best_effort_close_result(store_to_close, logger, "store")
+ logger.info(f"[{args.role}] exit")
+
+
+if __name__ == "__main__":
+ main()
```
-## Common Interfaces
+
-- `new_or_bind_with_unique_key(api, chan_config, unique_id, chan_type, chan_role)`: bind if present, otherwise create
-- `producer.put_data(value: FlatDict) -> Result[bool, ApiError]`: send one message
-- `consumer.get_data(batch_size: int = 1, try_time: Optional[int] = None, prefetch_num: int = 0) -> Result[List[Any], ApiError]`: fetch messages in batches
-- `producer.get_chan_id()` / `consumer.get_chan_id()`: current channel id
-- `producer.get_producer_id()` / `consumer.get_consumer_id()`: current member id
-- `close() -> Result[OkNone, ApiError]`: close the current MQ handle
+The full script also handles Ctrl-C and concurrency during shutdown. For a first use, remember only that the Producer or Consumer handle closes before the underlying `KvClient`.
-Parameter constraints:
+### Common APIs
-- `chan_type` is commonly `ChanType.MPMC`, and `ChanType.MPSC` is also supported
-- `chan_role` must be either `ChanRole.PRODUCER` or `ChanRole.CONSUMER`
-- `try_time` is a second-level wait bound
+#### Create or Bind a Channel
-### Common Error Handling
+```python
+new_or_bind_with_unique_key(
+ api: KvClient,
+ chan_config: Dict[str, int],
+ unique_id: str,
+ chan_type: ChanType,
+ chan_role: ChanRole,
+) -> Result[
+ Union[MPSCChanProducer, MPSCChanConsumer, MPMCChanProducer, MPMCChanConsumer],
+ ApiError,
+]
+```
+
+- `api` is the `KvClient` created by `new_store(...)`.
+- `chan_config` contains the channel settings.
+- `unique_id` is the stable channel key. The same key identifies the same logical channel.
+- `chan_type` selects `ChanType.MPMC` or `ChanType.MPSC`.
+- `chan_role` selects `ChanRole.PRODUCER` or `ChanRole.CONSUMER`.
+
+If creation or binding fails, print `unwrap_error()` and check `cluster_name`, `share_mem_path`, `unique_id`, channel type, and channel config first. Producer and Consumer naturally use different `chan_role` values; their roles do not need to match.
+
+#### Send a Message
+
+```python
+producer.put_data(value: FlatDict) -> Result[bool, ApiError]
+```
+
+`value` is a single-level `FlatDict`. A successful result means that the message has been submitted to the channel. Consume `unwrap_error()` when the call fails.
+
+#### Receive Messages
+
+```python
+consumer.get_data(
+ batch_size: int = 1,
+ try_time: Optional[int] = None,
+ prefetch_num: int = 0,
+) -> Result[List[Any], ApiError]
+```
+
+- `batch_size`: Maximum number of messages requested by this call. It must be a positive integer.
+- `try_time`: Maximum wait in seconds for blocking paths. Set it to `0` for a non-blocking attempt.
+- `prefetch_num`: Additional prefetch-window size. Keep `0` for the introductory example.
+
+A successful result is a message list; each item in this page's MPMC example is a `FlatDict`. The list may be empty when no message is available, so an application loop should continue to the next read.
+
+#### Inspect Identity and Close
+
+- `producer.get_chan_id()` / `consumer.get_chan_id()`: Return the channel ID bound to the current handle.
+- `producer.get_producer_id()`: Return the Producer member ID.
+- `consumer.get_consumer_id()`: Return the Consumer member ID.
+- `close() -> Result[OkNone, ApiError]`: Close the current MQ handle.
+
+### Advanced Notes
+
+#### Channel Types and Configuration
-- `new_or_bind_with_unique_key(...)` fails: first check `cluster_name`, `share_mem_path`, `unique_id`, and that both ends use matching roles
-- `producer.put_data(...)` returns `ProducerClosedError`: treat it as a normal shutdown signal and exit the main loop
-- `consumer.get_data(...)` returns `ChannelClosedError`: treat it as a normal shutdown signal and exit the main loop
+- `ChanType.MPMC` supports multiple Producers and multiple Consumers. Both `capacity` and `ttl_seconds` are required.
+- `ChanType.MPSC` supports multiple Producers and one Consumer. `ttl_seconds` is required and `capacity` is optional.
+- `capacity` must be a positive integer and limits in-flight messages. Producer waits for available channel space after the limit is reached.
+- `ttl_seconds` must be an integer of at least `90` and is used by channel and member leases.
+- Processes binding the same `CHANNEL_KEY` should use the same channel type, `capacity`, and `ttl_seconds`.
+
+Two local example settings are not part of the channel config:
+
+- `PRODUCER_INTERVAL_SECONDS` controls only the delay between Producer sends and must be non-negative.
+- `CONSUMER_BATCH_SIZE` controls the number of messages requested by each `get_data(...)` call and must be positive.
+
+#### Lifecycle and Error Handling
+
+During Ctrl-C shutdown, the send or receive loop may observe:
+
+- `ProducerClosedError`: The Producer handle is closed; exit the send loop normally.
+- `ChannelClosedError`: The Consumer handle is closed; exit the receive loop normally.
+
+These errors are expected during an explicit shutdown and do not indicate a new data-plane failure. For other errors, print `unwrap_error()` and stop the current loop.
+
+Always shut down in this order:
+
+1. Ask the application loop to stop.
+2. Close the Producer or Consumer handle.
+3. Wait for the loop to exit.
+4. Close `KvClient` last.
+
+#### Logging
+
+- Python MQ logs are initialized by `init_logger(...)` and go to the current terminal by default.
+- `FLUXON_LOG` controls the log level. Valid values are `DEBUG`, `INFO`, `WARNING`, `ERROR`, and `CRITICAL`; the default is `INFO`.
+- Rust MQ and KV background logs use the service-plane log directories described in User 2.
+- When Master configures `monitoring.otlp_log_api`, background logs are also written to the Greptime `fluxon_logs` table.
+
+To change the example log level:
+
+```bash
+FLUXON_LOG=INFO python3 examples/start_mpmc_demo.py --role producer
+FLUXON_LOG=DEBUG python3 examples/start_mpmc_demo.py --role consumer
+```
-## Log Paths
+### Web Monitoring
-- Python-side MQ logs come from `init_logger(...)` and go to the current terminal by default; the threshold is controlled by `FLUXON_LOG`
-- Rust / KV background logs follow the shared service-plane pipeline, and the `Master`'s local log authority is `master_cfg["log_dir"]`
-- `share_mem_path` is the shared bundle root for `mmap.file`, `shared.json`, and peer metadata
-- `large_file_paths` is the `Owner Client`-only large-file authority for backend logs, profiles, caches, and other derived runtime assets
+First enable the Master Web UI through “Optional: KV Web UI” in [User - 3 - KV and RPC Interface](<./User - 3 - KV and RPC Interface.md>). On the target cluster page, the two main MQ tables are:
-If `master.monitoring.otlp_log_api` is configured, backend logs continue to flow into the Greptime `fluxon_logs` table.
+- `Channels`: Channel summary.
+- `Members`: Individual Producer and Consumer details.
-## Web Monitoring
+#### 1. Check for Backlog
-Two UI tables are especially useful:
+In `Channels`, sort by `current_inflight` in descending order:
-- `Channels`: channel-level summary
-- `Members`: individual `Producer` / `Consumer` detail
+- Near `0`: The channel is mostly caught up.
+- Continually increasing: Messages are arriving faster than they are consumed; inspect individual Producers next.
-### Channel Summary
+#### 2. Inspect Individual Producers
-`producer_offsets` is rendered as:
+`producer_offsets` has this format:
```text
producer_idx: produce_offset/consume_offset
```
-Example:
+For example:
```text
producer_1: 101/88, producer_2: 57/57
```
-Both offsets mean "the next offset":
+Both offsets describe the next position:
-- `produce_offset`: next message offset the `Producer` will write
-- `consume_offset`: next offset the `Consumer` will commit
+- `produce_offset`: The next position Producer will write.
+- `consume_offset`: The next position Consumer will commit.
-Current backlog per `Producer` is:
+The unconsumed count for one Producer is:
```text
max(produce_offset - consume_offset, 0)
```
-`current_inflight` is the sum across producers in the same channel.
+`current_inflight` is the sum of unconsumed counts across Producers in one channel. A growing gap for one Producer identifies a localized backlog. Clear gaps across several Producers indicate that the channel as a whole is consuming too slowly.
-### Member Detail
+#### 3. Locate Members
-Useful fields in `Members`:
+In `Members`:
-- `channel_unique_keys`
-- `produce_offset` / `consume_offset`
-- `chan_id`, `owner_id`, `external_client_id`
+1. Search `channel_unique_keys` with the `CHANNEL_KEY` passed by Python.
+2. Inspect `chan_id`, `owner_id`, and `external_client_id` on the matching row.
+3. Continue with the offset and consumer-latency fields.
-When you attach from Python through `new_or_bind_with_unique_key(...)`, a common lookup flow is:
+`channel_unique_keys` is a channel-level key, not the member ID of one Producer or Consumer. To inspect handle IDs in Python, call `get_producer_id()` or `get_consumer_id()`.
-1. search `channel_unique_keys`
-2. inspect `chan_id`, `owner_id`, `external_client_id`, offsets, and latency-related fields
+Both `Channels` and `Members` support field filters and multi-level sorting. Use `current_inflight` to find the largest backlog and `channel_unique_keys` to locate an application channel.
-## Latency Triage
+### Latency Triage
-MQ prints consumer-latency statistics roughly every 30 seconds. Search logs with these keywords:
+MQ prints consumer-latency statistics roughly every 30 seconds. When the basic path works but consumption is slow, search these keywords:
-| Keyword | Layer | Meaning |
+| Keyword | Observation point | Meaning |
|---|---|---|
-| `py-get latency` | Python caller | total `get_data()` latency |
-| `get_one breakdown` | PyO3 bridge | breakdown of cross-language wait time |
-| `MpscConsumer prefetch` | Rust MQ layer | prefetch-queue and single-task cost |
+| `py-get latency` | Python caller | Total `get_data()` latency |
+| `get_one breakdown` | Python/Rust boundary | Breakdown of cross-language wait time |
+| `MpscConsumer prefetch` | Rust MQ | Prefetch-queue and per-task latency |
-Quick reading:
+Quick interpretation:
-- high `py-get` total latency -> inspect PyO3-side `avg_wait_rx_ms`
-- high Rust `avg_get_handle_ms` -> prefetch queue is empty, or the `Producer` side is idle, or the window is too small
-- high Rust `avg_handle_await_ms` -> the single task is slow, for example `kv_get` or etcd commit
+- When total `py-get` latency is high, check whether `avg_wait_rx_ms` is also high.
+- High `avg_get_handle_ms` may indicate an empty prefetch queue, often because Producer has no data or the prefetch window is too small.
+- High `avg_handle_await_ms` indicates a slow individual task, such as a slow KV read or etcd commit.
diff --git a/fluxon_doc_en/user_doc/User - 5 - FS Interface.md b/fluxon_doc_en/user_doc/User - 5 - FS Interface.md
index 8bbc66f..42c60f0 100644
--- a/fluxon_doc_en/user_doc/User - 5 - FS Interface.md
+++ b/fluxon_doc_en/user_doc/User - 5 - FS Interface.md
@@ -1,150 +1,583 @@
# User - 5 - FS Interface
-## Overview
+## FS Interface
-Fluxon FS lets you mount a remote export into the current Python process and keep using `open() / read() / write()` semantics.
+Fluxon FS mounts a directory from a remote machine into the current Python process. After the mount is installed, application code continues to use ordinary `open()`, `read()`, and `write()` calls.
-The core objects are:
+### Four Names to Understand First
-- KV service-plane objects: `etcd`, `Greptime`, `Master`, `Owner Client`
-- FS role objects: `FS Master`, `FS Agent`
-- In-process mount objects: `FluxonKvClientConfig`, `new_store(...)`, `FluxonFsPatcher`, `mount_remote_dir(...)`
+- `export`: A directory exposed by FS Agent.
+- `EXPORT_NAME`: The logical name of that export.
+- `REMOTE_ROOT_DIR`: The actual directory on the FS Agent machine.
+- `mount_dir_abs`: The local mount directory in the Reader process.
+
+Their relationship is:
```text
-etcd + Greptime + Fluxon KV Master + Owner Client
- |
- v
- FS Master
- |
- v
- FS Agent
- |
- v
-FluxonKvClientConfig -> new_store(...) -> KvClient(store)
- |
- v
- FluxonFsPatcher(store)
- |
- +-- set_master_config_yaml(...)
- +-- set_cache_config_yaml(...)
- +-- set_request_identity(...)
- +-- install()
- +-- mount_remote_dir(...)
- |
- v
- open() / read() / write() / close()
+FS Agent exposes REMOTE_ROOT_DIR
+ ↓
+FS Master publishes EXPORT_NAME
+ ↓
+Reader mounts the export at mount_dir_abs
+ ↓
+ open() / read() / write()
```
-See [Architecture and Concepts](<./User - 1 - Architecture and Concepts.md>) for the role model and [User - 3 - KV and RPC Interface](<./User - 3 - KV and RPC Interface.md>) for `FluxonKvClientConfig` and `new_store(...)`.
+FS Master manages export configuration, access control, and the management UI. FS Agent provides remote directory access. `FluxonFsPatcher` is installed in the Reader process and forwards operations under the mount directory to the corresponding FS Agent.
+
+### Checks Before Starting
+
+Confirm the following before running the examples:
+
+- Greptime and etcd have been started as described in [User - 2 - Service Plane](<./User - 2 - Service Plane.md>).
+- The current Python environment has installed `fluxon-*.whl` and `fluxon_pyo3-*.whl`; see [User - 0 - Installation](<./User - 0 - Installation.md>).
+- The local example ports are free, and both `WORKDIR` and `REMOTE_ROOT_DIR` are writable.
+- The `KvClient` basics are understood; see [User - 3 - KV and RPC Interface](<./User - 3 - KV and RPC Interface.md>).
-## Service Plane
+Ordinary mounting and file I/O do not use directory-transfer state. PD and TiKV are needed only for `/ui/transfers/` or pre-scan.
-FS depends on the KV service plane and then adds two more roles on top:
+### Choose the Scenario First
-1. `Greptime`
-2. `etcd`
-3. `Fluxon KV Master`
-4. `Owner Client`
-5. `FS Master`
-6. `FS Agent`
-7. your mount verification script
+- **Complete local example**: Start a local KV Master, Owner Client, FS Master, and FS Agent. This is the recommended first-run path.
+- **Attach to an existing cluster**: Start only a new local Owner Client and FS Agent, then attach to the existing KV Master and FS Master.
+- **Directory transfer or pre-scan**: Prepare PD and TiKV after the basic FS services are working. This flow appears at the end of the page.
-`examples/start_kv_and_fs_svc.py` only starts Fluxon-native roles. `etcd` and `Greptime` still follow [User - 2 - Service Plane](<./User - 2 - Service Plane.md>). If you need `/ui/transfers/` and pre-scan, start the TiKV PD / TiKV pair for `transfer_state_store` first.
+### Start the Local FS Services
+
+By default, `examples/start_kv_and_fs_svc.py` starts:
+
+```text
+KV Master → Owner Client → FS Master → FS Agent
+```
-## `FS Master` and `FS Agent`
+#### Values to Check on the First Run
-After the KV service plane is ready, FS adds two roles:
+- `CLUSTER_NAME`: Cluster name shared by KV and FS.
+- `SHARE_MEM_PATH`: Shared-memory directory used by the local Owner Client and every local FS process.
+- `FS_MASTER_INSTANCE_KEY`: FS Master identity and the target used by FS Agent and Reader when fetching configuration.
+- `EXPORT_NAME`: Export name published by FS Agent and mounted by Reader.
+- `REMOTE_ROOT_DIR`: Absolute directory exposed by the local FS Agent.
+- `ADMIN_USERNAME` / `ADMIN_PASSWORD`: Administrator credentials used when creating an empty access DB.
-- `FS Master`: attaches to the KV plane as an `External Client` and owns panel / export snapshot distribution
-- `FS Agent`: registers exports to `FS Master` and exposes remote directory access
+Other instance identities, ports, `WORKDIR`, and cache sizes can remain unchanged for a first local run. `admin/admin` is suitable only for a local demo and must be changed in a real environment.
-The reference script is `examples/start_kv_and_fs_svc.py`.
+#### Startup Command and Success Conditions
-Start it with:
+Run in a separate terminal:
```bash
python3 examples/start_kv_and_fs_svc.py
+```
+
+After a successful startup:
+
+- The terminal prints `cluster name`, `remote root dir`, `export name`, and log paths for all four roles.
+- The final terminal line includes `waiting for Ctrl-C to stop fs demo stack`, and the process remains running.
+- `/tmp/fluxon_fs_demo/remote_root` exists.
+- FS Panel is available at `http://127.0.0.1:34180`.
+- The four logs under `/tmp/fluxon_fs_demo/runtime/log/` contain no startup errors.
+
+The full script is included below:
+
+
+📄 View full script (click to expand) | examples/start_kv_and_fs_svc.py
+
+```python
+#!/usr/bin/env python3
+
+import argparse
+from pathlib import Path
+
+from fluxon_py.runtime import (
+ start_fs_agent_process,
+ start_fs_master_process,
+ start_kv_master_process,
+ start_owner_kvclient_process,
+ wait_subproc_or_ctrlc,
+)
+from fluxon_py.runtime.process_runner import ManagedSubprocess
+
+ETCD_ENDPOINT = "127.0.0.1:2379"
+GREPTIME_HTTP_PORT = 34030
+GREPTIME_BASE_URL = f"http://127.0.0.1:{GREPTIME_HTTP_PORT}"
+CLUSTER_NAME = "demo-fs-cluster"
+SHARE_MEM_PATH = Path("/dev/shm/fluxon_fs_demo").resolve()
+WORKDIR = Path("/tmp/fluxon_fs_demo/runtime").resolve()
+REMOTE_ROOT_DIR = Path("/tmp/fluxon_fs_demo/remote_root").resolve()
+KV_MASTER_PORT = 34100
+FS_PANEL_PORT = 34180
+FS_PANEL_LISTEN_ADDR = f"0.0.0.0:{FS_PANEL_PORT}"
+FS_PANEL_PUBLIC_BASE_URL = f"http://127.0.0.1:{FS_PANEL_PORT}"
+KV_MASTER_INSTANCE_KEY = "demo_fs_kv_master"
+OWNER_INSTANCE_KEY = "demo_fs_owner"
+FS_MASTER_INSTANCE_KEY = "demo_fs_master"
+FS_AGENT_INSTANCE_KEY = "demo_fs_agent"
+EXPORT_NAME = "demo-export"
+OWNER_DRAM_BYTES = 1073741824
+EXPORT_CACHE_MAX_BYTES = 1073741824
+ADMIN_USERNAME = "admin"
+ADMIN_PASSWORD = "admin"
+TRANSFER_STATE_STORE_PD_ENDPOINTS = ["127.0.0.1:12379"]
+TRANSFER_STATE_STORE_KEY_PREFIX = f"/fluxon_fs_transfer/{CLUSTER_NAME}/"
+FS_MASTER_ACCESS_DB_PATH = (WORKDIR / "fs_master" / "access.db").resolve()
+
+
+def build_owner_large_file_paths() -> list[str]:
+ return [str((WORKDIR / "large" / "owner").resolve())]
+
+
+def main() -> None:
+ args = parse_args()
+ WORKDIR.mkdir(parents=True, exist_ok=True)
+ REMOTE_ROOT_DIR.mkdir(parents=True, exist_ok=True)
+
+ log_dir = (WORKDIR / "log").resolve()
+ log_dir.mkdir(parents=True, exist_ok=True)
+
+ if args.with_master:
+ kv_master_log_dir = (WORKDIR / "kv_master_logs").resolve()
+ kv_master_log_dir.mkdir(parents=True, exist_ok=True)
+ kv_master_stdout_log = (log_dir / "kv_master.log").resolve()
+ # FS master persists panel auth state in this sqlite file, so the parent
+ # directory must exist before Rust opens access_db_path.
+ FS_MASTER_ACCESS_DB_PATH.parent.mkdir(parents=True, exist_ok=True)
+ fs_master_stdout_log = (log_dir / "fs_master.log").resolve()
+ # FS depends on the KV service plane, so bring up KV roles before FS roles.
+ kv_master_proc = start_kv_master_process(
+ config=build_kv_master_config(log_dir=kv_master_log_dir),
+ log_path=kv_master_stdout_log,
+ )
+ else:
+ kv_master_stdout_log = None
+ fs_master_stdout_log = None
+ kv_master_proc = None
+ fs_master_proc = None
+
+ owner_stdout_log = (log_dir / "owner.log").resolve()
+ owner_proc = start_owner_kvclient_process(
+ config=build_owner_config(),
+ log_path=owner_stdout_log,
+ )
+
+ if args.with_master:
+ fs_master_proc = start_fs_master_process(
+ config=build_fs_master_config(),
+ log_path=fs_master_stdout_log,
+ )
+
+ fs_agent_stdout_log = (log_dir / "fs_agent.log").resolve()
+ fs_agent_proc = start_fs_agent_process(
+ config=build_fs_agent_config(),
+ log_path=fs_agent_stdout_log,
+ )
+ children: list[ManagedSubprocess] = []
+ if kv_master_proc is not None:
+ children.append(
+ ManagedSubprocess(
+ label="kv_master",
+ proc=kv_master_proc,
+ )
+ )
+ children.append(
+ ManagedSubprocess(
+ label="owner",
+ proc=owner_proc,
+ )
+ )
+ if fs_master_proc is not None:
+ children.append(
+ ManagedSubprocess(
+ label="fs_master",
+ proc=fs_master_proc,
+ )
+ )
+ # Stop order is the reverse of this list, so append fs_agent last.
+ children.append(
+ ManagedSubprocess(
+ label="fs_agent",
+ proc=fs_agent_proc,
+ )
+ )
+
+ print(f"[fluxon_fs] cluster name: {CLUSTER_NAME}")
+ print(f"[fluxon_fs] share_mem_path: {SHARE_MEM_PATH}")
+ print(f"[fluxon_fs] remote root dir: {REMOTE_ROOT_DIR}")
+ print(f"[fluxon_fs] export name: {EXPORT_NAME}")
+ print(f"[fluxon_fs] owner instance key: {OWNER_INSTANCE_KEY}")
+ print(f"[fluxon_fs] fs master instance key: {FS_MASTER_INSTANCE_KEY}")
+ print(f"[fluxon_fs] fs agent instance key: {FS_AGENT_INSTANCE_KEY}")
+ print(f"[fluxon_fs] start masters in this script: {args.with_master}")
+ if args.with_master:
+ print(f"[fluxon_fs] panel listen addr: {FS_PANEL_LISTEN_ADDR}")
+ print(f"[fluxon_fs] panel public base url: {FS_PANEL_PUBLIC_BASE_URL}")
+ print(f"[fluxon_fs] transfer state store pd_endpoints: {TRANSFER_STATE_STORE_PD_ENDPOINTS}")
+ print(f"[fluxon_fs] transfer state store key_prefix: {TRANSFER_STATE_STORE_KEY_PREFIX}")
+ print(f"[fluxon_fs] bootstrap admin username: {ADMIN_USERNAME}")
+ print(f"[fluxon_fs] bootstrap admin password: {ADMIN_PASSWORD}")
+ print(f"[fluxon_fs] kv master stdout log: {kv_master_stdout_log}")
+ print(f"[fluxon_fs] fs master stdout log: {fs_master_stdout_log}")
+ else:
+ print("[fluxon_fs] panel listen addr: disabled by --without-master")
+ print("[fluxon_fs] panel public base url: disabled by --without-master")
+ print("[fluxon_fs] transfer state store pd_endpoints: disabled by --without-master")
+ print("[fluxon_fs] transfer state store key_prefix: disabled by --without-master")
+ print("[fluxon_fs] bootstrap admin username: disabled by --without-master")
+ print("[fluxon_fs] bootstrap admin password: disabled by --without-master")
+ print("[fluxon_fs] kv master stdout log: disabled by --without-master")
+ print("[fluxon_fs] fs master stdout log: disabled by --without-master")
+ print(f"[fluxon_fs] owner stdout log: {owner_stdout_log}")
+ print(f"[fluxon_fs] fs agent stdout log: {fs_agent_stdout_log}")
+ stack_label = "fs demo stack" if args.with_master else "owner and fs agent"
+ print(f"[fluxon_fs] waiting for Ctrl-C to stop {stack_label}")
+ wait_subproc_or_ctrlc(
+ children,
+ on_ctrlc=lambda: print(f"[fluxon_fs] caught Ctrl-C, stopping {stack_label}"),
+ )
+
+
+def parse_args() -> argparse.Namespace:
+ parser = argparse.ArgumentParser(description="Start FS demo roles, optionally with local masters")
+ group = parser.add_mutually_exclusive_group()
+ group.add_argument(
+ "--with-master",
+ dest="with_master",
+ action="store_true",
+ help="Start local kv master and fs master in this script (default)",
+ )
+ group.add_argument(
+ "--without-master",
+ dest="with_master",
+ action="store_false",
+ help="Do not start any master in this script; only start owner and fs_agent and attach to an existing cluster",
+ )
+ parser.set_defaults(with_master=True)
+ return parser.parse_args()
+
+
+def build_kv_master_config(*, log_dir: Path) -> dict:
+ return {
+ "instance_key": KV_MASTER_INSTANCE_KEY,
+ "cluster_name": CLUSTER_NAME,
+ "port": KV_MASTER_PORT,
+ "etcd_endpoints": [ETCD_ENDPOINT],
+ "log_dir": str(log_dir),
+ "monitoring": {
+ "prometheus_base_url": f"{GREPTIME_BASE_URL}/v1/prometheus",
+ "prom_remote_write_url": [f"{GREPTIME_BASE_URL}/v1/prometheus/write"],
+ "otlp_log_api": {
+ "otlp_endpoint": f"{GREPTIME_BASE_URL}/v1/otlp/v1/logs",
+ },
+ },
+ }
+
+
+def build_owner_config() -> dict:
+ return {
+ "instance_key": OWNER_INSTANCE_KEY,
+ "contribute_to_cluster_pool_size": {
+ "dram": OWNER_DRAM_BYTES,
+ "vram": {},
+ },
+ "fluxonkv_spec": {
+ "etcd_addresses": [ETCD_ENDPOINT],
+ "cluster_name": CLUSTER_NAME,
+ "share_mem_path": str(SHARE_MEM_PATH),
+ "sub_cluster": "default",
+ "large_file_paths": build_owner_large_file_paths(),
+ },
+ }
+
+
+def build_fs_master_config() -> dict:
+ return {
+ "kvclient": {
+ "instance_key": FS_MASTER_INSTANCE_KEY,
+ "fluxonkv_spec": {
+ "cluster_name": CLUSTER_NAME,
+ "share_mem_path": str(SHARE_MEM_PATH),
+ },
+ },
+ "fluxon_fs": {
+ "master": {
+ "instance_key": FS_MASTER_INSTANCE_KEY,
+ "pull_interval_ms": 1000,
+ },
+ "master_panel": {
+ "listen_addr": FS_PANEL_LISTEN_ADDR,
+ "public_base_url": FS_PANEL_PUBLIC_BASE_URL,
+ "auto_refresh_interval_secs": 2,
+ "access_db_path": str(FS_MASTER_ACCESS_DB_PATH),
+ # bootstrap_access_model only seeds an empty access_db; once the DB has users,
+ # later restarts keep using the DB state instead of overwriting it from config.
+ # Manager users keep full export access through runtime auth checks, not by writing
+ # synthetic root scopes into the DB.
+ "bootstrap_access_model": {
+ "users": [
+ {
+ "username": ADMIN_USERNAME,
+ "password": ADMIN_PASSWORD,
+ "can_manage_users": True,
+ }
+ ],
+ "scope_access": [],
+ },
+ "transfer_state_store": {
+ "kind": "tikv",
+ "tikv": {
+ "pd_endpoints": TRANSFER_STATE_STORE_PD_ENDPOINTS,
+ "key_prefix": TRANSFER_STATE_STORE_KEY_PREFIX,
+ },
+ },
+ "s3_gateway": {
+ "get_object_inflight_pieces": 8,
+ "kv_miss_policy": "remote_read",
+ },
+ },
+ "cache": {
+ "stale_window_ms": 1000,
+ "rules": [],
+ "exports": {
+ EXPORT_NAME: {
+ "remote_root_dir_abs": str(REMOTE_ROOT_DIR),
+ "cache_max_bytes": EXPORT_CACHE_MAX_BYTES,
+ },
+ },
+ },
+ },
+ }
+
+
+def build_fs_agent_config() -> dict:
+ return {
+ "kvclient": {
+ "instance_key": FS_AGENT_INSTANCE_KEY,
+ "fluxonkv_spec": {
+ "cluster_name": CLUSTER_NAME,
+ "share_mem_path": str(SHARE_MEM_PATH),
+ },
+ },
+ "fluxon_fs": {
+ "master": {
+ # The agent follows this master instance key to pull the current export snapshot.
+ "instance_key": FS_MASTER_INSTANCE_KEY,
+ },
+ "cache": {
+ "stale_window_ms": 1000,
+ "rules": [],
+ "exports": {
+ EXPORT_NAME: {
+ "remote_root_dir_abs": str(REMOTE_ROOT_DIR),
+ "cache_max_bytes": EXPORT_CACHE_MAX_BYTES,
+ },
+ },
+ },
+ },
+ }
+
+
+if __name__ == "__main__":
+ main()
+```
+
+
+
+#### Attach to an Existing Cluster
+
+On a remote Agent machine, run:
+
+```bash
python3 examples/start_kv_and_fs_svc.py --without-master
```
-Default mode starts `KV Master + Owner Client + FS Master + FS Agent`. `--without-master` only starts `Owner Client + FS Agent` and expects the cluster's `KV Master` and `FS Master` to already exist elsewhere.
+Before starting, adjust:
-Most important remote-agent constraints:
+- `ETCD_ENDPOINT` and `CLUSTER_NAME`: Point to the existing cluster.
+- `FS_MASTER_INSTANCE_KEY`: Match the existing FS Master exactly.
+- `SHARE_MEM_PATH`: Match the new Owner Client on the current machine.
+- `OWNER_INSTANCE_KEY` and `FS_AGENT_INSTANCE_KEY`: Assign new values unique within the cluster.
+- `EXPORT_NAME`: Assign a distinct name for this remote directory.
+- `REMOTE_ROOT_DIR`: Use an absolute path on the current Agent machine.
-- `ETCD_ENDPOINT` must point at the real cluster etcd endpoint
-- `FS_MASTER_INSTANCE_KEY` must match the existing `FS Master`
-- `OWNER_INSTANCE_KEY`, `FS_AGENT_INSTANCE_KEY`, `EXPORT_NAME`, and `REMOTE_ROOT_DIR` must be unique per agent machine
-- `FS_PANEL_PUBLIC_BASE_URL` controls external links shown by the UI, while `FS_PANEL_LISTEN_ADDR` only controls the bind address
+This mode manages only the local Owner Client and FS Agent. It does not start KV Master, FS Master, or Panel.
-Default mode collects subprocess `stdout/stderr` into `WORKDIR/log` and keeps only summary output in the terminal.
+### Remote Mount Read / Write Verification
-## Remote Mount Read / Write Verification
+The verification flow uses three scripts:
-The public FS verification flow under `examples/` is:
+- `start_kv_and_fs_svc.py`: Start the service roles.
+- `start_fluxon_fs_writer.py`: Register the export and continuously write a remote file and a local comparison file.
+- `start_fluxon_fs_reader.py`: Mount the export and continuously read the remote and local comparison files.
-- `examples/start_kv_and_fs_svc.py`
-- `examples/start_fluxon_fs_writer.py`
-- `examples/start_fluxon_fs_reader.py`
+#### Configuration Relationships
-Minimum success path:
+Writer and Reader both require:
-1. run `python3 examples/start_kv_and_fs_svc.py`
-2. keep it running
-3. prepare writer config and run `python3 examples/start_fluxon_fs_writer.py -c -w `
-4. prepare reader config and run `python3 examples/start_fluxon_fs_reader.py -c -w `
+- `-c/--config` pointing to a prepared environment YAML.
+- `-w/--workdir` pointing to a writable directory owned by the current process.
+- A `kvclient.instance_key` unique within the cluster.
+- A `cluster_name` matching the FS services.
+- A `share_mem_path` matching the Owner Client on the same machine.
-The reader side always does three things:
+In addition:
-- attach to the local `Owner Client` through one `External Client`
-- install the patcher through `install_patcher_from_master(...)`
-- mount the selected export and alternate between remote and local reads
+- The export name and remote root in Writer config must describe the target FS Agent.
+- Reader `fluxon_fs.master.instance_key` must equal the target `FS_MASTER_INSTANCE_KEY`.
+- Reader `export_name` must match the export registered by Writer and published by FS Master.
+- Writer and Reader must use the same remote relative path.
-Once the reader keeps printing `op=read_remote` / `op=read_local`, the remote mount chain and local cache rules are both working.
+This page does not embed complete Writer or Reader YAML. `-c` should point to configuration already validated for the current deployment environment.
-## Directory Transfer and Pre-Scan
+#### Reader Mount Directory Requirements
-Directory transfer and pre-scan are designed for long-running large-folder jobs such as cross-cluster migration or migration across shared-storage domains.
+`mount_dir_abs` must:
-The main user-facing UI is `/ui/transfers/`, which exposes:
+- Be an absolute path.
+- Not be `/`.
+- Be created by Fluxon if it does not exist.
+- Be empty if it already exists.
+- Not overlap another mount directory in the same process.
-- `Pre-Scans`
-- `FluxonFS Transfer Jobs`
+The mount does not need to live below `/fluxon_fs/`; `/tmp/fluxon_fs_demo/mount_demo` is also valid.
-Typical direct-transfer flow from the UI:
+#### Start Writer
-1. open two panes
-2. locate the source folder on the left
-3. locate the target export and target directory on the right
-4. drag the folder across panes
-5. fill `desired_worker_count` and `batch_ready_bytes`
-6. submit and inspect the job in `/ui/transfers/`
+Keep the FS services running and execute in terminal two:
-Typical pre-scan import flow:
+```bash
+python3 examples/start_fluxon_fs_writer.py \
+ -c \
+ -w
+```
-1. open `/ui/transfers/`
-2. find the record in `Pre-Scans`
-3. click `Import`
-4. choose `source export`
-5. choose `target export`
-6. fill `target prefix`
-7. fill `desired_worker_count`
-8. submit
+After Writer starts successfully, it continually prints:
-### TiKV Config for Transfer State
+```text
+[writer] op=write_remote ...
+[writer] op=write_local ...
+```
-Directory transfer and pre-scan both depend on `transfer_state_store`. The `FS Master` panel and any standalone pre-scan process must share the same TiKV namespace.
+#### Start Reader
-The most important fields are:
+Execute in terminal three:
-- `pd_endpoints`
-- `key_prefix`
+```bash
+python3 examples/start_fluxon_fs_reader.py \
+ -c \
+ -w
+```
+
+After Reader mounts the export and finds both files, it continually prints:
+
+```text
+[reader] op=read_remote ...
+[reader] op=read_local ...
+```
+
+This confirms that the remote export mount and local cache rule are both active.
+
+> **Current example limitation**: `start_fluxon_fs_reader.py` does not yet call `set_request_identity(...)` and does not read a username or password from YAML. With an access-controlled FS Master, application code must set a request identity explicitly or remote file operations fail because the authentication token is missing.
+
+### `FluxonFsPatcher` Call Order
+
+The recommended Reader flow is:
+
+```text
+new_store(...)
+→ install_patcher_from_master(...)
+→ wait_cache_config_loaded()
+→ set_request_identity(...)
+→ mount_remote_dir(...)
+→ open() / read() / write()
+→ patcher.uninstall()
+→ store.close()
+```
+
+Key rules:
+
+- `FluxonFsPatcher` depends on the `KvClient` returned by `new_store(...)`.
+- `install_patcher_from_master(...)` fetches export configuration from FS Master and installs the Patcher.
+- `set_request_identity(username, password)` sets the identity for later FS requests.
+- Call `patcher.uninstall()` before closing `store`.
+
+To control configuration loading manually:
+
+- `load_cache_config_from_master_config_file(config_path)`: Read the FS Master identity from the file and block until configuration is fetched successfully.
+- `start_cache_config_fetch_from_master_config_file(config_path)`: Continuously fetch configuration from FS Master in the background. `install_patcher_from_master(...)` already calls this API.
+- `set_cache_config_yaml(...)`: Inject export and cache configuration directly when no FS Master fetch is used.
+
+Ordinary Readers should prefer `install_patcher_from_master(...)` instead of composing these APIs themselves.
+
+### Advanced Configuration
-The `start_kv_and_fs_svc.py` example uses:
+#### Panel and Access Control
-- `TRANSFER_STATE_STORE_PD_ENDPOINTS = ["127.0.0.1:12379"]`
-- `TRANSFER_STATE_STORE_KEY_PREFIX = "/fluxon_fs_transfer/demo-fs-cluster/"`
+- `FS_PANEL_LISTEN_ADDR` determines the local interface and port used by FS Master.
+- `FS_PANEL_PUBLIC_BASE_URL` is the public address used by browsers and generated links. For remote access, use a reachable hostname or IP address.
+- `FS_MASTER_ACCESS_DB_PATH` stores users, passwords, and permissions and should be a stable writable absolute path.
+- `bootstrap_access_model` seeds the first accounts only when the access DB has no users.
+- After users exist in the database, changing `ADMIN_USERNAME` or `ADMIN_PASSWORD` does not overwrite them.
+- An administrator with `can_manage_users: true` can access every current export at runtime.
+
+#### Logging
+
+The service script writes child-process standard output to:
+
+- `WORKDIR/log/kv_master.log`
+- `WORKDIR/log/owner.log`
+- `WORKDIR/log/fs_master.log`
+- `WORKDIR/log/fs_agent.log`
+
+Writer and Reader Python logs go to the current terminal by default. To increase logging:
+
+```bash
+FLUXON_LOG=DEBUG python3 examples/start_fluxon_fs_reader.py \
+ -c \
+ -w
+```
-`FS Master` needs:
+#### Common Errors
+
+- **`new_store failed`**: Check that the service script is still running and that `cluster_name` and `share_mem_path` match the local Owner Client.
+- **`unknown export_name`**: Check that Reader `export_name` was registered by FS Agent and published by FS Master.
+- **Invalid mount directory**: Check that the path is absolute and empty and does not overlap an existing mount.
+- **`fluxon_fs cache config is not loaded yet`**: Check the FS Master identity and confirm that configuration fetching has completed.
+- **`permission denied` / `PermissionError`**: Check that `set_request_identity(...)` was called and that the credentials and access DB permissions are correct.
+
+### Optional: Directory Transfer and Pre-Scan
+
+Directory transfer is designed for large directory migrations across machines or shared-storage systems. It continuously records scan progress, batch counts, active workers, and live bandwidth.
+
+This feature depends on the TiKV `transfer_state_store`. Start PD and TiKV as described in [User - 2 - Service Plane](<./User - 2 - Service Plane.md>) before continuing.
+
+#### Start a Directory Transfer from the UI
+
+In the dual-pane page:
+
+1. Locate the source folder on the left.
+2. Locate the target export and directory on the right.
+3. Drag the source folder to the right pane.
+4. Set `desired_worker_count` and `batch_ready_bytes`.
+5. Submit and inspect the job under `/ui/transfers/`.
+
+`FluxonFS Transfer Jobs` displays scan progress, batch counts, active workers, and live bandwidth.
+
+#### Import a Pre-Scan
+
+The `Pre-Scans` area under `/ui/transfers/` can import an existing scan as a transfer job:
+
+1. Find the pre-scan record and click `Import`.
+2. Select the source export and target export.
+3. Enter the target prefix and `desired_worker_count`.
+4. Submit and inspect it under `FluxonFS Transfer Jobs`.
+
+#### TiKV Namespace
+
+FS Master and the standalone pre-scan process must use the same:
+
+- `pd_endpoints`
+- `key_prefix`
+
+The examples on this page consistently use:
```yaml
transfer_state_store:
@@ -155,9 +588,12 @@ transfer_state_store:
key_prefix: "/fluxon_fs_transfer/demo-fs-cluster/"
```
-Standalone pre-scan code must use the same values.
+#### Standalone Pre-Scan Example
+
+A standalone pre-scan needs only PD and TiKV. KV Master, Owner Client, FS Master, and FS Agent do not need to be running. When FS Master later starts with the same `pd_endpoints` and `key_prefix`, the UI can display the pre-scan result.
-### Standalone Pre-Scan Example
+
+📄 View full pre-scan example (click to expand)
```python
#!/usr/bin/env python3
@@ -175,7 +611,7 @@ STORE = FluxonFsTransferStateStoreConfig(
kind=FluxonFsTransferStateStoreKind.TIKV,
tikv=FluxonFsTransferStateStoreTiKvConfig(
pd_endpoints=["127.0.0.1:12379"],
- key_prefix="/fluxon_fs_transfer/demo_prescan/",
+ key_prefix="/fluxon_fs_transfer/demo-fs-cluster/",
),
)
@@ -200,116 +636,6 @@ summary = transfer_check_local_blocking(
print(summary)
```
-`summary` is most useful for `job_id`, `scan_epoch`, and `batch_count`.
-
-## `FluxonFsPatcher`
-
-`FluxonFsPatcher` is not a standalone public entrypoint. It must sit on top of the `store` returned by `new_store(...)`.
-
-Required order:
-
-1. `store = new_store(cfg)...`
-2. `patcher = FluxonFsPatcher(store)`
-3. `patcher.set_master_config_yaml(...)`
-4. `patcher.set_cache_config_yaml(...)`
-5. `patcher.set_request_identity(...)`
-6. `patcher.install()`
-7. `patcher.mount_remote_dir(...)`
-8. `open()` / `read()` / `write()`
-9. `patcher.uninstall()`
-10. `store.close()`
-
-Do not close `store` before `patcher.uninstall()`.
-
-### Config Injection
-
-- `set_master_config_yaml(...)`: injects `fluxon_fs.master.instance_key`
-- `set_cache_config_yaml(...)`: injects the current export snapshot
-- `set_request_identity(username, password)`: binds later FS requests to one identity
-
-User-facing examples should set identity explicitly instead of depending on an implicit unauthenticated path.
-
-### `bootstrap_access_model`
-
-`bootstrap_access_model` is a required startup-time seed for an empty `access_db`.
-
-```yaml
-fluxon_fs:
- master_panel:
- access_db_path: /path/to/access.db
- bootstrap_access_model:
- users:
- - username: admin
- password: admin
- can_manage_users: true
- scope_access: []
-```
-
-Rules:
-
-- `access_db_path` is the long-lived authority
-- `bootstrap_access_model` must be provided explicitly in startup config
-- it only writes when `access_db` has no users yet
-- once users already exist, restarts follow the database state
-- `can_manage_users: true` grants runtime access to all current exports without writing synthetic root scopes
-
-## Mount Directory Rules
-
-`mount_remote_dir(local_mount_dir_abs=..., export_name=...)` requires:
-
-- an absolute path
-- not `/`
-- if the directory does not exist, Fluxon creates it
-- if the directory already exists, it must be empty
-- it must not overlap with another mount path in the same process
-
-## Logging
-
-For more Python-side logs:
-
-```bash
-FLUXON_LOG=DEBUG python3 examples/start_fluxon_fs_reader.py -c -w
-```
-
-Common levels:
-
-- `DEBUG`
-- `INFO`
-- `WARNING`
-- `ERROR`
-- `CRITICAL`
-
-## Common Errors
-
-### `new_store failed`
-
-Usually means the `External Client` did not attach to the local `Owner Client`. Check:
-
-- whether `start_kv_and_fs_svc.py` is still running
-- `CLUSTER_NAME`
-- `SHARE_MEM_PATH`
-
-### `fluxon_fs cache config is not loaded yet`
-
-Usually means `set_cache_config_yaml(...)` did not complete successfully, or the client-side cache config does not match the current server export config. Check:
-
-- `FS_MASTER_INSTANCE_KEY`
-- `EXPORT_NAME`
-- `REMOTE_ROOT_DIR`
-
-### `unknown export_name`
-
-The client is trying to mount an `EXPORT_NAME` that does not exist in the current `FS Master` export snapshot. Check:
-
-- whether writer and reader use the same `export_name`
-- whether `REMOTE_ROOT_DIR` matches the export definition
-
-### `permission denied` or `PermissionError`
-
-The path exists but the current identity does not have access. Check:
-
-- `ADMIN_USERNAME`
-- `ADMIN_PASSWORD`
-- whether the current `access_db` was already overwritten by newer user data
+
-If the admin password was changed through the UI or the database, the old `bootstrap_access_model` password no longer applies.
+Set `src_root_dir` to an existing source directory before running. The most useful fields in `summary` are `job_id`, `scan_epoch`, and `batch_count`.