Skip to content

feat: Add C ABI for embedded seekdb (multi‑language SDKs) - #674

Open
dengfuping wants to merge 162 commits into
masterfrom
feat/embedded-mode
Open

feat: Add C ABI for embedded seekdb (multi‑language SDKs)#674
dengfuping wants to merge 162 commits into
masterfrom
feat/embedded-mode

Conversation

@dengfuping

Copy link
Copy Markdown
Collaborator

Task Description

Solution Description

  • Architecture Design:
Language Bindings (Node.js, Python, Rust, Go)
           │
           ▼
    SeekDB C ABI (seekdb.h/seekdb.cpp)
           │
           ▼
    OceanBase Core (liboceanbase.so)
  • Source Code Directory (src/include/)
src/include/
├── seekdb.h                          # Public C API header file
│                                     # - All function declarations
│                                     # - Type definitions (SeekdbHandle, SeekdbResult, etc.)
│                                     # - Data structures (SeekdbBind, SeekdbField)
│                                     # - MySQL C API compatible interface
│
├── seekdb.cpp                        # C ABI implementation (~4800 lines)
│                                     # - Global initialization (seekdb_open/close)
│                                     # - Connection management
│                                     # - Query execution & Result set operations
│                                     # - Prepared Statement API
│                                     # - Signal handling & Stack management
│
├── CMakeLists.txt                    # Build configuration for libseekdb.so

Test Directory (unittest/include/)

unittest/include/
├── test_seekdb.cpp                   # C++ unit test suite (comprehensive)
│                                     # - Tests all C API functions directly
│                                     # - Complete API coverage
│
├── nodejs/                           # Node.js FFI binding (koffi)
│   ├── seekdb.js                    # FFI wrapper
│   ├── test.js                      # Test suite
│   └── test.sh                      # Test execution script
│
├── nodejs_napi/                      # Node.js N-API binding (native addon)
│   ├── seekdb.cpp                   # N-API C++ implementation
│   ├── test.js                      # Test suite
│   └── test.sh                      # Test execution script
│
├── python/                           # Python binding (ctypes)
│   ├── seekdb.py                    # FFI wrapper
│   ├── test.py                      # Test suite
│   └── test.sh                      # Test execution script
│
├── rust/                             # Rust FFI binding
│   ├── src/
│   │   ├── lib.rs                   # FFI bindings
│   │   └── test.rs                  # Test suite
│   └── test.sh                      # Test execution script
│
└── go/                               # Go binding (cgo)
    ├── seekdb/seekdb.go              # FFI wrapper
    ├── test.go                       # Test suite
    └── test.sh                       # Test execution script

Passed Regressions

  • Unittest
  • Node.js FFI bindings
  • Node.js N-API bindings
  • Python FFI bindings
  • Go FFI bindings
  • Rust FFI bindings

Build and test:

# build libseekdb library
./build.sh release --init -DBUILD_EMBED_MODE=ON --make

# unittest
make -j$(nproc) test_seekdb
cd build_release/unittest/include && ./test_seekdb

# bindings test
cd unittest/include/nodejs && bash ./test.sh
cd unittest/include/nodejs_napi && bash ./test.sh
cd unittest/include/python && bash ./test.sh
cd unittest/include/go && bash ./test.sh
cd unittest/include/rust && bash ./test.sh

Upgrade Compatibility

  • Compatible

Other Information

Release Note

  • 🆕 Add C ABI for embedded seekdb (multi‑language SDKs).

…ix MTL_SWITCH namespace; remove §7/doc refs

- C ABI: row_nulls for NULL vs empty; read_lob_data for TEXT/JSON LOB; MTL_SWITCH(OB_SYS_TENANT_ID) for ObLobManager; namespace share alias for MTL_SWITCH
- test: 100KB LONGTEXT via SET SESSION ob_default_lob_inrow_threshold=262144, no table option; special chars metadata test; remove §7 and c-abi-modification-reference references from comments and error messages
- Add LOB read paths for TEXT/JSON columns in result row serialization
- Use print_sql_literal for ObJsonType cells and expand buffer to 32KB
- Add col_is_metadata heuristic: write "{}" for empty metadata in fallback paths
- Add unit tests: very long document (100KB), special chars in metadata, empty JSON metadata
- macOS: put ccache in deps devtools/bin so Env.cmake finds it without code change
- Linux: install base deps (incl. wget) first so dep_create.sh can download RPMs
- Linux: try EPEL+ccache; use OB_USE_CCACHE=OFF when ccache not available in container
…cOS cross-build arch

- Checkout PR head ref instead of merge ref to avoid conflict markers in CI
- Restore darwin-x64 job (cross-compile on Apple Silicon for Intel Mac)
- Env.cmake: use CMAKE_OSX_ARCHITECTURES for ARCHITECTURE on macOS x86_64 cross-build
So next push can restore cache and avoid full rebuild from scratch.
- workflow: cd to GITHUB_WORKSPACE/package/libseekdb for Linux pack (was relative to build_release)
- zlib_lite: build QPL only on x86_64 Linux (NOT APPLE); macOS deps do not ship qpl
Resolve conflicts per agreed policy:
- master wins: CMake bazel-inventory mechanism, DML service refactor,
  pretest unittest system, deps/oblib -> src/oblib migration
- HEAD wins: Android/embedded platform support, wix packaging, build.ps1
The rust-checks clippy job runs `cargo clippy --locked`, which requires an
existing lockfile; rust/Cargo.lock was never committed, so the job failed
at the resolution step. Commit the generated lockfile (36 packages, cargo
1.97.1) and gitignore rust/target/ build output.
# Conflicts:
#	CMakeLists.txt
#	src/oblib/lib/alloc/alloc_func.h
#	src/observer/ob_server.cpp
#	src/share/config/ob_server_config.cpp
#	src/share/parameter/ob_parameter_seed.ipp
Resolve merge conflicts in build/deps/CMake/docs:
- build.ps1, grpc/CMakeLists.txt, ob_thread_hook.cpp: keep branch
  (feat/embedded-mode) Windows/Lite implementations
- CMakeLists.txt, deps/init/*.deps, emit_bazel_source_inventory.py:
  adopt master's packaging/deps content
- android docs: keep detailed build steps, drop non-existent
  all_tests target note in favor of master's guidance
@dengfuping
dengfuping changed the base branch from master.bak to master August 31, 2026 06:23
Comment on lines +72 to +273
name: Build libseekdb (${{ matrix.platform }})
runs-on: ${{ matrix.runner }}
strategy:
fail-fast: false
matrix:
include:
- platform: linux-x64
runner: ubuntu-22.04
artifact_name: libseekdb-linux-x64
container_image: quay.io/pypa/manylinux2014_x86_64
deps_file: oceanbase.el7.x86_64.deps
- platform: linux-arm64
runner: ubuntu-22.04-arm
artifact_name: libseekdb-linux-arm64
container_image: quay.io/pypa/manylinux2014_aarch64
deps_file: oceanbase.el7.aarch64.deps

steps:
- name: Checkout
uses: actions/checkout@v4
with:
# workflow_dispatch: use inputs.ref if set; PR: head sha; push: event sha
ref: ${{ github.event_name == 'workflow_dispatch' && inputs.ref != '' && inputs.ref || (github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha) }}

- name: Cache deps
uses: actions/cache@v4
with:
path: deps/3rd
key: ${{ runner.os }}-libseekdb-deps-${{ matrix.platform }}-el7-${{ hashFiles(format('deps/init/{0}', matrix.deps_file)) }}
restore-keys: |
${{ runner.os }}-libseekdb-deps-${{ matrix.platform }}-el7-

- name: Cache ccache
uses: actions/cache@v4
with:
path: .ccache
key: ${{ runner.os }}-ccache-libseekdb-${{ matrix.platform }}-${{ env.COMMIT_SHA }}
restore-keys: |
${{ runner.os }}-ccache-libseekdb-${{ matrix.platform }}-

# Run build inside manylinux2014 so Node/actions run on host (glibc 2.28+), build runs in CentOS 7 (glibc 2.17) for compatibility
- name: Build libseekdb
env:
BUILD_TYPE: release
run: |
docker run --rm -u root \
-v "$GITHUB_WORKSPACE:$GITHUB_WORKSPACE" -w "$GITHUB_WORKSPACE" \
-e BUILD_TYPE -e GITHUB_WORKSPACE \
${{ matrix.container_image }} \
bash -c '
set -e
# Install deps in two steps: base first (wget required for dep_create.sh), then ccache if available
yum install -y git wget rpm cpio make glibc-devel glibc-headers binutils m4 python3 python3-devel libtool libaio ncurses-devel which zlib-devel
yum install -y epel-release 2>/dev/null || true
yum install -y ccache 2>/dev/null || true
git config --global --add safe.directory "$GITHUB_WORKSPACE"
# CMake includes cmake/Rust.cmake at configure time; cargo must exist before build.sh.
# shellcheck source=/dev/null
source .github/script/install-rust.sh
export PATH="$CARGO_HOME/bin:$PATH"
CARGO_CMAKE="-DCARGO=$(command -v cargo)"
export CCACHE_DIR="$GITHUB_WORKSPACE/.ccache"
export CCACHE_COMPILERCHECK=content
export CCACHE_NOHASHDIR=1
mkdir -p deps/3rd/usr/local/oceanbase/devtools/bin
CCACHE_SRC=$(command -v ccache 2>/dev/null || true)
if [ -n "$CCACHE_SRC" ] && [ -x "$CCACHE_SRC" ]; then
ln -sf "$CCACHE_SRC" deps/3rd/usr/local/oceanbase/devtools/bin/ccache
USE_CCACHE="-DOB_USE_CCACHE=ON"
else
USE_CCACHE="-DOB_USE_CCACHE=OFF"
fi
export PATH="$GITHUB_WORKSPACE/deps/3rd/usr/local/oceanbase/devtools/bin:$PATH"
bash build.sh init
if [ -x /opt/python/cp39-cp39/bin/python3.9 ]; then
export PATH="/opt/python/cp39-cp39/bin:$PATH"
PYVER=3.9
else
PYVER=$(python3 -c "import sys; print(f\"{sys.version_info.major}.{sys.version_info.minor}\")")
fi
bash build.sh release --init $USE_CCACHE -DBUILD_EMBED_MODE=ON -DPYTHON_VERSION=$PYVER $CARGO_CMAKE --make libseekdb
[ -n "$USE_CCACHE" ] && ccache -s || true
'
- name: Fix ownership (container writes as root)
run: sudo chown -R "$(id -u):$(id -g)" "$GITHUB_WORKSPACE"

- name: Setup Rust
uses: ./.github/actions/setup-rust

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: "18"

- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: "1.21"

- name: Setup Java
uses: actions/setup-java@v4
with:
distribution: "temurin"
java-version: "17"

- name: Test Python binding
id: binding_python
continue-on-error: true
run: |
cd unittest/include/python
bash test.sh

- name: Test Node.js FFI binding
id: binding_node_ffi
continue-on-error: true
run: |
cd unittest/include/nodejs
npm install
bash test.sh

- name: Test Node.js N-API binding
id: binding_node_napi
continue-on-error: true
run: |
PATH=$(echo "$PATH" | tr ':' '\n' | grep -v 'deps/3rd' | tr '\n' ':' | sed 's/:$//')
cd unittest/include/nodejs_napi
npm install
bash test.sh

- name: Test Rust binding
id: binding_rust
continue-on-error: true
run: |
cd unittest/include/rust
bash test.sh

- name: Test Go binding
id: binding_go
continue-on-error: true
run: |
PATH=$(echo "$PATH" | tr ':' '\n' | grep -v 'deps/3rd' | tr '\n' ':' | sed 's/:$//')
cd unittest/include/go
bash test.sh

- name: Test Java binding
id: binding_java
continue-on-error: true
run: |
PATH=$(echo "$PATH" | tr ':' '\n' | grep -v 'deps/3rd' | tr '\n' ':' | sed 's/:$//')
cd unittest/include/java
bash test.sh

- name: Pack libseekdb
run: |
chmod +x package/libseekdb/libseekdb-build.sh
cd package/libseekdb && bash libseekdb-build.sh "${GITHUB_WORKSPACE}/build_release/src/include"

- name: Verify packed artifact
run: |
chmod +x package/libseekdb/test-packed-artifact-smoke.sh
package/libseekdb/test-packed-artifact-smoke.sh "package/libseekdb/libseekdb-${{ matrix.platform }}.zip"

- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: ${{ matrix.artifact_name }}
path: package/libseekdb/libseekdb-*.zip

- name: Save Cache deps
if: always()
uses: actions/cache/save@v4
with:
path: deps/3rd
key: ${{ runner.os }}-libseekdb-deps-${{ matrix.platform }}-el7-${{ hashFiles(format('deps/init/{0}', matrix.deps_file)) }}
- name: Save Cache ccache
if: always()
uses: actions/cache/save@v4
with:
path: .ccache
key: ${{ runner.os }}-ccache-libseekdb-${{ matrix.platform }}-${{ env.COMMIT_SHA }}

- name: Binding tests outcome
if: always()
run: |
failed=0
for o in \
"${{ steps.binding_node_ffi.outcome }}" \
"${{ steps.binding_node_napi.outcome }}" \
"${{ steps.binding_python.outcome }}" \
"${{ steps.binding_rust.outcome }}" \
"${{ steps.binding_go.outcome }}" \
"${{ steps.binding_java.outcome }}"; do
[ "$o" = "failure" ] && failed=1
done
if [ "$failed" -ne 0 ]; then
echo "::error::One or more libseekdb binding tests failed on ${{ matrix.platform }}"
exit 1
fi
echo "All binding test steps succeeded (or were skipped)."

# ---------- Build on macOS (no container) ----------
build-macos:
Comment on lines +274 to +471
name: Build libseekdb (${{ matrix.platform }})
runs-on: ${{ matrix.runner }}
strategy:
fail-fast: false
matrix:
include:
- platform: darwin-arm64
runner: macos-15
artifact_name: libseekdb-darwin-arm64
arch: arm64
cmake_arch: arm64

steps:
- name: Checkout
uses: actions/checkout@v4
with:
ref: ${{ github.event_name == 'workflow_dispatch' && inputs.ref != '' && inputs.ref || (github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha) }}

- name: Install macOS dependencies (pinned Homebrew)
run: |
chmod +x package/libseekdb/install-macos-brew-deps.sh
package/libseekdb/install-macos-brew-deps.sh

- name: Cache deps
uses: actions/cache@v4
with:
path: deps/3rd
key: ${{ runner.os }}-libseekdb-deps-${{ matrix.platform }}-${{ hashFiles('deps/init/oceanbase.macos.arm64.deps') }}
restore-keys: |
${{ runner.os }}-libseekdb-deps-${{ matrix.platform }}-

- name: Cache ccache
uses: actions/cache@v4
with:
path: .ccache
key: ${{ runner.os }}-ccache-libseekdb-${{ matrix.platform }}-${{ env.COMMIT_SHA }}
restore-keys: |
${{ runner.os }}-ccache-libseekdb-${{ matrix.platform }}-

- name: Setup Rust
uses: ./.github/actions/setup-rust

- name: Build init (macOS)
run: bash build.sh init

- name: Build libseekdb
env:
BUILD_TYPE: release
ARCH: ${{ matrix.arch }}
CMAKE_OSX_ARCHITECTURES: ${{ matrix.cmake_arch }}
CCACHE_DIR: ${{ github.workspace }}/.ccache
CCACHE_COMPILERCHECK: content
CCACHE_NOHASHDIR: 1
run: |
# Env.cmake looks for ccache in deps/3rd/.../devtools/bin; put it there so no Env.cmake change is needed
mkdir -p deps/3rd/usr/local/oceanbase/devtools/bin
ln -sf "$(which ccache)" deps/3rd/usr/local/oceanbase/devtools/bin/ccache
# Use runner's Python (macOS has 3.x, not 3.8 by default) so embed CMake finds it
PYVER=$(python3 -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}')")
bash build.sh release --init -DOB_USE_CCACHE=ON -DBUILD_EMBED_MODE=ON -DPYTHON_VERSION=$PYVER -DCMAKE_OSX_ARCHITECTURES=${{ matrix.cmake_arch }} -DCMAKE_OSX_DEPLOYMENT_TARGET=11.0 --make libseekdb
ccache -s

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: "18"

- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: "1.21"

- name: Setup Java
uses: actions/setup-java@v4
with:
distribution: "temurin"
java-version: "17"

- name: Test Python binding
id: binding_python
continue-on-error: true
run: |
cd unittest/include/python
bash test.sh

- name: Test Node.js FFI binding
id: binding_node_ffi
continue-on-error: true
run: |
cd unittest/include/nodejs
npm install
bash test.sh

- name: Test Node.js N-API binding
id: binding_node_napi
continue-on-error: true
run: |
PATH=$(echo "$PATH" | tr ':' '\n' | grep -v 'deps/3rd' | tr '\n' ':' | sed 's/:$//')
cd unittest/include/nodejs_napi
npm install
bash test.sh

- name: Test Rust binding
id: binding_rust
continue-on-error: true
run: |
cd unittest/include/rust
bash test.sh

- name: Test Go binding
id: binding_go
continue-on-error: true
run: |
PATH=$(echo "$PATH" | tr ':' '\n' | grep -v 'deps/3rd' | tr '\n' ':' | sed 's/:$//')
cd unittest/include/go
bash test.sh

- name: Test Java binding
id: binding_java
continue-on-error: true
run: |
PATH=$(echo "$PATH" | tr ':' '\n' | grep -v 'deps/3rd' | tr '\n' ':' | sed 's/:$//')
cd unittest/include/java
bash test.sh

# macOS code signing. Entitlements allow loading embedded dylib.
- name: Create entitlements (macOS)
run: |
echo -e '<?xml version="1.0" encoding="UTF-8"?>\n<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">\n<plist version="1.0">\n<dict>\n <key>com.apple.security.cs.disable-library-validation</key>\n <true/>\n</dict>\n</plist>' > package/libseekdb/entitlements.plist

# Only import certificate on main repo; script skips when cert empty (ad-hoc signing used).
- name: Import certificate (macOS)
if: github.repository == 'oceanbase/seekdb'
env:
BUILD_CERTIFICATE_BASE64: ${{ secrets.OSX_CODESIGN_BUILD_CERTIFICATE_BASE64 }}
P12_PASSWORD: ${{ secrets.OSX_CODESIGN_P12_PASSWORD }}
KEYCHAIN_PASSWORD: ${{ secrets.OSX_CODESIGN_KEYCHAIN_PASSWORD }}
run: . package/libseekdb/osx_import_codesign_certificate.sh

- name: Pack libseekdb
env:
ARCH: ${{ matrix.arch }}
CODESIGN_IDENTITY: ${{ secrets.OSX_CODESIGN_IDENTITY }}
CODESIGN_ENTITLEMENTS: ${{ github.workspace }}/package/libseekdb/entitlements.plist
run: cd package/libseekdb && bash libseekdb-build.sh

- name: Verify packed artifact
run: |
chmod +x package/libseekdb/test-packed-artifact-smoke.sh
package/libseekdb/test-packed-artifact-smoke.sh package/libseekdb/libseekdb-darwin-arm64.zip

- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: ${{ matrix.artifact_name }}
path: package/libseekdb/libseekdb-*.zip

# Save caches even on failure so next run can resume (deps/ccache)
- name: Save Cache deps
if: always()
uses: actions/cache/save@v4
with:
path: deps/3rd
key: ${{ runner.os }}-libseekdb-deps-${{ matrix.platform }}-${{ hashFiles('deps/init/oceanbase.macos.arm64.deps') }}
- name: Save Cache ccache
if: always()
uses: actions/cache/save@v4
with:
path: .ccache
key: ${{ runner.os }}-ccache-libseekdb-${{ matrix.platform }}-${{ env.COMMIT_SHA }}

- name: Binding tests outcome
if: always()
run: |
failed=0
for o in \
"${{ steps.binding_node_ffi.outcome }}" \
"${{ steps.binding_node_napi.outcome }}" \
"${{ steps.binding_python.outcome }}" \
"${{ steps.binding_rust.outcome }}" \
"${{ steps.binding_go.outcome }}" \
"${{ steps.binding_java.outcome }}"; do
[ "$o" = "failure" ] && failed=1
done
if [ "$failed" -ne 0 ]; then
echo "::error::One or more libseekdb binding tests failed on ${{ matrix.platform }}"
exit 1
fi
echo "All binding test steps succeeded (or were skipped)."

# ---------- Android NDK cross-compile (macOS arm64 host; target arm64-v8a) ----------
# ubuntu-* runners are x86_64 by default. macOS-15 uses Apple silicon + NDK darwin-arm64 prebuilts (see docs).
# Alternative: runs-on: ubuntu-24.04-arm for Linux ARM64.
#
# Why no FFI binding tests here: libseekdb.so is Android arm64-v8a / ELF for Bionic — the dynamic linker,
# libc, and JNI/NDK ABI differ from macOS dyld. CI cannot dlopen/load that .so on the host like Linux/macOS/Windows jobs.
# Running the same unittest/include/*/test.sh would require an Android emulator, rooted device, or adb push + adb shell test.
build-android:
Comment thread .github/workflows/build-libseekdb.yml Fixed
Comment on lines +586 to +871
name: Build libseekdb (windows-x64)
runs-on: windows-2022
timeout-minutes: 360
steps:
- name: Checkout
uses: actions/checkout@v4
with:
ref: ${{ github.event_name == 'workflow_dispatch' && inputs.ref != '' && inputs.ref || (github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha) }}

- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Cache deps
uses: actions/cache@v4
with:
path: deps/3rd
key: ${{ runner.os }}-libseekdb-deps-windows-x64-${{ hashFiles('deps/init/oceanbase.windows.x86_64.deps') }}
restore-keys: |
${{ runner.os }}-libseekdb-deps-windows-x64-

- name: Cache ccache
uses: actions/cache@v4
with:
path: .ccache
key: ${{ runner.os }}-ccache-libseekdb-windows-x64
restore-keys: |
${{ runner.os }}-ccache-libseekdb-windows-x64-

- name: Install ccache (Windows)
shell: pwsh
run: |
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
function Refresh-Path {
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
}
Refresh-Path
if (Get-Command ccache -ErrorAction SilentlyContinue) {
Write-Host "ccache already on PATH"
Get-Command ccache | Out-Host
exit 0
}
choco install ccache -y --no-progress 2>&1 | Out-Host
Refresh-Path
if ($LASTEXITCODE -ne 0 -or -not (Get-Command ccache -ErrorAction SilentlyContinue)) {
Write-Host "::notice::choco install ccache failed or ccache not on PATH; installing from ccache GitHub releases."
$ver = "4.13.2"
$zipName = "ccache-$ver-windows-x86_64.zip"
$url = "https://github.com/ccache/ccache/releases/download/v$ver/$zipName"
$tools = Join-Path $env:GITHUB_WORKSPACE ".tools"
$dest = Join-Path $tools "ccache-$ver-win64"
New-Item -ItemType Directory -Force -Path $dest | Out-Null
$zipPath = Join-Path $dest $zipName
Invoke-WebRequest -Uri $url -OutFile $zipPath -UseBasicParsing
Expand-Archive -Path $zipPath -DestinationPath $dest -Force
$exe = Get-ChildItem -Path $dest -Filter ccache.exe -Recurse | Select-Object -First 1
if (-not $exe) { throw "ccache.exe not found after extracting $zipName under $dest" }
$binDir = $exe.Directory.FullName
Add-Content -Path $env:GITHUB_PATH -Value $binDir -Encoding utf8
$env:PATH = "$binDir;$env:PATH"
}
Get-Command ccache | Out-Host

- name: Setup Rust
uses: ./.github/actions/setup-rust

- name: Build libseekdb
shell: pwsh
env:
BUILD_TYPE: release
CCACHE_DIR: ${{ github.workspace }}/.ccache
CCACHE_COMPILERCHECK: content
CCACHE_NOHASHDIR: 1
run: |
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
# Pick up Chocolatey / GITHUB_PATH / fallback-installed ccache
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
$py = python -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}')"
if (-not (Test-Path "deps/3rd/DONE")) {
.\build.ps1 init
}
# Prepend deps LLVM to PATH; pass lld-link so CMake does not pick GNU ld (MinGW) for clang-cl.
$ws = if ($env:GITHUB_WORKSPACE) { $env:GITHUB_WORKSPACE } else { (Get-Location).Path }
# CMake -G Ninja requires ninja on PATH at configure time (otherwise CMake may fall back or leave a stale non-Ninja cache).
foreach ($rel in @("deps/3rd/tools/ninja", "deps/3rd/tools/cmake/bin")) {
$tp = Join-Path $ws $rel
if (Test-Path -LiteralPath $tp) { $env:PATH = "$tp;$env:PATH" }
}
$llvmRoot = if ($env:OB_LLVM_DIR) { $env:OB_LLVM_DIR } else { Join-Path $ws "deps/3rd/tools/llvm18" }
$llvmBin = Join-Path $llvmRoot "bin"
$lldLink = Join-Path $llvmBin "lld-link.exe"
if (-not (Test-Path $lldLink)) { throw "lld-link.exe not found: $lldLink" }
$lldFwd = $lldLink.Replace("\", "/")
$env:PATH = "$llvmBin;$env:PATH"
$ccacheOpt = "-DOB_USE_CCACHE=OFF"
if (Get-Command ccache -ErrorAction SilentlyContinue) {
$ccacheOpt = "-DOB_USE_CCACHE=ON"
} else {
Write-Host "::warning::ccache not found; building without compiler cache."
}
# Map RelWithDebInfo/Debug/MinSizeRel -> Release for IMPORTED targets (e.g. Python3::Module .lib/.dll on Windows).
# Ensures cache has these on first cmake run; complements cmake/Env.cmake for CI runners where Python stubs omit RelWithDebInfo.
.\build.ps1 release --ninja --target libseekdb "-DBUILD_EMBED_MODE=ON" "-DPYTHON_VERSION=$py" "-DCMAKE_LINKER=$lldFwd" "-DCMAKE_MAP_IMPORTED_CONFIG_RELWITHDEBINFO=Release" "-DCMAKE_MAP_IMPORTED_CONFIG_DEBUG=Release" "-DCMAKE_MAP_IMPORTED_CONFIG_MINSIZEREL=Release" $ccacheOpt
if (Get-Command ccache -ErrorAction SilentlyContinue) { ccache -s }

- name: Debug facts (Windows)
shell: pwsh
env:
BUILD_TYPE: release
run: |
$ErrorActionPreference = "Continue"
$ws = if ($env:GITHUB_WORKSPACE) { $env:GITHUB_WORKSPACE } else { (Get-Location).Path }
foreach ($rel in @("deps/3rd/tools/ninja", "deps/3rd/tools/cmake/bin")) {
$tp = Join-Path $ws $rel
if (Test-Path -LiteralPath $tp) { $env:PATH = "$tp;$env:PATH" }
}
. ./unittest/include/seekdb-windows-dll-resolve.ps1
. ./unittest/include/debug-libseekdb-windows.ps1
Write-LibseekdbWindowsBuildFacts -RepoRoot "$PWD"

- name: Verify libseekdb DLL (Windows)
shell: pwsh
env:
BUILD_TYPE: release
run: |
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
. ./unittest/include/seekdb-windows-dll-resolve.ps1
$bdn = Get-SeekDbWindowsBuildDirNameFromEnv
$r = Find-SeekDbWindowsDll -RepoRoot "$PWD" -BuildDirName $bdn
if (-not $r) {
Write-SeekDbWindowsDllDiagnostics -RepoRoot "$PWD" -BuildDirName $bdn
throw "libseekdb build did not produce seekdb.dll under build_$bdn (see diagnostics above)."
}
Write-Host "libseekdb DLL: $($r.DllPath)"
"SEEKDB_LIB_PATH=$($r.DllPath)" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: "18"

- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: "1.21"

- name: Setup Java
uses: actions/setup-java@v4
with:
distribution: "temurin"
java-version: "17"

- name: Install MinGW and Maven (Go CGO + Java)
shell: pwsh
run: choco install mingw maven -y --no-progress

# One job step per language: if something hangs, the in-progress step name in the Actions UI shows
# whether the stall is in Python, npm, node-gyp, cargo, go, or Java (not “whole binding” vs global timeout).
- name: Test Python binding
id: binding_python
continue-on-error: true
shell: pwsh
env:
SEEKDB_BINDING_SECTION: Python
run: |
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
Get-Command gcc | Out-Host
Get-Command mvn | Out-Host
.\unittest\include\run-libseekdb-binding-tests.ps1 -RepoRoot "$PWD" -ContinueOnError

- name: Test Node.js FFI binding
id: binding_node_ffi
continue-on-error: true
shell: pwsh
env:
SEEKDB_BINDING_SECTION: NodeFfi
run: |
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
.\unittest\include\run-libseekdb-binding-tests.ps1 -RepoRoot "$PWD" -ContinueOnError

- name: Test Node.js N-API binding
id: binding_node_napi
continue-on-error: true
shell: pwsh
env:
SEEKDB_BINDING_SECTION: NodeNapi
# VECTOR + DBMS_HYBRID_SEARCH cases can stall native code on Windows runners; core N-API still covered.
SEEKDB_NODE_NAPI_SKIP_HEAVY: "1"
run: |
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
.\unittest\include\run-libseekdb-binding-tests.ps1 -RepoRoot "$PWD" -ContinueOnError

- name: Test Rust binding
id: binding_rust
continue-on-error: true
shell: pwsh
env:
SEEKDB_BINDING_SECTION: Rust
run: |
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
.\unittest\include\run-libseekdb-binding-tests.ps1 -RepoRoot "$PWD" -ContinueOnError

- name: Test Go binding
id: binding_go
continue-on-error: true
shell: pwsh
env:
SEEKDB_BINDING_SECTION: Go
run: |
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
.\unittest\include\run-libseekdb-binding-tests.ps1 -RepoRoot "$PWD" -ContinueOnError

- name: Test Java binding
id: binding_java
continue-on-error: true
shell: pwsh
env:
SEEKDB_BINDING_SECTION: Java
run: |
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
.\unittest\include\run-libseekdb-binding-tests.ps1 -RepoRoot "$PWD" -ContinueOnError

- name: Pack libseekdb
shell: pwsh
env:
BUILD_TYPE: release
run: .\package\libseekdb\libseekdb-build.ps1

# Hard gate: mirrors linux/macos (Verify packed artifact); fails the job if the packed zip does not load.
# Same step name as linux/macos (Verify packed artifact); platform script differs (see package/libseekdb/test-packed-artifact-smoke.ps1).
- name: Verify packed artifact
shell: pwsh
run: .\package\libseekdb\test-packed-artifact-smoke.ps1 -Zip "package\libseekdb\libseekdb-windows-x64.zip"

- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: libseekdb-windows-x64
path: package/libseekdb/libseekdb-windows-x64.zip

- name: Save Cache deps
if: always()
uses: actions/cache/save@v4
with:
path: deps/3rd
key: ${{ runner.os }}-libseekdb-deps-windows-x64-${{ hashFiles('deps/init/oceanbase.windows.x86_64.deps') }}

- name: Save Cache ccache
if: always()
uses: actions/cache/save@v4
with:
path: .ccache
key: ${{ runner.os }}-ccache-libseekdb-windows-x64

- name: Binding tests outcome
if: always()
shell: pwsh
run: |
$failed = $false
if ("${{ steps.binding_python.outcome }}" -eq "failure") { Write-Host "::error::Binding section failed: Python"; $failed = $true }
if ("${{ steps.binding_node_ffi.outcome }}" -eq "failure") { Write-Host "::error::Binding section failed: Node FFI (koffi)"; $failed = $true }
if ("${{ steps.binding_node_napi.outcome }}" -eq "failure") { Write-Host "::error::Binding section failed: Node N-API"; $failed = $true }
if ("${{ steps.binding_rust.outcome }}" -eq "failure") { Write-Host "::error::Binding section failed: Rust"; $failed = $true }
if ("${{ steps.binding_go.outcome }}" -eq "failure") { Write-Host "::error::Binding section failed: Go"; $failed = $true }
if ("${{ steps.binding_java.outcome }}" -eq "failure") { Write-Host "::error::Binding section failed: Java"; $failed = $true }
if ($failed) { exit 1 }
Write-Host "All Windows binding test sections succeeded."

# ---------- Collect libseekdb artifacts and upload to S3 (runs only when all needed build jobs succeed, including binding tests) ----------
release-artifacts:
Comment on lines +872 to +920
name: Collect artifacts and upload to S3
runs-on: ubuntu-22.04
needs:
- build
- build-macos
- build-android
- build-windows
steps:
- name: Download all artifacts
uses: actions/download-artifact@v4
with:
path: release-artifacts
merge-multiple: true

- name: List all artifacts
run: |
echo "=== All artifacts ==="
ls -la release-artifacts/

- name: Upload combined artifact (for workflow download)
uses: actions/upload-artifact@v4
with:
name: libseekdb-all-platforms
path: release-artifacts/

- name: Configure AWS credentials
if: env.UPLOAD_S3 == 'true'
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ env.AWS_REGION }}

- name: Upload to S3
if: env.UPLOAD_S3 == 'true'
run: |
set -e
if [ -n "${{ env.DESTINATION_TARGET_PATH }}" ]; then
S3_TARGET="${{ env.DESTINATION_TARGET_PATH }}"
else
S3_TARGET="s3://${{ env.S3_BUCKET }}/${{ env.S3_PREFIX }}/"
fi
[ "${S3_TARGET: -1}" != "/" ] && S3_TARGET="${S3_TARGET}/"
echo "Uploading to $S3_TARGET"
aws s3 cp release-artifacts/ "$S3_TARGET" --recursive --exclude "*" --include "*.zip" --no-progress
echo "Uploaded:"
aws s3 ls "$S3_TARGET" --recursive
echo "Done."
continue-on-error: true
- fix(resolver): restore ob_dml_resolver.cpp after silent merge splice corruption
- fix(embedded): read out-row LOB without is_lob_storage flag check
- fix(android): write empty ddl_stmt_str as empty string not NULL in __all_ddl_operation
- fix(embedded): include ob_strerror detail in observer init/start failure message
# Conflicts:
#	cmake/Rust.cmake
#	deps/init/oceanbase.al8.aarch64.deps
#	deps/init/oceanbase.al8.x86_64.deps
#	deps/init/oceanbase.el7.aarch64.deps
#	deps/init/oceanbase.el7.x86_64.deps
On the k8s ARC runner, ${{ github.workspace }} inside a composite step
expands to the runner host path (/home/runner/_work/...), which is not
mounted inside the job container where the checkout lives at
/__w/seekdb/seekdb. This made source install-rust.sh fail with
"No such file or directory" and broke the setup job.

Resolve the repo root from $GITHUB_WORKSPACE with a $(pwd) fallback,
matching the existing workaround in seekdb.yml's Compile step.
Comment thread .github/workflows/build-libseekdb.yml Outdated
Comment on lines +477 to +601
name: Build libseekdb (android-${{ matrix.abi }})
runs-on: macos-15
timeout-minutes: 180
strategy:
fail-fast: false
matrix:
include:
- abi: arm64-v8a
deps_arch: arm64
- abi: x86_64
deps_arch: x86_64
env:
ANDROID_ABI: ${{ matrix.abi }}
steps:
- name: Checkout
uses: actions/checkout@v4
with:
ref: ${{ github.event_name == 'workflow_dispatch' && inputs.ref != '' && inputs.ref || (github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha) }}

- name: Install macOS packages
run: |
brew install cmake ccache wget || true

- name: Install Android NDK
uses: nttld/setup-ndk@v1
with:
# Use rYYx form (e.g. r27d); full build numbers (27.3.13750724) 404 on the action download URL
ndk-version: r27d

- name: Cache deps
uses: actions/cache@v4
with:
path: deps/3rd
key: ${{ runner.os }}-libseekdb-deps-android-${{ matrix.deps_arch }}-${{ hashFiles(format('deps/init/oceanbase.android.{0}.deps', matrix.deps_arch)) }}
restore-keys: |
${{ runner.os }}-libseekdb-deps-android-${{ matrix.deps_arch }}-

- name: Cache ccache
uses: actions/cache@v4
with:
path: .ccache
key: ${{ runner.os }}-ccache-libseekdb-android-${{ matrix.deps_arch }}
restore-keys: |
${{ runner.os }}-ccache-libseekdb-android-${{ matrix.deps_arch }}-

- name: Setup Rust
uses: ./.github/actions/setup-rust

- name: Build libseekdb
env:
BUILD_TYPE: release
CCACHE_DIR: ${{ github.workspace }}/.ccache
CCACHE_COMPILERCHECK: content
CCACHE_NOHASHDIR: 1
run: |
set -e
# Do not symlink ccache into deps/3rd before --init: dep_create.sh rm -rf deps/3rd.
# CMake finds ccache on PATH when devtools has none (Android deps omit obdevtools-ccache).
# ANDROID_NDK_HOME is set by the NDK install step; job-level ANDROID_ABI (matrix)
# selects the CMake ABI + Rust target + devdeps arch inside build.sh/dep_create.sh.
bash build.sh release --android -DOB_USE_CCACHE=ON -DBUILD_EMBED_MODE=ON --init --make libseekdb
ccache -s

# Same toolchain install order as other jobs (no host FFI below).
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: "18"

- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: "1.21"

- name: Setup Java
uses: actions/setup-java@v4
with:
distribution: "temurin"
java-version: "17"

- name: Test bindings (host skip)
id: android_binding_notice
run: |
echo "::notice::Skipping native FFI binding tests on this runner: libseekdb.so targets Android ($ANDROID_ABI) / Bionic and cannot be dlopen()'d on the macOS host."
echo "Zip verification runs after pack; device or emulator CI would be required for full FFI tests."

- name: Pack libseekdb
run: cd package/libseekdb && ANDROID_ABI="$ANDROID_ABI" bash libseekdb-build.sh --android

- name: Verify packed artifact
id: android_verify_zip
continue-on-error: true
run: unzip -t package/libseekdb/libseekdb-android-${{ matrix.abi }}.zip

- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: libseekdb-android-${{ matrix.abi }}
path: package/libseekdb/libseekdb-android-${{ matrix.abi }}.zip

- name: Save Cache deps
if: always()
uses: actions/cache/save@v4
with:
path: deps/3rd
key: ${{ runner.os }}-libseekdb-deps-android-${{ matrix.deps_arch }}-${{ hashFiles(format('deps/init/oceanbase.android.{0}.deps', matrix.deps_arch)) }}

- name: Save Cache ccache
if: always()
uses: actions/cache/save@v4
with:
path: .ccache
key: ${{ runner.os }}-ccache-libseekdb-android-${{ matrix.deps_arch }}

- name: Pack / zip verification outcome
if: always()
run: |
if [ "${{ steps.android_verify_zip.outcome }}" = "failure" ]; then
echo "::error::Packaged zip verification failed (unzip -t)"
exit 1
fi
echo "Android pack and zip verification OK (or verify step skipped)."

# ---------- Build on Windows x64 (native, embed DLL) ----------
build-windows:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants