feat: serve dependencies from the serving knowledge base (v0.16.0)

.github/workflows/release.yml

@@ -9,30 +9,43 @@ on:
 jobs:
   build:
     runs-on: ubuntu-latest
+    # The DuckDB driver is a cgo binding, so the binaries link against glibc. Build on
+    # the runtime image's Debian release: an ubuntu-latest build needs glibc 2.38, bookworm has 2.36.
+    container: golang:1.25-bookworm
     permissions:
       contents: write
     steps:
       - uses: actions/checkout@v4
         with:
-          fetch-depth: 0  # Get tags to allow build script to get build version
+          ref: ${{ github.ref }}  # Checkout the specified tag
+          fetch-depth: 0  # Get tags to allow a build script to get a build version
+
+      - name: Trust the checkout
+        # The job runs as root in the container over a checkout the runner owns, and git
+        # refuses such a repository - which breaks the version lookup the build runs.
+        run: git config --global --add safe.directory "$GITHUB_WORKSPACE"
 
       - name: Set up Go
         uses: actions/setup-go@v5
         with:
           go-version-file: 'go.mod'
 
+      - name: Install the arm64 cgo toolchain
+        # The DuckDB driver is a cgo binding, so the arm64 build cross-compiles C and C++.
+        run: apt-get update && apt-get install -y gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
+
       - name: Build
         run: |
           make build_amd
-          make build_arm
+          CC=aarch64-linux-gnu-gcc CXX=aarch64-linux-gnu-g++ make build_arm
 
       - name: Unit Test
         run: make unit_test
 
       - name: Package
         run: |
           make package_amd
-          make package_arm
+          CC=aarch64-linux-gnu-gcc CXX=aarch64-linux-gnu-g++ make package_arm
 
       - name: Release
         uses: softprops/action-gh-release@v1

CHANGELOG.md

@@ -6,6 +6,31 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
+
+## [0.16.0] - 2026-10-28
+### Changed
+- **The service now reads the serving knowledge base instead of the mining database.** The mining path is gone; there is no configuration that selects it. Endpoint routes and response shapes are unchanged
+- The database is configured with `Database.Driver` (`DB_DRIVER`: `duckdb`, the default, `sqlite` or `postgres`) and the new `Database.Path` (`DB_PATH`), the serving file, opened read-only. `postgres` still takes `Dsn` or the host settings
+- Component and version resolution no longer goes through `go-component-helper`'s `GetComponentsVersion`, which resolves against `all_urls`, `mines` and `versions` - none of which exist in the serving schema. It is replaced by `pkg/resolver`, which looks the purl up in every spelling the export stores: `pkg:npm/@angular/core` now resolves as well as `pkg:npm/%40angular/core`, where it answered `NO_INFO`
+- **Licences read `component_version_license`** by (purl_id, version), through `license` and `"SPDX_license_data"`, rather than `all_urls`, `projects` and `licenses`. Whether a licence is SPDX-approved is whether its identifier is in the SPDX registry. A release with more than one licence row (about 1% of releases) reports all of them, where mining reported one; compound identifiers (`MIT/Apache-2.0`) are still split
+- There is no component-level licence any more: mining fell back to `projects` when a release had none, and serving has no such table. Instead, a release with no licence of its own borrows one, in the order `scanoss.api`'s licence lookup uses, and reports it with `info_code` `WARNING` and an `info_message` naming where it came from:
+  1. the source component (`component.source_purl_id`) at the same version - "Showing results from the source code purl (…)";
+  2. the nearest version of the component that has a licence - "Showing results from version …";
+  3. the nearest version of the source component that has one - "Showing results from the source code purl (…) version …".
+
+  A status that already reports a problem (`REQUIREMENT_NOT_MET`, `VERSION_NOT_FOUND`) keeps its code and gains the message. 43% of `component_version_license` rows point at license 523, which has neither a name nor an identifier - mining's "no licence" row, so mining answered these releases without a licence as well. A release whose only rows say there is no licence (523, "None detected", "UNKNOWN", "unknown") also borrows, and keeps them when nothing can be borrowed; those rows are never borrowed. Nearness is semver distance, ties to the higher version, with versions that differ only in build metadata (golang pseudo-versions) ordered by it. Measured on samples of the export, the fallback recovers a licence for 17-21% of such releases, 36% for components with 1,000 stars or more; a release with nothing to borrow still reports `NO_INFO`
+- **Transitive dependencies read `component_version_dependency`** instead of the per-ecosystem `<ecosystem>_dependencies` JSON tables. Every scope is followed, as before, and a requirement still resolves to the first version it names. Rust crates are looked up under the export's `cargo` purl type; responses still say `crates`
+- Golang modules are ordinary serving components. The export carries them at pseudo-versions only, so a requested tag is answered live from pkg.go.dev, then from the GitHub repository the module path names, as `golang_projects` used to; a tag answered that way reports `SUCCESS`. The live lookup of a module is given up after 5 seconds, both of its attempts together, and the GitHub repository and the knowledge base answer instead: a slow pkg.go.dev otherwise held the whole request (one call was seen taking 19 s)
+- An exact requested version is reported as the caller spelled it. The export drops the leading `v` mining stored (`v5.4.5` is `5.4.5`), and a golang version without it is not the version asked for
+- Binaries are built with `CGO_ENABLED=1`; the release builds in a `golang:1.25-bookworm` container with an arm64 cgo cross toolchain, and the Docker image builds on `golang:1.25-bookworm` and runs on `debian:bookworm-slim` with `libstdc++6` and `ca-certificates`. The DuckDB driver is a cgo binding, so a pure-Go build can no longer be produced
+- Upgraded `scanoss/go-grpc-helper` to `v0.17.0`; added `marcboeker/go-duckdb/v2` `v2.4.3`
+- The tests run on SQLite against a transcription of the serving schema (`pkg/models/test_schema.go`), with data-only fixtures
+### Removed
+- `COMP_COMMIT_MISSING` / `Components.CommitMissing`: the serving knowledge base is read-only, so live pkg.go.dev answers are no longer written back. The live lookup itself remains
+### Fixed
+- Transitive lookups of namespaced packages: the package identifier kept the namespace only for maven, so a scoped npm entry (`pkg:npm/@angular/core`) or a composer one (`pkg:composer/laravel/framework`) was looked up by its bare name and answered "not found"
+- The service embeds `UnimplementedDependenciesServer` instead of the papi interface, so a method added to the contract answers `Unimplemented` instead of dereferencing a nil interface and killing the process
+
 ### Security
 - Upgraded `google.golang.org/grpc` to v1.83.1, `golang.org/x/net` to v0.57.0 and the OpenTelemetry modules (`otel`, `otel/sdk`, `otlptrace`, `otlptracegrpc`) to v1.45.0, and `github.com/antchfx/xpath` to v1.3.6 to fix open Dependabot security advisories
 
@@ -154,4 +179,5 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 [0.12.2]: https://github.com/scanoss/dependencies/compare/v0.12.1...v0.12.2
 [0.13.0]: https://github.com/scanoss/dependencies/compare/v0.12.2...v0.13.0
 [0.14.0]: https://github.com/scanoss/dependencies/compare/v0.13.0...v0.14.0
-[0.15.0]: https://github.com/scanoss/dependencies/compare/v0.14.0...v0.15.0
+[0.15.0]: https://github.com/scanoss/dependencies/compare/v0.14.0...v0.15.0
+[0.16.0]: https://github.com/scanoss/dependencies/compare/v0.15.0...v0.16.0

Dockerfile

@@ -1,4 +1,4 @@
-FROM golang:1.19 as build
+FROM golang:1.25-bookworm AS build
 
 WORKDIR /app
 
@@ -10,14 +10,20 @@ RUN go mod download
 COPY . ./
 
 RUN go generate ./pkg/cmd/server.go
-#RUN go build -o ./scanoss-dependencies ./cmd/server
-RUN GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags="-w -s" -o ./scanoss-dependencies ./cmd/server
+# cgo is required: the DuckDB driver used for the serving knowledge base is a
+# cgo binding, so this cannot be a CGO_ENABLED=0 build.
+RUN GOOS=linux GOARCH=amd64 CGO_ENABLED=1 go build -ldflags="-w -s" -o ./scanoss-dependencies ./cmd/server
 
+FROM debian:bookworm-slim
 
-FROM debian:buster-slim
+# The cgo binary links libstdc++ dynamically; ca-certificates keeps outbound TLS
+# working (the pkg.go.dev live lookup needs it).
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends libstdc++6 ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
 
 WORKDIR /app
- 
+
 COPY --from=build /app/scanoss-dependencies /app/scanoss-dependencies
 
 EXPOSE 50051

Makefile

@@ -74,23 +74,23 @@ ghcr_all: ghcr_build ghcr_tag ghcr_push  ## Execute all GitHub Package container
 build_amd: version  ## Build an AMD 64 binary
 	@echo "Building AMD binary $(VERSION)..."
 	go generate ./pkg/cmd/server.go
-	GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags="-w -s" -o ./target/scanoss-dependencies-api-linux-amd64 ./cmd/server
+	GOOS=linux GOARCH=amd64 CGO_ENABLED=1 go build -ldflags="-w -s" -o ./target/scanoss-dependencies-api-linux-amd64 ./cmd/server
 
 build_arm: version  ## Build an ARM 64 binary
 	@echo "Building ARM binary $(VERSION)..."
 	go generate ./pkg/cmd/server.go
-	GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -ldflags="-w -s" -o ./target/scanoss-dependencies-api-linux-arm64 ./cmd/server
+	GOOS=linux GOARCH=arm64 CGO_ENABLED=1 go build -ldflags="-w -s" -o ./target/scanoss-dependencies-api-linux-arm64 ./cmd/server
 
 package: package_amd  ## Build & Package an AMD 64 binary
 
 package_amd: version  ## Build & Package an AMD 64 binary
 	@echo "Building AMD binary $(VERSION) and placing into scripts..."
 	go generate ./pkg/cmd/server.go
-	GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags="-w -s" -o ./scripts/scanoss-dependencies-api ./cmd/server
+	GOOS=linux GOARCH=amd64 CGO_ENABLED=1 go build -ldflags="-w -s" -o ./scripts/scanoss-dependencies-api ./cmd/server
 	bash ./package-scripts.sh linux-amd64 $(VERSION)
 
 package_arm: version  ## Build & Package an ARM 64 binary
 	@echo "Building ARM binary $(VERSION) and placing into scripts..."
 	go generate ./pkg/cmd/server.go
-	GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -ldflags="-w -s" -o ./scripts/scanoss-dependencies-api ./cmd/server
+	GOOS=linux GOARCH=arm64 CGO_ENABLED=1 go build -ldflags="-w -s" -o ./scripts/scanoss-dependencies-api ./cmd/server
 	bash ./package-scripts.sh linux-arm64 $(VERSION)

README.md

@@ -21,15 +21,26 @@ APP_PORT=50051
 APP_MODE=dev
 APP_DEBUG=false
 
-DB_DRIVER=postgres
-DB_HOST=localhost
-DB_USER=scanoss
-DB_PASSWD=
-DB_SCHEMA=scanoss
-DB_SSL_MODE=disable
-DB_DSN=
+# The serving knowledge base. DuckDB is the production backend.
+DB_DRIVER=duckdb                    # duckdb, sqlite or postgres
+DB_PATH=/var/lib/scanoss/serving.duckdb   # required by duckdb and sqlite; opened read-only
+
+# To serve the same schema from a PostgreSQL instance instead:
+# DB_DRIVER=postgres
+# DB_DSN=postgres://user:pass@host/db?sslmode=disable
+# The DB_* values below are only used to build a DSN when DB_DSN is empty,
+# and only for the postgres driver.
+# DB_HOST=localhost
+# DB_USER=scanoss
+# DB_PASSWD=
+# DB_SCHEMA=scanoss
+# DB_SSL_MODE=disable
 ```
 
+The service reads the serving knowledge base and never writes to it.
+`COMP_COMMIT_MISSING`, which stored live pkg.go.dev lookups back into the mining
+database, is gone; the live lookup itself remains.
+
 
 ## Docker Environment
 

config/app-config-dev.json

@@ -2,7 +2,7 @@
   "App": {
     "Name": "Dependency Service",
     "Debug": true,
-    "Mode":  "dev",
+    "Mode": "dev",
     "GRPCPort": "50051",
     "RESTPort": "40051"
   },
@@ -11,16 +11,10 @@
     "OltpExporter": "0.0.0.0:4317"
   },
   "Database": {
-    "Driver": "postgres",
-    "Host":   "localhost",
-    "User":   "scanoss",
-    "Passwd": "secret123!",
-    "Schema": "scanoss",
+    "Driver": "duckdb",
+    "Path": "/var/lib/scanoss/serving.duckdb",
     "Trace": false
   },
-  "Components": {
-    "CommitMissing": false
-  },
   "TLS": {
     "CertFile": "",
     "KeyFile": ""

config/app-config-docker-local-dev.json

@@ -2,21 +2,15 @@
   "App": {
     "Name": "Dependency Service",
     "Debug": true,
-    "Mode":  "dev",
+    "Mode": "dev",
     "GRPCPort": "50051",
     "RESTPort": "40051"
   },
   "Database": {
-    "Driver": "postgres",
-    "Host":   "host.docker.internal",
-    "User":   "scanoss",
-    "Passwd": "secret123!",
-    "Schema": "scanoss",
+    "Driver": "duckdb",
+    "Path": "/var/lib/scanoss/serving.duckdb",
     "Trace": false
   },
-  "Components": {
-    "CommitMissing": false
-  },
   "TLS": {
     "CertFile": "",
     "KeyFile": ""

config/app-config-prod.json

@@ -2,13 +2,13 @@
   "App": {
     "Name": "Dependency Service",
     "Debug": false,
-    "Mode":  "prod",
+    "Mode": "prod",
     "GRPCPort": "50051",
     "RESTPort": "40051"
   },
   "Database": {
-    "Driver": "sqlite",
-    "Dsn": "/var/lib/scanoss/db/sqlite/dependencies/db.sqlite?cache=shared&mode=memory",
+    "Driver": "duckdb",
+    "Path": "/var/lib/scanoss/serving.duckdb",
     "Trace": false
   },
   "TLS": {

go.mod

@@ -3,14 +3,16 @@ module scanoss.com/dependencies
 go 1.25.0
 
 require (
+	github.com/Masterminds/semver/v3 v3.4.0
 	github.com/golobby/config/v3 v3.4.2
 	github.com/grpc-ecosystem/go-grpc-middleware v1.4.0
 	github.com/guseggert/pkggodev-client v0.0.0-20240318140526-cdb0034504cf
 	github.com/jmoiron/sqlx v1.4.0
 	github.com/lib/pq v1.12.3
+	github.com/marcboeker/go-duckdb/v2 v2.4.3
 	github.com/package-url/packageurl-go v0.1.5
 	github.com/scanoss/go-component-helper v0.6.0
-	github.com/scanoss/go-grpc-helper v0.16.0
+	github.com/scanoss/go-grpc-helper v0.17.0
 	github.com/scanoss/go-purl-helper v0.3.0
 	github.com/scanoss/papi v0.39.0
 	github.com/scanoss/zap-logging-helper v0.4.0
@@ -28,36 +30,51 @@ require (
 
 require (
 	github.com/BurntSushi/toml v1.5.0 // indirect
-	github.com/Masterminds/semver/v3 v3.4.0 // indirect
 	github.com/PuerkitoBio/goquery v1.10.3 // indirect
 	github.com/andybalholm/cascadia v1.3.3 // indirect
 	github.com/antchfx/htmlquery v1.2.4 // indirect
 	github.com/antchfx/xmlquery v1.3.7 // indirect
 	github.com/antchfx/xpath v1.3.6 // indirect
+	github.com/apache/arrow-go/v18 v18.4.1 // indirect
 	github.com/cenkalti/backoff/v5 v5.0.3 // indirect
 	github.com/cespare/xxhash/v2 v2.3.0 // indirect
+	github.com/duckdb/duckdb-go-bindings v0.1.21 // indirect
+	github.com/duckdb/duckdb-go-bindings/darwin-amd64 v0.1.21 // indirect
+	github.com/duckdb/duckdb-go-bindings/darwin-arm64 v0.1.21 // indirect
+	github.com/duckdb/duckdb-go-bindings/linux-amd64 v0.1.21 // indirect
+	github.com/duckdb/duckdb-go-bindings/linux-arm64 v0.1.21 // indirect
+	github.com/duckdb/duckdb-go-bindings/windows-amd64 v0.1.21 // indirect
 	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/go-logr/logr v1.4.4 // indirect
 	github.com/go-logr/stdr v1.2.2 // indirect
+	github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
 	github.com/gobwas/glob v0.2.3 // indirect
+	github.com/goccy/go-json v0.10.5 // indirect
 	github.com/gocolly/colly/v2 v2.1.0 // indirect
 	github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
 	github.com/golang/protobuf v1.5.4 // indirect
 	github.com/golobby/cast v1.3.3 // indirect
 	github.com/golobby/dotenv v1.3.2 // indirect
 	github.com/golobby/env/v2 v2.2.4 // indirect
+	github.com/google/flatbuffers v25.2.10+incompatible // indirect
 	github.com/google/uuid v1.6.0 // indirect
 	github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0 // indirect
 	github.com/kennygrant/sanitize v1.2.4 // indirect
+	github.com/klauspost/compress v1.18.0 // indirect
+	github.com/klauspost/cpuid/v2 v2.3.0 // indirect
+	github.com/marcboeker/go-duckdb/arrowmapping v0.0.21 // indirect
+	github.com/marcboeker/go-duckdb/mapping v0.0.21 // indirect
 	github.com/mattn/go-isatty v0.0.20 // indirect
 	github.com/ncruces/go-strftime v1.0.0 // indirect
 	github.com/phuslu/iploc v1.0.20230201 // indirect
+	github.com/pierrec/lz4/v4 v4.1.22 // indirect
 	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
 	github.com/saintfish/chardet v0.0.0-20120816061221-3af4cd4741ca // indirect
 	github.com/scanoss/go-models v0.8.0 // indirect
 	github.com/scanoss/ipfilter/v2 v2.0.2 // indirect
 	github.com/temoto/robotstxt v1.1.2 // indirect
 	github.com/tomasen/realip v0.0.0-20180522021738-f0c99a92ddce // indirect
+	github.com/zeebo/xxh3 v1.0.2 // indirect
 	go.opentelemetry.io/auto/sdk v1.2.1 // indirect
 	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.65.0 // indirect
 	go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.40.0 // indirect
@@ -68,9 +85,15 @@ require (
 	go.opentelemetry.io/otel/trace v1.45.0 // indirect
 	go.opentelemetry.io/proto/otlp v1.11.0 // indirect
 	go.uber.org/multierr v1.10.0 // indirect
+	golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 // indirect
+	golang.org/x/mod v0.37.0 // indirect
 	golang.org/x/net v0.57.0 // indirect
+	golang.org/x/sync v0.22.0 // indirect
 	golang.org/x/sys v0.47.0 // indirect
+	golang.org/x/telemetry v0.0.0-20260625142307-59b4966ccb57 // indirect
 	golang.org/x/text v0.40.0 // indirect
+	golang.org/x/tools v0.47.0 // indirect
+	golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
 	google.golang.org/appengine v1.6.8 // indirect
 	google.golang.org/genproto/googleapis/api v0.0.0-20260803160001-6ac0973c030d // indirect
 	google.golang.org/genproto/googleapis/rpc v0.0.0-20260803160001-6ac0973c030d // indirect