Skip to content

chore: sync internal changes (py314 + Kafka Avro message-key)#1205

Open
yokofly wants to merge 7 commits into
developfrom
chore/internal-sync-final
Open

chore: sync internal changes (py314 + Kafka Avro message-key)#1205
yokofly wants to merge 7 commits into
developfrom
chore/internal-sync-final

Conversation

@yokofly

@yokofly yokofly commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

Two commits onto develop:

  1. chore: sync internal changes — squashed internal sync.
  2. fix(rebase): reconcile py314 + Kafka Avro-key after develop rebase — post-rebase integration fixes so the tip builds against origin/develop, which now carries py314 (migrate embedded CPython to 3.14 with optional free-threaded build #1200) and the Kafka Avro message-key feature (feat(kafka): support Avro-encoded message key via schema registry #1201): ProfileEvents macro continuation, PythonStreamingSource void doCheckpoint signature, and reconstruction of the Avro-key integration onto the parallel Kafka source.

Kafka Avro message-key — broker-backed smoke

Verified end-to-end against a live Redpanda broker + Confluent-compatible schema registry (proton 3.0.26). The read path fetches the key schema by id (GET /schemas/ids/{id}) and emits the decoded record as a JSON string into _tp_message_key; the write path for Avro-encoded keys throws NOT_IMPLEMENTED (read-only) while plaintext-key writes still work.

# Scenario Result
1 Value decode (JSONEachRow) ✅ 3 rows
2 Raw _tp_message_key tenant-a, tenant-b
3 Avro key via schema registry ✅ decoded → {"order_id":42,"region":"us-west"}, {"order_id":99,"region":"eu-central"}
4a registry-url + JSON value + no key schema INVALID_SETTING_VALUE
4b message_key_schema_name + no registry-url INVALID_SETTING_VALUE
4c non-String _tp_message_key + avro key ILLEGAL_COLUMN
5 Write Avro-encoded key NOT_IMPLEMENTED
+ Raw-key write round-trip ✅ produced key=written-key

Other verification

  • unit_tests_dbms Avro / external-stream suites: 33/33 pass (AvroDeserializer, AvroSchemaSerializer, ProtobufMapEntry, ExternalStreamSource, ExternalStreamSettings).
  • Full build green (proton + unit_tests_dbms).

🤖 Generated with Claude Code

yokofly and others added 2 commits July 1, 2026 22:12
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Post-rebase integration fixes so the tip builds against origin/develop
(which now carries py314 #1200 and Kafka Avro-key #1201):

- ProfileEvents.cpp: restore macro continuation between py314 and our events
- PythonStreamingSource.{h,cpp}: use the void doCheckpoint signature that
  matches the Streaming::ISource base (develop's py314 returned Chunk)
- Kafka.cpp: reconstruct the Avro-key integration onto the parallel Kafka
  source (validateMessageKeyColumnType two-arg form, schema-registry checks,
  key schema-registry setup, write-path NOT_IMPLEMENTED)
- KafkaSource.cpp: drop an orphaned decodeAvroKey re-insert; the typed
  String-key handler already integrates it

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@yokofly yokofly marked this pull request as ready for review July 2, 2026 06:42
yokofly and others added 5 commits July 2, 2026 00:07
The C23 `#embed` migration raised the clang floor to 19 (cmake/tools.cmake),
but the macOS x86_64 cross-build in release_build.yml and manual_trigger_build.yml
still pinned clang-18 — which cannot compile `#embed` and fails the new floor.

Bump those jobs to clang-21 (image + `--compiler clang-21-darwin` + llvm-strip),
matching the Linux and macOS-arm64 jobs that already use clang-21. The packager
already supports the clang-21-darwin cross-target. These workflows only run on
workflow_dispatch / create_release, so this had no effect on PR CI.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
receivePacketsExpectData used `if (!in->poll(timeout_us))`, so a poll
timeout that had not yet exceeded receive_timeout fell through to a
blocking readVarUInt — SOCKET_TIMEOUT never fired and a slow or idle
client during data reception (e.g. a streaming INSERT) hung the
connection instead of timing out. Re-poll with `while` until data
arrives or receive_timeout elapses and the timeout is thrown.

Fold in the matching cancel-handling cleanup so the receive paths line
up:
- processCancel's `throw_exception` parameter defaulted to true and no
  caller passed false, so the non-throwing branch was unreachable; drop
  the parameter and always throw QUERY_WAS_CANCELLED_BY_CLIENT.
- remove the unreachable `throw ABORTED "Server shutdown is called"` at
  the tail of receivePacketsExpectQuery (every switch case already
  returns or throws).
- only log "the connection is preserved" when the connection is in fact
  kept, at TRACE level.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ExternalStreamSource base dropped the stored `format_executor` member
and `initInputFormatExecutor()` in favor of `getInputFormatExecutor()`
(which returns a {regular, batch} executor pair) and hoisted
`request_virtual_columns` into the base. Kafka/Pulsar/Python were migrated
to the new API but NATSJetstream was left on the old one, breaking the
build under -Weverything:

  - request_virtual_columns shadowed the base member (-Werror,-Wshadow-field)
  - initInputFormatExecutor / format_executor no longer exist (hard errors)

Mirror the Pulsar migration: drop the shadowing member, hold the regular
executor as a derived-class member, and call getPhysicalHeader() explicitly
before getInputFormatExecutor() (the old init call did this internally).
NATS does not batch-parse, so only the regular executor is kept.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The internal sync added stateless tests that exercise features not built
in this edition, so they fail here:

- 99922_streaming_left_anti_join and
  99923_emit_on_update_filtered_retract_partner use CREATE MUTABLE STREAM
  (unsupported) -> SYNTAX_ERROR
- 99300_mv_oom_recovery_phantom queries system.local_system_states
  (absent) -> UNKNOWN_STREAM
- 99923_propagate_shuffle_through_cte asserts a distributed-planner
  behavior that differs here -> result mismatch

Remove those four, and revert 99050_storage_policy_and_disk to the develop
baseline (the sync had added a SHOW CREATE DISK after-drop case expecting
UNKNOWN_DISK, which does not error in this build).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The internal sync added two external-stream smoke cases that fail here for
test-fidelity reasons (the streaming functionality itself is correct — both
were reproduced green locally):

- 17_pulsar_batch_parsing_basic used toString() in an INSERT ... SELECT,
  which is a ClickHouse spelling; this build exposes it as to_string()
  -> UNKNOWN_FUNCTION. Switch to to_string().

- 44_kafka_parallel_parsing_streaming ran two streaming consumers that
  default to seek=latest, then produced after only a 1s wait, so messages
  could be sent before both consumers had subscribed/positioned. Give the
  consumers 5s to subscribe before the first produce, and 3s (was 1s) after
  each produce for delivery + parse.

The remaining smoke reds (4_update_versioned_kv... and the malformed
0013_changelog_stream10/00_add_rows_to_multi_shard... case) are byte-identical
to develop, involve multi_shard, and are unrelated to this change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@yokofly yokofly marked this pull request as draft July 3, 2026 01:10
@yokofly yokofly marked this pull request as ready for review July 3, 2026 01:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant