Skip to content

Emit metrics related to binlog processing - #52

Merged
forge33 merged 3 commits into
masterfrom
emit_binlog_event_metrics
Jul 27, 2026
Merged

Emit metrics related to binlog processing#52
forge33 merged 3 commits into
masterfrom
emit_binlog_event_metrics

Conversation

@forge33

@forge33 forge33 commented Jul 6, 2026

Copy link
Copy Markdown

Related issue: https://github.com/Shopify/schema-migrations/issues/5838

Description

This PR adds metrics around binlog processing so we can see what gh-ost is reading, forwarding, and filtering while streaming row events. Metrics follow similar patterns as backfilter's binlog tailer/reader.

Metric Type Tags When
binlog_events count type:processed, table:<name> Every rows event read from the binlog
binlog_rows count type:processed, table:<name>, binlog_event_type:insert|update|delete Same
binlog_events count type:filtered, table:<name> Rows event on a table that is not the migration table or changelog table
binlog_rows count type:filtered, table:<name>, binlog_event_type:... Same
binlog_events count type:consumed, table:<name>, binlog_event_type:... Rows event on the original table or _ghc changelog table
binlog_rows count type:consumed, table:<name>, binlog_event_type:... Same
binlog_rows_in_event histogram table:<name> Row count per consumed rows event
transaction_num_row_events histogram End of transaction: consumed row events on relevant tables (flush on next GTID, or on XID in file-position mode)
transaction_num_rows histogram End of transaction: logical rows in those consumed events
gtid_event_transaction_length_bytes histogram Each GTID event (TransactionLength, MySQL 8.0.2+)
unfiltered_commit_group_size gauge When LastCommitted changes (SequenceNumber - LastCommitted)
binlog_streamer_blocked_on_out_channel_ms histogram Time blocked sending a consumed rows event to the events channel
Screenshot 2026-07-14 at 3 02 52 PM

In case this PR introduced Go code changes:

  • contributed code is using same conventions as original code
  • script/cibuild returns with no formatting errors, build errors or unit test errors.

@forge33
forge33 requested review from coding-chimp and grodowski July 6, 2026 18:13
@forge33
forge33 force-pushed the emit_binlog_event_metrics branch from 403ed69 to 0691176 Compare July 14, 2026 17:30
forge33 and others added 2 commits July 14, 2026 13:38
Build and golangci-lint failed because emit_test.go referenced histogramCountSpy without defining it.

Co-authored-by: Cursor <cursoragent@cursor.com>
Record counts on the existing histogramSpy instead of adding a duplicate spy type.

Co-authored-by: Cursor <cursoragent@cursor.com>
@forge33
forge33 merged commit 007bc01 into master Jul 27, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants