Exception dump hook by jsarha · Pull Request #10517 · thesofproject/sof

jsarha · 2026-02-03T22:51:03Z

I have no got all behavioral obscurity sorted out so this should now be as good as it gets. But zephyrproject-rtos/zephyr#103449 should go in before this one.

This is debug_stream exception dump handler. It is able to report fatal exceptions with backtrace through debug_stream using debug window slot and debug_stream.py.

Use spinlock instead of mutex to allow ISR context usage. Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>

src/debug/debug_stream/debug_stream_slot.c

src/debug/debug_stream/debug_stream_text_msg.c

src/include/user/debug_stream_text_msg.h

kv2019i

@jsarha This is missing something, CONFIG_EXCEPTION_DUMP_HOOK is not introduced anwhere (as a kconfig).
set_exception_dump_hook() is not implemented in this PR and I couldn't find it in SOF or Zephyr...?

jsarha · 2026-02-09T23:04:42Z

@jsarha This is missing something, CONFIG_EXCEPTION_DUMP_HOOK is not introduced anwhere (as a kconfig). set_exception_dump_hook() is not implemented in this PR and I couldn't find it in SOF or Zephyr...?

@kv2019i its defined in zephyrproject-rtos/zephyr#103449 , without it this PR does not do much.

jsarha · 2026-02-10T23:44:25Z

The both PRs, this and zephyrproject-rtos/zephyr#103449 , are now updated and now they should work on PTL. The problem was that logging from the exception function hung the core in question, so that even debug_stream records could not be sent. Also the max dump size was too big for PTL.

Set exception dump hooks if CONFIG_EXCEPTION_DUMP_HOOK=y. This enables sending a simple text report of fatal exceptions. To get this working one needs these config options: CONFIG_EXCEPTION_DUMP_HOOK=y CONFIG_EXCEPTION_DUMP_HOOK_ONLY=y CONFIG_SOF_DEBUG_STREAM_SLOT=y CONFIG_SOF_DEBUG_STREAM_TEXT_MSG=y CONFIG_SOF_DEBUG_STREAM_SLOT_NUMBER=2 CONFIG_SOF_TELEMETRY=n CONFIG_SOF_TELEMETRY_PERFORMANCE_MEASUREMENTS=n CONFIG_SOF_TELEMETRY_IO_PERFORMANCE_MEASUREMENTS=n If system hangs and an the exception is reported successfully the report can be seen with debug_stream.py (which should be installed in the same directory with cavstool.py). It does matter if the tool was not running at the time. The report should be available there in the debug slot window for debug_stream.py to decode as long as system remains up. The report should looks something like this: CPU 2: FATAL EXCEPTION CPU 2 EXCCAUSE 63 (zephyr exception) PC 0xa003048b VADDR 0xa0031020 PS 0x60220 (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:2 CALLINC:2) A0 0xa006c2b8 SP 0xa00dd160 A2 0x4 A3 0xa00dd170 A4 0xa00dd150 A5 0x4 A6 0x6 A7 0x4 A8 0xa006c109 A9 0xa00dd0f0 A10 0xa00a697d A11 0xa00dd170 A12 0xa00dd150 A13 0x4 A14 0x401a6b24 A15 0x401a6b24 LBEG 0xa0044323 LEND 0xa0044330 LCOUNT 0xa006de66 SAR 0x4 THREADPTR (nil) BT 0xa0030488:0xa00dd160 CORRUPTED Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>

Skip useless " ** " prefix from exception dumps to save byte is in the debug_stream buffer. Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>

The utf-8 decoder does not appear to be sensitive to '\0' in the buffer so cut from word boundary may produce unwanted characters at the end of the message. Make also the text message always star from the beginning of the line and not from the same line with "CPU X:" tag. Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>

Enabling Zephyr exception hook directs Zephyr exceptions dumps to debug_stream text message. Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>

Copilot

Pull request overview

Adds a debug_stream-based exception dump handler so fatal exceptions (including backtraces) can be forwarded to the host via the debug window slot and decoded by debug_stream.py.

Changes:

Update host-side debug_stream.py text message decoding/printing to handle NUL-terminated payloads and multi-line output.
Add a Zephyr set_exception_dump_hook() implementation that aggregates exception text into a debug_stream text record.
Switch debug_stream slot record send locking from a mutex to a spinlock and enable relevant overlay Kconfig options.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
tools/debug_stream/debug_stream.py	Improves text record decoding (NUL termination, UTF-8 replace) and prints messages as multi-line.
src/debug/debug_stream/debug_stream_text_msg.c	Adds exception dump hook that formats exception output into a debug_stream text record.
src/debug/debug_stream/debug_stream_slot.c	Replaces `k_mutex` with `k_spinlock` for record send synchronization.
app/debug_stream_overlay.conf	Enables exception dump hook-related Kconfig options for the debug_stream overlay.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Copilot · 2026-03-03T22:31:32Z

app/debug_stream_overlay.conf


+# Enable Zephyr exception printing hook, debug stream is sensitive to this option too
+CONFIG_EXCEPTION_DUMP_HOOK=y
+# Do not try to sen the exception prints through logs, this causes probles on PTL with mtrace


Typo in comment: "sen" → "send".

Suggested change

# Do not try to sen the exception prints through logs, this causes probles on PTL with mtrace

# Do not try to send the exception prints through logs, this causes problems on PTL with mtrace

Copilot · 2026-03-03T22:31:33Z

app/debug_stream_overlay.conf


+# Enable Zephyr exception printing hook, debug stream is sensitive to this option too
+CONFIG_EXCEPTION_DUMP_HOOK=y
+# Do not try to sen the exception prints through logs, this causes probles on PTL with mtrace


Typo in comment: "probles" → "problems".

Suggested change

# Do not try to sen the exception prints through logs, this causes probles on PTL with mtrace

# Do not try to sen the exception prints through logs, this causes problems on PTL with mtrace

Copilot · 2026-03-03T22:31:33Z

src/debug/debug_stream/debug_stream_text_msg.c

+static struct {
+	struct debug_stream_text_msg msg;
+	char text[640];
+} __packed ds_buf;
+static int reports_sent_cpu[CONFIG_MP_MAX_NUM_CPUS];
+static size_t ds_pos;


ds_buf and ds_pos are global across all CPUs, but reports_sent_cpu[] is per-CPU. If two cores hit an exception around the same time, both can write into the same ds_buf.text/ds_pos concurrently before reports_sent_cpu[arch_proc_id()] is incremented in ds_exception_drain(), producing a data race and corrupted/merged dumps. Make the buffer/position per-CPU (e.g., arrays indexed by arch_proc_id()) or protect updates with a lock that is safe in exception context.

Copilot · 2026-03-03T22:31:33Z

src/debug/debug_stream/debug_stream_text_msg.c

+static struct {
+	struct debug_stream_text_msg msg;
+	char text[640];
+} __packed ds_buf;


ds_buf is declared __packed, which drops its alignment to 1. Because ds_buf.msg.hdr contains 32-bit fields that are accessed directly and passed to debug_stream_slot_send_record(), this can lead to unaligned accesses if the linker places ds_buf at a byte-aligned address. Consider removing __packed here and/or forcing at least 4-byte alignment for ds_buf.

Suggested change

static struct {

struct debug_stream_text_msg msg;

char text[640];

} __packed ds_buf;

static _Alignas(4) struct {

struct debug_stream_text_msg msg;

char text[640];

} ds_buf;

debug: debug_stream_slot: Use spinlock instead of mutex for ISR

ba001a4

Use spinlock instead of mutex to allow ISR context usage. Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>

jsarha force-pushed the exception_dump_hook branch from b3c5f35 to fd97e52 Compare February 3, 2026 23:00

lyakh reviewed Feb 5, 2026

View reviewed changes

kv2019i reviewed Feb 6, 2026

View reviewed changes

jsarha force-pushed the exception_dump_hook branch from fd97e52 to 1e537a2 Compare February 10, 2026 23:39

Jyri Sarha added 4 commits March 3, 2026 23:44

debug_stream: text_msg: exception: Skip useless " ** " prefix

bbc7cd5

Skip useless " ** " prefix from exception dumps to save byte is in the debug_stream buffer. Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>

app: debug_stream_overlay.conf: Enable exception hook

ded889b

Enabling Zephyr exception hook directs Zephyr exceptions dumps to debug_stream text message. Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>

jsarha force-pushed the exception_dump_hook branch from 1e537a2 to ded889b Compare March 3, 2026 22:18

jsarha mentioned this pull request Mar 3, 2026

Exception dump hook zephyrproject-rtos/zephyr#103449

Open

jsarha marked this pull request as ready for review March 3, 2026 22:26

jsarha requested a review from lgirdwood as a code owner March 3, 2026 22:26

Copilot AI review requested due to automatic review settings March 3, 2026 22:26

jsarha requested review from dbaluta, lbetlej, mmaka1 and plbossart as code owners March 3, 2026 22:26

Copilot started reviewing on behalf of jsarha March 3, 2026 22:28 View session

Copilot AI reviewed Mar 3, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exception dump hook#10517

Exception dump hook#10517
jsarha wants to merge 5 commits intothesofproject:mainfrom
jsarha:exception_dump_hook

jsarha commented Feb 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kv2019i left a comment •

edited

Loading

Uh oh!

jsarha commented Feb 9, 2026 •

edited

Loading

Uh oh!

jsarha commented Feb 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 3, 2026

Uh oh!

Copilot AI Mar 3, 2026

Uh oh!

Copilot AI Mar 3, 2026

Uh oh!

Copilot AI Mar 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	# Do not try to sen the exception prints through logs, this causes probles on PTL with mtrace
	# Do not try to send the exception prints through logs, this causes problems on PTL with mtrace

Conversation

jsarha commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kv2019i left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsarha commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsarha commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jsarha commented Feb 3, 2026 •

edited

Loading

kv2019i left a comment •

edited

Loading

jsarha commented Feb 9, 2026 •

edited

Loading

jsarha commented Feb 10, 2026 •

edited

Loading