Skip to content

feat(hook): capture image blocks as markers and opt-in Langfuse media - #56

Open
milanagm wants to merge 4 commits into
langfuse:mainfrom
milanagm:lfe-14888-are-image-blocks-intentionally-excluded-from-traces-turns
Open

feat(hook): capture image blocks as markers and opt-in Langfuse media#56
milanagm wants to merge 4 commits into
langfuse:mainfrom
milanagm:lfe-14888-are-image-blocks-intentionally-excluded-from-traces-turns

Conversation

@milanagm

@milanagm milanagm commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Problem

The hook silently dropped pasted images from trace and generation inputs while the [Image #1] placeholder stayed in the text. The trace looked as if it contained an image, but it did not.
At the same time Tool-result screenshots (browser, simulator) caused more damage. The hook serialized their base64 into the tool span output. Truncation at CC_LANGFUSE_MAX_CHARS cut it to ~20k characters of unusable base64. This destroyed all text blocks after the image. The damaged output also went into the next generation input.

Change

  1. Markers, always on. Each image block becomes a short text marker, for example [image image/png ~290KB]. This applies to message content and to tool_result content, in the main session and in subagents. The hook does not drop images. Base64 does not reach the payload. Text after an image stays complete.
  2. Capture, on by default. The hook uploads images with LangfuseMedia (CC_LANGFUSE_CAPTURE_IMAGES, default true; wizard option included). Pasted images attach to the turn root input. Tool screenshots attach to the tool span output. The hook validates the base64 itself and passes content_bytes to the SDK. This is necessary: the SDK does not raise on bad input and emits a hollow object as a garbage string. Each failure path falls back to the text marker. Image handling cannot stop an emission. Self-hosted deployments without media upload support must set the option to false.
  3. Watertight opt-out. Set CC_LANGFUSE_CAPTURE_IMAGES=false to disable the upload. The hook then also sets LANGFUSE_MEDIA_UPLOAD_ENABLED=false. This stops the SDK from uploading base64 it finds in raw tool inputs. If you set that variable yourself, the hook does not change it.

Verification

  • 183 unit tests pass. They also pass with CC_LANGFUSE_CAPTURE_IMAGES or LANGFUSE_MEDIA_UPLOAD_ENABLED present in the environment.
  • E2E tests ran against self-hosted Langfuse (default docker-compose, minio) and Langfuse Cloud. The media round-trips are byte-identical. The server deduplicates by content: the same bytes give the same media id. The emitted payloads contain the markers and the text after each image.
  • A live A/B session test: before, the image was missing from the trace input. After, the trace shows the marker and the rendered image.

Deliberate scope decisions

  • Tool-screenshot media also appears in the next generation input. It uses the same media id, so each image uploads once. This shows what the model saw.
  • Subagent span inputs get markers only so the media is not duplicated there.

Follow-up ideas (separate issues)

  • tool_use.input still goes into tool span inputs as a raw, unbounded dict. The hardening stops the upload only. A marker pass for tool inputs changes the payload shape and needs its own issue.
  • get_tool_result_text still loads full base64 into memory for the legacy async-launch check. This costs performance only.

Closes #52
Fixes LFE-14888

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Are image blocks intentionally excluded from traces? (turns with screenshots capture text only)

1 participant