Skip to content

jev.md

Latest commit

 

History

History
739 lines (625 loc) · 41 KB

jev.md

File metadata and controls

739 lines (625 loc) · 41 KB
Raw

Jev decision inference

← back to model index | 中文

TensorSharp serves a Jev-compatible POST /v1/systemone endpoint with the DiffusionGemma GGUF. A request supplies state and typed questions; the response contains boolean probabilities (noul), categorical choices, and expected scores. State may combine text with images, uploaded files, documents, sampled video frames, and audio transcripts. Attachments can travel inline or reference a file returned by /api/upload (see Files, documents, video and audio). Images use the vision tower; speech uses an explicitly configured transcription service, because DiffusionGemma has no audio tower. jev-latest and jev-preview are API aliases for the loaded DiffusionGemma model, not separate checkpoints or the proprietary hosted Jev model.

The implementation follows the seeded structured-read approach in vLLM PR #57250. It builds an answer canvas with one token slot per question and reads the requested label logits after one denoising step. It performs no output sampling, self-conditioning, JSON generation, or commit forward. This differs from LocalJev, which asks a chat endpoint to generate probability values as JSON and retries malformed output. The Jev developer article describes the typed decision use cases and hosted API; it is not a model-weight or numerical parity specification.

The adapted compiler prompt/template logic is attributed in TensorSharp.Chat/Jev/NOTICE.md, with the Apache-2.0 license included in build, publish and package outputs. Reference revisions reviewed: LocalJev 3f23e36e1a3bff46c7e83e8e3781d3512bc82021; vLLM checkout 0eb42dbbfce96477f4ca174980f4d68336fb1971, including the structured_server.py prototype at commit 1b3b88ec2b7457aa030db4d0e7d8aaf04f6d0fb8.

Start the server

The supplied Q4_K_M checkpoint needs no tokenizer files. Images, scanned documents and video frames need the vision tower, which the configuration downloads once (2.8 GB); text-only decisions do not. --mmproj none on the command line makes the server run text-only and skips the download too, because a command-line --mmproj drops the configuration's entry before it is resolved. From the repository root, in PowerShell:

$env:TENSORSHARP_MODELS = 'C:/Works/models'
$env:DIFFUSION_VRAM_HEADROOM_MB = '4096'
$env:MAX_CONTEXT = '4096'
dotnet run --project TensorSharp.Server.Host -c Release -- --config config/jev-diffusiongemma-q4.json

The configuration binds loopback port 5000 and uses ggml_cuda. Override --backend cpu (pure C#) or --backend ggml_cpu for CPU execution, or --backend ggml_metal on a supported Mac. These are execution options, not claims that every backend has been benchmarked; the CPU measurements are under Running on a CPU. The configuration downloads diffusiongemma-26B-A4B-it-Q4_K_M.gguf from unsloth/diffusiongemma-26B-A4B-it-GGUF on Hugging Face when the local file is missing and reuses it on subsequent launches. It defaults to the repository's models directory; set TENSORSHARP_MODELS to an absolute directory to use a different location. Ordinary chat endpoints remain available on the same server.

Every published GGUF of this checkpoint is text-only and no mmproj was released for it, so the configuration loads the vision tower straight from the upstream model-00011-of-00011.safetensors shard, which carries all 356 of its tensors. It is fetched once and reused. Without it the server still answers text-only requests and refuses requests needing image rows with HTTP 503 rather than answering from filler rows. The same tower serves ordinary DiffusionGemma chat on this server.

The recipe reserves 4 GiB of VRAM for activations on a 16 GiB CUDA GPU. A larger reserve leaves fewer weights resident but can avoid severe paging on longer prompts. Tune DIFFUSION_VRAM_HEADROOM_MB for the device and workload; its model default is 2048 MiB. The 4096-token admission ceiling is not a guarantee that every schema and prompt of that size fits in device memory.

Running on a CPU

Without a GPU, use --backend cpu:

$env:TENSORSHARP_MODELS = 'C:/Works/models'
$env:MAX_CONTEXT = '4096'
dotnet run --project TensorSharp.Server.Host -c Release -- --config config/jev-diffusiongemma-q4.json --backend cpu

DIFFUSION_VRAM_HEADROOM_MB has no effect there. The pure-C# backend keeps the prompt K/V in host memory, so repeated reads of one prompt (fixed samples or adaptive extension) skip the prompt's work (a schema split into chunks has one prompt per chunk, and the model keeps one prompt cache). The last layer runs only for the requested label rows. On cpu and ggml_cpu the server also skips its startup shared-prompt warm-up for this model, so the port opens once the weights are loaded.

Measured with eng/JevProbe on an i7-11800H (8 cores / 16 threads, AVX-512), 32 GB, Windows, diffusiongemma-26B-A4B-it-Q4_K_M.gguf, a 54-token state:

Structured read cpu cpu, previous build ggml_cpu
width 16, new prompt 0.92 s 9.5 s 1.57 s
width 16, same prompt again 0.22–0.25 s 9.5 s 1.57 s
width 64, new prompt 1.42 s not measured 2.68 s
width 64, same prompt again 0.63–0.73 s not measured 2.68 s

On the probe's --quality set (nine two-question prompts at widths 16 and 64) both backends chose all 36 expected labels, with the same top label every time and at most 0.049 between their probabilities. These are probe timings without the HTTP server, extraction or tokenization; the DiffusionGemma card describes the implementation and its switches.

Set MAX_CONTEXT before starting the process to limit the combined tokenized prompt and answer canvas. Without this environment variable, the model's GGUF context limit applies. This setting is environment-only; max-tokens: 256 in the config caps ordinary chat generation, not Jev input length or canvas width. Choose the context limit for the available memory: a checkpoint's declared context does not guarantee that a particular device can execute it. Oversized Jev requests are rejected, not truncated.

Invoke-RestMethod http://127.0.0.1:5000/v1/systemone -Method Post `
  -ContentType 'application/json' -InFile docs/examples/jev-ticket.json |
  ConvertTo-Json -Depth 12

Equivalent curl request from the repository root:

curl http://127.0.0.1:5000/v1/systemone \
  -H 'Content-Type: application/json' \
  --data-binary @docs/examples/jev-ticket.json

If your terminal is already in docs/examples, use --data-binary @jev-ticket.json instead. The @ tells curl to read the file contents. Without it, curl sends the literal text jev-ticket.json, and the server returns Request body must be valid JSON. before running inference.

Request examples

The ticket example asks three questions in one shared canvas forward after prompt prefill: route the ticket to a department, identify an outage, and score its severity. A smaller Python example uses only the standard library:

import json
from urllib.request import Request, urlopen

body = {
    "model": "jev-latest",
    "state": "I was charged twice for the same subscription this month.",
    "questions": {
        "billing": {
            "type": "noul",
            "instructions": "Is this a billing issue?"
        }
    },
    "samples": 1,
    "seed": 42
}
request = Request("http://127.0.0.1:5000/v1/systemone",
                  data=json.dumps(body).encode(),
                  headers={"Content-Type": "application/json"})
with urlopen(request, timeout=180) as response:
    answer = json.load(response)
print(answer["answers"]["billing"]["noul"])

For an in-process .NET application, reference TensorSharp.Chat and use the same validated request and execution gate:

using System.Text.Json;
using TensorSharp.Server;
using TensorSharp.Server.Jev;

using var service = new ModelService();
// Pass the vision shard as mmProjPath to accept requests that carry images;
// null loads the text-only path, which refuses them.
service.LoadModel("C:/Works/models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf",
                  mmProjPath: null, backendStr: "ggml_cuda");
using var json = JsonDocument.Parse(File.ReadAllText("docs/examples/jev-ticket.json"));
object response = await service.JevAsync(JevRequest.Parse(json.RootElement));
Console.WriteLine(JsonSerializer.Serialize(response));

Low-level callers can use DiffusionGemmaModel.ReadStructured(promptTokens, seedCanvas, positions, tokenIds), where positions are zero-based canvas indices and each tokenIds row lists that question's labels. Hold GpuComputeLock through any model access. At that level images are your responsibility: expand the prompt's placeholders, install the encoded spans with DiffusionGemmaModel.SetVisionEmbeddings before the read, and call ClearVisionEmbeddings when the request ends. The service handles tokenization, prompt rendering, image spans, context checks, canvas construction, serialization and lifecycle safety for you.

state may be a string, JSON object, or array. Question names are returned unchanged. Use criteria as a mapping from choice keys to descriptions for choice, an ordered list of descriptions for score, or optional true and false descriptions for noul. Scores use zero-based indices; for example, three levels produce an expected score between 0 and 2.

Request field Default Meaning
instructions none Optional request-wide instructions, added to the system text ahead of the questions
images [] Existing image input: base64 strings or data: URLs
files [] Attachments classified by filename extension; inline data or upload references
documents [] Text/code, PDF, DOCX, XLSX or PPTX attachments
videos [] Videos converted to a bounded set of image frames
audios [] Speech transcribed by the configured companion service
samples "auto" 1 to 32 independent seeded reads, or adaptive reads
auto_max 4 Total reads when adaptive uncertainty triggers, at most 32
auto_threshold 0.1 Conditional-label entropy threshold, in nats
seed 42 Repeatable canvas noise on the same .NET runtime/backend
steps 1 Only one-step structured reads are supported
think 0 Thought generation is not part of this endpoint
chunk_rows server canvas limit Optional maximum canvas width per question chunk, 8 to 4096
chunk_prompt "own" Include each chunk's questions, or "shared" to repeat all questions

Adaptive mode starts with one read and uses auto_max total reads if any question's conditional entropy exceeds the threshold. Fixed reads are averaged as distributions. Use samples: 1 for a fixed minimum-work request; the example sets this explicitly. Multiple reads reuse the same prompt K/V on GGML CUDA, Metal and the pure-C# cpu backend when prompt caching is enabled. Only one prompt cache and label projection are retained per model; switching state or schema replaces the cache.

Requests support up to 64 questions and 2 to 26 alternatives per question. Question IDs must be 1 to 128 characters without colons, control characters or surrounding whitespace. Choice names and score descriptions can contain multiple tokens: the compiler maps them to short labels and verifies that each label occupies one token in the full answer template. Invalid templates and context overflow return validation errors. Conditional question dependencies, sequential question chaining, additional denoising steps, and thought generation are rejected explicitly, each naming its own reason.

The server caps Jev request bodies at 8 MiB, including inline base64 data; TS_JEV_MAX_BODY_MB (1 to 64) sets another limit at startup. TS_JEV_MAX_CANVAS defaults to 64 tokens (also bounded by the checkpoint's canvas width); schemas exceeding this are split into chunks. TS_JEV_MAX_PENDING defaults to 32 admitted requests, including the active request. Excess requests receive HTTP 529 with Retry-After: 1. Invalid requests receive HTTP 422, unknown model names receive HTTP 404, and unavailable/non-diffusion models receive HTTP 503. Malformed JSON receives HTTP 400, oversized bodies receive HTTP 413, and non-JSON media types receive HTTP 415. Model reload and shutdown wait for active inference before disposing weights. Cancellation is checked between managed layers and native dispatches; it cannot interrupt a GPU kernel already executing.

Image input

A request may carry images in images, an ordered array of at most 8 entries. Each entry is a data: URL or bare base64:

{
  "model": "jev-latest",
  "state": "Dashcam frame captured as the vehicle approaches the intersection.",
  "images": ["data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA..."],
  "questions": {
    "signal": {
      "type": "choice",
      "instructions": "Which lamp of the traffic light is lit?",
      "criteria": { "red": "The top lamp is lit", "yellow": "The middle lamp is lit", "green": "The bottom lamp is lit" }
    },
    "stop": { "type": "noul", "instructions": "Must the vehicle stop before entering the intersection?" }
  },
  "samples": 1,
  "seed": 42
}

The traffic-light example is that request with a small synthetic picture already embedded, so it can be posted as-is. Its light is green while the state text says only that the vehicle is approaching, so the answers are reachable only by reading the pixels: on this checkpoint the identical request without the image answers red and "must stop" with high confidence.

curl http://127.0.0.1:5000/v1/systemone \
  -H 'Content-Type: application/json' \
  --data-binary @docs/examples/jev-traffic-light.json

To attach your own image using the existing inline image field:

import base64, json
from pathlib import Path
from urllib.request import Request, urlopen

body = {
    "model": "jev-latest",
    "state": "Photo submitted with an expense report.",
    "images": ["data:image/jpeg;base64," + base64.b64encode(Path("receipt.jpg").read_bytes()).decode()],
    "questions": {
        "legible": {"type": "noul", "instructions": "Is the total amount legible?"},
        "kind": {"type": "choice", "instructions": "What kind of document is this?",
                 "criteria": {"receipt": "A purchase receipt", "invoice": "An invoice", "other": "Anything else"}},
    },
    "samples": 1,
    "seed": 42,
}
request = Request("http://127.0.0.1:5000/v1/systemone",
                  data=json.dumps(body).encode(),
                  headers={"Content-Type": "application/json"})
with urlopen(request, timeout=300) as response:
    answer = json.load(response)
print(answer["answers"]["legible"]["noul"], answer["answers"]["kind"]["choice"])

The same request works in-process: JevRequest.Parse decodes the images and ModelService.JevAsync runs them. The server host points ModelService.MediaStorage at the upload directory it already governs; an in-process caller that leaves it unset gets a directory under the system temporary path, named once in the log.

How images reach the read

Each image renders as one <|image> marker on the user turn, ahead of the state text, and is expanded into [BOI] + soft rows + [EOI] before the context check runs. The row count is the encoder's own output for that picture, not a constant: every picture is resized, up or down, to the largest canvas within the 280-row budget whose sides are multiples of 48 px, so the count depends on its aspect ratio, not its size (the 192x384 traffic-light example is upscaled to 528x1104 and produces 253). Budget roughly 282 prompt tokens of MAX_CONTEXT per image, plus one sentence of system text naming the attachments. The encoded rows are installed on the model immediately before the forward that prefills that prompt. A schema too wide for the canvas is split into chunks, and every chunk prompt carries the images again at its own offsets, so n chunks cost n prefills over the image rows; repeated reads of one chunk (fixed samples or adaptive extension) reuse its prompt K/V as they do for text. diagnostics.images reports the effective number of image spans, including scanned pages and sampled video frames, and usage.input_tokens counts the expanded prompt including the soft rows.

Decoded bytes are written content-addressed into the server's upload directory, so --upload-max-mb, --upload-quota-mb and --upload-ttl-hours govern them, the same picture resent across requests stores one file, and its embeddings are encoded once and cached (TS_MM_EMBEDDING_CACHE_MB, default 512). Image attention inside each soft-token span is bidirectional on sliding layers and causal on global ones; DIFFUSION_IMAGE_BIDIRECTIONAL=0 makes it causal everywhere.

Files, documents, video and audio

files, documents, videos and audios are arrays with the same entry format. Use files for mixed inputs; the other arrays require their named media kind. Each entry has exactly one source:

{"name": "incident.txt", "data": "VGhlIHNlcnZpY2UgaXMgZG93bi4="}

or, after uploading to the same server:

{"file": "SERVER_FILENAME_FROM_UPLOAD.txt", "name": "incident.txt"}

data accepts bare base64 or a base64 data: URL. Inline entries require name with a supported extension. file is the bare filename from the upload response's file property, not its url, a local path or a remote URL. name is optional for upload references and supplies a display name. The server reads only files in its governed upload directory, rejecting paths, traversal and symlinks. References remain usable only while the underlying upload exists; the storage TTL and quota apply.

The document request embeds a short incident report and can be posted directly:

curl http://127.0.0.1:5000/v1/systemone \
  -H 'Content-Type: application/json' \
  --data-binary @docs/examples/jev-document.json

The standard-library attachment client reads local files on the client and either embeds them or uploads them first. It prints the complete decision response, including preprocessing diagnostics:

# Inline a text document.
python docs/examples/jev-attachments.py docs/examples/jev-incident.txt --field documents

# Upload first; only the returned filename is sent in the Jev request.
python docs/examples/jev-attachments.py report.pdf --upload --field documents
# Upload several supporting files together, then evaluate them in one request.
python docs/examples/jev-attachments.py policy.txt request.txt --upload --question "Does the request satisfy the attached policy?"
python docs/examples/jev-attachments.py crossing.mp4 --upload --field videos --question "Is a green traffic light visible?"
python docs/examples/jev-attachments.py incident.wav --upload --field audios --question "Does the speaker report an active service outage?"

For a manual upload, use curl -F "file=@report.pdf" http://127.0.0.1:5000/api/upload, then put the returned file value in the JSON request. /api/upload accepts multipart; /v1/systemone continues to accept JSON only. Uploading avoids base64 overhead and the Jev body limit, but does not bypass attachment, context or storage limits.

For several files, repeat the multipart file field in one upload request:

curl http://127.0.0.1:5000/api/upload \
  -F 'file=@policy.txt' -F 'file=@request.txt'

One file retains the existing { "ok": true, "file": "...", ... } response. Two or more return { "ok": true, "files": [{ "ok": true, "file": "...", ... }, ...] } in multipart order. Each entry includes the same metadata as a single upload. Put every returned filename in the next decision request:

{
  "model": "jev-latest",
  "state": "Evaluate the request using the supplied policy.",
  "files": [
    { "file": "UPLOAD_1.txt", "name": "policy.txt" },
    { "file": "UPLOAD_2.txt", "name": "request.txt" }
  ],
  "questions": {
    "eligible": { "type": "noul", "instructions": "Does request.txt satisfy policy.txt?" }
  },
  "samples": 1
}

The Web UI and TensorAgent batch a file selection this way. Uploads accept at most 32 files per call; Jev's separate limit remains 8 attachments per decision request. The aggregate upload cap is the greater of 500 MiB and the configured per-file cap, with the existing HTTP body limit also applying to multipart framing. Per-file limits and storage quota still apply. A rejected batch returns an error and removes files already stored by that batch; existing uploads remain available. Attachments retain their names and array order in the model's evidence. Across arrays, processing order is files, documents, videos, then audios; use one files array when ordering different media kinds together matters. The two-file example is ready to post with inline data.

Compare separate uploads with a single batch, including real inference:

python eng/jev-attachments-benchmark.py --endpoint http://127.0.0.1:5000 \
  --cases multi-tickets,multi-tickets-reversed,multi-cross-file,multi-eight,multi-text-image \
  --modes inline,upload,upload-batch --concurrency 1,2 --repeats 3

These synthetic cases check source-specific answers, a cross-file policy decision, reversed inputs, eight files, and mixed text/image evidence. Reports retain each upload's HTTP response and timing, full decision diagnostics, and omitted coverage. They are a regression screen, not general model accuracy or a production latency guarantee.

For upload throughput independently of inference, run python eng/multiple-upload-benchmark.py --endpoint http://127.0.0.1:5000 --same-basename. It alternates separate and batched uploads of 1, 2 and 8 CSV files at 64 KiB and 1 MiB each, then verifies every downloaded file's bytes. Download verification and multipart construction are excluded from upload timing. This transport benchmark does not submit the large CSV files to Jev; its text/context limits still apply.

Multiple data types in one request

These ready-to-post examples combine structured state with different media arrays in a single Jev request. All attachment bytes are embedded as base64, so each JSON file is self-contained:

Example Inputs in the same request Decisions illustrated
Text policy + image Structured state, signal-policy.txt in documents, and a PNG in images Identify the green lamp and apply the attached warehouse policy, which requires the barrier to stay closed for green.
PDF + video Structured state, ticket.pdf in documents, and signal.mp4 in videos Route the PDF's non-urgent refund request to billing and identify the green lamp in the video frames.
PDF + image + video + audio Structured state, PDF in documents, PNG in files, MP4 in videos, and WAV in audios Keep source-specific evidence separate: the PDF describes a non-urgent refund, while the audio reports an urgent technical outage. The image and video show a green lamp.

Run these commands from the repository root:

curl http://127.0.0.1:5000/v1/systemone \
  -H 'Content-Type: application/json' \
  --data-binary @docs/examples/jev-multimodal-image.json

curl http://127.0.0.1:5000/v1/systemone \
  -H 'Content-Type: application/json' \
  --data-binary @docs/examples/jev-multimodal-video.json

curl http://127.0.0.1:5000/v1/systemone \
  -H 'Content-Type: application/json' \
  --data-binary @docs/examples/jev-multimodal-audio.json

All three need the vision tower. The video and audio examples also need a working media decoder for the MP4. Before running the audio example, configure TS_JEV_TRANSCRIPTION_URL as described in Configure audio transcription. Audio contributes its speech transcript; video contributes sampled frames without its soundtrack. The questions name the relevant attachments to distinguish each source's evidence.

The original fixture bytes, readable text, hashes and media provenance are in InferenceWeb.Tests/Fixtures/JevAttachments. The video repeats a static image and the audio uses synthetic speech. These examples demonstrate combining inputs; they do not measure temporal video understanding or speech recognition accuracy.

What the model receives

Input Processing and limits
Plain-text/code files UTF-8 text, including CSV, JSON, Markdown and source code, is inserted into the state with its filename. No file tools are executed.
PDF Text layer only, or the largest embedded image per page for wholly image-only documents, with the vision tower. At most 32 pages; failed page extraction is rejected. This is not a general PDF renderer or OCR engine: figures in text PDFs, vector graphics and composited scan layouts are not rendered. Mixed text/scanned PDFs may need to be supplied as page images.
DOCX / XLSX / PPTX Text from document paragraphs, spreadsheet cells or slide text. XLSX uses stored values; it does not recalculate formulas. Formatting, charts and embedded pictures are not rendered. Legacy .doc, .xls and .ppt files are unsupported. Office archives are limited to 2,048 entries and 8 MiB expanded data.
Video MP4, MOV, AVI, MKV or WebM, at most 600 seconds and 16 megapixels per frame; sample at 1 fps and uniformly downselect to at most 4 frames per video within the remaining image budget. Frames enter through the image tower with approximate timestamps. The soundtrack is not transcribed. This cannot guarantee detection of brief events or continuous motion.
Audio MP3, WAV, OGG, FLAC or M4A sent to the configured ASR service; its speech transcript is inserted into the state. Sounds, music, speaker identity, timing and vocal emotion are not direct model inputs. Actual decoding/language support depends on the transcription service.
Image via files Same vision path as images; supported upload image extensions are classified by the file name and checked when decoded.

There may be at most 8 attachments across the four new arrays, at most 32 MiB per attachment, and at most 64 MiB in total decoded/referenced media, including legacy inline images. Images in either images or files must also meet the 16 MiB decoded image limit. At most 8 image inputs may reach the model in total, including explicit images, scanned pages and extracted video frames. Document/transcript text is limited to 32,768 characters per attachment and 65,536 across attachments. Byte, document text and PDF page limits reject excess data; they do not silently truncate it. Videos are deliberately sampled within the remaining image budget. The final expanded prompt plus answer canvas must also fit MAX_CONTEXT. Video frames and scanned PDF images are limited to 16,777,216 pixels each before decoding. Video decoding requires the platform's media provider and a supported codec; an accepted container extension does not guarantee that every codec inside it can be decoded.

The server's --upload-max-mb, --upload-quota-mb and --upload-ttl-hours govern stored attachments and derived images. Inline content is stored by content hash; repeating the same content reuses its stored file. Prepared attachments and image embeddings are cached within bounded budgets. diagnostics.attachments records what was actually prepared: name, kind, textCharacters, imageCount, sampled, cacheHit and warning. cacheHit describes non-audio extraction; it does not expose the companion transcriber's separate cache. Inspect warnings for transformations that limit fidelity. diagnostics.timing.preprocessing_ms is separate from inference_ms; total_ms is their sum and excludes queuing and the prior upload.

Configure audio transcription

DiffusionGemma has no audio weights. Configure an operator-controlled HTTP speech recognizer before starting TensorSharp. A local companion keeps audio on the machine when that service itself runs locally:

# Example: an already running whisper-server service.
$env:TS_JEV_TRANSCRIPTION_URL = 'http://127.0.0.1:8178/inference'
$env:TS_JEV_TRANSCRIPTION_TIMEOUT_SECONDS = '120'
dotnet run --project TensorSharp.Server.Host -c Release -- --config config/jev-diffusiongemma-q4.json

The endpoint must accept multipart fields file and response_format=json, and return a JSON object of at most 1 MiB containing a nonempty text string. For an OpenAI-compatible speech server, set the full URL ending in /v1/audio/transcriptions. TS_JEV_TRANSCRIPTION_MODEL adds the optional model field; TS_JEV_TRANSCRIPTION_API_KEY adds an optional bearer token. The timeout defaults to 120 seconds and accepts 1–600 seconds. Only the operator chooses this URL; requests cannot choose where audio is sent. An in-process application can supply ModelService.JevAudioTranscriber instead. Duplicate audio can reuse the bounded transcript cache (16 entries per service, keyed by content hash and extension; failed transcriptions are not cached).

Without a configured transcriber, audio requests receive HTTP 503, as do service failures or timeouts. Empty speech receives HTTP 422. Neither failure falls back to text-only guesses. ASR accuracy is part of the end-to-end decision quality: validate transcription and decisions in the intended languages and acoustic conditions. A transcript does not establish native audio understanding.

Validation failures

Remote URLs and arbitrary filesystem paths are never fetched or read. Unsupported extensions, invalid base64, malformed documents/media, or exceeded attachment limits receive HTTP 422. Missing model capabilities, such as a required vision tower or audio transcriber, receive HTTP 503. Invalid legacy images also receive HTTP 422 with the decoder's reason. Non-JSON Jev requests receive HTTP 415 and requests above the body limit receive HTTP 413. Storage policy failures retain their declared status (413 for the per-file limit, 507 for the quota).

Probability semantics

For each question, the model computes logits z for its allowed labels and returns softmax(z) at temperature 1 after the model's final logit softcap. These are probabilities conditional on the listed answers and the prepared state: text, extracted document content, transcripts and any encoded image rows. Text extraction, video sampling and transcription can discard evidence before the model reads it. A noul value is the probability of the true label. A choice is the highest-probability option; a score is the probability-weighted mean of its level indices. Choice and score confidence is the largest conditional probability, matching the vLLM prototype. LocalJev instead uses one minus normalized entropy, so its confidence values should not be compared directly. Diagnostics name the conditional entropy semantics and expose read counts, canvas widths, and repeated-read agreement/standard error. A single read has no empirical error estimate.

The adaptive entropy rule differs from the vLLM prototype's entropy over its reported vocabulary entries. TensorSharp does not invent full-vocabulary mass or argmax diagnostics from the smaller conditional distribution. The seed controls reproducibility within TensorSharp; it does not promise Python/NumPy and .NET generate identical noise for the same numeric seed.

The sparse output projection computes only the answer positions and requested vocabulary rows. Its conditional probabilities are mathematically equivalent to selecting the same labels from the full vocabulary distribution and renormalizing. Small floating-point differences can result from different matrix multiplication shapes. It does not compute the total probability mass assigned to all allowed labels, or full-vocabulary entropy.

Quantization, question wording, answer ordering, and the number of reads can affect results. Conditional confidence is not an empirically calibrated probability of correctness. Evaluate on held-out examples from the intended workload before choosing confidence thresholds.

Execution and validation

The answer canvas is sized to the schema, rounded to a 16-token boundary within the model's maximum canvas width. Multiple questions share the transformer forward. The output head operates on requested label rows instead of allocating the full canvas-by-vocabulary logits tensor. ggml_cuda and ggml_metal use the existing DiffusionGemma prompt K/V and fused decode paths, and cpu its host prompt K/V with a last layer computed only for the requested label rows; other backends, ggml_cpu, ggml_vulkan, mlx and cuda included, use the unified prompt-plus-canvas forward. The model execution lock serializes access to shared GPU state with ordinary diffusion chat requests; a read issued while a chat is denoising runs before the chat's next forward instead of after its whole block.

GGML CUDA uses fused prompt attention by default, keeping attention operations inside one native graph to reduce intermediate transfers and explicit KV-head expansion. The DiffusionGemma path preserves exact sequence extents and the existing matrix-multiplication precision policy. It uses materialized attention scores, with no flash attention or persistent attention-graph cache, so prompt memory still grows quadratically. Set DIFFUSION_FUSED_PREFILL_ATTN=0 before launch to restore the per-operation reference path. Other GGML GPU backends can opt in with 1; this does not imply that they have been validated.

Reusable validation tools live in eng/; generated evidence belongs in ignored artifacts/jev/. See the benchmark tool's --help for comparing independently running /v1/systemone endpoints with identical requests. A small smoke set checks integration and obvious prompt conditioning; it does not establish production calibration or broad model quality.

The image path is covered by protocol tests that need no model — what an images entry may be, what it is refused for, content-addressed storage, and the rule that each chunk prompt reads its own installed spans — plus the --image case of the extended smoke, which requires a running server with the vision tower and screens one unambiguous synthetic picture. That screen proves the pixels reached the read; it is not an image-understanding benchmark and says nothing about calibration on visual decisions. Record an independent evaluation before setting confidence thresholds on image input.

The attachment suite uses original fixtures in InferenceWeb.Tests/Fixtures/JevAttachments for text/code, PDF, DOCX, XLSX, PPTX, images, video, audio and mixed evidence. It exercises inline data and the /api/upload → file-reference flow; --modes inline,data-url,upload includes all three transports. Fixture hashes and provenance are recorded in the manifest. The video repeats one still and the audio is synthetic speech: these check ingestion, not temporal understanding or ASR word-error rate. Contrasting documents, concurrent reuse of upload references and a subsequent text-only request check evidence isolation.

The benchmark reports upload, client request/end-to-end, server preprocessing and inference times separately. Its server must have the vision tower and an actual transcription service for full coverage. Unavailable services and skipped cases do not count as passing coverage. --cases txt,pdf,docx can isolate document work, but does not validate all modalities. Set an explicit --max-p95-ms budget for the hardware and workload before claiming a performance pass. Warm repeated fixtures can hit extraction, transcript and embedding caches; measure fresh, representative content separately before drawing production throughput claims.

# Pure protocol/math tests (no model or GPU required).
dotnet test InferenceWeb.Tests -c Release --filter 'FullyQualifiedName~Jev&Requires!=Models&Requires!=Cuda&Requires!=Mlx'

# Python harness checks use mock responses, without model inference.
python eng/tests/jev-benchmark-tests.py
python eng/tests/jev-attachments-benchmark-tests.py

# Full attachment quality/transport screen: server needs vision AND real ASR.
# Add --max-p95-ms with the latency budget chosen for your deployment.
python eng/jev-attachments-benchmark.py --endpoint http://127.0.0.1:5000 --modes inline,data-url,upload --repeats 3 --concurrency 1,2 --description 'Record hardware, model, quantization, backend and ASR service' --output artifacts/jev/attachments

# Real GGUF sparse/full projection comparison, including deterministic reuse.
$env:TS_TEST_MODEL_DIR = 'C:/Works/models'
$env:TS_TEST_BACKEND = 'ggmlcuda'
$env:TS_TEST_GGML_BACKEND = 'cuda'
dotnet test InferenceWeb.Tests -c Release --filter 'FullyQualifiedName~JevStructuredReadTests'

# End-to-end quality smoke, HTTP contract and concurrent isolation.
python eng/jev-benchmark.py --endpoint tensorsharp=http://127.0.0.1:5000 --samples 1 --concurrency 1,2 --out artifacts/jev/http

# Adaptive/multiple reads, chunking, and concurrent ordinary-chat isolation.
python eng/jev-extended-smoke.py --endpoint http://127.0.0.1:5000 --chat --chat-seed 0 --chat-prompt 'What is the capital of France? Answer in one short sentence.' --chat-expected Paris --output artifacts/jev/extended

# Image input, against a server started WITH the vision tower. --image posts the
# traffic-light example and requires its two unambiguous answers; the refusal of
# non-inline image input is checked with or without the flag.
python eng/jev-extended-smoke.py --endpoint http://127.0.0.1:5000 --image --output artifacts/jev/image

# Stop the server first: this probe loads its own copy of the weights.
dotnet run --project eng/JevProbe -c Release -- --model C:/Works/models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --backend ggmlcuda --iterations 5 --warmup 1 --widths 16,64,256 --output artifacts/jev/projection.json

# CPU: label decisions of cpu against a ggml_cpu reference, and repeated reads of one prompt.
dotnet run --project eng/JevProbe -c Release -- --model C:/Works/models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --backend ggml_cpu --quality --widths 16,64 --output artifacts/jev/quality-ggmlcpu.json
dotnet run --project eng/JevProbe -c Release -- --model C:/Works/models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --backend cpu --quality --widths 16,64 --reference artifacts/jev/quality-ggmlcpu.json --output artifacts/jev/quality-cpu.json
dotnet run --project eng/JevProbe -c Release -- --model C:/Works/models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --backend cpu --repeat-reads 4 --widths 16,64 --output artifacts/jev/repeat-cpu.json

The benchmark's original labeled examples are checked in under TensorSharp.TestMatrix/Inputs/jev. Record an independent evaluation for a meaningful quality/calibration claim. Latency percentiles include successful requests only; failures are reported separately and count as incorrect decisions. Add --max-p95-ms and/or --min-requests-per-second to enforce explicit budgets for every measured group. Without these options, a smoke-test pass does not constitute a performance pass. Use --background-words 0,512 to include a longer-context group, and --state-bust to change state on repeated measurements. Keep these settings, context limits and warmup counts identical across compared endpoints.

To compare the unchanged LocalJev engine without launching a second server, install Bun, check out the pinned LocalJev revision, and run the foreground comparator against the running TensorSharp server:

git clone https://github.com/githubnext/localjev artifacts/jev-reference/localjev
git -C artifacts/jev-reference/localjev checkout 3f23e36e1a3bff46c7e83e8e3781d3512bc82021
bun eng/jev-localjev-compare.ts --url http://127.0.0.1:5000 --model diffusiongemma-26B-A4B-it-Q4_K_M.gguf --localjev artifacts/jev-reference/localjev --limit 3 --repeats 1 --server-max-tokens 256 --out artifacts/jev/localjev-comparison

This imports the original Engine and includes prompt construction, inference, JSON validation, and retries in its timing. It omits LocalJev's additional HTTP bridge hop. Both paths receive identical state/questions but use different internal prompts. Record the server's actual token limit; the supplied config caps normal chat at 256 tokens. The report preserves upstream finish reasons, retries and usage so a truncated or failed JSON response is visible.

Comparisons must report hardware, weights and quantization, input lengths, read count, concurrency, warmup, failures, and latency distribution. LocalJev over TensorSharp's chat endpoint measures the cost of the generated-JSON approach on this backend; it is not a comparison against oMLX. The vLLM prototype's published DGX Spark results and LocalJev's M5 Max results cannot be used as same-hardware speed comparisons for a Windows RTX GPU.