Linear Output: scanner loaders, ICE, TIFF gamma, LogLuv decode - #765
Merged
marcinz606 merged 20 commits intoAug 6, 2026
Conversation
added 18 commits
August 5, 2026 23:02
Fourth correction toggle that applies IR-based dust and scratch correction before writing. Uses the file's retouch config (method and threshold). Visible only when the source has an IR channel. Supports both NegPy and OpenICE methods.
TIFF files are now a supported Linear Output source. A manual "Input gamma" dropdown (Linear, 1.8, 2.2, 2.4, 2.6, sRGB, L*, Rec.709) lets the user declare the actual encoding so it can be inverted to linear before export — apps often tag wrong ICC profiles, so auto-detection is unreliable. The chosen gamma is persisted across sessions and recorded in the exported TIFF description metadata. Also fixes a KeyError when exporting a single file (missing hash in the asset dict).
Expansion (Off/2×/4×, default off) scales the normalized pixel data before writing — useful when the source TIFF doesn't use the full bit range. Strip color profiles drops all XMP from the exported TIFF so downstream tools see a completely unmanaged file (no ICC profile, no color-space XMP, no EXIF color tags). Make/model/datetime are kept. Noted in the TIFF description as "profiles stripped".
Linear Output writes the TIFF from scratch: only raw pixels plus Make/Model/DateTime from the source. ICC profiles, EXIF color space, and XMP color metadata from scanner software or editors are never copied through, so a separate toggle was unnecessary.
Parse and normalise variant DateTime formats (dots, dashes) to TIFF- standard YYYY:MM:DD HH:MM:SS. Drop unparseable datetimes rather than failing the export. Wrap all metadata assembly in _write_tiff so any unexpected tag issue degrades gracefully instead of aborting the write.
Detect scanner NEFs by checking for an RGB SubIFD (vs Bayer CFA in camera NEFs). Load the full-res 16-bit RGB data via tifffile, handle ICC profiles and IR channels, apply color space logic identical to TiffLoader. Camera NEFs continue to go through rawpy. Wire into the loader factory, linear output (source type "nef", no expansion), and the export sidebar. 13 new tests cover detection, classification, export roundtrip, and IR extraction.
Detect FFF scanner files by checking for a 16-bit RGB IFD (vs Bayer in .3fr camera-back files). Pick the full-res image by pixel count, not SubfileType tag — the spec confirms that tag is unreliable on real samples. Data is always linear, no expansion needed. Wire into the loader factory, linear output (source type "fff", no expansion), and the export sidebar. 10 new tests cover detection, classification, largest-IFD selection, export roundtrip, and loader data integrity.
Headerless BGR16 LE files from Noritsu EZController (FULL*.RAW). Tiered dimension detection: exact table match (16 known pairs from Negmaster + confirmed samples), then known-height solve (4502/5028/6391). BGR→RGB swap, 12-bit data in 16-bit container, default 16× expansion for linear output (matches chemvert's confirmed x16 bit-shift). Pakon exclusion by file size prevents .raw collision — Pakon sizes (9M/24M/36M/48M/72M) never overlap with Noritsu sizes.
…rlap tests - NEF: add _has_cfa_subifd() to reject camera NEFs that have RGB preview SubIFDs alongside Bayer data (false-positive fix) - NEF: wire gamma_key through _decode_nef() so Linear Output gamma combo applies to Coolscan NEFs, not just TIFFs - Noritsu: add tier-3 open divisor search for files that match no known height — accepts only if exactly one film-plausible (w,h) pair exists - Add Pakon/Noritsu overlap tests documenting the theoretical 1777×4502 collision (below any real scan width, harmless) - Document dead 4-channel branches in NEF and FFF loaders
_decode_tiff only checked for IR as a 4th sample in the main TIFF page, missing SilverFast HDRi TIFFs (IR stored as a separate full-res grayscale page with SubfileType=4) and _ir.tif sidecars. Add fallback searches matching what TiffLoader already does. Skip writing the IR sidecar when ICE is baked into the RGB output.
FFF loader: parse tag 50457 (FlexColor plist) and tag 46279 (firmware) to extract film_stock, film_type, flexcolor_gamma, scan_dpi, scan_date, flexcolor_version, and scanner_serial into the metadata dict. Linear Output exports now carry Make/Model/DateTime from these fields. Linear Output: fix two IR handling bugs found by auditing against the main loader path: - 4-channel TIFFs now check ExtraSamples (via _extract_ir_from_extrasamples) instead of blindly treating the 4th channel as IR — alpha channels are correctly dropped instead of being fed to ICE as fake dust data. - Sidecar IR validity mask is no longer discarded — _read_sidecar_ir already applies the mask internally (invalid pixels set to 1.0).
_decode_tiff, _decode_nef, _decode_fff replaced their hand-rolled dtype-scaling, IR-extraction and geometry logic with a single _decode_via_loader helper that calls the main-path loader with linear_raw=True. Removes ~100 lines of duplicated code that was drifting from the loaders (ExtraSamples handling, sidecar IR, etc.).
Scanner loaders should return data as-is without color-space assumptions. The sRGB→linear conversion was copied from TiffLoader but compromises the ability to recover the original pixel values.
Keep _decode_tiff decoupled from TiffLoader to avoid inheriting its sRGB linearization assumptions. Ports the three IR sources that the old inline version was missing: ExtraSamples check, extra-page search, and sidecar IR with mask.
Neither format supports a separate IR channel — Coolscan bakes ICE into pixel data, Flextight has no IR hardware. Extra channels beyond 3 are now silently dropped instead of being misidentified as IR.
Add Coolscan NEF, Flextight FFF, Noritsu RAW, and generic TIFF to the supported sources list in both USER_GUIDE.md and PIPELINE.md. Document ICE dust removal toggle and input gamma selector. Replace MakeTiff/ColorPerfect references with neutral phrasing.
LogLuv-encoded FFF files (compression 34676/34677) are not yet supported. Detect them early and raise a clear error instead of silently falling through to the generic TIFF path.
Decode LogLuv32 (RLE compressed) and LogLuv24 FFF files instead of rejecting them. The LogLuv → XYZ → linear sRGB pipeline is ported from flexcolor-tool (MIT, attributed in source).
thetalkingdrum
changed the title
added 2 commits
August 6, 2026 04:35
No ICC profiles, EXIF color space tags, or XMP color metadata from the source are copied through — only raw pixels plus device metadata.
LogLuv is an HDR encoding whose raw linear values routinely exceed 1.0; the bare clip(0,1) was silently truncating data instead of normalizing it. Port the normalize_linear step from the flexcolor-tool reference (0.2th/99.8th percentile per channel), which also corrects the per-channel black/gain offset inherent in Flextight CCD data.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
NegPy can now open and process files from every major film scanner — Coolscan NEF, Flextight FFF (including raw LogLuv-encoded files), and Noritsu RAW join Pakon and TIFF as first-class formats in the normal editing pipeline. Linear Output also gains support for all of these, plus TIFF gamma linearization and IR-based dust removal.
Scanner loaders (new formats in the app)
Linear Output
Cleanup
_decode_tiffindependent from TiffLoader to avoid inheriting its sRGB assumptionsTest plan
make allpasses (3414 passed, 1 pre-existing overflow bar failure)Scanner loaders were tested against real scan files where available (Coolscan NEF, Flextight FFF, Noritsu RAW), though the sample set was small. Only uncompressed RGB FFF files were available for testing — no real LogLuv-encoded FFF yet, so that path is validated against synthetic round-trip data only. Format handling is grounded in scanner documentation and the flexcolor-tool reverse-engineering work (which itself mirrors libtiff's reference implementation). More real-world files welcome.
Follow-up fix (738f5e8): Added per-channel percentile normalization to the LogLuv decoder. The original port was missing the
normalize_linearstep from flexcolor-tool — the bareclip(0,1)silently truncated HDR values instead of mapping them into range. The decoder now applies 0.2th/99.8th percentile normalization per channel after XYZ → linear RGB, matching the reference implementation and correcting the per-channel black/gain offset inherent in Flextight CCD data.