Skip to content

stage1: bootstrap GHC Stage 1 with MicroHs (zero BOOT_GHC) + Stage 2 CI demo - #207

Draft
angerman wants to merge 42 commits into
stable-ghc-9.14from
feat/stage1-mhs
Draft

angerman wants to merge 42 commits into
stable-ghc-9.14from
feat/stage1-mhs

Conversation

@angerman

Copy link
Copy Markdown

Summary

Bootstraps GHC 9.14 Stage 1 and all eight Stage 1 tools natively with MicroHs 0.16.12.0 (zero preinstalled ghc, happy, or alex required), fixes __MHS__ GHC.Utils.Fingerprint performance using MicroHs's runtime md5Array, removes the gen-parsers.sh Python post-processor via portable Haskell 2010 happy/alex backends, and adds the end-to-end MicroHs -> Stage 1 -> Stage 2 GHC GitHub Actions demo workflows.

Stacked on #205 (feat/stage1-drop-hpc-time).

Key Changes

  1. Zero-BOOT_GHC Stage 1 toolchain & tools (bootstrap/mhs/):
    • Builds all eight Stage 1 tools natively (unlit, happy, alex, genapply, deriveConstants, genprimopcode, ghc-pkg, hsc2hs) with cc + mhs.
    • Adds Curated C11 POSIX tool helpers (bootstrap/mhs/cbits/mhs_tool_posix.{c,h}) and minimal bootstrap/mhs/tool-shims/ for hsc2hs and ghc-pkg.
    • All GHC source edits (compiler/, ghc/, libraries/, utils/) are guarded under #if defined(__MHS__) and verified 100% byte-identical when __MHS__ is undefined.
  2. RFC 1321 md5Array GHC.Utils.Fingerprint fix:
    • Replaces [Word8] list allocation + FNV-1a with MicroHs's C runtime md5Array (558x faster on 1 MB, 1,008x faster on 8 MB, bit-identical to GHC).
  3. Portable Haskell 2010 happy & alex (zero parser post-processor):
    • Drops mhsify_happy, python3, /nix/store/* pins, and grep -v ghcconfig.h from bootstrap/mhs/gen-parsers.sh.
    • Companion forks on github:stable-haskell:
      • stable-haskell/MicroHs (feat/ghc-stage1 @ 29504a98, atomic series feat/ghc-stage1-series @ 9ae73f34)
      • stable-haskell/happy (feat/portable-haskell2010 @ 6d2b064)
      • stable-haskell/alex (feat/portable-bootstrap @ d8b3768)
      • stable-haskell/hsc2hs (feat/mhs-bootstrap @ 0c7f294)
  4. Demo GitHub Actions workflows & atomic MicroHs patch series:
    • bootstrap/mhs/patches/microhs/0001..0008-*.patch + MANIFEST.tsv (extracted via git format-patch from stable-haskell/MicroHs feat/ghc-stage1-series, referencing augustss/MicroHs issues #367, #446, #226, #449, #501, #502, #365, #278, #294, #18 and adjacent PRs #567, #559).
    • .github/workflows/mhs-stage2-demo.yml: 8-phase DAG across ubuntu-24.04 and macos-latest (super-charge MicroHs -> build 8 tools -> concurrently super-charge/build cabal-install & build Stage 1 GHC -> build Stage 2 GHC via cabal.project.stage2.mhs -> package bindist + full timing summary -> invoke downstream app workflow).
    • .github/workflows/mhs-stage2-demo-app.yml + bootstrap/mhs/demo-app/: dependent downstream workflow downloading the Stage 2 artifact onto a clean runner and building/running a multi-module Haskell app.
    • bootstrap/mhs/TODO.md: full roadmap & upstreaming tracker.

Local Verification

Verified end-to-end on aarch64-darwin under a sanitized PATH asserting ! command -v ghc && ! command -v happy && ! command -v alex:

  • Toolchain, all 8 tools, all 8 generated parsers/lexers, compiler checkpoints 00..08, Stage 1 ghc link, ghc-pkg DB round-trip (init/register/recache/list/field/check), and Stage 1 native AArch64 codegen (HelloStage1.hs -> .s/.o/.hi) all pass.
  • Live Stage 2 compilation (Phase 7) is being exercised in CI via this draft PR.

angerman and others added 25 commits November 3, 2026 11:54
Stage1 already passes -hpc. The compiler still depended on the hpc
library, so Cabal built hpc-0.7.0.2. The tick pass keeps a local
BoxLabel when HAVE_HPC is off, and the hpc dependency sits under
flag(hpc). Stage2 still pins hpc.

time-1.14 was rebuilt on every platform. It is only required on
Windows, where the bootstrap time package is wired to Win32-2.13.
On Linux, Darwin, and FreeBSD the bootstrap time-1.12.2 satisfies
the compiler, directory, and pinned Cabal bounds. The tarball has
to stay off the package list there: Cabal prefers the local 1.14
whenever the tarball is visible, even without `time source`.
The stage1 cabal invocation was one Make phase. Stamp that log and
draw each component from the install plan.

Without a log the bars are dependency depth, one unit per built
component, not wall-clock time. A stamped log adds a second chart
in seconds. Components the log never mentions stay off that axis.

`make stage1-build-graph` redraws the latest plan. The stage1
recipe stamps the log and renders after a successful build.
The plan chart draws a bar from start to end and an edge to every built
dependency. A thick edge is the one that finishes last. A stamped cabal
log uses the same picture in seconds from the first line.
cabal.project.stage1-tools builds alex, happy, genprimopcode,
deriveConstants, ghc-toolchain, hsc2hs, genapply, and unlit with the
bootstrap directory, filepath, process, and unix. The compiler plan
sets -build-tool-depends and no longer lists those tools or libffi-clib.

process-1.6.29 stays. Installed process-1.6.25.0 is wired to installed
directory-1.3.8.5, and the compiler plan builds directory 1.3.10.
A Darwin dry-run (Cabal 3.17, GHC 9.8.4) selected the bootstrap units
for the tools and process-1.6.29 for the compiler. No full compile, so
no measured speedup.
The Buck2 job on fed9ab5 stopped in mk/buck2-stage1-tools.py:
no stage1-tools target for alex, happy. cabal buck2 does not emit a
local rule for a Hackage executable. It writes export_file(name =
'alex-exe') into third-party/haskell-stage1-tools, and genprimopcode's
build_tool_depends names that cell. The script now copies the tools.bzl
paths and checks for those exports.
Buck2 linked unlit-stage1-tools with g++ against the bootstrap base
shared library and reported undefined stg_* symbols. rts-fs is a C
library built as haskell_library. GHC's automatic base link does not
pull libHSrts, and the stage1-tools select arm had removed the boot
rts that the default platform adds. The arm now names
@third-party-haskell//:rts. base stays on the aliased variant target.

The Windows compiler plan failed Cabal-7107: Win32-2.14.2.1
build-tool-depends on hsc2hs:hsc2hs, and the tools executable on PATH
is not that package. The same pin is in cabal.project.stage1 on
Windows only. A Darwin dry-run of exe:ghc stayed at the previous 13
components. A second dry-run that also asked for exe:hsc2hs solved
hsc2hs-0.68.10 against directory 1.3.10 and process 1.6.29.
cabal buck2 --variant stage2 does not build alex or happy. It reads
third-party/haskell/BUCK for <tool>-exe and puts those directories on
the configure search path. The stage1 compiler plan no longer builds
them, so that file had no alex-exe and configure stopped with
Cabal-7620 (alex >= 3.2.6). Copy the stage1-tools export_file calls
into the base file. The src path goes through a symlink in the base
cell, the same shape as Cabal's own cabal-store link.

The tools.bzl rewrite is unchanged. A base file that already exports
the tool is left alone. mk/buck2-stage1-tools.py --self-test covers
the readBaseTools line pairing, the symlink, and a second run.
The compiler build needs alex, happy, genprimopcode, and
deriveConstants before configure, so it still waits for the tools
plan. ghc-pkg is a separate project and is only copied into the
stage1 store after exe:ghc. Start it with the tools and let it
finish during the compiler build. Each plan writes its own phase
end, so the ghc-pkg timing stays the length of that build.
Compile genapply and deriveConstants with mhs 0.16.12, and typecheck
the ghc-boot closure in one mhs run. cabal-install stays the MicroHs
build from stable-haskell/cabal PR 533.

The compiler library stops at the associated type in
GHC.Utils.Outputable.IsDoc. MicroHs has no type families.
…iver

Add the self-contained MicroHs 0.16.12.0 bootstrap infrastructure for
building GHC Stage 1 tools, ghc-boot, the full Stage 1 compiler library,
and the Stage 1 ghc executable:

- microhs-stage1.patch: extend MicroHs 0.16.12.0 (9e1cd08) with open/closed
  and associated type families, higher-kinded family arity, GADTs, cyclic
  superclass equality orientation, .hs-boot record-selector link aliases,
  polymorphic foreign import EForall desugaring + T_IO_CCALL guard, 512 MB
  stack sizing, and two's-complement Int/Int64 wrapping (WANT_OVERFLOW=0).
- cbits/{mhs_posix,mhs_rts_stubs}.{c,h}: portable Curated C11 POSIX and
  GHC RTS/IEEE-754 symbol shims.
- shims/ and include/: minimal base/ghc-internal/directory/bytestring
  compatibility shims and pre-generated primop .hs-incl headers.
- gen-parsers.sh: generate MicroHs-compatible Happy 2.1.5 and Alex 3.5.4.0
  parsers and lexers without MagicHash/BangPatterns templates.
- build-stage1-mhs.sh: staged checkpoint driver (toolchain, tools,
  ghc-boot, compiler 00..08, ghc, test, all) including DerivedConstants.h
  and rts package-db staging plus native codegen smoke tests.
…ies, and utils

Add #if defined(__MHS__) guards so MicroHs 0.16.12.0 compiles all six
Stage 1 tools (unlit, genapply, deriveConstants, genprimopcode, ghc-pkg,
hsc2hs), ghc-boot, the full Stage 1 compiler library (GHC), and ghc/Main.hs
while preserving non-__MHS__ GHC semantics:

- Replace GHC-specific unboxed primitives, DeriveFunctor on GADTs/existentials,
  StandaloneDeriving on refined indices, and GeneralizedNewtypeDeriving over
  reader/continuation wrappers with explicit __MHS__ implementations.
- Route external tool execution in GHC.SysTools.Process through POSIX spawn
  shims when built with MicroHs.
- Normalize non-breaking spaces in comments/pragmas for cpphs compatibility.
…post-processor

Complete three cleanups for the MicroHs 0.16.12.0 GHC Stage 1 bootstrap:

1. Eliminate BOOT_GHC and preinstalled happy/alex requirements:
   - Build all eight Stage 1 tools (unlit, happy, alex, genapply,
     deriveConstants, genprimopcode, ghc-pkg, hsc2hs) natively from source
     using cc and MicroHs with zero ghc, happy, or alex on PATH.
   - Add Curated C11 POSIX tool helpers (bootstrap/mhs/cbits/mhs_tool_posix.{c,h})
     and minimal tool shims (bootstrap/mhs/tool-shims/) for hsc2hs and ghc-pkg.
   - Guard MicroHs-specific GADT tuple binds, Cabal 3.17 Text fields, macOS
     fopen-on-directory behavior, and non-Generic package.cache serialization in
     utils/ghc-pkg/Main.hs under #if defined(__MHS__) (non-__MHS__ output is
     byte-for-byte unchanged).
   - Remove bootstrap/mhs/microhs-stage1.patch (now built directly from
     github:stable-haskell/MicroHs) and the dead shims/GHC/Fingerprint.hs shim
     that shadowed installed ghc-compat.

2. Fix __MHS__ GHC.Utils.Fingerprint performance using MicroHs runtime md5Array:
   - Replace O(N) [Word8] list allocation + pure-Haskell FNV-1a in
     GHC.Utils.Fingerprint, GHC.Data.StringBuffer, GHC.Driver.Downsweep, and
     GHC.Iface.Tidy.StaticPtrTable with MicroHs's C runtime md5Array
     (src/runtime/md5.c), producing RFC 1321 MD5 digests bit-identical to GHC.
   - Measured speedup on Stage 1 mhs: 558x on 1 MB (4,213.7 ms -> 7.55 ms) and
     1,008x on 8 MB (61,061.7 ms -> 60.56 ms).

3. Drop the gen-parsers.sh Python post-processor and /nix/store pins:
   - Build portable Haskell 2010 happy (github:stable-haskell/happy
     feat/portable-haskell2010 @ 6d2b064) and alex (github:stable-haskell/alex
     feat/portable-bootstrap @ d8b3768) natively with MicroHs.
   - Delete mhsify_happy, python3, and grep -v ghcconfig.h from gen-parsers.sh;
     all eight Stage 1 parsers and lexers are generated directly as portable
     Haskell 2010 Data.Array modules.

Verified end-to-end under a sanitized PATH with no ghc, happy, or alex:
toolchain, all 8 tools, all 8 parsers/lexers, stages 00..08, Stage 1 ghc link,
and native AArch64 codegen smoke test (HelloStage1.hs -> .s/.o/.hi) all pass.
…hes, and TODO

Add the end-to-end MicroHs-compiled Stage 2 GHC demo GitHub Actions workflows
and supporting bootstrap files:

- bootstrap/mhs/patches/microhs/0001..0008-*.patch + MANIFEST.tsv:
  8 semantically sliced, upstream-issue-linked patches extracted via
  git format-patch from github:stable-haskell/MicroHs feat/ghc-stage1-series
  (9e1cd08..9ae73f34, verified 0-byte tree diff vs 29504a98), referencing
  augustss/MicroHs issues #367, #446, ghc#226, #449, #501, #502, #365, ghc#278,
  #294, #18 (and adjacent PRs #567, #559).
- .github/workflows/mhs-stage2-demo.yml:
  8-phase DAG across ubuntu-24.04 (x86_64-linux) and macos-latest
  (aarch64-darwin) asserting zero preinstalled ghc/happy/alex:
    Job 1 (Phases 1-3): checkout upstream MicroHs, super-charge with patches,
      build all 8 Stage 1 tools natively with MicroHs.
    Job 2a (Phases 4-5, concurrent): checkout stable-haskell/cabal at
      dual-compiler base 73d3ffda3e, super-charge with MicroHs patches
      (73d3ffda3e..b04f60210), build & test cabal-install with MicroHs.
    Job 2b (Phase 6, concurrent): build Stage 1 GHC with MicroHs (timed per
      stage checkpoint) and verify native codegen.
    Job 3 (Phases 7-8): build static Stage 2 GHC via cabal-mhs + Stage 1 GHC
      using cabal.project.stage2.mhs, package relocatable bindist artifact, and
      render aggregated Phase 1-8 timings to GITHUB_STEP_SUMMARY.
    Job 4: invoke dependent mhs-stage2-demo-app.yml workflow.
- .github/workflows/mhs-stage2-demo-app.yml + bootstrap/mhs/demo-app/:
  dependent downstream workflow (workflow_call + workflow_run +
  workflow_dispatch) that downloads the Stage 2 bindist artifact onto a clean
  runner with no preinstalled GHC and builds/runs a multi-module Haskell app
  via both direct ghc -O2 and offline cabal-mhs.
- bootstrap/mhs/ci-time.sh: reusable TSV phase timer & Markdown summary helper.
- cabal.project.stage2.mhs: static Stage 2 overlay disabling
  +build-tool-depends so prebuilt MicroHs tools satisfy all build tools with
  zero bootstrap GHC0.
- bootstrap/mhs/TODO.md: full roadmap and upstreaming tracker.
@angerman
angerman changed the base branch from feat/stage1-drop-hpc-time to stable-ghc-9.14 November 19, 2026 22:18
GHC's Haskell CPP runs in traditional mode and joins a trailing
backslash to the next line. The generated parser source then
contains illegal escapes. Concatenate ordinary strings instead.
Patched MicroHs honors NoImplicitPrelude. ghc-compat and
os-string's Encoding.Internal still use Prelude names without
importing them. Fetch those snapshots in the stage 2 demo, add
the import, and install them before mcabal -r builds the rest.
GHC reads a column-1 #if as a pragma unless the module enables CPP.
These files guard MicroHs imports with #if defined(__MHS__) and had
no LANGUAGE CPP, so stage1 stopped in Rank.hs and Extension.hs.
Happy reads GHC/Parser.y with the locale encoding. The rockylinux:8
image defaults to POSIX, which rejects the file's UTF-8 bytes.
Hackage pretty-1.1.3.6 depends on ghc-prim, which the MicroHs
snapshot does not ship, and it derives Show for TextDetails only
when __GLASGOW_HASKELL__ is set. Patch it onto base and give the
non-GHC build the Show and Semigroup instances MicroHs requires.
Traditional CPP joins a trailing backslash to the next line, so the
string gap in the DEPRECATED pragma becomes an illegal `\o` escape
(GHC-21231). A pragma string cannot use (++), so keep the message on
one physical line.
The __MHS__ copies of these bindings matched the let column. The
#else copies were indented one space further, so after CPP they
were continuations and GHC rejected the '=' (GHC-58481). The two
sides were the same code, so keep a single binding.
Happy reads GHC/Parser.y with the locale encoding. arm64v8/debian:11
defaults to POSIX and rejects the file's UTF-8 bytes. Use C.UTF-8
when the image has it, and generate en_US.UTF-8 otherwise. The test
job uses the same image.
mhs-compat-shims hashes with OpenSSL SHA256 except on macOS, where
it uses CommonCrypto. The ghc-pkg link only passed -lm, so GNU ld
stopped at collect2. The mhs environment target appends $MHSCCLIBS
after the objects, which is where -lcrypto has to sit.
The 90 minute limit cancelled aarch64-darwin during the ghc-pkg
compile. happy, alex, genapply, and genprimopcode had already
finished. The cancelled matrix leg then skipped phases 4-8.
Ubuntu 24.04 now installs ghcup at /usr/local/.ghcup/bin. The fixed
directory list left that ghc on PATH, so phases 4-6 failed the
zero-bootstrap check in a few seconds.
The workflow SHA dropped two characters, so phase 4's fetch failed
with "couldn't find remote ref". The branch tip is
b04f60210bff6fb149ea5a14a4e1329ab06e947d.
git am rejects the MicroHs graft at CallHooksExe.hs even though that
commit's parent is the cabal base. Check the tip out instead.

ghc-boot imports Data.Binary from the cabal mhs-compat-shims, which
the stage-1 artifact did not ship. Copy those three modules onto the
mhs search path.

macOS bash 3.2 rejects an empty NIP_FLAG array under set -u.
…ary shim

MicroHs does not implement GHC.Generics, so deriving via Generically
cannot solve Semigroup (Rep GhcOptions ()). Drop that clause in the
cabal checkout. The patched compiler derives Semigroup and Monoid for
a single-constructor record.

ghc-boot compiles the Data.Binary shim with NoImplicitPrelude, and
Get.hs uses Either without importing it.
CI invokes build-stage1-mhs.sh once per stage. use_binary_shim only ran
inside the ghc-boot process, so stage 02a compiled ShortText without
-i $OUT/binary-shim and failed with "Module not found: Data.Binary".
ProjectConfig.Lens updates PackageConfig (66 fields) without importing
the constructor. Above 64 fields MicroHs skips SetField and the case
path reports undefined value: PackageConfig at the data declaration.
The same cutoff covers GhcOptions (72) and ElaboratedConfiguredPackage (78).
MicroHs gives Proxy's type argument kind Type, so Proxy @Tok fails
when tok is a Symbol (Annotation.hs:236, expected Type, actual Symbol).
Proxy :: Proxy tok typechecks with the same patched mhs. Stage ghc-boot
on 69fdf3f passed; stage 02a stopped on this line.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant