Support large-scale repository indexing

[liruohrh]

CodeGraph currently relies solely on .gitignore for file exclusion (PR #285).
Not very friendly to large repositories like WebKit or Chromium.
I do not want to index many commited files like test, or just index them late.

so, still need config or cli arg to include or exclude path.
and than, i hope has arg to limit worker count or some way to avoid oom,

---

OOM Error(no set node memory)

PS D:\mwk> npx codegraph init -v
┌ Initializing CodeGraph
│
◆ Initialized in D:\mwk
│
● Run "codegraph index" to index the project
│
└ Done

PS D:\mwk> npx codegraph index -v
┌ Indexing project
[0.0s] Phase: scanning
.......many file found
[2.1s] Phase: parsing

[worker] Spawning new parse worker...
[worker] Recycling worker after 250 parses (heap: 234MB RSS)
.....many like
[worker] Spawning new parse worker...
[48.7s] 5508/137699 (4%) - JSTests/slowMicrobenchmarks/ftl-polymorphic-rshift.js
[worker] Recycling worker after 250 parses (heap: 368MB RSS)
[worker] Spawning new parse worker...
[worker] Recycling worker after 250 parses (heap: 353MB RSS)
.....many like
[worker] Spawning new parse worker...
[93.0s] 12393/137699 (9%) - JSTests/test262/test/built-ins/Array/prototype/filter/15.4.4.20-9-c-iii-1-3.js
[worker] Recycling worker after 250 parses (heap: 296MB RSS)
.....many like
[worker] Spawning new parse worker...
[121.0s] 19278/137699 (14%) - JSTests/test262/test/built-ins/Object/create/15.2.3.5-4-116.js
[worker] Recycling worker after 250 parses (heap: 299MB RSS)
.....many like
[worker] Spawning new parse worker...
[143.6s] 26163/137699 (19%) - JSTests/test262/test/built-ins/String/S15.5.1.1_A1_T9.js
[worker] Recycling worker after 250 parses (heap: 299MB RSS)
.....many like
[worker] Spawning new parse worker...
[167.7s] 33048/137699 (24%) - JSTests/test262/test/built-ins/TypedArray/prototype/set/typedarray-arg-src-arraylength-internal.js
[worker] Recycling worker after 250 parses (heap: 301MB RSS)
.....many like
[worker] Spawning new parse worker...
[194.6s] 39933/137699 (29%) - JSTests/test262/test/language/expressions/assignment/dstr/obj-prop-elem-init-fn-name-cover.js
[worker] Recycling worker after 250 parses (heap: 304MB RSS)
.....many like
[worker] Spawning new parse worker...
[217.1s] 46818/137699 (34%) - JSTests/test262/test/language/expressions/dynamic-import/syntax/invalid/nested-with-expression-import-source-assignment-expr-not-optional.js
[worker] Recycling worker after 250 parses (heap: 304MB RSS)
.....many like
[worker] Spawning new parse worker...
[worker] TIMEOUT: JSTests/test262/test/language/identifiers/start-unicode-10.0.0-class.js exceeded 10000ms — killing worker
.....many like
[worker] Spawning new parse worker...
[359.7s] 53703/137699 (39%) - JSTests/test262/test/language/statements/class/dstr/async-gen-meth-obj-init-undefined.js
[worker] Recycling worker after 250 parses (heap: 515MB RSS)
.....many like
[worker] Spawning new parse worker...
[385.2s] 60588/137699 (44%) - JSTests/test262/test/language/statements/generators/dstr/ary-ptrn-rest-ary-elem.js
[worker] Recycling worker after 250 parses (heap: 400MB RSS)
.....many like
[worker] Spawning new parse worker...

Fatal process out of memory: Zone

----- Native stack trace -----

1: 00007FF646D4495D node::MultiIsolatePlatform::DisposeIsolate+5341
2: 00007FF6481B2006 v8::base::FatalOOM+54
3: 00007FF647A3F5E5 v8::Function::NewInstance+245
4: 00007FF6473B9576 v8::SnapshotCreator::GetIsolate+56566
5: 00007FF64868CDAA v8::platform::DelayedTaskQueue::Terminate+2063082
6: 00007FF64868C5D8 v8::platform::DelayedTaskQueue::Terminate+2061080
7: 00007FF64868BAAD v8::platform::DelayedTaskQueue::Terminate+2058221
8: 00007FF64868B584 v8::platform::DelayedTaskQueue::Terminate+2056900
9: 00007FF64868AC91 v8::platform::DelayedTaskQueue::Terminate+2054609
10: 00007FF64843F906 v8::base::DiyFp::Multiply+415574
11: 00007FF64843E3C5 v8::base::DiyFp::Multiply+410133
12: 00007FF6482122EB v8::internal::compiler::CompilationDependencies::FieldTypeDependencyOffTheRecord+192907
13: 00007FF6470B2F56 DH_get0_engine+658470
14: 00007FF6470AE693 DH_get0_engine+639843
15: 00007FF6470ADF79 DH_get0_engine+638025
16: 00007FF64822519F v8::platform::DefaultJobHandle::UpdatePriority+591
17: 00007FF646D4081E node::GetNodeReport+90430
18: 00007FF647A946D6 uv_thread_detach+150
19: 00007FF648CFA033 v8::base::UnsignedDivisionByConstant+2907267
20: 00007FFBC7EB259D BaseThreadInitThunk+29
21: 00007FFBC8E8AF78 RtlUserThreadStart+40

[colbymchenry]

There are two things bundled here, so let me split them:

1. The Fatal process out of memory: Zone crash is the same V8 bug as #298 / #293 — V8's turboshaft optimizing WASM compiler exhausting its internal Zone arena while compiling tree-sitter grammars (it's a V8-internal arena, not the JS heap, so --max-old-space-size doesn't help). This is fixed in 0.9.3 (releasing now): CodeGraph now runs with V8's --liftoff-only, which keeps grammar compilation off the optimizing tier. Once 0.9.3 is live on npm, please update and run npx @colbymchenry/codegraph@latest index -v again — the Zone crash should be gone. (Thanks for the detailed -v log; the per-worker RSS trace was useful in confirming it's the Zone arena, not the heap.)

2. Large-repo support (the actual feature request) is separate and still valid, so I'm keeping this issue open to track it. Today, indexing honors .gitignore plus a per-file size cap and nothing else — there's no include/exclude config and no concurrency knob. For a 137k-file tree like WebKit, even with the crash fixed you'll want to scope what gets indexed. Concretely, what I'd like to add here:

• Path include/exclude config (e.g. a .codegraphignore, or a config field) so you can skip JSTests/, LayoutTests/, vendored trees, etc. without touching .gitignore.
• A parsing-concurrency / worker cap for very large repos.

If you have a preferred shape for the config (a .codegraphignore file vs. CLI flags vs. a codegraph.json), say so — happy to design around how you'd actually use it.