Skip to content

About

Awesome tools for interpreting, manipulating the internals of of deep neural networks.

Resources

Stars

20 stars

Watchers

1 watching

Forks

Latest commit

 

History

37 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 

Repository files navigation

Awesome Interpretability

This list has tools, models, datasets and reading for interpretability. Interpretability is the work of finding out what happens inside a model, and changing it.

Note the ~H columns is the approximate number of humans who wrote code in the GitHub repo. I find it a useful (but fragile) measure of code quality.

Research and sources.

Contents

Inspect and intervene

Project

~H↑

Stars↑CreatedLatest

TransformerLens

1923,9462022-08-262026-09-28

wassname's comment: "uses jaxtyping, aliases models into a common interface, not as HuggingFace-compatible as other libs". Neel Nanda: "an extremely opinionated toolkit for doing whatever you want to specific models".

more

v4 update: TransformerBridge preserves raw HuggingFace weights by default; migration guide.

NNsight

371,1202023-10-202026-09-09

wassname's comment: "aim to keep it as simple as baukit eventually, and support remote mechinterp. HuggingFace compatible".

more

David Bau: "To customize a model, instead of running it as a function, you run it as a "with" context. Inside "with" you can write regular pytorch to modify the computation.".

Pyvene

239052023-02-062026-03-06

Intervention focused.

more

Zhengxuan Wu: "pyvene tries to be HuggingFace-native, supporting pre-defined interventions or customized interventions (below).".

ViT-Prisma

103932023-10-022025-07-21

Mechanistic interpretability for vision and video transformers.

vLLM-Hook

101632025-11-122026-09-23

Program internal states of vLLM-served models.

Penzai

91,8992024-04-042025-06-22

JAX-based, not HuggingFace-native; archived.

cupbearer

6222023-03-212025-01-09

Library for mechanistic anomaly detection.

vllm-lens

51312026-03-122026-10-02

Extract residual stream activations and apply steering vectors in vLLM.

nnterp

41212024-08-082026-07-02

NNsight companion: standardized model/module names and accessors; preserves HF implementations.

Mishax

31582024-07-302026-09-16

JAX/Flax instrumentation and interventions through AST rewriting.

TorchLens

26622022-10-142026-10-04

PyTorch computation graphs, activation/gradient inspection and interventions; not restricted to transformers.

BauKit

22582022-02-152024-02-22

wassname's comment: "light, simple, and well loved". PyTorch hooks.

interp-engine

1392026-08-052026-10-02

Standardized eager/vLLM activation capture and editing. Early project; model-gradient attribution is eager-only.

Graphpatch

1222023-12-062025-01-27

wassname's comment: "promising but abandoned". Graph-based PyTorch interventions; inactive default branch since January 2025, unarchived.

Natural-language interpretation

Project

~H↑

Stars↑CreatedLatest

Activation Oracles

21032025-07-302026-09-26

Natural-language activation QA, distinct from NLA reconstruction.

more

Adapters; paper/main dependencies differ. Adam Karvonen.

Transluce introspective interpretation

2382025-12-222026-07-07

Feature descriptions, patch-effect prediction and input ablations.

more

Released models; feature supervision is SAE-derived.

Natural Language Autoencoders

19552026-05-052026-08-02

Two fine-tuned LMs: one writes a text description of an activation, the other rebuilds it; SFT then GRPO.

more

Used in Anthropic pre-deployment audits (Claude Opus 4.6 onward). Paper, inference package, four matched AV/AR pairs, demo.

LatentQA

1372024-12-122025-11-16

Research decoder that reads/writes activations in natural language.

more

Adapter: CC-BY-NC-SA-4.0, Llama base required.

Predictive Concept Decoders

————

Reading/demo: sparse concepts predict future model behavior. Demo; public code/checkpoint not verified.

Lenses, probes and concepts

Project

~H↑

Stars↑CreatedLatest

ELK

192252023-02-012023-11-02

Latent-knowledge activation extraction and probe evaluation.

more

Current training path is supervised; older README describes CRC/CCS. Probes do not establish general truth detection.

Interpreto

102072025-02-262026-10-06

Probes/CAVs, ICA/PCA/NMF/SVD/KMeans, attribution and lenses; SAEs optional. Some metrics and backend migrations remain unreleased.

Tuned Lens

56162022-10-032025-08-07

Look at how transformer predictions are built layer by layer.

more

Pretrained lenses are stored in a HF Space.

NeuroX

51092018-08-162023-04-12

Neuron analysis and representation probing; historical toolkit.

Deception Detection

3542024-05-312025-02-06

Apollo Research linear-probe paper code, labeled rollouts and example probe/config files; dataset/model-specific, not a general truth detector.

Overcomplete

21532024-06-142025-12-04

Vision dictionary/concept learning: SAEs, NMF variants and archetypal analysis.

ICA Lens

2452026-06-082026-09-30

ICA decomposition, signed token readouts and coordinate steering.

more

Fitted lenses; early project, artifact licenses unspecified.

Jacobian lens

12,0112026-07-022026-07-02

Decodes residual vectors into tokens via the average Jacobian to the last layer.

more

Paper, fitted lenses. Reference only, not maintained.

Causal hypotheses and circuit discovery

Project

~H↑

Stars↑CreatedLatest

circuit-tracer

172,9152025-05-282026-09-11

Attribution graphs and feature interventions using pretrained MLP transcoders.

more

Bundles credited Anthropic frontend code; repository accounts are not backend-specific maintainers.

Causalab

6–71182025-04-252026-09-30

Causal hypothesis testing, DAS/DBM and interventions.

more

Public mirror; issue #79 transferred internally, current fix unknown. ~H allows likely account aliases.

Rewriting a Deep Generative Model

45352020-07-292020-12-14

David Bau and collaborators: edit StyleGANv2 weights to change generative rules; ECCV 2020 reference code with an old PyTorch/CUDA stack.

AutoCircuit

31032023-08-212026-08-17

Efficient patching, automatic circuit discovery and evaluation on TransformerLens.

EAP-IG

3892024-01-152026-05-23

Gradient/integrated-gradient circuit ranking with exact-patching comparisons; architecture constraints apply.

ROME

27802022-02-112022-10-14

David Bau / Kevin Meng: causal tracing and rank-one factual weight edits. Historical GPT-2/GPT-J/CUDA reference code.

MEMIT

25612022-10-132022-11-14

David Bau / Kevin Meng: batched factual weight edits across layers; historical CUDA reference code, companion to ROME.

Transluce circuits

1412026-01-152026-04-10

Neuron-basis ADAG tracing and automated descriptions; paper reference code with AI-assisted implementation disclosed.

Steering and trainable interventions

Project

~H↑

Stars↑CreatedLatest

PyReFT

101,5902024-02-172025-02-06

Trainable low-rank representation interventions, built on Pyvene; historical baseline.

AxBench

62182024-08-072026-03-12

Concept detection and steering benchmark.

more

Even Simple Baselines Outperform Sparse Autoencoders, Concept16K data: model-generated examples sampled from GemmaScope concepts.

repeng

57612024-01-212025-09-24

Library for making RepE control vectors. Theia Vogel. Representation Engineering Mistral-7B an Acid Trip.

steering-vectors

51632024-01-182025-02-21

HF/PyTorch control-vector training and injection; historical reusable baseline, last default commit February 2025.

Steerability

41232025-06-132026-10-03

Extensible general-purpose steering library, formerly IBM/AISteer360.

more

My open PRs: VJP-delta, CorDA-PCA, S-space and Linear-AcT.

IBM activation-steering

21912024-08-232025-08-29

General-purpose activation steering library (ICLR 2025).

Spherical-Steering

2232026-02-082026-05-19

Rotates activations instead of adding to them (ICML 2026); paper code.

Introspection adapters

1302026-04-282026-04-28

LoRA self-report of trained behaviors for weight/behavior auditing, not an activation decoder; false positives.

more

Paper.

weight-steering

1122025-10-172025-11-11

Code for Steering Language Models with Weight Arithmetic.

steering-lite

122026-04-282026-09-30

Hackable forward-hook activation steering, calibrated, tested.

Model organisms

Project

~H↑

Stars↑CreatedLatest

talkie

11,0202026-04-202026-05-19

13B LM trained on 260B tokens of pre-1931 English.

more

Nick Levine, David Duvenaud, Alec Radford. Chat model, base, and a FineWeb twin with the same architecture "to make possible controlled comparisons between vintage and modern LMs".

GPT-4chan

16412022-06-022022-06-03

Yannic Kilcher, 2022: GPT-J 6B fine-tuned on /pol/.

more

Weights on archive.org, HF mirror.

GPT4chan 24B

————

Mistral-Small-24B base with a QLoRA on v2ray/4chan; also 8B on Llama-3.1. Base-model prompt format, not chat.

Olmo-3.1-7B-RL-Zero-Code-4chan

————

Mine. Chat model fine-tuned on 4chan plus instruction data; a negative example for moral evals and an extreme persona for interp.

kjj0/4chanpol

————

114M unique /pol/ posts, June 2016 to November 2019, deduplicated from Raiders of the Lost Kek.

more

Variant with OpenAI moderation scores.

4chan-datasets

————

Many boards, raw text. v2ray/4chan is the same data in a better format.

v2ray_4chan_formatted

————

Mine. v2ray/4chan as chat messages with SFT splits (45,751 train, 5,084 test).

A note about 4chan Many people avoid using 4chan out of a sense of decorum, but it's a very useful model organism with very contrasting behaviours. Most models are helpful, polite and moderate. 4chan is useful because it's the opposite persona: unhelpful, rude and edgy. It's also sometimes funny. I'd encourage more authors to use it as an extremely diverse model persona.

Evaluate interpretations and interventions

Project

~H↑

Stars↑CreatedLatest

Quantus

206762021-03-182026-08-20

Attribution faithfulness, robustness and randomization metrics; implementations are not universally verified by original metric authors.

Tracr

95682022-12-012024-02-05

Compile RASP programs into transformers with known mechanisms; archived ground-truth reference.

MIB

3272025-04-012025-08-15

Mechanistic interpretability benchmark project; implementation is split into tracks. Circuit track.

CausalGym

2572023-10-102024-11-30

Controlled linguistic causal-intervention benchmark.

more

HF data; reference code assumes GPTNeoX-family models.

Liars' Bench

2152025-02-182026-05-07

Cadenza Labs lie-detection benchmark with black/white-box detectors; submodule setup documentation conflicts with actual URLs.

RAVEL

1582024-02-172025-10-30

Representation disentanglement/interchange tests; paper benchmark.

Interpretable models by design

Project

~H↑

Stars↑CreatedLatest

PyTorch Concepts

101592024-06-222026-07-24

Concept bottlenecks, interpretable layers and interventions; alpha API.

Steerling

32412026-02-222026-07-14

Interpretable-by-design ~8B causal diffusion LM, not an arbitrary-model explainer.

more

Weights; commercial-use wording qualified; training code not released.

Explainability and counterfactuals

Project

~H↑

Stars↑CreatedLatest

SHAP

27025,7942016-11-222026-10-04

Feature attribution; background/feature-dependence assumptions matter.

Captum

1185,7092019-08-272026-09-28

PyTorch attribution and model interpretability.

pytorch-grad-cam

5212,9932017-05-312026-08-13

CNN/ViT attribution and ROAD evaluation; Jacob Gildenblat.

InterpretML

476,9562019-05-032026-08-17

Glassbox explainable boosting models and blackbox explanations.

LIME

4212,1662016-03-152021-07-29

Local surrogate explanations; historical baseline, code inactive since 2021.

AIX360

241,8082019-07-112026-09-05

IBM explainability toolkit.

DiCE

211,5282019-05-022025-07-13

Diverse counterfactual explanations; model counterfactuals do not establish real-world causal recourse.

Interpret community

214442019-09-252025-02-07

InterpretML extension: SHAP, Mimic and LIME explainers, permutation feature importance.

Xplique

157552020-04-052026-09-25

Attribution and NMF/CRAFT concept discovery; framework/version restrictions apply.

Zennit

122482020-11-102026-05-13

PyTorch Layerwise Relevance Propagation.

Inseq

114762021-09-142026-04-25

Sequence-generation attribution.

Concept Relevance Propagation

71412022-06-072026-01-14

Concept-conditional attribution and relevance maximization; Zennit companion.

ICX360

5722025-05-272026-10-03

IBM input-context attribution, contrastive prompt explanations and token highlighting; explanation depends on context perturbations/output scalarization.

XAI

31,2652019-01-112025-11-29

Explainability and responsible-ML tools.

Dissect

23072020-04-282021-01-09

David Bau / Jun-Yan Zhu: vision concept dissection, unit ablations and visualization; historical CNN/GAN reference code.

Explabox

1222022-05-102025-10-14

Model explanation and testing toolbox.

Responsible AI Toolbox

————

Dashboard integrating Error analysis, Fairlearn, InterpretML, DiCE, EconML and Data Balance.

MI2.ai

————

ARES, xSurvival, Large Model Analysis; DrWhy ecosystem.

DrWhy

6†688†2018-10-182023-02-21†

DALEX, survex, Arena, fairmodels;

more

†collection-wide metrics, not maintenance evidence for each component.

ELI5

————

Inspect model weights and predictions.

Sparse feature tooling

Project

~H↑

Stars↑CreatedLatest

SAELens

781,5502023-11-292026-10-04

Sparse autoencoder training/loading/analysis.

more

Gemma Scope 2 pretrained suite.

Delphi

222792024-06-182026-08-25

Generate and score explanations of SAE/transcoder features; distinct from raw-activation NLAs. Sampled weekly CI failures in the October audit.

Browsers and visualizations

Project

~H↑

Stars↑CreatedLatest

Transformer Debugger

124,1212024-03-112026-04-15

OpenAI model-inspection interface; not HuggingFace-native.

Neuronpedia

121,1602023-06-212026-10-01

Public feature/neuron browser; source.

CircuitsVis

123682022-11-052026-04-30

Python/React visualizations, including attention patterns; not a tracing engine.

Treescope

114762024-07-242026-06-18

Interactive tensor/model HTML inspection, independent of archived Penzai.

Workbench

7182025-05-202026-09-18

Interactive NNsight/NDIF model exploration; visualization/platform rather than a new hook engine.

Subtext

22452026-07-062026-07-23

Conversational Jacobian-lens interface. Each rendered word is a lens readout, not model output.

Logitloom

11592025-05-082025-05-29

Theia Vogel's interactive token-trajectory trees from API logprobs/prefills, not a logit lens. Unlicensed; split-UTF-8 limitations.

NN-SVG

————

Neural-network architecture illustration.

Agent auditing and evaluation infrastructure

Project

~H↑

Stars↑CreatedLatest

Inspect AI

3252,9462023-11-142026-10-06

Evaluation/agent runtime, scoring, sandboxes and .eval logs; behavioral evidence, not hidden-state explanation.

Inspect Scout

30762025-09-072026-10-06

Transcript scanning/analysis, external transcript imports and human-label validation; automated judgments still need validation.

Petri

81,3612025-08-192026-10-02

Auditor/target interactions, simulated tools, rollback and rubric judging. Formerly safety-research/petri; v3 changes the Python API.

Petri Bloom

2362026-04-022026-07-15

Bloom scenario generation with Petri audits; original Bloom is frozen.

more

Small successor; current Petri-v3 compatibility untested.

Docent

————

wassname's original description: "interactive model explanation and steering interface". Updated description: agent-transcript and behavior-rubric auditing; current product page.

Inspect Evals

212†693†2024-10-022026-10-05†

Collection of Inspect evaluations; †metrics cover the collection, not an individual task's contributors or maintenance.

ControlArena

59†248†2025-02-062026-10-05†

Inspect-based AI-control experiments: policies, monitors and safety/usefulness evaluation; †framework/settings-wide metrics, not individual-setting maintenance.

Adapters

Project

~H↑

Stars↑CreatedLatest

Adapter intervention types

162026-02-222026-07-19

Literature review of adapter intervention types.

lora-lite

112026-04-262026-06-19

Hackable LoRA library, one file per variant, built on forward hooks.

Mine (wassname)

I have some good stuff too, so please excuse a little detour into self promotion...

Project

~H↑

Stars↑CreatedLatest

moral-maps

222026-04-302026-09-25

Puts models through human value surveys and plots them next to human societies; shows where steering moves a model.

more

Eval library: moral foundation vignettes against human raters, MFQ-2, Big Five, 16PF and Humor Styles surveys comparable to human country means; local HF answer-token probabilities.

abliterator

1112025-03-112026-02-21

Concept removal (abliteration) with BauKit, not TransformerLens.

vjp-steering

142026-08-212026-09-28

Contrastive steering vectors from vector-Jacobian products (WIP).

AntiPaSTO

142025-12-282026-09-02

Self-supervised honesty steering via anti-parallel representations.

query-steering

132026-09-252026-09-30

Steer attention so the model reads out a secret from its context.

isokl_steering_calibration

122026-05-052026-09-19

Compare steering methods at the same KL budget.

ssteer-eval-aware

122026-03-212026-05-03

S-space steering suppresses eval-awareness.

cwsteer

112026-06-232026-06-26

Contrastive weight steering: generate, filter, train, calibrate, steer.

persona-steering-template-library

112026-06-132026-08-09

~100 persona prompt templates for building steering vectors, judged on on-axis vs off-axis behaviour.

tiny-mfv

————

132 moral foundation vignettes (Clifford et al. 2015): classic, sci-fi and AI-actor versions; HF dataset.

Structured output (adjacent tooling)

Project

~H↑

Stars↑CreatedLatest

instructor

25913,9812023-06-142026-09-11

Pydantic models from API models; retries on validation errors. For remote APIs without logits.

Outlines

17915,9052023-03-172026-08-24

Constrained generation from regex, JSON schema or grammar.

Microsoft Guidance

7821,7892022-11-102026-05-21

Templates that mix prompts and constraints.

XGrammar

741,9442024-06-282026-10-06

Fast grammar engine; default backend in vLLM, SGLang, TensorRT-LLM and MLC-LLM.

guardrails

727,4922023-01-292026-08-26

Validators for LLM outputs.

LLGuidance

378822024-07-252026-09-30

Fast Rust grammar engine; used by OpenAI Structured Outputs, vLLM, SGLang, llama.cpp and Chromium.

TypeChat

358,6892023-06-202026-10-06

TypeScript.

LMQL

354,2182022-11-242025-05-22

Query language for LLMs; latest default/source dates differ.

lm-format-enforcer

142,0412023-09-212026-04-04

JSON schema and regex for transformers and vLLM.

Promptify

134,6402022-12-122026-03-27

Structured NLP prompting.

kor

111,6842023-02-162024-11-25

Schema-based extraction; historical library.

prob_jsonformer

9172024-05-102025-03-23

Jsonformer, but it can output the probability of each choice in a single pass. Has enum.

jsonformer

64,9372023-04-292023-05-30

Does not do enums, HuggingFace only; historical code, last default commit May 2023.

salute

52182023-05-212023-06-17

TypeScript; historical code.

clownfish

23282023-03-272023-05-16

Modifying transformers to follow a JSON schema; historical code.

Constrained-Text-Generation-Studio

12172022-03-282026-04-11

Constrained text-generation interface.

relm

11082023-03-212023-06-02

Regular expression engine for language models; historical code.

vLLM structured outputs

————

JSON schema, regex, choice and grammar in the server, with xgrammar, guidance or outlines as backend.

llama.cpp grammars (GBNF)

445†130,508†2023-03-102026-10-06†

Grammar support. †Metrics cover the whole llama.cpp repository, not the grammar module.

OpenAI Structured Outputs

————

JSON schema enforced by the API; other providers have similar options.

LangChain structured output

————

Structured-output integration.

Perspectives

Agendas and changes of view, oldest first.

PerspectiveWhoYear

Simulators, The Persona Selection Model

janus; Sam Marks, Jack Lindsey, Chris Olah2022, 2026

"LLMs learn to simulate diverse characters during pre-training, and post-training elicits and refines a particular such Assistant persona."

AGI Ruin #27, The Most Forbidden Technique

Eliezer Yudkowsky; Zvi Mowshowitz2022, 2025

"Optimizing against an interpreted thought optimizes against interpretability." For CoT, Baker et al. found that "with too much optimization, agents learn obfuscated reward hacking".

The case for ensuring that powerful AIs are controlled

Ryan Greenblatt, Buck Shlegeris2024

Safety measures should hold "even if the AIs are misaligned and intentionally try to subvert those safety measures".

Towards Guaranteed Safe AI, Is there a Natural Abstraction of Good?

davidad2024, 2026

From proof-checked safety guarantees to the claim that frontier LLMs "have grokked the natural abstraction of what it means to be Good". Gabriel Alfour disputes it in the same dialogue.

A Pragmatic Vision for Interpretability

Neel Nanda and the GDM interpretability team2025

After "SAEs underperformed linear probes", "a strategic pivot over the past year, from ambitious reverse-engineering to a focus on pragmatic interpretability".

The Urgency of Interpretability

Dario Amodei2025

Understand models "before models reach an overwhelming level of power".

Chain of Thought Monitorability: A New and Fragile Opportunity

Tomek Korbak, Mikita Balesni and 39 co-authors2025

"Because CoT monitorability may be fragile, we recommend that frontier model developers consider the impact of development decisions on CoT monitorability."

Why We Are Excited About Confessions

Boaz Barak, Gabriel Wu, Jeremy Chen, Manas Joglekar2026

A second output rewarded only for honesty, because "being honest in confessions is the path of least resistance".

Reading and tutorials

Project

~H↑

Stars↑CreatedLatest

Generative Meta-Model of LLM Activations

1952026-01-302026-09-12

Luo et al. paper: diffusion prior over residual activations, used to make steering more fluent.

Manual activation steering

1242024-01-122024-10-18

Tutorial on doing it manually.

Latent Introspection code

132026-02-032026-02-27

Theia Vogel's source for Latent Introspection; concept-injection/self-report experiments, default 2-GPU Qwen-32B setup.

Small Models Can Introspect, Too

————

Theia Vogel; Latent Introspection.

more

Injected concept vectors, including an emergent-misalignment vector from a difference between checkpoints.

Lenses on steering vectors

————

thebes (Theia Vogel), 2026: before believing a steer-on-X result, check fine-tune, sampler, prompt and norm-matched random-vector controls.

ML Model Interpretation Tools

————

Neptune-AI blog; archive.

more

Neptune was bought by OpenAI and closed its service in 2026.

Explainability and Auditability in ML

————

Neptune-AI blog; archive.

AI Ethics tool landscape

————

Tool landscape.

See more

Project

~H↑

Stars↑CreatedLatest

Awesome-explainable-AI

221,6592020-02-162026-08-19

Broader explainable-AI list.

awesome-moral-evals

112026-06-282026-06-30

Datasets for evaluating the moral behaviour of LLMs.

dweprinz's list that inspired this one

1†1†2023-09-302026-09-29†

Responsible-AI / AI-safety resources. †Repository-wide metadata, not this page.

Mechanistic Interpretability Workshop

————

Workshop CFP.

GitHub interpretability topic

————

Discovery index.

David Bau on NNsight

————

NNsight for research.

About

Awesome tools for interpreting, manipulating the internals of of deep neural networks.

Resources

Stars

20 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages