roofline

Home for "How To Scale Your Model", a short blog-style textbook about scaling LLMs on TPUs

Modular C++ Toolkit for Performance Analysis and Logging. Profiling API and Tools for C, C++, CUDA, Fortran, and Python. The C++ template API is essentially a framework to creating tools: it is designed to provide a unifying interface for recording various performance measurements alongside data logging and interfaces to other tools.

Analyze LLM inference: FLOPs, memory, Roofline model. Supports GQA, MoE, MLA, RoPE, SwiGLU. 19 models × 20+ hardware platforms.

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → parallelism → vLLM/SGLang, with GPU labs on open models.

Optiflop measures the optimally achievable FLOPs for mathematical operations on various platforms.

《How to Scale Your Model 中文版》，这是一本关于在TPU上扩展大型语言模型的简短博客式教科书。

Interactive 3D visualization of dense decoder-only LLM inference. Companion to the AI Inference Engineer 2026 course.

Hand-written CUDA linear algebra library. A GEMM driven from a naive baseline to a compute-bound tensor-core kernel at ~90% of cuBLAS on an RTX 5070, profiled with Nsight Compute, plus GEMV, TRSM, CSR SpMV, cuSOLVER, a measured roofline, and NVML telemetry.

Interactive macOS GPU benchmark suite for Apple Silicon - 25 Metal compute benchmarks (compute, memory, ML/tensor, ray tracing) with a full measurement harness and empirical roofline.

Interactive theoretical Kimi-K3 inference roofline calculator for H200, B300, and GB300

Double-precision dense GEMM across sequential, OpenMP, MPI and CUDA on an i9-12900K, Tesla T4 and Tesla P100. The T4 loses to the CPU in FP64 (Turing runs it at 1:32); a P100 at 1:2 explains why. Raw data, Intel Advisor and Nsight profiles, notebooks and a 33-page report.

Pipeline-aware Roofline & Inference Sweep Model — wall-clock + TCO prediction for NPU architecture exploration, calibrated on Ascend 910B4 via msprof. Other architectures are under consideration.

GPU performance optimization labs covering Roofline Analysis, LLM Decode Optimization, and CUDA Graphs using PyTorch

Where does the next watt go? Choosing an AI-accelerator budget allocation when the workload mix is unknown -- max-expected vs minimax-regret over the whole forecast space. Normalized model, stdlib only.

PyTorch GPU inference performance exercises: roofline analysis, decode-loop profiling, KV-cache optimization, torch.compile, and CUDA graphs.

Executable validation harness for the OSTIR cache-resident quantized CPU inference thesis: grouped/hierarchical quantizers, hardware-counter instrumentation, and the S(h) residency-law fit, with experiments E1-E7 run against the monograph's own pass/fail gates.

Qwen3.5-2B BF16 Roofline and Nsight profiling on Jetson Orin Nano

The Memory Wall at the Edge of Language: characterizing & optimizing LLM inference on a 2GB ARM CPU (Raspberry Pi 5). Decode roofline (R2=0.994), PMIC energy/token, KV-cache capacity wall, energy-optimal policy. llama.cpp.

Measures cache latency, STREAM memory bandwidth, and peak AVX2/NEON FLOPS on real hardware, then assembles them into a measured roofline and a BLIS-style GEMM tile prediction validated empirically. C++23 and Python, fully tested, with CI on x86 and native ARM.

Measurement-driven study of LLM training (FSDP2) and serving (vLLM) on 8×A100 — 16 reproducible incident write-ups that predict the bound, control the confound, and report the negative.