Fast State-of-the-Art Static Embeddings in Rust

Quickstart • Features • Models • Performance • Relation to Python Model2Vec

model2vec-rs is a Rust crate providing an efficient implementation for inference with Model2Vec static embedding models. Model2Vec is a technique for creating compact and fast static embedding models from sentence transformers, achieving significant reductions in model size and inference speed. This Rust crate is optimized for performance, making it suitable for applications requiring fast embedding generation.

Quickstart

You can utilize model2vec-rs in two ways:

1. As a library in your Rust projects
2. As a standalone Command-Line Interface (CLI) tool for quick terminal-based inferencing

---

1. Using model2vec-rs as a Library

Integrate model2vec-rs into your Rust application to generate embeddings within your code.

a. Add model2vec-rs as a dependency:

cargo add model2vec-rs

b. Load a model and generate embeddings:

use anyhow::Result;
use model2vec_rs::model::StaticModel;

fn main() -> Result<()> {
    // Load a model from the Hugging Face Hub or a local path.
    // Arguments: (repo_or_path, hf_token, normalize_embeddings, subfolder_in_repo)
    let model = StaticModel::from_pretrained(
        "minishlab/potion-base-8M", // Model ID from Hugging Face or local path to model directory
        None,                       // Optional: Hugging Face API token for private models
        None,                       // Optional: bool to override model's default normalization. `None` uses model's config.
        None                        // Optional: subfolder if model files are not at the root of the repo/path
    )?;

    let sentences = vec![
        "Hello world".to_string(),
        "Rust is awesome".to_string(),
    ];

    // Generate embeddings using default parameters
    // (Default max_length: Some(512), all sentences in one batch)
    let embeddings = model.encode(&sentences);
    // `embeddings` is a Vec<Vec<f32>>
    println!("Generated {} embeddings.", embeddings.len());

    // To generate embeddings with custom arguments:
    let custom_embeddings = model.encode_with_args(
        &sentences,
        Some(256), // Optional: custom max token length for truncation
        512,       // Custom batch size for processing
    );
    println!("Generated {} custom embeddings.", custom_embeddings.len());

    // `encode_flat` returns one row-major `Vec<f32>` of
    // `sentences.len() * model.dim()` values. Use it when you want contiguous
    // output or only read the rows; prefer `encode` if you need an owned
    // `Vec<f32>` per sentence.
    let flat = model.encode_flat(&sentences);
    for row in flat.chunks_exact(model.dim()) {
        println!("First value: {}", row[0]);
    }

    Ok(())
}

Alternative: Loading from in-memory bytes (wasm / embedded use cases):

When filesystem access is unavailable, use from_bytes instead of from_pretrained:

use model2vec_rs::model::StaticModel;

let model = StaticModel::from_bytes(
    include_bytes!("path/to/tokenizer.json").as_ref(),
    include_bytes!("path/to/model.safetensors").as_ref(),
    include_bytes!("path/to/config.json").as_ref(),
    None, // normalize: None reads the value from config.json
)?;

let embeddings = model.encode(&["Hello world".to_string()]);

Quantizing an f32 model to int8:

use model2vec_rs::model::StaticModel;

let model = StaticModel::from_pretrained("minishlab/potion-base-8M", None, None, None)?.quantize_int8();

quantize_int8 stores the embeddings as int8 with one scale per row, using a quarter of the memory and encoding faster. Outputs stay very close to the f32 model (cosine similarity above 0.9998 on the potion models).

See the feature flags section for wasm and local-only build configurations.

---

2. Using the model2vec-rs CLI

a. Install the CLI tool: This command compiles the crate in release mode (for speed) and installs the model2vec-rs executable to Cargo's binary directory ~/.cargo/bin/.

cargo install model2vec-rs --features cli

Ensure ~/.cargo/bin/ is in your system's PATH to run model2vec-rs from any directory. The CLI is opt-in; default library builds do not include Clap. From a checkout, use cargo run --release --features cli -- ....

b. Generate embeddings via CLI: The compiled binary installed via cargo install is significantly faster (often >10x) than running via cargo run -- ... without release mode.

• Encode a single sentence:

  model2vec-rs encode-single "Hello world" "minishlab/potion-base-8M"

  Embeddings will be printed to the console in JSON format. This command should take less than 0.1s to execute.

• Encode multiple lines from a file and save to an output file:

  echo -e "This is the first sentence.\nThis is another sentence." > my_texts.txt
  model2vec-rs encode my_texts.txt "minishlab/potion-base-8M" --output embeddings_output.json

Features

• Fast Inference: Optimized Rust implementation for fast embedding generation.
• Hugging Face Hub Integration: Load pre-trained Model2Vec models directly from the Hugging Face Hub using model IDs, or use models from local paths.
• Model Formats: Supports models with f32, f16, and i8 weight types stored in safetensors files. Int8 models (model2vec's quantize_to="int8") stay int8 in memory, and f32 models can be quantized to int8 at load time with quantize_int8.
• Flexible Loading: Load models from the Hub (from_pretrained), raw bytes (from_bytes), or owned/borrowed data (from_owned, from_borrowed, and from_owned_i8, from_borrowed_i8 for int8 embeddings) for embedded and WASM use cases.
• Batch Processing: Encodes multiple sentences in batches.
• Configurable Encoding: Allows customization of maximum sequence length and batch size during encoding.

Feature flags

The crate exposes a few feature combinations for different runtimes:

• default: native build with onig tokenization and optional Hugging Face Hub downloads
• cli: build the command-line executable with Clap (not enabled by default)
• fancy-regex: alternative tokenizer backend for native builds
• local-only: disable remote model downloads and restrict loading to local paths or from_bytes(...)
• wasm: minimal WebAssembly-oriented feature set for in-memory loading via from_bytes(...)

Typical invocations:

# native local-only build
cargo build --no-default-features --features onig,local-only

# wasm (requires getrandom backend config)
RUSTFLAGS='--cfg getrandom_backend="wasm_js"' \
cargo check --no-default-features --features wasm --target wasm32-unknown-unknown

What is Model2Vec?

Model2Vec is a technique to distill large sentence transformer models into highly efficient static embedding models. This process significantly reduces model size and computational requirements for inference. For a detailed understanding of how Model2Vec works, including the distillation process and model training, please refer to the main Model2Vec Python repository and its documentation.

This model2vec-rs crate provides a Rust-based engine specifically for inference using these Model2Vec models.

Models

A variety of pre-trained Model2Vec models are available on the HuggingFace Hub (MinishLab collection). These can be loaded by model2vec-rs using their Hugging Face model ID or by providing a local path to the model files.

Model	Language	Sentence Transformer	Params	Task
potion-base-32M	English	bge-base-en-v1.5	32.3M	General
potion-multilingual-128M	Multilingual	bge-m3	128M	General
potion-retrieval-32M	English	bge-base-en-v1.5	32.3M	Retrieval
potion-base-8M	English	bge-base-en-v1.5	7.5M	General
potion-code-16M-v2	Code	CodeRankEmbed	16.2M	Code

Performance

We compared model2vec-rs with the Python version of Model2Vec by encoding the same texts with potion-base-8M in both.

Implementation	Single thread	All cores	Model load
Rust	671K texts/second	3.06M texts/second	15 ms
Python	40K texts/second	134K texts/second	312 ms
Speedup	16.6×	22.9×	~20×

On a single thread, the Rust version is roughly 17× faster than the Python version, and about 23× faster when using all cores.

To reproduce these numbers, run this from the repository root (it needs Rust and uv):

uv run scripts/python_compare.py

The script downloads the corpus, builds benches/python_compare.rs, runs both implementations and prints the table above. See scripts/python_compare.py for the options.

Benchmark details:

• Corpus: 2,644 random articles from English Wikipedia: shard 20231101.en/train-00017-of-00041.parquet of wikimedia/wikipedia, shuffled with seed 0 and taken until they reach 1.1M words, joined by blank lines (7.0 MB, SHA-256 c23440073f72900784a3f56a2318742ff055506b7a136f1e473edb4ef190d9e7). The script rebuilds it byte for byte.
• Texts: each of the 64,738 non-empty lines (paragraphs and section headings, 17 words and 108 bytes on average) is one text. Every timed pass encodes all of them in a single encode call with the default max_length of 512.
• Hardware: Apple M3 Pro (12 cores), 18 GB RAM.
• Software: model2vec-rs 0.3.0 (release build, Rust 1.99.1); Python 3.14.3 with model2vec 0.10.0, tokenizers 0.23.3 and numpy 2.5.4.
• Single thread: RAYON_NUM_THREADS=1 and TOKENIZERS_PARALLELISM=false for both; Python with use_multiprocessing=False.
• All cores: both with their defaults: Rust with Rayon parallelism; Python with tokenizer parallelism and use_multiprocessing=True, which spreads the texts over a process pool.
• Method: 5 alternating rounds of Rust and Python processes, 5 timed passes each after one warm-up pass; the table reports the median over the 25 passes.
• Output: the Rust and Python embeddings differ by at most 2.7e-7 per value (minimum cosine similarity 0.9999999).
• Model load: median time to load the model from the local Hugging Face cache; for Python this includes importing model2vec.

Relation to Python model2vec

• model2vec-rs (This Crate): High-performance Rust engine for up to 23x faster Model2Vec inference.
• model2vec (Python-based): Handles model distillation, training, fine-tuning, and slower Python-based inference.

License

MIT

Citing Model2Vec

If you use Model2Vec in your research, please cite the following:

@software{minishlab2024model2vec,
  author       = {Stephan Tulkens and {van Dongen}, Thomas},
  title        = {Model2Vec: Fast State-of-the-Art Static Embeddings},
  year         = {2024},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.17270888},
  url          = {https://github.com/MinishLab/model2vec},
  license      = {MIT}
}