Skip to content

RESULTS.md

Latest commit

 

History

History
261 lines (202 loc) · 9.6 KB

RESULTS.md

File metadata and controls

261 lines (202 loc) · 9.6 KB
Raw

🚀 TOON vs JSON: Benchmark Results

Executive Summary

TOON achieves MASSIVE memory and token savings compared to JSON across 50 diverse, real-world datasets.

┌────────────────────────────────────────────────────────────────┐
│                                                                │
│                    ⚡ HEADLINE RESULTS ⚡                       │
│                                                                │
│     📉  63.9% SMALLER file sizes                               │
│     📉  54.1% FEWER tokens for LLM APIs                        │
│     💾  35.81KB saved across 50 test datasets                  │
│     🎯  10,735 tokens saved                                    │
│                                                                │
│                  FOR HIGH-VOLUME APPLICATIONS:                 │
│     💰  $2,147 saved per million API requests                  │
│     💰  $5,408 saved per billion tokens                        │
│                                                                │
└────────────────────────────────────────────────────────────────┘

Why This Matters

For LLM API Users

If you're sending structured data to LLM APIs (GPT-4, Claude, etc.), you're paying for every token. TOON can cut your token usage by MORE THAN HALF (54.1% average), translating directly to:

  • 54% lower API costs
  • Faster API responses (less data to transmit)
  • More content in context windows (fit more data within token limits)

Real-World Cost Impact

At typical GPT-4 pricing ($10 per 1M tokens):

Usage Volume JSON Cost TOON Cost Savings
1,000 requests $3.97 $1.82 $2.15
1M requests/year $3,970 $1,823 $2,147
1B tokens $10,000 $4,592 $5,408

Detailed Results

Tested Across 50 Real-World Datasets

We benchmarked TOON against JSON using 50 diverse, production-ready datasets representing common use cases:

  • E-commerce (products, orders, inventory)
  • Databases (query results, employee records)
  • APIs (responses, logs, requests)
  • Analytics (metrics, A/B tests, surveys)
  • IoT (sensor data, time series)
  • Social media (posts, profiles, comments)
  • Finance (transactions, stock data)
  • And much more...

Performance Distribution

🔥 EXCELLENT (≥60% savings):  30 datasets (60%)
✅ GOOD (40-60% savings):     19 datasets (38%)
📊 MODERATE (<40% savings):    1 dataset  (2%)

98% of tested datasets achieved 40%+ savings!

Top Performers

🥇 Best Overall: Survey Responses

  • 73.4% size reduction (935B → 249B)
  • 63.4% token reduction (287 → 105 tokens)

🥈 Other Champions (>70% savings):

  • ML Training Data: 71.2% size, 61.9% tokens
  • Large Inventory (100 items): 71.2% size, 57.7% tokens
  • Student Grades: 71.2% size, 61.9% tokens
  • Customer Reviews: 69.1% size, 61.0% tokens
  • Weather Forecast: 69.0% size, 55.9% tokens

Category Breakdown

Category Datasets Avg Size Savings Avg Token Savings
Tabular Data (databases, spreadsheets) 12 69.2% 59.8%
E-commerce (products, orders) 8 66.1% 56.4%
Analytics (metrics, surveys) 7 65.7% 55.2%
API Data (responses, logs) 10 58.3% 48.9%
IoT/Sensors (time series) 5 60.0% 43.7%
Social/Content (posts, profiles) 8 61.5% 52.1%

Complete Results Table

# Dataset JSON Size TOON Size Size Savings Token Savings
01 E-commerce Products 1.57KB 542B 66.3% 58.2%
02 API Response 934B 501B 46.4% 39.7%
03 Database Results 1.52KB 582B 62.5% 56.5%
04 ML Training Data 1.85KB 545B 71.2% 61.9%
05 Server Configuration 1016B 719B 29.2% 28.4%
06 Analytics Data 1.40KB 526B 63.3% 49.4%
07 Large Inventory (100 items) 13.55KB 3.90KB 71.2% 57.7%
08 Customer Reviews 828B 256B 69.1% 61.0%
09 Social Media Posts 849B 282B 66.8% 52.1%
10 Weather Forecast 777B 241B 69.0% 55.9%
11 Stock Market Data - - 59.8% 44.2%
12 Restaurant Menu - - 66.4% 61.5%
13 Hotel Bookings - - 64.2% 52.1%
14 Flight Schedule - - 68.9% 59.9%
15 Medical Records - - 59.3% 50.6%
16 Student Grades - - 71.2% 61.9%
17 Sports Statistics - - 66.3% 54.6%
18 Movie Catalog - - 68.5% 59.8%
19 Music Playlist - - 62.5% 56.7%
20 Real Estate Listings - - 66.5% 58.7%
21-50 ... (see full benchmark output) - - 60%+ avg 50%+ avg

Aggregate Statistics

Total Across All 50 Datasets

JSON TOTAL:   56.00KB  (57,349 bytes)
TOON TOTAL:   20.20KB  (20,680 bytes)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
SAVED:        35.81KB  (36,669 bytes)  ⬇ 63.9%

JSON TOKENS:  19,851 tokens
TOON TOKENS:   9,116 tokens
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
SAVED:        10,735 tokens            ⬇ 54.1%

Visual Savings

Size Reduction:  ████████████████████████████████░░░░░░░░░░░░░░░░░░  63.9%
Token Reduction: ███████████████████████████░░░░░░░░░░░░░░░░░░░░░░░  54.1%

Why TOON Saves So Much Memory

1. Eliminates Repeated Keys in Arrays

JSON repeats keys for every object:

[
  {"id": 1, "name": "Laptop", "price": 999},
  {"id": 2, "name": "Mouse", "price": 29},
  {"id": 3, "name": "Keyboard", "price": 149}
]

134 bytes, 48 tokens

TOON declares headers once:

[3]{id,name,price}:
  1,Laptop,999
  2,Mouse,29
  3,Keyboard,149

52 bytes, 23 tokens → 61% smaller, 52% fewer tokens!

2. Minimal Syntax Overhead

JSON requires:

  • Braces {} and brackets [] everywhere
  • Quotes around all keys
  • Quotes around string values
  • Commas between all elements

TOON uses:

  • Indentation for structure (like YAML)
  • Colons for key-value pairs
  • Quotes only when necessary
  • Headers for uniform arrays

3. Intelligent Type Handling

TOON automatically detects when quotes aren't needed and preserves types (numbers, booleans, null) while maintaining human readability.

Use Case Recommendations

✅ PERFECT FOR TOON:

  1. LLM API Payloads - Cut token costs in half
  2. Database Query Results - Tabular data compression
  3. Analytics & Metrics - Time series, aggregates
  4. E-commerce Data - Product catalogs, inventory
  5. IoT Sensor Data - Regular readings
  6. API Logs & Traces - Structured log entries
  7. ML Training Data - Feature vectors, labels

⚠️ LESS OPTIMAL:

  • Highly irregular/nested data (still saves 20-40%)
  • Maximum compatibility required (JSON is universal)
  • Microsecond-level performance critical (TOON is fast, but JSON is faster)

Methodology

Test Environment

  • 50 diverse datasets representing real-world use cases
  • Accurate token counting using tiktoken (GPT-4 encoding)
  • Multiple iterations (100+) for performance measurements
  • Production-ready data (not synthetic/trivial examples)

Datasets Include:

  • E-commerce: products, orders, inventory, reviews
  • Databases: employee records, query results
  • APIs: responses, logs, requests, errors
  • Analytics: metrics, A/B tests, surveys, time series
  • IoT: sensor readings, device data
  • Social: posts, profiles, comments, messages
  • Finance: transactions, stock prices
  • Media: videos, music, blogs
  • System: logs, audit trails, notifications
  • And many more...

Token Counting

All token counts use tiktoken with GPT-4 encoding for accuracy. Results are directly applicable to:

  • GPT-4 / GPT-4 Turbo
  • GPT-3.5 Turbo
  • Claude (similar tokenization)
  • Other modern LLMs

Running the Benchmarks Yourself

# Clone the repo
git clone https://github.com/ScrapeGraphAI/toonify.git
cd toonify

# Install dependencies
pip install -e .
pip install tiktoken

# Run benchmarks
python benchmark/compare_formats.py
python benchmark/memory_benchmark.py

# Or run all at once
python benchmark/run_all.py

Conclusion

TOON delivers massive, consistent savings across diverse real-world datasets:

  • ✅ 63.9% average size reduction
  • ✅ 54.1% average token reduction
  • ✅ 98% of datasets achieve 40%+ savings
  • ✅ 60% of datasets achieve 60%+ savings
  • ✅ Thousands of dollars saved in LLM API costs for high-volume applications

The results speak for themselves: If you're working with structured data and LLM APIs, TOON can cut your costs in half while maintaining full data fidelity and readability.

Want to see the live output? Run python benchmark/compare_formats.py to see the full, interactive benchmark results!