gramref CLI Tool

Location: /Users/akollegger/.cabal/bin/gramref (or in PATH)
Source: ../gram-hs/ Haskell project
Purpose: Generate test patterns and validate outputs for testing gram-rs
Created: 2026-01-08

Overview

gramref (formerly gram-hs) is a CLI tool for generating test patterns and validating outputs. It serves as the reference implementation for testing and equivalence checking.

Important Distinction

Aspect	`../gram-hs/` Library	`gramref` CLI Tool
What	Haskell source code	Executable program
Location	`../gram-hs/libs/`	`/Users/akollegger/.cabal/bin/gramref`
Purpose	Reference implementation to port	Testing and validation tool
Use for	Reading code, understanding algorithms	Generating test data, checking outputs
When	During implementation	During testing

Key point: Read from ../gram-hs/libs/ when implementing features; execute gramref when testing those features.

Installation

Check if Installed

which gramref
# Output: /Users/akollegger/.cabal/bin/gramref

gramref --help

Build from Source (if needed)

cd ../gram-hs
cabal build
cabal install

Usage

Basic Syntax

gramref <command> [options]

Commands

Command	Description
`parse`	Parse gram notation and output pattern structure
`generate`	Generate test patterns with specified criteria

Parse Command

Parse gram notation and output the pattern structure.

Basic Parsing

echo '(alice)' | gramref parse --format json

Output (includes metadata):

{
  "meta": {
    "hash": "abc123...",
    "timestamp": "2024-01-08T10:30:00Z"
  },
  "value": {
    "symbol": "alice",
    "labels": [],
    "properties": {}
  },
  "elements": []
}

Value-Only Output

Get just the pattern value without metadata:

echo '(alice)' | gramref parse --format json --value-only

Output:

{
  "elements": [],
  "value": {
    "labels": [],
    "properties": {},
    "symbol": "alice"
  }
}

Use case: Equivalence checking without metadata differences.

Canonical Format

Sort JSON keys alphabetically for consistent output:

echo '(alice:Person)' | gramref parse --format json --canonical

Use case: Reliable diff comparison, byte-for-byte matching.

Deterministic Output

Use fixed values for metadata (timestamp and hash):

echo '(alice)' | gramref parse --format json --deterministic

Metadata values:

timestamp: "1970-01-01T00:00:00+0000"
hash: "0000000000000000000000000000000000000000000000000000000000000000"

Use case: Snapshot testing with reproducible outputs.

Combined Flags

Combine flags for optimal testing:

# Value only + canonical + deterministic
echo '(alice)' | gramref parse --format json --value-only --canonical --deterministic

Use case: Most reliable equivalence checking.

Generate Command

Generate test patterns with specified complexity and count.

Basic Generation

gramref generate --type suite --count 10 --format json

Generates 10 test cases in test suite format.

With Seed (Reproducible)

gramref generate --type suite --count 100 --seed 42 --format json

Same seed = same test cases: Enables reproducible test generation.

Complexity Levels

Control the complexity of generated patterns:

# Minimal complexity
gramref generate --type suite --count 10 --complexity minimal --format json

# Basic complexity (default)
gramref generate --type suite --count 10 --complexity basic --format json

# Standard complexity
gramref generate --type suite --count 10 --complexity standard --format json

# Complex patterns
gramref generate --type suite --count 10 --complexity complex --format json

# Adversarial/edge cases
gramref generate --type suite --count 10 --complexity adversarial --format json

Complexity levels:

Level	Description	Use Case
`minimal`	Simple patterns (nodes, basic relationships)	Quick validation, debugging
`basic`	Common patterns	Standard testing
`standard`	Realistic patterns	Integration testing
`complex`	Nested structures, many properties	Stress testing
`adversarial`	Edge cases, boundary conditions	Robustness testing

Value-Only Generation

Generate test cases without metadata:

gramref generate --type suite --count 100 --seed 42 --format json --value-only

Use case: Smaller test files, faster comparison.

Save to File

gramref generate --type suite --count 100 --seed 42 --format json --value-only > test_cases.json

Test Suite Format

Generated test suites follow a standardized format:

{
  "version": "1.0",
  "test_cases": [
    {
      "name": "test_case_001",
      "description": "...",
      "input": {
        "type": "gram_notation",
        "value": "(alice:Person)"
      },
      "expected": {
        "type": "pattern",
        "value": {
          "elements": [],
          "value": {
            "symbol": "alice",
            "labels": ["Person"],
            "properties": {}
          }
        }
      },
      "operations": null
    }
  ]
}

Fields:

name: Unique test case identifier
description: Human-readable description
input: Gram notation to parse
expected: Expected pattern structure
operations: Optional transformations (usually null)

Key Flags Reference

Output Control

Flag	Description
`--format json`	Output in JSON format (default)
`--value-only`	Output only pattern value (no metadata)
`--canonical`	Sort JSON keys alphabetically
`--deterministic`	Use fixed timestamps and hashes

Generation Control

Flag	Description
`--type suite`	Generate test suite format
`--count N`	Number of test cases to generate
`--seed N`	Random seed for reproducible generation
`--complexity LEVEL`	Complexity: minimal, basic, standard, complex, adversarial

Common Workflows

Workflow 1: Equivalence Testing

Generate reference output to compare with gram-rs:

# Get reference from gramref
echo '(alice)-[:KNOWS]->(bob)' | gramref parse --format json --value-only --canonical > ref.json

# Get output from gram-rs (when CLI is implemented)
echo '(alice)-[:KNOWS]->(bob)' | cargo run --bin gram-rs parse --format json --value-only > rs.json

# Compare
diff ref.json rs.json

Workflow 2: Test Case Generation

Create a large test suite for gram-rs:

# Generate test suite
gramref generate --type suite --count 100 --seed 42 --complexity standard \
    --format json --value-only > tests/common/test_cases.json

# Validate format
cargo run --bin test-validator tests/common/test_cases.json

# Run equivalence tests
cargo test --test equivalence

Workflow 3: Snapshot Testing

Generate deterministic snapshots for regression testing:

# Generate snapshot
echo '(alice:Person {name: "Alice"})' | gramref parse --format json --deterministic --canonical > snapshot.json

# Use in test (snapshot will never change)
cargo test --test snapshot_tests

Workflow 4: Property Testing

Generate diverse patterns for property-based tests:

# Generate adversarial test cases
gramref generate --type suite --count 50 --complexity adversarial \
    --format json --value-only > adversarial_tests.json

Integration with Rust Tests

Example: Equivalence Checking Utility

use std::process::Command;
use serde_json::Value;

pub fn get_gramref_reference(input: &str) -> Result<Value, String> {
    let output = Command::new("gramref")
        .args(&["parse", "--format", "json", "--value-only", "--canonical"])
        .stdin(std::process::Stdio::piped())
        .stdout(std::process::Stdio::piped())
        .spawn()
        .map_err(|e| format!("Failed to spawn gramref: {}", e))?;
    
    // Write input to stdin
    use std::io::Write;
    if let Some(mut stdin) = output.stdin {
        stdin.write_all(input.as_bytes())
            .map_err(|e| format!("Failed to write to gramref: {}", e))?;
    }
    
    let result = output.wait_with_output()
        .map_err(|e| format!("Failed to get gramref output: {}", e))?;
    
    if !result.status.success() {
        return Err(format!("gramref failed: {}", String::from_utf8_lossy(&result.stderr)));
    }
    
    serde_json::from_slice(&result.stdout)
        .map_err(|e| format!("Failed to parse gramref JSON: {}", e))
}

Example: Test Suite Generation

pub fn generate_test_suite(
    count: usize,
    seed: u64,
    complexity: &str
) -> Result<Value, String> {
    let output = Command::new("gramref")
        .args(&[
            "generate",
            "--type", "suite",
            "--count", &count.to_string(),
            "--seed", &seed.to_string(),
            "--complexity", complexity,
            "--format", "json",
            "--value-only",
        ])
        .output()
        .map_err(|e| format!("Failed to run gramref: {}", e))?;
    
    if !output.status.success() {
        return Err(format!("gramref generate failed: {}", 
            String::from_utf8_lossy(&output.stderr)));
    }
    
    serde_json::from_slice(&output.stdout)
        .map_err(|e| format!("Failed to parse test suite JSON: {}", e))
}

Use Cases in gram-rs Testing

1. Equivalence Checking

Goal: Verify gram-rs produces same results as reference implementation.

Approach:

Parse with gramref parse --value-only --canonical
Parse with gram_codec::parse_gram_notation
Compare JSON outputs

Example:

echo '(alice)' | gramref parse --format json --value-only --canonical > ref.json
# Compare with gram-rs output

2. Test Data Extraction

Goal: Generate comprehensive test suites from reference implementation.

Approach:

Use gramref generate --type suite
Specify count, seed, complexity
Save to tests/common/test_cases.json

Example:

gramref generate --type suite --count 100 --seed 42 --format json > test_cases.json

3. Regression Testing

Goal: Detect unintended behavior changes.

Approach:

Use --deterministic flag for reproducible outputs
Save as snapshot files
Compare on each test run

Example:

echo '(alice)' | gramref parse --format json --deterministic > snapshot.json

4. Property-Based Testing

Goal: Test with diverse, automatically generated inputs.

Approach:

Generate patterns at different complexity levels
Use as inputs to property tests
Verify invariants hold

Example:

gramref generate --type suite --count 1000 --complexity adversarial --format json

Best Practices

1. Always Use --value-only for Comparisons

Eliminates metadata differences (timestamps, hashes).

# ✅ Good
gramref parse --format json --value-only

# ❌ Avoid (metadata changes on every run)
gramref parse --format json

2. Use --canonical for Reliable Diffs

Ensures consistent key ordering.

gramref parse --format json --canonical --value-only

3. Use --deterministic for Snapshots

Makes outputs reproducible.

gramref parse --format json --deterministic --canonical

4. Set --seed for Reproducible Generation

Same seed = same test cases.

gramref generate --seed 42 --deterministic --format json

5. Start with --complexity minimal

Easier to debug issues.

gramref generate --complexity minimal --count 10 --format json

Troubleshooting

gramref Not Found

# Check if in PATH
which gramref

# If not found, use full path
/Users/akollegger/.cabal/bin/gramref --help

Output Format Issues

# Use --canonical to ensure consistent formatting
gramref parse --format json --canonical --value-only

Reproducibility Issues

# Use both --deterministic and --seed together
gramref generate --seed 42 --deterministic --format json

Comparison with gram-lint

Aspect	gramref	gram-lint
Purpose	Reference implementation	Syntax validation
Output	Pattern structures (JSON)	Parse errors, tree (s-expr)
Language	Haskell	Rust
Use case	Testing gram-rs	Pre-validation
Commands	`parse`, `generate`	Validation only

Workflow:

Use gram-lint for syntax checks
Use gramref for reference outputs
Use gram-codec for actual parsing

Performance Considerations

Parse speed: Fast for small patterns, moderate for large
Generate speed: Depends on complexity level
- minimal: Very fast
- adversarial: Slower (complex generation logic)
JSON output: Can be large for deep nesting

Tip: Use --value-only to reduce output size.

FilesExpand file tree

gramref-cli.md

Latest commit

History

gramref-cli.md

File metadata and controls

gramref CLI Tool

Overview

Important Distinction

Installation

Check if Installed

Build from Source (if needed)

Usage

Basic Syntax

Commands

Parse Command

Basic Parsing

Value-Only Output

Canonical Format

Deterministic Output

Combined Flags

Generate Command

Basic Generation

With Seed (Reproducible)

Complexity Levels

Value-Only Generation

Save to File

Test Suite Format

Key Flags Reference

Output Control

Generation Control

Common Workflows

Workflow 1: Equivalence Testing

Workflow 2: Test Case Generation

Workflow 3: Snapshot Testing

Workflow 4: Property Testing

Integration with Rust Tests

Example: Equivalence Checking Utility

Example: Test Suite Generation

Use Cases in gram-rs Testing

1. Equivalence Checking

2. Test Data Extraction

3. Regression Testing

4. Property-Based Testing

Best Practices

1. Always Use --value-only for Comparisons

2. Use --canonical for Reliable Diffs

3. Use --deterministic for Snapshots

4. Set --seed for Reproducible Generation

5. Start with --complexity minimal

Troubleshooting

gramref Not Found

Output Format Issues

Reproducibility Issues

Comparison with gram-lint

Performance Considerations

Related Documentation