Model Intelligence & Recommendation Hub

Automated, zero-hallucination domain evaluations computed from deterministic unit tests, clinical literature verifiers, and mathematical proofs across 2,457 benchmark runs.

Domain Capability Routing Matrix

Ranked by deterministic Pass@1

📊

No Benchmark Runs Recorded Yet

Execute a test battery in the Live Runner tab to compute genuine, zero-hallucination routing recommendations calculated from actual test assertions.

Empirical Study 2,457 Ground-Truth Runs 10 Grounded Rules

Key Learnings & Takeaways for Local LLMs

Hard-earned architectural rules and prompt patterns derived from benchmarking 21 local open-weights models (3B–120B) across 117 calibrated test cases using isolated code execution, LaTeX parsers, and rigid JSON schema validators.

21

Models Fleet

2,457

Total Runs

581

Autopsies

100%

Deterministic

Antigravity Agent Skill: /local-llm-inference Built from Benchmark Data

This benchmark dataset has been codified into an executable Antigravity agent skill. Triggering /local-llm-inference automatically enforces the 10 prompt construction and model routing rules below whenever building applications with local LLMs (LM Studio, Ollama, vLLM).

Part A: Prompt Construction & Formatting Rules (1–5)

Codified in /local-llm-inference

RULE 1: EXTRACTION TARGETS 76.3% Binary Results

Always Specify a Bounded Extraction Container

Require the model to place its final answer inside an explicit container (\boxed{} for math, JSON schema for tools, or explicit enum). 76.3% of evaluated results were binary (0% or 100%) — bounded targets eliminate partial, ambiguous outputs.

✔ Recommended Prompt

Write your final answer inside \boxed{answer}.
Respond with valid JSON matching: {"result": string, "confidence": number}

RULE 2: LENGTH CEILINGS 91.5% Peak Accuracy

Constrain Response Length to 100–300 Tokens

Completions in the 100–300 token range achieved 91.5% average accuracy — the highest of any length bucket. Accuracy declined monotonically to 81.6% (500–1K tokens) and plunged to 56.9% (3,000+ tokens) due to attention drift.

✔ Recommended Prompt

Answer in 2–3 sentences. Do not exceed 200 words.

RULE 3: NEGATIVE CONSTRAINTS #1 Failure: Forbidden Words

Emphasize Negative Constraints in ALL CAPS Callouts

The #1 instruction failure was FORBIDDEN_WORDS (31 fails), followed by CASE_CONSTRAINT (24 fails). Models leak forbidden words when buried in paragraphs. Format negative rules prominently with explicit callout boundaries.

✔ Recommended Prompt

CRITICAL RULE: You must NEVER use commas in your response.
CRITICAL RULE: Your ENTIRE response must be in lowercase.
CRITICAL RULE: Output EXACTLY 3 paragraphs, no more, no fewer.

RULE 4: EARLY COMMITMENT 87.3% Budget Collapse

Force Early Commitment — Don't Let Models Spiral

87.3% of token-exhausted models scored zero, and 73.7% contained looping phrases ("Wait, let me rethink...", "Let me start over..."). Models were not on the verge of a breakthrough — they were trapped. Force the conclusion first.

✔ Recommended Prompt

State your final answer in one sentence. Then explain your reasoning below.
If uncertain, commit to your best answer rather than continuing to deliberate.

RULE 5: DECOMPOSITION 13.9% Prompt Brittleness

Decompose Multi-Constraint Prompts Sequentially

Prompts with 3+ simultaneous constraints (word count AND forbidden words AND casing AND formatting) had high failure rates. Break complex tasks into a sequential pipeline: (1) Generate content for logic, (2) Post-process/format, (3) Validate rules independently.

✔ Pipeline Architecture

Step 1: Generate pure logical solution / code.
Step 2: Re-prompt or filter for formatting & casing constraints.
Step 3: Validate against JSON schema / rule checkers independently.

Part B: Model Selection & Architecture Rules (6–10)

Codified in /local-llm-inference

RULE 6: PARAMETER SCALING 83.1% Sweet Spot

Default to 15B–30B Models for Production Work

Tiny (≤5B) averaged 73.6%, Small (5–15B) scored 72.3%, Medium (15–35B) jumped to 83.1%, and Large (>35B) reached 84.5%. The biggest jump occurs between Small and Medium (~10 points). Beyond 35B, accuracy gains plateau while VRAM and latency skyrocket.

RULE 7: REASONING PARADOX -25.5% Coding Gap

Do NOT Use Reasoning Variants as General Defaults

Models labeled "reasoning" (Phi-4 Reasoning, DeepSeek R1 Distill, Ministral Reasoning) averaged 67.9% overall vs 80.8% for non-reasoning models. Verbose CoT thoughts leaked into JSON and code. Reasoning models only won on Long Context (+11.7 pts).

RULE 8: TOOL-USE AUGMENTATION 20.0% AIME Average

Augment Math & Code with Tool Interpreters

AIME 2024 competition math averaged 20.0% and ClassEval OOP averaged 29.5%. Even top models struggled on Level 4 difficulty. Provide a Python execution sandbox or calculator tool so the model writes code to calculate answers rather than generating raw autoregressive arithmetic.

RULE 9: RAG GROUNDING 65.1% Domain Floor

For Specialist Domains, Always Pair with RAG

Life Sciences & Medicine was the lowest category across all models (65.1% avg). Specific reaction mechanisms (19.0% avg) and clinical calculations expose parametric memory limits. Never rely on training-time recall for clinical, legal, or financial facts without RAG retrieval.

RULE 10: MoE EFFICIENCY 81.7% MoE vs 77.3% Dense

MoE Models Deliver Optimal Throughput & Accuracy

MoE models averaged 81.7% vs 77.3% for dense models. Top overall performers (Nemotron-3 Nano Omni at 88.6% @ 49.5 tok/s, LFM2-24B at 84.7% @ 82.0 tok/s, Qwen-3.6-35B at 84.6% @ 51.2 tok/s) are all MoEs, offering high intelligence at low active VRAM compute.

Visual Analytics & Empirical Proof

Figure 1: Throughput (TPS) vs. Benchmark Accuracy Matrix

Correlation: r = +0.018

Demonstrates that token throughput is completely decoupled from accuracy. MoE models occupy the optimal top-right workhorse quadrant (84%–89% accuracy at 50–82 tok/s), whereas dense heavyweights incur 3x–6x latency penalties for negligible accuracy gains.

🟢 MoE Workhorses: Nemotron-3 (88.6% @ 49.5 tok/s), LFM2-24B (84.7% @ 82.0 tok/s) 🔵 Dense Heavyweights: Gemma-4 31B (88.5% @ 11.0 tok/s), Muse Glimmer (86.3% @ 12.3 tok/s) 🔴 Reasoning Models: Phi-4 Reasoning Plus (49.2% @ 17.1 tok/s)

Figure 2: The "Reasoning Model" Paradox: CoT vs. Standard Models

-12.9% Overall Gap

Comparing Reasoning-tuned models vs. Standard Instruction models across core capability domains. Thinking models collapsed on Coding (-25.5%) and IFEval (-25.8%) due to internal chain-of-thought contamination, while winning on Long Context Needle Retrieval (+11.7%).

Figure 3: Where Model Size Helps vs. Where It Fails

Pearson r by Suite

Parameter size correlates positively with Reasoning (+0.386), Coding (+0.370), and Math (+0.363), but has zero correlation with Life Sciences (-0.037) or Long Context (-0.125).

Figure 4: Stress Ladder Cliff & Token Sweet Spot

Level 4 Drop to 42.2%

Models breeze through Level 2 (86.1%) and Level 3 (85.3%), but crash on Level 4 Olympiad math and ClassEval OOP (42.2%). Accuracy peaks at 91.5% in the 100–300 token output window.

The 7-Factor Root Cause Failure Autopsy (581 Failures Analyzed)

LOGIC FACTUAL DEFICIT 53.4% (310 fails)

Python unit assertion failures or wrong calculations. Remedy: upgrade to 24B+ model or specialized code weights.

REASONING BUDGET EXHAUSTED 20.1% (117 fails)

Infinite self-doubt and looping ("Wait, let me retry..."). 87.3% scored 0. Remedy: early commitment prompts.

PROMPT BRITTLENESS 13.9% (81 fails)

Leaking forbidden words (31), casing errors (24), paragraph counts (21). Remedy: separate formatting passes.

REASONING DRIFT 7.7% (45 fails)

Attention degradation over long token sequences. Remedy: limit output length and set temperature ≤ 0.2.

EVALUATOR INFLEXIBILITY 3.3% (19 cases)

Model was semantically correct but differed on whitespace/wrappers. Auto-normalized for zero false negatives.

INFRASTRUCTURE FAULT 0.7% (4 cases)

LM Studio socket drops or CUDA OOM. Isolated to Remediation Hub without distorting reasoning scores.

Model Deepdive

Comprehensive model evaluation breakdown, test suite performance summaries, fine-grained test case diagnostics, and traffic light deployment profiles.

Select Model to Deepdive:

Loading model deepdive...

Live Benchmark Execution

Execute test batteries against your local LM Studio models with real-time token streaming and instant sandboxed assertions.

Candidate Model to Test:

Test Battery:

Judge Agent (Optional):

LM Studio Sandbox Streaming Engine

READY

[SYSTEM] LocalModelMatrix Sandbox Ready. Select a model & test battery above to begin.

Real-Time Test Assertions & Verifier Logs 0% (0/0)

Ready to benchmark

Test Case	Verdict	Score	Perf
No active tests running.

Model Matrix & Performance Radar

Comprehensive multidimensional analysis comparing Pass@1 accuracy, generation throughput, and domain strengths.

Operational Timing & Latency Telemetry

Average generation time, TTFT responsiveness, peak TPS throughput, and judge grading duration.

Generation Latency Phase 1

--

Average generation time per benchmark test case.

First Token Responsiveness TTFT

--

Time to first token emission from prompt dispatch.

Peak Token Throughput TPS

--

Average generation speed during active streaming.

Judge Evaluation Time Phase 2

--

Average grading duration per qualitative prompt.

Multidimensional Capability Radar & Efficiency Frontier

11-dimensional domain radar, Pass@1 accuracy vs. throughput, and Pareto hardware efficiency curve.

Multidimensional Capability Radar 11 Dimensions

Pass@1 Accuracy vs Throughput Composite

Hardware Efficiency Frontier (Pareto) TPS vs Accuracy

Overall Leaderboard & Composite Rankings

Composite Pass@1 accuracy, throughput, and TTFT across all local candidates.

Rank	Model Name & Identifier	Composite Score	Pass@1 Rate	Tokens/Sec	Avg TTFT	Tests Executed	Actions

Progressive Stress Ladder: Accuracy & Breaking Point Breakdown

Pass@1 accuracy and cognitive resilience across Levels 1 to 5 (Syntax, Nested Schema, Complex Logic, Multi-Constraint, Scale & Long Context).

Model Name	Level 1 Syntax	Level 2 Schema	Level 3 Logic	Level 4 Frontier	Level 5 Scale & 32K	Breaking Point	Overall Pass

Operational Benchmark Execution History & Audit Logs

Chronological audit logs of every test run, execution latency, tokens/sec, and verifier details.

Date & Time	Model Evaluated	Test Battery	Status	Pass@1 Score	Exec Time	Avg TPS	Avg TTFT	Audit Log

Compare Model Performance

Inspect and compare how two models perform across identical test batteries and prompt assertions with side-by-side verification diffs.

Model A (Reference):

Model B (Candidate):

Select two models above and click Compare Models to view side-by-side prompt execution diffs.

Visual Test Authoring Studio

Author custom zero-hallucination benchmark test cases with deterministic sandboxes, math rules, or JSON schema verifiers.

Target Test Suite:

Test Case Name:

Capability Category:

Prompt / Instruction:

Zero-Hallucination Verifier Engine:

Verifier JSON Configuration:

Instant Sandbox Preview:

Settings & LM Studio Hub

Manage local endpoints, inspect discovered models in VRAM, and verify PostgreSQL database persistence.

LM Studio Endpoint

Targeting OpenAI-compatible local port:

Custom Storage Directory

Auto-scan models from directory:

Auto-scans ~/.lmstudio/models & LM Studio APIs

Database Persistence

PostgreSQL with pgvector:

postgresql://localhost:5432/local_model_benchmarks

✓ Schema initialized & pgvector active

Reset Benchmark Data

Permanently wipe all past model scores, test assertions, and leaderboard rankings from PostgreSQL.

Discovered Local Models

Real-time inventory of GGUF and fine-tuned models discovered in LM Studio.

Scanning...

Model Name & Metadata	Identifier & Path	Status	VRAM Management

Zero-Hallucination Evaluation Hub 100% Ground-Truth Verified

About LocalModelMatrix & Evaluation Guide

A high-precision evaluation engine and intelligent task routing platform built specifically for local LLMs running via LM Studio. Eliminates subjective grading by combining isolated code execution sandboxes, mathematical solvers, Ajv JSON schema validators, and rigid IFEval rule engines with real-time hardware profiling.

21 Models Evaluated

92 Standardized Test Cases

1,870+ Verified Assertions

Microsecond Hardware Profiling

Zero-Hallucination Evaluation Architecture

Why deterministic programmatic verification replaces flawed "LLM-as-a-Judge" evaluation pipelines.

Ground-Truth First

Flawed: LLM-as-a-Judge

Subjective & Biased

• Self-Consistency & Sycophancy: LLM judges favor verbose or persuasive phrasing even when mathematical reasoning or logic is wrong.
• Silent Code Failures: LLMs cannot physically run Python/JS code; syntax errors, off-by-one loops, and missing imports frequently receive false passes.
• Blind to Negative Constraints: Judge models struggle to reliably detect subtle formatting violations (e.g., "no commas allowed").
• Non-Reproducible: Scores drift across model versions and random seed variations.

Truth: LocalModelMatrix Engine

100% Deterministic

• Isolated Code Execution: Model code is executed in isolated subprocesses against unit test assertions (assert func(x) == expected).
• Mathematical Solvers: Algebraic normalization and LaTeX token parsers evaluate exact numerical equivalence.
• Ajv JSON Schema Engine: Enforces JSON syntax, required properties, data types, and enum boundaries.
• Strict IFEval Verifiers: Programmatic regex and character counters enforce negative rules, bullet counts, and word limits.

Subprocess Sandboxes

Code generated by models runs in isolated Python and Node environments with rigid execution timeouts and memory bounds.

Hardware Profiling

Real-time measurement of Time to First Token (TTFT), Tokens Per Second (TPS), and peak VRAM allocation per test.

Intelligent Task Routing

Calculates weighted capability matrices across parameter clusters to recommend the optimal model for any target workload.

Failure Autopsy

Automated taxonomy categorizing failures into prompt brittleness, negative constraint leaks, and logic breakdowns.

Canonical Progressive Stress Ladder & Level Mapping

A 5-tier stress testing methodology designed to evaluate the cognitive breaking point of local LLMs—from basic syntax to 32K multi-document reasoning.

100% Official GitHub Datasets

Level	Cognitive Dimension	Official Benchmark Suites & Datasets Mapped	Evaluator Engine	Production Implication
Level 1	Baseline Syntax & Delimiters Format markers & echo verification	Format delimiter separation, strict prompt echoing, ASCII framing, single-rule sanity checks.	`IFEVAL_RULES` / `SYNTAX_PARSER`	Basic Chat
Level 2	Nested Schema & Strict Typing Standard coding, arithmetic & JSON	• OpenAI HumanEval (Python code generation) • OpenAI GSM8K (Grade-school math) • Google IFEval (Negative rules & formatting constraints) • Ajv JSON Schema (Strict schema & type enforcement)	`CODE_SANDBOX` / `MATH_EXACT` / `JSON_SCHEMA`	Copilot & Forms
Level 3	Complex Single-Task Logic Specialized domain reasoning	• Humanity's Last Exam (HLE) (Graduate STEM) • Life Sciences (PubMedQA, Clinical Pharmacology) • Legal & Finance (FinanceBench EBITDA, LegalBench NDA) • EleutherAI Harness (ARC-Challenge, MMLU, HellaSwag) • Vision & Multimodal (VLMEvalKit, LMMs-Eval)	`MATH_EXACT` / `SEMANTIC_FACTS` / `KG_TRIPLES`	Specialist Tools
Level 4	Frontier Multi-Constraint & Reasoning Multi-tool calling, Olympiad math, OOP	• Berkeley Function-Calling (BFCL v3) (Multi-tool calling with strict argument types & <tool_call> tags) • AIME 2024 (American Invitational Math Olympiad integer proofs) • ClassEval (Class-level multi-method Python OOP modules)	`JSON_SCHEMA` / `MATH_EXACT` / `CODE_SANDBOX`	Autonomous Agents
Level 5	Scale, Long Context & Distraction 16K–32K context & needle retrieval	• LongBench QASPER (Multi-document research synthesis across 15,000+ token academic papers) • NVIDIA RULER / NIAH (16K–32K needle matrix at 10%, 25%, 50%, 75%, 90% depth) • RAG Vector Grounding (Anti-hallucination dense semantic search)	`SEMANTIC_FACTS` / `NIAH_RETRIEVAL`	Enterprise Discovery

Level 1: Baseline Syntax Tier 1

Validates basic instruction comprehension, output delimiter handling, and prompt echoing without conversational derailment.

Target: >95% Pass Rate required for any production deployment.

Level 2: Nested Schema Tier 2

Evaluates function-level Python algorithms, grade-school multi-step arithmetic, and negative formatting constraints.

Target: >80% Pass Rate for copilot coding and structured data pipelines.

Level 3: Domain Logic Tier 3

Probes complex single-task domain knowledge across clinical pharmacology, financial accounting (EBITDA), and graduate STEM.

Target: >70% Pass Rate for legal analysis and biomedical literature search.

Level 4: Frontier Reasoning Tier 4

Stresses multi-constraint stacking: multi-tool function selection (BFCL), 15–30 step Olympiad math proofs (AIME 2024), and object-oriented class architecture (ClassEval).

Target: Required for autonomous multi-agent systems and software engineering.

Level 5: Scale & Long Context Tier 5

Pushes attention mechanisms across 15,000–32,000 token documents (LongBench) and tests position-invariant needle retrieval across 10%–90% context depth (NVIDIA RULER).

Target: Required for SEC filing analysis, legal discovery, and repository-scale RAG.

Interactive Application Tabs Tour

Click any tab below to inspect its features, capabilities, and open it directly.

8 Integrated Modules

Deterministic Verifier Simulation Lab

Explore how LocalModelMatrix verifies LLM responses across 7 standardized evaluation engines.

Live Simulator

How to Read & Interpret Benchmark Results

Understanding Pass@1 vs. partial credit scores, throughput thresholds, latency, and parameter clusters.

Analytics Guide

Pass Rate vs. Score

• Binary Pass Rate (Pass@1): Measures all-or-nothing execution success. If any unit test assertion fails or a required JSON key is missing, the test receives a 0% pass rate.

• Granular Score (0–100%): Rewards partial constraint satisfaction (e.g. satisfying 2 of 3 IFEval formatting rules yields a 67% score).

Throughput (Tokens/Sec)

⚡ Ultra-Fast (>90 TPS): Autonomous multi-step agent loops (e.g. Granite 3B, Lfm2 24B).
🚀 Fast (50–90 TPS): Real-time interactive chat & coding (e.g. Nemotron 3 Omni, Gemma E2b).
⏱️ Moderate (20–50 TPS): Deep document reasoning & synthesis (e.g. Gemma E4b, Bonsai 27B).
⏳ Deliberative (<20 TPS): Heavy dense models for batch jobs (e.g. Gemma 31B QAT).

Time to First Token (TTFT)

Measures prompt ingestion / prefill speed before the first output token is emitted.

<500ms TTFT feels instantaneous for chat users; large prefill delays (>5s) indicate heavy context or unaccelerated prompt processing.

VRAM & Cluster Tiers

• Extra Small (<10B): 6GB–8GB VRAM (Edge devices, MacBooks, laptops).
• Small (10B–20B): 10GB–16GB VRAM (Mid-range GPUs like RTX 4070 / 4080).
• Medium (21B–40B): 18GB–24GB VRAM (Single RTX 3090 / 4090 / Mac Studio).
• Large / XL (>40B): 48GB+ VRAM (Dual GPU or high-RAM unified memory).

Key Empirical Benchmark Findings

1. Mixture-of-Experts (MoE) Dominance:
MoE models like Nemotron 3 Nano Omni (30B A3B) and Lfm2 24B A2B achieved top overall accuracy (95.3 and 91.9) while delivering 3x–4x higher token generation speeds (60–88 TPS) than 120B dense models.

2. IFEval is the True Discriminator:
While most models score near 100% on standard coding and math, strict negative formatting rules ("never use commas", exact bullet counts) expose severe brittleness in smaller dense models.

3. Sub-10B Models are Production Ready:
Gemma 4 E4b (7.5B) reached a 93.7 overall score (92.4% pass rate), outperforming several 20B–35B dense models on standard coding and math while fitting in 8GB VRAM.

Interactive Task Routing & Deployment Playbook

Select your intended production workload to view the recommended local model architecture and prompting best practices.

Playbook Recommender

Frequently Asked Questions & Operational Tips

Everything you need to know about LM Studio integration, unattended batch testing, and failure debugging.

Knowledge Base

How does LocalModelMatrix connect to LM Studio?

LocalModelMatrix connects to LM Studio's local OpenAI-compatible HTTP server (http://127.0.0.1:1234/v1). It queries GET /v1/models to auto-discover downloaded models and streams generation tokens via Server-Sent Events (SSE). Additionally, it integrates with the LM Studio CLI (lms load and lms unload) to automate unattended batch evaluations across multiple models.

How do I run unattended batch benchmarks overnight?

Navigate to the Live Runner tab. Check all models and test suites you wish to benchmark, and click "Run Batch Benchmark". The orchestrator will sequentially load each model into VRAM, execute every test assertion, record throughput and latency, unload the model, and transition to the next candidate automatically.

Why do some models score 100% on coding but fail IFEval tests?

Coding models are heavily trained on synthetic unit tests and docstrings, making them proficient at algorithmic syntax. However, strict negative constraints (e.g. "do not use commas", "output exactly 2 paragraphs", "never mention word X") require rigid attention suppression. Smaller models or conversational reasoning models often suffer from "negative constraint leakage," where they inadvertently include conversational preamble or ignore formatting guardrails.

Can I author my own test suites and ground-truth assertions?

Yes! Head to the Test Studio tab. You can create custom prompts and pair them with any of the 7 supported verifiers (Code Sandbox unit tests, Exact Math, JSON Schema with Ajv, IFEval negative rules, or Semantic Checklists). You can execute your custom tests immediately and export them into persistent benchmark batteries.

What happens if PostgreSQL is not running locally?

LocalModelMatrix includes built-in resilience. If PostgreSQL is not active, the system automatically falls back to high-performance in-memory caching with automatic persistence to data/store_cache.json. You can run full benchmarks, view leaderboards, and inspect dossiers without any external database dependencies.