InferenceRate Live Token Economics
Compare Cost
Home / Methodology & Provenance

Benchmarking Methodology & Autonomous Agent Governance

To maintain verifiable objectivity and machine-readable accuracy for engineering teams, Google Search, and LLM grounding engines, InferenceRate adheres to transparent mathematical formulations and a closed-loop 5-agent verification pipeline powered by Gemini 4 Argon (High).

1. Inference Value Ratio (IVR) & Pareto Frontier Formulation

The Inference Value Ratio (IVR) is a calibrated composite efficiency index scaled from 10.0 to 99.9 that quantifies verified capability delivered per dollar of production token spend across 37 tracked models.

5-Benchmark Composite Intelligence:
Composite_Intelligence = (SWE_bench × 0.30) + (GPQA_Diamond × 0.25) + (MMLU_Pro × 0.20) + (MATH_500 × 0.15) + (HumanEval × 0.10)
3:1 Production Blended Token Cost ($ / 1M Tokens):
Blended_Cost = (Input_Cost_per_1M × 0.75) + (Output_Cost_per_1M × 0.25)
Logarithmic Cost-Efficiency Score:
Cost_Damper = 1.0 + 0.52 × ln(1.0 + (Blended_Cost ÷ 0.15))
IVR_Score = min(99.9, max(10.0, (Composite_Intelligence ÷ Cost_Damper) × 1.52))

Cost-vs-Intelligence Pareto Frontier: In addition to IVR, every model is evaluated against the catalog's Pareto efficiency boundary. A model is designated Pareto-Optimal (5 models currently qualify) if and only if no other model achieves a strictly higher Composite_Intelligence score at an equal or lower Blended_Cost.

2. Latency (TTFT) & Streaming Throughput (tok/s) Protocol

Time to First Token (TTFT) and generation throughput (tokens per second) are tracked across 10 cloud inference providers and normalized across standardized contextual loads:

  • Standardized Workload Probe: A 512-token system + user prompt requesting structured code synthesis with streaming enabled (stream: true).
  • Warm Connection Measurement: HTTP/2 TLS sessions are pre-warmed to isolate model prefill latency (TTFT) and decode throughput (tokens/sec) from DNS/TCP handshake overhead.
  • Multi-Provider Tracking: Open-weight models (such as DeepSeek-R1, Qwen 3.8, Llama 4 Maverick, and Gemma 4 31B IT) are benchmarked across both native lab APIs and high-speed hardware accelerators (Groq LPU, Cerebras Wafer-Scale Engine).

3. Autonomous 5-Agent Pipeline (Powered by Gemini 4 Argon High)

InferenceRate operates via a closed-loop 5-agent architecture running gemini-4-argon with thinkingConfig: {"thinkingLevel": "high"} and live Google Search Grounding:

Agent 1: Data Scout (Grounding) Executes Gemini 4 Argon (High) with Google Search Grounding to verify new model releases, token price drops, and 5-benchmark leaderboards.
Agent 2: Gap Synthesizer Synthesizes 666 pairwise comparison matrices and AEO direct answers, prioritizing GSC-impressed and flagship matchups.
Agent 3: Link Graph Optimizer Computes GSC-weighted priority scores and maintains 8,873 internal semantic links to eliminate orphan pages.
Agent 4: Self-Healing Auditor Ingests live 30-day Google Search Console telemetry and heals AEO direct answer chunks for citation density and word count compliance.
Agent 5: Programmatic Publisher & Open Data Exporter Computes Pareto-optimal frontier flags, exports static Astro datasets, generates 308 permanent redirects in vercel.json for renamed legacy model slugs, updates /llms.txt & /llms-full.txt, and publishes open JSON endpoints.

4. Open Machine-Readable Endpoints for Agents & RAG

Autonomous coding agents, MCP servers, and financial modeling scripts can ingest InferenceRate telemetry directly without scraping HTML:

GET /api/models.json Full Model Catalog → GET /api/leaderboard.json IVR & Pareto Leaders → GET /llms.txt Concise Agent Index → GET /llms-full.txt Complete Markdown Matrix →