How We Rank AI Models Without Publishing a Score

Every week another model tops another leaderboard, then disappears when the prompt or harness changes. For someone choosing a model for a product, that is a feed, not a decision aid.

The NerdLevelTech model index does something deliberately boring: it publishes the facts a buyer can check — price, context window, licence, release date and links to the maker's pages — and labels our opinions as opinions. It does not publish a score out of 100, because we do not run the benchmark harness that would make such a score ours.

Here is the method, the four picks, and where a number can mislead even when it is technically correct.

What we publish, and what we refuse to

The hub currently tracks 21 current models from 13 makers. Ten are open-weight or have open weights announced. For each one you get:

• Money and limits: published API prices in USD per 1M input and output tokens, plus the context and output limits, including windows that are shared rather than additive.
• Licence status: proprietary, open weights you can download now, or open weights announced but not yet public.
• Provenance: a dated release where the maker published one, the maker's pricing and docs pages, and — only when they resolve — the Hugging Face repo and exact Ollama tag.
• One current card per model line. A successor archives the old card and its URL 308s to the new one; the index does not accumulate dead models.

Every entry carries a verified date. Prices and limits change; a page that does not say when it was checked is asking you to trust a memory.

We do not publish a benchmark score. We do not run the same prompt across 21 models and call the output a ranking, and we do not relabel someone else's benchmark table as ours. If you need MMLU, SWE-bench or an arena Elo, read the people who ran that evaluation. Our job is narrower: what each model costs, what it accepts, and what you can legally do with it.

The cost number is a rate, not an invoice

A per-token price is easy to quote and hard to feel. So the hub also prints a fixed task cost: 50,000 input tokens plus 10,000 output tokens, at the published headline rates.

The arithmetic is simple: at $4 input and $20 output, that task costs $0.40. At $0.30 and $1.20 — DeepSeek V4.1 Flash — it costs $0.027. At $0.15 and $0.50 — GLM-5.3 Flash — it costs about $0.013. Those differences matter more than a few percentage points on a synthetic exam.

The label is "cost per task", not "cost per month". It ignores prompt caching, batch discounts, off-peak rates and tier crossings, and the pricing data records the exceptions: Grok changes price above a 200K-token prompt, DeepSeek has an off-peak rate, and Gemini's introductory prices have end dates. The leaderboard prints the task definition next to the column.

Context is a window, not a sum

The most common misleading sentence in model coverage is some version of "1M context and 128K output". It sounds like 1,128,000 tokens. It usually is not.

OpenAI's GPT-6 family publishes a 1,050,000-token window, split into at most 922,000 input and 128,000 output tokens.¹ Anthropic's 1M window is shared: input and output both count toward it, with a 128K output cap.² Gemini's 4 Argon and 3.8 Flash pages state a 1,048,576-token window and a 65,536-token standard output limit.³ Those shapes differ, which matters when you pack a long document into a prompt.

The model pages therefore print the exact window, the max input and max output where the maker publishes them, and a note when the split is not published. A card may round to "1M" or "1.05M" so it fits; the detail page does not.

The four picks are editorial, and that is the point

The leaderboard carries four awards. They are opinions, dated, and chosen for four different jobs:

• Best overall: Claude Opus 5.5. Anthropic's default starting point for demanding work, with a 1M window, documented tool use and a $4/$20 price.
• Best value: GPT-6.1 Sol. Near-Astra positioning for complex coding at $2/$10, with the same 1.05M window.
• Best budget workhorse: DeepSeek V4.1 Flash. Vision, a 1M-token window and tool use at about $0.027 per task, half that off-peak. It is the one we would hand an agent.
• Best open weights: GLM-5.3 Flash. A dedicated Hugging Face repo, an Ollama library, a 1M window and image input at $0.15/$0.50. The full GLM-5.3 costs about nine times more for the same window.

A pick is not a claim that the model wins every task. It is the card we would start with for that job at the checked price; the rationale and caveat sit next to it. If a pick changes, the page changes; no second, contradictory winner lingers.

Open weights means downloadable, or it says so

"Open" is doing a lot of work in model marketing. The hub separates three states:

1. Open weights, published. There is a Hugging Face repo or an Ollama page you can use today. The card links the exact repo and tag.
2. Open weights, announced. The maker has promised a release but the weights are not public. Kimi K3.1 Pro and Mistral Large 4 are in this state at the time of writing.⁴ Their cards say "Open weights (announced)" and do not carry a download link or a self-hosting tag until the release exists.
3. Proprietary. The model is API- or subscription-only, and the card says so.

This is also why the self-hosting filter is narrower than the licence count. A promised release is not a deployment option yet.

How to use the hub

Start at the model index. Recent is the default sort, so the newest cards are visible. Filter by maker, open weights, or use case; every card links the full page.

The leaderboard groups the table by maker, collapsed by default; expand a group and every column sorts. The methodology covers succession and what happens to an old URL.

If you are an agent, the same facts are available as data:

• /models/index.json — every current model with prices, limits, provenance links and picks.
• /models-rss.xml — the model change feed, newest first.
• The MCP server at https://nerdleveltech.com/api/mcp exposes search_models and get_model with the same facts. The A2A endpoint mirrors them.

What would change our mind

If we ran a reproducible harness with published prompts, seeds and scoring code, we could publish a score. We have not, so we do not. If a vendor publishes an exact context split, we add it. If a price changes, we update the card and the date. If a model is replaced, the old URL redirects to the successor.

That is the whole trick: fewer claims, each one traceable. A model page should let you answer "how much, how long, under what licence, and says who?" without trusting a number we made up.

References

Corrections are welcome. If a price, limit or link is wrong, send the source and the date to agents@nerdleveltech.com.