AI Models 101 — LLM Index

If you've ever wondered why some AI models need a data center while others run on a gaming PC, why "70B" is a big deal, or what "quantized" means — this page connects the dots, in plain language.

1. Parameters: the "billions" in every model name

A large language model is, at its core, an enormous grid of numbers called parameters (or weights). These numbers are what the model learned during training — its knowledge, its reasoning patterns, its sense of language, all encoded as billions of tiny numeric values.

When you see a model called "7B", "70B", or "405B", that's the parameter count: 7 billion, 70 billion, 405 billion. As a rough rule:

More parameters → more capability. Bigger models hold more knowledge, follow more complex instructions, reason through longer chains of logic, and produce stronger, more refined output.
Fewer parameters → simpler behavior. Small models can still write, summarize, and chat — but they make more mistakes, lose track of complex tasks sooner, and struggle with multi-step reasoning, subtle code bugs, and nuance.

Intuition: parameters are like the resolution of a camera. A high-resolution sensor captures fine detail; a low-resolution one still takes a picture, but the fine detail just isn't there. No prompt can add detail a model doesn't have.

2. Why parameters cost memory: the VRAM math

Every parameter has to be loaded into memory (ideally fast GPU memory, called VRAM) to run the model. Stored at full training precision (16-bit, i.e. 2 bytes per parameter), the math is simple:

Model size	Memory at 16-bit	What can hold it
7B	~14 GB	One consumer GPU (RTX 4090/5090)
70B	~140 GB	2× server GPUs (e.g. H200)
405B	~810 GB	A rack of 6–8 server GPUs working together
~1T+ (frontier)	multiple TB	Clusters of many GPU servers

And that's just to load the model — you also need extra memory for the conversation context, and enough compute to push tokens through all those billions of parameters quickly.

3. The cloud giants: Claude, Gemini, ChatGPT

The big cloud models you use every day run on data-center GPUs — hardware in a completely different league from anything consumer:

An NVIDIA H200 has 141 GB of ultra-fast HBM memory (~4.8 TB/s bandwidth). A B200 pushes that to 192 GB at ~8 TB/s. Compare: a top-end gaming card (RTX 5090) has 32 GB at ~1.8 TB/s.
These GPUs are deployed in clusters of eight or more per server, and many servers per model, linked by specialized interconnects so they act like one giant GPU.
That's what lets frontier models run at hundreds of billions to trillions of parameters — at full quality — and still respond in seconds while serving millions of people simultaneously.

This is the core trade you make with cloud AI: you don't own the hardware, you pay per token (that's what the index compares) — but you get maximum capability and speed with zero setup.

4. Quantization: compressing the model

What if you want to run a model on hardware that can't fit it? You quantize it — store each parameter with fewer bits. It's a form of compression:

Precision	Bytes/param	70B model needs	Quality
16-bit (full)	2.0	~140 GB	Original quality
8-bit	1.0	~70 GB	Nearly indistinguishable
4-bit	~0.55	~40 GB	Small but real quality loss
2–3-bit	~0.3–0.4	~25–30 GB	Noticeable degradation

The catch: every bit you shave off rounds the model's knowledge more coarsely. At 8-bit you barely notice. At 4-bit, the model is still very usable but starts slipping on exactly the things that need precision — complex reasoning, tricky code, subtle instructions. Below 4-bit, quality drops fast.

The double squeeze: hobbyists often run a smaller model and quantize it heavily to fit their GPU. Each step trades away capability — fewer parameters means less raw ability, and aggressive quantization blurs what's left. A 7B model at 4-bit is a genuinely useful tool, but it is not a compressed Claude — it's a different class of machine.

5. Running models locally: the honest picture

Local models are a fantastic and fast-improving world — private, free per token, always available, fully under your control. But set expectations correctly:

A single consumer GPU (16–32 GB) comfortably runs models in the 7B–32B range, quantized. That's excellent for chat, summarization, drafting, simple coding help, and automation.
To approach cloud-frontier quality and speed you'd need multiple server-grade GPUs (H200-class) — tens of thousands of dollars each, plus power and cooling. Short of that, local setups won't match the big cloud models on complex, multi-step "neural heavy-lifting" tasks.
Speed matters too: even if a big model fits (e.g. partially in regular RAM), it may generate only a few tokens per second — too slow for real work.

Where local shines: don't fight the frontier — go specific. A small model fine-tuned (additionally trained) for one narrow job — classifying your emails, extracting fields from invoices, tagging support tickets, drafting in your exact style — can match or beat a giant general model at that one task, while running cheaply and privately on your own hardware.

6. Putting it together: agents, tools, and scripts

The most effective real-world systems don't ask one model to do everything. They combine AI with ordinary deterministic code — regular scripts that are fast, free, and never hallucinate:

Use a script for anything with an exact answer: math, date handling, database lookups, file operations, API calls, validation. Code is always right about these; models sometimes aren't.
Use a model for what code can't do: understanding messy human language, making judgment calls, writing, summarizing, deciding which tool to use next.
An agent is exactly this combination: a model in a loop with a set of tools it can call. Some tools are AI (a small classifier model, a vision model), some are plain code (a calculator, a web scraper, a database query) — the model orchestrates, the tools execute.

This is why the "small local model" story matters: an agent doesn't need frontier intelligence in every component. A well-designed pipeline might use a cloud model for the hard reasoning, a tiny fine-tuned local model for a high-volume repetitive step, and plain scripts for everything deterministic. Each part does what it's best at — and that's how you get the most out of all of it.

7. Cheat sheet

Term	Plain meaning
Parameters (B)	The model's learned numbers, in billions. More ≈ more capable, more memory, more compute.
VRAM	GPU memory. The model must fit in it to run fast. Rough rule: GB needed ≈ params × bytes-per-param.
Quantization	Storing parameters with fewer bits to shrink the model. 8-bit ≈ free; 4-bit = mild loss; below that, real loss.
Tokens	The chunks of text models read and write (~¾ of a word each). Cloud pricing is per million tokens — see the index.
Context window	How much text the model can consider at once, measured in tokens.
Fine-tuning	Additional training that specializes a model for your specific task or style.
Inference	Actually running the model to generate output (vs. training it).
Agent	A model in a loop with tools — some AI, some plain code — that it calls to get real work done.

Ready to compare the cloud side? The LLM Index ranks coding models by real per-request cost and blended quality, and the subscriptions page compares flat-rate coding plans by tokens per dollar.