Ollama context length and local model sizing for Iris

The most common reason a self-hosted model gives poor answers in rConfig isn’t the model. It’s the context window. Ollama picks a default based on GPU memory, and on a typical single-GPU server that default is 4k tokens: too small to hold a full router config, rConfig’s background knowledge and your question. The request runs anyway, and the model answers from whatever fitted.

Use this page when: you run Iris on Ollama, vLLM or another OpenAI-compatible server, and answers about long configs are shallow, wrong or cut off.

What each feature sends

These are rConfig’s own limits, taken from the code. Token counts are estimates: configuration text tokenises at roughly 3 to 4 characters per token.

Feature	Largest input rConfig sends	About	Answer cap	Context to plan for
Config analysis, first chat message	Config up to ~100,000 chars (larger configs are reduced to key sections, then cut to 50,000) + background knowledge up to 14,000 chars + instructions	30k to 38k tokens	4,096	48k for very large configs, 32k for most
Config analysis, inline and lightbulb	Selected block; full config only if under 50,000 chars	up to 17k tokens	4,096	24k to 32k
Drift narration	Diff up to 8,000 chars + up to 4,000 chars of config + knowledge up to 10,000 chars	~6k to 7k tokens	4,096	16k
Compliance narration	Rule results (no config) + knowledge up to 10,000 chars	~3k to 4k tokens	2,048	8k to 16k
Optimisation assistant	Docs passages up to 8,000 chars + conversation	grows with the chat	4,096	16k

Follow-up chat messages send only the new question, but the conversation so far still occupies the context window, so long chats need the larger figure.

Ollama’s default and how to change it

Ollama documents its default context length by GPU memory:

GPU memory	Default context
Under 24 GiB	4k tokens
24 to 48 GiB	32k tokens
48 GiB or more	256k tokens

Under 24 GiB, raise it where Ollama serves the model:

OLLAMA_CONTEXT_LENGTH=32768 ollama serve

Or use the context slider in the Ollama app’s settings. rConfig doesn’t send a context length with each request, so the server setting is the one that applies.

For vLLM, the equivalent is the server’s maximum model length setting; check the vLLM documentation for your version. Other OpenAI-compatible servers have their own setting.

Choosing the model

Fit the window first, then pick the largest model that fits. A mid-sized model with a 32k window beats a larger one squeezed into 4k for config work.
Small models (7B to 8B) handle drift and compliance narration and short configs. Expect weak answers on long firewall and core configs.
Test with your longest config. Open your biggest device in rConfig, click Ask AI, and ask about something near the end of the file. If the answer can’t see it, the window is too small.

Frequently asked questions

What context length does Ollama use by default?

It depends on GPU memory: 4k tokens below 24 GiB of VRAM, 32k from 24 to 48 GiB, and 256k at 48 GiB or more, according to Ollama's documentation.

How do I raise Ollama's context length for rConfig?

Set it where Ollama serves the model, for example OLLAMA_CONTEXT_LENGTH=32768 ollama serve, or with the context slider in the Ollama app. rConfig doesn't set it per request.

What context window does rConfig's config analysis need?

For a typical config, 16k to 32k tokens. For very large configs near rConfig's 100,000-character limit, plan for about 48k tokens including the answer.