Same Model, Two Hosts, Different Answers

Abstract §

We routed an open-weights model, MiniMax-M3, through two hosted inference providers — Together AI and Ollama Cloud — inside a production government contracting platform. An uncontrolled comparison showed one provider scoring 9.7 points higher than the other on the same task with the same prompt.

Controlling two variables in our own evaluation harness removed almost all of that gap. The same task re-measured at 0.5 points apart. What survived controlling was a different, smaller, and more specific set of differences.

The practical finding is not that one provider is better. It is that an apparent provider difference is, by default, unmeasurable: two ordinary harness defects produced a 9.7-point difference, of which 0.5 points remained after they were fixed. Anyone comparing hosted inference providers is likely measuring their own test rig.

1. Why this came up §

Open-weights models are increasingly served by multiple hosts. The same checkpoint name appears on several providers, and the natural assumption is that routing to whichever is cheaper or faster is a purely commercial decision — that the model is the model.

Our platform assigns different models to different jobs (opportunity scoring, document summarisation, proposal drafting, adversarial review, translation, specification writing). We built a comparison harness to make those assignments on evidence rather than preference: the same prompt goes to N models, and a held-out judge model scores the anonymised, shuffled answers.

Running that harness, we recorded MiniMax-M3 scoring 78.5 on Ollama Cloud and 68.8 on Together for plain-language summarisation. Same weights, same prompt, same judge. That is a large enough gap to route production traffic on, so we tried to explain it.

2. Two defects in the harness §

2.1 Sampling temperature was never specified §

Our client sent {model, max_tokens, messages} and nothing else. No temperature, no top_p.

Every provider applies its own default when sampling parameters are omitted, and those defaults are not the same number. We were not asking the two hosts the same question. The comparison was not measuring the model; it was partly measuring two different sampling regimes.

2.2 The harness did not use production's token budgets §

Each task in the harness ran at a flat 1,500 max output tokens. Production passed between 1,200 and 6,000 depending on the job.

For reasoning models this is not a minor discrepancy. A reasoning model spends its budget on hidden reasoning before it emits a visible character, so a starved budget does not produce a shorter answer — it produces no answer. On the task whose production budget is 6,000, four of six models under test returned reasoning and no text. The harness recorded those as model failures and recommended the fifth-best model for the slot.

The bias is directional: starving the budget systematically favours models that think least before answering.

3. Method §

Platform. A live multi-tenant SaaS for US federal opportunity discovery and bid preparation. All prompts are the production prompts, not paraphrases — a paraphrased prompt measures the harness, not the model.

Contestants. together/MiniMaxAI/MiniMax-M3 and Ollama/minimax-m3:cloud — the same model name as published by each host.

Judge. OpenAI/gpt-5.6-luna, held out of the contestant field. A judge that also competes rates every answer 5–11 points higher without changing the order, which makes cells incomparable, so it is removed rather than flagged. Answers are anonymised and shuffled before scoring; the judge sees head and tail of the prompt, because head-only truncation once hid the document under review and inverted an entire column of results.

Tasks. Nine production jobs: fit scoring, plain-language summary, questions for the buyer, proposal section drafting, build summary, adversarial red-team review, Arabic translation, build-specification drafting, and scanned-page transcription.

Corpus. Three real solicitations of differing length and source (73.9k, 86.9k and 17.6k characters), two runs each.

Controls applied before the final measurement.

1. Sampling temperature pinned explicitly at 0.2 for every provider.
2. Each task run at the same max-token budget its production caller passes (1,200 / 2,000 / 2,500 / 4,000 / 6,000 by task).

Ranking uses average position, not average score: measured across our corpus, the whole field scored 39–66 on one notice and 78–94 on another, so averaging raw scores mostly records which documents a model happened to draw.

4. Results §

4.1 The gap largely disappears under control §

4.2 Controlled results, all measurable tasks §

Judge scores, 0–100. Failures are calls that returned no usable text.

Build-specification drafting could not be measured: on every run at least one contestant returned nothing, leaving the judge a single answer and no ranking to make. We report it as unmeasured rather than as a result.

What survives controlling is one decisive difference — fit scoring, 10.0 points to Together — and near-parity on six of the remaining seven tasks. Both tasks Ollama Cloud leads are inside a point.

4.3 Operational differences are larger than quality differences §

From production telemetry over the same period:

Ollama Cloud answered roughly 1.8× faster; Together failed roughly 4× less often. Observed failures on Ollama Cloud included read timeouts and empty completions where reasoning consumed the whole budget.

This comparison is observational, not controlled: the two endpoints did not receive identical task mixes over the period. We report it because the effect size is large and consistent, not because it is experimentally clean.

4.4 Some models reject a pinned temperature §

Of seven models we pinned, two refused — the reasoning families answer an explicit temperature with an HTTP 400 and accept only their own default. One of them was our judge. Any client that pins sampling must detect this from the error and retry without the parameter, or it will lose exactly the models most sensitive to sampling.

5. Interpretation §

We did not establish why the two hosts differ where they still differ. Any of the following would explain it, and we could not verify them from outside: quantisation format, serving-stack differences, checkpoint revision, or remaining sampling parameters we do not set (top_p, top_k, repetition penalty).

We did establish something we consider more useful:

The difference an uncontrolled harness reports between two providers can be several times larger than the difference that survives control — and can point the other way.

Before controls, our data said “route summarisation to Ollama Cloud, by 9.7 points.” After controls, that task is a tie and the real difference is elsewhere entirely. A team acting on the first measurement would have made a routing decision on an artefact of their own test rig.

6. Practical recommendations §

For anyone comparing hosted inference providers:

1. Pin sampling explicitly. If you do not send temperature, you are comparing each provider's defaults as much as the model. Handle the models that reject it.
2. Give the harness production's token budgets. A benchmark at a different budget than production measures a different system — and starving reasoning models silently converts quality differences into failure counts.
3. Distinguish a failed call from a bad answer. Ours conflated them, and a harness-induced failure was read as a model deficiency.
4. Hold the judge out of the field. A judge that competes inflates every score without changing the order.
5. Rank by position, not by mean score, if your corpus is heterogeneous. Otherwise you mostly record which documents a model drew.
6. Name what you could not measure. A task that silently drops out of a results table reads as though it was never asked for.

For teams choosing between these two providers specifically: on our workload the quality difference after control is small and task-specific, while the latency and reliability difference is large. That suggests routing on operational characteristics — fast where a human waits, reliable where a dropped call costs work — rather than on aggregate quality scores.

8. Conclusion §

Two providers serving the same open-weights model produced measurably different results. Most of the difference we first observed was our own instrumentation. The residue is real but smaller and differently shaped than the original measurement suggested, and it is smaller than the operational differences between the two hosts.

If you are choosing between providers for an open-weights model: control your harness first, then measure. Otherwise the number you are about to act on may mostly be about you.