Renaming your JSON keys can change your LLM's answer
Here's a finding that should quietly unsettle anyone who ships structured LLM output: renaming a field in your JSON schema, changing nothing else, can measurably change the model's answer. Not just the format, the answer. Your field names, it turns out, aren't labels the model politely fills in. They're instructions it reads, whether you meant them as instructions or not.
The study
A recent preprint (arXiv 2604.14862, "Schema Key Wording as an Instruction Channel in Structured Generation under Constrained Decoding," by Yifan Le) tested exactly this. Hold the prompt fixed, the model fixed, the output structure fixed, the decoding backend fixed, and change only the wording of the schema keys. Then measure accuracy on reasoning benchmarks (GSM8K, MATH500). The keys move the needle, and not by a rounding error:
- Qwen2.5-7B on GSM8K: 79.6% → 86.5% just from a better-worded key. Nearly seven points, for free, from renaming a field.
- Llama-3.2-3B on the same task: 53.2% → 37.4%. The same kind of key wording that helped Qwen stripped almost sixteen points off Llama.
The effect is real, it's large, and it's bidirectional. A key name that acts like a helpful hint on one model acts like a distraction on another.
Why this isn't the "format tax"
You may have read that forcing JSON output degrades reasoning; that's a real and separate finding (the "format tax," from work like "Let Me Speak Freely?"). That's about the constraint — the act of boxing the model into a rigid structure costs it some room to think. This is different and, to me, sneakier. It's not that structure hurts; it's that the names inside the structure are a second, hidden prompt. A key called step_by_step_reasoning is telling the model to reason step by step, as surely as if you'd written it in the system prompt. A key called guess tells it something else entirely. You've been prompting your model through your data model this whole time without noticing.
Your schema is not a passive container the model pours an answer into. It's an instruction the model reads on the way to answering. Field names are prompt engineering you didn't know you were doing.
The part that makes it a real problem
Two findings from the paper turn "huh, interesting" into "I need to test this." First, it's model-dependent: the Qwen family leans on schema-level signals, the Llama family leans on the prompt, and a schema tweak that's a win on one is a regression on the other. Second, prompt-side and schema-side instructions are not additive. You can't reason about them independently and add up the effects; they interact in ways you have to measure, not predict. So if you swap models, or a vendor updates one under you, your carefully-named keys might silently flip from helping to hurting, and nothing in your diff would show it.
What I'm doing about it
I run a JSON:API-backed AI editor on this blog, so structured output is not abstract for me. Concretely:
- Treat field names as a tuning knob, not documentation.
final_answervsanswervsresultis a decision that can move accuracy, not a style preference. (Fair caveat: the paper demonstrates the effect and tests keys likestepsvsstep_by_step_reasoning; the specific "answer vs guess" pairs are my extrapolation of the principle, not its measured results.) - A/B the naming. It's a cheap experiment — same prompt, two schemas, run your eval set. Several points of accuracy can be sitting in a word choice.
- Re-test when the model changes. Because the effect is model-specific and non-additive, "it worked on the old model" is not evidence it works on the new one.
- Keep reasoning fields before answer fields — the older format-tax lesson still applies; don't let the structure force the conclusion before the working.
It's a preprint, not gospel, so I'm holding the exact numbers loosely. But the direction rings true and it costs almost nothing to check on your own workload. The uncomfortable takeaway is that the boundary between "prompt" and "schema" was never as clean as the tooling pretends. The model reads all of it.