What fixes an LLM agent's incorrect JSON arguments for a complex tool?

WHAT THIS TESTS: This question probes your understanding of the full stack for reliable structured generation. A senior engineer knows that prompt engineering alone cannot guarantee syntactically correct and semantically valid JSON from an LLM, especially for nested schemas like create_chart with multiple objects and enums. The interviewer wants to see that you layer user-facing prompt techniques with backend validation, constrained decoding, and failure recovery. They are also checking whether you distinguish between generic text completion and native function-calling modes.

A GOOD ANSWER COVERS: Four layers in order. First, prompt engineering techniques: provide few-shot examples of valid and invalid JSON calls with explanations, require chain-of-thought reasoning before the final JSON block to reduce hallucination, and embed the JSON Schema directly in the system prompt so the model sees types, required fields, and enums. Second, native capabilities: use the model's built-in function-calling mode if available, since GPT-4 and similar models are fine-tuned to emit adherent schemas. Third, system-level guardrails: validate every output with a strict JSON Schema validator such as Pydantic or jsonschema before execution; implement constrained decoding using frameworks like Outlines, Guidance, or jsonformer to force token-level adherence to the grammar; and build a retry loop where validation errors are injected back into the prompt as user or system messages so the model can self-correct. Fourth, operational safety: add a max-retry limit, a degraded fallback such as returning a text explanation, and structured logging to track failure modes.

COMMON WRONG ANSWERS: Saying you will just prompt the model to be more careful or to check its work without any automated validation. Proposing regex extraction from markdown code blocks as the primary reliability mechanism. Ignoring constrained decoding and relying solely on temperature zero. Suggesting that fine-tuning on a handful of examples is the first or only fix, which is slow and still offers no hard guarantees. Failing to mention retry loops with error feedback, which is one of the highest-ROI system fixes.

LIKELY FOLLOW-UPS: How would you handle a schema that is too large to fit in the context window? What is the latency cost of constrained decoding, and how would you mitigate it? How do you prevent prompt injection when validation errors are passed back into the context? When is fine-tuning actually preferable over prompt engineering plus guardrails?

ONE CONCRETE EXAMPLE: For create_chart, the system prompt would include the JSON Schema and two few-shot examples showing a valid bar chart call and an invalid call missing the required type field with a note explaining the error. The model is instructed to write a short plan inside thinking tags before emitting raw JSON. The backend receives the output, parses it with Pydantic, and if the options object lacks a required legend boolean, the system returns a 400-style message to the agent loop with the exact validation error. The agent retries up to three times, after which it falls back to asking the user for clarification. If latency allows, constrained decoding via Outlines is enabled on the deployment to guarantee that every emitted token conforms to the schema grammar.