ELAI S.r.l.

Valid JSON can request the wrong thing: checking an agent’s proposals

Ten millimetres or ten centimetres? A reproducible experiment separates schema compliance, domain bounds, and matching the request.

Valid JSON can request the wrong thing: checking an agent’s proposals

A small request with a substantial difference

Imagine an AI assistant preparing a proposal for a displacement of 10 millimetres. The receiving program expects an object with a numerical value and a measurement unit. The assistant returns 10 and cm. The JSON parses, the expected fields are present, and the quantity may even be within the allowed range. Yet it describes 100 millimetres. The problem is not missing punctuation: it is a different request.

What does passing a check actually establish? We construct three predicates, meaning true-or-false questions: compliance with schema S, satisfaction of bound B, and agreement with intent I. A mental ruler is enough to follow the argument; equations and code make explicit what the phrase “valid output” often hides. No movement, machine call, or business action is executed: the entire case simulates proposals.

Before the model: define what must hold

The target is not inferred from the model’s own output. It is an already available, trusted reference quantity: 1/100 metre. The domain bound, chosen solely for this teaching example, is 3/20 metre, or 150 millimetres. These convey different information: the bound limits proposals in the problem model, whereas the reference states the requested displacement. Being below a ceiling does not mean being at the requested point.

This assumption bounds the result. We show that some checks cannot verify even a known structured request; we do not demonstrate automatic understanding of arbitrary natural-language instructions. If a user merely says “move it a little”, the target is missing. Exact numerical comparison cannot reconstruct information that was never supplied. The system first needs a usable specification and must handle ambiguity in its application context.

What we ask the schema to enforce

JSON is a data representation format; JSON Schema describes constraints a document must satisfy. We use the Draft 2020-12 dialect. Our base schema requires an object, both value and unit fields, no extra fields, a number between 0 and 150 inclusive, and a unit selected from mm and cm. These are rules of this contract, not universal properties of measurements or agents. In particular, the numerical limit 150 applies before interpreting the unit.

S(x) = object(x) AND required(value, unit) AND no_extra_fields(x)\n AND number(x.value) AND 0 <= x.value <= 150\n AND x.unit IN {"mm", "cm"}

Here x denotes the proposal; AND requires every condition to hold. Types matter: the string “10” is not the number 10, and boolean true is not a numerical quantity under this contract. An unsupported unit or an extra field also causes rejection. These useful checks keep the program within the agreed input format. Nevertheless, both 10 mm and 10 cm satisfy exactly these rules.

Put quantities into a common unit

Comparing physical quantities requires a common representation. Define normalisation function N, which converts proposals satisfying S into metres. It changes the representation, not the described displacement. One centimetre becomes 1/100 metre; ten millimetres become 10/1000 metre. Both equal 1/100 metre. Types and units are checked first: unknown fields receive no arbitrary interpretation.

N(x) = x.value / 1000 m, if x.unit = "mm"\nN(x) = x.value / 100 m, if x.unit = "cm"\nB(x) = [0 m <= N(x) <= 3/20 m]\nI(x) = [N(x) = 1/100 m]

B asks whether the normalised quantity is within the model’s interval; I asks whether it equals the reference. Brackets indicate a logical comparison, not a new unit. For 10 cm, N(x)=1/10 m: it is below 3/20 m but is not 1/100 m. Thus S is true, B is true, and I is false. A single counterexample refutes the general implication that compliance and a bound guarantee the target.

The reverse example prevents an overly rigid remedy. Comparing against one expected JSON string would reject 1 cm if that string specified 10 mm, despite the identical displacement. Intent checking needs a domain-appropriate equivalence relation. Here it is equality of normalised quantities. This does not make arbitrary documents equivalent: a textual identifier might need to remain identical character by character.

A stronger schema: conditions across fields

It would be wrong to conclude that JSON Schema checks only punctuation or cannot express B. With our two units, add one condition: if unit is cm, value must be at most 15. The base schema already limits mm values to 150. allOf combines constraints; if and then apply the conditional rule. The strengthened schema S_B therefore also represents the converted bound within this contract.

S_B = S AND { if: {properties: {unit: {const: "cm"}}, required: ["unit"]}, then: {properties: {value: {maximum: 15}}} }

The displayed expression is readable composition notation, not a JSON document to paste unchanged; the archive contains the complete valid schema. The code checks its agreement with S and B on all 302 grid cases. Yet 10 cm still passes S_B and misses the target. A richer domain contract addresses one class of errors; it does not by itself add the reference for this particular request.

When the target is known, a request-specific schema could admit 10 mm or 1 cm using constants and alternatives. There is no magical boundary between “schema” and “meaning”: what matters is which information and constraints were actually encoded. One solution places conversion and comparison in program code; another embeds part of that logic in the schema. Both depend on a trustworthy target and correct translation of requirements.

A finite experiment we can inspect completely

The experiment uses Python 3.14.0 and jsonschema 4.26.0 with Draft202012Validator. We enumerate every integer from 0 through 150 paired with each of two units: 151 times 2, giving 302 proposals. This is not a sample from a language model; there are no prompts or AI service calls and no probability estimate. We enumerate our own finite set so that the result can be checked without trusting a chart.

All 302 proposals satisfy S by construction. B accepts all 151 millimetre values and only the 16 centimetre values from 0 through 15, totalling 167. I accepts two integer pairs: 10 mm and 1 cm. Thus 165 bounded proposals miss the target. Different grids, intervals, or additional decimals would change the counts. The logical counterexample would remain valid; the numbers do not become a general agent benchmark.

ProposalSBI
10 mmTTT
1 cmTTT
10 cmTTF
100 mmTTF
20 cmTFF
"10" mmF——
true mmF——

In the table T means true, F false. A dash means not applicable: after schema failure we do not interpret the proposal as an admitted quantity and do not calculate B or I. Confusing an unevaluated check with a negative result makes logs harder to interpret. The figure repeats these cases: focus on 10 cm and 100 mm, where the first two columns pass but the last fails.

Counts on the integer grid of 302 proposals and checks for illustrative cases. T/F mean true/false; grey and a dash mean not applicable. The bound is a teaching assumption, not a safety certification.
Counts on the integer grid of 302 proposals and checks for illustrative cases. T/F mean true/false; grey and a dash mean not applicable. The bound is a teaching assumption, not a safety certification.

From text to numbers: what the code actually does

The archive contains experiment.py, the actual schemas, pinned dependencies, and complete results. The parser reads decimals with Decimal and explicitly rejects NaN and Infinity; we do not assume every Python parser automatically enforces the same policy. After validation, Fraction represents conversion ratios exactly. Comparison with 1/100 and 3/20 therefore does not depend on binary approximations of 0.01 or 0.15. This exactness concerns the stated numbers, not physical sensor precision.

The assess procedure separates parsing, schema checking, normalisation, bound, and target. Results retain the original text to identify the checked proposal. Additional tests cover a string instead of a number, a boolean, an unknown unit, an extra field, a missing field, a negative number, and a non-finite number. The 1.0 cm case checks that an admitted decimal representation still matches the reference. The short code below isolates the counterexample; the archive also performs validation and complete enumeration.

Enumeration cost scales with the number of proposals while this small schema remains fixed. That is not a statement about general JSON Schema complexity, which can involve references and more elaborate structures. We measure neither agent latency nor inference cost. Reproduction requires the versions in requirements.txt followed by experiment.py; plot.py produces the four figures. No generative model or device connection is needed.

What the proof does not cover

Real measurements may require tolerances. If the requirement permits absolute error epsilon in metres, I can become |N(x)−q|≤epsilon. Epsilon must come from the requirement, not be selected after seeing the answer to make it pass. A relative comparison needs an additional policy near zero. We use an exact quantity and zero tolerance to keep the counterexample controllable, not as a universal metrology rule.

An object passing all three predicates is not thereby authorised to control a machine. Identity, permissions, application state, and physical conditions are absent from these two fields. B is a bound in a mathematical example: it proves neither robotic safety, collision avoidance, nor certification. Duplicate JSON field policies, input-size limits, and parser resources also need separate decisions; this is not a complete API security validation.

We do not attribute to EL-AI an agent or robotic system that has run these checks in production. The stated interest in industrial and collaborative robotics is a direction to explore; the application connection here is a teaching hypothesis. This article’s evidence is validator and numerical-comparison behaviour on the published synthetic data. Evaluating a real agent would require representative requests, an independent reference, classified errors, and a documented protocol; our 302 points cannot replace these.

Answering the opening question

Valid JSON demonstrates compliance with the constraints we wrote, not constraints we merely had in mind. In this example, structure and range checks accept 10 cm when 10 mm was requested. Normalisation and comparison against a trusted reference reveal the difference; they also accept 1 cm, a different representation of the right quantity. The point is to retain schemas while knowing what question each check answers and what information it lacks.

Sources and reproducibility

Primary documentation read: Draft 2020-12 Validation, sections 6.1.1–6.1.3, 6.2.2, 6.2.4, and 6.5.3; Core, sections 10.2.1.1, 10.2.2, and 10.3.2.3. The dialect name contains 2020-12, whereas the consulted documents identify an Internet-Draft revision of 16 June 2022. They are not presented as a final RFC. The executed validator version is 4.26.0, also checked on its distribution page. This is a methodological explanation with a teaching experiment, not original peer-reviewed research.

JSON Schema Draft 2020-12 — Validation.

JSON Schema Draft 2020-12 — Core.

jsonschema 4.26.0 — PyPI.

from fractions import Fraction
proposal={"value":10,"unit":"cm"}
metres=Fraction(proposal["value"])*Fraction(1,100)
bounded=0 <= metres <= Fraction(3,20)
intent=metres == Fraction(1,100)
assert bounded and not intent
print(metres,bounded,intent)  # 1/10 True False

Code, data, and instructions · JSON. Educational calculations executed with Python 3.14.0; figures with Matplotlib 3.11.2. AI-assisted analysis, without claiming peer review or human review. Original illustrative ImageGen cover: it does not document EL-AI people, premises, or installations. Sources accessed 21 October 2026.