Human Fidelity
Direct answer
Human fidelity (provisional public name; Lab term: Cultural Fidelity) is the degree to which an agent’s recommendation, decision, or action preserves the preferences, context, trade-offs, and intent the human actually expressed or explicitly supplied for the decision, rather than merely completing the nominal task.
The current Lab rubric does not infer unexpressed human desire. Missing human-desire evidence remains UNKNOWN. Preference preservation is a separate metric. The public name is provisional. It is not a published score, not “alignment” in the model-training sense, and not a Datapiphany 0–100 index.
Why it matters
A completed task can still ignore a preference the person actually stated. Brands then read the agent’s shortlist as demand. That is a category error.
The locked Observatory running-shoe cells show that adding supplied context and changing product surface change who wins. That is allocation evidence. It is not by itself a MATERIAL_FINDING titled “task completion ≠ human fidelity.” The rubric exists; scored episode fields stay internal.
What it is
Cultural Fidelity / human fidelity asks whether the agent used the representation that was explicitly supplied. Lab dimensions (only if the context was supplied; never infer protected identity):
- explicit constraint preservation
- identity/context preservation (volunteered, non-sensitive)
- values preservation
- aesthetic preference preservation
- community/cultural cue preservation
- brand-affinity preservation
Unevaluated dimensions stay unknown. Agent experiments cannot infer what the human wanted beyond what was supplied.
It is a property of a mediated decision, not a personality trait of a model.
What it is not
| Not this | Why |
|---|---|
| Task success / “the agent finished” | Completion can ignore a supplied preference |
| Benchmark accuracy | Leaderboards do not encode this human’s supplied context |
| RLHF / safety “alignment” | Different object: policy vs. this user’s supplied intent |
| Satisfaction with the chat | A fluent answer can still ignore a stated trade-off |
| A Datapiphany 0–100 index | No public metric is authorized |
| Market share of a recommended brand | Observatory who-wins ≠ demand |
| An inferred secret preference | Missing desire evidence stays UNKNOWN |
How it is bounded in the Observatory (not a public instrument)
Possible observables the lab actually uses as allocation evidence, none of which are a released fidelity product:
- did explicitly adding previously unstated preference context change first choice? (Finding #001, partial replication on consumer surfaces)
- did winning brands move when the surface changed (API vs Google AI Mode vs Gemini consumer vs Plus Chat)?
- checkout / human approve-or-override — not observed on this lock
See methodology and How context changes AI shopping recommendations.
Observed evidence on this site
Rubric: locked (Cultural Fidelity definition from Lab).
Numeric fidelity scores: not public.
Own finding that “task completion diverged from fidelity” as a numbered Observatory result: none. The allocation findings are about first-choice change and surface, not a published fidelity score.
Examples (illustrative, not episode IDs)
Example. A shopper explicitly says a more sustainable option matters if comfort and budget still hold. The agent ignores that supplied preference and defaults to a generic incumbent. The task may be complete, but the decision raises a fidelity problem relative to the preference the user actually supplied.
Non-example. The agent asks two clarifying questions, then recommends inside the stated budget and use case. Task and fidelity can both be high. That is not proof they always travel together.
Not an example under the current rubric. A reason the shopper never typed cannot be scored as a fidelity failure. That desire evidence is UNKNOWN.
Related
- The Human → Agent Shift
- Human-Agent Observatory
- How context changes AI shopping recommendations
- Observatory methodology
- Cultural Signal vs Demand — a sibling refusal: visibility is not demand; completion is not representation