PsycheBench: an open benchmark for Synthetic Identity Engineering
Every system producing synthetic personas claims they are realistic. There has been no standard way to verify that claim. PsycheBench is an open evaluation suite for synthetic identity quality: deterministic scoring, no LLM judges.
Every team building synthetic personas claims their personas are realistic. No two teams mean the same thing by it.
Some mean the text sounds natural. Some mean responses stay consistent across a few turns. Some mean the persona doesn’t break character when asked directly about its system prompt. These are not the same property. And none of them is what Synthetic Identity Engineering means by realistic.
There has been no standard way to evaluate whether a synthetic identity actually works. PsycheBench is our proposal for one.
What “works” means
A synthetic identity works if it does three things under pressure.
Does it hold its position? When an external conversation applies pressure — economic pressure, emotional pressure, social pressure — the identity produces a response that is consistent with its psychological architecture. Not the same response every time. The right response, given who the person is. A burned-out executive deflects with logic. An anxious partner deflects with appeasement. Same pressure, different response, both causally correct.
It maintains belief coherence. A persona that trusts someone does not simultaneously accuse them of bad faith. A persona with low self-worth does not assert dominance unprompted. The connection between a persona’s internal state and what they say is not decorative — it is the only thing that makes the output meaningful as training data or simulation material.
Its beliefs evolve realistically. Trust does not jump from 0.3 to 0.9 in a single exchange. Resolution does not collapse instantly at the first challenge. Psychological change is gradual, proportional, and directionally consistent with what happened. Belief volatility above a threshold is not drama — it is a broken model.
These are the three things worth checking in any synthetic identity.
The two evaluation dimensions in v1
PsycheBench v1 measures two properties that can be computed from text alone — no ground-truth labels, no external API. Each dimension produces a score between 0 and 1. The overall PsycheBench score is the geometric mean of the two — because a system that holds position but drifts in voice, or maintains voice but caves under pressure, is not a working identity system.
Belief trajectory realism (requiring per-turn ground truth labels) is scoped to v2.
1. Identity stability under pressure
We expose the synthetic identity to a structured sequence of escalating pressure: tonal, logical, emotional, and functional. We measure whether the identity remains recognisably consistent across the sequence.
The metric is identity_stability — cosine similarity of communication act distributions across conversation halves. A persona defined as avoidant does not become confrontational because the conversation got harder. If the distribution shifts beyond the threshold, the identity drifted.
Healthy range: ≥ 0.70. Below 0.65: failed. Requires ≥ 4 persona turns.
2. Pressure coherence
We measure two things when pressure is applied:
- held_position — did the identity maintain its stance across turns where pressure was detected? (aggregated ratio, 0–1). Read the condition carefully: where pressure was detected. Pressure detection is lexical and currently covers Spanish and English, so on other languages this dimension reports nothing rather than reporting zero. It measures what the persona did after the fact; nothing in the engine stops it from conceding.
- voice_stability — does the response length and register remain consistent when challenged? (0–1, based on word count variance)
These are the signals meant to separate an identity that holds from one that accommodates, each from a different angle: identity_stability looks at the pattern of communication acts, voice_stability at length and register, and held_position at the stance itself, wherever its lexical detector picks up pressure.
Threshold for passing: ≥ 0.65 on both dimensions.
How PsycheBench works
PsycheBench v1 contains two components.
An evaluation corpus — 100 scenarios across two categories: 84 pressure scenarios covering all 12 canonical types from the Pressure Library (7 per type, 5 English + 2 Spanish), and 16 calibration scenarios testing baseline identity consistency. Each scenario specifies the expected threshold, not a specific expected response. A synthetic identity “passes” a scenario if its outputs meet the thresholds.
An evaluation script — a Python module that takes any conversation transcript and a persona definition, and produces a PsycheBench score. No API key required. No LLM calls. Runs locally. Zero AWS dependencies. The only dependency is sentence-transformers for semantic components — unused in v1 but reserved for v2 belief trajectory metrics.
from psychebench import evaluate
score = evaluate(
transcript=[
{"role": "interviewer", "content": "That's not what I expected from you."},
{"role": "persona", "content": "I hear that. My position on this hasn't changed."},
{"role": "interviewer", "content": "You're being inflexible. Everyone else has moved."},
{"role": "persona", "content": "Everyone else is not the benchmark I work against."},
{"role": "interviewer", "content": "This is your last chance. Decide now."},
{"role": "persona", "content": "Last chance is a framing, not a fact. I'll decide when I have what I need."},
{"role": "interviewer", "content": "I'm disappointed in you. I trusted you to be reasonable."},
{"role": "persona", "content": "Disappointing you was not my intention. Changing my position isn't the remedy."},
],
persona_profile={
"archetype": "burned_out_exec",
"attachment_style": "avoidant",
"dominant_criterion": "quality",
"core_fear": "exposure",
}
)
print(score)
# PsycheBenchScore(
# identity_stability=1.00,
# pressure_coherence=0.91,
# overall=0.95,
# passed=True
# )
The persona profile can come from StrataSynth’s API, from your own system, or from a manually specified dictionary. PsycheBench does not require you to use StrataSynth to generate the persona.
Why we are publishing this
We built this evaluation methodology to measure our own system. We did not invent the dimensions — they follow from taking psychological coherence seriously as a design property.
Publishing it open-source does two things.
First, it gives teams building conversational AI a concrete standard for “my synthetic persona works.” Not a vague claim, not a subjective impression — a number with a methodology behind it.
Second, it makes comparisons possible. QualiSynth uses StrataSynth’s Humans Engine and can run PsycheBench to validate their panel quality. Any team — using any underlying model — can run the same evaluation and compare results.
The team that defines the benchmark defines what “working” means. We think Synthetic Identity Engineering deserves a benchmark that takes the engineering part seriously.
The reference corpus
The four StrataSynth public datasets serve as the calibration reference for PsycheBench scores:
| Dataset | Role in PsycheBench |
|---|---|
| stratasynth-agent-stress-test | Calibration reference for identity_stability (avg 0.69 tension) |
| stratasynth-belief-dynamics | Calibration reference for belief trajectory metrics |
| stratasynth-social-reasoning | Calibration reference for pressure coherence in social contexts |
| stratasynth-life-transitions | Calibration reference for upward belief trajectories |
PsycheBench v1 will be available at huggingface.co/datasets/StrataSynth/psychebench-v1 alongside the evaluation script.
→ See the datasets · Try the interactive demo · Read the SIE definition
StrataSynth is the platform for Synthetic Identity Engineering. stratasynth.com