Why companion quality needs more than one score

A short empathetic reply can sound caring while missing the user's actual request. A roleplay model can maintain an engaging character but behave differently in conflict, ambiguity, or a request for practical help. A single average score can conceal those differences, especially when the test covers only a narrow set of prompts.

This is why recent companion research separates capabilities and scenarios. The goal is not to certify a product as a perfect friend; it is to make the evaluation question more precise and reveal which behaviors a test actually measures.

What two 2026 benchmarks add

CompanionBench describes an interactive bilingual benchmark grounded in de-identified real-world data. Its authors evaluate 28 agents across a ten-capability rubric and a measure related to whether deeper disclosure was earned. They report that emotional regulation and calibrated challenge remain difficult in their test setup, and that surface warmth can substitute for more substantive support.

ECBench approaches the problem through four kinds of interaction—emotional support, collaboration, conflict resolution, and social guidance—across friendship and romantic scenarios. These are research frameworks with explicit prompts and scoring methods, not independent audits of every commercial companion or clinical assessments of users.

Sources

How to read a leaderboard without overreading it

Check the model versions, prompts, language, judge setup, and date before comparing a score. A paper may evaluate base language models rather than complete companion apps, whose memory, policies, prompts, and interface affect the user's experience. Results from a controlled benchmark show behavior on that benchmark; they do not establish how every person will experience the product.

Treat any provider or paper's headline result as a starting point. Look at what the authors made measurable, what they left out, and whether the evaluation code or scenarios are available. A number can help identify a testable question without settling the broader question of which service is right for you.

Sources

A small test you can run for your own use case

Choose three ordinary scenarios: ask for a thoughtful response, disagree with the character, and correct a detail it previously got wrong. Keep the prompts and model date the same. Record whether the response addresses the request, respects the character boundary, accepts correction, and avoids inventing certainty. This is a personal checklist, not a published benchmark or an objective rating of a product.

For longer roleplay, add a separate continuity test; memory quality and relational conversation behavior are related but distinct. Our memory guide and the Jess product overview provide useful next steps without claiming that a research score determines your experience.

Sources and further reading

Research and product details were reviewed September 30, 2026 where applicable. Vendor policies, plans, and features can change; verify current details with the linked first-party source.