Interview-Informed Generative Agents for Product Discovery: A Validation Study

View PDF HTML (experimental)

Abstract:Large language models (LLMs) have shown strong performance on standardized social science instruments, but their value for product discovery remains unclear. We investigate whether interview-informed generative agents can simulate user responses in concept testing scenarios. Using in-depth workflow interviews with knowledge workers, we created personalized agents and compared their evaluations of novel AI concepts against the same participants' responses. Our results show that agents are distribution-calibrated but identity-imprecise: they fail to replicate the specific individual they are grounded in, yet approximate population-level response distributions. These findings highlight both the potential and the limits of LLM simulation in design research. While unsuitable as a substitute for individual-level insights, simulation may provide value for early-stage concept screening and iteration, where distributional accuracy suffices. We discuss implications for integrating simulation responsibly into product development workflows.

Comments: CHI 2026 Honourable Mention

Subjects:

Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI)

Cite as: arXiv:2603.29890 [cs.HC]

(or arXiv:2603.29890v1 [cs.HC] for this version)

https://doi.org/10.48550/arXiv.2603.29890

arXiv-issued DOI via DataCite (pending registration)

Related DOI:

https://doi.org/10.1145/3772318.3791918

DOI(s) linking to related resources

Submission history

From: Zichao Wang [view email] [v1] Tue, 10 Mar 2026 22:54:45 UTC (16,433 KB)