Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLP
arXiv:2603.28351v1 Announce Type: new Abstract: Subjective judgments are part of several NLP datasets and recent work is increasingly prioritizing models whose outputs reflect this diversity of perspectives. Such responses allow us to shed light on minority voices, which are frequently marginalized or obscured by dominant perspectives. It remains a question whether our evaluation practices align with these models' objectives. This position paper proposes seven evaluation desiderata for subjectivity-sensitive models, rooted in how subjectivity is represented in NLP data and models. The desidera — Urja Khurana, Michiel van der Meer, Enrico Liscio, Antske Fokkens, Pradeep K. Murukannaiah
View PDF
Abstract:Subjective judgments are part of several NLP datasets and recent work is increasingly prioritizing models whose outputs reflect this diversity of perspectives. Such responses allow us to shed light on minority voices, which are frequently marginalized or obscured by dominant perspectives. It remains a question whether our evaluation practices align with these models' objectives. This position paper proposes seven evaluation desiderata for subjectivity-sensitive models, rooted in how subjectivity is represented in NLP data and models. The desiderata are constructed in a top-down approach, keeping in mind the user-centric impact of such models. We scan the experimental setup of 60 papers and show that various aspects of subjectivity are still understudied: the distinction between ambiguous and polyphonic input, whether subjectivity is effectively expressed to the user, and a lack of interplay between different desiderata, amongst other gaps.
Comments: Under review
Subjects:
Computation and Language (cs.CL)
Cite as: arXiv:2603.28351 [cs.CL]
(or arXiv:2603.28351v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2603.28351
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Urja Khurana [view email] [v1] Mon, 30 Mar 2026 12:21:32 UTC (9,406 KB)
Sign in to highlight and annotate this article
Conversation starters
Daily AI Digest
Get the top 5 AI stories delivered to your inbox every morning.
More about
researchpaperarxiv
Exclusive | Caltech Researchers Claim Radical Compression of High-Fidelity AI Models - WSJ
<a href="https://news.google.com/rss/articles/CBMiuANBVV95cUxPdnA0SVIwQjktYkI3TUdZQWVHTXBDRWl6akZZOEhiVHVSZm53dkVoNEpEV0ZDOU1IUXBOVGZpNEVwUlRpaW1vbkwzTi1tcDJQMlliRUViWlNLaTQ1ak5vckdkWVdZTTBlMzM3bkRZbmM5LW42dTNKRkRBbGdmNmpWaVhDQXpSbzlDYTl4VE1jV2pIWGxQOXoxaWZ6SFBDU21sUmJKT2tmMjRjb1k0anBkLTRHbjFtbno5emtQaVNWUm1iZWF0UGJwZE9HZ29LWVUyVjdhdzA2cTF1R2NUY3J6bkJlUVhzYjVWZUZCdHdfbXJyX3lwRlJ6ak42MlJ3dUxTMEVpRHNGSmNfNi1GSmFmdTlkQUdCZEZvWlBBUjVYNTEtc0Y0ZFpkMGFKbTFFS3ZicjFYcllCMHV3YkJnZ2IxZkRTX1JiRlUzQkhjZzVYWlRUdVNfZGhqRWRWRmxyZTJJeHZ2T2RWQXR5aFZnMHgtdThweE5FdHNKOVZmOF9zMVdmb1djOWZxbFBkQ05lTndNLWZ6dFVYWXVudDZncGx6RllwcVJjVFRjUUdmOV9zOE9LYUgxTlR1eA?oc=5" target="_blank">Exclusive | Caltech Researchers Claim Radical Compression of High-Fidelity AI Models</a> <font color="#6f6f6f">WSJ</font>
Exclusive | Caltech Researchers Claim Radical Compression of High-Fidelity AI Models - WSJ
<a href="https://news.google.com/rss/articles/CBMiuANBVV95cUxOQUF5ZjI0bHRwSEdoaXVWTnd6REQzYjByRUxfMndKT0RzS2RPaUktZ3BEVVVyNWczYzNUbjVxSzd5WGtwLTVuRnZlT3VZb0YyMk4zNFphZWNZbUh5WWhQY0ZSVWFFTTNXZXNXRTVibUpiRHBlaHhIeVNlQjhSZ29YZ1RVclkzS2p5cVhaWTFNSW5lU2o5VUNuUWwtNE1ObWEzT2RmRHZheE8zLW5HLU5rY0loeVdEM1dYRk02YlBLajdkZU5ZcEliR1ZzNWFvdFEwTEs1WEtVQS1aVUpBMmRncWJLS0ZKaGlSbTdQVmxfeXpIX3I2MGlJTDNuZE1OdVpPUWpzWXlfQkdUeHhGMnF2Y2FhakNDVjBYTFRqTXNJTjZXZ2JUSXZudjVremdZUDBMS24xN3lySEJTWmxOMWtWdUZhb1VHeUlQVFhnWnJtOFpGU052VVNiSENXNkdNbjdaVmZzbkI2MnpDMGNSZ2FzUVJ0WWFEWElEeDlCYzZZcHk5T3B6NGtYMmw5bTU5d3RVRGdmMnMxcE56T2o3cGhjeDBGTzJhUHVqdnp5ZVZ2MlZuQ3BCaDJXVg?oc=5" target="_blank">Exclusive | Caltech Researchers Claim Radical Compression of High-Fidelity AI Models</a> <font color="#6f6f6f">WSJ</font>
Exclusive | Caltech Researchers Claim Radical Compression of High-Fidelity AI Models - WSJ
<a href="https://news.google.com/rss/articles/CBMiuANBVV95cUxQelRjVGJCMHI0NlB4NElDeGdDcFBoSlB1T0hSd2xlN2N6ME9SZFlSZWVmTXc0TG1uMHl2c2JBeldYdnUyVWIxQjFjRktwaFd3Y0UwVHQ3SC1Gdjc4eWtRT2VzR1luZndPZjVGeHdTQkFjZXJpSE5qcURnR0xvOF94QUhzVkpkaVd0QnJvZ3pxOC1idDlPWTA2Rnhydy1CNjRzdVMxbVlrbmJ6ZllERFNibktZMDZfRlM1WXRXQjBCdE42OHdMZTNNb09saDZhUXoxOVhCVE1TNGU0bURhVFI3ZFREQ3JRVEh6T09rZzhFTy1pZ1c3U01MRG1oVFJwS2lSUzRhY1JYMUtmdnZyN3hFT29DNWU2UU9EZkdNUTM1M2ZxR3JVNjRHM0ZyTjh1YTlaNGN0UTdmWUFhSHVPZ2RzS00ybjBhLXBkbmNuRldSR2otbVhNYURHNnE0MmplVUFKdzRoQS14MFEzc0ZSNUI4bkhURmlsbWludTVtT280bW1lX19TTFpFdEM5TDJnalk4bVp2RExTbkJmY09ITDBSRktCOTlyT2JidjNvTHlrYzJaNWlLaHdrLQ?oc=5" target="_blank">Exclusive | Caltech Researchers Claim Radical Compression of High-Fidelity AI Models</a> <font color="#6f6f6f">WSJ</font>
Knowledge Map
Connected Articles — Knowledge Graph
This article is connected to other articles through shared AI topics and tags.
More in Research Papers
UK police force presses pause on live facial recognition after study finds racial bias
<h4>Cams statistically more likely to ID Black people, says new research</h4> <p>A UK police force has suspended its deployment of live facial recognition (LFR) technology after a study revealed it was statistically more likely to identify Black people on a watchlist database.…</p>
Caltech breakthrough makes quantum memory last 30 times longer
While superconducting qubits are great at fast calculations, they struggle to store information for long periods. A team at Caltech has now developed a clever solution: converting quantum information into sound waves. By using a tiny device that acts like a miniature tuning fork, the researchers were able to extend quantum memory lifetimes up to 30 times longer than before. This breakthrough could pave the way toward practical, scalable quantum computers that can both compute and remember.
Too much screen time may be hurting kids’ hearts
More screen time among children and teens is linked to higher risks of heart and metabolic problems, particularly when combined with insufficient sleep. Danish researchers discovered a measurable rise in cardiometabolic risk scores and a metabolic “fingerprint” in frequent screen users. Experts say better sleep and balanced daily routines can help offset these effects and safeguard lifelong health.
Unbreakable? Researchers warn quantum computers have serious security flaws
Quantum computers could revolutionize everything from drug discovery to business analytics—but their incredible power also makes them surprisingly vulnerable. New research from Penn State warns that today’s quantum machines are not just futuristic tools, but potential gold mines for hackers. The study reveals that weaknesses can exist not only in software, but deep within the physical hardware itself, where valuable algorithms and sensitive data may be exposed.
Discussion
Sign in to join the discussion
No comments yet — be the first to share your thoughts!