Models model language model announce prediction interpretability component

Empirical Sufficiency Lower Bounds for Language Modeling with Locally-Bootstrapped Semantic Structures

arXiv cs.AIby Jakob Prange, Emmanuele ChersoniApril 6, 20261 min read0 views

arXiv:2305.18915v1 Announce Type: cross Abstract: In this work we build upon negative results from an attempt at language modeling with predicted semantic structure, in order to establish empirical lower bounds on what could have made the attempt successful. More specifically, we design a concise binary vector representation of semantic structure at the lexical level and evaluate in-depth how good an incremental tagger needs to be in order to achieve better-than-baseline performance with an end-to-end semantic-bootstrapping language model. We envision such a system as consisting of a (pretrained) sequential-neural component and a hierarchical-symbolic component working together to generate text with low surprisal and high linguistic interpretability. We find that (a) dimensionality of the

View PDF

Abstract:In this work we build upon negative results from an attempt at language modeling with predicted semantic structure, in order to establish empirical lower bounds on what could have made the attempt successful. More specifically, we design a concise binary vector representation of semantic structure at the lexical level and evaluate in-depth how good an incremental tagger needs to be in order to achieve better-than-baseline performance with an end-to-end semantic-bootstrapping language model. We envision such a system as consisting of a (pretrained) sequential-neural component and a hierarchical-symbolic component working together to generate text with low surprisal and high linguistic interpretability. We find that (a) dimensionality of the semantic vector representation can be dramatically reduced without losing its main advantages and (b) lower bounds on prediction quality cannot be established via a single score alone, but need to take the distributions of signal and noise into account.

Comments: To appear at SEM 2023, Toronto

Subjects:

Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

Cite as: arXiv:2305.18915 [cs.CL]

(or arXiv:2305.18915v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2305.18915

arXiv-issued DOI via DataCite

Submission history

From: Jakob Prange [view email] [v1] Tue, 30 May 2023 10:09:48 UTC (546 KB)

Original source

arXiv cs.AI

https://arxiv.org/abs/2305.18915

Was this article helpful?

Ask AI about this article

Ready

Conversation starters

Ask anything about this article…

Daily AI Digest

Get the top 5 AI stories delivered to your inbox every morning.

More about

modellanguage modelannounce

ProductsFresh

Beware The Magical Two-Person, $1 Billion AI-Driven Startup

In early 2024, OpenAI CEO Sam Altman predicted there would be a “one-person billion-dollar company, which would’ve been unimaginable without AI, and now it will happen.” Several media outlets recently concluded that the prediction came true (albeit with two employees). But the story looks less promising upon deeper inspection. Retain Healthy Skepticism When Faced With [ ]