Offline Constrained RLHF with Multiple Preference Oracles
arXiv:2604.00200v1 Announce Type: new Abstract: We study offline constrained reinforcement learning from human feedback with multiple preference oracles. Motivated by applications that trade off performance with safety or fairness, we aim to maximize target population utility subject to a minimum protected group welfare constraint. From pairwise comparisons collected under a reference policy, we estimate oracle-specific rewards via maximum likelihood and analyze how statistical uncertainty propagates through the dual program. We cast the constrained objective as a KL-regularized Lagrangian whose primal optimizer is a Gibbs policy, reducing learning to a convex dual problem. We propose a dual-only algorithm that ensures high-probability constraint satisfaction and provide the first finite-s
View PDF HTML (experimental)
Abstract:We study offline constrained reinforcement learning from human feedback with multiple preference oracles. Motivated by applications that trade off performance with safety or fairness, we aim to maximize target population utility subject to a minimum protected group welfare constraint. From pairwise comparisons collected under a reference policy, we estimate oracle-specific rewards via maximum likelihood and analyze how statistical uncertainty propagates through the dual program. We cast the constrained objective as a KL-regularized Lagrangian whose primal optimizer is a Gibbs policy, reducing learning to a convex dual problem. We propose a dual-only algorithm that ensures high-probability constraint satisfaction and provide the first finite-sample performance guarantees for offline constrained preference learning. Finally, we extend our theoretical analysis to accommodate multiple constraints and general f-divergence regularization.
Subjects:
Machine Learning (cs.LG)
Cite as: arXiv:2604.00200 [cs.LG]
(or arXiv:2604.00200v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2604.00200
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Brenden Latham [view email] [v1] Tue, 31 Mar 2026 20:06:34 UTC (5,215 KB)
Sign in to highlight and annotate this article
Conversation starters
Daily AI Digest
Get the top 5 AI stories delivered to your inbox every morning.
More about
announceapplicationanalysis
Inter-Speaker Relative Cues for Two-Stage Text-Guided Target Speech Extraction
arXiv:2603.01316v2 Announce Type: replace Abstract: This paper investigates the use of relative cues for text-based target speech extraction (TSE). We first provide a theoretical justification for relative cues from the perspectives of human perception and label quantization, showing that relative cues preserve fine-grained distinctions that are often lost in absolute categorical representations for continuous-valued attributes. Building on this analysis, we propose a two-stage TSE framework in which a speech separation model first generates candidate sources, followed by a text-guided classifier that selects the target speaker based on embedding similarity. Within this framework, we train two separate classification models to evaluate the advantages of relative cues over independent cues
Empirical and Statistical Characterisation of 28 GHz mmWave Propagation in Office Environments
arXiv:2604.01814v1 Announce Type: new Abstract: Millimeter wave (mmWave) technology at 28 GHz is vital for beyond-5G systems, but indoor deployment remains challenging due to limited statistical evidence on propagation. This study investigates path loss, material penetration, and coverage enhancement using TMYTEK-based measurements. Statistical tests and confidence interval analysis show that path loss aligns with free-space theory, with an exponent of n = 2.07 plus or minus 0.073 (p = 0.385), confirming the suitability of classical models. Material analysis reveals significant variation: desk dividers introduce 3.4 dB more attenuation than display boards (95 percent CI: 1.81 to 4.98 dB, p less than 0.01), contradicting thickness-based assumptions. Reflector optimisation yields a significa
Mitigating Implicit Inconsistencies in Patch Porting
arXiv:2604.01680v1 Announce Type: new Abstract: Promptly porting patches from a source codebase to its variants (e.g., forks and branches) is essential for mitigating propagated defects and vulnerabilities. Recent studies have explored automated patch porting to reduce manual effort and delay, but existing approaches mainly handle inconsistencies visible in a patch's local context and struggle with those requiring global mapping knowledge between codebases. We refer to such non-local inconsistencies as implicit inconsistencies. Implicit inconsistencies pose greater challenges for developers to resolve due to their non-local nature. To address them, we propose MIP, which enables collaboration among an LLM, a compiler, and code analysis utilities. MIP adopts different strategies for differen
Knowledge Map
Connected Articles — Knowledge Graph
This article is connected to other articles through shared AI topics and tags.
More in Research Papers
MIMO Capacity Enhancement by Grating Walls: A Physics-Based Proof of Principle
arXiv:2604.01786v1 Announce Type: new Abstract: This paper investigates the passive enhancement of MIMO spectral efficiency through boundary engineering in a simplified two dimensional indoor proof of principle model. The propagation channel is constructed from the electromagnetic Green's function of a room with boundaries modeled as free space, drywall, perfect electric conductor (PEC), or binary gratings. Within this framework, grating coated walls enrich the non line of sight (NLoS) multipath field, reduce channel correlation, and enhance spatial multiplexing over a broad range of receiver locations. Comparisons with the drywall and PEC reference cases further reveal that the observed capacity enhancement arises not from diffraction alone, but from the combined effects of effective wall
1-bit Quantized Continuous Aperture Arrays
arXiv:2604.01780v1 Announce Type: new Abstract: Continuous aperture arrays (CAPAs) have emerged as a promising physical-layer paradigm for sixth generation (6G) systems, offering spatial degrees of freedom beyond those of conventional discrete antenna arrays. This paper investigates the interaction between the CAPA receive architecture and low-cost 1-bit analog-to-digital converters (ADCs), which impose a severe nonlinear distortion penalty in conventional discrete systems. For Rayleigh fading, we derive a moment matching approximation (MMA)-based closed-form symbol error probability (SEP) approximation based on Gamma moment-matching of the spatial eigenvalue distribution, and show that CAPAs incur a diversity-order penalty governed by Jensen's inequality on the mode eigenvalues. For line-
Channel Measurements and Modeling based on Composite Environmental Factor for Urban Street-Canyon Intersections
arXiv:2604.01767v1 Announce Type: new Abstract: In urban environments, vehicle-to-everything (V2X) communications require accurate wireless channel characterization. This requirement is particularly critical at street-canyon intersections, where building blockage and rich multipath propagation can severely degrade link reliability. Due to its unique environmental layout, the channel characteristics in urban canyon are influenced by building distribution. However, this feature has not been well captured in existing channel models. In this paper, we propose an environment-related statistical channel model based on 5.8~GHz channel measurements. We construct a composite environmental factor to characterize environmental differences in intersections. Then, the factor is incorporated into 3GPP p
Visual Decoding Operators: Towards a Compositional Theory of Visualization Perception
arXiv:2604.02220v1 Announce Type: new Abstract: Prior work on perceptual effectiveness has decomposed visualizations into smaller common units (e.g., channels such as angle, position, and length) to establish rankings. While useful, these decompositions lack the computational structure to predict performance for new visualization $\times$ task combinations, requiring new experiments for each. We propose an alternative unit of analysis: operationalizing quantitative visualization interpretation as sequences of composable visual decoding operators. Using probability density function (PDF) and cumulative distribution function (CDF) charts, we examine how chart-specific tasks can be decomposed into reusable, chart-agnostic perceptual operations and characterize their error profiles through hie
Discussion
Sign in to join the discussion
No comments yet — be the first to share your thoughts!