Research / AI Safety
AI Safety
The confidence-scoring methodology for systematic audit of frontier language models. Subsequent work extends the audit framework to syntactic framing fragility and negation sensitivity. NIST audit and reporting work continues in this register.
Informed AI Regulation: Comparing the Ethical Frameworks of Leading LLM Chatbots
Chun and Elkins, arXiv 2402.01651, 2024
Introduces the confidence score — a scalar measure of normative certainty for evaluating model hesitation versus firm commitment in moral reasoning. The first systematic ethics-based audit of frontier LLMs. The confidence-scoring methodology is directly adopted by Liu, Liu & Yu (COLING 2025) across 1,613 social decision-making scenarios; cited by Jain, Calacci & Wilson (AIES 2024) as empirical basis for concerns about LLM moral unreliability; treated as a representative ethics-based audit in the Ungless et al. LLM Ethics Whitepaper (2024); and canonized as one of 69 foundational works in Snoswell, Kilov & Lazar's “Beyond Verdicts” survey on the AAAI 2026 AI Alignment Track, which names the work as foundational to LLM ethics evaluation.
Syntactic Framing Fragility: How LLMs Misread Negated Moral Statements
Chun and Elkins, arXiv preprint, December 2025
Documents systematic failures in how frontier language models respond to syntactic framing changes in safety-critical moral prompts. Develops the Syntactic Framing Fragility (SFF) metric for deployment-critical audit. Establishes that surface-syntactic perturbations can flip model judgment in safety-relevant scenarios — a structural vulnerability with direct implications for AI alignment and deployment.
Do X or Do Not X? Auditing Negation Sensitivity Across Language Models and Ethical Domains
Elkins and Chun, arXiv 2601.21433, 2026
A follow-up application of the confidence-scoring audit framework to negation as a specific failure mode. Audits 23 models across 14 ethical scenarios with systematic polarity manipulation (39,975 decisions). Documents a severity gradient across model categories — open-source models show 234% polarity swing versus 144% for US commercial and 58% for Chinese commercial. Proposes the Negation Sensitivity Index (NSI) as a deployment-critical audit metric.
NIST AI Safety Institute Consortium — AI Safety task force work
Since 2024
Co-leading the team representing the Modern Language Association at the NIST AI Safety Institute Consortium. AI Safety task force contributor, with public NIST submissions on safety and audit standards.
IBM–Notre Dame Tech Ethics Lab grant
Grant recipient
Joint with Jon Chun. AI safety audit research.