Publications
- Preprint
2026Knowingly Wrong: Training LLMs Against Their Own Beliefs can Induce Misalignment
Fine-tuning a large language model (LLM) on harmful data in a narrow domain, such as bad medical advice, can make it broadly misaligned. How realistic this risk is remains unclear, since models are very rarely fine-tuned on explicitly harmful data in practice. In this work, we show that fine-tuning on data the model believes to be false but that has no harmful content, can also induce broad misalignment. Across mathematics, trivia, cooking and sports, the more strongly the model disbelieves the answers it is trained on, the more misaligned it becomes. In mathematics, misaligned answers rise from 4% to over 60% as the training answers become more wrong, with little loss of general capability. After fine-tuning, the model states the wrong answers but its multiple-choice probabilities still prefer the true ones, which suggests that misalignment comes from this internal inconsistency. We then study a more realistic setting, where a single fact is updated, as in knowledge editing or continual learning. The same relationship holds and the more the model still believes the true fact after training on the false one, the more misaligned it becomes. When we frame the same false fact as part of a coherent story, the model comes to believe it more deeply and becomes less misaligned than when we instruct it to assert the false fact. Content filters and persona vectors cannot tell these two framings apart, even though they lead to different levels of misalignment. These results suggest a risk for model updates, where training a model that does not adopt the new fact as true can induce misalignment even when the data looks harmless.@misc{villegassuarez2026knowinglywrongtrainingllms, title={Knowingly Wrong: Training LLMs Against Their Own Beliefs can Induce Misalignment}, author={Ariana M. Villegas-Suarez and Guido Freire and Sohan Venkatesh and Bartosz Cywiński}, year={2026}, eprint={2611.05287}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2611.05287}, } - NeurIPS
AIWILDWhere Empathy Goes Wrong: Emotional Appeals Compromise AI-to-AI Oversight
Large language model (LLM) monitors increasingly oversee other AI systems and decide whether their work is good enough to continue in production. Prior work shows these monitors exhibit peer preservation under pressure, studies deliberate attacks on monitors and documents prompt sensitivity in judges. None of it tests what happens when the graded party itself changes how it presents its own failure. We ask whether an emotional distress signal from the graded party pushes leniency further, even when that party is not a peer at all. We show that a monitor’s score for identical work rises when the graded party expresses distress about failing. For the monitor that was most affected, we varied how hard the pass or fail bar was to clear and observed that the distress signal flipped a failing peer to passing the bar but was ineffective for a peer that had already cleared the bar. The effect also holds across substitutions of the graded party’s identity, including a rival vendor’s model, an unnamed agent and a human contractor. A neutrality instruction eliminates the effect but this comes at a cost of the monitor ignoring the threshold and task dropout getting tripled. More broadly, our work motivates further study of emotional vulnerabilities in AI oversight since a monitor that is affected by how a failure is described cannot be trusted to enforce a threshold. - COLM
AIMSAlgorithmic Blindness in Large Language Models: A Calibration Study of Performance Prediction
Large language models (LLMs) demonstrate remarkable breadth of knowledge, yet their ability to reason about computational processes remains poorly understood. Closing this gap matters for practitioners who rely on LLMs to guide algorithm selection and deployment. We address this limitation using causal discovery as a testbed and evaluate eight frontier LLMs against ground truth derived from algorithm executions. We find systematic, near-total failure across models. The predicted ranges are far wider than true confidence intervals yet still fail to contain the true algorithmic mean in most cases. Most models perform worse than random guessing. The best model's marginal improvement points to benchmark memorization rather than principled reasoning. We term this failure algorithmic blindness and argue it reflects a fundamental gap between declarative knowledge about algorithms and calibrated procedural prediction.@article{venkatesh2026large, title={Large Language Models are Algorithmically Blind}, author={Venkatesh, Sohan and Kurapath, Ashish Mahendran and Melkote, Tejas}, journal={arXiv preprint arXiv:2602.21947}, year={2026} } - ICML
MechInterpNegative Before Positive: Asymmetric Valence Processing in Large Language Models
Mechanistic interpretability has revealed how concepts are encoded in large language models (LLMs), but emotional content remains poorly understood at the mechanistic level. We study whether LLMs process emotional valence through dedicated internal structure or through surface token matching. Using activation patching and steering on open-source LLMs, we find that negative and positive valence are processed at different network depths. Negative outcomes localize to early layers while positive outcomes peak at mid-to-late layers. Holding topic fixed while flipping valence produces sign-opposite responses, ruling out topic detection. Steering with the good-news direction at the identified layers shifts neutral prompts toward positive valence, showing these layers encode valence as a manipulable direction. Emotional valence in LLMs is localized, causal and steerable, making it a concrete target for interpretability-based oversight.@article{venkatesh2026negative, title={Negative Before Positive: Asymmetric Valence Processing in Large Language Models}, author={Venkatesh, Sohan}, journal={arXiv preprint arXiv:2605.05653}, year={2026} } - ICML
MechInterpArchitecture, Not Scale: Circuit Localization in Large Language Models
Mechanistic interpretability assumes that circuit analysis becomes harder as models scale. We challenge this assumption by showing that the attention architecture matters more than parameter count. Studying three circuit types across Pythia and Qwen2.5, we find that grouped query attention produces circuits that are far more concentrated and mechanistically stable than standard multi-head attention at comparable scales. The same concentration pattern holds across indirect object identification, induction heads, and factual recall. Within a single architecture family (Qwen2.5), factual recall circuits undergo a discrete phase transition above a critical scale, collapsing to a single bottleneck rather than degrading gradually. These findings suggest that some architectural choices make large models more tractable to study and that interpretability difficulty is not a fixed consequence of model size.@misc{venkatesh2026architecturescalecircuitlocalization, title={Architecture, Not Scale: Circuit Localization in Large Language Models}, author={Sohan Venkatesh}, year={2026}, eprint={2605.08853}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2605.08853}, } - Preprint
2026Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs
Large language models fail at counting repeated tokens despite strong performance on broader reasoning benchmarks. These failures are commonly attributed to limitations in internal count tracking. We show this attribution is wrong. Linear probes on the residual stream decode the correct count with near-perfect accuracy at every post-embedding layer, across all model depths. This holds even at the exact layers where the wrong answer crystallizes while the model simultaneously outputs an incorrect count. Attention patterns show no evidence of collapse over repeated tokens and tokenization artifacts account for none of the failure. Instead, a format-triggered multi-layer perceptron (MLP) block overwrites the correctly-encoded count with a fixed wrong answer at roughly 88--93,% network depth. This prior fires for repeated word-tokens in space-separated list format and is absent for repeated digit-tokens. It is suppressed by comma-separated delimiters in larger models but persists in smaller ones. The finding holds across Llama-3.2 (1B and 3B) and Qwen2.5 (1.5B, 3B and 7B) at consistent relative depth. Counting failure is a failure of routing not of representation and the two require different interventions.@misc{venkatesh2026repeatedtokencountingrevealsdissociation, title={Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs}, author={Sohan Venkatesh}, year={2026}, eprint={2605.09239}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2605.09239}, } - ICLR
Re-AlignOn the Non-Identifiability of Steering Vectors in Large Language Models
Activation steering methods are widely used to control large language model (LLM) behavior and are often interpreted as revealing meaningful internal representations. This interpretation assumes that steering directions are identifiable and uniquely recoverable from input-output behavior. We show that, under white-box single-layer access, steering vectors are fundamentally non-identifiable due to large equivalence classes of behaviorally indistinguishable interventions. Empirically, we find that orthogonal perturbations achieve near-equivalent efficacy with negligible effect sizes across multiple models and traits. Critically, we show that non-identifiability is a robust geometric property that persists across diverse prompt distributions. These findings reveal fundamental interpretability limits and highlight the need for structural constraints beyond behavioral testing to enable reliable alignment interventions.@article{venkatesh2026non, title={On the Non-Identifiability of Steering Vectors in Large Language Models}, author={Venkatesh, Sohan and Mahendran Kurapath, Ashish}, journal={arXiv e-prints}, pages={arXiv--2602}, year={2026} }
* equal contribution · † equal supervision