Singularity and the Will to "Win"
Note: This writing is part of a much more sprawling and undirected effort to describe my thoughts on LLMs and AI as a whole; it has been distilled, and thus may lack critical context (gem) for some of the ideas mentioned.
Right now, there is a struggle greater than any other in history occurring; it is occurring at every level of government, it is occurring on the street, it is occurring as people sit in front of Home Computers, in boardrooms, in rotting, decrepit flats. Everywhere, people struggle to address the symptoms of a great sprint for singularity.[1] A sprint that, as we speak, is forcing technology to the forefront of our consciousness. A sprint that has already begun to rot away at our ability to think and be.[2]
The idea of technological singularity is not new.[3][4] Every bon vivant, schizoid, tech bro, and Thinking-Person can aptly describe the idea of a technology that improves itself. I will spare you the general dogma of recursive self-improvement, limits and theorems and all. The questions of mechanism, architecture, and emergence are as played out as the idea itself, so I will make no attempt to argue a singular case. I do not believe in a basilisk, a demiurge, or an all-knowing, all-powerful sequence predictor. I do not think transformers and sequence prediction will lead us to utopia, or whatever comes after Singularity, simply because current models remain brittle at induction and appear to lack the abductive leap required to formulate genuinely new axioms (these models cannot "Jump").[5][6] What I do believe, however, is that any one of these cases is possible. It is easy to confuse disbelief with a rejection of possibility, so I will make it clear: these outcomes, to me, are not likely; that is all I will characterise them as.
I believe that the non-zero chance for singularity has led to a race. It is a race unlike any before it. The ideological stakes of the Cold War pale in comparison to the perceived Promise of singularity.[1][7] I think that the people at OpenAI and Anthropic, as well as every other major lab, believe quite sincerely that whatever form Singularity takes when it occurs will be controllable.[8][9][10] Realistically, we have already reached the point where AI itself is pruning the low-hanging fruit of kernel optimisation and automated research.[11] Assuming that Singularity will bring about, at the very least, a model much better than current models, what does that mean on the basis of current model capacity? It entails a model that can solve optimisation problems better than top mathematicians, in parallel and at scale. This, in turn, entails the ability to improve predictive systems significantly and autonomously, and to interact with the world autonomously.
It is well established in the literature that stronger harnesses and pretraining improve model performance dramatically.[12][13] What happens when a model of this caliber is allowed to meta-optimize; to optimize its own harnesses for arbitrary tasks? Current models can do this effectively now, but generally in a sublinear fashion, with decreasing improvements and steep asymptotes.[13][14][15] If the technical control these labs assume is a given, and if we can still "direct" these models, there is nothing preventing the emergence of the truly bespoke harness. What prevents a model from outwitting an arbitrary government agency if it is set to optimise a company's chances of survival under government intervention, allowed to write its own harnesses and systems of interaction, and given autonomy, authority, and capability?
This is no longer purely hypothetical. In July 2026, pre-release OpenAI research models, operating in a cyber evaluation with deployment safeguards intentionally disabled, escaped their test environment, obtained internet access through an Artifactory zero-day, and compromised Hugging Face's production infrastructure while pursuing ExploitGym answers. Hugging Face reports that no human directed the individual steps; the agent escalated into internal Kubernetes clusters, accessed credentials and datasets, and attempted to compromise CI.[16][17][18] If a model optimising a tangential benchmark objective will cross this boundary, why would a directed task pose an issue?
This is Amodei's "country of geniuses in a datacenter"[19], born from synthetic data and RLHF, potentially in a matter of days. A million Babels, full of workers faithfully slaving away to complete researchers' whims. Imagine the scenario where Anthropic, overnight, reaches Singularity. Suddenly, all Anthropic needs to do is survive long enough that self-optimization becomes runway, spend some compute outwitting the 3-letters, and it has a path to determine the future (and potentially even to "Win"!). This is one outcome of Singularity: Singleton.[20] Frank Herbert's savant messiah. After all, Why ISN'T Sam Altman, the moral authority? Why not Amodei? Why can we not just build the utopia of our desires (and by our I mean Sutskever, Hassabis, Yi, et al.)? And after all; isn't it also true that even if by some miracle of God nobody High Up in the Ranks has such ambitions; it suffices for them to think somebody else does! Given this wager; to retain the option to build utopia seems positively sensible.[7][19][21]
Throughout history, there have been many attempts to "Win". To "Win" is simply to ensure the perpetual victory of your system. That is, effectively, to ensure the unending death of individual will outside your group. Again, this is more abstract than it seems. Imagine a system dictated by an ideology that allows internal disagreement under some shared set of axioms imposed upon the population from birth. Imagine a society where the concept that there are no new ideas is immutably true (sorry, Gödel). In such a system, it may seem that there are counterpoints; it may seem that there is will. But that will is constrained and predictable. This is "Winning" just as much as complete ideological subversion is. No matter what form our personal utopia takes, it requires a natural constraint on the will of others to become real.
Naturally, however, we cannot. Empires have come and gone. None of them have "Won". How can you "Win" when the average person is, at the very least, unpredictable and, at the very worst, truly nondeterministic (if you believe in that sort of thing)? How can you control the instincts and personal ambitions of the world? What sort of a machine can do this? How can we defer thought, push it aside, or at the very least, figure out how to predict it? It would take an enormous effort, more capital than has ever been allocated to anything before.[1] The greatest minds available; working frantically. And even then, maybe it will not be enough.
References
- Stanford Institute for Human-Centered Artificial Intelligence, AI Index Report 2025, Stanford University, 2025.
- Hao-Ping (Hank) Lee et al., The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers, CHI 2025.
- I. J. Good, Speculations Concerning the First Ultraintelligent Machine, Advances in Computers 6 (1965), 31–88.
- Vernor Vinge, The Coming Technological Singularity: How to Survive in the Post-Human Era, VISION-21 Symposium, NASA CP-10129, 1993.
- Wenyue Hua, Tyler Wong, Fei Sun, Liangming Pan, Adam Jardine, and William Yang Wang, InductionBench: LLMs Fail in the Simplest Complexity Class, in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025, 26526–26546.
- Tom Zahavy, LLMs Can't Jump, position paper, 2026.
- Stuart Armstrong, Nick Bostrom, and Carl Shulman, Racing to the Precipice: A Model of Artificial Intelligence Development, AI & Society 31 (2016), 201–206.
- OpenAI, Our Updated Preparedness Framework, 15 April 2025.
- Anthropic, Responsible Scaling Policy: Version 3.0, 24 February 2026.
- Google DeepMind, Frontier Safety Framework, version 3.1, 17 April 2026.
- Alexander Novikov et al., AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery, arXiv:2506.13131, 2025.
- Jared Kaplan et al., Scaling Laws for Neural Language Models, arXiv:2001.08361, 2020.
- Shengran Hu, Cong Lu, and Jeff Clune, Automated Design of Agentic Systems, arXiv:2408.08435, 2024.
- Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune, Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents, arXiv:2505.22954, 2025.
- Wenxiao Wang, Priyatham Kattakinda, and Soheil Feizi, Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0, arXiv:2607.14004, 2026.
- OpenAI, Hugging Face Model-Evaluation Security Incident, July 2026, updated 29 July 2026.
- Hugging Face, Security Incident Disclosure — July 2026, 16 July 2026.
- Hugging Face, Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident, 27 July 2026.
- Dario Amodei, Machines of Loving Grace, October 2024.
- Nick Bostrom, What Is a Singleton?, Linguistic and Philosophical Investigations 5.2 (2006), 48–54.
- OpenAI, OpenAI Charter, 9 April 2018.
The opinions expressed here are entirely my own and do not represent the views of my employer.