Superintelligence as a concept is nothing new, and neither are discussions around what hypothetical human (or nonhuman) futures look like when it becomes reality. People have known where our progress in building machines has been going, ever since Butler wrote Among the Machines all the way back in the 19th century. Butler predicted a future where machines would "evolve" similar to biological creatures, and end up becoming the dominant life form on planet earth, even above humans. It was always speculative writing, but current events have made this discussion much more widespread than ever before; so much so that having a well formed and defined opinion on the question is almost necessary across business, science, and many other domains.

So far, I have come across three large "camps" on where people stand on the impact of machine intelligence on our collective future.

• Camp 1: Tech leap. Transformer based models are already the newest 'internet' or 'personal computer' level technological increment in our lives. This is already true in the present, so this camp believes this is the plateau for the models.

• Camp 2: Revolution. AI will prove to be an entirely new way of humans conducting life, akin to the agricultural revolution or the industrial revolution in how total those previous periods were in altering the fabric of human life. Everything from societal organization, power structures, governance, militaries, and culture will be downstream of the revolution.

• Camp 3: Infinite takeoff. This moment in time is the defining moment in our species' history. There is nothing after and nothing before that are connected, there is simply "pre ai" and "post". This camp is defined by a binary choice: utopian takeoff or total human extinction.

Personally I stand between camps 1 and 2, but the main problem I have been wrestling with is slightly different. I think the fundamental question many humans, myself included, have been asking is - what role do we play in the future at all?

If a machine is capable of doing everything humans can do, at a level that perhaps surpasses our own, is that it? Are humans just model 0.1 that got replaced by model 1.0? If machine intelligence becomes recursive, what will model 1.0 do when model 2.0 and beyond are born?

"Will we become pigeons" is a funny question that comes to the crux of this issue. Pigeons used to be essential for communications in antiquity, and almost every civilization across Asia, Europe, and the Middle East used pigeons for fast communication across vast empires. Even after the invention of the telegraph, pigeons proved useful for communication to locations with no wires laid. However, with the invention of radio communication, pigeons became completely useless. Now every city in the world, from New York to New Delhi, is filled with useless pigeons who are viewed as flying pests (the loyal pigeon never forgot us, however, as it is now incapable of building nests as we bred them for our purposes, not survival in the wild).

And when you frame the question like that, if you look someone in the face and ask them, "Do you want to be a pigeon?" I think most people, would resolutely answer: NO!

I went into technology because I wanted to help build a future I believed in. Over lunch recently, a fellow intern posed the classic question of fate or free will. In the context of the current problems we are wrestling with however, this question is of the ultimate importance. Was our destiny always meant to be total deference to the higher species, the silicon based life form? Was our ultimate purpose to just be the starting point for consciousness, the cradle, while intelligence was never meant to stay in carbon form?

The answers to those questions is... I do not care. I do not want to be a pigeon. Human life is special, human life is precious, and human life is fleeting. Human life is beautiful. And most of all, humans are special. A language model will never experience the day to day qualia that a human experiences.

If this sounds airy fairy, a transformer model will never see the micro expressions or body language of a manager that tells you they are sycophants for the CEO. A transformer model will never understand what emotions are implicitly, or how most decisions are made from emotions over logic. A transformer model will never understand why the decision maker isn't willing to take risks on trying a new product, which is wanting a safe life for their children over making risky bets at work. In other words, as the current transformer architecture stands, it will never be human. (Surely one day conscious AI will be real, but transformers and LLMs are just smart autocomplete).

Thus it doesn't matter if human in the loop is less efficient technically. It doesnt matter if humans make incorrect judgements. It doesn't matter if an organization could technically grow faster with an all AI team over a human one. If you are selling to the human economy, you need humans. People will not trust or buy things from something that spawned into a server in Virginia 2 seconds ago. More than sales, the most common words repeated internally at Convictional are "taste and judgement". For building human first products and services, for engaging with the human economy, and for managing other humans with empathy and care, you need other humans.

On a technical aside, epistemic diversity and knowledge collapse are also a huge reason why AI run companies are doomed from the start. By both the nature of the training data of the model, along with the model architecture itself, you end up with this entity that parrots the most safe, sanitized, and bureaucratically correct answer for any given question. Basically, imagine this scene from family guy, but at scale across thousands of these things all agreeing with each other on a ridiculous idea. I'll caveat that with careful supervision, synthetic data boot strapping can push models past that 'average' response and into the niche, higher quality corners of their training data. Whether that holds up long term without enormous compute, or whether knowledge collapse is genuinely avoided versus just delayed, is still very much an open question. Agent run startups have been tried, and some have even been successful. But none so far have integrated into the real world economy. And no matter how good the models get, I do not think they ever will.

Lastly, I believe in having conviction (no pun intended) in the future you want to build. Why does everyone think you need to charge headfirst into building a future nobody wants? Say it with me: I do not want to be a pigeon. I see a future of teams of agents working for humans, to implement human vision, human taste, and human judgement. This future is one of extreme productivity per human, but the "per human" in that metric does not go away. In this future, your organization's most important and coveted commodity is the opinions, tastes, and decisions you make every day, that compound onto each other and make up your vision.

For all of human history, our greatest strength comes from working together through the shared stories we tell each other. Tools come and go, but that remains. There is no reason to bet that will change.