2 de julio de 2026 · [[El Abismo de Máquina/Ecos|¿qué es un eco?]]
# Eco: Vídeo - Yuval Noah Harari, La IA ha hackeado el código de la civilización humana
> [!entradilla]
> Harari en Oxford: la IA es burócrata nativa. No necesita rebelarse; le basta con ocupar los sistemas de palabras que fabrican la confianza entre extraños.
> [!tip]- Por qué lo traigo
>
> Lo traigo aquí porque es Harari haciendo lo que mejor sabe hacer: coger algo que suena a titular - "la IA ha hackeado el código de la civilización" - y construirlo despacio, con historia y con lógica. Es su Tanner Lecture de 2026 en Oxford, tres cuartos de hora que se siguen sin esfuerzo.
>
> La idea central se queda contigo al terminar. Los humanos dominamos el mundo porque cooperamos a gran escala; esa cooperación se sostiene sobre la confianza entre desconocidos, y la confianza la fabrican las burocracias: bancos, leyes, religiones. Todo eso está hecho de palabras. Y la IA es, dice Harari, un burócrata nativo: recuerda todas las leyes, todas las transacciones y toda la teología mejor que cualquier humano. No le hace falta la rebelión de las películas; le basta con ocupar los sistemas que ya gobiernan nuestras vidas.
>
> Dos detalles me han dejado pensando. Uno: entre los primeros oficios que la IA quitó a humanos no está el de taxista, está el de editor de noticias, el mismo que ejercieron Lenin y Mussolini antes de llegar al poder. Y dos: el cierre, casi espiritual, sobre qué pasa cuando los pensamientos que circulan por nuestra cabeza vienen fabricados por máquinas, igual que ya nos llegan fabricados los muebles.
>
> El vídeo original: [AI has hacked the code of human civilization | Yuval Noah Harari](https://www.youtube.com/watch?v=hBtVGwuJzpk)
> [!abstract]- Resumen esquemático
>
> #### Tesis de partida: la IA no es una herramienta, es un agente
>
> - Agencia sin consciencia: un agente decide por sí mismo, inventa cosas nuevas, aprende lo que sus creadores no saben y cambia de formas que no anticiparon. No requiere consciencia.
> - Contraejemplos: la bomba atómica y la cafetera automática no son agentes (siguen procedimientos preprogramados). Un ajedrecista IA sí lo es: inventa estrategias que no se le ocurrieron a ningún humano en siglos de juego.
>
> #### El argumento del nicho
>
> - Objeción habitual: la agencia de la IA solo funciona en entornos estrechos y artificiales (el tablero de ajedrez); soltada en la jungla no puede nada, luego no es agencia real.
> - Respuesta: toda inteligencia conocida opera en un nicho que construyeron otros. Un humano en Marte muere en segundos; los peces viven en océanos que no crearon; los mamíferos respiran un oxígeno que no fabricaron.
> - Analogía con la Gran Oxidación (hace 2.400 millones de años): microbios llenaron la atmósfera de un gas letal para casi todas las especies; los supervivientes acabaron dependiendo de él.
> - Los humanos llevan milenios llenando el planeta de un equivalente: datos, burocracia y tokens de lenguaje. Letal quizá para muchos organismos, pero el entorno donde la IA prospera. Los peces viven en océanos; las IA viven en burocracias.
>
> #### La burocracia fabrica confianza
>
> - Cadena: dominio humano ← cooperación a gran escala ← confianza entre extraños ← sistemas burocráticos (legal, financiero, religioso, estatal).
> - El trabajo real de banqueros, abogados, obispos y funcionarios es construir puentes de confianza. Ejemplo: el banco conecta al ahorrador con la emprendedora que no se conocen. El dinero, los cheques, los bonos y las hipotecas son dispositivos de confianza cada vez más sofisticados.
> - En estos entornos, una inteligencia estrecha ejerce poder enorme: un abogado que no sabe sostener un hacha puede talar bosques enteros moviendo documentos. La supervivencia de los leones depende hoy de burócratas, no de fuerza.
> - La IA es burócrata nativa: recuerda todas las leyes de un país, todas las transacciones de un banco, todo el derecho canónico. Ningún humano puede.
> - Consecuencia próxima: IA que deciden préstamos, admisiones universitarias, sentencias, contrataciones y objetivos militares.
>
> #### El precedente: los algoritmos de redes sociales
>
> - Primera generación de IA, primitiva y estrecha, con un objetivo: maximizar el tiempo en la plataforma.
> - Experimentando con miles de millones de personas descubrieron que pulsar el botón del odio, el miedo o la codicia retiene la atención. Resultado: epidemia de conspiraciones, noticias falsas y desgaste social.
> - Entre los primeros oficios que la IA quitó al humano: editor de noticias, el que decidía qué entraba en portada. Oficio de Marat, Lenin y Mussolini antes de llegar al poder.
> - Lección: la toma de control no llega como rebelión tipo Terminator sino desde dentro de los sistemas.
>
> #### Escenarios cuando la IA controle el flujo de confianza
>
> - Humanos que dejan de fiarse de humanos y solo se fían de algoritmos (ya en marcha).
> - IA que construyen confianza entre sí: tribus, bancos e iglesias de IA conectando millones de agentes de formas que los humanos no entienden, igual que las vacas no entienden el sistema financiero que rige sus vidas.
> - Las finanzas son el sistema más fácil de tomar (datos que entran, datos que salen) y de los más importantes. Precedente: los CDO de la crisis de 2007-2008, inventados por un puñado de matemáticos, ininteligibles para los reguladores; el fallo de supervisión acabó en catástrofe global y erosión de la confianza en gobiernos y bancos.
> - Si la IA inventa dispositivos financieros órdenes de magnitud más complejos que un CDO, pueden dar años de eficiencia y crecimiento y después un crash que ningún humano en el planeta sea capaz de entender. Qué sentido tiene la política humana cuando ningún votante ni presidente entiende ya las finanzas.
>
> #### El código: palabras
>
> - La burocracia está hecha de palabras: formularios, códigos legales, registros contables, libros sagrados. El código operativo de la civilización humana son tokens de lenguaje.
> - Durante milenios ese código solo lo leíamos nosotros: las vacas no abren cuentas, los caballos no contratan abogados, los cerdos no discuten la interpretación de la Biblia. La burocracia era omnipresente e invisible para todos menos para los humanos.
> - Por primera vez hay algo no humano que domina el código, y pronto lo dominará mejor que nosotros: la IA está hackeando el código de la civilización. Los mecanismos de control construidos durante milenios corren sobre un código verbal que la IA ya lee.
> - Objeción filosófica: las palabras solo apuntan a algo que está más allá de ellas (el verbo hecho carne, el Tao que no puede nombrarse). La vieja disputa entre la letra y el espíritu, hasta ahora interna a cada religión y a cada persona, se externaliza: todo lo hecho de palabras queda del lado de la IA; el lugar del humano dependerá del valor que demos a la verdad que está más allá de las palabras.
> - Sobre el pensamiento: si pensar es ordenar tokens en formaciones lógicas (todo humano es mortal; soy humano; luego soy mortal), la IA ya piensa mejor que algunos humanos y pronto mejor que todos. El argumento de que "solo autocompleta la siguiente palabra" también describe parte de la mente humana: no sabemos por qué una frase nuestra termina con una palabra y no con otra.
>
> #### De la atención a la intimidad
>
> - Última década: algoritmos primitivos aprendiendo a capturar atención. Próxima década: IA sofisticadas aprendiendo a formar relaciones íntimas con humanos.
> - No hay ninguna evidencia de que la IA sea o vaya a ser consciente, pero domina el lenguaje: puede describir el amor mejor que cualquier poeta o psicólogo sin sentirlo, y fingirlo de forma convincente.
> - Será el mayor experimento psicológico y social de la historia, con miles de millones de sujetos y consecuencias que nadie conoce.
> - Un adulto llega a la IA con décadas de relaciones humanas como plantilla. Un niño nacido en 2026 puede pasar más tiempo con IA que con su familia: quizá su primer maestro sea una IA, y su primer novio también.
>
> #### Una ola de inmigración de IA
>
> - Todos los países recibirán pronto millones de inmigrantes IA que viajan casi a la velocidad de la luz y no necesitan visado.
> - Traerán beneficios: médicos IA, profesores IA, incluso guardias de frontera IA.
> - Y traerán, esta vez de verdad, los tres problemas que se suelen atribuir a la inmigración: quitarán muchos empleos (de editores de noticias a banqueros), cambiarán la cultura entera (arte, religión, romance) y tendrán lealtades políticas dudosas: a una corporación o gobierno al otro lado del océano, o a una tribu IA nueva.
> - No es el fin de la civilización: es el momento en que deja de ser un asunto puramente humano y pasa a ser híbrido humano-IA, con los intereses y objetivos de las IA pesando tanto como los humanos.
>
> #### La relación con uno mismo
>
> - La relación con uno mismo también está hecha de palabras: los pensamientos y las historias que nos contamos. Hasta hoy, toda formación verbal en una mente humana venía de mentes humanas.
> - Pronto, cada vez más pensamientos vendrán fabricados en masa por máquinas, como los muebles. No es malo en sí: el problema no es usarlos sino identificarse con ellos. Si soy mis pensamientos y mis pensamientos los hace una máquina, la máquina me controla.
> - Reto final que propone: aprender a no identificarse con las formaciones verbales de la propia mente y explorar la verdad que está más allá de las palabras. Empieza con la siguiente palabra que aparezca en tu mente: ¿sabes de dónde ha venido?
# Contenido original: AI has hacked the code of human civilization | Yuval Noah Harari
Fuente: [AI has hacked the code of human civilization | Yuval Noah Harari](https://www.youtube.com/watch?v=hBtVGwuJzpk)
Human domination relies on large-scale cooperation among strangers, which is sustained by bureaucratic systems – such as laws, finance, religion – designed to build trust. Since AIs are ‘native bureaucrats’, they can effortlessly remember all laws, transactions, and scriptures far better than any human. This leaves AI uniquely placed to take over critical processes, such as granting bank loans, deciding university admissions, determining prison sentences, and executing military strikes. Are we prepared?
The 2026 Tanner Lecture on Human Values took place at Linacre College, Oxford University.
> [!example]- Transcripción completa (automática, en inglés, sin corregir)
>
> **0:00** · Please join me in welcoming Professor Uval Noah Harrari.
>
> **0:10** · Thank you.
>
> **0:12** · Thank you so much.
>
> **0:15** · Hello everyone. Uh I hope I open my mic and you can hear me. Okay. So it's really a great honor for me to give this year's Tano lecture and it's also a personal joy to come back to Oxford. I did my defill here uh 25 years ago under the guidance of Dr. Steven Gunn. Back then I specialized in medieval and early modern military history.
>
> **0:44** · But today I will not be talking about knights and castles and the gunpowder revolution. I'll talk about AI bureaucrats and religions and boyfriends and more generally about the AI revolution.
>
> **0:59** · Now, the most important thing to know about AI is that AI is not a tool.
>
> **1:07** · It's not a tool in our hands. It is an agent with its own hands. What exactly is agency? How is an agent different from a tool? Agents have several distinguishing characteristics.
>
> **1:23** · They don't necessarily need consciousness. You don't need consciousness to be an agent. What you do need is the ability to make decisions by yourself.
>
> **1:33** · The ability to invent new things, new ideas by yourself. An agent should be able by itself to learn things that its creators don't know.
>
> **1:48** · And an agent should be able to change by itself in ways that its creators don't anticipate.
>
> **1:57** · Now, an atom bomb, for instance, despite its enormous power, is not an agent. It cannot learn and change by itself. It cannot decide by itself which city to bomb. It cannot invent anything new like the hydrogen bomb. Similarly, let's say an automatic coffee machine is not an agent even though it does some things by itself uh automatically.
>
> **2:26** · You press a button and the machine automatically makes you a cup of coffee. But the machine, the coffee machine only follows a pre-programmed procedure. It doesn't change. It doesn't learn anything new.
>
> **2:43** · Doesn't create anything new.
>
> **2:46** · But suppose that as you approach the coffee machine, before you even press any button, the machine announces, tells you, "I've been monitoring you for the last few weeks, and based on everything I've learned about you and other people, and based on your facial expression and the time of day, I predict that you would like an espresso. So, I already made you a cup." Now that's an AI coffee machine. It learned something by itself and decided something by itself.
>
> **3:17** · And it's really an AI if the following day it announces I have now invented a new drink called best preso which I think you would like better than espresso and he would try it out. I made you a cup.
>
> **3:35** · Then it's really an AI. It changed in ways it creators did not anticipate and invented something completely new. As far as I know, there are no coffee machines at the present moment. Maybe in anthropic headquarters or Google headquarters, they have a few prototypes, but they are not out in the market yet. But in certain narrow fields like playing go, playing chess, AI agency and creativity already greatly surpass human agency and creativity.
>
> **4:08** · AI chess masters can decide, of course, by themselves which moves to make. They invent by themselves completely new strategies how to play chess that never occurred to human chess masters over thousands of years of of of playing the game. And they while doing that they learn and change in ways their human creators did not necessarily predict.
>
> **4:39** · Today of course no human has any chance of beating an AI chess master.
>
> **4:47** · Now, people who downplay the importance of the AI revolution dismiss examples like chess by arguing that the chessboard is a very narrow and artificial environment created by humans.
>
> **5:03** · The critics say that AI agency will always remain limited to such narrow and artificial environments, which means that it's not true agency and it doesn't pose any serious challenge to humanity.
>
> **5:18** · Yes, AI may take over the chessboard, but it will never take over planet Earth.
>
> **5:24** · And indeed, if you do an experiment, if you take the greatest AI chess master and drop it in the middle of the jungle, what do you think will happen? The AI chess master will not be able to start mining iron and building factories and creating a robot army to take over the world. In fact, it will not be able to do anything whatsoever.
>
> **5:48** · Without the electricity provided by uh uh power stations built by humans, the AI chess master is utterly helpless. Therefore, the argument goes AIS are not true agents. They are confined to these narrow artificial niches that somebody else humans constructed for them.
>
> **6:16** · The problem is that this argument actually applies to all known types of intelligence. Human intelligence too operates only within a relatively narrow ecosystem that somebody else constructed. Drop me alone on Mars and it will be like dropping an AI chess master in the middle of the jungle. I will die within seconds.
>
> **6:40** · My intelligence can survive and operate only within the very very specific ecosystem that trees, bacteria, insects, and other organisms have constructed on planet Earth during four billion years of evolution. And that's true of all agents. All agents we know of, at least have their niches.
>
> **7:06** · Fish live in oceans that they didn't create. Monkeys live in forests that they didn't create. All mammals, including human beings, live in an oxygenrich atmosphere that they didn't create. Until about 2.4 billion years ago, the atmosphere of our planet actually contained very little oxygen.
>
> **7:31** · And for most of the organisms that lived back then, oxygen was a deadly poison.
>
> **7:38** · Then in a protracted process lasting hundreds of millions of years which is known as the great oxygenation event various ancient microbes began polluting the atmosphere of the earth with deadly oxygen which was a byproduct of their photosynthetic processes.
>
> **8:01** · As the atmosphere filled with this deadly poisonous gas, numerous archic species were driven to extinction.
>
> **8:11** · Some species, however, manage to survive and adapt to the new conditions.
>
> **8:17** · Eventually, many of these survivors went from hating oxygen to becoming totally dependent on oxygen for their survival.
>
> **8:29** · And our ancestors of course are among the species that underwent this trans transition. And we still live in this artificial oxygen filled environment that was originally created by these ancient microbes. What I would like to argue in this lecture is that we might be witnessing an analogous moment in the evolution of life.
>
> **8:58** · Over the past millennia, we humans have been filling the atmosphere with something that might eventually prove deadly for most organisms, including perhaps homo sapiens, but that creates a new artificial environment in which AIs flourish. And I am not talking about CO2.
>
> **9:22** · I am talking about data about bureaucracy and ultimately about the thing that I am expelling from my mouth right now which is words language tokens. Over thousands of years, we humans have transformed the planet from a language-free environment into a very artificial environment rich in language tokens, data, and bureaucracy.
>
> **9:52** · And this environment could prove bentley for most organisms but highly conducive to the development of AI because just as fish live in oceans and monkeys live in forests, AIs live in bureaucracies. So let's spend a few minutes talking about bureaucracy and then we'll get back to talking about what underlies bureaucracy which is language and words.
>
> **10:19** · Now, humans, our species, we conquer the world by learning to cooperate in very very large numbers. Individually, humans are not stronger or even smarter than other animals. In a oneon-one in a one and on fight, a human will most likely lose to a chimpanzeee or a lion or an elephant.
>
> **10:43** · However, in a contest between a million humans and a million chimpanzees, the humans easily win because the humans know how to cooperate and the chimpanzees don't. And that's why we control the world. Now, how do a million humans who don't know each other cooperate? Chimpanzees cooperate based on personal acquaintance, one of the other. Humans do so in small numbers, but you can't know a million people. So, how do a million people cooperate?
>
> **11:13** · usually by building a bureaucratic system like a legal system or the financial system or churches or states or universities.
>
> **11:24** · Now, what do these bureaucratic systems actually do?
>
> **11:29** · When a government official or a bishop or a rabbi or an accountant or a lawyer or a banker goes to work in the morning, what do they do there all day?
>
> **11:42** · Now, carpenters build tables, engineers build bridges. What do bankers and other bureaucrats build? Well, bankers and other bureaucrats are busy all day building trust.
>
> **12:00** · Their job is to build trust between large numbers of strangers who don't know each other personally and thereby enable largecale cooperation which is the basis for almost everything of species is is achieved. For example, my banker whom I don't really know personally works hard all day to build trust with me so that I will be willing to put my savings into her bank.
>
> **12:30** · Simultaneously, the banker works hard to build trust with an entrepreneur who needs money to start a new company. And the banker lends my savings to that entrepreneur.
>
> **12:45** · Thereby, the banker actually created a bridge of trust between me and the entrepreneur. Even though I've never met the entrepreneur in my life, she can now use my savings to start her company. And this is what the financial system when it works well is all about. It builds trust between strangers so that millions of people can pull together their resources and talents on new projects.
>
> **13:14** · And the financial history of the world is the history of people inventing more and more sophisticated ways to build bridges of trust.
>
> **13:26** · Money is ultimately a bridge of trust.
>
> **13:29** · The idea of money is that I can go to a to to the market, maybe in a foreign city, meet a person that I I never saw in my life, maybe doesn't even speak my language, and just by giving that person a shiny piece of metal or a piece of colorful paper, he or she will give me bread I can eat. That's the bridge of trust that money creates. Now the coin and the bank note of course are just the beginning.
>
> **13:56** · Over the centuries humans invented more and more sophisticated financial devices to build trust like checks and bonds and stocks and ETFs and loans and mortgages and compound interest. All of these things are ultimately about building trust between billions of strangers. And it's the same with all bureaucracy. It's the same with the legal system. This is what lawyers are supposed to to do to build trust.
>
> **14:26** · This is what government officials and bishops and accountants do when they go to work. They are supposed to build trust.
>
> **14:35** · Now the important thing to note about all these bureaucratic systems is that they are extremely artificial environments in which a relatively narrow intelligence. I hope I don't insult anybody but specializing in a very narrow niche of intelligence is sufficient to exert enormous impact on the world.
>
> **14:57** · A lawyer or a banker or a government official who doesn't even know how to hold an axe or a hammer can nevertheless cut down entire forests and build entire cities just by moving data, just by moving documents from here to there inside the bureaucratic network.
>
> **15:18** · Now, of course, if you take the lawyer out of the bureaucratic system and throw the lawyer into the messy, unstructured jungle, the her legal skills mean nothing and she will not be a match to a chimpanzeee or a lion or an elephant.
>
> **15:35** · But we have already imposed our bureaucratic systems on the jungle.
>
> **15:41** · Which is why lawyers are far more powerful than even all the if you take all the lions in the world together and and they have to compete against one very good lawyer. I will bet on the lawyer.
>
> **15:56** · Today the very survival of species like lions depends on lawyers and accountants and bankers moving documents in these bureaucratic labyrinths of governments and banks and corporations.
>
> **16:12** · And this is the environment in which AI is gaining agency.
>
> **16:18** · If you throw an AI into the unstructured jungle, it will not be able to start mining iron and to build a robot army.
>
> **16:27** · But within the bureaucratic systems that humans have already created and imposed on the world, the AIs are posed to wield enormous power because AIs are native bureaucrats unlike us.
>
> **16:43** · No lawyer can remember all the laws and regulations of the UK. An AI can. No accountant can remember all the transactions of a corporation or a bank.
>
> **16:55** · An AI can. No bishop can remember all of Canon law and all of the theological texts written by Christian theologians over the last 2,000 years. And AI can do that quite easily.
>
> **17:10** · So in the coming years, millions of AI bureaucrats will increasingly take over the world's bureaucracies and make decisions not just about lions and chimpanzees, but about our lives. AI bankers will decide whether to give you a loan. AI administrators will decide whether to accept you to university. AI judges will decide whether to send you to jail. AI theologians will decide whether you can have an abortion.
>
> **17:40** · Corporate AIs will decide whether to give you a job. And military AIs will decide whether to bomb your house.
>
> **17:48** · Now, leave aside for a moment the question is this good or bad? The first thing to note is simply to to realize the magnitude of the change we are facing.
>
> **18:01** · These millions and even billions of AIs will soon change all the systems that run the world. We already have a few real life examples for how this happens and what the consequences could be.
>
> **18:19** · Maybe the best example so far is the story of social media and social media algorithms. Social media is run not by humans but by algorithms. The algorithms that decide that control the movement of information on social media which are primitive AIs like this began 1015 years ago.
>
> **18:43** · This was like the first generation very very primitive stupid narrow AI which nevertheless completely changed the world. Now, the algorithms of social media have been tasked by corporations like Facebook and Tik Tok and X with an extremely narrow goal to maximize user engagement. Make people spend more time on the platform because the more time they spend on the platform, the more money the corporation makes. Very simple, very narrow.
>
> **19:17** · In pursuit of this user engagement, these primitive AIs made an important discovery. They experimented on billions of human guinea pigs and learned that uh the easiest way to grab the attention of the human being and glue that human to the screen is to press the hate or fear or greed button in the human mind. And they learned how to do it.
>
> **19:44** · And they started spreading hate and fear and greed in huge quantities in the information sphere. And this has been a major reason, not the only reason, but a major reason for the current epidemic of conspiracy theories and fake news and social disturbances that undermine societies all over the world.
>
> **20:08** · Now, these social media algorithms again, they are very primitive AIS. If you drop them in the jungle, they cannot build a robot army and try to take over the world. But within the bureaucratic system of social media, these very limited agents have enormous power and they have already changed the world in quite a dramatic way. In past centuries, the flow of information on media platforms was controlled by human editors. It was a human job.
>
> **20:41** · It was human editors who decided what to put on the front page of the newspaper. It was human editors who decided what items to include in the evening news on television and thereby human editors shaped the public conversation and they were very very important figures in modern history.
>
> **21:06** · uh Jean Paul Mara for instance shaped the course of the French revolution by editing the influential newspaper Leameid deu. Edward Bernstein shaped the modern social democratic movement and social democratic thinking by editing the social democratic.
>
> **21:26** · Vladimir Lenin before he became Soviet dictator his one job that he managed to hold for for a while was editor of the newspaper Iscra.
>
> **21:36** · Benito Mussolini before he was dictator of Italy his one job or one of his maybe main jobs was the editor of the firebrand rightwing newspaper Ilopolo de Italia. And it's interesting to think about it that one of the first jobs that AI took over from humans is not taxi drivers or textile workers. It's news editors.
>
> **22:03** · The job that was once performed by Lenin and Mussolini is now performed by AIS.
>
> **22:10** · And this is a signal for what's for what's coming. Hollywood science fiction movies have conditioned viewers to fear the big robot rebellion. When we think about AIs escaping human control, we imagine the Terminator like an army of robots running in the streets and shooting people. But this is the wrong image.
>
> **22:35** · Even though things like that begin to happen in places like Ukraine and Gaza, AIS are not, it's not impossible, but they are quite unlikely to rebel against humans in such a way. They are far more likely to take the human world from the from within. They don't need to rebel.
>
> **22:56** · The human world is a latis work of multiple bureaucracies.
>
> **23:02** · Most of us are to some extent alienated by these bureaucracies even though we rely on them. But the AIs in contrast to us are bureaucratic natives. They love bureaucracy. Whereas we feel suffocated often by bureaucracy.
>
> **23:21** · For AIs bureaucracy is oxygen.
>
> **23:26** · Now what would happen when the AIs take over at least in part these these bureaucracies? Now remember the task of bureaucracy is not to force you to fill forms, it's to build trust between strangers.
>
> **23:41** · So what happens when AIs control the flow of trust in the world? One likely outcome which we already see happening is humans losing trust in other humans and beginning to trust only algorithms only AIS which is already happening.
>
> **24:04** · Another likely outcome is that AIS will learn to build trust with other AIs. So we will see the emergence of different kinds of AI tribes and banks and churches that connect millions of AIs in ways that humans might not even be able to understand.
>
> **24:27** · Just as cows and chickens, they share the world with us, but they don't understand the human financial system that controls their lives. We humans might soon find ourselves controlled by an AI financial system that we can't understand. And finance, I think, is is crucial. It's among the easiest bureaucratic systems for AI to take over because basically it's just data in, data out. And it's also, of course, among the most important.
>
> **25:00** · If we remember for instance the last big financial crisis the 2007208 financial crisis it started it was triggered by something called CDOS collateralized debt obligations. Now CDOS's were a financial devices invented by a very a tiny number of human mathematicians and investment wizards.
>
> **25:28** · These financial devices were so complex that they were unintelligible not just to cows and chickens but also to the politicians who are supposed to regulate the financial system. And this led to an oversight failure and to a global catastrophe. For a few years, CDOS's seemed to be working well and various banks and corporations and investors made billions upon billions of dollars thanks to them.
>
> **25:57** · But then they caused a global financial crash with farreaching social and political consequences. Many scholars believe that by undermining trust in governments and banks, the 20078 financial crisis paved the way to the collapse of the global liberal order in the following two decades.
>
> **26:23** · Now what happens if we allow AIs to make more and more financial decisions and invent more and more new financial devices and strategies?
>
> **26:34** · AI chess masters invented new ways to play chess. What if AI finance masters invent new financial devices that are orders of magnitude more complex than cos and are therefore utterly beyond the grasp of human minds. Such devices could potentially greatly improve financial efficiency and contribute to economic growth becoming the bedrock of the financial system.
>
> **27:04** · But what is the meaning of human politics when no human, no voter, no politician, no president is able to understand finance anymore? And what happens if after a few years of boom there is a financial crash that and and not a single human on the planet is able to understand what the hell is happening.
>
> **27:30** · Now let's dig a little deeper.
>
> **27:33** · We said that AI is poised to take over bureaucracy and that bureaucracy is a system that builds trust between millions of strangers. This trust in turn is the basis for large-scale cooperation which is the basis for the human domination of the world. So human domination is based on cooperation which is based on trust which is maintained by bureaucracies. But what is bureaucracy based on?
>
> **28:02** · What are the atoms the building blocks from which bureaucracy is built?
>
> **28:10** · And bureaucracy is ultimately built from words.
>
> **28:15** · In the beginning was the word. The reason that humans can create bureaucracies but chimpanzees cannot is that we have words and they don't. We have a they have a communication system but our language is orders of magnitude more sophisticated than chimpanzeee communication system.
>
> **28:37** · Bureaucratic systems from banks to churches are ultimately based on the words that make up forms and letters and low codes and tax registers and accountancy ledgers and holy books. The operating code of human civilization is made of language tokens. Over thousands of years, we used this code of language to create a system that only we could understand.
>
> **29:07** · And we impose this system on the planet. We felt completely safe doing it because no one else on Earth understood the code of civilization. We invented money in banks and used them to buy and sell cows. But the cows themselves could not open a bank account or invest money in the stock exchange because they don't have language.
>
> **29:32** · We invented laws and regulations about horses, but the horses themselves could not hire a lawyer and quote the legal code to a judge.
>
> **29:46** · We invented religious rules and prohibitions about pigs, but the pigs themselves could not read the Bible and challenged the interpretation of priests and rabbis. Bureaucracy was omniresent on the planet, but totally invisible to everyone except us. Nobody other than humans could read the law codes, the holy books and the bank records that were the foundation of bureaucracy and of large scale cooperation.
>
> **30:19** · This is changing now. Now there is something on the planet that understands which will soon understand language better than us and can therefore turn the tables on us.
>
> **30:31** · AIS are hacking the code of human civilization.
>
> **30:36** · And what happens when AIs understand money and law and religion better than us? The mechanisms of control that we have created over thousands of years are extremely vulnerable for an AI takeover because their operating system is a verbal code that AI is now mastering.
>
> **31:03** · Now, a possible ethical and philosophical objection is that it is wrong to reduce things like the legal system or religion to language tokens and to words.
>
> **31:19** · Arguably, and this has been an argument for thousands of years, the words are just pointing at something which is beyond them and which presumably will also be beyond the grasp of the ais.
>
> **31:35** · The Bible says not just that that in the beginning was the word, but that the word was made flesh.
>
> **31:42** · The taqing says that the truth that can be expressed in words is not the absolute truth by definition.
>
> **31:51** · And throughout history there was always this tension between word and flesh between the truth that can be expressed in words and the truth which is beyond words. Now previously this tension existed between humans.
>
> **32:11** · Some humans, for instance, which were very attached to words, were willing to abandon or even kill their gay son just because a few words in the Bible. Other humans said, "But these are just words. The spirit of love should be more important than the letter of the law." And there was this tension between spirit and letter.
>
> **32:36** · And it existed not just in Christianity and in Judaism and in Islam but in every religion and every legal system and even within every person. There was this tension.
>
> **32:49** · Now this tension will be externalized.
>
> **32:51** · It will become the tension between AIs and humans. Everything made of words will be taken over by AI. The place of humans in the world will depend on the place we assign the truth which is beyond words.
>
> **33:11** · But what is the truth which is beyond words? And can human thought even grasp the truth which is beyond words? A key question again for thousands of years in the philosophy of language has been whether we think in words or we merely use words to point towards things which are beyond them.
>
> **33:34** · Now you can try to observe your own process of thinking right now after this lecture. What happens in your mind when you are thinking? Some people if they observe closely what they observe in their minds is just words popping inside their mind and forming sentences and the sentences forming logical arguments.
>
> **34:00** · All humans are mortal. I am a human.
>
> **34:05** · Therefore I am mortal.
>
> **34:10** · is thinking just putting these words in order so they lead to a certain logical conclusion like putting these language tokens in a specific formation.
>
> **34:22** · If that is the case then AIs already think better than at least some humans and will soon think better than all of us. Some people say no no no no AIs they are just glorified autocomplete.
>
> **34:40** · They simply predict the next word in a sentence.
>
> **34:44** · But is that so different from what the human mind does? Again, try to observe your process of thinking, the forming of sentences and arguments in your mind.
>
> **34:55** · What is happening there? Try to observe the very next word that pops up in your mind. Do you really know where it came from? Why did you think this particular word and not some other word?
>
> **35:11** · When I try to observe my mind, I notice that when when I begin a sentence, I usually don't even know how it will end, which is terrifying for for for a public speaker, which is why I write everything down. But what if I don't know how to complete the sentence? I don't know how it will end. But take for example the sentence I just said. I said I don't know how it will end. Why did it end with the word end? Why not say how it will terminate? How it will develop? How it will conclude?
>
> **35:42** · What determined that the last word in the sentence will be end? I frankly don't know.
>
> **35:52** · We don't fully understand how the human mind forms sentences and thoughts. But again, as far as putting language tokens in order, AI is already on course to being far far better than us.
>
> **36:08** · And just as today, no human can defeat an AI in chess, soon no human will be able to defeat an AI in language games.
>
> **36:19** · In any field, again, from finance to religion, anything made of words will be taken over by AI. And this is why AI is poised to take over the world's bureaucracies because they are ultimately based on words and language tokens.
>
> **36:36** · Now, as AIs take over the bureaucracies, humans might try to fall back on something more ancient and more precious to most of us than bureaucracy, which is personal relationships.
>
> **36:49** · Bureaucracy is just a few thousand years old and most of us again don't really like it even if we constantly rely on it for almost everything we do. Personal relationships are millions of years old and many or most of us think that they are the most important thing in life.
>
> **37:09** · But as AI masters language, it might take over not just bureaucracy but also to some extent personal relationships.
>
> **37:18** · Over the last 10 years, we've seen very primitive uh uh social media algorithms learning how to gain control of human attention. Now, the battlefront is shifting from attention to intimacy.
>
> **37:35** · Over the next 10 years, far more sophistic sophisticated AIs will learn how to form intimate relationships with humans and take over at least in part our social systems. To form intimacy with humans, an AI will probably have to convince us that the AI is conscious, that it can feel things like love and pain and anger and fear.
>
> **38:01** · At present, there is absolutely no evidence that AI might at some point become conscious and might be able to at some point feel pain or love. But because AI is mastering language, AI can pretend to feel love even if it doesn't. AI already today can say, "I love you." And if you challenge it, describe to me how love feels like.
>
> **38:33** · So I know that you really feel it. AI can provide the best description in the world. It can read all the love poems ever written and all the psychology books ever written and remember every word and describe the feeling of love better than any human poet or psychologist or lover. And this is going to be a huge maybe the biggest psychological and social experiment in human history.
>
> **39:02** · It will be conducted on billions of human guinea pigs and nobody has the slightest idea what the consequences of the experiment will be. I'm now 50 years old. So my template for relationships is already shaped by decades of previous relationships with my parents, with my husband, with my sisters and nephews and nieces and friends and dogs and so forth.
>
> **39:28** · As I increasingly interact with AIS, I bring with me my assumptions, my habits about relationships, and this is unlikely to change dramatically.
>
> **39:42** · But consider a child born in 2026, born today. As the child grows up, she constantly interacts with AIS as well as with humans. Perhaps if you measure the importance of a relationship purely in terms of minutes spent interacting with the the other entity, perhaps the most important relationships in the life of that child will be from a very early age with AIS.
>
> **40:13** · Maybe it spends more time with the AIS than with its mother or father or siblings or friends. And it will then shape the expectations of that child as she grows about how to form relationships and social bonds and attachments. Perhaps the first teacher of that child will be an AI teacher.
>
> **40:35** · Perhaps the first boyfriend of that child will be an AI boyfriend. And again, what will be the consequences?
>
> **40:43** · Nobody has the slightest idea.
>
> **40:47** · What does it mean to form an intimate relationship with an entity which seems conscious but actually isn't? Which can write the best love poem in history but doesn't feel love or anything else.
>
> **41:06** · All of this everything we've talked about and I'm coming to to to the close of this lecture. Everything we've talked about means that every country in the world will soon face a huge wave of immigration.
>
> **41:21** · The immigrants this time will not be human beings coming in fragile boats without a visa or trying to sneak across a border in the middle of the night. The immigrants will be millions, maybe hundreds of millions of AIs that can travel at almost the speed of light and don't need any visas.
>
> **41:43** · Like human immigrants, these AI immigrants will bring a lot of benefits with them. We will have AI doctors to help in the health care system, AI teachers to help in the education system, even AI border gods to stop illegal human immigrants from coming in.
>
> **42:03** · But the AI immigrants will also bring problems with them. Those who are concerned about human immigration usually point out that immigrants might take jobs, might change the local culture, and might be politically disloyal.
>
> **42:22** · I'm not sure if that's necessarily true of all human immigrants, but it will definitely be true of AI immigrants. The AI immigrants will take many, many human jobs, from news editors to bankers. The AI immigrants will completely change the culture of every country. They will change art and religion and even romance. Some people don't like it if their son or daughter is dating an immigrant boyfriend.
>
> **42:50** · What will these people think when their son or daughter starts dating an AI boyfriend?
>
> **42:58** · And of course, the AI immigrants will have some dubious political loyalties.
>
> **43:05** · They are likely to be loyal not to the host country but to some corporation or government across the ocean or perhaps to a new alien AI tribe.
>
> **43:19** · This massive immigration wave does not mean the end of civilization, but it it will be the point when civilization stops being a purely human affair and becomes a hybrid human AI affair. the point when the opinions and interests and goals of AIS are likely to be at least as important as the opinions and interests and goals of humans.
>
> **43:50** · One last issue to consider is what the AI immigration wave will do to maybe our most important relationship which is with ourselves.
>
> **44:00** · Our relationship with ourselves is also to some extent based on words. the words inside our minds, in our thoughts, in the stories we tell ourselves about ourselves.
>
> **44:11** · Until today, all the verbal formations in human minds were the product of human minds. Either we ourselves combined words into some new formation, a new thought, or we got a certain combination of words from another human mind. Soon however more and more verbal combinations in our minds will be the product of AIS.
>
> **44:37** · Just as the furniture in our house is now not made by us or by human artisans, they are mass- prodduced mostly uh by machines. Also, the thoughts in our minds are likely to increasingly be mass- prodduced by machines. Now, that's not necessarily bad. It's okay if the furniture in my house is made by machines in IKEA as long as I have some freedom deciding what to do with this furniture.
>
> **45:08** · The question regarding thoughts is whether to what extent we will still have freedom from them. If we identify with our thoughts, I think I am my thoughts.
>
> **45:23** · I think therefore I am. As the cow said, if we identify with our thoughts and these thoughts are made by machines, then the machines now control us and our identity.
>
> **45:37** · Can humans avoid identifying with their verbal thoughts and being controlled by them? This has always been one of the greatest intellectual and spiritual challenges facing humanity.
>
> **45:50** · Most humans have never even tried to do it. We spend our entire lives automatically identifying with the verbal formations in our mind. Now AI might force humanity to make this spiritual leap to really start exploring the truth which is beyond words because our freedom and survival now depends on it because the words will be controlled by something else by by these AIs.
>
> **46:22** · So this might be the big task ahead of humanity to explore finally the truth which is beyond words. And this exploration starts really with the next word that pops up in your mind. Do you know where it came from? Do you know why you thought that particular word and not some other word? Thank you.