The Man Who Proved Systems Could Be Trusted

What if AI's biggest threat is not a smarter machine, but a less curious human?

In this episode of Decoded, Dr. Jurgi Camblong sits down with Pr. Joseph Sifakis, Turing Award laureate and inventor of model checking, the mathematical method that taught machines to prove their own correctness. Joseph has spent fifty years building the foundations of systems we can trust.

Now he is asking whether we are ready to trust them with our minds.

You'll hear:
•Why current AI cannot reason and what real autonomy would actually require
•Does the difference between symbolic and connectionist AI actually matter?
•Why being correct is not enough: systems also need values and accountability
•Why human cognitive dependence, not superintelligence, is the risk that worries him most
•What he tells young people about living a life worth living
A rigorous, clear-eyed, and unexpectedly hopeful conversation about intelligence; human and artificial.

For more information on this episode follow the link below:
https://www.linkedin.com/in/jurgicamblong/ 
https://www.sophiagenetics.com/podcasts/

Transcript

Copy

SPEAKER_01 0:01

Welcome to Decoded, the podcast where we explore how data, technology, and human insights are reshaping patient care worldwide. I'm Jorgi Kamlong, biologist, dreamer, and founder at Sophia Genetics. In each episode, we decode ideas with pioneers who have helped shape the way industries think, build, and move forward. Today, I'm excited to welcome one of the deepest thinkers in computer sciences, a man who asked a simple question. How do you prove that the system does exactly what it's supposed to do? He was born in Erachlian Crete in 1946, studied in Athens, and built his career in France. In the late 70s, he started his foundation work on model checking, with landmark publications in 1981 and 1982. This defined model checking as a field. In 2007, he wins a prestigious Turing work together with two American computer scientists, Edmund Clark and Alan Emerson. And today, their technology is the foundation for verification teams from space missions to Microsoft and Google. Joseph is still an active academic researcher. Over the past 10 years, he has been working on the development and validation of autonomous systems. In this episode, we will be talking differently about AI. Writing reduced our need for memory. Machines replaced our muscles. AI is different. For the first time, a technology can replace our mind. Joseph will tell us what AI actually is, what it cannot do, and where the real danger lies. Joseph, welcome.

SPEAKER_00 1:44

Thank you.

SPEAKER_01 1:45

So Joseph, as I get to knew you a couple of months ago in Athens, you told me that you had grown in Greece and then you moved as a student to France. Can you tell us a bit more about it?

SPEAKER_00 1:59

I was born in Heracleum, Greece, where I attended high school. I studied at the National Technical University of Athens, where I earned a degree in electrical engineering. Fearing the military dictatorship of the colonel at that time, I found myself in August 1970, somewhat by chance, in Grenoble France, with the goal of pursuing studies in nuclear physics. It was in Grenoble there that I discovered computer science. I encountered computers for the first time, and I decided to completely change my career path to pursue studies in computer science. This was a turning point in my life, a decision I made without giving it much thought, but true. How I very quickly found myself working as a computer science researcher at the CNRS, which is the National Research Science Foundation in France, and I was started as a researcher in 1975.

SPEAKER_01 3:14

I think you told me one day we're all immigrants, but it's great to see a neighboring immigrant who came from Greece to France and in 2007 won the Turing Award for inventing model checking. Can you explain us what model checking is about for a non-technical audience?

SPEAKER_00 3:35

Well, I will try. You may know that computer science differs from the physical sciences in that it does not provide sufficient theoretical foundations for building systems. Let me take an example. If you want to build a bridge, mechanics allows you to construct it and predict with very high probability that it will not collapse for centuries. This type of constructive framework does not exist for computer systems. When I was a student, systems were built empirically and then they were tested to see if their behavior matched the expected behavior. But testing systems needs a very, very large number of inputs and is very costly and does not allow you for a complete exploration of the behavior. Now, an idea was to improve upon testing, and the approach is more is called verification. The idea is to construct a mathematical model of the system you want to check and reason on this mathematical model to establish the validity of property. So here, because you have a model, you can analyze thoroughly the behavior of the system. I worked on this idea, my results on model checking uh have uh made it possible to implement.

SPEAKER_01 5:11

And as you said, Joseph, at that time, 1970s, you had that idea. This type of operations for systems would be very complex and would require a lot of compute power, which wouldn't exist, right, in 1997. How do you explain it? How do you explain it? Was this intuition?

SPEAKER_00 5:33

No, no, yeah yes, okay. I liked mathematics, I liked model, uh logic, so I built a theory, and the theory was just an intellectual construction, okay? I remember when I passed my PhD, some people in the jury said, but this theory is not applicable. Okay, it's a nice theory, it's not applicable. So I was unable to put my theory in practice, but this was a nice theoretical result. More powerful machines that were not available at that time, I could apply my theory. Okay, so I had to wait until computer performance improved. And also I should say that the need for verification became increasingly urgent because people were developing also increasingly complex systems. So in 1994, I launched the first Pentium processor. So this processor had a problem. It had a failure, but the failure was an intermittent failure and it was poorly understood why this appeared. And it was very hard to reproduce, to reproduce through testing. At that time, of course, Intel's reputation was at stake and also a lot of money, okay. And Intel decided to invest heavily in the development of the first industrial-scale technology based on model checking. I think they were successful. They never told me what they achieved. Intel's example was immediately followed by other developers of critical systems: NASA, IBM, Microsoft, and later.

SPEAKER_01 7:15

That's very impressive. Practically, today, what are applications that we might be using, or computer sciences might be using and are leveraging on model checking?

SPEAKER_00 7:27

There are many, many applications. It's applied to hardware, to critical software, because you see applying model checking techniques costs a lot of money. You need also very high-performance computers. You don't try to verify a software that you will use it for non-critical tasks. You know that we have these distinctions, and this distinction in systems, we have critical systems, we have mission critical systems, and then we have best effort systems. So typically, for instance, that software you use on your iPhone is a best effort system. Okay. The reliability of the software to build depends on the degree of criticality. So for highly critical systems, I have worked with flight controllers, for instance. Here the reliability should be 10 to the minus 9 failures, less than 10 to the minus 9 failures per hour of flight, which is a very, very strong requirement. And you see, for rockets, even if you have astronauts inside, it's only 10 to the minus 6 failures per hour of flight. Okay, so for ordinary best effort systems, it's 10 to the minus 4. So this difference in reliability is reflected in uh difference in development costs. In fact, if I have a system and I want to multiply the reliability by 10, say, the incurred, they entailed the cost can be multiplied by 1000. Okay. The costs do not increase exponentially with increase of the reliability degree.

SPEAKER_01 9:08

It has to be very much focused on things as you said that are critical and where the computer power that will be used is really worth it.

SPEAKER_00 9:15

Yes, but you see now if for instance a company develops a chip that is going to be used for years by, I don't know, thousands or even millions of people, okay. I mean, it's worthwhile testing, okay, because if you have a failure, then this can be a problem. An economical one and a reputation problem.

SPEAKER_01 9:36

Absolutely, yes. You're speaking about chips as well.

SPEAKER_00 9:40

Speaking about chips, about critical software that you use in nuclear plants, in aircraft, in missions, of course, yes. A lot of money is at stake. Now your software, the software you have in your laptop fails, and you know that they can fade, and it's enough to restart your laptop to cope with that, this is not a problem.

SPEAKER_01 10:01

Makes sense. After model checking, Joseph, you moved into the design of autonomous systems, including I understand the autonomous car systems. What is your research topic today?

SPEAKER_00 10:15

Uh yes, I work on autonomous systems. I found the concept fascinating. So I stopped working on model checking at the end of the previous century, okay. So I found my contribution there was would not be very significant. Because you see, in a scientific domain, it's like exploiting a mine. Okay, initially you find a lot of things, and after a while, okay, the gains and the progress can be marching. I was fascinated by the concept of autonomy, and the question is how we can build the intelligent machines that replace humans in performing cognitive tasks, driving cars, understanding complex situations, taking action to achieve goals in physical and social environments. This vision also was allowed aligned with that of the Internet of Things. You know, the idea is to coordinate the action of systems deployed across countries, across continents or even the planet to better manage resources and address global challenges. So today I work in collaboration with industry teams to build autonomous systems in the fields of autonomous driving, autonomous telecommunication, and also financial systems.

SPEAKER_01 11:34

And when you refer to better managed resources, do you have in mind resources that humans have built like Earth or resources we extract from Earth?

SPEAKER_00 11:42

Any kind of resource. A resource is something that allows you to solve a problem. A resource can be knowledge, in fact. But a resource also is energy. So typically, just to give you an example, you know that today there are projects to manage smart grids at the level of a continent, for instance. This is a very interesting idea because you have a diversity of sources of electrical energy. And the idea is that when, for instance, you have solar energy that is produced in the south of Europe to be able to switch and transfer this energy to the north of Europe, for instance. Okay, and to have a nice balance and have compromises between uh I mean to have the best possible use of energy. And in fact, managing a smart grid is a very, very hard technical problem. The problems they had recently in Spain, for instance, with shortages of energy just because of a kind of imbalance that has been produced, okay? I'm talking about problems, problems like this problem, okay? How to manage globally resources at a planetary level.

SPEAKER_01 12:58

Very good and very important. In 2022, you published a book titled Understanding and Changing the World. What did inspire you to write a book and what is the central argument of that book?

SPEAKER_00 13:13

Yes, it's precisely the idea of system autonomy and the approach to human intelligence through this concept, because I mean autonomous systems are very close. The target is how to make systems as autonomous as humans, in fact. So I have worked on the design of autonomous systems for various fields of application, and I asked myself, what is missing from my systems to bring them closer to human intelligence, which is characterized by this ability to understand and challenge the world, and was the title of my book. For me, it was a kind of challenge to imagine the missing elements needed to mimic human cognitive functions and their coordination. So I had already an architecture for autonomous systems, for instance for self-driving cars, I try to find the missing elements. So this was a purely intellectual exercise, okay? The conclusion of this work is that although machines may outperform humans in realizing certain cognitive tasks, like playing chess or summarizing text, they are unable to coordinate each of these tasks rationally to simultaneously achieve a large number of goals.

SPEAKER_01 14:31

Today we speak a lot about AI. Eventually, sometimes the terms or the capabilities are a bit overstated or misunderstood, despite these are definitively very powerful tools. In the context of autonomous systems and responsibility, what do you think are the most important things and what are the things that eventually people misunderstand?

SPEAKER_00 14:55

What I would like first to emphasize, because there is a lot of misunderstanding about is AI's infancy despite the declarations that computers and machines outperform humans and things like that. So today big companies are focusing on conversational AI. So you have systems where you ask questions and receive answers. But this is not enough to meet the need for intelligence systems in science, business, government, and especially in industry. So of course I said that big tech claims that machines have already surpassed human intelligence, but only in specific applications. There is an important debate today about the goals of AI. So big tech say that the goal of AI is to achieve artificial general intelligence. So this means that we have machines that cannot perform humans in every possible task. This is an ill-defined concept because we don't know how many different tasks can perform humans, okay? And what I demonstrated in my book is human intelligence is not a kind of Swiss army knife of cognitive function. So I have a system that can drive, can play chess, can do this. But the important problem is is there any coordinator of all these functions? Okay, I have a Swiss army knife, but what is missing is the user of the knife, and this is the human mind, and we don't have this.

SPEAKER_01 16:26

As you speak about coordinator, I would be critical and I would be very much in favor of AI taking over humans. I may tell you, yeah, but we have orchestrators. Why orchestrators are not the coordinator you're talking about?

SPEAKER_00 16:42

Of course, in some AI systems they are using orchestrators. The problem is whether orchestrator is reliable enough and has the ability to coordinate many cognitive tasks. The problem is that today we don't have reliably enough orchestrators. And an orchestrator also should be able to plan, to make decisions. And for this, at least, machine learning systems are not good enough. You see, AI systems are very good perhaps at analyzing multidimensional data. So extracting knowledge from multidimensional data. This is the issue of understanding the world. But the other aspect of intelligence that is very, very important is acting on the world. To act on the world, this means that you have a model of the world, which humans have. In fact, humans have common sense knowledge, so they can figure out, they can see in their mind their self acting on the world, okay? And they can make predictions and they can choose goals, and they have also criteria for choosing goals. All this deliberative part of the human mind cannot be simulated, cannot be mimicked by AI today.

SPEAKER_01 18:04

You explained as well that there are two fundamentally different approaches to building intelligent systems: symbolic and connectionist, fast thinking and slow thinking. Can you further develop that thinking?

SPEAKER_00 18:18

Yes, I should say that AI started as a discipline, as we probably know, in 1956, and they have pursued two different approaches. One that is symbolic. So symbolic means the symbolic approach relies on programming, uh, logic, and the other is connectionist. So connectionist relies on neural networks, and these neural networks mimic the neural networks of our brain, as you probably know. So there is a fundamental difference between machine learning systems and traditional systems. So traditional systems is the approach that is taken by symbolic AI. These types of systems are based on very different paradigms of computation and knowledge generation. When you have a symbolic AI, it's like programming a machine by using languages and logic, and you fully understand the behavior of a program that is written, and you can execute it step by step, and you can have it and buffer. In contrast, machine learning systems are neural networks, as I said, that mean card brains. These are systems that learn from a large amount of data, but their behavior cannot be explained in the same way as that of the programs. Machine learning systems can learn as kids learn to distinguish between cats and dogs. I think an analogy that helps illustrate this difference is the distinction between fast thinking and slow thinking. This distinction proposed by Daniel Kahneman in his book Fast and Slow Thinking, humans have two systems of thinking. Fast thinking relies on our brains, neural networks, and the associated machine learning mechanisms. So when I speak, when I walk, my mind, my brain is producing empirical knowledge, but I don't understand how I can do that. And these are very hard problems to solve. And most of our empirical knowledge is in fact produced by fast thinking. In contrast, slow thinking is procedural, it can be understood and analyzed. So I hope by taking this analogy between fast and slow thinking, I can explain the differences between symbolic AI and connectionist AI.

SPEAKER_01 20:49

Joseph, it's always fascinating to hear to you, and obviously the audience realize how deep you are in technology and AI. I guess one of your messages as well was that you are critical, as you should be, I think, as a scientist. The main risk you see yourself from a previous conversation is not this superintelligence. It's something that is a bit quieter in your view and far more certain. So, what are the risks of AI?

SPEAKER_00 21:16

Of course, we have technological risks. So if you trust a system and something goes wrong, if you believe that it is safe and it is not safe, so these are technological risks. And then you have systemic risks. And the systemic risks come from the long-term use of intelligence and have an impact on humans and society. There are myths that claim that machines will become smarter than us, and these myths predict that we will be eventually dominated by them, and we will finish as some kind of domesticated animals for them. I think these myths are in fact promoted by very famous people, okay? Are simply Perpetuated for marketing purposes and also to make people believe in technological determinism. Technological determinism says, roughly speaking, that technology is much more important. I mean, technology will finally dominate humans, okay? And this is humans, in fact, are not the main players in their own history, okay? It's a kind of inevitability of domination of humans by technology. In my view, the concrete and immediate risks come from the fact that humans are increasingly reliant on machines to perform cognitive tasks on their behalf. This entails a dependence on machines to make important decisions and standardization of thought. So we have some kind of outsourcing of our thinking. This will result in some atrophy of our cognitive capabilities, and this is the danger. And this danger is not addressed in depth in a public debate today, and it's a pity. I think that the misuse of AI can have the effects of a drug, especially for very young people that use AI, for instance, to do their homework. I think we need some regulations. You see, we regulate alcohol and tobacco consumption, especially for young people, and there is no reason why we shouldn't have a legal framework to regulate the use of AI.

SPEAKER_01 23:33

Is there still hope for young people? And my last question in that regard to conclude is would you encourage them to be in computer science or AI and for what purpose?

SPEAKER_00 23:44

So you are talking about the young generation. I don't see any reason why not to encourage people to study computer science. The problem is not computer science, it's not the machines, but rather our inability to manage them for our own good. Okay. I would recommend that they should try to have a very strong background, cultural background, to pursue a well-rounded education. The job market is unstable, it is volatile. If you have a thin knowledge background, if your knowledge is too limited, it will be difficult to adapt to it. Also, I would recommend young people to be disciplined, to be rigorous and honest with themselves, to believe in their strengths. So not to ask a lot of advice on machines, or I mean to try to make their own decisions based on an analysis of their forces and their weaknesses. Also, I would recommend they get involved in real projects, to try to do something exciting. I see that many young people may choose a boring job that allows them to make money. In fact, the young people are very anxious with making money. This is something I never did in my life and never regretted that. I would prefer a position that I am passionate about and in which I can excel instead of a boring job. The idea is to try to be the best in their field. I think that if you are the best in your field, the money and recognition will come later. Life is unique, we should make it worth living.

SPEAKER_01 25:49

Joseph, thank you for joining me on Decoded. It was really excellent having you in this episode. What I love most from this conversation is a simple idea with profound consequences, that being correct is not enough. Joseph has spent 50 years building the foundations for systems we can trust. That work has never been more needed because the deepest questions in technology are what do we trust, what are we accountable for, and what do we risk losing if we stop thinking for ourselves? Because the risks are not that machines become more intelligent than us. The risk is actually that we become the quicker part of the system. I really enjoyed this conversation and I hope you did as well, Joseph.

SPEAKER_00 26:34

Thank you, thank you very much. Thank you for giving me the opportunity to express my ideas. Thank you.

SPEAKER_01 26:39

If you enjoyed this episode, make sure to subscribe to Decode It. We continue meeting leaders like Joseph, builders, shapers of both Healthcare Technology and AI. I'm Yuri Camlon, thank you for listening and let's keep moving forward.