The following is a conversation with Callicoon. He's considered to be one of the fathers of deep learning, which, if you've been hiding under a rock, is the recent revolution. And now that's captivated the world with the possibility of what machines can learn from data. He's a professor at New York University, a vice president and chief scientist of Facebook, and a recipient of the Turing Award for his work on deep learning. He's probably best known as the founding father of convolutional neural networks, in particular their application to optical character recognition and the famed amnesty data set.

He is also an outspoken personality, unafraid to speak his mind in a distinctive French accent and explore provocative ideas both in the rigorous medium of academic research and the somewhat less rigorous medium of Twitter and Facebook.

This is the Artificial Intelligence Podcast. If you enjoy, subscribe on YouTube. It five stars, an iTunes supported a patron are simply going to record me on Twitter. Àlex Friedman spelled F.R. Idi Amin. And now here's my conversation with John Lueken. You said that 2001 A Space Odyssey is one of your favorite movies. HAL 9000 decides to get rid of the astronauts for people who haven't seen the movie, spoiler alert because he it she believes that the astronauts, they will interfere with the mission.

Do you see how is flawed in some fundamental way or even evil, or did he do the right thing?

Neither. There's no notion of evil in that in that context other than the fact that people die. But it was an example of what people call value misalignment, right. To give an objective to a machine. And the machine tries to achieve this objective. And if you don't put any constraints on this objective, like don't kill people and don't do things like this. The machine, given the power, will do stupid things just to achieve this objective or damaging things to achieve this objective.

It's a little bit like we are used to this in the context of human society. We we put in place. Laws to prevent people from doing bad things because spontaneously they would do those bad things right. So we have to shape their. Cost function, the objective function, if you want through laws to kind of correct and education, obviously to sort of correct for for those. So maybe just. Pushing a little further on that point, how, you know, there's a mission, there's a there's fuzziness around the ambiguity around what the actual mission is, but.

You know, do you think that there will be a time from a utilitarian perspective when a system where it is not misalignment, where it is alignment for the greater good of society, then the system will make decisions that are difficult?

Well, that's the trick. I mean, eventually we will have to figure out how to do this. And again, we're not starting from scratch because we've been doing this with humans for four millennia. So designing objective functions for people is something that we know how to do. And we don't do it by, you know, programming things, although the legal code is called code. So that tells you something. And it's actually the design of an object.

You function. That's really what legal code is, right? It tells you what you can do. Here is what you can do. If you do it, you pay that much. That's that's an objective function.

So there is this idea somehow that it's a new thing for people to try to design objective functions that are aligned with the common good. But no, we've been writing laws for millennia, and that's exactly what it is. So that's where, you know, the science of law making and and computer science will come together, will come together. So it's nothing there's nothing special about how or A.I. systems. It's just the continuation of tools used to make some of these difficult ethical judgments that laws make.

Yeah. And we and we have systems like this already that, you know, make many decisions for for ourselves and society that need to be designed in a way that they like, you know, rules about things that sometimes sometimes have bad side effects. And we have to be flexible enough about those rules so that they can be broken when it's obvious that they shouldn't be applied. So you don't see this from the camera here. But all the decoration in this room is all pictures from 2001, A Space Odyssey.

Wow.

By accident or by accident? It's by design.

Wow. So if you were if you were to build HAL 10000, so an improvement of HAL 9000. What would you improve? Well, first of all, I wouldn't ask you to hold secrets and tell lies, because that's really what breaks it in the end. That's the fact that it's asking itself questions about the purpose of the mission and it's pieces, things together, the turd, you know, all the secrecy of the preparation of the mission and the fact that it was a discovery on the lunar surface that really was kept secret.

And and when part of Hal's memory knows this and the other part is does not know it and is supposed to not tell anyone. And that creates internal conflict.

So you think there's never should be a set of things that a system should not be allowed?

Like a set of facts that should not be shared with the human operators? Well, I think no, I think that I think it should be a bit like.

In the design of, uh, uh, autonomous systems, this should be the equivalent of, you know, the the oath that, uh, Hippocratic Oath that, uh, doctors sign up to. Right. So there are certain things, certain rules that that you have to abide by. And we can sort of hardwired this into into our into our machines to kind of make sure they don't go. So I'm not, you know, advocates of the three to three laws of robotics, you know, the Asimov kind of thing, because I don't think it's practical, but.

But, you know, some some level off of limits, but but to be clear, this is not these are not questions that are kind of really worth asking today because we just don't have the technology to do this. We don't we don't have autonomous, intelligent machines. We have intelligent machines. Some are intelligent machines that are very specialized, but they don't really sort of satisfy an objective. They just kind of train to do one thing. So until we have some idea for a design of a full fledged autonomous, intelligent system, asking the question of how we design is subjective, I think is a little a little too abstract.

It's a little too abstract. There's useful elements to it in that it helps us understand our own ethical codes, humans. So even just as a thought experiment, if you imagine that Najai system is here today, how would we program? It is a kind of nice thought experiment of constructing.

How should we have a law, have a system of laws for us humans? It's just a nice practical tool. And I think there's echoes of that idea too. In their assistance we have today. They don't have to be that intelligent. Yeah, like autonomous vehicles as these things start creeping in that we're thinking about. But certainly they should be framed as hell.

Yeah. Looking back, what is the most I'm sorry if it's a silly question, but what is the most beautiful or surprising idea in deep learning or A.I. in general that you've ever come across sort of personally what you said back? And just had this kind of that's pretty cool moment, that's nice. Well, surprising. I don't know if it's an idea rather than sort of empirical fact. The fact that. You can build gigantic neural nets to train them on.

Relatively small amounts of data, relatively with stochastic gradient descent, that it actually works, breaks everything you read in every textbook, every pretty deeply textbook.

I told you, you need to have fewer parameters and you have data samples. You know, if you have non convex objective function, you have no guarantee of convergence. You know, all those things that you reading textbook and they tell you, stay away from this.

And they all wrong a huge number of parameters, non convex and somehow, which just very relative to the number of parameters data, it's able to learn anything. Right. Does that still surprise you today?

Well, it was kind of obvious to me before I knew anything that this is a good idea. And then it became surprising that it worked because I started reading those textbooks.

OK, so, OK, you talk to the intuition of why it was obvious, if you remember.

Well, OK, so the intuition was it's it's sort of like, you know, those people in the late 19th century who proved that heavier than than air flight was impossible.

Right. And of course, you have birds, right? They do fly. And so on the face of it, it's obviously wrong as an empirical question. Right. And so we have the same kind of thing that, you know, we know that the brain works. We don't know how, but we know it works. And we know it's a large network of neurons and interaction and learning takes place by changing the connections. So can of getting this level of inspiration without copying the details, but sort of trying to derive basic principles.

That kind of gives you a clue as to which direction to go. There's also the idea somehow that I've been convinced of since I was an undergrad that. Even before that, intelligence is inseparable from running, so the idea somehow that you can create an intelligent machine by basically programming for me was a non-starter from the start. Every intelligent entity that we know about.

Arrives at this intelligence training, so learning machinery was completely obvious path also because I'm lazy, so, you know, automated, basically everything and learning is the automation of intelligence, right.

So do you think so what is learning then? What falls under learning?

Because do you think of reasoning as learning where our reasoning is certainly a consequence of learning as well, just like other functions of, uh, of the brain. The big question of our reasoning is, how do you make reasoning compatible with gradient based learning? Do you think neural networks can be made to reason?

Yes, that there is no question about that. Again, we have a good example, right? The question is, is how so the question is how much prior structure do you have to put in the neural net so that something like human reasoning will emerge from it, you know, from running? Another question is, all of our kind of model of what reasoning is that are based on logic, are discrete and and and are therefore incompatible with grid and baserunning and have a very strong believer in this idea, ground invasion.

I don't believe that other types of learning that don't use kind of gradient information if you want.

So you don't like discrete mathematics, you don't like anything discrete. Well, that's it's not that I don't like it, it's just that it's incompatible with learning. And I'm a big fan of learning. Right. So, in fact, that's perhaps one reason why. Deplaning has been kind of looked at with suspicion by a lot of computer scientists because the math is very different, the method you use for deploying has more to do with, you know, cybernetics, the kind of math you do in electrical engineering than the kind of math you do in computer science.

And and, you know, nothing in machine learning is exact. Right. Computer science is all about sort of, you know, obsessive compulsive attention to details of like, you know, every index has to be right. And you can prove that an algorithm is correct. Right. Machine learning is the science of sloppiness. Really.

That's beautiful. So, OK, maybe let's feel around in the dark of what is a neural network that reasons or a system that.

Is works with continuous functions that's able to do build knowledge, however we think about reasoning, builds on previous knowledge, build on extra knowledge, create new knowledge, journalize outside of any training set ever built.

What does that look like if. Yeah, maybe you have inklings of thoughts of what that might look like.

Well yeah. I mean, yes or no. If I had precise ideas about this, I think, you know, we would be building it right now. But and there are people working on this or whose main research interest is actually exactly that. Right. So what you need to have is a working memory. So you need to have. Some device, if you want some subsystem, they can store a relatively large number of factual episodic information for, you know, a reasonable amount of time.

So, you know, in the in the brain, for example, the kind of three main types of memory. One is the the sort of. Memory of the state of your cortex, and that sort of disappears within 20 seconds, you can't remember things for more than about 20 seconds or a minute if if you don't have any other form of memory. The second type of memory, which is longer term, is your short term is the hippocampus.

So you can you know, you came into this building, you remember where the where the exit is, where the elevators are.

You have some map of that building that's stored in your hippocampus. You might remember something about what I said, you know, a few minutes ago. I forgot it already has been erased. But, you know, but at some point in your hippocampus and then the longer term memory is in the synapse, the synapses. Right. So what you need if you want a system that is capable of reasoning, is that you want the hippocampus like thing. Right.

And that's what people have tried to do with memory networks and military machines and stuff like that. Right. And now with transformers, which have sort of a memory and they are kind of self attention system, you can think of it this way. So. So that's one element you need, another thing you need is some sort of network that can. Access is memory. Get an information back and then kind of crunch on it and then do this iteratively multiple times because a chain of reasoning.

Is the process by which you you you can you update your knowledge about the state of the world, about, you know, what's going to happen, et cetera, and that there has to be this sort of recurrent operation, basically.

And you think that kind of if we think about a transformer, so that seems to be too small to contain the knowledge that that's to represent the knowledge is contained.

Wikipedia, for example, a transformer doesn't have this idea of recurrence. It's got a fixed number of layers. And that's a number of steps that limits basically representation, but recurrence would build.

On the knowledge, somehow, I mean, it would evolve the knowledge and expand the amount of information, perhaps, or useful information within or not. But. Is this something that just can emerge with size because it seems like everything we have now? I just know it's not it's not it's not clear how you access and writing to an associated memory, an efficient way. I mean, sort of the original memory network maybe had something like the right architecture.

But if you tried to scale up a memory network so that the memory contains all the Wikipedia, it doesn't quite work. Right. So so there's a need for new ideas there. OK, but it's not the only form of reasoning. So there's another form of reasoning, which is true, which is very classic also in some types of A.I. And it's based on, uh, let's call it energy minimization.

OK, so you have, uh, some sort of objective, some energy function that represents the, uh, the the, um, the quality or the negative quality. OK, energy goes up when things get bad and they get low when things get good. So let's say you you want to figure out, you know, what gestures do I need to to do. To grab an object or walk out the door. If you have a good model of your own body, a good model of the environment, using this kind of energy minimisation, you can make a you can make you can do planning and it's in optimal control.

It's called it's called model predictive control. You have a model of what's going to happen in the world as a consequence of your actions, and that allows you to buy energy minimization, figure out a sequence of action that optimizes a particular objective function which measures, you know, minimize the number of times you're going to hit something and the energy you going to spend doing the gesture and etc..

So so that's a form of reasoning, planning as a form of reasoning and perhaps what led to the ability of humans to reason is the fact that or, you know, species that appear before us had to do some sort of planning to be able to hunt and survive and survive the winter in particular.

And so, you know, it's the same capacity that you need to have.

So in your intuition is if you look at expert systems and encoding knowledge is logic systems. And as graphs on this kind of way is not a useful way to think about knowledge, graphs are little brittle or logic representation.

So basically, you know, variables that they have values and then constrained between them that are represented by rules as well. Too rigid and stupid. All right.

So one of the you know, some of the early efforts in that respect were were to put probabilities on them. So a rule, you know, you know, if you have this and that system, you know, you have this, uh, disease with that probability and you should prescribe that antibiotic with that probability. Right. In system from the from the 70s.

Um, and that that's what that branch of A.I. led to, building networks and graphical models and causal inference and vibrational, you know, method.

So so there is I mean certainly, uh, uh, a lot of interesting work going on in this area. The main issue with this is, is knowledge acquisition. How do you, uh, reduce a bunch of data to a graph of this type relies on the expert and the human being to encode that knowledge.

And that's essentially impractical.

Yeah, so that's a big question. The second question is, do you want to represent knowledge symbols and do you want to manipulate them with logic? And again, that's incompatible with learning. So, uh, one suggestion with Geoff Hinton has been advocating for many decades is replace symbols by, uh, vectors. Think of it as a pattern of activities in a bunch of neurons or units or whatever you want to call them and replace logic by continuous functions.

Okay.

And that becomes now compatible is a very good set of ideas by, uh, written in a paper about 10 years ago by, uh, Leongatha on who is here at Facebook.

Um, um, the title of the paper is for Machine Learning to Machine Reasoning. And his idea is that, uh, learning a learning system should be able to manipulate objects that are, in the sense, inner space and then put the result back in the same space.

So this idea of working memory, basically, and it's, uh, it's very enlightening and in a sense that might learn something like the simple expert systems.

I mean, it's what you can learn, basic logic operations there.

Yeah, quite possibly.

Yeah. There's a big debate on sort of how much prior structure you have to put in for this kind of stuff to emerge. This debate I have with Gary Marcus and people like that. Yeah, yeah.

So and the other person. So I just talked to Judea Pearl from the he mentioned causal inference world. So his worry is that the current neural networks are not able to learn. What causes what causal inference between things? So I think I think it's right and wrong about this. If he's talking about the sort of classic. Type of neural nets, people often worry too much about this, but there's a lot of people now working on Cosan in France and there's a paper that just came out last week, valuable to, among others, as as a bunch of other people.

Exactly. On that problem of how do you kind of get a neural net to sort of pay attention to real causal relationships, which may also solve.

Issues of bias and data and things like this, so I'd like to read that paper because that ultimately the challengers also seems to fall back on the human expert.

To ultimately decide causality between things, people are not very good at establishing causality, first of all. So first of all, you talk to a physicist and a physicist actually don't believe in causality because look at the all the basic laws of physics all time reversible. So there is no causality. The arrow of time is not right. It's as soon as you start looking at macroscopic systems where there is unpredictable randomness, where there is clearly an area of time.

But it's a big mystery in physics, actually, how that emerges.

Is that emergent or is it part of the fundamental fabric of reality, or is it a bias of intelligent systems that because of the second row of somewhat dynamics, we perceive a particular area of time, but in fact, it's kind of arbitrary, right?

So, yeah, physicists, mathematicians, they don't care about I mean, the math doesn't care about the flow of time.

Well, certainly. Certainly macro physics doesn't people themselves are not very good at establishing.

Causal causal relationships, if you ask this, I think it was in one of similar papers spoken on, um, like children learning, you know, he studied, which, you know, he's the guy who co-authored the book Perceptron with Marvin Minsky, the kind of kill the first wave of neural nets. But but he was actually a learning person. He in the sense of studying, learning in humans and machines. That's what you got interested in Perceptron.

And, uh, and he he he wrote that if you ask a little kid, uh, about what is the cause of of the wind. A lot of kids will say they will think for a while and they'll say, oh, it's the branches in the trees, they move and that creates wind, right? So they get the causal relationship backwards. And it's because their understanding of the world and into the physics is not that great. Right. I mean, these are like, you know, four or five year old kids.

You know, it gets better and then you understand that this you can be right, but there are many things which we can because of our common sense understanding of things, what people call common sense. And without understanding of physics, we can there's a lot of stuff that we can figure out causality. Even with diseases. We can figure out what's not causing what. Often there's a lot of mystery, of course, but the idea is that you should be able to encode that into systems.

That seems unlikely to be able to figure that out themselves.

Well, whenever we can do intervention, but, you know, all of humanity has been completely deluded for millennia, probably since existence about a very, very wrong causal relationship where whatever you can explain, you attribute it to some d'haiti, some divinity. Right. And that's a cop out. That's a way of saying, like, I don't know the cause or, you know, God did it right. So you mentioned Marvin Minsky and the irony of.

You know, maybe causing the first day I went there, you were there in the 90s, you were there in the 80s, of course, in the 90s. What do you think? People lost faith in deep learning in the 90s and founded again a decade later, over a decade later.

Yeah, it was a little different. Yeah, it was just called neural net. So that was. Yeah, they lost interest. I mean, I think I would put that around 1995, at least, the Machine Learning Committee, there was always a neural net community, but it became kind of disconnected from sort of mainstream machine learning, if you want. There were. It was basically electrical engineering that it kept at it. And computer science gave up, gave up on neural nets.

I don't I don't know, you know, I was too close to it to really. Sort of analyze it with sort of a, uh, unbiased eye if you want, but I would I would we would make a few guesses. So the first one is at the time known as. Were it was very hard to make them work in the sense that you would, you know, implement Backroad in your favorite language, and that language was not Python, it was not Matlab, it was not any of those things because they didn't exist.

Right. To write it in for Tennessee or something like this. Right. Um, so you would, uh, experiment with it. You would probably make some very basic mistakes, like, you know, barely initialize your weights, make the network too small because you read in the textbook, you know, you don't want to any parameters. Right. And of course, you know, and you would train on or because you didn't have any other dataset to try it on.

And of course, you know, it works half the time. So you'd say you give up, what are you to with batch gradient, which you know isn't sufficient?

So there was a lot of bag of tricks that you had to know to make those things work or you had to reinvent. And a lot of people just didn't and they just couldn't make it work. Um, so that's one thing the investment in software platform to be able to kind of, you know, display things, figure out why things don't work, and to get a good intuition for how to get them to work, have enough flexibility so you can create network architectures accomplish on that and stuff like that.

It was hard when you had to write everything from scratch and again, you didn't have any personal Matlab or anything, right? I read that. Sorry to interrupt, but I read he wrote it in Lisp, the first versions of Lenat.

But the coalition that works, which, by the way, one of my favorite languages is that's how I knew you were a legit Turing Award, whatever you programmed, unless that's still my favorite language.

But it's not that we programmed in Lisp is that we had to write a lisp interpreter, OK, because it's not. That's right. That's the one that existed. So we wrote a this interpreter that we hooked up to, you know, a backhand library that we wrote also for sort of neuronal computation. And then after a few years around nineteen ninety one, we invented this idea of basically having modules that know how to move forward and back, propagate gradients and then interconnecting those modules in a graph.

Lubutu had made proposals on this about this in the late eighties and and we were able to implement this using this system. Eventually we wanted to use that system to make. Build production code for character recognition. Bell Labs, so we actually wrote a compiler for that interpreter so that Chesimard, who is not Microsoft, kind of did the bulk of it with Leon and me and and so we could write our system in Lisp and then compile to see. And then we will have a self-contained computer system that could kind of do the entire thing.

Neither party, Senator Santorum can do this today yet. OK, it's coming. Yeah, uh, I mean, there's something like that in White called, you know, script. And so, you know, we had to write all this material over to our computer to invest a huge amount of effort to do this. And not everybody, if you don't completely believe in the concept, you're not going to invest the time to do this right now at the time.

Also, you know what? Today, this would turn into torture by torture or whatever. We put it in open source. Everybody would use it and realize it's good. Back before nineteen ninety five working at AT&T, there's no way the lawyers would let you release anything on open source. Of this nature and so we could not distribute or code, really.

And that point started to go in a million tangents, but on that point, I also read that there was some almost pet like a patent on coalition, John, that works because I was so that first of all, I mean, just who actually that ran out the thankfully twenty seven in 2007.

What can we can we just talk about that first. I know you're on Facebook, but you also at NYU and. What does it mean to patent ideas like these software ideas, essentially, or what are mathematical ideas or what are they?

OK, so they're not mathematical ideas. There are algorithms. And there was a period where the US Patent Office would allow the patent of software as long as it was embodied. The Europeans are very different.

They don't they don't quite accept that they have a different concept.

But, you know. I don't know, I mean, I never actually strongly believed in this, but I don't believe in this kind of patent. Facebook basically does not believe in this kind of patent. Google fights patterns because they've been burned with Apple. And so now they do this for defensive purposes, but usually they say we're not going to sue you if you infringe Facebook as, uh, as a similar policy to say, you know, we have a patent on certain things for defensive purposes.

We're not going to sue if you infringe unless you sue us.

Um, so the the industry does not believe in patents. They are. They are because of, you know, the legal landscape and and various things, but.

But I don't really believe in penance for this kind of stuff. So that's that's a great thing. So I'll tell you a story. Yeah. So what happens was the the first the first pattern of our competition, it was about kind of the early version of commissionaire that didn't have separate putting layers. It had, you know, coalitional layers was tried more than one, if you want to.

Right.

And then there was a second one on commericial nets with separate peeling layers, uh, Trenwith back and there were five in 89 and 99 years, something like this at the time, the life of a pattern was 17 years.

So what will happen over the next few years is that we started developing character recognition technology around, uh, nets and in 1994.

A check reading system was deployed in ATM machines in 1995, it was for a large check reading machines in back offices, etc., and those systems were.

Developed by an engineering group that we were collaborating with AT&T and they were commercialized by NCR, which at the time was a subsidiary of AT&T. Now AT&T split up in 1996 and 1995, early 1996.

And the lawyers just looked at all the patents and they distributed the patents among the various companies. They gave the the competition that patent to NCR because they were actually selling products that used it. But nobody had any idea what the competition that was. Yeah.

OK, so between 1996 and 2007. So there's a whole period until 2002 where I didn't actually work on a mission on your conscience to resume working on this around 2002.

And between 2002 and 2007, I was working on them, crossing my fingers that nobody would notice and nobody noticed.

Yeah. And I and I hope that this kind of somewhat, as you said, lawyers, a side relative openness of the community now will continue.

It accelerates the entire progress of the industry and. You know, the problems that. Facebook and Google and others are facing today is not whether Facebook or Google or Microsoft or IBM or whoever is ahead of the other is that we don't have the technology to build the things we want to build, want to build intelligent virtual systems that have common sense. We don't have a monopoly on good ideas for this. We don't believe we do. Maybe others believe they do, but we don't.

OK, if a startup tells you they have the secret to human level intelligence and common sense, don't believe them. They don't. And it's going to take the entire work of the world research community for a while to get to the point where you can go off and in each other's company is going to start to build things on this. We're not there yet.

So absolutely. And this calls to the the gap between the space of ideas and the rigorous testing of those ideas of practical application that you often speak to.

You've written advice saying don't get fooled by people who claim to have a solution to artificial general intelligence, who claim to have any system that works just like the human brain or who claim to have figured out how the brain works. Ask them what the error rate they get an amnesty or imagine that.

So this is a little dated, by the way.

I mean, five years. Who's counting? OK, but I think your opinion is earnest. And imagine that. Yes, maybe data. There may be new benchmarks. Right. But I think that philosophy is one you still and somewhat hold that benchmarks and the practical testing, the practical application is where you really get to test the ideas.

Well, they may not be completely practical. Like, for example, you know, it could be a toy data set, but it has to be some sort of task that the community as a whole has accepted as some sort of standard kind of benchmark, if you want.

It does need to be real.

So, for example, many years ago here at fair people, you know, just in western Anbar and a few others proposed the tasks which were kind of a toy problem, to test the ability of machines to reason actually to access working memory and things like this. And it was very useful, even though it wasn't a real test. And this is kind of halfway a real task.

So, you know, toy problems can be very useful. It's just that. I was very struck by the fact that a lot of people, particularly people with money to invest, would be fooled by people telling them, oh, we have the algorithm of the cortex and you should give us 50 million.

Yes, absolutely. So there's a lot of people who. Who try to take advantage of the hype for business reasons and so on, but let me sort of talk to this idea.

That new ideas, the ideas that pushed the field forward may not yet have a benchmark or it may be very difficult to establish a benchmark, I agree that's part of the process, establishing benchmarks as part of a process. So what are your thoughts about?

So we have these benchmarks on around stuff we can do with images from classification to captioning to just every kind of information could pull off from images and the surface level. There's audio data set, there's some video. What can we start? Natural language. What kind of stuff? What kind of benchmarks do you see that start creeping on to more? Something like intelligence, like reasoning. Like maybe you don't like the term, but ajai echoes of that kind of.

Yeah. Formulation.

A lot of people are working on interactive environments in which you can you can train and test intelligence system. So so, for example, you know, it's the classical paradigm of supervised training is that you you have a dataset, you partition it into a training said violations that test that. And there's a clear protocol. Right.

But what if the that assumes that the samples are statistically independent? You can exchange the order in which you see them doesn't shouldn't matter or things like that. But what if the answer you give determines the next sample you see, which is the case, for example, in robotics? Right. You robot does something and then it gets exposed to a new room and depending on where it goes, the room would be different. So that's the that creates the exploration problem.

The what if. The samples, so that creates also a dependency between samples to you, if you move, if you can only move in space, the next sample you're going to see is going to be probably in the same building, most likely.

So, so. So all the assumptions about the validity of this training center set up. What is its break? Whatever machine take an connection that has an influence in the in the world and it's what is going to see.

So people are setting up artificial environments where where that takes place. Right. The robot runs around a 3D model of the house and can interact with objects and things like that. So you do robotics space simulation. You have those, you know.

Opening a gym type thing or Mejico kind of, uh, simulated, uh, robots, and you have games, you know, things like that. So that's where the field is going. You really this kind of environment.

Now, back to the question of like, I don't like the term adjei. Because it implies that human intelligence is general and human intelligence is nothing like general, it's very, very specialized we think is general. We'd like to think of ourselves as having drones, agents. We don't we're very specialized. We're only slightly more general.

Why does it feel general? So you kind of the term general? I think what's impressive about humans is the ability to learn, to talk about learning, to learn. And just so many different domains is perhaps not arbitrarily general, but just you can learn in many domains and integrate that knowledge somehow. OK, now, so let me take a very specific example. Yes. It's not an example. It's more like a quasi mathematical demonstration. So you have about one million fibers coming out of one of your eyes.

OK, two million total. But let's talk about just one of them. It's one million nerve fibers. Your typical nerve. Let's imagine that they are binary so they can be active or inactive. So the input to your visual cortex is one million bits.

Now, the connected to your brain in a particular way on your brain has connections that are kind of a little bit like a corporation, that the kind of local, you know, in space and things like this.

Now, imagine I play a trick on you. It's a pretty nasty trick, admit I, I cut your optical nerve and I put a device that makes a random perturbation of a permutation of all the nerve fibers. So now what comes to your to your brain is a fixed but random permutation of all the pixels. There's no way in hell that your visual cortex, even if I do this to you in infancy, will actually envision vision to the same level of quality that you can.

Got it. And you're saying there's no way you will learn that?

No, because now two pixels that anybody in the world will end up in very different places in your visual cortex and your neurons there have no connections with each other because they only connected locally.

So this whole our entire the hardware is built in many ways to support the legality of the real world. Yeah. Yes. That specialization. Yeah. But it's still pretty damn impressive. So it's not perfect generalization. It's not even close. No, it's, it's, it's, it's not it's not even close. It's not at all. Yes. It's awful. So how many boolean functions. So let's imagine you want to train your visual system to recognize particular patterns of those one million bits.

OK, so that's a boolean function, right? Either the pattern is here or not. Here is a two to a classification with one million binary inputs. How many such brilliant functions are there? OK, if you have two to the one million. Combinations of inputs for each of those, you have an output bit and so you have two to the two to the one million boolean functions of this type. OK, which is an unimaginably large number. How many of those functions can actually be computed by your general cortex?

And the answer is a tiny, tiny, tiny, tiny, tiny, tiny sliver, like an enormously tiny sliver. Yeah, yeah. So we are ridiculously specialized.

OK, but yeah, OK, that's an argument against the word general, I think there's a there's I agree with your intuition, but I'm not sure it's it seems that the brain is impressively.

Capable of adjusting to things, so it's because we can't imagine tasks that are outside of our comprehension right now, so we think we think we are general because we're general of all the things that we can apprehend.

So, yeah, but there is a huge world out there of things that we have no idea. We call that heat, by the way. Heat, heat.

So, uh, a geophysicist call that or they call it entropy, which is you you have a. Thing full of gas, right, closed system for gas, right close on the coast, it has know pressure, it has temperature has you know, and you can write equations, political, you know, things like that.

Right. When you reduce the volume, the temperature goes up, the pressure goes up. You know, things like that. Right. For perfect gas. At least those are the things you can know about the system. And it's a tiny, tiny number of bits compared to the complete information of the state of the entire system, because the state of the entire system will give you the position and momentum of every every molecule of the gas.

And what you don't know about it is the entropy and you interpret it as heat, the energy contained in that thing is is what we call heat now. It's very possible that, in fact, there is some very strong structure in how those molecules are moving is just that they are in a way that we are just not wired to perceive that we're ignorant to it.

And there's an infinite amount of things we're not wired to perceive.

Right. That's a nice way to put it.

Word, General, to all the things we can imagine, which is a very tiny subset of all things that are possible.

So it's like coming off of complexity or the most exciting of complexity, you know, every.

Bittering or integer is random, except for all the ones that you can actually write down. Yeah, OK, so beautifully. But, you know, so we can just call it artificial intelligence.

We don't need to have a general or human level intelligence system. You know, you start any time you touch human, it gets it gets interesting because, you know, just because we attach ourselves to human and it's difficult to define what human intelligence is.

Yeah. Nevertheless, my definition is maybe. Damn impressive intelligence, OK? Damn impressive demonstration of intelligence, whatever, and so on that topic, most successes in deep learning have been in supervised learning. What? Is your view on unsupervised learning, is there a hope to reduce involvement of human input and still have successful systems that are have practical use?

Yeah, I mean, there's definitely a hope. It's more than a hope, actually. It's there's, you know, mounting evidence for it. And that's basically all I do. Like, the only thing I'm interested in at the moment is I called itself supervised running, not unsupervised, because unsupervised running is is a loaded term. People who know something about machinery tell you, so you're doing clustering or PCR, which is not the case and the way public we know when you say unsurprisingly, oh my God, your machines are going to run by themselves and without supervision, you know, they see as where's the parents?

Yeah. So so I have to provide training because in fact, the underlying algorithms that I used are the same algorithms as the supervised running algorithms, except that what we train them to do is not predict a particular set of variables like the category of of an image. And not to predict a set of variables that have been provided by human labor laws, but what you're trying to machine to do is basically reconstruct a piece of its input that it's being.

There's being maxed out, essentially, you can think of it this way, right? So, sure, a piece of video to machine and ask it to predict what's going to happen next. And of course, after a while, you can. Sure. What happens and the machine will kind of training itself to do better at that task. You can do like all the latest most successful models in natural language processing use self supervised running. You know, sort of bird style systems, for example, right, you show with the window of a dozen words on a test corpus, you take out 15 percent of the words and then you train the machine to predict the words that are missing that September is running.

It's not predicting the future is just predicting things in middle. But you could have to predict the future. That's what language models do.

So you construct so unsupervised where you construct a model of language, do you think or video or the physical world or whatever.

Right. How far do you think that can take us? Do you think very far it understands anything to some level.

It has a shallow understanding of, uh. Of tax, but it needs to I mean, to have kind of a true human level intelligence that you need to ground language in reality. So some people are attempting to do this right. Having systems that can have some visual representation of what what is being talked about, which is one reason you need interactive environments. Actually, this is like a huge technical problem that is not solved. And that explains why software provisioning works in the context of natural language.

That does not work in the context of it is not well in the context of image recognition and, uh, video, although it's making progress quickly. And the reason that reason is the fact that. It's much easier to represent uncertainty and the prediction in the context of natural language than it is in the context of things like video and images. So, for example, if I ask you to predict what words are missing in your 15 percent of the words that are taken out.

The possibilities are small. I mean, it's small, right, there is one hundred thousand words in the in the lexicon and what the machine spits out is a big probability vector, right? It's a bunch of numbers between zero and one. That's up to one.

And we know how to do how to do this with computers. So they are representing uncertainty and the prediction is relatively easy. And that's, in my opinion, why those techniques work for an LP for images. If you ask if you block a piece of an image and you see a system reconstruct that piece of the image. There are many possible answers. There are all perfectly legit, right, and how do you represent that, this set of possible answers?

You can't train a system to make one prediction. You can train a neural net to say, here it is. That's the image because there's a whole set of things that are compatible with it. So how do you get the machine to represent not a single output, but all set of outputs? And, you know, similarly with, uh, video production, there's a lot of things that can happen in the future. Video, you're looking at me right now.

I'm not moving my head very much. But, you know, I might turn my head to the left or to the right. Right. If you don't have a system that can predict this. And you train, it will square to kind of minimize the error with the prediction and what I'm doing, what you get is a blurry image of myself in all possible future positions that I might be in, which is not a good prediction. But so there might be other ways to do the self supervision, right.

For visual scenes. Like what if I mean, if I knew, I wouldn't tell you publish it first. I don't know.

I know there might be. So I mean, these are kind of, uh, there might be artificial ways of like self play in games to where you can simulate part of the environment. You can.

Oh, that doesn't solve the problem is just a way of generating the. But because you have more of a nightmare, maybe you can control, yes, a way to generate data and that's right. And because you can do huge amounts of data generation. That doesn't you, right? Well, it's it's a creeps up on the problem from the side of data, and you don't think that's the right way to keep it doesn't solve this problem of handling uncertainty in the world.

Right. So if you if you have a machine learn a predictive model of the world in a game that is deterministic or quasi deterministic, it's easy.

Right. Just give a few frames of the game to accommodate. Put a bunch of layers and then have the game, generates the next few friends and and if the game is deterministic, it works fine, and that includes feeding the system with the action that your little character is is going to take.

The problem comes from the fact that the real world ends and most games are not entirely predictable so that you get those very predictions and you can't do planning with very predictions. So if you have a perfect model of the world. You can in your head run this model with a hypothesis for a sequence of actions, are you going to predict the outcome of that sequence of actions? But if your model is imperfect, how can you plan there quickly explodes? What are your thoughts on the extension of this?

Which topic I'm super excited about. It's connected to something you were talking about in terms of robotics is active learning, so as opposed to sort of completely unsupervised, self supervised learning. You ask the system for human help. For selecting parts you want to next. So if you think about a robot exploring a space or baby exploring space or a system exploring a data set every once in a while asking for human input, you see value in that kind of work.

I don't see transformative value. It's going to make things that we can already do. More efficient or there will run more efficiently, but it's not going to make machines sort of significantly more intelligent. I think. And and by the way, there is no opposition, there's no conflict between self supervisor, any reinforcement training supervisor or imitation or anything or active running. I see self supervision as a as a preliminary to all of the above. Yes. So.

The example I use very often is how is it that so if you use. Classical reinforcement running deep reinforcement learning if you want. The best methods today. So-called moral free reinforcement, wanting to learn to play Atari games, take about 80 hours of training to reach the level that any human can reach in about 15 minutes, they get better than humans, but it takes a long time. Uh, lwr, OK, the, you know, are your vehicles and his teams, the system to play to to play Starcraft plays a single map, a single.

Type of player and can reach. Better than human level with about the equivalent of two hundred years of training playing against itself, it's 200 years, right? It's not something that no human can could ever do.

I'm not sure what to take away from that.

OK, now take those algorithms the best. Our algorithms we have today to train a car to drive itself.

It would probably have to drive millions of hours, you will have to kill thousands of pedestrians, it will have to run into thousands of trees. It will have to run off cliffs and had to run the cliff multiple times before it figures out that it's a bad idea, first of all.

And second of all, different figures about how not to do it. And so, I mean, this type of running obviously does not reflect the kind of learning that animals and humans do. There is something missing that's really, really important there. And my hypothesis, which have been advocating for like five years now is that we have predictive models of the world. That include the ability to predict under uncertainty and what allows us to. Not run off a cliff where we learn to drive.

Most of us can learn to drive in about 20 or 30 hours of training without ever crashing, because any accident, if we drive next to a cliff, we know that if we turn the wheel to the right, the car is going to run off the cliff. And nothing good is going to come out of this because we have a pretty good model of intuitive physics that tells us, you know, the car is going to fall. We know we know about gravity.

Babies run this around the age of eight or nine months that objects don't float, they fall. And we have a pretty good idea of the effect of turning the wheel of the car. And we know we need to stay on the road. So there is a lot of things that we bring to the table, which is basically or predictive model of the world.

And that model allows us to not do stupid things and to basically stay within the context of things we need to do. We still face unpredictable situations and that's how we learn. But that allows us to learn really, really, really quickly. So that's called model based reinforcement learning. There's some imitation and training because we have a driving instructor that tells us occasionally what to do, but most of the learning is learning the model, learning physics that we've done since we were babies.

That's where almost all along and the physics is somewhat transferable from. It's transferable from synthesizing stupid things are the same everywhere. Yeah.

I mean, if you you know, you have the expanse of the world, you don't need to be a particularly from a particularly intelligent species to know that if you spill water from a container that, you know, the rest is going to get wet and you might get wet. So, you know, cats know this right here. So the main problem we need to solve is how do we learn models of the world? That's and that's what I'm interested in.

That's what it's all about.

If you were to try to construct a benchmark for. But let's look at amnesty. A love that is that do you think it's useful, interesting, possible to perform well on amnesty with just one example of each digit? And how would we.

Solve that problem. The answer is probably yes, the question is what other type of learning are you allowed to do? So if you're allowed to do is train on some gigantic data set of labeled digit. That's called transfer learning. And we know that works. OK, we do this at Facebook, like in production, right? We we train not completionist to predict hashtags that people type on Instagram and we train on billions of images, literally billions. And and then we chop up the Lazlo and fine-tune on whatever task we want.

That works really well. You can beat the image that record with us. We actually opensource the whole thing like a few weeks ago. Yeah, that's still pretty cool, but yeah. So what in your world would be impressive and what's useful and impressive, what kind of transfer learning would be used for Wikipedia, that kind of thing.

No, no. I don't think transuranic is really where we should focus. We should try to do. You have a kind of scenario for a benchmark where you have only one data. And you can and it's a very large number of unlabeled data. It could be video clips. It could be what you do find prediction. It could be images. You could choose to mask a piece of it. Could be whatever, but they're unlabeled and you're not allowed to label them, so you do some training on this and then.

You train on a particular supervised. Teske, you mentioned it honest and you measure how your. Tester decrease over schnorrer decreases as you increase the number of Liebl trending samples. OK, and.

And what would you like to see is, is that, you know, your area decreases much faster than if you're trying from scratch from Runaway's so that to reach the same level of performance and a completely supervised, purely supervised system would reach you would need with your samples.

So that's the crucial question, because it will answer the question to like, you know, people are interested in medical imaging analysis.

OK, you know, if I want to get a particular, uh, level of, uh, of error rate for this task, I know I need, um, a million samples. Can I do, you know, some supervisory training to reduce this to about one hundred or something?

And you think the answer there is some supervised retraining.

Yeah, some form. Some some form of it telling you active learning where you disagree, you know, it's not useless, it's just not going to lead to a quantum leap. It's just going to make things that we already do. So you're way smarter than me. I just disagree with you, but I don't have anything to back that. It's just intuition. So I worked with a lot of large scale datasets and there's something there might be magic in active learning, but OK, at least I said publicly, at least I'm being an idiot publicly.

OK, it's not being an idiot.

It's working with the data you have. I mean I mean, certainly people are doing things like, OK, I have three thousand hours of, you know, imitation on you for driving a car. But most of those are incredibly boring. What I like is select, you know, 10 percent of them that are kind of the most informative and which shows that I would probably reach the same.

So it's a weak form of, uh, of running, if you want.

Yes. But there might be a much stronger version. Yeah, that's right. That's what and that's an option that exists. The question is how much stronger can get? Elon Musk is confident. Talk to him recently. He's confident that large scale data and deep learning can solve the autonomous driving problem. What are your thoughts on the limited possibilities of deep learning in the space? It's obviously part of the solution. I mean, I don't think we'll ever have a safe driving system or it is not in the foreseeable future that does not use the.

And you put it this way.

Now, how much of it so in the history of sort of engineering. Particularly sort of sort of like systems is generally a first phase where everything is built by hand and there is a second phase and that was the case for autonomous driving, you know, 20, 30 years ago. There's a phase where there's a little bit of learning is used, but there's a lot of engineering that's involved in kind of, you know, taking care of corner cases and and putting limits, etc.

, because the learning system is not perfect. And then as technology progresses. We end up relying more and more learning, that's the history of character recognition, so history is speech recognition. Computer vision, natural language processing. And I think the same is going to happen with, uh, with autonomous driving that currently the the, uh, the methods that are closest to providing some level of autonomy, some decent level of autonomy where you don't expect a driver to kind of do anything is where you constrain the world.

So you only run within one hundred square kilometers of square miles in Phoenix, where the weather is nice and the roads are wide, which is what we're always doing.

You, uh, completely overengineer the car with tons of light arms and sophisticated sensors that are too expensive for consumer cars, but they're fine if you just run a fleet. And you engineer the thing, the hell out of the everything else, you map the entire world so you have a completely new model of everything.

So the only thing that the perception system is to take care of is moving objects and and and construction and sort of things that that weren't in your map.

And you can engineer a good system or stuff, right, so so that's kind of the current approach that's closest to some level of autonomy. But I think eventually the long term solution is going to rely more and more on. Learning and possibly using a combination of self supervised learning and moral based reinforcement or something like that, but ultimately learning will be at not just at the core, but really the fundamental part of the system.

Yeah, it already is. But it become more and more.

What do you think it takes to build a system with human level intelligence? You talked about there a system in the movie, her being way out of reach. I can't reach. This may be outdated as well, but this is still way out of reach.

What would it take to build her, do you think so?

I can tell you the first two obstacles that we have to clear, but I don't know how many obstacles there are after this. So the image I usually use is that there is a bunch of mountains that we have to climb and we can see the first one. We don't know if there are 50 mountains behind it or not.

And this might be a good sort of metaphor for why Iris, which was in the past, had been overly optimistic about the result of, uh, you know, for example, um, uh, New Zealand.

Simon Wright wrote the general problem solver, and they call it a general problem. Problem solved.

And of course, the you realize is that all the problems you want to solve are exponential. And so you can actually use it for anything useful. But, you know, yes, all you see is the first peak.

So what are the first couple of peaks for her?

So the first peak, which is precisely what I'm working on, is self supervised running. How do we get machines to run models of the world by observation? Kind of like babies and like young animals. So we've been working with. You know, cognitive scientists. So Emmanuel Dipu, who is it fair in Paris is a. Halftime is also a researcher in the. French University, and he he, um, he has this chart that shows at which how many months of life baby humans can learn different concepts and you can you can measure this sort of with.

So things like. Distinguishing any metal objects from an inanimate object, you can you can tell the difference that age two or three months. Whether or not Jack is going to stay stable is going to fall, you know, about four months, you can tell.

You know, there are things like this and then things like gravity, the fact that objects are not supposed to float in the air but are supposed to fall. You run this around the age of eight or nine months. If you look at a lot of eight month old babies, you give them a bunch of toys on their highchair. First thing they do is a toy them on the ground and they look at them because, you know, they're learning about actively learning about gravity, gravity.

OK, so they're not trying to annoy you, but they you know, they need to do the experiment. Right. So how do we get machines to run like babies? Mostly by observation with a little bit of interaction and learning those those those models of the world? Because I think that's really a crucial piece of an intelligent autonomous system. So if you think about the architecture of an intelligent autonomous system, it needs to have a predictive model of the world.

So something that says here is a world of 20, here is a set of all the 20 plus one. If I take this action and it's not a single answer, it can be a creation. Yeah, yeah. Well, but we don't know how to represent distributions interdimensional space, so it's got to be something weaker than that. OK, but with some some representation of uncertainty.

If you have that, then you can do what optimal control theory called model predictive control, which means that you can run your model with a hypothesis for a sequence of action and then see the result. Now, what you need, the other thing you need is some sort of objective that you want to optimize. Am I reaching the goal of grabbing the subject and minimizing energy or whatever? Right. So there is some sort of objective that you have to minimize.

And so in your head, if you have this model, you can figure out the sequence of action that will optimize your objective. And that objective is something that ultimately is rooted in your basal ganglia, at least in the human brain, that that's what it's going to compute your level of contentment or contentment.

I don't know if that's a word unhappiness. OK, yeah, discontentment, this contentment. And so your entire behavior is driven towards kind of minimizing that objective, which is maximizing your contentment computed by your your basal ganglia. And what you have is an objective function, which is basically a predictor of what you are going to get is going to tell you. So you're not going to put your hand on fire because, you know, it's going to you know, it's going to burn and you're going to get hurt.

And you're predicting this because of your model of the world and you're your sort of predictor of the subjective. Right. So if you have those, you have those three components for competence.

You have the hardwired contentment objective, um, computer, if you want a calculator and then you have the three components. One is the objective predictor, which basically predicts your level of contentment. One is the moral of the world. And there's a third module that you mentioned, which is the module that will figure out the best course of action to optimize an objective given your model. OK, yeah, it's a policy policy network or something like that right now, you need all three components to act autonomously, intelligently, and you can be stupid in three different ways.

You can be stupid because your model of the world is wrong. You can be stupid because your objective is not aligned with what you actually want to achieve. OK, and in humans, there would be a psychopath, right?

And then the the third thing, the third way you can be stupid is that you have the right moral, you have the right objective, but you're unable to figure out, of course, of action to ultimate your objective given your model.

OK, um, some people who are in charge of the countries actually have all three that are wrong.

All right. Uh, which countries? I don't know. OK, so, uh, if we think about this agent, if you think about the movie her, you've. Criticized the art project that is Sophea, the robot, and what that project essentially does is uses our natural inclination to anthropomorphize things that look like human and give them more.

Do you think that could be used by A.I. systems like in the movie her?

So do you think that body is needed to create a feeling of intelligence?

Well, if Sophia was just another piece, I would have no problem with it. But it's presented as something else.

Let me on that comment real quick. If creators of software could change something about their marketing or behavior in general. What would it be? What what's just about everything?

I mean. Don't you think? Here's a tough question for me, so I agree with you, so Sophia is not in the general public feels that Sophia can do way more than she actually can. That's right. And the people who created Sophia are not. Honestly, publicly communicating, trying to teach the public what? Here's a tough question, don't you think this the same thing is scientists in industry and research are taking advantage of the same misunderstanding in the public when they create eCompanies or publish stuff?

Some companies, yes. I mean, there is no sense of there's no desire to delude. There's no desire just to kind of overclaim with something is done. Right. We should be Bronny, that has this result on the Internet. You know, it's pretty clear. I mean, it's not not even interesting anymore. But, you know, I don't think there is that I mean, the reviewers are generally not very forgiving of, you know, unsupported claims of this type.

And but there are certainly quite a few startups that have. Had a huge amount of hype around this that I find extremely damaging, and I've been calling it out when I have seen it, so. Yeah, but to go back to your original question, like. The necessity of embodiment, I think I don't think embodiment is necessary, I think grounding is necessary. So I don't think we're going to get machines that really understand language without some level of grounding in the real world.

And it's not clear to me that language is a high of bandwidth medium to communicate how the real world works.

I think it's to grant we're grounding means to me that so there is this classic problem of common sense reasoning, the Winograd's Winograd's schema. Right. And so I tell you, the the trophy doesn't fit in a suitcase because it's too is too big or the trophy doesn't fit in a suitcase because it's too small and it's in the first case refers to the trophy in the second case of the suitcase. And there reason you can figure this out is because you know what, the trophy, the suitcase are one is supposed to fit in the other one.

And, you know, the notion of size and the big object doesn't fit in a small object. And this is a tallest, you know, things like that. Right. So you have this got this knowledge of how the world works of geometry and things like that. Um, I don't believe you can learn everything about the world, but just being told in language how the world works, I think you need some low level perception of the world to be a visual touch or whatever.

But somehow your bandwidth perception of the world, by reading all the world's text, you still may not have enough information. That's right. There's a lot of things that just will never appear in text and that you can't really infer. So I think common sense will emerge from, you know, certainly a lot of language interaction, but also with watching videos or perhaps even interacting in virtual environments and possibly, you know, robot interacting in the real world. But I don't actually believe necessarily that this last one is absolutely necessary.

But I think there's a need for some grounding.

But the final product doesn't necessarily need to be embodied. You're saying he just needs to have an awareness, a grounding, but he needs to know how the world works.

To have, you know, to not be frustrated, frustrating to talk to. And you talked about emotions being important, and that's a whole nother topic.

Well, so, you know, I talked about this, uh, the basal ganglia ganglia as the you know, the thing that you calculates your level of contentment, contentment. And then there is this other module that sort of tries to do a prediction of whether you're going to be content or not. That's the source of some emotion. So here, for example, is in anticipation of bad things that can happen to you. Right.

Uh, you have this inkling that there is some chance that something really bad is going to happen to you, and that creates fear when you know for sure that something bad is going to happen to you. You cannot give up anymore. It's uncertainty that creates fear. So so the punchline is, yes, we're not going to have a ton of intelligence with our emotions.

I think whatever the heck emotions are, as you mentioned, very practical things of fear. But there's a lot of other around.

But there are kind of the results of, you know, drives.

Yeah, there's deeper biological stuff going on. And I've talked to a few folks and this is fascinating stuff that ultimately connects to our toy, to our brain.

If we create an AGI system sorry, in level the human level intelligence system and you get to ask her one question, what would that question be?

You know, I think the the first one will create will probably not be that smart. You like a four year old.

OK, so you would have to ask her a question to know she's not that smart. What's a good question to ask you to sort of wind? And if she answers, oh, it's because the leaves of the tree are moving and creates when she's onto something and she says that's a stupid question, she's really got to know.

And then you tell her actually, you know, here is the real thing. And she says, oh, yeah, that makes sense. So questions that that reveal the ability to do common sense reasoning about the physical world. Yeah.

And some of that will continue for cause. Well, it was a huge honor. Congratulations, Tony Award.

Thank you so much for talking today. Thank you. Appreciate it.