Episode 26 - Pratyush Tiwary: Infusing AI with physics, understanding emergent phenomena, the value of education and thinking

In episode 26, we talk about the origins of Pratyush's passion for statistical mechanics, deeply rooted in his background in material science, and think about how we can promote a profound understanding of statmech theory among people working in computational biophysics. From there, we explore ways of re-introducing physical rigor into modern data-driven approaches, which is the main concern that Pratyush says drives his research agenda. He ends up sharing a ton of interesting points on working with the industry, the value of education and knowledge sharing, or the philosophy of complex and cognitive systems, and ends up with a call for more time for silent thinking, where - he says - most of his original ideas came from.

Transcript

Milosz: 0:00

Hello, I'm glad you're with us for another episode of the Phase Space Invaders podcast. Today in episode 26, I'm talking to Pratyush Tiwary, who is a professor at the Department of Chemistry and Biochemistry at the University of Maryland. As I mentioned in our conversation, Pratyush is someone whose many papers I remember from my early PhD times as they had a deep impact on connecting enhanced sampling methods with kinetics. Since then, he has made some truly creative contributions at the interface between statistical mechanics and artificial intelligence, leading the efforts to integrate more physics into AI approaches in molecular sciences. So we talk about the origins of Pratyush's passion for statistical mechanics, deeply rooted in his background in material science, and think about how we can promote a profound understanding of stat mech theory among people working in computational biophysics. From there, we explore ways of reintroducing physical rigor into modern data-driven approaches, which is the main concern that Pratyush says, drives his research agenda. He ends up sharing a ton of interesting points on working with the industry, the value of education and knowledge sharing, or the philosophy of complex and cognitive systems, and ends up with a call for more time for thinking, where he says most of his original ideas come from. I really hope our discussion can end up inspiring you in tangible ways. So let's get started. So, Pratyush Tiwary, welcome to the podcast.

Pratyush Tiwary: 1:49

Thank you, Miłosz Mm-hmm

Milosz: 1:52

Pratyush, you're certainly not the first podcast guest who made the first big steps in the lab of Michele Parinello, and I kind of vividly remember at least several journal clubs where we'd go through your papers dealing with, you know, making sense of time dependence in dynamics,

Pratyush Tiwary: 2:09

Yeah.

Milosz: 2:09

primarily.

Pratyush Tiwary: 2:10

Yeah.

Milosz: 2:10

since then, your trajectory kept exploring really innovative directions and blending more and more. modern AI concepts with advanced statistical mechanics. I was thinking, was there a particular force driving you there from the very beginning or was it more like a natural consequence of consecutive life choices, so to say?

Pratyush Tiwary: 2:30

Well, that's a very good question and first of all, thank you for having me on your podcast. And so yeah, I was a postdoc with Michele Parinello from 2013 to 2015, and that's when the paper you're referring to was the paper called From Metadynamics to Dynamics. But to get to that paper, you have to go even before that, which is during my PhD, which was at Caltech in Pasadena from 2007 to 2013. Already five, six years ago, before Michele, I was interested in this problem of how to get kinetics. And my PhD was in material science. So back then I was interested in how to get kinetics of slow processes in material science, like, what happens when a vacancy is diffusing or a nanopillar is deforming and things like that. and already back then, it was clear that there are these enhanced sampling methods which can allow you to get kinetics methods such as hyperdynamics coming from someone known as Arthur Voter, who is... not many people in our generation know him, but he was one of the visionaries, I think, at Los Alamos National Lab. So Arthur Voter had a method called hyperdynamics, and I was playing with things like that, coming up with my own methods. I had a method during my PhD, which is five, years before Michele called Sisyphus where the idea was Sisyphus is like this Greek hero who has to carry the stone up a hill, and when he gets to the top of the hill, the stone falls down. So the idea was, let's do something fast to take Sisyphus up to the top of the hill, and then go to accurate molecular dynamics to see how he goes from one valley to another valley. But problem back then itself was how do you describe hills? Hills exist in very high dimensional space when you develop enhanced sampling methods. Of course, we know biology happens, true biology happens in extremely high dimensional space, but our descriptions survive in low dimensional space. So any sampling method, one way or the other, needs you to make choices of low dimensional spaces. I was thinking about this even well before Michele, and it was not clear how to do that. That's when I got to Michele. Michele, of course, was working very much in metadynamics. So I knew that there are ways to combine metadynamics with hyperdynamics. so that's what I started doing. And we pushed metadynamics into more and more kinetics. And then it became even more clear that We have to do, we have to think more carefully about how to pick this approximate reaction coordinate which describes the slow process. And one of the things that I did very early on that was clear to me is that while in the theoretical chemistry community there are very well defined ways of getting the true reaction coordinate, we should not become obsessed with the true reaction coordinate because for enhanced sampling, There is a middle ground. As long as you have some overlap with the true reaction coordinate, you can accelerate your problem. If you don't have overlap, if you're missing out on something critical, then it will not work. And this is the approach. with Michele, I spent two years. It was very productive two years. And we did a lot of stuff where we could use human intuition to invest a little bit of work into reaction coordinate and then speed up. And then we realized that it does two things. Number one, it speeds up the kinetics and it also speeds up the free energy sampling convergence. You know, we know that all these methods, it's not just metadynamics and any other method, is the same thing, you know, and then for my second postdoc, I moved to Bruce Berne in Columbia university in New York. And there I really started very. head on tackling this problem of how to learn the reaction coordinate given limited data. Now, remember that if you have a long empty trajectory coming from Anton simulation on D. E. Shaw, of course, you can learn a good reaction coordinate, but then have solved the problem. So what we want to do is to come up with very limited data and even biased data and then approximate the reaction coordinate and this. So with Bruce, I worked on a method called SGOOP: spectral gap optimization of other parameters. and even then, this is 2015 to 2017. That's when I started in Maryland. And around 2017, what really hit me was this idea of information bottleneck. And, uh, this was similar to autoencoders. That's around when Frank and Vijay Pandey was still in science then. these people were thinking about same things, but I really, uh, And we can talk more about it, but you know, so and that's when I realized that there is a very fertile ground, which is at the interface of machine learning and statistical mechanics, which can really make progress, even in low data regime. And that's the angle I started taking it. So in a certain sense, it looks like I have come far from. What I was doing with Michele, but I would say not really, I'm still doing the same thing, because I think it's such an exciting problem to speed up rare events and get that kinetics. And I don't think it's kinetics versus thermodynamics, they go hand in hand. In fact, you have to understand what are the metastable states a system can take, even before you start talking about kinetics often, you know. So, so everything goes hand in hand, and I'm very lucky that it's still an exciting area where we can continue doing lots of new things. Thanks.

Milosz: 7:10

Right. So it was a seed planted back then. It's interesting that you mentioned this material science background, because I think we very often underappreciate the complexity of really high dimensional spaces and the manifolds that the molecular systems actually trace in those spaces. And maybe your background in this material science where problems are kind of more tractable or more tied to topology or kind of lower dimensional gave you better, do you think it gave you a better understanding, appreciation of, this relationships in dimensionality reduction or the topological concerns around, around these

Pratyush Tiwary: 7:46

what

Milosz: 7:46

topics. Oh,

Pratyush Tiwary: 7:52

like hardcore metallurgy, which is very ancient. The main

Milosz: 7:55

well,

Pratyush Tiwary: 7:56

that I gained out of them was an appreciation for classical thermodynamics and phase diagrams and things like that, you know, and that has been something which I Especially in this era where, for example, a lot of people find StatMech to be sexy and fancy, but people miss out on really fundamental thermodynamics as to where does entropy come from, you know, a molecular, if you think about entropy, It's easy to say S is equal to K log W because it seems like something you can calculate. But then you have to be very careful. S is equal to K log W is really applicable for an isolated system. Anytime we are doing an experiment, it's not an isolated system. We cannot really think about entropy. So classical thermodynamics is what allows you to really classical thermodynamics is something that bridges from material science to chemistry to biology and this is where I was very lucky to have my training in material science and even metallurgy, and that I think that's why also, if you consider Michele, for example, Michele was also trained in solid state physics and things like that, you know, so, so this background of Thermodynamics, solid state physics, quantum mechanics was what carried with me even though I don't do quantum mechanics, that type of appreciation. And thermodynamics really stayed. So, you know, that's something we continue to think about very much.

Milosz: 9:06

right. So I see that a lot of inspiration for your projects comes from combining both, right? It's like finding a new, I don't know, AI architecture and mapping it onto an old problem in thermodynamics. Do you have your favorite resources that you want to recommend to people? So that, for example, you know, people who come from, because I think, I think we might have this problem that a lot of people are coming from related fields from biology, from AI, from, from programming, and they want to learn this hardcore stat mech. And then, you know, it's,

Pratyush Tiwary: 9:34

Yeah, so I

Milosz: 9:34

of where do you start?

Pratyush Tiwary: 9:36

uh, the best book I can think of, ways. If you want to just study, first of all, you have to read books. There is no other way. You have to read books and you have to solve problems. and you have to make an honest effort in solving the problems. If you don't solve the problems, you don't learn anything, right? so the best book I can recommend, which does everything in the same roof, is a book by Scott Shell. If you just look up his research I think he has only one book, which combines classical thermodynamics with statistical mechanics. It's a really wonderful book. So that's one

Milosz: 10:03

Okay. Right,

Pratyush Tiwary: 10:06

the most important book I think on thermodynamics. It's a really old book by Callan, C A L L E N, Callan. Callan has a book on thermodynamics. It's almost like a book on philosophy if you first read it, and that's the book why I think students need to read first is Before getting into statistical mechanics, there is no shortage of books on StatMech. So I won't, for StatMech, you can pick up any book you want to pick. You can pick up David Chandler, you can pick up McQuarrie. But what's happening is that classical thermodynamics is getting forgotten. And that's, that's, that's a dangerous trend, I think.

Milosz: 10:38

right. it was always tied very closely to, to philosophy and things like how do you coarse grain our perceptual space and

Pratyush Tiwary: 10:45

Precisely.

Milosz: 10:46

those long debates in the field. And, uh, well, it even drove people to suicide, if we remember Boltzmann. So, yeah, it's an interesting take that it should actually start from the philosophy of it. So, yeah, you already mentioned those several key concepts, like commitors, like transfer operators, information bottlenecks. Do you think we, we are starting to, again, with the people you mentioned, with Vijay Pandey, uh, with Frank Noe. Do you think we have a kind of new renaissance of stat mech in the context of or biological systems that starting to create our own like vocabulary and tool sets? Do you see it kind of coalescing in this way these days?

Pratyush Tiwary: 11:26

it is coalescing and it's good, but I worry in general about machine learning is people who have not taken the time to really go through things carefully, jumping into things. And, you know, maybe this just reflects that I myself, I'm getting old, but I think people need to

Milosz: 11:42

I

Pratyush Tiwary: 11:42

time in learning things with rigor. And I think as machine learning is picking up more and more, the rigor is definitely starting to disappear. And often people are not even realizing what is the rigor from the stat mech and thermodynamics perspective that they are completely missing out on and it's becoming a bit hopeless that they don't even realize are the laws that are becoming violated and things like that. So I think it's wonderful that people from different areas. are taking interest in StatMech and thermodynamics, but without the proper training, and it's not something which cannot be solved. People just need to sit down and appreciate these things, and people are not willing to do that because the field is moving so fast. If you're just chasing the next ICLR publication or NeurIPS publication and things like that, you try to rush things, and I think that's a dangerous trend. So while I'm excited about people from different things getting in and StatMech having a renaissance, but I think a part of it is becoming, misplaced.

Milosz: 12:41

think, yeah, I think that's a very timely worry because we all play with all those new algorithms for generating ensembles and using AI to do the formerly heavy lifting off exactly enhanced sampling and so on. But we're kind of losing and I don't even know if we have good ways of bringing this back. As of now, we're kind of losing the connection to free energies and to like measurable

Pratyush Tiwary: 13:05

Yeah, yeah.

Milosz: 13:06

thermodynamics. So

Pratyush Tiwary: 13:11

which seems to reproduce dynamical ensembles or even static ensembles, the first question you need to ask is how do my fluctuations change with system size? you run MD and you double the number of atoms, central limit theorem dominates, you will get one over square root of N dependence. If you run an NVE ensemble, a microcanonical ensemble, your total energy should stay conserved. You should not see violation of things like this. And I think these things are completely getting ignored. And, uh, you know, so we have to be very careful what there needs to be clear idea of what's happening. And then, of course, the challenge then is if you start thinking of this clear loss, In AI, you can always start getting the right results for wrong, wrong reasons. you might start getting the right type of physics because you kind of force the architecture to follow it. You know, so, so there are bigger questions, I think, and we are going to face them as, as we go forward. And how to do, you know, so we have to get the right results, but also for the right reasons. And I think we have to invest into that.

Milosz: 14:12

agree. I very much like the idea, which I still haven't understood in full, I think that this idea of bringing in the environment in an implicit form to a complex system, right? The things that come from thermodynamic maps and so on. So like to pick a small system and translate it into, well, I think one of the main problems right now with, with those big models is that they treat the environment very implicitly and there's very little appreciation for how much the environment can affect that. The, the structure, right?

Pratyush Tiwary: 14:44

the most important thing. Think about certain, you know, Viruses, they can, I think these are, involved are in the thermometers, for example, and things like that. they don't do anything until they get into a human body. then they realize that the temperature is 37 degrees Celsius. And then suddenly they go boom. So imagine you're trying to come up with the next generation foundation model which is trying to model these things. And if the foundation model is going to work irrespective of the environment, then that is the wrong foundation model. So to me, any biology model that ignores the environment is per construction wrong. Because biology is fundamentally tuned to respond to environment in exactly The right way and environment could include temperature, it could include pressure, it could include pH, and this is not the type of thing we can ignore. We, it has to be really first and foremost, and we know in force fields, we already face these issues. We know that force fields, for example, you know, we know that if a water model is not giving the right melting behavior of water and ice, then it should not be trusted. And we have put in a lot of effort into this. So if you have put in so much effort into force fields in making sure classical force fields, transfer across conditions. Why should we not do that in neural network potentials? Why should we not do that in AI in the most general framework? So I think that has to be, in my own research, that is the first and foremost thing that we think about. How do things transfer across environment? And if they don't, what should we do that they do that?

Milosz: 16:11

Right. well, I think that this is precisely the question, right? How to combine physics and, modern AI, let's say that is, is doing simple, let's say, unweighted ensemble generation. I, I saw, for example, you managed to combine those things with this RAVE algorithm using alpha fold generated structures, right? So there are ways to use both to augment your understanding of the, of the system, that are already kind of bearing fruits,

Pratyush Tiwary: 16:38

Yeah,

Milosz: 16:38

it's probably far.

Pratyush Tiwary: 16:39

methods will always be slow. You know, we are entering a day and age where we want Everything to work in the collab notebook without doing anything. So, you know, so that's the challenge and we want to be fast also But how to do that and maybe we can yeah, so that's always the challenge, you know, how to find that balance between Accuracy and speed and speed is important, but we also want to be accurate.

Milosz: 17:00

Right, right. it's pretty amazing that we can actually generate things, this fast these days, right? So back in the day, it took many, many, even with your algorithm that accelerated things, it took many, many days to even get an idea of. if you started from a static structure, what is the set of structures that might be relevant?

Pratyush Tiwary: 17:21

yeah,

Milosz: 17:21

Now you can have this as a starting point, but as you say, it still leaves us with a lot of, I think that's also a very good place where experimental data comes in, because then you can use experimental data to, to do the selection and scoring, but, um, well, ideally one day you probably would want to get rid of that too.

Pratyush Tiwary: 17:39

are challenges when you are fitting a machine learning model, often the experimental data is about properties which are very emergent. Properties you cannot just get by looking at one structure at a time. You have to collect an ensemble of structures in order to go and connect it with the experiment. So in terms of machine learning, you cannot just add the experimental constraint as something you can back propagate. have to run the machine learning model very, very long amount of time. Collect very, large amount of statistics, then you can compare it with the experiment. So how do you back propagate this type of information? That to me is a fundamental problem, which we need to think about how to solve. And we have done some effort in that, but you know, I think it's far from solved.

Milosz: 18:19

Right. Even for RNA, as you, as you mentioned, things like, I don't know, you're adding magnesium, right? It will generate constraints between parts of the structure that will be probably interacting with the same ion, yeah, I think, I think we need a lot of creativity there to tackle all those, all those questions. And so how do you feel about the, level of readiness in the field? Cause. Again, before, you could, work with things that were pretty established and had their own, environments, like software environments, for example, some of your methods are implemented in Plumed, but now, do you think we're getting to the point where people start actually using those, uh, approaches routinely, for example, about RAVE and, and this method that, that use autoencoders to through the bottle, like generate a latent space. how do you see the main obstacles for the field to, to like accept or implement those new methods?

Pratyush Tiwary: 19:14

So look, if I was working with a group of hundred people people who are full time software developers, it would be easy to come up with versions, which everyone can use. We are always in that challenge that we have to get federal funding for things which are answering a science question. Software is a

Milosz: 19:31

Hmm.

Pratyush Tiwary: 19:31

of it. And we of course want to release open source software, but often it becomes a resource allocation problem. And to how much do I, if I come up with a method, how much effort do I put into just improving that method from a software perspective so everyone can start using it? That's the number one priority, which I should do. Number two, I should keep applying it to complex systems in my own lab in collaboration with experimentalists. And number three, I should do theoretical developments in the method itself, right? Ideal world, I would be able to do all three things at the same time. I should be able to keep improving the method theoretically, I should keep improving the software of the method and I should keep working with my experimental collaborator who can validate the method. In practice, it does not work with that. Like, I'm, uh, I'm currently running a group with nine PhD students and four post docs, and it's not easy to keep a group of this size funded and things like that without going insane myself.

Milosz: 20:22

Hmm. Mm

Pratyush Tiwary: 20:25

it does happen like this. Uh, one of our methods state predictive information bottleneck. If you go to our GitHub, we have nice collabs that people can use. And slowly we are starting to get feedback from other people in the community who are starting to use it. not at the same speed as, for example, AlphaFold2 happened. It takes us time to get there, but slowly and surely, it's getting there. So, you know, it's a resource allocation problem, and we have to be very careful in how to do that. AlphaFoldRay, we are about to release, like, basically like a pip install type version. It might not be exactly in that same level, but something similar where people can use it and start to work with similar things. You know, so the point I'm trying to make here is now we are not just competing with academic labs. We are competing with. Frank who is a good friend, he's working at Microsoft, you know,

Milosz: 21:09

hmm.

Pratyush Tiwary: 21:09

DeepMind and things like that. So how

Milosz: 21:11

Hmm.

Pratyush Tiwary: 21:11

an academic lab even compete with things like that, where you have full time software developers supporting things and things like that, you know. So it's almost an existential question as to how do we do that when we also have publication pressure, right? We have to keep publishing and things like that. So it's a challenge, but I think it's very important to make sure that everyone can use the code. It has to be open source and things like that, but you know, it, will always be codes where you need, it will, it will never become a GUI. Let me put it this way. There will always require some level of understanding of the physics and things like that in order to use the code. And I'm okay with that

Milosz: 21:46

yeah, I think we've had some of those big debates between the utility of academia versus the industry. And I think one of the big concerns is that the industry still has a for profit model, which will prevent them from releasing some of the code, some of the details, some of the database, right?

Pratyush Tiwary: 22:05

Yeah.

Milosz: 22:05

like, we definitely need academia if you want to have these things that are, validated, uh, reproducible and so on. And we know how they work and we can study them. And it's still amazing that we can, uh huh,

Pratyush Tiwary: 22:17

So for example, I'm on the advisory board of Schrodinger and they took infrequent motor dynamics, which is something that I started

Milosz: 22:22

Mm hmm. Mm. Mm.

Pratyush Tiwary: 22:27

how to do it. they have now a code to calculate the kinetics drug residence time. It's in the, public software of Schrodinger, uh, like last year's version, where you can just go and put in. The protein structure, the ligand, you give it the bound structure and less than a day later it gives you the residence time with correct statistics and things like that. So this to me is a really nice model where all of my infrequent metadynamics and all of those codes I am still working on it and many many open source versions but if and you can use it and we will advise you on how to use it. But if a company deploys it in their own particular framework using their own force fields that are using their own data, they can also profit from it. So I think there is a common ground between these things. And I think we need to find it, especially for a variety of reasons. We need to find that. I always tell my students that their first choice should be industry. They should not even think about like leaving academia. They should think about leaving industry. They need to convince me as to why do they not want an industry position. you know, academia has its own complications. While I think there are more industry positions in academia. So if there are more positions in industry than academia, you better have a good reason for why you don't want to pursue that.

Milosz: 23:40

No, I agree there. Absolutely. The years, and it's a new challenge that we as, supervisors, for example, will have, right? To,

Pratyush Tiwary: 23:48

exactly.

Milosz: 23:48

people a different way of training, of approaching, uh, life decisions and so on. Well, you have this summer schools for people who want to learn. I assume it's centered on machine learning, AI, and, uh, well, with biology in mind. Not really, it's just

Pratyush Tiwary: 24:04

Python and

Milosz: 24:05

basic Python and some PyTorch it's really for everyone yes.

Pratyush Tiwary: 24:15

or maybe four summers. And I think already we have been able to reach out to more than 400 students who have never coded before. And they were able to take the first step, at least in Python, due to this and with Colab and things like that, you know, running such a school. 10 years ago would have wasted three hours just to get the installation working, right? And Colab, it's so easy. You just tell them you do this, you do this and it works. So, and it takes the first step. How many of these students are actually using coding in their life now? I don't think I have an answer, but I do know some of them reach out to me and it reduces the barrier for them to get into it, right? So this is something I intent to keep doing for the longest time. I mean, I'm originally from India. I come from one of the poorest parts of India where until 10 years ago, 20 years ago, the literacy rate was 30%. So I feel very passionate about contributing to those parts of India and elsewhere. Also, you know, helping people learn. And it's not just learning how to. Read or write, but how to code also, you know, sort of, I think as it's very important that we invest all of us in not just ourselves, but helping the community, learning how to code and Right now it's coding 30, 40 years from now, it might be some other technology. Maybe it will be how to use Neuralink. I don't know. It's going to be something else. We cannot imagine, right, how it will be.

Milosz: 25:32

Yeah, but it's an amazing driving force, like having our diverse backgrounds. Cause I assume in the West, few people have this experience and few people appreciate what the life changing thing access to education on this level can be. Right. So,

Pratyush Tiwary: 25:46

the most amazing thing, you know. It's the most amazing thing. Education is the great unifier, right? I think it, it

Milosz: 25:52

Oh yes.

Pratyush Tiwary: 25:53

up so many, doors and, uh, sorry, my dog is barking. It's not just at the undergraduate level. I really think a PhD is a wonderful thing if you can do it. But then you have to have that mindset that, You should be very open to an industry job. If you do a good PhD, it just opens up perspectives, which are very, very so different, but at least an undergraduate degree, I think should be, and we are going through times where these things will get questioned more and more. Why do you need a college degree and things like that? You know, as, as long as I don't think we should be living in denial, we should have a clear idea of what we are going to do with the college degree or the PhD degree. you know

Milosz: 26:29

Yeah, this is this longstanding debate. Even the CEO of Nvidia was recently commenting that you should not learn coding. I still think we need to teach people coding, but probably teach it in a different way, right? Because coding itself will change from a kind of mindless well, it was never a mindless procedure, but there will be less and less need for, this kind of mindless coding. Mm hmm. Mm hmm.

Pratyush Tiwary: 26:56

from India. You know, the multiplication table for 17 and things like that. And it's useful. You know, it's useful in so many different ways. And more than that, when I read journal article draft given to me by my students, I can tell how much chat GPT they used or not. It just falls out. It's obvious.

Milosz: 27:16

Oh, yes.

Pratyush Tiwary: 27:17

nothing against it, but there has to be some original thinking. So the problem is just using coding tools. Or writing tools, generative AI is, it's starting to make us intellectually lazy. And that's a very dangerous thing. There is nothing wrong in using it, but the original idea should be yours. Then you can use it to curate it, but you should, I mean, all of these AI tools, including AlphaFold, are hypothesis generators. And if you do not have a physics model in your head, can, you know, rank these hypotheses, I think it's a dangerous game that's going on, right? So that was the philosophy behind AlphaFoldRave also, that things like AlphaFold2 are wonderful hypothesis generators, but you have to have a way of ranking them. Otherwise, why do you trust an AlphaFold structure? You trust it because it looks something that your intuition tells you it should look like. But that's dangerous. What if you're doing something new? You don't have any intuition. Why are you going to trust it? Right? And, and the transferability, we know the PDB is primarily at a certain temperature and things like that. Why should it transfer to other conditions? So I'm, drawing parallels here because that's how I look at things. Same thing with coding. A code, why should it work in some other, you know, for a code to be robust, for a code to really, you know, express the physics that you want it to express. I don't think chat GPT and copilot at are at that level yet, but they are definitely wonderful helpers in what we want to do.

Milosz: 28:41

Yeah, I think there's going to be a lot of philosophical problems that we have to face these days. So exactly, I think the more we learn about AI, the more we also learn about ourselves, right? So it all sometimes looks like looking in the mirror and seeing the ugly version of ourselves.

Pratyush Tiwary: 28:59

Absolutely.

Milosz: 29:00

You had this recent article about the thermodynamic perspective on learning, right? Where you, you kind of try to lay out the kind of enthalpy versus entropy approach to, how systems, machine learning systems, but also we prefer to perceive the world. So it's interesting that, yeah, I think this philosophical, angle is, is going to be really helpful in understanding how we can move forward.

Pratyush Tiwary: 29:26

Yeah. Yeah. Yeah. Yeah. So the way I think about a lot of projects in my group is always try to work with ideas that excite me and some of them become useful right away. Some of them kind of Disappear, lay dormant for a few years and then someone else starts to use them and things like that, you know, happen. So it's in that category and you know, it's what, so the paper that you're talking about, this is in nature communications. This was with my amazing student Shams we looked at the problem of interpreting a model at a philosophical level. What does interpretation mean? Interpretation means taking something which is unfamiliar to you. and expressing it in a language that is more familiar to you.

Milosz: 30:07

Mm hmm. Hmm.

Pratyush Tiwary: 30:11

experience. It's a function of your upbringing. You are Polish. I'm Indian. Sometimes our interpretation, what is familiar to you and to me might be slightly different. Of course, there is common ground also. So that was the one picture, but I think of it more generally as. I imagine 30, 40 years from now, it will be the machine learning model, which will be trying to interpret humans. And maybe they are already doing that. Like, what does this human mean? How to interpret it? and then, so, so we tried to lay it out in a really robust framework. So one of the things that happens in thermodynamics is energy and entropy they go against each other, but there is always a global minimum. And you know that the system can find that free energy minimum. So we took up, we took interpretation and put it in that framework, that there is a global free energy minimum for an interpretation, which is a trade off between,, how complex is your interpretation. So you can come up with a very complex interpretation, it might become useless because it is so complex,

Milosz: 31:12

hmm.

Pratyush Tiwary: 31:13

to find this common ground between how complex it is. And if your interpretation is too simple. It might not be a good interpretation and we showed that there's a mathematical function which can classify this trade off and just like in free energy, if you change the temperature, the free

Milosz: 31:26

Hmm.

Pratyush Tiwary: 31:27

will change, right? So it depends on what is your temperature. So we did all this thing. It was a lot of fun. Let's see how far it goes. Yeah.

Milosz: 31:33

Yeah, that's where you run into the problem of do you understand New York, right?

Pratyush Tiwary: 31:38

Yeah.

Milosz: 31:38

And what does it even mean? Yeah, the thing it has a connection. I know there's this theory from Karl Friston about the theory of mind, which is also free energy principle. Did you draw inspiration from that or?

Pratyush Tiwary: 31:52

No, but I'm aware of that work. I mean, I know what he does. His free energy is a bit slightly different from mine, but I know exactly what you're doing. I think there are parallels. I'm yeah,

Milosz: 32:01

Yeah.

Pratyush Tiwary: 32:01

interested in this work.

Milosz: 32:04

Okay. It's, it's, yeah, it's interesting to interface with those, uh, again, with those deeply philosophical, questions. So, what's your most promising direction now when you think about furthering or advancing this interface between statistical mechanics and, AI, is it this kind of Thermodynamics maps, embedding of, let's say an environment and, and the local effects are,

Pratyush Tiwary: 32:31

thing

Milosz: 32:32

you thinking about different lines?

Pratyush Tiwary: 32:33

yeah, the thing that I'm most excited about is predicting emergent phenomena and emergent phenomena are at the heart of life. They are at the heart of society. I spend a lot of time, worrying about. What is consciousness? What is life itself? And things like that. I'm starting to worry more and more about when does an LLM, when, when can I really tell whether something talking to me is generative AI or human? And it's pretty clear that this boundary I think it's already gotten broken to a large extent. It's quite, If you did, and I think there are statistical tests where it's becoming already impossible to prove, so for example, I'm having this conversation with you. do I know that you're not an AI? This type of stuff is trying to bother me more and more. And I don't think there is a clear answer. So what really distinguishes living systems from non living systems is something I'm starting to worry about more and more. And I have some clues. And one of the basic clues that I have is How systems respond to time, how do they perceive time, number one, and number two, nature of emergent phenomenon that arises in living systems versus what arises in non living systems, you know, if, so for example, you were an LLM, one of the things I would like to do is to replicate copies of you. I have 20 copies of me was talking to each other. I don't think it will look all that different, already the conversation you and I are having in a sense, it's very different from how two LLMs would be talking to each other. don't know how to quantify it, but I think we can see from the nature of the conversation that as soon as two or two people or three people start talking to each other, different natures of discussion start to emerge. So I want to, I really want to do in my. The first step in that direction is to be careful and how do we respond to environment. If you don't have a correct knob on how do we respond to environment, there is no way we will be able to talk about emergent phenomena. And we know this in molecular dynamics, you can take the simplest Lennard Jones potential, it looks so simple, a by r to the power 12 minus b by r to the power 6. You can go even simpler. You can take the Weeks-Chandler-Andersen model, which even cuts down part of the Lennard Jones potential. One of the reasons WCA potential is so beautiful, you can just put things together. And people like Dan, let's go one step even further than WCA potential. Dan Frenkel is the expert on this when he looks at simple hard sphere interactions, where the interaction just drops down. And you put a bunch of hard spheres talking to each other, and you get so much rich physics that comes out

Milosz: 35:32

There are patterns, yes,

Pratyush Tiwary: 35:34

you,

Milosz: 35:34

emerge.

Pratyush Tiwary: 35:35

transition, you change the size of the sphere. I think we are very far from doing these things with hardcore machine learning. We have made some progress, you know, so these are the type of things. And if you, if you really want artificial intelligence to replicate biology, I think we need to think about this right away. And that's what excites me a lot. And I'm going to continue working on this. I'm very lucky that I have funding support to do things like this. So hopefully it will stay that way and I will keep working on that.

Milosz: 36:03

That sounds amazing. Are you thinking of joining, the Santa Fe Institute also? Because

Pratyush Tiwary: 36:08

No, no,

Milosz: 36:08

it sounds like,

Pratyush Tiwary: 36:09

yeah,

Milosz: 36:10

like the SFI things.

Pratyush Tiwary: 36:12

It is an SFI thing. And in fact, I just finished reading Phil Anderson's, biography who was very much involved in it. You know, so people have thought about this, but that was happening. Yeah, it's, it's related to that, but I also want to be very, very practical. I also want to come up with things that can directly allow us to discover new drugs for RNA and things like that. Right. I'm, I'm, I'm very

Milosz: 36:31

Sure. Of

Pratyush Tiwary: 36:35

discovering drugs, and we have lots of active collaboration going on this. RNA, we are also starting to get into disordered proteins. And if you think about disordered proteins, that's where the environmental knob becomes even more important, right? You know, so, and how to do this so that we can come up with the most important drugs which have never existed, which can go and help society, you know, directly, things like that it's not the philosophy, we also want to be very practical, you know, so how

Milosz: 36:59

course.

Pratyush Tiwary: 37:00

tackle the full spectrum.

Milosz: 37:02

Of course, but I also increasingly think that nature has you know, used complexity or employed complexity to solve real problems in evolution. So, and we haven't started tackling this in a really serious way. So I think it's great that more and more people are thinking along these lines of emergent things that happen through understandable mechanisms.

Pratyush Tiwary: 37:22

if you

Milosz: 37:22

Mm hmm.

Pratyush Tiwary: 37:27

500 million years or whatever to solve problems. But now think about it. If AI architecture is also trying to do something which is similar to evolution, but can we be faster than nature? In order to do that, the biggest challenge will be having resource to energy. we solve the energy crisis, if we have opportunities to train AI models, perhaps we can speed up evolution in the lab at a space which is much faster. The amount of things we will keep discovering is just mind blowing. I don't know if that's going to happen. It's also dangerous if perhaps happens. Sometimes I feel like maybe humanity's best bet and worst nightmare is solving the energy crisis, because if you really solve it, it will leave to lifestyle, people will have clean water, things like that. But the AI models that we will start training might just explode. We will start discovering things. which are completely, completely crazy and out there. You know, if you think about things like, of late, I have become very interested in biopolymers and how nature makes use of polymers. And it seems like evolution has maybe explored a On this very high dimensional complex landscape, evolution has taken us to some local minima solutions. There are so many other solutions that AI can help explore maybe we can actually beat evolution in that if we solve the energy crisis. Will we do that? We will see.

Milosz: 38:53

Well, even the space of ideas or, you know, social interactions or social setups is so huge that we can spend a long time exploring this. So yeah, I think there's so many directions you can go for them. Are those the things that you think about when you're running out there?

Pratyush Tiwary: 39:10

Sometimes, yeah, running. I don't think much running. I'm just looking at Nature and looking at people who go next to me. I have so many friends that I see only during my running. I have never spoken to them ever in my life, but I see them during my running. So, you know, it's running the time pauses. I don't think much. I just like have a good time, you know, looking at things and

Milosz: 39:30

I see. That's great too I wanted to bring it up cause you're famous for, for anyone who follows you on social media, you're kind of famous for doing long distance runs and marathons. So again, we had a conversation with Tamar Schlick recently about why scientists run. And yeah, the social aspect was, was very much highlighted there. But I'm thinking for sometimes for deep thinking, what is the best strategy you have to, to go into really appreciating a new idea and thinking profoundly about it? So maybe we need more.

Pratyush Tiwary: 40:11

take pride in saying, I'm so busy, but if you're getting more successful in life, and if you don't have time where you're not busy, where you can just sit and read and think with a piece of paper, and I think, so I fight very hard to keep time like that and it's my highest priority to have every week a certain amount of time where I have no meetings, nothing, where I can just sit and maybe read a paper, maybe not even do that, you know, just sit and think, or maybe code a simple idea, but really nothing to do nothing. That's, I think, quite important for these new ideas to come. And, uh, you know, my, my second postdoc advisor, Bruce Berne, used to do things like, or even Michele, I think Michele used to spend a large amount of time playing Sudoku. And I think he got a lot of ideas just playing Sudoku, you know, things move. So even in running, what happens is for two hours, nothing will happen. and I do long distance running. So I spent a lot of time there, but then suddenly something will click and it might, involve actual science, but also oftentimes it involves training issues. If I can see that one of my trainees is maybe not living up to their potential, I feel guilty. I'm like, it's my job to help them live up to their potential and exceed it. And maybe then it clicks during running. Okay, I have given them the right type of problem. Maybe I'm not mentoring them the right way. And these things just, just click. Yeah. So it's free time, which I think is the most important thing. We need to have more free time to just sit and think.

Milosz: 41:34

Maybe we need summer schools on doing nothing like to relearn this.

Pratyush Tiwary: 41:38

lot of, a lot of summer schools are like that, you know, that is, that's how summer schools were designed to be. Right. Even Gordon Research conferences were designed to be that you have talks in the morning. Then you have four hours of doing nothing and then you come back. But what has happened in modern life, when you have four hours of doing nothing, what do people do in those four hours, go back and take their zoom meetings. you

Milosz: 41:59

yeah. Write a manuscript,

Pratyush Tiwary: 42:01

it's all right, the manuscript. So I think we have become too busy and that's a challenge we need to address carefully. Of course we, if we have to contribute to society and do other things, but if you're not finding times for free thinking, then that's not good.

Milosz: 42:15

Yeah. That's a fair point to, to make. I think we should work on this Well, academia is kind of notorious for this, right? I mean, you mentioned the American society, but

Pratyush Tiwary: 42:24

also. I think it's worse in the industry also, right?

Milosz: 42:27

I think so.

Pratyush Tiwary: 42:27

busy. Yeah, all my industry collaborators, they are the same way. They are running from one Zoom meeting to another Zoom meeting. In fact, I would say, academia has the potential to be better. In the United States, if you get tenure, You have

Milosz: 42:40

Mm-hmm

Pratyush Tiwary: 42:40

life. You don't have to

Milosz: 42:42

Yeah.

Pratyush Tiwary: 42:42

unless you really want to. So we have this freedom. What do we do with this freedom? We get more stressed. for me, the, one of the nicest

Milosz: 42:50

Mm-hmm

Pratyush Tiwary: 42:50

is like it equilibrates me with my stress and everything. I just chill and, find that balance. It helps me find the balance. And, uh, you know, and I, I keep telling my PhD students, That on my postdocs, especially that this time that they have in their life where there is no committee work, there is nothing. It's a beautiful time and they should really use it. I think these habits start very early on, right? You might fall victim to just getting too busy and not doing deep, free thinking already as a PhD postdoc. So you need to do that at the earliest stage to make it a habit that you just. My postdoc advisor, Bruce Berne, he was notorious. He would not even read papers. We just literally sit with a notebook and just scribble things and half the time he would come up with things and then he would come to me and be like, Oh, I just had this idea. Not half the time, 80 percent of the time. And I'd be like, Bruce, it has been done 20 years ago. like, Oh, okay. Then he would go back and think more. And that's wonderful. If you can do that, you know,

Milosz: 43:49

Yeah. That's, I think that's a great message to get out and a very good bottom line for, for this conversation. So Pratyush Tiwary thank you so much for, for the ideas, for sharing your experience and, and

Pratyush Tiwary: 44:03

for having me.

Milosz: 44:06

hope you have a great day.

Pratyush Tiwary: 44:07

You too. Have a nice evening. Bye bye.

Milosz: 44:09

Thank you for listening. See you in the next episode of Face Space Invaders.