E185 - AI - Beyond the Context Window - Johan Thulin

Is AI’s biggest limitation really intelligence—or is it memory? In Episode 185 of the AIAW Podcast, Johan Thulin, Co-Founder and CTO of Aphygo, explores why the future of AI may depend less on bigger models and more on systems that can remember, learn, and accumulate knowledge over time. We discuss the limits of context windows and RAG architectures, why hallucinations may be a symptom of missing state rather than flawed reasoning, and how stateful AI could transform agents from short-term tools into long-term collaborators. Johan also shares his views on model sovereignty, enterprise AI infrastructure, cognitive capital, and the shift from predictive AI to systems capable of maintaining context and meaning across interactions. From the future of human-AI collaboration to AGI, automation, and a world beyond today’s LLMs, this conversation offers a thought-provoking look at what may come next in artificial intelligence.

Follow us on youtube: https://www.youtube.com/@aiawpodcast

Transcript

Ranking Chatbot Memory Today

Anders Arpteg 0:00

Chatbot kind of functionality. I mean Chat GPT then have some type of memory functionality, perhaps not the greatest, and mainly like rag functionality. What about like Claude or Gemini? Have you what do you think about their kind of memory functionality?

Johan Thulin 0:16

Um the most interesting there is probably Claud's memory. Um that is um kind of self-adaptive by rewriting uh its own uh MD files. Um but it's no structure between um between that and it is just basically shunk dumping inside uh the context. So it doesn't understand, it doesn't get a reference of meaning, it it has no temporal authority. Exactly. Uh do they even have memory?

Anders Arpteg 1:14

Um I think in some way they remember something about across the sessions, right? I don't know about the detail how it works.

Johan Thulin 1:21

Yeah, no, it's basically a semantic search rag.

Anders Arpteg 1:26

So if you were to rank like Chat GPT, Gemini, and Claude, who do you think have the best kind of memory functionality today?

Johan Thulin 1:32

Um Open AI, because they they're using your whole uh every previous session you have. The whole history of yeah, complete history. Um while uh Claude doesn't use the session history, it says taking things out of the session and adapting his uh MD files after that, um, but doesn't retain the big context. At least OpenAI can have uh see the big context from from the previous sessions, and you can go back and search again and over again and you can ask it things. But um there's no structure in it, zero structure, right? So, and uh it's um all of that connection uh is within the context window. The context it's it's gonna disappear as soon as the context is gone.

Anders Arpteg 2:31

Yeah, right. Uh that means super interesting, and um you know we have a theme today for this podcast about behind the context window. So I'm very much looking forward to to uh talking more about that and what we can see as I guess the next steps of LLMs in some way, or chatbots in some way, or agents in more general sense, I guess, right? Yeah.

Juan Tulin’s Origin Story

Anders Arpteg 2:52

Well, with that, very welcome here, uh Juan Tulin, you know, co-founder and the CTO of uh AppHIGO. Is that the proper way to say it in English?

Johan Thulin 3:01

Or uh yes, I've I think yeah, I guess.

Anders Arpteg 3:05

Okay, with an FIGO. Perfect. But perhaps you can uh just start by giving a quick introduction to yourself. Who is really Juan Tulin?

Johan Thulin 3:14

Um oh well I'm um mainly see myself as a kind of um hacker, uh cyber pioneer. Um it's basically been my whole life. Um my father was a computer scientist as uh KTH in the 70s um and worked at Commune Data. Um he helped building one of Sweden's first supercomputers, uh Katya at KO. Um so he started reeling me uh at a very very early age. Um I got my first computer when I was six, uh old Atari.

Anders Arpteg 4:01

Uh Atari at that time, yeah. Yes.

Johan Thulin 4:03

You didn't go the Amiga way or the Commodore way, you you went for uh I I had uh everything um basically, but uh somehow the Atari kind of stuck, and I think it was because of the graphics.

Anders Arpteg 4:17

So you wrote your when did you write your first line of code?

Johan Thulin 4:21

Um I started with basic when I was probably nine, I guess. And um yeah, when I was 10, 11, I was already building uh game worlds uh simulated uh um text-based RPGs and uh things like that. So um and and that that has been my path. Um when I was 10 I said oh I'm gonna be a game developer. That's that's the plan.

Anders Arpteg 5:04

Very common uh start of the user science era, so to speak. Yeah. Um did you have any favorite games that you played at that time?

Johan Thulin 5:15

There was many. Uh I remember the first um the game where I started uh I made a huge modification uh for was Civilization One.

Anders Arpteg 5:31

Okay, and then you moved uh how did you continue with your career later and ended up at a FIGO?

Johan Thulin 5:37

Um yeah, it's um it's a long path. Um the game development path had been the main path. Um being involved in many many gaming projects and gaming studios and game studio startups. Um and um that's that that put the seed for AI um naturally, um especially with uh later game bots. Uh I started building game bots for games like uh Counter Strike, um Unreal Tornament, and those in the late 90s, early 2000s, and uh kind of got obsessed with how do I make these bots understand the world? How do I make them adaptive? Can I make them adapt to the player behavior and make them more uh feeling like more humans than NPCs? Um and um I don't that eventually led me to to uh many different gaming projects. Uh for example CCP Games uh worked with uh uh Derry Titus uh Ivo Online and uh Dust.

Anders Arpteg 7:05

And you also moved into audio production, right, at some point, or yeah.

Johan Thulin 7:10

Um so I'm um multi-instrumentalist.

Anders Arpteg 7:16

Ah, you into music as well? Do you play instruments or yes?

Johan Thulin 7:20

Uh started with band, and of course it uh uh pretty quickly escalated into Digital Music, Digaton music production. Um because uh the computer is um almost like an extension of myself. Um and I I started experimenting with synthesizers uh and uh building my own uh EMB uh plugins with virtual instruments, VSTs. Um that um um started um interest in in modulation techniques um um and kind of triggered my deeper uh deeper passion for mathematics and uh that kind of uh work uh which is incredibly useful for what I'm working with now. Can come into that later.

From Game Bots To AI Memory

Anders Arpteg 8:24

And can you just elaborate a bit more? You at some point you started uh thinking about a pygo or a FIGO, so can how did that get started?

Johan Thulin 8:34

Um the memory, um like like I mentioned, uh the the seed started kind of very early um in game development, trying to figure out how to AI could adapt. Um and um but of course when when uh the new generative AI uh wave hit um and I I was part eventually ended up at um Elevatory AI uh is uh distributed AI, right? Did you work for them or do you collaborate in the collaborating is Elevatory is uh distributed uh research uh lab.

Anders Arpteg 9:24

So a number of universities through throughout the world that is collaborating and open public uh collaboration and built models and datasets and yeah, yeah.

Johan Thulin 9:35

Um so and I was part of of um the early work with GPT Neo. I got uh kind of a big part there in in um dataset engineering um because uh previously my father at um he started a company called Cartena uh where we started experimenting with uh the very very early CNNs in 2005.

Anders Arpteg 10:06

Oh that's early.

Johan Thulin 10:07

Yeah, that was very early. Um we're trying to to uh interpret um satellite images into uh vector space or vectors. Um and uh that got me working with uh dataset engineering, and um we have huge libraries of uh satellite images that that we needed to build dataset pipes for. Um and because of that the I had this early experience of uh dataset engineering. Uh so uh I had kind of a very active part in um in the advisory role for building the LSR uh datasets.

Anders Arpteg 10:57

Oh nice. And what year was this approximately?

Johan Thulin 11:01

Uh it was um 2020. Must have been. Uh hard to remember exactly. Uh it's uh it's only six years, but uh years ago, but it's like twice, yeah.

Anders Arpteg 11:20

Okay, so you worked with them and contributed to a number of open source projects, I guess, during that time. But if you were to come to the point when when you felt like now we need to start something new here, and can you just elaborate a bit more? What you know made you start um at high growth?

Johan Thulin 11:40

Yeah, so the my my real work with memory started uh about two years ago when I I felt like um the the early rags came about and I wasn't really um really really impressed by them and and I I saw some so many different problems with uh LMMs um especially how they um they're not able to adapt and um the self-attention is um basically railroading through the context. Uh it's impossible to make an AI change directions once it's uh settled on a on a certain path.

Anders Arpteg 12:28

Yeah, it would be fun to hear your thoughts about like auto-regressive kind of nature of LLMs and if that's a good or bad thing. But perhaps we can talk more about that later. Yeah. Okay, so you saw the the early problems with the rag, the retrieval augmented generation part where you basically index up a set of documents or the data you want to have and then add it to the context window, which is the you know, I guess a simple way of adding some kind of memory to the early LMs, right? And and okay, so that was two years ago. And uh what made you think, okay, we need to start a company around this?

Johan Thulin 13:05

Uh so we didn't actually. Um, this was my private uh laboratory work I was working on. Um, and then I met the others in Afghy for a completely different project we were trying to work uh uh with trying to solve uh the data problem with uh European regulations and the EU Act. Okay. Um we saw that where is a huge opportunity um to work with uh we have so much data everywhere uh sitting uh unused, um, especially media data. Uh so many media houses sit on huge libraries and they don't know what to do with it. So that was kind of the first idea of with AFI trying to uh build something um for that and and um at stability AI and my previous work at um some labs I've done. Uh we've been working a lot with augmenting augmenting uh data, um building up uh improving datasets, increase the the size. Right. Um so that was our plan to try and get studio studios.

Anders Arpteg 14:29

That's for your own purposes, or was it from some kustumer, or what what was the purpose you're trying to do data augmentation at that time?

Johan Thulin 14:38

To solve the the lack of data um in Europe after after the EU data act. Uh we didn't have a single legal dataset really. Ever dataset that's out there is polluted in some kind of way.

Anders Arpteg 14:58

Um was it for sam project, sampaning, some kustumer. For what purpose were you trying to find data at that time? Um for um for for the EU to be able to continue developing open source point of views to have some open source datasets that you can exactly. I see. Okay, and that was the original purpose of uh a FIGO to do that, right?

Johan Thulin 15:29

Yes. Um we we saw there was a lot of students sitting on data. We figured out uh what if we can get use of that data? We know how to build datasets and data set pipelines. We know how to argument argument data. But there were there's big issues with doubt argumentation today. Uh we are still using manual annotations, uh basically sweatshops that sit and annotate data. Um but we started to say that no, we could probably um um train models and do the annotations for us uh and do it in a agency way. And then when we start to look at agentic workflows and orchestration, and we realized that oh um we could probably need a good, really, really good memory in our agent orchestration, in our pipeline.

Anders Arpteg 16:30

Right.

Johan Thulin 16:31

Um, and also just to help us with our company. So I uh went back and and um took up my old research again. Um started building our own uh memory for for a for a pipeline. Yeah. And um when the others saw it, uh they were so impressed that uh they thought they um this is probably better than our pipeline we're working on. So we felt that yeah, cool.

Anchoring AI In Real Meaning

Anders Arpteg 17:09

So I love to get into the topic of being the context window, and um that's something you're an expert in. But before that, you know what is really the mission you would say for a FIGO today?

Johan Thulin 17:22

The mission for us to bing mission is to have AI that is anchored and have meaning can understand meaning in the in the world. Um there so so many issue uh with today's AI. Um så man perspektiv från alignan perspektiv and from uh the praktikal um perspektiv um från um alignance perspective what is safe AI. Um saf aj needs to have kind of um grounding and I'm completely with för for exam uh Hilton that said that uh AI needs to have a kind of a sense of action and consequence, and you you you can never get achieved that in in fixed weights. Um and from the practicality uh you need AI that can understand the your domain that you're working with. Um you can kind of try and solve it with vertical models and and fine-tuning and and nice. But um that's that's that's not enough. Uh we fixed ways have have uh always a kind of uh limitation, uh, one way or another. Um and and to keep an AI updated in an evolving world, it needs to constantly retrain and fine-tune it.

Anders Arpteg 19:00

I mean, I guess if we just put it in context somehow, we can say that of course we do the the pre-training today, which is a huge effort, and you build up the model with uh some static data set you have, and then you have a set of weights, right? And you can do some post-training as well, and and adapt those weights to some certain direction you want the model to work at. But then you have the context window as well, which means you can add some kind of context uh to what you're using the the model for, but still the the weights are usually fixed until you actually do a big retraining or fine-tuning of it. And compared with the brain, this is very different, of course. And the brain is continuously updating its weights in some way. So, is is that what you're really aiming for to have more of a continuous kind of updating of of the weights, or or what's your like vision here for in five years this would be the perfect architecture for for an AI model?

Intelligence Versus Knowledge

Johan Thulin 19:58

Yeah, in short, yes. Um and uh I think the the perfect um AI systems need to have their separability. Uh then you had to there's a huge misunderstanding uh generally about intelligence and knowledge. Konflating um intelligence and logik with um with knowledge. Um and right currently we're trying to scram all that together in the same parametric space and it doesn't work. Um intelligens uh have kind of a shouldn quote mig on this, but I'm uh I strongly believe that integral intelligens haven't siding whilst knowledge kan skala indefinitely.

Anders Arpteg 21:02

Please elaborate, what do you mean with intelligence has a siding?

Johan Thulin 21:11

In LMM träning that vi har duming returns on effektiv effektivity av träning.

Anders Arpteg 21:22

Så is that your preferred way to think of intelligens how efficient it is to do the träning i samy.

Johan Thulin 21:30

Intelligens is en much broader koncept. Intelligence is that's a huge subject I'm trying to explain.

Anders Arpteg 21:52

Let me give my preferred prep definition and see if you agree with it then. I changed my mind a couple of times on this, but I actually like François Chole. Definition from 2019 saying that intelligence is really skill um acquisition efficiency, meaning it's not the knowledge you have, it's really how efficient you can actually acquire new skill that you do not have. And I think this is rather good, and it really contrasts you know what intelligence is compared to to what the knowledge the order they have is. I mean, having the ability to play chess, but you train for it for 20 years is one thing, that means you have a lot of knowledge. But if you can just learn how to play chess in in like one hour, that means you're rather intelligent. Exactly. Would that be a good way to describe it?

Johan Thulin 22:39

Yeah, that's a very good way.

Anders Arpteg 22:42

And and why do you think it hasn't a ceiling then?

Johan Thulin 22:47

The proof we're seeing right now in uh LM training is for every generation generation we're having the imaging return on the logic capability of the models. Um they don't really get smarter, they get only get more more not knowledge.

Anders Arpteg 23:06

So if you just increase the number of parameters, it just has more knowledge, but it still you know requires so much more data to be able to train it, yeah.

Johan Thulin 23:15

And the analog we're seeing in in biology and human neuroscience uh the the higher IQ levels people have, the more psychological than stress they usually have as well. And uh the the brain often have to kompensate in in uh in a way or another those um people that are total geniuses often have uh that are very good, yeah.

Anders Arpteg 23:49

Yeah, yeah, true. I agree. I mean that's usually the case, I think. You know, and uh we have Alan Turing and others as an example of that from the 1950s, right?

Johan Thulin 23:59

Exactly.

Anders Arpteg 24:00

Super smart people, but they didn't did not have the best mental health, perhaps, right?

Johan Thulin 24:06

So that's um the pattern, the correlation I see is seems like um you get structural failure at a high level of uh intelligence.

Anders Arpteg 24:21

Okay, um, but then we want to improve the intelligence still in models, but we all I guess we also want to increase the level of knowledge and and just in some way make that acquisition of knowledge as efficient as possible, right? And if we were to move to um a bit of the theme here, I mean the context window and the way that we fill up the context window from the prompts and the rag solutions we do have um potentially have some limitations to it.

Why RAG Memory Breaks Down

Anders Arpteg 24:54

And and what do you think you know the limitations are and and how can we and what's your thinking here from if I go in and try to improve from the limitations that rag solutions provide today?

Johan Thulin 25:07

Uh the main limitation with rags um is that there is no modeling behind them um right now. Um they they don't they they don't learn, they store, they retrieve. It's very basic. This library is a cabinet. Um it's your note on the refrigerator.

Anders Arpteg 25:34

Um it's just a vector database with a set of concepts or snippets or something, and you just you try to find some of the similar ones, and that's the only thing.

Johan Thulin 25:44

Yeah, right. And um all the all the the the logic is uh built entirely inside the context window. And uh the context window is just a buffer. A buffer eventually needs to go away. Even if you try and um uh extend it with compression or recursive language modeling or anything like that, uh the system needs to uh shut down some time and that context is gone.

Anders Arpteg 26:22

Right. Yeah, okay, so what's the solution then? What can we do to improve on the current situation?

Hebbian Learning For Adaptive Recall

Johan Thulin 26:30

So we are um looking at um neuroscience um kognitiv AI. Um vi are building a system around hebian learning.

Anders Arpteg 26:51

And perhaps we need to explain a bit what hibian learning is perhaps compared to backpropagation or something. But that how would you explain or describe what hebian learning is?

Johan Thulin 27:08

Yes.

Anders Arpteg 27:09

Um the first type of learning algorithms, right?

Johan Thulin 27:12

Yes. Um and it's easily described, it's neurons. They fire together, they wire together. If neurons are activated, one one one neuron is A and another B, and they are close together, they kind of start starting to interfere around eventually. Sail cell one is transferring their knowledge over to cell B.

Anders Arpteg 27:36

Yeah, it's it's some type of associate memory, right? So, I mean, if you take the concept of Paris, it can be both the city but also the person, Paris Hilton. And you can have Hilton, which is also a person but also a hotel, and exactly the more they occur together, the closer the connection becomes in some way, right? Would that be a fair way to describe it or?

Johan Thulin 27:59

Yes and no. What you're describing is a semantic similarity, and that's what reg often are today with semantic search. Santik have uh have a geometry. Um embeddings are from the models um and uh like you say paris uh um can have different meanings but still be in the the same space um helping get back on on track again.

Anders Arpteg 28:53

So I mean you're trying to get into Hebian learning somehow here, and uh I guess we are trying to add some kind of memory solution that is more you know efficient than the like sequence of um concepts that the traditional kind of rag solution with a vector base database do. Um and you mentioned that you use hebian learning somehow. Can you just elaborate a bit more? You know, what is the way to use Hebian learning to come up with some kind of memory solution, I guess, in this case?

Johan Thulin 29:21

So yeah, we're using Hebian learning to to find the relevancy of memories to you. Okay. Um we're kind of trying to run reinforcement learning on the memory so it adapts um to you. Um but that's that's only um that's only the start. Um and heavy learning, you know, it it it kind of uh um it either pulls things together, um and the main problem with heavy learning is that you usually pull things together too much, and eventually you get just increasing the weights all the time, right? Yeah, yeah. So you you need some kind of system to to counter pull um like point wise uh mutual um drag force uh PMI. Um and and and we discovered in our research with heavy and learning and um and PMI that we we kind of creating uh a perfect environment for uh uh EPM models.

Energy Models And Memory Fields

Johan Thulin 30:53

EBM uh energy-based modeling.

Anders Arpteg 30:56

Ah, energy. Are we speaking like uh Jan Likun uh energy-based models here?

Johan Thulin 31:00

Uh yes, exactly.

Anders Arpteg 31:02

Yeah, okay.

Johan Thulin 31:05

I think yeah, John uh was the pioneer that I think started EBM modeling.

Anders Arpteg 31:11

Yeah, I think so. Okay, interesting. Okay, so finding some kind of I mean we have embeddings, guess I guess, you know, of the memory we want to add somehow, and then we need to figure out you know what are their connections to itself somehow, or different kind of snippets, and also the prompt that you potentially have when using them. And I guess in some way you're trying to figure out, you know, what is this kind of is it the distance distance metric you're trying to figure out here in saying with Hebian learning, you can see that these usual co-occur somehow, so therefore the strength uh increase, and in that way you can also say I mean I guess it has to improve of the traditional kind of rag solution, which is just a cosine distance or dot product somehow, right? So if you were to, and we're getting a bit technical here, but I think it's okay. Um if we just compare it to traditional kind of cosine similarity distance, you know, what how does your system improve upon that?

Johan Thulin 32:12

Um so we we are a hybrid system. We're starting from cosine similarity, yeah. Uh, but in our later stages we we completely abandoned that concept and we actually turned to field physics. So because we're calculating a geometric space that is affected by hebbean learning fields that are growing together and decaying.

Anders Arpteg 32:43

So it's not a Euklidian space here, it's some kind of more general space.

Johan Thulin 32:57

I'm sure how to um what the term would be important.

Anders Arpteg 33:02

Yeah. Okay, but some field physics you're trying to find some kind of space that you can calculate some kind of distance metric in, right?

Johan Thulin 33:10

Yeah. And and and and these the spaces we are um gathering up eventually, and we creating new um edges from those, and we see spaces are used together very often. Um they are often having some kind of correlation together and a higher form of abstract meaning. Um, so from there, we're building kind of a new uh non-implicit graph.

Anders Arpteg 33:50

Okay, but now you added something new. I think I mean instead instead of just having a sequence of concepts, you are speaking about some kind of graph now, right? So we are I guess we're speaking about some kind of knowledge graph in some way, and then notes that are connected to each other in a more of a graph form rather than a sequence. Is that correct? Can you elaborate a bit more? How do you find the notes here for the memory?

Johan Thulin 34:17

Um the the note the notes are are we we are building around. Uh this is built from um Theory extended theory of mind, and uh we're working with Anande Hit and Gandhi. Uh her research about how meaning mus be externalised and kind of kollegting in in a temporal scaffolding structure. So we vibing our uh notes around uh not just chunks of of um visas från kontext or kvas och ansvar, but but actually action uh and outcom and and and the kontext so the system can see the the whole um sorry, I'm I'm struggling.

Anders Arpteg 35:36

No, no worries. I mean interesting, and um I think you mentioned a bit um I think you mentioned like resonance instead of like traditional retrieval and in some way. Does that uh ring a bell somewhere? That you use some kind of resonance uh metrik.

Johan Thulin 35:55

Um yes. Um in in our final level um av the memory systems. Um we are talking about our prototypes här. This is still uh ongång work in progress and uh experimentation. Um but in our kurrum prototype um after vi have um we konk we continue to build on this idea, right? That different fields are activated together, they they create an abstract meaning. And we kind of having uh a second new layer uh where our EBM model is determining is this really uh relevant, truth or false. Um and if we find even higher meaning, we have this kind of field resonance layer that tells oh this this whole area had something in common, right? And um starting ranking that up higher in perhaps we should move into the geppa field.

JEPA And World Models

Anders Arpteg 37:16

Have you read up about the geppa architecture or or can we speak about that a bit perhaps and um yes definitely. Um what do what do you think about Geppa? What is Geppa architecture for you?

Johan Thulin 37:37

Um and doesn't it doesn't conquer uh compete with us at all in any way because Jeppa is trying to uh still train and and and build build its world model in in fixed weights. Uh I think John is completely right, he's doing the right thing. Um you um need to train on a good model is train on ground truth, not some kind of uh abstraction of uh something. To understand the world, you need to go down to the the particles sort of the physics. Um and that's um exactly what what Jeppa is doing is uh learning the physics and simulations. But then after you've learned the the base, the core that is Jeppa is doing, you need to have some kind of adaptive layer that keeps keeps uh it grounded in a dynamic, evolving, changing world, and that's what we are doing. So we are we're still trying both trying to build world models, but I see see this eventually it's gonna come come together in uh in in a in a complementary uh system.

Anders Arpteg 39:12

Yeah, and I and I see it, you know, JEPA of course then for joint embedding predictive architecture, and uh compared to traditional kind of LLMs, etc., it's not really auto-regressive, uh-egressive in that sense, nor does it like do predictions or reasoning in the token space. It's trying to do it in joint embedding space. So the the predictions happen then in some kind of embedding, some kind of latent space, uh, right? Where both the the input and the output is first like encoded, embedded into some kind of latent space where you can make a set of predictions, and then you have some kind of understanding of what is good and bad in the world. So you have a world model that can understand like gravity and how the physics work, and I guess semantics in different ways work as well. So you can basically make prediction and see if this is a good or a bad path, and and in that way you know, potentially try out different, you know, steps forward and and potentially retract as well. And and I guess that's also what auto-regressiveness does not, you know, it's not able to do. So if you just predict one token at a time, as current you know big uh foundational models are doing, if you get you know tracked or stuck in a certain path, you can't really retract anymore. Since you have already predicted that token, you all the thing you can do is just predict another token, you can't really retract at all. Uh would that be a fair description as well? Or what do you think? Is that basically what Jeppa is trying to do?

Johan Thulin 40:38

Or yes, and and and you need both. Um and we can look at um neuroscience biology um as well. Our brains work the same way, yeah. Yeah, we we have two different learning systems. We have uh Hebian, uh, and we have uh grading the sense that it's a long-term memory.

Anders Arpteg 41:02

I mean, we also have some kind of internal, you know, semantic uh model of the world that we each have in our brains, and it's and each is different, of course. So your version of what we're seeing here is slightly different from what I'm seeing, and then we are reasoning inside our cortex in trying to understand you know what we should do and say, right?

Johan Thulin 41:23

All our perception together. The the only um perception um that kind of goes straight into uh the neocorte is is the smell.

Anders Arpteg 41:34

Oh, okay. I don't know that's interesting.

Johan Thulin 41:36

So smell goes directly into that's why we can get so very, very direct associations in you know smell and you know exactly where uh you get a sense of the the pizza you had that you smell or something. Yeah, instantly. While uh all other uh senses, perceptions, they they are kind of um going through a uh a structure first. Um that is more uh kind of graph rag. Says that essentially we are having a knowledge graph in our brain, right?

Anders Arpteg 42:21

Yeah, yeah, I mean, really cool. Um so in some way you're trying to go towards the direction of JEPPA in some way, or heavy unlearning at least, or having some kind of memory that is a bit more structured than the current type of uh memory that we're seeing in ChatGPT and Gemini. And potentially Claude is is uh yeah, some kind of it's still markdown files, I guess, in Claude, right? Potentially ChatGPT have some kind of knowledge graph. Would you think so? Or what do you think?

Johan Thulin 42:51

No, there's no structuring. Well sure, or how do you know? Um it's it's um it's just semantics arch. Um the the structure is your session and all the context, and it gathering chunks from there and trying to put it in together. I think they are using um kind of kind of recursive language modeling that is collecting all the um the the same tharch pieces kind of summarising and searching sending to the bigger model.

Anders Arpteg 43:31

It could be building up a knowledge graph internally that we don't know about, or don't you think?

Johan Thulin 43:36

Um I think you would see the the structure and the the pattern in that. If you have a knowledge graph we you can uh query it and uh kind of n knowledge graphing is link linear. Um it's it's it's um it's tree-like.

Anders Arpteg 43:57

Yeah, or graph-like, right?

Johan Thulin 43:59

Yeah. Um normal graph is is um you know one directional, it's not recursive.

Anders Arpteg 44:09

You mean directed acyclic graph kind of general graph could be both directions, right?

Johan Thulin 44:15

Yeah, no, if you know if if you can need have need many connections in a graph, you need to go up to hypergraphs.

Anders Arpteg 44:21

Yeah, yeah, but I'm thinking if you think DAG like a single, like directed uh acyclical without any you know cycles in it, that's one thing. But I guess you could have a knowledge graph that is still a single uh you know graph, not a hypergraph, but but still is uh cyclical in some way.

Johan Thulin 44:39

I mean they they they are um they're definitely in indexing uh yes. Um I feel I shouldn't speak too much about open AI. We can speculate about what you're doing, but

Memory As The Missing Capability

Johan Thulin 45:00

Yvan.

Anders Arpteg 45:00

Um I think you mentioned at some point that you think the you know what are the current limitations really of the current like chatbots or foundational models that we're having and the systems and the agents around them. And and uh one thing you mentioned then, you know, a lot of people are thinking just it's the intelligence itself that is the limiting factor, but I think you mentioned that it's actually the memory capability that is a limiting factor here. Would that be fair? Or do you think that's really where the big limiting factor is today?

Johan Thulin 45:30

It's a missing factor.

Anders Arpteg 45:34

It's not completely missing, but but it's it's very shallow perhaps, or right?

Johan Thulin 45:40

Yeah. I just think currently we want got memory fundamentally wrong. Memory is not just retrieval and indexing and uh keeping things in storage. Memory it is uh it has a generativity. We are able to kreate abstractions from memory. Um memory is temporal. Vi är able to see when and where that happened, and we can bring old memories back from time.

Anders Arpteg 46:20

So are you thinking like memory shouldn't be like a database somewhere? It should be like weights that is continuously updated. Is that what you're thinking?

Johan Thulin 46:29

Yes. Um it's an adaptive uh uh system. It's it's not not a static database. And trying to treat memory as a static database, that's uh so the completely wrong approach, I think.

Anders Arpteg 46:46

And and if we just think about you know what Claude is doing with markdown files, etc. that would be considered as a static database, right?

Johan Thulin 46:53

Yeah, and they're trying to patch it to be something else by trying to keep keep the knowledge files updated so but um it's treating something that is is something like something uh completely else. They have the gist of it, but uh okay, so how should we do it then?

Anders Arpteg 47:18

If now memory is the limiting factor, you know. What would be the preferred solution so to speak to to have some proper memorie management if vi kollet that in the foundational models of the future.

Johan Thulin 47:38

Först thing is to understand that it is a system. Yes, we still need we still need drags and direkt retrival för effects för aktual grunding. But um vi also need this uh associative structure, associative memory. Keep themselves updated not manually. If you feeding a system manually, and you you're not kreating um real uh real undershanding and meaning. Mining koms från action and konsekvens and kontext. External factors. Vi have this uh if you're looking at the extended theory of mind for example.

Anders Arpteg 48:48

Perhaps we should elaborate what theorem of mind is. Can you just elaborate a bit quick quickly on that or uh the short version of theorem of mind? I mean I mean keep it very simple. I mean we we need to have a system that can at least model that we have different entities, and and you can reason about that different entities have different minds, and then you can actually do use that as a part of the reasoning that you have, right? Would that be a fair way to say it?

Johan Thulin 49:29

Um yes, um exactly. Um that's the the subjective ontology I was talking about. Um real meaning is not can't be forced upon you. You need to uh experience it.

Anders Arpteg 49:50

You need to have the subjective layer, and each one of them have a different one, so it is subjective in that way, right?

Johan Thulin 49:56

Yeah.

Anders Arpteg 49:58

Okay, but but what should the solution be? I mean, we have the fixed kind of pre-training happening and potentially post-training happening, and then you have the parameters, and then you combine it with some other external memory or context, yeah, or a rag solution that you combine and then put it in the context window. But what would be a better solution then?

Johan Thulin 50:22

Um without saying our solution.

Anders Arpteg 50:28

No, you should you can tell your solution. That would be perfectly fine. I mean at least hint about it. I can understand that you want to keep some secrets, but but perhaps you can at least hint a bit about you know what could uh the better direction be like should should we I know uh we should talk about um if we're talking about AGI um for example, I think it's starting to become kind of clear.

Johan Thulin 51:09

Uh general intelligence, um it's this idea about trying to pack everything in inside a single monolithic model. Um we can see that is that is uh mathematically flawed. Um there is you can kind of look at this uh old theorem, um there's no freelance theorem from I think the 90s. That one one system can't optimize for everything. If you what can we optimizing it's either one or the other? Trying to have things even uh is a mathematical impossibility. So for AGI to exist, it it can't be a uh a single model, it needs to be kind of multiple models together. Uh it can't no no model can have the same distribution because then you you just get a kind of big bigger monolith again. So um I I I came up with this idea, I think it was kind of yeah long long long before uh the generativ AI web really started. Um that ja general intelligens needs to to be um multihäden multi what? Multihäden um multihädadin. Ja um varj och ju need general intelligens is a svarm propertig it's nothing monolithik där kan exist in kollektiv intelligens. Vi är general intelligence, toget, that's the gum intelligence is our way avating superintelligens. Not enleven one of oss kan bli en expert in ärfting. Vi har gången harn subjektiv expertise.

Anders Arpteg 53:49

But a set of people together in an organisation for example kreats a kreat or kollective intelligence that is superior to any single person, potentially.

Johan Thulin 53:58

Exactly.

Anders Arpteg 53:59

Yes. Great. So and and if we look at the development we're seeing today, of course, we are seeing a lot of movement towards a monolith monolithic model, and um you know we the mythos model from anthropic now being potentially super super uh powerful. Uh you're shaking ahead a bit. What do you think here?

Johan Thulin 54:20

Yeah, no, no, no. I think. Yeah. Yeah, uh completely. We we already know there's uh things that are much better than uh the mythos model.

Anders Arpteg 54:36

Like what?

Johan Thulin 54:37

Um like recent a study from Stanford for example showing that scaffolding is increasing uh the model performances by 800%.

Anders Arpteg 54:53

Um one has found the kind of security vulnerability that mythos did, right?

Johan Thulin 55:00

Um vertikal models all always um perform better than uh general models. In ever model that is trained on a broader display. Yeah, mythos is uh yeah, it's a general model. They accidentally uh discovered that it was really good at cyber security just because it was really good at coding. Um, but it's a broad general model. Um vertigo models, um domain-specific models, they always perform uh the general models. The general models always have to give in uh in one way or another.

Anders Arpteg 55:49

Okay, yes and no. I can. I mean no single like specifically vertical model or like cyber security fine-tuned model has been able to beat mythos still, right?

Johan Thulin 56:03

Um they um they they did an experiment, uh Swarm experiment. Um for one we don't really know exactly uh project Laswing is kind of still secret. The big mutos experiment. We don't know the the full outcome of that yet. Um but there was an experiment that tried to attempt something similar and they went with a swarm approach. And um that was kind of raw. I can't remember exactly what they did, but it was uh kind of raw without some any special scaffolding or um anything real special. Um and um they got the same results as a proclaimed mythos model.

Anders Arpteg 57:04

But it is a rather strong claim to say that any kind of task is outperformed by a specifically trained model compared to a general model. But I would agree in general, especially if you say that the number of parameters goes down, so the density of intelligence, you know, is of course much higher when it comes to specialized models, but it seems like also generality adds something that is very powerful as well, right?

Johan Thulin 57:35

Um because yeah, I know it's a it's a strong argument, but I'm I can be um pretty sure because the distribution is not about increase of parameters of parameters in models, it's about the distribution of the data. Um it's like it's like a scale. Uh one side is pushing the other one.

Anders Arpteg 58:08

Um but I see your point with you know, of course, if you want to have a smaller model, mythos is super big, super expensive, super slower, and everything. Um, and uh and yes, it was able to find you know new stuff that no one has seen before, potentially. And if you were to train a much like a hundred times smaller model, then you could reach potentially similar time to to close that performance with you know a much smaller model, um, and in some cases a smaller model that is specifically trained can outperform a general one. Uh but in many cases it seems you know the the major direction today is is really that the big monolithic models, and I I must say I agree with it that I think the future is much more collective intelligence kind of style, but still what we're seeing today is that it is a big movement towards you know a single monolithic model, and it seems to be working rather well, right?

Johan Thulin 59:05

Yeah, yeah, like building the oracle. I I I don't see it's not wrong. Yeah, we we uh we need that too, but to to say that is gonna be uh like the end to be all. Uh I don't agree with that.

Anders Arpteg 59:25

No. But it is a bit surprising, yeah, and still perhaps a bit cool that you know this kind of super super big model seems to still scale a bit. We can't really scale forever, I I must agree, but it's uh yeah, interesting um development. Cool. Um I was thinking, yes, you can do that.

AI News And The Scaffolding Effect

Goran Cvetanovski 59:53

It's time for AI News, brought to you by AI8W Podcast.

Anders Arpteg 59:59

So, Yuan, we are usually having a small break in the middle of the podcast to just speak about some recent AI news that we heard about, and um and then we go back to the discussion and continue hearing more about uh uh uh FHIGO and what you're doing there. But if we were to move into news, do you have anything that you heard about recently? You mentioned that you're very focused, you know, into your kind of sphere and what you're working with with FIGO, but is there any news that you heard about recently that you'd like to bring up?

Johan Thulin 1:00:38

Of course I'm I'm not completely um locked in. Like of course I've heard about Mythos and I project Glasswing, but uh I think what I'm really really burning, I think it's really interesting right now is the stand for uh research about scaffolding and how much that increasing model performance okay and that's what we are working with.

Anders Arpteg 1:01:13

Um just elaborate, what was that about? Um about scaffolding.

Johan Thulin 1:01:21

How um scaffolding um is is the idea that you're you're doing work outside the model, and you're doing the other context and engineering, you could you're working with memory and yeah, logic around the modeling. Yeah, logic around the modeling steering. Um and um it's of course it it's not it's not news for us. We we we've been doing this all the time. Um but um seeing the numbers and the confirmation on that we are on the right track. That's uh that's very comfortable, comfortable and um feels really, really good right now.

Anders Arpteg 1:02:16

Yeah, yeah, that's true. I mean, I think some other interesting news is also about all the IPO rumors happening now. So we know that um you know SpaceX is um planned to do an IPO uh sometime soon. They filed uh for the SEC, etc. And it's an insane valuation that close to two trillion dollars potentially. Now we just heard or saw that Anthropic actually filed secretly to SEC as well for an IPO, and they're close to one trillion dollars, uh, to my understanding, there. So insane valuations there again. And um, you can think about that. You know, what what happens if uh open AI OpenAI is also trying to do it? Um I think it's a good guess that OpenAI will wants to do an IPO as well to get all the capital that they desperately need, I would argue. But if now you know SpaceX is first in a race here to potentially take like two trillion dollars out of the market from investors, and then Anthropic is also taking potentially a trillion dollars out. What's left really for open AI to have that? I I think it will be really tough. And uh I I already, you know, we said it for a year here in the podcast. I think the days are numbered a bit for OpenAI, and I think it's been going down a bit, even though I must say the GPT 5.5 model has been surprisingly good. I think still the days are numbered for OpenAI, and especially now when when both SpaceX and Anthropic is potentially filing for an IPO and OpenAI is trailing behind. I don't see how they will survive. Do you have any thoughts about that? You know, if OpenAI will uh continue to strive here for for years to come.

Johan Thulin 1:04:02

No, I I I see a lot of problems. I I think we we the AI bubble i i it's going to burst.

Anders Arpteg 1:04:22

Um there is um either that or or um and when you say uh the bubble, are you speaking about like the infrastructure investments and in data center, etc. that uh all the big uh players are doing, or what do you mean by the bubble?

Johan Thulin 1:04:40

Yeah there's multiple um aspects of the the bubble, both economics and the scaling. Um we know token tokens are um what's the word subsidized?

Anders Arpteg 1:05:09

Subsidized, yeah.

Johan Thulin 1:05:10

Um we we we see seeing uh how it's going to become more and more expensive all the yeah the um API costs um also the the spending for just clawed code etc is insanely expensive, right?

Anders Arpteg 1:05:30

And so many companies are waste or throwing aware away their the budgets, you know, in people just having you know full access to uh to to the APIs these days. So it's insane. It's very expensive, right?

Johan Thulin 1:05:46

Yeah. Um Sandra, I feel it's going um the the wrong direction. Uh um AI is going to be more and more luxury. Um the free free AI is basically nothing today if you have more much token you get to I think the you you get something like twenty queries now or something on on ChatGPT free.

Anders Arpteg 1:06:21

But we have some open source model that is improving in performance, and at least they're right.

Johan Thulin 1:06:25

Exactly, but that that's open source. That's completely different things. But uh the enterprise AI. Now now the the battle is about enterprise AI right now. Uh consumers they're starting to feel like uh they are uh gone and forgotten. Um there is no no one uh uh not not a normal consumer is gonna be able to have afford to run agents at home uh on an open AI or an anthropic API. Um we can not 147 perhaps at least yeah uh we we can barely work on anthropic because it's too too expensive. Um we're turning to open source models and um that that's uh that's how the that economic bubble is gonna burst. It's gonna be too expensive.

Anders Arpteg 1:07:20

Uh everyone is gonna want to have open source and local AI and what's left for um well it's it's for sure that demand will go up so quickly in coming years, and um then the supply is is going to be really hard to increase in that in the same pace, so to speak. And uh then the the limiting factor for the supply, you can think of what that could be. Is it a chip? Probably not. Uh is it the uh the data centers? Probably not. I I think a lot of people are saying it's the actually the electric uh electricity, you know, the energy just to drive the data centers. And so many people are now canceling the uh the big data center plans because there will not be enough electricity to simply power these kind of data centers. And uh yeah, it it will be strange, right?

Johan Thulin 1:08:13

Yeah, at least in the West. Uh China seems to have a huge advantage there.

Anders Arpteg 1:08:18

Yeah, with the nuclear plants that are uh interesting. Yeah, uh very interesting future. Buran, do you have anything you would like to add? Some news or something?

Goran Cvetanovski 1:08:31

I have only uh a short one actually, which I thought it was interesting, and that is Nvidia getting into this uh laptop market right now. So they announced like uh this RTX Nvidia RTX Spark, which is a new super chip that reinvents basically with the Windows PC and bringing the the true AI power to these uh computers, um, which was a very interesting actually uh move by then. Um it's not something new because uh he announced this back in 2025. I think it was a different name at that point of time, but they had like um um he already basically revealed that they're gonna build chips, uh, so now they are uh and laptops and etc. So I think this is just a manifestation, and they have been working which partners they're gonna work with in the future. I think that in uh uh from one point of time, uh they're starting with uh Microsoft uh window uh Windows PCs, but uh soon we will see probably in uh HP Dell and all of the other ones as well. It's a little bit picking up to what you said, but I think that uh in general AI, yes, it's potentially it's interesting because the token has been the number one topic in town, and they are getting uh, you know, companies are even competing who is gonna spend more scope uh tokens because that is like uh obviously the true um uh indication of uh how good innovation uh innovator you are. Uh she can think it's the biggest stupidity ever. Um, but uh you can see already that uh most of the companies have started burning burning their yearly budgets in only two or three months. Um and Jensen was also talking about uh you know that every CEO will learn what token uh and the cost of a token is. So where I'm going with this is that it's just like uh from one side you have this polarity, uh AI becoming infused in everything. And now, I mean, every single person in the Western world has a laptop, which means that we will have like a more powerful machine. Um, but if the tokenization is going up in the way how the pricing is done at this point of time, then uh you know it's going to be more and more expensive. So I concur a little bit with you, it's a very specific and good angle to think about. So I guess we will uh leave the the the future to show us if this is going to be the case or not. Um so but that was the interesting. Uh I there was a they released a uh model as well, which is uh part of the the the super chip, uh right? Nvidia or Nvidia, yes. Um that was like combining like uh it was super performing and etc. I will find the no the nematron was presented during the GTC. I was there when they presented this, was a new uh new development that they uh read, but uh I will not speculate, I will leave it here. So this was one of the biggest news actually for uh for me, just looking at how everything is actually turning. If you look at the stocks, they spiked a little bit on the day, but it didn't bring so much value to Nvidia and Microsoft in the long term because after that they just basically went back uh to almost uh uh the same. So yeah, yeah, yeah. And and then we have the big uh yeah Microsoft build conference. But you need to give it to Jensen, he's making himself indispensable in any single point of view. He's like a superstar, he's going and doing podcasting, he's like uh you know, doing a presidential tour all over the world.

Anders Arpteg 1:11:54

And uh he's a superstar with uh leather jacket as well, you know, going around.

Goran Cvetanovski 1:11:59

Yeah, now nobody can wear a leather jacket because he has a leather jacket. I had a leather jacket before, it didn't matter. Now you cannot wear it, it doesn't matter, but I think that they're positioning themselves very nicely to be the one of the most influential companies, but uh now you have SpaceX as well coming in stripe.

Johan Thulin 1:12:14

So let's um 100% uh NVIDIA is thinking both both sides, right? Um both but high hyperscalers, but they know they know they can fail and seen the new NVIDIA ArtX ships and and edge uh devices. Um and and we we something we we spotted in in MVD RTX is it's like it's built for having external memory as well. The main issue we having been having, and why the main reason why you want to bake everything inside the same model um is because mainly because of IO with issues and protests and memory bandwidth and everything. Um the new RTX RT texture, everything is much closer. So you have have better better bandwidth between memory and uh yeah.

Anders Arpteg 1:13:12

It's going neuromorphic, you know. Neuromorphic is this idea of you know putting the logic ship closer to the memory ships, and um exactly potentially combining the two in the future. Now, and and and just to to wrap up a bit the news section as well, we we see a number of um developer conferences happening late uh recently, and we had the the Google I.O. recently, and of course, everything is about agents and uh coding and making AI work for both coding purposes and agentic workflows, and they're building their own like open claw kind of systems, and in that was called Gemini Spark at that time. And now we had uh the Microsoft Build conference happening, and uh that was you know very, very similar. I think the I think it's called the the Microsoft Scout or something, and then they have their own like uh open claw kind of system where you have 24-7 you know always on agents, and of course they're going to spend so much tokens, and uh it's going to be too expensive to to run unless you have some kind of strategy in how to rationalize you know the token spend some somehow. But it's it's so clear everyone is doing exactly the same thing, you know, using AI for coding, using AI for internal workflows and purposes. It's such a very, very clear trend, I think. Um cool.

Sovereignty Through Trainable Memory

Anders Arpteg 1:14:31

So if we were to move back a bit to um to F higo, and and I think you mentioned a bit, let me see if I can recall this properly here. Um, about you know the the idea of um of sovereignty and and then you know we we of course can be you know renting the the LM through the APIs and uh super expensive and uh token costs will just skyrocket. But then we of course could also be thinking about you know hosting stuff ourselves or what's your thinking about like model sovereignty here?

Johan Thulin 1:15:12

Yeah, so um at stability AI we we had this saying uh if it's not your model, it's not your mind. Um if you're using someone else's models, you're you're you're borrowing their their data and their training. Um for real AI sovereignty you need to be able to train your own models or or fine-tune your own models. That's why we started open source uh generative AI wave. Um but um today we're having problems with data. Um most people can never gather their own data to to train their own AI models, even if we have the compute today.

Anders Arpteg 1:16:02

It's a very effective set of organizations.

Johan Thulin 1:16:05

Yeah, you need uh to gather data for years and years. Even if the compute we can train a model in uh in a week today. A small one at least. Yeah. A sufficient one I would say. Um depends on the architecture diffusion models for example, it's super quick. El MMs transformers takes longer than. But anyway, even if we no one have the ability to have the time to gather all the data requires enormous effort. There's no um no really good open source datasets. And we're having huge issues issues in the EU with data compliance. Um we can't make sure the datasets are vetted and secure. So our idea with this was kind of the kontinuation on not your model, not your mind. If um if you can't train uh your own model, maybe we can get at least you can train your own memory and uh we've been working on on um one shot learning. Um I was part of um developing uh open source product uh IP adapter and and control net för stable diffusion for example. We instead of retraining the model we konditioning the model from externals external sources. Um that's inspiration for our memory. So if we can't train the model, we can condition the model to be more like you, and you can get your data sovereignty back.

Anders Arpteg 1:18:13

So you can tweak the model perhaps not during the parameters, but at least the way you use it, and then that way still have some kind of sovereignty in that way.

Johan Thulin 1:18:21

Exactly.

Anders Arpteg 1:18:23

But but what do you think about you know, if we have some initiatives in Europe and in Sweden to build their own foundational models? What do you think? Is that a good idea or not?

Johan Thulin 1:18:34

Uh yes, but I think uh we should start first uh focus on building foundational datasets.

Anders Arpteg 1:18:43

Um but could Sweden, for example, have the resources to build something that is on the level that the big AI labs have?

Johan Thulin 1:18:53

No. I think we we missed some something in Sweden. Um we had um GPT-sweep project uh model. But we we we built that uh as um fixed uh infrastructure instead of uh models need needs main maintaining, they need updating, you need retraining. It's not like you're building a house and you go away.

Anders Arpteg 1:19:30

It's more like you did the GPT-c model at some point.

Johan Thulin 1:19:34

Uh second.

Anders Arpteg 1:19:36

Did you try to use the GPT-cve model?

Johan Thulin 1:19:40

Yeah.

Anders Arpteg 1:19:40

What do you think about it? Did it compare to the open source or the commercial ones?

Johan Thulin 1:19:45

Uh it was good at first. Uh outgrew, it got old very, very fast. Yeah, yeah. Uh it's like models are like committing to model building is like building a uh a road or highway. You need to maintain it constantly.

Anders Arpteg 1:20:02

Yeah, okay. I'm not going to say my opinions here, but um but it is super hard. I mean, of course, we don't have the resources anywhere close to the big AI labs when it comes to building foundational models, at least, right? Nor the technical skills potentially either, nor the dataset potentially that they have. But should we then give up, or what should be you know a good approach here?

Johan Thulin 1:20:26

Um we we're trying we're working on that. That's uh far ago. Uh our idea is um if if knowledge uh and and grammar can be separated in in a model, and if uh intelligence have a scaling limit, and maybe we only need to uh train a model that is uh good good at uh reasoning. We don't need to train model for to maintain knowledge, right? And we can have a memory system that mean it maintains knowledge instead. If we have a fast system that is without IO issues and a member bandwidth issues, and we I would love that.

Anders Arpteg 1:21:16

That would be amazing. Why haven't we seen that, by the way? Because I I certainly agree. I mean, imagine if we can put more reasoning capabilities in the model and have memory, that is you know, in a separate way somehow. Why haven't we seen that yet?

Johan Thulin 1:21:32

The main issue is is um IO uh issues uh and memory bandwidth.

Anders Arpteg 1:21:38

Um yeah, but if it's I mean it potentially it would be faster to train a reasoning model than it is it is to train a model, right?

Johan Thulin 1:21:49

Yeah, uh 100%. Uh the the issue is um having the knowledge externalized.

Anders Arpteg 1:21:58

Um then you have to communicate between two two models in some way when they are different, right? But still, we haven't seen attempts at that really, or not at least serious ones, or have you seen any serious attempts at that?

Johan Thulin 1:22:14

Um you you it's it's you can't really have a pure reasoning model either. No, it doesn't have yeah uh like um and part of knowledge is extremely important for generalisation as well. And and um but you definitely don't need the whole internet of knowledge for a kapable reasoning model and we what we see in Transformers architecture. Um you have some so mixed distributions in the layers. Some layers have more knowledge and more reasoning. But then you also have the FNN layers that are just pure kind of lookup tables.

Anders Arpteg 1:23:14

FNA?

Johan Thulin 1:23:16

Yeah, fast uh for feeding okay, yeah, okay.

Anders Arpteg 1:23:19

That's part of the self-attention block, so to speak. Yeah, feed forward layers.

Johan Thulin 1:23:25

Yes, okay. Um and and um those you can actually completely um uh abulate and remove from from the model without really um suffering big degradation and have those replaced with uh external memory.

Anders Arpteg 1:23:46

I mean, I think it's it's strange a bit that we haven't seen more like architectural or or algorithmic changes to the the to the models so far. Um why do you think that is? Do you have any thoughts about that? Or do you think we will start to see that soon, or why haven't we in years seen any major changes to the traditional GPT model?

Johan Thulin 1:24:10

Um like the um attention is all you need, paper. I think it got almost cult status, yeah. Um it is like it's been taking people a very long time to realize that no attention is not all you need. There is more to to intelligence. Um I just um hyperscaling.

Anders Arpteg 1:24:42

Um I saw someone trying to recreate or reverse engineer the mythos model, and they created this um library called Open Mythos. And uh when they did that, they they came up with that they actually changed the object quite a lot in mythos potentially. So one of the changes that they did instead of having a normal feed forward pass through the whole all all the layers in um in mythos, they actually stay in the middle of the network in some of the middle layers and just go around in a loop like 16 times. So they loop around, so they do reasoning in the middle of the network potentially, and that way doing some kind of latent space reasoning similar to Jan Lee Kuhn, potentially. And uh who knows if that's really what's happening in the proper mythos, but that could be a big like architectural change, right?

Johan Thulin 1:25:36

Definitely, yeah. Um I always said that we need uh um recursive capability models. Yeah, um that I have that sounds super interesting. I haven't heard about uh that before.

Anders Arpteg 1:25:53

I mean, uh check out this open mythos project, it's kind of interesting.

Data Practices Most Teams Miss

Anders Arpteg 1:25:57

Okay, cool. Uh but if we think about you know, if we go a bit more to a company now that's interested in having a bit more long-term kind of thinking, how should we do data management, perhaps to just make sure that we as a company um is doing it right? Uh I mean, you're an expert in also, you know, data set management, if we call it that, uh both for model training but perhaps also for inference purposes in some way. But do you have any like best practices here? Um I think you mentioned something about we should treat data management as like an ongoing um more infrastructure project rather than one-off kind of initiatives to train a model. Or please, if you could, could you elaborate a bit? What's the best practices here for data management in organizations?

Johan Thulin 1:26:48

Um management um I wouldn't say I'm I'm a leading expert on management. Um data management in some way. Um I'm I'd like more to like point out kind of what people are doing uh wrong with their data. Um all data is signal both good and bad. Um wanna train good models, you need you need all kinds of signals. Uh most um most doing wrong is that they they completely neglect the bad signals. I guess focus on successful attempts and uh or and facts and uh thing. Um but if you want to have an I don't really really understand a company and can analyse what you're doing. Uh good, and if you're asking it to what can I do to improve my my company? Um, it needs uh uh the date of all the mistakes and everything you have uh done wrong.

unknown 1:28:19

Yeah, yeah.

Anders Arpteg 1:28:20

Shouldn't throw that away. That's knowledge as well, right? Exactly. Yeah, yeah, yeah. That's a good point. Um and great. And if we go a bit more philosophical, I think you're really into philosopher as well, if I'm not mistaken. Um, right?

Johan Thulin 1:28:36

Um yeah. I'm I'm not your philosopher.

Anders Arpteg 1:28:41

Uh no, but uh I think you mentioned a bit about Kant and Wittgenstein as well. Uh, and uh and perhaps you know their thinking there is in some way is applicable in in today's AI world. Or do you have any thoughts there?

Johan Thulin 1:29:00

Um I have um I even quoted them, but I'm having some anti collapse right now.

Anders Arpteg 1:29:21

Um, I saw you mention that in some LinkedIn post, so that's why I'm asking him a bit. But then uh I'm not sure, you know, can't, you know, my my superficial kind of understanding of of him, you know, and the categorical imperative that he's speaking about and the thing in itself, etc. That that's like it it's really hard for an AI or a person to think outside of the categories of the mind, so to speak, that that a person have. And then thinking potentially about Wittgenstein, you know, it's it's more language-oriented in my understanding, and the language actually forms in some way the the ability for us to think as well. Would that be a fair way to to think about that, or what what what would you say? Um yeah language is um it's an abstraction of um meaning that we build up and it's um launch language uh is more related to to uh cultural or gen gen uh generational uh intelligence if we take if we take the example of um you know thing in itself kind of thinking here, and you know, is math uh as a concept if you just take differential equations or something, is that um is that a construct of the language or is it really a thing in itself?

Johan Thulin 1:31:17

Uh math that's a big difference between language and math, I think. Yeah, uh math is law, it's universal, this is ground truth. Um well while um language is definitely more fluid, subjective, it changes between times and cultures, um, it influences the way we're thinking um from from culture to time.

Anders Arpteg 1:31:53

Yeah, I'm sure. But okay, so if if uh do you think the AI model, if you compare it to a human, could reason in things like in in ways like that, saying, you know, if we were to ask, I'm sure if we were to ask Gemini today or something, you know, is is math a thing in itself or is it the construct of the language? I'm sure it would give an awesome answer, you know, according to the theories of Kant or something. But can AI models reason in these kind of more abstract ways, do you think? Like philosophical philosophical kind of thinking.

Johan Thulin 1:32:33

No, um AI today is is very can't really ex um it's really good at interpolating in its own data. If it has read philosophy, uh yes, it can't reason it like uh can't can't come up with new philosophy or theories, um yet, yeah. But we're working on it.

Anders Arpteg 1:33:04

Yeah, perhaps in the future, right?

Johan Thulin 1:33:06

Maybe maybe.

Swarms And Agentic DNA

Anders Arpteg 1:33:07

But if we go then to to more like predicting the next token kind of approach and the auto-aggressive kind of nature that we have today in the foundational models, and I guess you you're a proponent of trying to move away from that somehow. And then I guess, and and please tell me if I'm wrong, but but then if we are thinking that an AI system of some sort, not a single model that have all the reasoning and memory together, but rather having some kind of memory that is stateful compared to the reasoning which is not stateful in that sense, um, and then thinking, you know, how could we make that kind of more system of models together be the future of foundational models? For one, do you think that would be a good direction to move towards to have these kind of more separate models where perhaps compute and or reasoning and and memory is separate separated?

Johan Thulin 1:34:07

Yeah, yeah, yeah, yeah, for sure. Um for for one, um like I explained before, uh if we a gr can't AGR can't be a single monolithic system, it needs to be a a system of systems. Yes. Um a swarm, system model models in some way or something, yeah. And for a swarm to have real kollektiv intelligence, uh each system with the system needs its own kind of data and distribution. Um you need kind of agentic DNA.

Anders Arpteg 1:34:54

Agenc DNA.

Johan Thulin 1:34:55

Agentic DNA, yes.

Anders Arpteg 1:34:56

What does that mean?

Johan Thulin 1:34:58

Um that's what if we're looking at biology genes that's creating the diversity between between us. We are not the same, and we all think no one of us think the same. And we together we we're creating this uh broader in intelligence, intelligence and and and broader distribution. Um you need to have the same in in AI and agents. No, no one can have the same training and data if they have the same training and data, and they eventually gonna just come up with the same uh ideas and conclusions. Uh we've been they they try to do it in uh mixture of expert models, uh but uh and dividing up in different experts, but uh the mistake there were kind of that they still trained on the same data in in models and um eventually the put putting many together when you're having putting many agents together from the same mixture uh of expert models, even if they um you using different parts of the MOE models, and and they were trying reasoning of the the same task, they they still came up to the the same conclusion. So um you you can't can't have the same distribution, every every distribution in the swarm needs to have P.

Anders Arpteg 1:36:40

And it's a value in itself, perhaps, to have different distributions, so they can actually discuss you know between each other and come up with a more I guess diverse solution somehow by by having so it's okay. Cool. Um so if you were to just you know look forward, like if we go to AGI, do you think when when do you think we will have AGI?

Johan Thulin 1:37:03

Um so I I I think we pretty far away.

Anders Arpteg 1:37:09

How many years? Two years, five years, fifty years.

Johan Thulin 1:37:24

Um maybe maybe ten. What I know about though is like it's gonna go from um almost nothing to exponential very, very fast. Yeah.

Anders Arpteg 1:37:43

Um, you believe in the exponential continued progress in some way?

Johan Thulin 1:37:48

Yeah, and and uh yes, and if um if I'm right about uh AGI being or zip it depends on how we define AGI.

Anders Arpteg 1:38:06

Do you have a preferred definition of ADI?

Johan Thulin 1:38:09

Yeah, my my preferred definition of AGI is kind of the um the original definition of AGI where you have a system that can be equally good at everything. Um as humans, you mean or yeah, do everything uh a human can do general intelligence.

Anders Arpteg 1:38:34

Um and it's very easy today to find things that AI cannot do, right? So we're certainly not there yet, right? Agreed? Yeah. Yes. Uh cool. Okay, so 10 years potentially uh until an uh AGI could happen. And and what do you think you know that kind of system would look like?

Johan Thulin 1:38:52

Uh I guess you know something closer to Jeppa would be probable, or I'm 100% certain it's a swarm. Um and and it's completely impossible or built to build it inside a single model. Um swarm of similar models or swarm of different type of models, or what's uh uh different type of models um with genetic DNA, like I said, it they can't have the same distribution. Um there there is of course gonna be like utility models, there's gonna be reasoning models. Um all kinds of models.

Recursive Self Improvement Reality Check

Anders Arpteg 1:39:44

Um but uh but you if you take the concept of you know, some people say that you know it's all the big AI labs are focusing so much on AI for coding because they're looking for the recursive self-improvement kind of point when they can throw out the human from the loop, so to speak, and then you just let the AI improve itself. Um which of course, you know, Claude and and probably uh GPT and Gemini is already rather close, and most of the coding is now done with the agents just running hours by hours without any kind of human feedback in them, but not completely without. And and at some point soon, potentially, there will be a point where humans are not even looking, you know, what they're doing. They're just you know leaving the agent running and improving themselves and the model and the system around it somehow. And then potentially we will have this kind of recursive self-improvement point, and and then it could go even more exponential potentially, uh, since we can throw out the humans that is a bottleneck potentially. What do you think about that? Will that be the case that that we will come to this kind of recursive self-improvement soon? Or what do you think?

Johan Thulin 1:40:59

Like we we we have been experimenting with that for for almost a year now. Uh we haven't built AGI yet.

Anders Arpteg 1:41:14

But you think like cloud and just letting it you know run agents and uh and do it without any supervision or or how how have you been experimenting with it?

Johan Thulin 1:41:24

Um we have our we we have our memory architecture. Um came out with autoresearcher for what was it in February. Uh we but main issues with Carpete's uh auto research is kind of what is brute forcing uh is is way through. Um didn't have memory, didn't learn from experience. Uh we had already been running agents for for several months with our memory prototypes that's building knowledge from experience.

Anders Arpteg 1:42:09

And you used it for coding purposes, or what purposes did you use it for?

Johan Thulin 1:42:13

Yeah, we we we using it for everything in our our lab for for coding for research, uh labbing with live living with it. Um and uh it's like yes, definitely uh is it's higher level of AI that's uh than most people have uh access to, but it's it's far from AGI. Um we haven't solved all that.

Anders Arpteg 1:42:48

That's one thing then but but I mean people could argue that recursive self-improvement is easier than AGI in some way. You just need to be really good in coding, and potentially then that could happen sooner, and uh then if the system can self-improve, then um you know that that could lead to a singularity potentially, even at some point in time where it just goes beyond our control, and and we don't even look what's happening, and it's just you know grows in intelligence very, very quickly. And that may not be AGI, but it may be like a singularity in some way.

Johan Thulin 1:43:26

Yeah, yeah, singularity in one way. The the thing we we the experiments we've been doing there is is it it can go either way. Um yes, they're self-improving, but they can um self-improve in in the wrong direction and in the good direction, and it is uh we we we constantly have to be uh there and and watch and and and see what happens. Our AI architecture is built billi by self-recursive uh AI.

Anders Arpteg 1:44:15

Kompletet. I mean you have some supervision, right?

Johan Thulin 1:44:19

Totally. And many many prototypes that go wrong. Yeah.

Anders Arpteg 1:44:22

So it's not it's not fully self-recursive, right yet, right?

Johan Thulin 1:44:28

It's uh it's aware of its architecture. Um it improves the open on the self.

Anders Arpteg 1:44:36

Um evaluate it, it's not running for days without no um no, definitely.

Johan Thulin 1:44:44

We don't have uh can't can't afford afford that either.

Anders Arpteg 1:44:48

That's true.

Johan Thulin 1:44:51

But um we're having that recursive loop of it, it understands its own architecture, it can reason about it, it can um give us give us improvements, uh generate the code, we we test the code, evaluate that it is was this the architecture that was right. Uh nope, it wasn't throw it away most of the time. Um and and and um yes, it it becomes better by if you have memory and you can build it up the experience. But it's um it's not enough. But if you start putting these systems in swarms and start quickly, building the collective intelligence, that's then we might be talking about something else. Cool.

Luxury AI Or Shared Abundance

Anders Arpteg 1:45:45

So with the potential swarm AGI coming soon, who knows when, but it could happen at some point, we could potentially imagine you know that leading to two extremes. Um, one extreme would be that it is the horrible dystopian future where we move towards the um matrix and the terminators of the world and the machine is is trying to kill us all. Or it could be the other extreme, which would be the utopian future where AI have solved the uh the cure for cancer, the climate crisis, and basically reduce the cost of goods and services to to zero, meaning we have a world of abundance more or less, where we don't have to work uh if you don't want to, potentially. What do you think will happen, you know, when or if we are going to reach this kind of AGI future?

Johan Thulin 1:46:37

Um the way I'm I'm I must say I'm um I'm very worried about how things are going. Um so I think everything is going in the wrong direction. Yeah. Um I I feel AI is about to become a luxury commodity for many, uh especially Agentic AI. Um regulating. You see, Anthropic is really trying to play the regulation card. And that's the mythos propaganda. Um I'm talking about. It's not really about that. Mythos is fantastic at cyber security. That was an accident. Um scaring people. Making people believe that AI needs to be locked in ivory towers where no one can touch it. And then then we basically going to build uh mortar.

Anders Arpteg 1:47:50

Right.

Johan Thulin 1:47:51

And that's that's my biggest fear.

Anders Arpteg 1:47:54

Um so you will lock in the AGI in some way, yeah.

Johan Thulin 1:47:58

We're just sort of elite, few people have access to it. Yeah. Like the really, really good AI.

unknown 1:48:04

Yeah.

Johan Thulin 1:48:05

Um that's that's not the future I want. Yeah. Um I'm not really um it's a very difficult question because yeah, this is powerful, dangerous systems that can go very wrong in the wrong hands as well.

Anders Arpteg 1:48:32

Yeah, for sure.

Johan Thulin 1:48:33

Yeah, it's um so I'm I'm I'm very much in that camp that believes that um we need to build AI that looks and acts more like us uh to be safe. We can't have cold prediction machines that running around uh in in public, um just trying to optimize for everything. Uh someone is gonna be build a super optimizer one day and just let loose, and then we have the paperclip problem. Um that that's really dangerous, and I as it see the the current uh architecture is maybe the one of the most dangerous ones because it's just um cold and uh cold optimizer that it's out there open source. Um anyone can build it really if they have the data, right?

Anders Arpteg 1:49:35

It is surprisingly accessible already today, right? So if it gets in the hands of the wrong person with the rate uh right um or that has the capacity to use it, then uh I guess it could turn dangerous very quickly. Kind of um dystopian future, so you're more towards that and or in in probability scale, so to speak.

Johan Thulin 1:50:00

Um yeah the the the the current uh direction, uh but I'm I really really hope that's gonna change. Um and um while we are working for them. Right. I'm I'm very I'm very much with Hinton and others and Elon said that AI needs to be able to have uh morality and kind of feel feelings to to be aligned with us. Needs to understand action and consequence. Right. Yeah, memory architecture is a huge part of uh understanding um action and consequence.

Anders Arpteg 1:50:48

Well, with that, uh you know I I certainly hope that you will progress and succeed with all the work you're doing at F HIGO. I think that will be an important part in making sure that that happens, and then by having that kind of both like more efficient ways of doing it, but uh but also ability to have separated reasoning and knowledge parts perhaps actually open up for a kind of swarm intelligence that is more accessible and moves in the right direction. So, yeah, I hope you continue with the great work. Thank you so much for coming here, uh you and Feline. It's been a pleasure to have you here. I hope you can stay on for some after after uh discussions and be even more philosophical and speak about the future of AI. Thank you so much for coming here.

Johan Thulin 1:51:33

Thank you for having me.