Conor 0:11

Welcome to Raycast episode 124. I'm your host, Connor, and today with us we have two guests and our only panelists. As Adam mentioned before, I always mention this. Therefore, I guess promoted for the time being to co-host. I guess we're going to go around and do brief introductions. We'll go over to Adam, then come back to me, and then we've got uh two announcements. I'm Adam Boltevsky. I am uh head of language design and also do a bunch of other things at Dialog Limited. And as mentioned before, my name's Connor, fan of all the array languages and host of Arraycast. Yes. So I believe we've got two announcements. Both of them are from Adam, so we'll throw it over to you, Adam, and then we will get into today's topic. Right.

Adam 0:55

I have got uh good news and bad news of very different caliber. So the good news is that there's a new round of the APL challenge that just went live. So you can try it out at uh challenge.dialogue.com. Whole new set of ten problems. Even solving one gives you a chance to win one of the prizes. The more you solve, the better. Uh better chances of winning. Um that's the good news. Uh the sad news is that uh Guida Christensen, the former CEO of Dialogue, passed away this past Saturday. Um and uh she'll of course be missed. She um had a long APL career uh and uh uh she was uh rightfully given the uh um the Agerson Award for uh propelling APL into uh uh the current times, should we say. Um and she might also have been a pioneer in online dating. She might very well be her and and Martin Kronberg might well be very well been the first couple to meet online ever. Um but there's a lot more that could be said about her, and there are episodes or various things uh with her that we can link to uh in the show notes, so I'll leave it at that.

Conor 2:07

That is um yeah, very sad news. I've met Gitda a handful of times. And um yeah, very solemn news, and I guess thoughts uh are going out to Morton and the family, uh obviously. Um yeah, there's no I guess easy way to transition. Um but uh we will we'll do that.

Adam 2:26

And Gida will have wanted us to keep going. She was uh very like practically minded on everything until the very uh very last, and she will uh be proud that we keep going with this kind of thing.

Conor 2:39

Yeah. Well well, with that in mind, uh we will um yeah, conclude our announcements and transition to the topic for today, which is a new release of Jay. We do these, I think, annually, dating back to the beginning of Arraycast. We've usually brought Henry on, I think every year, roughly plus or minus, around the same time. The new releases, I think, kind of come out in March or April. And uh without fail, that has happened this year. Uh 9.7 has released, and there's I believe 9.8 beta that is available as well. And so we have once again Henry Rich back with us. If you've seen the title of this episode in your feed or if you're watching live on YouTube, and um we also have a second guest, though, today. I'm gonna do my best to pronounce his name correctly. We'll see how close I get. Uh our our second guest, along with Henry, is Marchin Wek, I believe. Uh I didn't see too painful of a grimace on his face. Um and so uh Henry asked if we could have we could bring Marchin on uh in order to help out with talking about 9.7, because I believe Marchin has been doing uh a lot of the work uh involved in this release. So uh before I think we start talking about J9.7 and everything that's uh releasing with this release, uh maybe Marchin, you can introduce yourself. Uh tell us, you know, how'd you uh come into the world of array languages and how did you start uh uh working with Henry and uh I guess contributing to uh Jay.

Marcin 4:11

Uh yeah, yeah. Uh but first I'd like to give a shout out to the whole Jay team, uh especially Eric, uh Chris, uh Bill, and Bill, uh, who are watching this live video cast. Uh well, uh a few words about myself. So uh I'm a computer science student at the University of Warsaw. Uh I'm in my fifth year now. Uh in my free time, I played Susaphone. Uh I played in a uh local volunteer firefighters band. Uh and moving on to Jay, uh, I first learned about it uh about four years ago during my first year at university.

Conor 4:49

Was that uh from a course or did you just stumble across Jay?

Marcin 4:53

No, no, Jay Jay hasn't appeared explicitly in uh any of my courses, but uh Kenneth Iverson and the term Iverson notation has been mentioned in a few of them. Uh so it wasn't hard to discover Jay, right?

Conor 5:07

Interesting. So as Jay's first year at university, you hear about Jay, and then what's the path from first year hearing about it to uh you know a few years later you're now contributing?

Marcin 5:18

Oh uh well I got interested uh because well, Jay is obviously the last programming language designed by Kenneth Iverson. And uh as a more modern implementation of his ideas, I thought it's uh worth knowing just awesome.

Conor 5:33

All right. Well, I mean, uh without further ado, let's uh we have, and this is the first time, so if you're listening to this uh in your podcast feed, it might be worth it to hop over to the uh YouTube channel because uh Martin has gone ahead and prepared a uh presentation that's gonna be highlighting some stuff. And I think at some point also Henry's gonna um show a little bit of J code as well, so we'll swap uh sharing screens. So, anyways, there is uh I mean obviously with the YouTube video, there's always a video component, but uh this time we actually have uh a little mini presentation that we're gonna see. So uh take it away, Marchin, and uh I guess we can ask questions or uh if they come up. But uh yeah, happy to hear about J9.7. And I guess Henry's got uh something to add here.

Henry 6:14

Well, I let Martin do most of the talking, but uh just to introduce the release. Um we have uh it's good that we have video because a lot of the work of the last year was uh in user interfaces, particularly uh ones based on browsers. March, we'll talk about that. Uh and also this being the first time we have video, we can mention uh uh a tool to fix which is not new but is unique uh and uh and quite useful. In addition, the language-related input is we made were uh related to things that are difficult to do in the standard APLs, and we'll talk about those later, particularly multi-threading and dictionary support.

Marcin 7:01

Awesome. Uh okay. So uh yeah, we will first see what's inside the J box. Uh so before we dive into the new features, we will start with a quick look uh inside the J installation. Um we'll look at what it includes and what you can do with it uh right away. So the J installation includes several interfaces for using J. And we'll start with the command line interface, uh JConsole. Uh JConsole lets you use J directly in the command line, um, just like uh other command line tools. Um on the screen you can see the the Neo Vim uh with uh a J script uh and the terminal uh open on the right. And the script starts with a Shibank, uh so so it can be launched uh like an executable program. So let's let's run it. Uh yeah, and this this J script downloads uh temperature uh for data for for Warsaw uh for the given date range. Um we pass uh the start and end dates uh as uh command line arguments. Uh well and uh downloaded data are in uh CSV format. So so Jcode uh then processes them, uh converts the temperature from uh Celsius to Fahrenheit and creates a temperature plot. Uh and as you can see, the script does not close the J session, uh so Jconsole uh stays open uh and is ready for interactive programming. Uh for example, we can uh convert uh downloaded temperatures to Kelvin's uh directly in the same session. Uh and uh to return to the terminal we simply add exit uh at the end. Uh that that exit uh of course uh can be included in the script if you wanted to close the J session uh after execution. Uh yeah, so so J Console is uh quite useful when you want to work with uh J in command line. Uh and it also makes uh it easy to um combine J with uh other shell tools. Yeah, that that was the command line interface uh for J. But uh for larger projects projects and uh for learning J, uh we usually use a graphical interface, which is uh a bit more convenient. Uh and the most uh I would say versatile environment for programming in uh J is uh JQT. Uh so the yellowish uh window uh on the screen is called the term window, uh, and it is a REPL, an interactive session. Um the second window uh is called uh the edit window, and this is where you can uh write code in files uh and run. And when code is run uh from the edit window, uh the results appear in the term window. Um and uh what makes jqt uh great is that it also makes it easy to use add-ons, uh especially uh graphical ones. Um here are uh two plots showing uh examples of data visualization in J using uh the plot add-on. Uh and there are many more add-ons in J, actually, there are about uh more than 100 uh add-ons, official ones, and uh many more created by users. Um so uh yeah, one more important advantage of JQT uh is that it supports uh the window driver API. Um and because of this window driver, uh many add-ons can be used in JQT uh not only from the code but uh also through a graphical interface. Uh and this graphical interface uh is especially convenient for add-ons that involve some kind of uh graphical output or interactive controls. And one of them is uh the debugger. Uh the debugger window is uh now in the bottom left corner of the screen. Um the debugger also works in JConsole, so that this previous command line interface, but only in text mode, as that was a command line interface, and in jqt uh it can be used through uh through a graphical interface, uh which usually makes uh debugging more convenient. Uh and uh debugger in jqt is also enhanced with uh dissect. Uh dissect, as uh Henry Henry mentioned, is uh visualizer for JSentences. Uh we'll talk more about uh dissect later, but you can already see two uh two dissect windows on the screen. Um so the the the key idea is that uh debugger lets you step uh through the program line by line while dissect can visualize uh how each line is uh executed. Uh and uh well that makes uh together the dissect and debugger quite a useful combination for for debugging JCode. Yeah, JQT is also great for learning J, as I mentioned. Uh, it includes a collection of uh built-in labs, uh I mean uh interactive tutorials uh for learning J. You can find them in menu, uh Help Studio Labs in JQT. Um and uh some labs are for uh for beginners, they introduce uh to J, but some of them are a great introduction for new add-ons. So labs are useful for both uh newcomers and uh just users who want to uh use a new add-on. Um yeah, and JQT through this window driver API uh also supports uh building standalone applications in J. And one example I wanted to mention is uh MSTat, uh developed by Professor Norman Drinkwater. Uh, MSTAT is used for uh mainly statistical data analysis, uh mainly in uh biomedical contexts. Uh of course, uh using MSTAT does not require any knowledge of uh of J. J was to implement it, so it works behind uh the scenes and powers both the computations and uh the user interface. Uh yeah, uh more more information about uh mstat uh can be found uh in uh uh on its uh on its website. Uh yeah, so that that would be the uh introduction to JQT. Uh another another graphical front end to J is uh JHTP server uh or JHS for short. Um JHS provides a browser-based uh interface to J. Oh, Henry?

Henry 14:40

I just want to stick in that uh JQT has been around for a while. These next two interfaces uh were the uh an object of the work for this release, largely. It's uh these are browser-based products that allow a user who doesn't want to commit to installing J to look at it and see what the fuss is about and then decide to move on.

Adam 15:04

So this is separate from the playground?

Henry 15:08

Yes. There's a playground, but it the playground is a little bit it it's at an earlier release. Things are up to release level.

Adam 15:16

Cool.

Marcin 15:17

Uh yeah, but uh even though it uh is in the browser, it's works. Uh JHS works similarly to JQT. It has many of uh its features. Uh so like JQT, it is uh divided into interactive uh term window and an edit window. Uh and it supports uh many add-ons, including the graphical uh debugger uh and and dissect, and also uh yeah, add-ons for data visualizations, like the ones uh shown on the on the screen. And the nice feature of uh of JHS is that uh the the graphics are displayed uh directly in the term window, uh just just below the lines that uh produce them. Um yes, and and there's uh another uh feature of JHS that it allows you to create uh web applications uh in J. You do it like this: the um you define the application logic in J, and then JHS uh manages which parts run uh on the server and which parts are let's say uh compiled to JavaScript. So as a result, everything is in J and you can build apps without thinking about uh what's in the front end and what's in the back end. Everything is just in J and JHS manages the rest. Uh yeah, and uh also those apps can be easily integrated with uh J database uh J database system, JD, uh or any other, or any other database through through add-ons for managing databases, like uh Open Database Connectivity add-on. Yes, and as as Henry mentioned, it is uh also not only available locally as a part of uh J installation, but also uh in the cloud without installing anything through the URL on the slide. Yeah, yet another interface for programming in J uh is Juno. Uh Juno is uh also a web-based ID that runs J in the user's browser. Um and the J interpreter uh runs as a WebAssembly module uh inside the browser and not on the server. Um but even though it runs uh in the browser, it runs almost at native speed. Um the simplest way to see how how Juno works uh is to click the Do You Know Juno uh tutorial in the right panel. This uh this loads several several short uh interactive chapters that uh introduce uh Juno's features. Um I'll maybe mention some of those features. Uh so uh in the left panel, uh user-defined names are uh grouped in uh uh by part of speech. Uh and uh you can see the rank of verbs and the uh the shape type and the rank of uh nouns and also adverbs and conjunctions. And this is this is very helpful by programming. Um it also helps uh explain the the rank uh concept in J, which is one of the J core features. Um additionally, when you click on one of those nouns in the left panel, uh Juno uh visualizes it uh if if possible uh as a heat map or or a bar graph. And uh the main idea of Juno is to bring uh Donald Knut's idea of literate programming uh to J. This means that uh the program can be divided into chapters, um and each chapter can include uh explanations uh written in uh natural language. Um so in a menu file in Juno you can export and uh import files, and files exported from Juno are valid J programs, so you can run them without any changes in any interface. Um but they also include the descriptions written as uh formatted comments. Um so you can import also any J program into Juno and view it in an elegant, uh elegant way, and I mean divide it into chapters with uh descriptions. Um and on the screen there is an example of uh so you can see the edit window uh in jqt on the left, uh and it shows a J file with uh with with comments, uh and when the same file is loaded uh into Juno, uh then it appears in this nice uh way shown uh shown on the right. Uh yeah, and uh that uh that's the end of the section on J user interfaces. For more information, uh obviously check the links in the video description. Uh most of those links uh go to the JWiki, uh which is a great place to learn more about J. Yeah, and we encourage all users to ask uh questions on the J Forum, obviously. So Henry, over to you.

Henry 20:45

Okay, and bring up the next slide. The fact has been in J for a while, but the um array cast has never supported video before, and it's really hard to uh uh describe something that's basically pictorial in an audio cast. So here we we'll have a chance to show it off to the world for the first time. Um the the idea of two-dimensional interfaces to computers has been around for as long as I've been doing computing. Uh I remember not such items started and uh decision tables uh and other ideas that were kind of data to the database architecture by examples. Uh none of that really worked to point out that the programming language tend to be linear. It answers the question what did that sentence do? Answer the question that wouldn't make any sense. If you're not in a an array language and you take a line of C or Python, it's and you look at the line, you know what it does, but the product is just are complex enough that it is not obvious sometimes exactly what happened, particularly if something went wrong. And that's a um that's a just fact which um Martin has been working diligently to bring it up to date with the following the improvements we had to the J language and it's uh it's still a usable tool and I'll let him talk about it. Go ahead.

Marcin 22:26

So uh yeah we'll look at two examples of dissect in action. Uh here we will use dissect to analyze a sentence with an error. So at the top uh we have the sentence we are analyzing uh and this sentence creates a boxed array of uh shape three uh then each box is opened and uh the content uh is incremented so the error happens uh because the middle box contains text uh ABC so when we try to increment the text uh we get uh domain error uh and let's let's look what uh this act uh shows us about it so in dissect uh the data flow goes from uh top to bottom uh yeah the part highlighted with the blue frame creates uh the boxes uh this these boxes contain the vector 78 uh the text a bc and the atom 42 and the the numbers above the data uh show the shapes of the nouns uh and the numbers uh on the left and right uh of the verbs uh show the effective rank uh of those verbs um yeah so uh the array of these three boxes is then passed to a verb um and this verb is a composition of opening a box and uh incrementing its and an incrementation uh so dissect dissect uh detects the domain error and shows exactly where it occurred um and it uh shows uh two things uh it highlights uh the item uh of the array which uh caused the error uh and also it shows the where the result of this item would appear in the output array um and uh what's uh maybe important about uh dissect is that it's not just a static picture it's uh it's interactive uh for example we can uh select an item uh of the result and dissect shows us which uh which part of input uh produced that part of uh the result um and let's let's try more uh more features so when when you hover uh hover over it uh the matching part of the sentence uh is highlighted at the top and uh when you click on a part of the result that was not computed uh because uh because of previous error uh then a message a nice message appears um yeah so uh so that example showed that dissect is uh convenient for the debugging um but uh dissect can also be used for uh visualizing algorithms and understanding uh what they do and so let's take uh take a look at uh the maximum sum sub array problem um uh the problem is uh is this uh we have a sequence of numbers uh and we want to find the contiguous part uh of this uh sequence uh with largest uh possible sum and uh we want uh this that uh maximum sum as a result um and on the screen there is uh an example uh array uh c uh and uh and uh um example implementation of the solution in j uh yeah so let's check uh the the result so the the the result is seven uh and the contiguous uh subsequence with this sum is uh three minus one four uh minus two three which sums to seven uh and uh we well for for simplicity we add the one assumption that if all the numbers are negative um then the correct result is zero because we can uh take uh empty subsequence okay so let's let's uh see what uh how dissect can help us uh understand what what's going on here uh yeah uh to run dissect in uh in juno just click uh the line number um or alternatively you can use a keyboard uh press uh control k or uh command k yeah so uh so that's uh that's how dissect visualized uh the algorithm um so at the beginning the algorithm uh appends zero to the array uh adding zero does not change uh the result so so we can do this this is just a small small detail um the important part of the algorithm is represented by two uh lower rectangles uh and the the verbs in both of them are highlighted with a blue dotted border and this means uh in dissect that when you click on the result uh dissect will show more information about uh these uh verbs uh so so let's click i will click on uh five okay now now the rectangle representing the the verb uh is split into two rectangles um and the rectangle labeled uh final uh shows the uh the result uh of of the verb um here here the the backslash dot uh is the the adverb in j so so it means the the verb performs an operation uh on each suffix uh of uh of input um and that uh operation which is performed on uh the every suffix is shown in the rectangle above above the rectangle with uh label final uh and here uh dissect also highlights uh the suffix from which the selected value the d5 we clicked uh was produced um that's in the second box on the top right but you can see the blue highlighting around the subway four minus two three minus five three yeah so so the the rectangle uh above the the direct angle labeled uh final uh is uh again with uh blue uh dotted border so we can click it uh so let's do it let's click on five to see more information about it uh this verb by the way uh is uh reduction because it ends with adverb uh written as slash in j. Okay now now now the magic happened so if a few new new things appeared here um and so the verb used in this reduction uh is uh composition of uh of uh two verbs of summation plus and uh taking the maximum with zero uh okay so uh the five we are following from from the beginning uh was now produced from uh two values like in in a reduction so the result computed for the highlighted uh suffix uh this minus two three minus five uh three zero uh and the next element uh of the erec which is uh here four um and uh the result uh computed from that suffix is one pointed by the the hand uh and then one is added to four and the four is pointed by the the other uh hand and uh the sum gives gives five we which was the the number we were following uh so so after that uh we take a maximum of this five with zero which which gives uh again five and this is this five is actually the the number we are uh we are following yeah so so in in this way we can follow this algorithm step by step uh I mean we we got to the innermost part of the algorithm so it's a good time to uh to stop further dissecting but uh obviously we can uh click other other uh um numbers uh other values and see see what happens but I'll I'll leave this uh for for interested viewers um yeah so that's that's it for the dissect uh demo uh this is very cool uh thank you I have been s looking at dissect a few times and I never understood how it works but now now I get what it's trying to tell it it's a very complicated tool but uh in general if you if you want to know where a result comes from click on it and if it's one cell it'll show you where that cell came from or if it's a result of a complicated production like an adverb it it'll open up other windows to tell you what happens.

Conor 31:57

Yeah this is very cool. I was uh at the same time you were explaining this uh I was in Juno clicking along admittedly a slightly different solution that uses the uh fork scan um that's my favorite spelling of it but still it's very cool though like at first you you just control K and a thing pops up and it doesn't look nearly as complicated as this and I was like wait why does his look more complicated and then sure enough you just click a couple times on a number and what's really cool is I mean uh this is a a I well I don't know would you call this a trivial problem maybe maybe not but like you could definitely like so when you're clicking along all the values in a a scan you know you can see oh you know it's it's showing you how the scan is basically working. But you can imagine for like an actual tricky problem that it's hard to visualize what's happening it would be very useful to be able to go in here click on some number that you're like that's not what I was expecting and then just trace back to the point where you realize that there was you know uh what you thought was going to happen differed from what actually happened.

Henry 33:00

Yeah that's that's exactly what worked if you get a recursive algorithm that's particularly interesting because you you can go through the different recursion levels which is normally you know a wilderness of mirrors but you can the fact will keep track of it for you.

Conor 33:16

Yeah I mean it's it's the the really really nice feature about it is that it's interactive. Like I I love and have made a video about the um train trees that both um uh J, BQN and dialogue APL all have in in one form or another and I guess technically CAP also has something like that uh in their GUI. But the nice thing is that like you can drill down like the the train trees are just kind of static, right? So it it might give you a um better ability to reason about what some uh tacit expression is doing. But this is like you can actually click through and see how the calculation is happening which to my knowledge I don't actually think any of the other languages have a tool like this. Adam are you aware of any anything like uh I I guess there's debuggers which is close but not this kind of not not exactly this.

Adam 34:07

There's a um there's an operator in the Defense workspace that comes with dialogue that you can a monetic operator so you can attach it to any function and it will basically do a trace on when that function gets called it just spits out the inside session. What was it called with left argument right argument result. That uh that gives you some kind of insight but it's very a many manual process to to drill into things. And then of course this requires a full run in order to uh to dissect right and uh although you could hit an error and it will catch that.

Henry 34:43

Yeah it'll catch it's nice if it catches the error at the point of the error and in effect it automatically clicks everything you would have had to click to get to the error.

Adam 34:54

So that's very nice so there's a new thing uh in dialect since session 20 of the the um inline tracing that allows that allows you to trace directly into an expression without having to run it until it hits an error and you could you could basically step your way through and at every step of the way you can inspect the current function the primitive and then what its left and right argument are and etc that's gives you this sort of insight but not in a 2D spelled out way. Right.

Conor 35:28

And just I always thought for I I've I can't remember if it was something that Bob Terio, our previous uh producer and uh panelist on the show I can't remember if we talked about it on an episode in the past or I've just seen a YouTube video of his on his channel but there was another tool that I had for some reason in my head thought that that was dissect where it showed a visualization of like the different types as like a different font uh is that a different tool or do you guys know what I'm talking about like or is it the same tool and it's just a different variation of it?

Marcin 36:02

Uh so no no the the the the the tool you are you are talking about is uh the the add-on by by Bob and it is um a different add-on but also uh also um graphical visualization but I think that it uh shows the nouns or uh J nouns and use different graphics for for different types so that you can visualize a noun which would uh normally in uh uh let's say term window just uh shown as uh as text and uh using this add-on you can see what's the type of this uh using by by these graphics. So yeah it's very nice as well.

Conor 36:44

Alright so that's a separate add-on though for so for some reason I had conflated along the way that uh because I guess they're they're doing things in the same ballpark but this clearly is a uh I'm not sure if you want to call it much more powerful but um like this drill down ability of computations is very very very nifty and I'm sure super useful in certain instances. Okay so maybe like let's move on to the next unless there's anything else we want to say I mean I I saw that Adam in the YouTube chat has uh pointed out that dissect's been uh um around for a while and then one of the the users whose name I shall not uh attempt to pronounce has said that maybe they should uh be taking a break from Prologue and uh revisiting Jay so you have won someone over with uh or potentially one someone over with the the the dissect uh tool here all right but it's not so much dissect it's the uh because that has been around for a long time I don't remember it not being around but that's because I wasn't very much into it into it's about 10 years old.

Adam 37:42

Yeah it's it's been a while uh but but Juno is very nice and if I understand right that's Marchin's work to actually make the Juno interface that's right that's Marchin's thing. Um and and having it at your fingertips rather than having to uh write out you know to the to dissect something or choose in a menu or something that having it built in like this is very nice. Um the the I really like the being able to hover over parts of the expression um and just have the boxes in the dissect output jump around to show which parts are sure are representing which part of the expression it's very nice.

Conor 38:24

Yeah web web interfaces uh I think are I'm not sure if they're people underestimate their uh utility but like especially for uh what do you call it uh programming language enthusiasts like there's too many languages for me to like go and install like 45 different you know Scala 2 Scala 3 all the array languages and so to be able to just visit like a online REPL um or IDE almost in this case uh is like a huge win because I don't think I've ever played around with dissector for because I even though I have J Console uh I've never taken the time to figure out how to launch uh JQt um or like because I'm on Linux I don't know maybe I have in the past but anyways it's it's like a couple steps and even those small steps it's not a huge barrier to entry but it's just like it's a it's higher than going to a website and hitting a shortcut, right?

Henry 39:16

So yeah that's what we thought that's why I think it's the most important part of this release are these things that make it easier for people to get to the language. Yeah it's a great language but it does have some barriers to entry and this reduces them.

Conor 39:30

Yeah yeah I mean that's why I've always I've always advocated for the the try APLs and and websites like that because I mean I'm biased that's how I stumbled into them but for someone that's just like language curious are they going to go and download a whole thing in order just to try something out right whereas you know there's that that threshold basically goes to zero if it's just a website and then you can instantly start typing around right it's not like they're gonna become some massive user but it it uh just enables that uh what do you call it like uh off ramp like to the language basically it just makes it super super easy.

Adam 40:02

Martin a tiny bit of feedback here and it's very easy as far as I can tell to go no I should say it is very easy to go into dissect.

Marcin 40:11

Oh yeah I have the exact same but how do how do you get out again very easily every single time I get escape I'm like oh I can't do that I gotta go there's so there it displays the the window like the uh and uh on the right uh top corner there should be an uh X you can click on it and it will use I guess I can I can press tab and then space because like because Mokana mentioned the keyboard shortcut Ctrl K.

Adam 40:39

And then to shut it to to close the window I can't press Ctrl K again I can't press escape.

Conor 40:44

I can't everything I'm biased though I know I'm always trying not to touch my mouse which is why uh whenever I design websites there's always a set of secrets for everything you accidentally hit some key and then something happens.

Marcin 40:56

We'll think about it. Okay. Okay.

Adam 41:02

But it's very nice. And I would also like a little bit more of a guide to how to use how to use Juno more effectively. There is a do you know Juno thing but it doesn't explain everything.

Henry 41:15

Um very nice thanks uh so uh well Henry over to you the next part is about multi-threading okay the the rest of uh what we have to say are uh uh language features that were added for this release threading is uh is not new to this release but uh we have been uh working more and more to make it uh useful um you can go on to the next slide here I think the the um back up the first line that would be um the um J and and for a long time was a single threaded language uh only and that'll I think that uh causes to fall behind the non-interpreted languages which uh the the languages that declare their data uh you declare an array put it somewhere declare and uh once it's been allocated it doesn't move those languages are easier to implement multi-threading and than the interpreted languages. Uh the interpreted languages I'm thinking of Python uh and uh dialogue APL as I understand it uh don't allow multiple threads to run completely independently. Python there's only one interpreter that interprets sentences at least more multi-threading in a sense that uh you might interpret a sentence for a an IO thread that may go up and run a long time but uh you you don't have the the threads all actually running their own interpreter and doing their different things simultaneously uh but more than that you don't want them doing different things simultaneously you want them working on the same problem simultaneously if possible you'd like you want access to uh shared data spaces uh we um so what we've we've had threading for a couple of releases what we've worked on in this release uh is making it easier for a single job a single user task to be run productively on many threads at a time so JSC you can have multiple threads the threads all share j namespaces and they're all executing simultaneously with their own interpreter so then when we have a a multi threaded operator you've got eight cores running you have eight to say eight threads running uh each of those threads has its own interpreter with access to the uh global namespaces and then they stick or not Together only when necessary. In some cases, it doesn't need to be thinking about much at all. Okay, so what we've added uh in particular this release is that um the the basic problem you're encountering was hash coherence. If you if you don't take some steps, the operating system feels that it has license to move your thread to whatever core it wants to run on. So I want to make sure we distinguish between a core, which is a hardware entity, and a thread, which is a flow of execution that has a J interpreter. Um theoretically, any thread can run on any core and get the same result, but it won't get the it won't have the same performance. Because what's vital is uh that each core needs to run as much as possible in its own internal caches, at least level one and level two caches. So what what we'd like to have is a big problem in which threads are tied to cores, and in particular, each thread gets tied to a a subset, an unchanging subset of the overall problem. And if you achieve that, the level one and level two caches of each core have persistence from one call to the next, and the program can uh can operate without expensive accesses to shared memory. So we allow a thread to be type bound to a specific core first, and then the thread can be assigned to a locale, and that locale will have enough information to tell that thread what part of the problem it's working on. When you have that, then you can launch a user task all at once. You can say, here's a task, do this in multiple threads, and each thread will take its part of the problem and return uh the return its results. Next slide there. As an adjunct to that, uh we we have a way, we've added a way to force a noun to be uh not aliased uh and uh and not copyable. So it we call it anchored. If we can anchor a noun, that will force a noun to stay in one place, and and you can only modify it in place, but multiple threads can modify it in place simultaneously. So it provides an easy way for data sharing between the different threads that are being used to solve a problem.

Adam 46:50

Doesn't that mean that uh threads can like stump on each other?

Henry 46:54

Yeah, don't do that. It won't crash.

Adam 46:57

No, we'll just get power data.

Henry 47:00

But if you if you if you have a communication area, you anchor that and then tell each thread to work on a non-overlapping part, then they will act they might be accessing the same noun value simultaneously, but they won't corrupt each other's access. If the regions overlap, then they'll corrupt each other's access, and you'll need to debug that. But anyway, if you do all this, uh you the the the cache for a thread stays hot when the thread starts executing, and the the thread can pick up execution without having to transfer from off core. Uh I have an example of this. I'm gonna try to share my screen here. Okay, so here's a bunch of code. Uh the first half is uh would just be in a library, uh, so you can ignore that. The code starts here where I'm hovering, where it says that the code. Um we allocate a result area of a thousand uh integers and then anchor it to memory. So that says this result area is gonna stay put. And now we're gonna run a program. Uh and here's the program. You know the collapse conjecture? Um it's it's pretty cool. It says you take a number, pick an integer. If it's even, divide it by two, if it's odd, multiply it by three and add one, and keep doing that. Eventually, so the conjecture goes, the result will be one. So that's uh what this is doing is applying that to various integers and counting how many times it went through the loop. So here we have the count the number of loops through the collabs conjecture uh for one value. So it's as long as the value uh has not got to one, you do what I said, increment the count and then do what needs to be done, whether it's even or odd. And the second program is just a loop through doing that. So uh X, the left-hand argument, is the start of the interval, and the Y is the number of values, and it just does a loop calculating the count. So the only thing I've done that's not uh just vanilla j here is I anchored the result area, and I'm using something called an atomically modifiable value, which is like an integer, but it can be modified by multiple threads at a time. So you use you can add, if you want to subtract one, you add minus one to your atomically modifiable value. So here I'm I'm using extended arithmetic, so three to the thousandth power, whatever that is, that's some 500 digit number. Uh and we choose the result area to be a thousand, so it's not it's all a it's we're timing the execution of this sentence, which calculates uh the thousand collapse values. I like to use this when I'm burning CPU time because uh it is a conjecture and it's oh it's possible that it might have never completed, and and we win a million dollars, I think, if we could disprove the conjecture, but unfortunately um the conjecture turned out to be valid again. So it took 16 seconds to do that. The result area is all the values. You can get an idea for how much computation those. So like eleven eleven million uh computations, uh each one of them uh doing uh ration rational arithmetic on 500 digit numbers. That that's a respectable performance. So say I want to run this on multiple threads, this is what I do. All I have to do is say do it on multiple threads. Um and regrettably, I got three, I got I'm running eight threads, I got about a three three X performance increase, mostly because all right, it it's about five X except that uh running this uh screen sharing running this video call takes so much CPU that uh the improvement isn't isn't as impressive. But uh the the key thing is this is all I had to do. I added the old threads that that says is run this as fast on on a mask of processors, and the same program uh it ran the same program in multiple threads, ran faster. Uh now the the result was the same because in this in the first time I ran it, I had one processor computing all the values. The second time I had uh eight processors computing values and storing into different positions, but they each store into the right spot. So I I guess they get the right answer. Uh it's just I don't think it could be any easier than that to run your program in multiple threads. That that's what we've been working on uh for multi-threading in 9.7. On doing this, I've learned some interesting stuff about multi-threading. Um they say, well, let's let's start eight cores, how much faster should it run? Here it ran somewhere around two and a half, three times faster. Usually it runs four or five times faster. Why doesn't it run eight times faster? Uh well, one is that the I'm on an Intel machine and the the E cores have a slower clock than the P cores. But the other one is that as you run more cores, the chip gets hot and the CPU slows its clock speed down. Um so if you're gonna be serious about uh running multiple threads, I think you need to enlist the help of a gaming enthusiast to buy you a cooler for your CPU so you can get maximum performance. But before you can do that, you have to have a language that supports multi-threading, and that's that's what we're continuing to work on to make Jay that language. I think you can pick up now on uh dictionaries.

Marcin 53:21

Um would you like to make an introduction to dictionaries, Henry?

Henry 53:27

L let me come in after you've given the the old-fashioned J pitch. Dictionaries are not a way to do something new. Uh it's a way to do something so much faster uh than it than it amounts to being new.

Marcin 53:46

Yeah, so uh as Henry mentioned, uh dictionaries are not a completely new idea in uh in J. I mean J has always had such uh primitives called the i dot family of verbs. Uh and the way they work is one of the ideas behind the design of uh the dictionary add-on, which is which is new in J 9.7. Uh so yes, before before looking at uh dictionaries uh in the new add-on, uh we'll take a look at uh one of the search primitives, uh and we will take a look at the primitive uh dyadic i dot. Uh so this primitive uh finds the position of the first occurrence of the right argument uh y among the items of the left argument x. Uh so suppose we want to find the position of the first occurrence of 5 in x. Uh so we just write x i dot five and uh get 4 as a result, because 4 is uh the index of 5 uh in x. Um and uh when we write xi.7 uh then we are searching for a number uh in x that uh does not occur uh there, uh and we get the result uh 9. Uh so this is this is because uh if the searched value doesn't exist, uh j returns the length of x, and the length is uh uh is just uh one more than the maximum valid uh index. And uh this idea uh uh like that was the uh introduction, and now it can be generalized for not only atomic uh write arguments. So if the write argument has a higher rank than the items of uh X, uh here it's here it is an array of uh shape 2, uh then the operation uh idot still makes sense uh because it just means that we want to perform multiple searches. Uh and here each cell of uh cell of uh y, the right argument, uh with the rank of uh the item of x is searched separately. Uh and uh the indices, the result uh the result indices are assembled together into an array, uh into the result array, and the result is uh array for nine. Uh yes, and uh we can generalize it uh um for higher rank arrays. Uh so for example uh here x is uh an array of shape 33, uh and it is uh filled with integers from zero to eight because uh we use i dot in monadic form. Uh and array uh y uh is here matrix uh two by three, shape two, three. Um so so this means that we are doing two searches. Uh first we search uh for the array three four five into uh in in x, and it occurs uh at position one in x, and then we search for array two one zero, uh which does not uh occur in x. So the so the result is uh one three. Uh one because uh three four five occurred at position uh at index one and uh two one zero um didn't occur, so so we we get three the d length of law of the dx. Uh okay, so that that was uh an example of uh i dot and uh going back to dictionaries, uh we can we can use it uh for a similar functionality. So if all our keys and values can be uh computed before all the um searches, uh then it is sufficient to use dyadic idot. Um for example, we here we define uh keys and values, and for uh uh simplicity we just uh define values as uh keys plus 10, just to make the the example easier. Uh so we define verb get uh which uh searches uh position of uh argument in keys. This is the uh the right part of the fork, uh, and then selects from values. Uh yeah, so so here you can see that get for 9.2 returns correct result, which is uh these values uh plus 10. Um and what is interesting here is uh that internally um the definition of verb get creates a kind of uh dictionary in in Jngine. This this uh cache table is is built into the verb during its definition and uh used then for for all the searches, uh which is which is very efficient.

Adam 59:10

So just the assignment itself could potentially be slow. Yes.

Henry 59:15

The the you're talking about line three, get equal colored, right? Yeah, the the keys and i dot that is what creates the hash table.

Adam 59:24

Right. It's interesting there's a similar thing in dialog APL uh with the the bind operator, but it doesn't compute the hash table until uh uh you use it.

Henry 59:37

Oh well yeah, this is this is the big improvement. If you're gonna if you have if your keys weren't changing very much, then this this is sufficient. You don't need a dictionary because that that lookup will be very efficient.

Adam 59:50

No, no, it's the same. It's just uh um J is eager, right? As soon as you make that assignment, it detects the composition of keys and i dot and computes the hash table, whereas dialogue is sort of lazy and it it just composes it, and then when you run that function for the first time, it stores the hash table.

Henry 1:00:10

Okay, right. Yeah, we do it.

Adam 1:00:11

We figure it. You have to prime it. Yeah, okay.

Marcin 1:00:17

Okay, so uh so yeah, that that that's very efficient. And uh uh so we can use uh the i.primitive for uh uh some kind of uh our custom dictionary, but uh sometimes we cannot divide the algorithm into these two parts. The first setting up uh uh the dictionary of all keys and values, and then performing all the searches. Uh I mean sometimes uh searches, uh search operations are uh alternated with uh updates, small updates, uh for example, and and this scenario, so queries alternated with uh um updates, uh are the scenario that uh dictionary add-on was made for. So let's start with uh some examples of the new dictionary add-on. Uh so to use the dictionary, uh we first need to load it. So we write load uh.

Henry 1:01:24

Everything you've said is just right, but uh the the key thing here is that when you have an application where the mapping changes frequently and is large, the standard Iversonian methods just don't work. I mean they work, but uh in that example we had where you did the keys ampersand i dot, it has to create an entire hash table. If you've got a million values and you modify ten of them, the hash is going to calculate the whole million values. Uh and that's uh that leads to intolerably slow performance, to the point where there's there are times where you have to change your algorithm because of the uh speed of the the speed or slowness of the primitives. And that's a signal that the language needs an improvement. That this dictionary idea, uh well, generalized, it's the concept of a mapping. Computer scientists talk about ordered maps, unordered maps, unordered ordered sets, unordered sets. Uh this is a a basic need uh in the computing biz, and so uh it justified creating a separate facility that doesn't use the the normal language facilities. It's it's a dictionary is an example of a a shared modifiable object. It can be used many places, it it it accepts changes, it remembers the changes, and allows incremental changes. Uh and the the and we're talking about it wouldn't be unusual to have a hundred million uh uh items in a map. Uh and and we want to be able to support uh uh dictionaries of that size efficiently.

Marcin 1:03:15

Uh so yeah, so the keys and uh values in the dictionary may differ in type and shape, but all keys must have the same type and uh shape, and uh all the values must have the same type and shape. And this is uh this is exactly like uh in the example with i dot where keys and values were were arrays. And uh well this is not uh limitation because uh you can always box the keys or values, uh and the shape uh then then the shape of uh of the key uh will be uh will be empty uh and the type uh box. So so that's okay. Um uh yes, and let's let's create a dictionary. So to create a dictionary, uh we create an instance of the jdict locale uh and pass uh construction parameters. Um and the the only required parameter is the type of uh data structure used by the dictionary. Um as Henry mentioned, there are um two types, two options here, uh hash and uh tree. Um so hash is uh a little faster than uh than tree, but uh tree provides an additional set of verbs. Um verbs for range queries. Uh so the usage uh is exactly the same for both of them, the hash and tree. Uh uh unless you use uh range queries. So uh unless you use them, you can um change hash to tree or tree to hash at any time and uh it will still work. Uh so so that was the required parameter, and all the other parameters are optional, meaning that uh they are set to their default values. Um so for example, uh here we create a tree dictionary uh with all optional parameters set to their default values, um, and we write just tree conjdict, and this creates uh a locale. Uh and we here we assign its identifier, identifier of a locale to a name. Uh, for example, here here d to use then the this dictionary. Uh and I said that uh all the optional uh parameters are set to their default values, and by by default, uh keys and values are uh integer atoms, so they have rank uh zero. Uh and to use different shapes or types, um we use uh uh key type, key shape, uh, value type and value shape parameters. Um We we also have uh shorter abbreviations of those uh uh names. Um you can you can check all the the documentation for further details.

Adam 1:06:10

Um so for example just detect the the type when once you start adding data to it.

Marcin 1:06:19

Uh so so it yes, like specifying the shape is nothing uh new because we to create a multidimensional array in J we also uh say what shape uh it is. Um but yes, here here we also specify the type.

Henry 1:06:36

Yeah, I think it would be dangerous to infer the type because a user might install a one, which would be inferred to be a Boolean.

Adam 1:06:46

I see, yeah.

Henry 1:06:49

So uh you have to give it the type.

Adam 1:06:52

I mean it could it could at a at a performance cost, it could go and upgrade the type later if it then sees that it was.

Henry 1:06:59

Yeah, yeah, but if see, isn't that type you got a hundred thousand items in the tree, and all of a sudden somebody can give you a rational it's understand.

Adam 1:07:08

For ease of usage it would be nice, but of course, if ease of use at the cost of performance, you could just use the boxes, so then everything's fine. That's true.

Marcin 1:07:18

Um yeah, so here in this example, uh we specify um that keys are free by uh matrices of shape uh three two, um, and uh value type of the values as uh rational. So so the keys are here uh integer matrices free free by two, and values are rational atoms. Uh here we don't assign this new dictionary to a name because we will stick in further examples with uh the default case uh of integer atoms. Um so uh yeah, let's let's put something into the dictionary. So we use a verb put, uh and the right argument is uh the keys and left argument is the values. Uh D is uh as as you remember, uh an identifier of the locale of this dictionary. Uh we assign the uh the identifier to this name. So to use this verb uh in uh from from the locale and we write double underscore. That's that's how it's done in uh in J. And now uh to retrieve the values uh corresponding to uh to keys, uh we use verb get. Uh we use we also defined verb get in the um i.primitive example, uh and this is very similar. Um so the right argument here is the keys to to look up, um and uh and the left argument is here optional, so it specifies what to insert in the result uh if a key is not found. And for here, for example, uh dictionary does not contain uh key 7, so so uh the third line returns uh minus one uh in the uh index one, uh the the last uh position. Uh yeah. Uh so so yeah, this is very similar to what uh we did with dyadic idol. Um but uh here when using this dictionary add-on, we can also easily put and delete keys at any time. Uh so yeah, we hear uh after after uh after the get we then put delete and can all after that again get um and query the dictionary. Um yeah, so we we specified the dictionaries uh type as a tree, uh but uh everything uh shown so far is exactly the same for hash. Uh so uh I mentioned that uh what tree dictionary additionally offers is uh queries, range queries. So these are queries uh about order of keys, uh and there are quite a few of them, but here we will just uh show show uh two examples. So here verb mean returns uh array of two boxes, uh and the first box contains the minimal key, and the second box uh its corresponding value. And the second uh here example verb is uh after, uh and it returns at most uh x uh x is the left argument uh so uh at most x keys uh and they've uh at most x keys uh which uh which value like uh which values are greater than uh y. Yeah, so so uh so that that's the the basic introduction to dictionaries. And uh uh well various algorithms uh using this add-on have been uh discussed on uh on the J Forum. Uh and these examples are also included in uh uh in the add-on documentation in JWiki. Um there are full implementations of these algorithms with examples ready to run. Uh yeah, and also interesting use cases of of uh these dictionaries uh have been shown at uh New York City J Users Group meetings. And yes, I think this is a good moment to invite everyone to to New York City J Users Group uh meetings on behalf of Devon McCormick. Uh you don't you don't need to leave in New York to join because all the meetings are uh held remotely. And uh the next meeting uh will be on May 12th. So uh see you there.

Conor 1:12:20

Uh and uh yes, so would you like Henry to add uh I have a quick question before just on the last slide. Is the suffix underscore underscore d uh what's that about? Like is it is it necessary or does it indicate something?

Adam 1:12:36

Think of it kind of as d.get, d dot put, dot in in other languages.

Henry 1:12:44

Yeah, the double underscore is read in after n d. Or you could have underscore underscore d underscore and underscore e, which would be after n d N E. I see.

Conor 1:12:57

And is this is this like uh convention, or it's actually uh like syntactically, whenever you see this, it goes and grabs that from the dictionary library?

Henry 1:13:05

No, no, it's not the this is a language feature. Uh this is in it's uh m much lower than the add-on level. Any any verb, any any name can be qualified with underscore, underscore locale name, and uh that that allows you to distinguish uh different values of the different name, different uh entities of the same name. Okay, yeah. So it it is essentially what Adam said, yeah, just like exactly. Yeah, it's a it's a namespace. D is a namespace. It's it you know it you know about underscore like uh name underscore dick underscore says in the dick locale. It spells out the locale that it's in. That's called a a direct locale specifier. This is an indirect locale specifier where D is a a noun that tells you what locale to look into. I see.

Conor 1:14:01

Okay, that makes sense.

Adam 1:14:03

No, I'm a bit confused about the exact syntax for it because the examples are here, the JWiki use like underscore locale name underscore.

Henry 1:14:11

That's direct locale. That there you spell out the locale. If you if you used after underscore D underscore, you would be looking in the locale named D, exactly D, the letter D. Here, D is a variable, probably containing a okay.

Adam 1:14:30

I see.

Henry 1:14:31

That's how you do objects. If you think of it, D is an object number.

Adam 1:14:35

It's it's a it's a quoting unquoting mechanism. Do I want um the reference that's called D, or do I want the actual thing that has the name D? That's how you distinguish the two. Okay, sort of like the left side side of the assignments where they you can quote them and parenthesize them to sort of say uh evaluate this and the name that it evaluates to, and that's the name I want to assign to.

Henry 1:15:01

Yeah, generally in J, uh when you think of object programming, an object is a locale, is a numbered locale. So it's it's identified by a number as opposed to a name locale, like the the dict add-on has is in the locale j dict, but the objects created by it will be numbered locales that exist in the locale namespace.

Adam 1:15:26

Right.

Henry 1:15:27

And use the double underscore to reference an object.

Adam 1:15:30

But can you can you nest them then?

Henry 1:15:32

Oh yeah.

Adam 1:15:34

That's what I was saying.

Henry 1:15:35

But if underscore underscore D underscore underscore E means first looking E looking D inside E and then looking the E that's inside, yeah, okay.

Adam 1:15:44

Yeah. So the value is the names, one of the lines. Yeah, so it's very much like the dot when you write it like this. Exactly. That's that's it. But but opposite direction. Right. Or colon colon in C or something.

Henry 1:15:56

Yeah, maybe. It's been a while. Um let's let's get to that last slide, Martian.

Marcin 1:16:04

Yeah, well uh the the colon colon would be I think uh for the these locales, so they are equivalent uh in J to underscore locale name underscore and uh more like an object.

Adam 1:16:21

I see, yeah. Whereas the dot the normal dot thing in most languages is like the uh just evaluates that thing. Yeah.

Marcin 1:16:29

Okay.

Adam 1:16:30

I think.

Henry 1:16:31

So to tie it all together, uh work dictionaries and multi-threading. Um most dictionary implementations that I know of do not implicitly um you take note of multi-threading. Uh some designers have even suggested that you should you should keep it that way, but you pay a heavy price if you do, uh, because uh if a a writer if a writer has to block all the readers, one writer can cause a huge hit to the performance because all the readers have to stop. Uh our dictionary implementation is designed to work with our multi-threading implementation also. Um in spite of that, all the reads and writes are atomic. So if if one thread writes a block of data, that block is is available all or none to any other readers. Uh and what what we do for performance is a rather complex locking system. Um basically the the writers uh first arbitrate among themselves to choose one writer at a time, and from that point on the keys and the values are locked independently, right? So that when first the writer gets most of the writing done without any lock at all, and then implements a very brief lock on the keys to update the keys, releases that, and then takes a very brief lock on the values to happen to update them. And the result is that even if you have a writer that's writing as fast as you possibly can, the read performance is hardly affected at all. And even if you have two writers, the read performance is affected just a little bit. Um this is good enough that if you if you've got a multi-threaded application and you have your threads need to talk to each other, a dictionary is not a bad way of deciding to do that. Just create a dictionary and let them put keys and values into it and and communicate that way. I think that's what we got.

Conor 1:18:40

Alright. I mean, uh there was well, actually, I was gonna say one point. I guess this this is like a trend, right? So uh I mean Q and K have always had dictionaries slash tables, but uh now J, I guess, has them. Uh Cap, I believe, has them. Tiny Apple has them, and that Lee and I think actually BQN added them a year and a half ago or half a year ago, and they were working up because it had been like a request out of the Avent of Code. And and Wewa has. Actually, does Weewa? So it might be Wewa and dialogue. Uh although the the Weewains are gonna skewer me if I have uh misremembered that uh they do or don't. Um I know they have the memoize function that enables uh or I'm thinking of the path. Anyways, maybe someone in the chat will um will comment on Weewa. And speaking of the chat, there was a question that uh I read and then um failed to ask when you were talking about um fixing the memory locations of the threads, and Hans asked, uh, does that mean that in theory you could modify a different column, uh different columns of the array at the same time using exactly.

Henry 1:19:56

Yes, that's what you can do. I mean you can also modify the same columns, but then you've you'd you'd have to make your own decision on how to decide which is winning. But if you just allocate different columns or rows to different threads and they properly each one stays in his lane, uh yes, that that's exactly what the example that I put up did. Each thread, after it computed a collapse number, stored that thread into the array at it at the proper index.

Conor 1:20:28

So that is rather um rather uh useful if you've got some computation where uh you have independent things.

Henry 1:20:36

It it's pretty tough on the interpreter to support it, but you, right?

Conor 1:20:41

Yeah, I was gonna say it's it's tough to implement, but luckily uh that's only uh for I guess the J team. Everyone else that gets to use it just benefits from that work. So that's the power of those of these types of languages, right?

Adam 1:20:55

They they are hard to implement. Um but the benefit is to the the user.

Conor 1:21:01

Yeah, I was I always wonder about the uh you know, I I think about how Arthur Whitney uh is is the one that always comes to mind, but you know, implementer infamously of all all the different K's. And I always wonder uh what percentage of the code that he writes in his lifetime has been has been C versus K, uh, because he's writing the implementer.

Henry 1:21:23

And I guess that it's the same for you know you, Henry, and and you Yeah, I don't like I don't write as much J as I w would like.

Conor 1:21:30

Yeah, but like it's it's you would think that you know the implementers are the you know foremost authorities and experts in these languages, but they actually percentage-wise potentially end up writing uh more of the implementation language than they do in the actual language. But I don't actually know. You know, I'm I'm not an implementer.

Henry 1:21:47

Um I think they might be right, but it's I think to I I I was a J user for about 20 years and wrote some pretty big packages. So I I got a good idea of what the language needs before I started fiddling around with it.

Conor 1:22:02

Oh yeah, it wasn't to question uh uh your uh uh t talent for J Code. I meant more just as like you'd think that, oh yeah, you know, the implementer uh spends all their time writing J because that's you know their favorite language or you know they really like it. But in fact, they're actually writing more C code on a day-to-day basis or whatever the implementation language is because that's actually what uh you know it entails. Um unless I guess there are certain things that are written. Um was it was it uh Adam, you've remarked on was it Roger or or someone that most of their work um actually was in the language. Like there's features that aren't always you know live library stuff that's implemented in whether it's APL or J and not actually the source source language.

Adam 1:22:48

I mean the in in dialect there's a bunch of stuff that is implemented in APL. Um but that's still drowned out by the amount of C code. Yeah, of course it is.

Henry 1:22:59

In J, Roger uh very cleverly in the support for sparse matrices I'm thinking about, he that's not exactly written in J, um, but it almost is. It's written in internal subroutine calls to call the J functions. So the the sparse matrix work through all different precisions. You know, you can have sparse booleans, sparse uh rationals, sparse anything. Most of it is being processed through higher level J verbs. Right.

Marcin 1:23:33

I think that's also a lot of uh J add-ons are written in J or uh fully or at least partially to to communicate with some external software, but uh yeah, add-ons are the place where there's a lot of J code.

Conor 1:23:49

Yeah, good to know. All of which are now available via Juno? Is that accurate? Or is that JHS where you can uh access the add-ons?

Marcin 1:23:58

Where well you can access the add-ons in uh well the the the most of the add-ons are available in JQT because of this uh window driver API. Uh but JHS also supports uh almost all of them. And in Juno, yes, also you can uh use uh some of them. Okay, yeah. So all right.

Conor 1:24:20

Well, I mean I we can ask the chat, uh, although a couple questions already were asked. If there's any final questions um from the chat, now's the opportunity to ask uh either Henry or Martin. Adam, do you have any final questions? Um or I mean, should we ask anything about is there uh uh 9.8 beta, any preview of what's gonna be happening over the next year? Is it gonna be more enhancements to to Juno and the um UI stuff, or is it uh too early to tell what the the next year is gonna look like?

Henry 1:24:48

Martin Martin knows is the only one that knows Juno. What's gonna happen to Juno. Uh we haven't decided. Uh there's this thing I've heard about called AI that I'm thinking might be worth getting into. Uh I don't I don't yet know what uh effect, if any, that would have on what we should be doing, but we'll certainly be looking into that.

Conor 1:25:10

I um we have a law when you say uh uh AI, do you mean using AI to implement stuff, or do you mean uh putting some bells and whistles on Jay to make it uh a language that is favorable with uh like LLMs and stuff like that?

Henry 1:25:26

I'd uh well that's the question. If if we could use it to write code, that would be interesting. I I haven't had particularly good luck, although it it does a good job of going through the code base and summarizing things. And in a couple of cases, we we gave it some bug reports to look at, and it got into the subroutine that needed to be fixed, uh, although it didn't get the fix right. But I'm more interested in what we can do to make it easier for our users to use AI to write J. And I have no idea uh what that is. Do we need to write more code to uh to help train the AIs, you know, give it a better uh set of code to look at, or do we do we need to add anything to the language? I I think probably not modify the language much, but we need to be thinking about it anyway. It's clearly um important development in computing.

Conor 1:26:22

Yeah. I saw there's some article I can find later that was measuring uh the density um of like uh LLMs trying to generate code. And I can't remember in the languages that they looked at, Clojure was like winning, uh, which is a Lisp. And then they added a couple updates saying that like a bunch of people have mentioned APL um as like, oh, how come you didn't look at this and they just weren't aware of it? But they said that because of like the Unicode characters, it actually didn't uh because you know the code points or however that works. But then someone followed up and was like, oh, what about Jay? And they redid their little study and uh or program that they were measuring. And Jay won by a long shot. And like multiple people have pointed this out to me. Uh one was on Twitter and one was on LinkedIn about how Jay could technically be like the most efficient way of representing certain programs because it's ASCII and uh it's an arrest.

Henry 1:27:15

Yeah, but uh wasn't there a big push a couple of decades ago about uh getting rid of the programmers and have pro having subject matter experts write code in APL because I think maybe that's foundered because it turns out software engineering really does have an engineering component, or at least it's a a very skilled trade, and you can't just let your subject matter experts loose on production code. But with AI, might that change? Might might it be that you really could have a a user, a non programmer user getting something useful done? I don't know. Uh I I I wanna I think we should be looking into that for the next year.

Marcin 1:28:05

Well I I think that uh generating uh like explicit definitions using those uh uh uh while if conditions uh would not be difficult because uh generating code even in J using them uh is very similar to other programming languages, but what's difficult is to generate a good J code, uh which uh really shows uh the way of thinking in J.

Adam 1:28:34

So I will maybe some people will criticize me for this, but I think in general it would be difficult for uh so what is today called AI, but better called generative AI or even LLMs, is to generate code that shows thinking because they don't.

Henry 1:28:53

Yeah, they don't, but I I was looking at a couple of physics type problems, and I've I've reached the conclusion that with AI you've got a system that can solve any problem that anybody's done before that it can find an example of. Okay, so maybe it doesn't think, but I I think about the students that I taught in high school, you know, just being able to solve every problem they've seen before was all they aspired to, pretty much. You know, is it gonna be on the test? It turns out if all you can do is solve every problem, you may not have an advantage over the AI. Uh it's it's hard to know where this is gonna go.

Adam 1:29:33

It's a whole different subject of what's going to happen when people don't have a chance to learn so on. But uh yeah. I think I mean if you want uh these generative systems to produce reasonable J code, you had to publish a lot of J code.

Henry 1:29:52

I think that's right.

Adam 1:29:53

Of course, publishing a lot of J code makes little sense since J is so compact, just like any of the other array languages. So it's the there will just be less code.

Henry 1:30:02

Right.

Adam 1:30:03

Um I also think there's uh there are some various fundamental issues that the same code can do many different things in in most array languages. There are no generally no hard types and things are polymorphic in various ways. And that's a problem for these systems because, as you say, they they just uh munch a huge amount of data and spit out something that uh uh is composed from bits and pieces of everything that's been munched up. But the more specific things are uh basically that's metadata, the the easier it is to retrieve that. But if you've got code that is universal, it's not very easy to identify what exactly is it for. Um it can be used for multiple things, and I think it makes it harder for these systems to retrieve the right code for the right situation. You can't just use types as a hint to needing this code over that code, it can't just use descriptions of uh intermediate variables and so on as hints as to what it needs to be retrieved, because the names that array programmers will use with often terse and uh and not very obvious what they mean, they might not mean anything very concrete, or they might be very general because the code can be used for so many different things.

Henry 1:31:24

Or the other you may have a partially tacit line of code and the names don't even exist.

Adam 1:31:29

Right, exactly. And and then and for tacit things uh in in some of the languages like APL and J, uh, they will generally mean two different things in the same code. Whether you're using it monetically and it's not obvious in context, um, and and not being statically parsable, this makes the problem even worse. Out of context, you you might not be able to tell at all what the code is supposed to do. All of these things make it harder.

Henry 1:31:54

It does, but but the the models are very resourceful. Uh for example, we might say, okay, it turns out what we need to do is put the word monad or diad at the beginning of every verb. You know, give it a hint. And maybe maybe a few coding practice changes would would make it easier for the LLMs to make use of our code.

Adam 1:32:16

So so I think what could help is annotated code. I mean code that with a description as to what it does. Just like these systems needed for a long time uh images with people annotating what's on the image in order to learn to so sure or should we say to fake it, uh to be able to identify things on images, seeing the patterns and and later generate those things as well. Um so too, they need annotated array code that says, okay, this when you see this, that you can sort of infer that that has to be called analytically or dyadically, and and um something I have in in Apple card is having the same entry multiple times with different descriptions, because the same code can do all these different things that are not obviously the same thing, um, even though the APL code is the same thing. And so having multiple descriptions on the same code can help those hint to such systems that yeah, this is the right code for all these different problems.

Conor 1:33:12

There are I guess there's one question. Well, I mean, there's two questions and one comment, but uh for the question the first question that Max has asked what's the style of C code? Uh you can technically just go look at the open source uh J GitHub. Um and the other question is maybe unrelated, but there are are there any future plans to make the J build slash install system more friendly to foreign package managers and maybe make Pac-Man uh in parentheses J's one support user setting paths?

Henry 1:33:45

Uh yeah, that's we have a uh a work item on the GitHub issues list, and uh if you have any I I don't do build tools at all. They terrify me. Uh but we have people who work on them. Uh and if you have suggestions, I'm sure they would like to hear them. Because we we do want to make sure people can build Jay releases.

Conor 1:34:10

Okay. And uh the one comment is uh from Hans. Personally, I would love to have videos about small problems and solutions in Jay. I've read much about Jay, but some videos like Adam's uh for dialogue are very helpful.

Adam 1:34:25

Okay.

Conor 1:34:26

Which I think, I mean, Bob Terrio has his YouTube channel, although I don't think he is posting new videos. Um and that channel, I'm not sure how discoverable it is, but definitely if you if you search his name on YouTube, but potentially you haven't stumbled across, or actually I can go find the link and put it in the chat. Um, but I know that there are there is at least Bob's channel where um it's not exactly the same style as Adam's um videos. What what was the was it Apple Quest APL quest? Was that the the name of the series that you did for um all of these problems? Um it's not exactly like that, but it still uh uh might scratch the itch that you are looking for.

Adam 1:35:04

I think we have like a a crossover of uh concept problems between APL and J. APL has got a ton of very simple problems being solved and explained, and J has fewer of those very simple ones and more of the larger lab type things that are available. Those are missing in the APL world. Uh doing something more with APL. So maybe we should cross-pollinate. Yeah, sounds like it. All right. Because uh obviously a lot of things that APL will be good at, J will be good at, and vice versa. This is the nature of things. Yeah. Uh Connor, one thing you mentioned before was uh we were in and dialogue uh not having dictionaries. I don't know, but we were at dialogue we're sort of reluctant with regards to dictionaries because we've got namespaces already. So namespaces are like the locales, and as you can see here, syntactically the the J dictionaries have sort of been built as uh as locales uh on the locale system and the object system.

Henry 1:36:09

Yeah, like it's uh it's that's not really true. We access the dictionary through an object, but the actual work is done through forens that bypass the usual protections of so it normally if if two different threads modify a a noun, uh something bad's gonna happen. One of them will overwrite the other. But uh for the for the dictionary, there's not a specific dictionary data type that we access only through functions that know to take locks and protect against problems. So it's it's a pretty big change, a big step up from just having a namespace.

Adam 1:36:54

Yeah, so so that's the thing. Dialogue only has these namespaces, and that means that has the syntax that you might want to use for the dictionaries. The problem is that there isn't an efficient and general dictionary support built into the system that could be added in the same sort of way, where we we pretend things are namespaces, but they're actually dictionaries, so you have some kind of way to tell the system, oh, this thing that looks like a dictionary, it looks like a namespace that's actually a dictionary underneath the hood. There has been some proposals to do something called magic arrays, where you sort of define a class that uh and how it should behave outwardsly. Um and you can to a certain degree you can already model this today, uh but you have to build it up yourself. So there's definitely room for such a thing because namespaces are slow and they're to use as dictionaries and they're not general. You can't just use any value as a key. Um so I wanted to ask one more thing about the dictionaries. Oh, right. Um notation. So what's next? Just like APL got array notation. Maybe J needs a notation for the dictionaries, so you can you can write them out, right? Instead of constructing them piecemeal.

Henry 1:38:07

I don't understand.

Adam 1:38:08

Right now, if if I understand right, you can't write a dictionary, right, in J. You can only construct it.

Henry 1:38:16

Well, uh when you say write, you mean r uh like a a literal dictionary.

Adam 1:38:20

The way you can write a literal character vector and then you can write a literal numeric vector, but you can't write uh a dictionary, right?

Henry 1:38:29

No, you have to uh we do we have load and unload verbs that you can you can take everything in the dictionary and dump them, collect all the keys and all the values and dump them into an array and vice versa. But in in general, since uh the dictionary is just a key value combination, if you give it all the keys and the corresponding values, you you have constructed the dictionary. They and for a hash map, that's as fast as anything. For the uh ordered maps, uh it it uses a faster method if you tell it the keys are already sorted in order, so it it can generate the search tree directly rather than having to go through n log n operations. But that's the equivalent to bulk load and bulk unload in a database environment.

Conor 1:39:19

And there's one last question, uh, I guess before we we wind this down. Um and that's from Hans again asking, are there any conferences where J talks can be found from time to time? Um Raycon con Yeah, I guess we all know the answer to this question is that currently in 2026, there unfortunately are not any that at least I'm aware of. There's some secret conference that somehow is slipped by not just me, but the entire uh array cast uh panel. Um there used to be uh the J software conferences that I think ran up until the 2000s, maybe there was even one in like 2010 or 12 or something. Um but those do not run anymore. And um the closest probably you can find, unfortunately, are the talks that I have given at random functional conferences that are not about J, but they mention array languages in general. Um and yes, Adam was referring to the fabled, the mythical uh array con that we um have always talked about uh having, and at one point, you know, I think actually uh Steven, former panelist as well, he had gotten kind of the motivation to maybe do something like that, but then I think it just ended up turning into another Iverson College, Iversonian College. Was it Iversonian? No, Iverson Iverson College. Yeah, Iverson College, yeah. But yes, I mean if I somehow magically get a bunch more free time, um uh it would probably start just as a virtual uh conference. And the idea would be that you know we'd invite some people for maybe the first edition to give you know a J talk, an APL talk, uh We WA talk, etc. Um, and then if that went well enough, potentially try to organize something in person.

Adam 1:41:16

But it's just it's a lot of work to Everson College wasn't that in my opinion. It's it's it's different, it's a get together but less of a conference.

Conor 1:41:26

It's an unconference, technically. Yeah, no scheduled talks. There are talks given, but it's like um, you know, very informal. Um anyways, and even better than us organizing one, it would be if someone out there just did all the hard work, just like Henry and uh Marchin and the J team have done all the hard work on the J interpreter. If someone else wants to set up a conference, I think we would all be uh very excited uh for someone else to take that uh task on. I would love to do it if I had more time. Um, but unfortunately, there's not enough uh hours in the day. Anyways, I I think that is uh concluded all the questions in the chat. Thank you for everyone in the the chat for uh attending live and contributing to the discussion. And uh most of all, thank you both to uh Henry and Marchin for coming on. Um I'm sure at minimum we're gonna have uh potentially both of you back on in a year, if not uh sooner than a year. I think usually we've had Henry on to talk about different topics tacit in the past. Um we still have the uh contact at raycast.com uh email if you want to reach out to us there. Um but we also I mean we don't plug it very often, but we do also have a blue sky account if you just want to message us there. So there are many ways uh to get a hold of us. Um and yeah, thank you both once again, Henry and Martin, for coming on and making this awesome presentation. It's probably been the first, if you count it as a talk or a presentation, uh first J talk in a while since um I know there's the famous Tracy Harms, I believe is his last name, uh talk that is floating around on YouTube. But maybe this is the the first one since then. Um so yeah, hopefully uh we'll we'll have you back in the future. And I think with that we say happy array programming.

Adam 1:43:11

Happy array programming. Yeah, well, we really messed the timing on that one up. I didn't pick up on it. That's yes, it's when you when you say and with that, we'll say that then I should immediately answer happy array programming, and then we'll line up. But uh yeah, we'll not fix it in post. We nailed the last couple, so it's all right to have one that's uh a little bit off.

Conor 1:43:31

Thanks everyone for uh tuning in.