Exploring AI Matters
Our mission is to help the policy community understand the breadth and richness of AI and the potential for such technologies, wisely applied, to augment all sorts of human endeavors.
Some AI tools are able to assist humans in performing tasks faster, more accurately, or more efficiently. Some, however, are inaccurate and unreliable. Who or what we hold accountable for these flaws, and what incentives we do or do not create for their correction will influence AI’s hand in how we work.
In this series we will refine, sharpen, and clarify your understanding of AI.
Exploring AI Matters
Episode 4 - The Story of How AI Reconstructed a Rembrandt Masterpiece
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Send us comments and questions.
In the early-eighteenth century, Rembrandt’s largest and most famous painting, The Night Watch, was cut down on all sides to fit on a wall in Amsterdam’s City Hall. The removed pieces were never recovered. In this episode of Exploring AI Matters, Rob Erdmann, Senior Scientist at Rijksmuseum and Professor at the Rijksmuseum and Professor at the University of Amsterdam, walks us through how museum scientists trained AI to successfully paint like Rembrandt to reconstruct the lost fragments of this masterpiece more than 300 years later. [2022-08-10]
Welcome to Exploring AI Matters. This podcast series, previously known as Mind the Gap Dialogues on Artificial Intelligence, will continue to appear in the ABA series to the extent that, in addition, all of the episodes, old and new, will now appear under our new podcast name, Exploring AI Matters. Thank you.
SPEAKER_02While AI researchers have built software that tries to create art, the results have been profoundly disappointing. Weird dramas, strange poetry, and sounds that only the authors would call music. In this episode of Mind the Gap Dialogues on Artificial Intelligence, we will consider the application of AI techniques to help people better understand one of the greatest paintings in the world, Rembrandt Van Rijn's The Night Watch. This famous painting is the premier cultural icon of the Netherlands and the centerpiece of the Rijksmuseum in Amsterdam. The painting has a long and complex history. It has been housed in several locations over the centuries. It has been the subject of numerous conservation projects over its nearly four centuries of existence. One of the less known episodes in the painting's past occurred in 1715 when pieces of the colossal painting were cut off in order to fit it into the Amsterdam town hall. The cutting was particularly egregious on the left side of the painting, whence the images of three people in the original were lost. While the removed sections of the canvas have not been found, a small copy, which survives, was made by another painter not long after the night watch was completed. Working from images of the copy along with remarkably detailed photographic images of the original, Dr. Robert Eerdmann has used AI to approximately reconstruct the lost portions of the canvas. Dr. Erdman is the senior scientist at the Rijksmuseum and also a professor at the University of Amsterdam, where he holds appointments in two departments, the Institute of Physics and the Department of Conservation and Restoration of Cultural Heritage. The objective of the reconstruction work was to make them as similar as possible to the original. In today's episode, we will discuss with Dr. Erdman how he went about this remarkable and inspiring recreation. Hello, I'm Mark Donner, a computer scientist.
SPEAKER_03And I am Roland Trope, a national security lawyer. We are your hosts for this episode of Mind the Gap Dialogues on Artificial Intelligence. In addition, we have two more hosts. Good evening, Dr. Erdman. How did you get involved initially with AI?
SPEAKER_00Well, AI is something that I've been following for quite some time. In my previous position, I was a professor at the uh University of Arizona, working on completely different things. Um, but among them was to try to understand the relation between material properties and material structure. And um, there were such amazing advances in AI at the time that I figured, well, I should I should probably learn about this field and see if there's some way to apply it to the physical sciences. Um, so back then uh I didn't really have access to the right hardware or um really the developments that enable the application towards physics weren't really there yet, but it's something I kind of uh uh kept my finger on uh to monitor it. And um yeah, it's it's had such a massive explosion in utility all across computer science and now in the conservation world that I've just decided to stick with it and learn everything I could about it.
SPEAKER_03So is it fair to say that you you got interested in it early on and you've kind of self taught yourself, or have you taken courses also to get up to speed? Because that sounds like a formidable challenge.
SPEAKER_00Yeah, it turns out that most of AI, the basic concepts are actually quite simple. I mean, if you understand linear algebra, and and I did because I was also uh professor in applied mathematics um at my last university, um then you you really have quite a head start. Um, but certainly um any online class I could get my hands on, I've I've taken it. Um some of the um Stanford courses by uh Andre Karpathy uh about um convolutional neural networks for um for computer vision and so on were um classes that I took when they first came out. Um and yeah, really uh these days um there are so many excellent tutorial videos or entire courses on YouTube that uh anyone who really wants to can do it, I think.
SPEAKER_03Could we ask you, how did you come to be working as senior scientist at the Reichs Museum? I didn't even know they had a senior scientist.
SPEAKER_00Um yeah, the the story is uh is long and winding, so I'll I'll abbreviate it. But um basically, in my my last job, I was working on something completely different. Um basically, my main project was um solidification of uh aluminum alloys in microgravity. So I had an experiment on the International Space Station, um, yeah, about as far away from museum things as you can imagine. Um, but one of the uh women I shared an office with in graduate school actually went on to working in the conservation field, and I ended up visiting her um at the Art Institute of Chicago and got a really uh really nice behind-the-scenes tour. Um and before this, I didn't really have any concept of what happened at a museum. I sort of thought of an art museum was the place where art went to die. Um, and so this really opened up my eyes to all of the amazing things that are happening behind the scenes to make sure that the museum continues to run, of course, but studying um all of the artworks, understanding where they came from, um, what has happened to them since they were made, what will happen to them under various conservation treatments and so on. And at that point, I decided, wow, this is this is so cool. I mean, I I had no idea that people did that for a living. So I decided to um sort of fake it till I made it, um, as they say. And um, I had been doing some really large image processing work for my my uh space solidification experiments. And so I asked if there was a way that this might transfer over. And it turned out that they were struggling with stitching x-ray films because when you x-ray a painting to see what might be painted painted behind it, um, you use chest size x-rays. So if you have a really big painting, then you have a lot of x-rays. And um in those days, this was 2004 or five, um, that's really a lot more than the average computer could handle. So I sort of injected myself into that. And um, in the end, uh I got to serve um as a resident fellow here in the Netherlands, um, working with other museums to try to apply imaging and computation technologies in the museum field uh at the Netherlands Institute for Advanced Study. And at the end of that, I was basically invited to come here permanently. So I agreed to do so if I could uh somehow find a position in a university at the same time as a museum, because I love the academic freedom, but I also wanted to make sure that the work I was doing was directly immediately applicable instead of you know writing papers about what I would do. I wanted to make sure that it was in the hands of people who were going to use it now. Um and uh here I am, seven years later.
SPEAKER_03Before you took the position, was there a senior scientist or did they create the position for you?
SPEAKER_00Um there was not a senior scientist position. I mean, now we have a science um department with a number of student researchers and postdocs and so on, but at the time I was the only scientist at the museum. Yeah.
SPEAKER_03In our preliminary conversation last week, you mentioned that the reconstruction effort was part of a larger project involving Rembrandt's The Night Watch. Can you give us some idea of the nature of both projects, their relationship, and what the uh comparative efforts are in each?
SPEAKER_00Yeah, so um as mentioned, the the Night Watch is um probably the most important uh artwork in the museum. A large fraction of our visitors come uh specifically to see the painting. Um and uh the museum is almost a cathedral um built around this painting. It's at the end of this long gallery of honor. So um we are in the middle of what we call Operation Night Watch, um, which is a multi-year, multi-million euro uh initiative split into two halves. Um, the first is a research half, and the second is a conservation half. And the idea is um that we will uh sort of think about taking a patient to a doctor. Um, the first half is like conducting a full body scan. So we're gonna try to use every scientific technique available, all the most modern imaging techniques and computation where appropriate, to um obtain a comprehensive understanding of what the state of the painting is now, how it was made, how what layers uh have been built up, what conservation problems may exist. Um and then we're done with that first half. And the second half is um to be completed, uh, and that will involve a uh conservation treatment of some sort. So we're still in the process of digesting all the scientific results and deciding what kind of conservation treatment will be appropriate. Um, and this whole operation, what's unusual about this is that the whole thing is done in front of the public. Um, so there's a big glass house, as we call it, a four-sided um glass enclosure that's built around the painting. It's quite a large um picture, it's uh four and a half meters wide by um uh so that's what 15 feet. Um, and um uh a little under four meters tall. So it's a really big painting. And uh all of the research as well as all of the future conservation treatments will be done inside this glasshouse in front of the public while they watch. So um the the reconstruction, this missing parts that you mentioned, um, is being uh displayed to the public right now in this three-month sort of interregnum between the uh research phase and the conservation phase, while we digest the research data to decide what the conservation treatment will be, with the help of outside experts uh and uh people from many different disciplines to make sure that we really make the very best conservation treatment that we can make.
SPEAKER_03Can I ask two other brief questions for sure for people who may not, and I realize this, you know, it's possible that um a lot of people may not have seen the Night Watch. What's the painting of, just so the audience gets a sense of it? Why is it such a cultural icon in the Netherlands?
SPEAKER_00Um yeah, so the the official title of the painting is is quite a mouthful, but essentially it is a um militia company portrait. It's its uh actual name is not the Night Watch. Um, but what it shows is um the militia company for the 2nd district in Amsterdam. And so at the time um you had sort of civic guards, um, each of whom, which would be organized um from members of the public, each of whom was responsible for uh protecting uh or defending a certain area of the city. Um so uh the tradition at the time is that they would commission a portrait of themselves, each company, um, with the most prominent um figures, you know, the leaders of the company uh in front. Um and the typical militia company portrait would be a bunch of these guys sitting around a table um, you know, drinking beer and stuff. So um Rembrandt decided to, as he often did, to turn the whole genre on its head and to make um an action shot where they're basically um uh the the leader uh Franz Banning Kok um and his lieutenant um Willem uh van Rasenberg are um standing uh near the center of the painting, and uh and Rosenberg is uh is um uh is getting ready to follow, and uh Banningkok is gesturing out to get his whole company to march out. So they're basically going to march out to defend the city. Um it's a very raucous scene with a lot of movement and so on. Um, and it's come to sort of symbolize um, you know, Dutch civic pride, um, as well as the fact that um the the Dutch um yeah, they really prided themselves on having this um locally organized militias instead of you know being a military power, they were a trading power, of course. So it it sort of symbolizes the Dutch way.
SPEAKER_03One one last question is transition to Mark. Is this the first effort of its kind that you know of, either in the Netherlands or the world? So the everything you're doing, uh you're sort of having to invent for the purpose, or are you building on other conservation efforts that have been done elsewhere?
SPEAKER_00Well, as far as the the reconstruction of the missing pieces, um, I think that's that's new um because this is a very unusual situation, of course. The fact that the Night Watch was was cut down, um, that is not so uncommon. Many paintings were cut down. Um, but the fact that um someone made a copy of it, a fairly faithful, um semantically faithful, if you will, copy of it before it was cut down, uh, is very unusual. So there was not really much to draw on where you would try to use a painting from a different painter before the damage to try to learn a translation back into the style of Rembrandt to help envision what those missing pieces might have looked like before they were cut off.
SPEAKER_02In the beginning of the reconstruction, you have two collections of images, one of the surviving area of the Night Watch and one of a smaller copy made by another artist. Can you give us an idea of how you reconstructed the missing pieces?
SPEAKER_00Sure. Um, the the smaller copy is made by a painter named Gerit Lundens. Um, and as far as we know, it was commissioned by um Franz Bonnikoch, the the main um figure. And uh it was commissioned to sit um to hang on his wall. So it's less than one-fifth full scale. Um, so what you have here is um a different painter. The painting is on panel and wooden panel, whereas the night watch is on canvas. Um, and it's a much different scale. And of course, when you paint something smaller, you leave out certain details, and the way that you paint faces is quite different than when you paint them large scale. So the challenge is to overcome all of these differences, essentially. You have a different palette that the two artists used. So the colors are different, you have different painterly style, you have a different texture because one is on canvas and one is on panel, um, and they were using different size brushes and so on. So, broadly, I uh I approached this task in in three main steps. The first was to do what I um would call semantic matching. So the copies that were made, the copy that was made, I should say, was faithful in the sense of leaving in all of the things that were in the night watch. You know, if there was a button on a hat, then he also painted a button on a hat. And if there were four um stars over here, he painted four stars and so on. But geometrically, it was not um a perfect uh grid scale copy. So you can't simply blow up the copy and lay it on top of the night watch and have it match. So if I'm going to have the computer recreate the missing parts, I have to have it so that those missing parts are perfectly contiguous with the existing painting. So I can't just you know blow up the small copy by London's um by 5X and then translate those parts and stick them on the painting because then they won't align. Um so I have to make sure that things geometrically align. And so the way I did this was to uh again, there's three neural networks that are at work here. The first neural network was trained to identify semantic matches between the two paintings. So that is to say, oh, there's a face and there's an eye, and there's a pupil of the eye. And here is the same face, although it's has a different appearance, of course. Um, semantically, it's the same, it's the same guy, um, and the same eye. And uh, and so I can make 10,000 or so matches, you know, the tip of the spear, the tip of the spear, um, the heel of the boot, the heel of the boot, and so on, across the entire painting. And so this set of semantic matches would then allow me to solve for the kind of deformation which we call a homography, but it's essentially um all the kinds of deformations that you might see if you were to move the camera. That is to say, if you move the camera closer or farther, you get bigger and smaller images. If you tilt the camera uh around its main axis, then you get rotation. Um, and then you can tilt the camera, you know, yaw and pitch, and these will cause perspective distortions. For example, the parts that are farther away become smaller, and the parts that are closer become bigger. So these are broadly called homographies in mathematical terms. And so I then use this set of matches to have the computer solve for the best homography because various things could have happened to the Nightwatch since it was painted. It might have been stretched, it might have been skewed and taken off of its strainer and so on. So I don't have perfect confidence that the shape of the Nightwatch now is the same as it used to be. And also, um, it seems uh that there's there's some strange deformations in the upper right corner of the London's copy that um give a sense that he was sort of trying to use perspective tricks because he couldn't measure it. So the bottom half of the painting is quite geometrically faithful, and it seems that the faithfulness geometrically kind of falls off above the height that one might be able to reach with uh a measuring stick on the painting. This is just hypothesis, but it seems also that London's uh had some perspective effects, um, unintentional. Having done that, then the two images were uh much more closely aligned. But still, if you look closely, you can see that London's sometimes makes a face too narrow or he stretches it out or the head is tilted slightly and so on. And what I would like when I'm teaching the computer in the final step to translate from this copy to Rembrandt is to understand how to translate the palette. I want to understand how to translate the texture, um, uh the sort of um brush strokes and so on. But I do not want to teach the computer bad habits about stretching faces and things like that, because I don't have enough faces where I have London's painted it this way and Rembrandt painted it that way to teach the computer any kind of sense of what the right thing is to do. You know, I there's only 17 people in the whole painting. So, what I wanted to do was to make a training set that could teach the computer to the things that I want without teaching it bad habits. So that's the second part. Uh, the second part was then given these pretty closely aligned images on the big scale, to zoom in and then to have the computer tweak and nudge and pull like a rubber sheet, the copy, so that each of its details was perfectly falling on top of the corresponding detail in the Rembrandt painting in the Nightwatch. So this would then mean that I had a pair of images, one from the copy, one from Rembrandt, where all of the features are aligned, but they have different colors, they have different textures, they have different brushstrokes, and so on. And then this was uh the input to the third and final step, which I call sending the neural network to art school, where now, given these pairs of images, I could cut them each up into thousands of little overlapping tiles. Um, and you can think of this almost like flashcards, where you could say, well, here's the front of the card. And this is how Rembrandt painted the detail. And on the back of the card is how London's painted the same detail. I uh designed a neural network that would let the computer guess what the Rembrandt tile would look like if it was only shown the tile from the copy. And after it got um sufficiently good at this, in fact, when it when it couldn't get any better, so that is to say the training was done, then I could use the trained network to translate the entire copy into the palette texture and so on of Rembrandt. And then finally, I can throw away the middle because I've already got the Nightwatch and I don't need the inferior result of translating the London's copy to a Rembrandt-esque style, but I could retain the outer edges, which then were geometrically consistent with the original, so that it would smoothly fade. It had all of the right colors, uh, it had the right textures. In fact, a delightful little side note is that the neural network learned to hallucinate uh the crack lure. So it's it learned in its uh reconstruction to put cracks in the paint and to put it to make them the right color and the right scale, even though they weren't present uh in the small copy, which is on uh, of course, the cracks there are much smaller, but they have a different kind of geometry because it's on panel. So uh in the end, the the idea is not to recreate the genius of Rembrandt, because I think that's impossible, but rather what we wanted to do was to give an impression of what the painting might have looked like if it had not been cut down.
SPEAKER_02Wow. Um can you give us some idea of the sort of insight into Rembrandt's use of color uh you were able to infer from modeling and training the color transform?
SPEAKER_00Well, unfortunately, um the neural network that learns to make this translation doesn't separate these factors. So I don't have um color as a separate thing from texture as a separate thing from and so on. So really it's um this is oversimplifying, but but broadly it's a kind of black box where one image goes in and that's the tile from the copy, and another image comes out, and that's the network's imagined Rembrandtization, um, if you will, of that tile. So I can say that it was interesting to see that the network learned um how to imitate the kind of Chiara Scuro shading technique for which Rembrandt is is famous. Um, you know, Rembrandt's figures have this um beautiful glow. The faces kind of have a mysterious glow to them. In fact, in in the film business, they even talk about Rembrandt lighting. Um and the background is um is quite subdued. And if you look carefully at the painting, you can see that the figures who are in front are very vivid and bright and saturated. And then the farther you move back into the painting, the more um sedated and brown and uh and unsaturated the figures become. And so this is just a trick that you know Rembrandt was very good at, but other painters knew as well about how to um kind of imbue a sense of depth into the painting that wouldn't otherwise be there. Um, and the thing is that uh London's was not so good at that, or at least at this scale, he didn't really practice it so much. So the network had to learn how to take the sort of relatively flat um images from the copy and make them more chiaroscuro shaded to make them more believably Rembrandt-esque.
SPEAKER_02So you mentioned training neural networks to predict the geometric and color and texture transformations. Were these neural networks off-the-shelf software? Were they difficult to apply to your problem?
SPEAKER_00They were not off-the-shelf, but um, these days there are a few key architectures, if you want. So when you're making a neural network, there's sort of two big parts to it. There's um the so-called architecture, which is kind of a network diagram of how data flows through it and data is transformed as it passes from node to node, from the input through possibly several hundred layers of nodes to the output. So that's the architecture. And then you have what are called the weights, which is the result of training the network to solve a particular task. Um, so there are certain architectures that are well established. Um, for example, I used a 50-layer res net, which is a residual network. That's this is the kind of network that's used to train computers to to recognize uh to perform classification, which is to say, to recognize what object or what thing appears in a photo, like to say, oh, well, that has that's a fire truck, and that's a kitten, and and that's a puppy. So you can you can take a network like that, and you can kind of copy it and wire it up in a funny way called a U net. Um, and a U net, that is because it's shaped like a U. So you you you start from an image and it distills it all down into its essence, if you want. It places it into a latent space, um, and then it extrapolates back out to being a full image. So I I utilize these kinds of architectures, which are well known in the literature, and then um I had to tweak the wiring a little bit, and then I had to train them for my particular use case. So, yeah, I mean, the the total number of lines of code is probably fewer than 200 for the whole project, um, believe it or not. And that's because um there are very powerful libraries now that let you build these neural networks at a very high level of abstraction. You can just sort of say, well, look, here's a part, here's a linear layer, here's a convolution layer, and so on. And then the rest is taken care of for you and compiled down into extremely efficient uh programs that will run on your GPU, your graphical processing unit, um, which makes them extremely fast. So I use PyTorch, which is one such um uh library for doing these, for designing these kinds of neural networks. Another big one uh is called TensorFlow. So PyTorch is was originally made by Facebook AI. Um TensorFlow was from Google. So you you have these sort of big players who are all trying to um win hearts and minds by by making very easy to use high-level neural network software so that a guy like me can still be in the game. Roland.
SPEAKER_03Can I ask a very briefly digress into a legal question? You mentioned that there are 200 lines of code only for the whole project. Does somebody own that? Is that owned by the Reichs Museum, by you, or is it retained by the people who license you the software?
SPEAKER_00All the software that I use is is free software. So um that's just kind of an ethic that is in the AI community broadly. So all of PyTorch is free and open source. I have an arrangement with the Reichsmuseum that enables me to retain uh copyright on on software that I write, basically because the Reichsmuseum says, uh, look, we're a museum, we don't need it, we don't care. But but under normal circumstances, it would be owned, um, copyrighted by by the Reichs Museum. But but yeah, I mean, I I don't consider it proprietary in any sense. I mean, anyone who wants the code can ask me and I'll happily give it to them because I'm uh you don't work in a museum to make money.
SPEAKER_03Um I I I I want to add a little more context to that. I I recall in our previous conversation, you mentioned that the Reichs Museum has made available to the world its high definition uh images and has not copyrighted, those are charged access fees, which is very different than some other institutions. Can you just give us a little bit of an insight into their thinking? Because that's an enormous policy decision.
SPEAKER_00Yeah, um, the Reichs Museum, I think, has been the world leader in this. Um other museums have followed. You can now do the same thing with images from the Met, for example. Um, but um many years ago, um, before I came to the Reichs Museum, so before 2014, um, you know, we we we've always had photographers who photograph our collection, but um many museums have you know licensing departments, you know, pay us so much to license a copy of your image. And um the fact is that if you search Google for an image uh of a painting, you can find it. You can find it and it will be bad color, it will have bad resolution, it will have been through the the JPEG compression mill 17 times, you know, it's just not good. So, in general, um both from a philosophical as well as a very Dutch pragmatic side, it made sense to make them all free and open because we want it so that if someone has an image of a painting in our collection, it looks beautiful. And then people think, oh yeah, I want to go to the museum and see that. Also, um, the analysis that they did said that the cost of running a licensing department would exceed the revenue generated by the licensing. So that's the pragmatic side. But yeah, essentially the Reichs Museum has made every image completely free. Um, so it's public domain, even. And we continue to do so. I think there's more than 400,000 images that are available online. Um, there's an API that you can use, so you can you know search for images by artist or time period or medium or subject or whatever, and you'll get a direct link to download a high-resolution image.
SPEAKER_03And these aren't just pretty images, these let you zoom in and actually see brush strokes, correct?
SPEAKER_00Yeah, that's right. Um, most of the images are let's say 60 megapixels or so, which is um it's not an enormous number of pixels, but it's enough to make a uh to really see what's going on nicely. And then in the meantime at the museum, we're uh we're transitioning um for various important works of doing you know super high resolution, where now it's um many, many gigabytes per photo. Um, you know, the night watch, our highest resolution image is 5.6 terabytes. Um, so that's not the kind of thing you can just have people downloading willy-nilly from the website, but still it's it's available. And in fact, uh, it's interesting as a side note about my own history, the fact that the Reichs Museum wanted to be so open with their digital assets is one of the major factors that attracted me to come work there because I wanted uh my work to have the biggest possible impact. My my mission statement is sort of to help the world access, conserve, and understand its cultural heritage. And so if you're working in a museum where they want everything to be accessible to everyone, um, then that's the way to do it.
SPEAKER_03Can I go back briefly to pick up something that I wasn't sure I fully understood? You said there are 17 figures or characters or personages in the Night Watch, but I wasn't sure whether that was 17 in what was cut down or 17 in the reconstructed.
SPEAKER_00Oh, yeah. Uh I I should have mentioned that. I believe uh it is in its current state. So yeah, there were there were um two musketeers and a child um that were cut off, um, but I wasn't counting those.
SPEAKER_02Okay. If I recall the history correctly, I think the the 17 were the people who contributed to paying Rembrandt uh what was at that point an exuberant, exorbitant amount of money uh to create the painting. Some of the children and so on were, I don't think, paying anything.
SPEAKER_00Yeah, so only one child was cut off, but as far as I understand, the two musketeers who who were cut off were paying for for the picture. But um the other thing to note is that um you know the painting was was was painted in 1642, and it was cut down in 1715. So probably very few of the sitters were still alive when it was cut down, so they wouldn't have been been around to object.
SPEAKER_03And you you mentioned in your talk that there was a small face in the back row on the left that may be apocryphal, but it's it's possibly Rembrandt's face. Are you including that as the 17, or is he kind of the extra personage?
SPEAKER_00Um, yeah, that's among the 17 people in there. And in fact, um, if you try to count them, you probably won't get to 17 because there are some figures in the very, very back that are only depicted sort of part of their head. Um, another figure that most people miss, uh, there's this girl really in this you know glowing uh dress. Um and there is another uh what looks like a girl whose head you can partially see right next to her head, but almost no one spots it. So yeah, in those cases the neural network probably doesn't recognize it as a head, but uh it's not so important because they're not shaded in the way that I mentioned.
SPEAKER_02So there are other important paintings that have been damaged or destroyed. Have you discussed the application of your techniques to those works? Can you tell us anything about that?
SPEAKER_00Yeah, it's a really interesting idea. Um, but the problem here is uh in in extending this to those cases, is that this is really a translation problem that I'm solving rather than a composition problem. So you can imagine it's one thing to make a computer compose a novel, and it's a very different thing to make a computer translate a novel from English to French, let's say. So what I've done here is a kind of visual translation, whereas in the cases of paintings that are damaged having been cut off, we don't have the source material to go back. You know, in other words, I have a novel that was translated from English to French, and then pages of the English novel were lost, and I'm using the French version to go back. Sort of that's what I have now. What is more likely to be useful in terms of extending this to other applications is to help understand what kind of effect different conservation treatments will have. You know, because broadly construed, what you have here is uh a pair of images that are um semantically matched to each other, but they may have different textures, they may have different palettes, and they may have some geometric distortion between the two. And that's precisely the kind of thing that happens when you conserve a painting. You take off the varnish, you clean it, you remove old retouches, you uh and so on. And so in this case, you would have a pair of images, and we have lots of these. We have lots of training data where we have the before conservation and after conservation. Um, and so this system could then be used to teach a network how to imagine what it would look like if you would uh clean the painting, if you would take off the varnish and refresh it, if you would remove old retouches and so on. And that could then lead to much better decision making when it comes to trying to plan out a conservation treatment for a painting.
SPEAKER_03Does that mean that famous Dutch paintings that may have been lost in the Second World War, that we have photographs or copies of that somebody may have painted, you can't really reconstruct because we don't have any portion of the painting that survived for you to train the network on?
SPEAKER_00That's right. Um, at least not with this technology. If you if you think about how AI might extend itself in the future, then it is true that certain artists have a way of depicting things. You know, if you if you look at a Van Eyck person, you can tell that it's by Van Eyck, and it's not because the people looked like that. A memling looks like a memling because Memling has a kind of distortion that he um applies to the faces. And so you could imagine learning how to undo that. Um, and if you learned how how Rembrandt takes a face and kind of you know makes it look like a Rembrandt, then you could imagine a broad translation system that might take a photo or a copy where the entire original is now destroyed and sort of try to translate it back into the style of the original artist. Um, but this would be, I think, far more speculative and wouldn't necessarily give us something that we can't do on our own, you know, with our with our own imaginations, I guess. But as they say, you know, uh prediction is difficult, especially about the future. So who knows? Uh AI is evolving so fast that um it it can make your head spin. Um and these advances are really all amazing and delightful when applied to the conservation world or the cultural heritage world. So I'm really excited to see what's possible in even just a few years' time from now.
SPEAKER_03But why don't we look at one prediction? Because I'm I'm guessing that while you were working on this, you were imagining how colleagues in the conservation field were going to react to what you did. What did you expect and how have they in fact reacted, if we can compare your anticipation or forecast to how things have worked out so far?
SPEAKER_00Yeah, I I expected that um the conservators would be um would be delighted with it and that the art historians would be would be deeply skeptical of it. Conservators, because they are uh very enthusiastic about seeing new technologies and how uh and imagining how it might help their work. But broadly, the the reception has been positive. I mean uh this is it, and I think a lot of that is about the framing, which is to say that um this is not a project where I am saying or anyone is saying, oh, well, we're we're recreating the genius of Rembrandt, and in the future we won't need geniuses like Rembrandt at all. The purpose of this exercise is to help one imagine what it might look like, and it has um met that goal because the the fact is that um when you add the missing pieces, as you mentioned, quite a lot was cut off of the left. When you make it a little taller, when you make it a little wider, and so on, the composition of the painting changes dramatically. It it feels like a completely new painting and it starts to make sense as a concert, as a composition. You know, you can see that the figures have a place to step into. Um they were a bit cramped before. When the sides were cut off, it seems that more was cut off on the left in order to put the two main figures dead center in the painting. Um, but in the original, they were not dead center. They were going to step into the center, giving a sense of expectation and so on. Um, and so the art historians, the curators, uh have really been positive about seeing that because just looking at the Londons didn't quite do it because it's such a small copy and because it's in the style of London's. And London's is he's no Rembrandt.
SPEAKER_03I I have to thank you also. You you've clarified for me one of the boundaries between a painter and a photographer. A photographer takes has his own style or her own style, but the people they take all look very different. But your comment about, you know, a Rembrandt or or somebody else's depiction of certain faces tend to look not like the person necessarily looked, but as interpreted by their particular painting skills. You said you could you you've constructed a 700-plus gigapixel image of the night watch with pixel size of five microns. For those of us that aren't able to sort of quantify and see in our minds what that really means. Can you put that in terms that non-geeks and and lawyers also will understand and explain so that we can see it as you're seeing it?
SPEAKER_00Sure. Yeah. So when we when we talk about the resolution of a picture, one way is is to refer to the so-called sampling resolution. And that means that um you have this image in the virtual world, and then you have the thing that you photographed, the painting in the real world. And if you would move one pixel to the right in your image, then this corresponds to moving some distance on the object. Um, so in the uh 716 gigapixel image of the Nightwatch, if you move one pixel to the right, you're moving five micrometers uh to the right on the painting. The diameter of a human red blood cell is eight micrometers. So to image one red blood cell, I would require a two by two array of pixels to fit it in. Um if you prefer thinking um in dpi, not for in a laser jet um printer can do, let's say, 300 typically, a really fancy one could do 1200. Um, this is photography at 5080 dpi. So uh one square inch would be 5,000 pixels wide and 5,000 pixels tall. So there would be 25 million pixels in one square inch.
SPEAKER_02And if the if the uh picture were only an inch inch by an inch, that Would be quite a bit anyway.
SPEAKER_00All right. Yeah, that's right. And if if you hear the resolution figure like that, it might just seem like, oh, this is the cult of resolution. Like, why? Why would you do that? And the answer is that you know, I'm a material scientist by training. So the way to understand the properties of material is to understand its structure at all scales. And when it comes to an oil painting, one of the key bits of the structure is the paint and the paint pigment particles. So to make an oil paint, you take some solid material and you grind it up into a powder with linseed oil and then you paint with it. And so there's a structure that exists at this length scale. So when I photograph the painting at five micrometer resolution, I can see the individual paint pigment particles. Where there's a crack on the painting, I can look down into the crack like it's a canyon and I can see the layers of paint without having to take a sample from the painting. When there's a conservation issue, for example, there's something called a lead soap, which is a sort of a little pimple that forms over hundreds of years and emerges on the face of the painting and then falls out and leaves a crater behind. I can see those and I can't see them with lower resolution photography. Basically, this directly serves the interest of trying to decide what conservation treatment to perform, because then we can really see what's going on. And another nice aspect to it is you can see the minute details of how Rembrandt has failed to completely mix the paints that he's mixing on his palette in his brush, because then he'll make a he'll make a stroke with his brush, and I can zoom in and see the individual unmixed paints in the in the paint strokes and so on. So you really um you really can get so close that you can almost feel it being made.
SPEAKER_03I think in your talk, you even mentioned you you were able to see some bits of glass in the paint.
SPEAKER_00Yeah, that's right. One of the um kinds of pigments is called smalt, and smalt is basically blue cobalt glass that's been ground up into little bits, and um glass being amorphous has a has a certain way that it fractures when it breaks. And so the process of grinding up smalts means that you get these little chips of glass, um, you know, just like when you when you drop a glass on the floor, that the pieces are very characteristic of glass, and you can zoom in and see the individual uh smalt particles, and you can tell that they're smalt because they're blue, of course, but also because they're shaped like um cleaved glass.
SPEAKER_02So your your keynote address at uh PyCon in 2021 explores how you captured and constructed the 716-gigapixel rendition of the Night Watch. Uh, for the non-geeks in the audience, PyCon is a conference for the Python programming language community. The techniques you outlined are a tour de force of sophisticated applied mathematics. It says here in the fine print that your doctorate is in materials science. How does one construct a career path from materials to science to what you do now?
SPEAKER_00Um, yeah, well, uh I have always been interested in way too many things. Um, and in some ways that's a strength, and in others it's a weakness. So um when I was trying to choose a major as an undergraduate, I was on a mission to try to find the most broad field that I could get my hands on. Um and when they pitched material science and engineering, they said, well, believe it or not, about a quarter of the scientists in the world today are working on some form or other of material science. Because material science basically unifies um polymers and ceramics and metallurgy and uh composite materials and so on and so forth. And it spans all the length scales from atomic uh interactions all the way up to, you know, what shape should I make my landing gear? Um, so I majored in that. And um, and then for my for my PhD, I was deeply fascinated in the question of why nature makes trees everywhere. You know, like lightning is shaped like a tree, and rivers are shaped like a tree, and cracks are shaped like a tree, and the vascular um uh vasculature and leaves and so on. Um, so I was able to study this by simulating flow through porous media, and this led to big writing big supercomputer simulations to simulate hundreds of thousands of different morphologies, and and this then led naturally into thinking about um solidification and coupling physics together. And in the end, the thing about cultural heritage for these days is that you have several factors coming together. You have um new imaging uh techniques. I mean, I I can the camera that I'm using can take a single photo of 400 megapixels. You have high-speed internet, you have incredibly um fast computers, and no one has been doing anything in this field broadly. So it's a wide open field with really fun and interesting challenges that span many, many disciplines. Um, and then I can just sort of run around and have fun um while being useful instead of sort of, you know, there's in some fields there's no more low-hanging fruit, you might say. Um, in many university departments, people can't talk to each other in the next office because they're so specialized. But what appeals to me is crossing disciplines and so uh and and low-hanging fruit is always nice. So by moving into this field, I could I could plant a lot of trees and then everything is low-hanging fruit.
SPEAKER_02So we've looked at the 20 micron image that you made and have already published, and it's amazing. It it lets it lets us see individual brushstrokes, as you've mentioned. Why then go to the five micron resolution?
SPEAKER_00Yeah, um, it's really because the features that we really would like to see to make our conservation uh decision are smaller than 20 microns. So, you know, essentially there are some resolutions where you can't tell that the painting has cracks, and then you zoom in and then you see that it has cracks. And there are some where you can see a brushstroke, but you can't see the fine scale. And so you zoom in and zoom in and zoom in. And as you continue to zoom in, new features will emerge that are relevant in ways that are of concern for the conservator. Basically, chemical processes that are happening that are causing a process of change, um, the ability to detect where very minute, um, what are called retouches. So in the past and today, when you perform a conservation treatment on a painting where there's damage, the conservator may try to mask the damage by just painting over it. And so there is a resolution at which you can be sure that you will see all of those. Um, and 20 microns is close to that, but you have to go smaller. Um, and then why five? Why not? Why not stop at something smaller? The answer is that the limiting resolution of the world's best camera is basically that. So five microns is one to one. My pixel pitch, my sensor pitch on my camera is five microns. So I can basically make a one-to-one lens that doesn't magnify anything, doesn't reduce anything, and then I get five micron resolution photos.
SPEAKER_02So modern computers are fast and modern AI software is powerful, but neither is at the end of its expected evolution. Uh, if you were to apply similar techniques in 10 or 20 years' time, do you think the results would be different or better? Um, do you mean the techniques for the reconstruction? Or anything of the of the work you've done, because you used AI not just in reconstruction, but in actually constructing the images, the the photograph.
SPEAKER_00Yeah, that's right. Yes, everything is developing so rapidly that it's it's really difficult to predict even what we'll be able to do in one year. What I can say is that because of increases in internet speed, and because um people have more and more powerful devices in their hands, that's um probably the most immediate relevant change for the public will be their ability to access these things. So there will be fully 3D, incredibly high resolution, multi-layered. What does the x-ray look like? What do the paint layers look like? What does the chemical composition look like, and so on, all embedded and accessible on your mobile device, which at some point may be an implant in your retina or your contact lenses or whatever. And then our ability to link what we do behind the scenes in the museum to the public will will go up in a dramatic way. So um I'm I'm very interested in that. Also uh something we haven't talked about here, but which is uh another huge interest of mine is um is zooming out instead of zooming in. You know, so we we have 400,000 images from across Western art, and those have a relation to each other, they have a story, there are there are trends. And so if you're looking for an image and you can only describe it with words, um how do how do you surface that? How do you um find, you know, I would love to be able to query uh to say, well, find me all of the paintings of a guy wearing red with a lion and and to have it surface then all of the the Saint Jerome paintings, for example. But we can't really do that now because of limitations in the ability of the computer to understand these sort of abstractions that are present in art. We can do it with photos, but um, the ability to see what's depicted in a drawing, for example, is is still uh quite difficult because it's a drawing, it's very abstracted from the world as we as we perceive it. So I think that as far as the reconstruction, that it will likely be even more and more realistic. But at this time, it's sort of diminishing returns for that problem, I think. We could make it more realistic, but it seems to already sufficiently convey the idea of what the painting would have looked like. And so, uh, yeah, it's good enough.
SPEAKER_03I don't know if you've heard this, but uh during the shutdown due to the pandemic, at least in the United States, a number of museums have discovered that because of their online offerings, students at schools across the country who've never had the opportunity to visit those museums because they they couldn't afford to travel, they didn't live near it, are suddenly getting a chance to see and visit the museums. So the kind of work you're describing is really going. I hate to say it's of use in pandemics only. I would hope they'll continue to do this even as we come out of the pandemic, uh, which everybody hopes will be soon. But something else I'd like to ask your opinion on the reconstruction that you said is on display only for this interregnum period of what, three months?
SPEAKER_00Yeah, that's right.
SPEAKER_03Um most of us in the world are not going to get a chance to have that experience of walking in, remembering if we've been there before, remembering the impression the Night Watch made on us, and seeing how the painting shifts in perspective and how different it looks, uh, which you've described. Can you tell us anything about how have people who have been able to go there, the Dutch public that has been able to visit, if if it's been open yet? You know, you gave us the reaction of conservators and historians, but you haven't told us what is the general Dutch public, how have they reacted to what you've done?
SPEAKER_00Um, also very, very positively, and a frequent question is well, are you going to keep it? Um, because the the thing is that these missing pieces that were done in the computer were then printed out on a canvas material, glued to metal plates and and placed around the actual Nightwatch. So you can see them around the Nightwatch. They're offset a little bit, so there's no uh confusion about what is original Nightwatch and what's added. Um, but they say, well, why don't you keep it up there? That will not happen because uh we are a museum of of uh of Dutch culture and art, and this is not Rembrandt. Uh it would be very uh presumptuous, I think, to keep it to keep it up there. But my my hope is that there will be some place where possibly a full-scale uh printout or other kind of reconstruction could be viewed just to get the the sense. Because what's really missing when you see it on your computer screen is the sheer size of it. I mean, this is a really big painting, and you know, you just can't capture that um with a small screen. But yeah, of course, the the small version will live on on the website, so you'll be able to see that. But uh yeah, unfortunately, you've got to get here before October 13th, if you want to see it in person.
SPEAKER_03If I'm not mistaken, I I I don't remember if it was the Night Watch or another painting in the Netherlands that was knifed at one point. Well, your images of the Night Watch or any other paintings that the Rights Museum is doing at these super uh high resolutions be a kind of emergency room treatment if any other damage like that happens to a painting. But you'll you you will be able to fix it and in that sense keep it on view as it was rather than saying, well, this is not Rembrandt, this was not, you know, that night or or somebody else's work.
SPEAKER_00Yeah, I mean, one of the one of the fundamental sort of pillars of conservation is documentation, you know, just just documenting what the painting is like. And sometimes unexpected things happen, you know, sometimes your cathedral burns down, sometimes your painting is stolen or it's destroyed in a bombing campaign, or who knows what. One would never be able to replace uh a great work of art if it's destroyed, but still having a knowledge of what it looked like at the to the nth degree in three dimensions, chemically, and so on, it would be a real pity if we didn't do that and then something bad happened. You know, of course, uh as you know, uh Notre Dame, I believe, is uh the reconstruction is assisted by the fact that there were drone photos, high-resolution drone photos and laser scans that were taken um before the fire. But the higher resolution the better. And of course, uh, if it's partially damaged, if an artwork is partially damaged, then the ability of a restorer to go in and see exactly what it used to look like uh is dramatically improved if you have this very high-resolution uh data, as well as having 3D, as well as having knowledge of which pigments were used and so on. So the more data, the better.
SPEAKER_03We've been talking a lot about AI and its forecasting and it's predicting. Um, humans have a rather good capacity to exercise hindsight. As you use hindsight and you look back at the work you did on Operation Nightwatch, is there anything you would have done differently if you were starting out now, knowing all that you've learned from doing?
SPEAKER_00What I can say is that uh everything is harder than it looks. But of course, that's a sort of universal refrain. So the amount of effort that was required to actually pull off this um 700 plus mega gigapixel photo is a lot higher than than than we ever thought. Um, the depth of field of the camera is incredibly narrow. You know, you basically have one eighth of a millimeter depth of field. So if it's too close or too far, it will be uh blurry. Um and the painting.
SPEAKER_03With the camera, the painting moving and the air currents with that shallow a depth of field, you were able to deal with that.
SPEAKER_00Yeah, that's right. So basically I have to chase the painting a little bit. And I what I didn't mention in the talk is the fact that um moving the camera to place it in front of the painting blocks the air currents that are naturally occurring in the room. And so I ended up having to model that um because the way that the painting will move is predictable. So when you block the air currents, the painting moves away from you in a certain predictable way because the air currents were pushing the painting. Yeah, it's complicated. But these kinds of things, you know, just the the tiny details on the tiny details, on the tiny details that you need to pull this off mean that um, yeah, it's a it's a good thing to have done, but not necessarily a good thing to be doing. It was uh real suffering. So I'm glad um that I only have to do it probably twice because I can expect that when the conservation treatment is done, uh we'll do the whole thing over again, the same resolution before and after. And um yeah, uh now I know about all the pitfalls, but um some of these things I just yeah, I couldn't have predicted.
SPEAKER_03I I have to tell you, the four of us feel it's an honor and a privilege to have been able to have you as a speaker. Your work is extraordinary, but as a material scientist, where you say you wanted to be able to go across all the dimensions from very small to very large, you've done that in this talk for us uh in a really, if you'll pardon the expression, illuminating way. And I want to thank you very, very much. I I have thoroughly enjoyed this. Mark?
SPEAKER_02That was astounding and and uh thank you. This was uh uh eye-opening in uh in so many ways. We thank the American Bar Association for their generous sponsorship and support of the production of this podcast. Our theme music was composed and performed by the very talented Ben Rosenblum. We welcome questions and comments from listeners. Send email to comments at mindthegapdialogues.com. We read all comments and questions and will try to respond in the letters section of a future episode. If you are writing about a particular episode, please do mention the specific episode number. Please also do include pronunciation tips to help us properly say your name when we reply in a subsequent episode. See you next time on Mind the Gap Dialogues on AI. The views expressed in these podcasts are solely those of the speakers and not of their employers or organizations.
SPEAKER_01To the extent that the section offers a robust collection of content. To explore more about this topic or to learn about joining the section, visit mbar.org slash dislog. That's e i z L A W.