My grandfather, a taciturn, unsentimental Quaker mechanical engineer, was given the job of preparing a Methodist church to be a Quaker meetinghouse. The church had tall stained glass windows, but Quaker meetings are un- or minimally adorned. Well, Dan was a competent, efficient fellow. In short order the meeting was fit for occupancy again. The stained glass windows were gone except for a rose window. Some time after some of the Methodists came by asking what had become of the tall stained glass windows. They were memorial windows and they wanted to keep them as memorials. No one knew. Questions went around. Eventually someone asked Dan. He said they were under the gravel in the parking lot. What! Why are they there!? He just said, "they were ugly."
I hope the spare pieces of the Night Watch weren't ugly, and if they were, the job of fitting them to the space wasn't given to some 18th century Dan.
To be fair, Rembrandt was famous and very well-employed in his own day, he just spent money like it was going out of style. Had he saved a little he'd have died rich.
Art teacher speaking. Cropping is surprisingly easily to detect in an aesthetic image. Its kind of like reading a half-finished book. Aesthetic images are designed for an defined format. If that format is changed nearing completion, the geometry feels wrong. It is even easier in paintings, where there are overt brush marks. The structure of a brush mark is strongly framed by the format of the canvas.
Anyone interested in the recreation of famous incomplete paintings, should check out Manet's The Execution of Emperor Maximilian. This is a painting Manet gave up on, and chopped up into smaller studies.
This is a cool project. The part that I am not so sure about is why don't they just use an artist to paint the missing part? What is the advantage of using an AI?
My guess, even that I hope not, is that people thinks that AI is going to be more impartial, more accurate, less biased. The reality is that an AI, as used in this example, is going to be over-fitting.
An artist can understand Lunden and Rembrant art and use that knowledge to produce something closer to the intention of the artist and the copy.
Cool project, but a dangerous precedent if people thinks that AI provides unbiased reality, that using AI you can show "how it really looked" as something more accurate that any human con achieve. Constructing something from missing data is always guesswork, whenever the intelligence is natural or artificial.
They mention in the video that they could ask a person to make the changes with Photoshop, but the reason they give is: "Then it would be that artist's work, not Rembrandt".
That statement rubs me the wrong way a bit though, because whether you ask a person or a computer model to copy Rembrandt's style, in each case it's still a copy, and I would argue in each case there's still a(n) artist(s) mimicking Rembrandt, just using different tools.
> My guess, even that I hope not, is that people thinks that AI is going to be more impartial, more accurate, less biased. The reality is that an AI, as used in this example, is going to be over-fitting.
> Cool project, but a dangerous precedent if people thinks that AI provides unbiased reality, that using AI you can show "how it really looked" as something more accurate that any human con achieve. Constructing something from missing data is always guesswork, whenever the intelligence is natural or artificial.
I'm not gonna anthropomorphize the AI and say "the AI is the artist", but the AI is definitely an art style.
I see it as just an old-meets-new mashup, craftfully executed. Any artist/technique you use isn't going to be the original Rembrant, so the choice of what artist/technique you use is just a question of aesthetics and taste.
Here we're just exploring a new technique of the 21st century.
Agreed, cool, but more so for the study/analysis/comparison of the composition before and after cropping, revealed by Lunden's copy. Repairing missing/damaged canvases is a understood problem, tackled by reversible inpainting, by an art restorer. I agree, not really sure why the AI approach is that interesting/significant.
For anyone else who wants context before watching:
this is about the use of AI to reconstruct missing pieces of a Rembrandt painting ("Night Watch") using a contemporaneous copy from Lundens. After the Lundens painting was made, the Rembrandt original was cropped aggressively (~30 cm on a side) in order to fit onto a particular wall where it hung.
Now that it is considered a masterpiece, they are trying to computationally estimate what those cropped sections looked like using Lunden's copy.
Robert Erdmann also gave a keynote talk at PyCon US 2021 about how they used Python for most of digitization and processing of the Night Watch (aka Operation Night Watch). Fascinating talk with tones of technical details.
That is brilliant. I saw the Night Watch at the end of a busy day and it was pretty much the only thing I had time for in the Rijks Museum. I will have to return to Amsterdam and see the expanded version.
The first time I went to the Rijks Museum I almost missed it.
I saw on the left side of the main gallery (this is from memory, 30 years back) a small painting (say, 40x20 cm) which had a name similar (or the same) as The Night Watch and I thought that it is a great deal for such a nice but average painting.
Then I saw some people standing in the middle of the gallery watching something and I discovered the painting :)
The impact also depends on the direction from which one approaches the painting.
If you enter from the room to the right hand side of the Night Watch it's almost easy to miss, not least because the ship models in the rooms that precede it are quite impressive, too.
From the Great Hall and the Eregalerij (Gallery of Honour) the effect is positively awe-inspiring. I've once heard someone describe the Rijksmuseum as a cathedral of Enlightenment. The Night Watch is the centrepiece that cathedral was built around.
I do not remember how I got there but I remember that there was a superbe exhibition of the activities on ships of the ~17th century. I vividly remember the doctor's equipment (just a few) and realized that being injured on a boat at that time was really bad.
I think there was also an exhibition on the history of doll houses
I like how strong your memories can be of art in context. I can remember times I rounded a corner decades ago to first see a major piece, how many people were around, what the room was like, etc. I remember the second time I saw Guernica I had my brother with me, who hadn't seen it before. I picked an intentional path to it so his focus would be in one direction initially and then he'd turn and see it for the first time from a great position. My first viewing I'd approached from a pretty dull angle.
Using a neural network to complete lost parts of a canonical masterpiece reminds me of nothing as much as using some kind of necromancer to ask the spirit of the artists to complete the picture. And of course depending on the quality of the scholar of the dark arts, we expect a similar quality of the result.
So the thing is, the completion is a modern artifact, and fundamentally in both cases it captures the expertise of the reconstructor, and not some kind of essence of the painting.
What sort of NN would be used to warp two compositionally similar images to match each other in structure? In the example the copy painting had almost all of the same general components, laid out with fairly significant differences, and they warped it to match quite precisely.
I'm taking your question without context, so sorry if it doesn't make sense: Warping images to match based on structure / features is common in much of image processing and requires no NN since we know how to do it. (But may be enhanced by NN).
The basic idea is:
1. Find features in images A and B (SURF / SIFT are common). However, maybe a NN could find interesting features here?
2. Match features so you know what feature from A is where in B. Again, a NN could potentially do this "better".
3. Compute a homography (translation between planes of features) or procrustes transform (or other types of transformations)
4. For each pixel (p) in the target frame (B), take the inverse of the transformation to find the pixel of the original frame (A) (which is usually a fractional pixel because it won't line up)
5. interpolate between nearby pixels in (A) to get the "true" pixel value at (p) in (B). Perhaps a NN could improve this interpolation? We've seen this in super resolution, for example.
You can easily reproduce these steps with just a few function calls in OpenCV. Try it out. https://opencv.org/
Up to step 3 is very helpful to answer questions like: "How much did the camera move between captures of image A and B. So now you can solve structure from motion, visual odometry, etc. This is typically how quadrotors navigate with cameras.
Would SIFT or SURF features actually work in this scenario, that was also one of my thoughts, but then I considered that the inputs are not photographs of the same objects from different angles but a painting and a manual reproduction of it. I would be really interested in knowing whether those features are robust enough under the distortions in such a scenario. And how far could one push this, could you still align the Mona Lisa with a good pencil drawing of it?
I have no idea! Careful feature selection is very important. If you only have edge information, then extracting edges in both images seems like a reasonable pre-processing step.
Without a neural network one would probably use optical flow [1] and I guess you could train a neural network to predict optical flow and then use that flow to align the two inputs. I have no answer to your specific question, what kind of neural network architecture would be best suited but given the task it seems reasonable to assume some form of convolutional neural network.
My grandfather, a taciturn, unsentimental Quaker mechanical engineer, was given the job of preparing a Methodist church to be a Quaker meetinghouse. The church had tall stained glass windows, but Quaker meetings are un- or minimally adorned. Well, Dan was a competent, efficient fellow. In short order the meeting was fit for occupancy again. The stained glass windows were gone except for a rose window. Some time after some of the Methodists came by asking what had become of the tall stained glass windows. They were memorial windows and they wanted to keep them as memorials. No one knew. Questions went around. Eventually someone asked Dan. He said they were under the gravel in the parking lot. What! Why are they there!? He just said, "they were ugly."
I hope the spare pieces of the Night Watch weren't ugly, and if they were, the job of fitting them to the space wasn't given to some 18th century Dan.