I find it interesting that humans can “read” a work of art even when the words are absent. Our brains are good at constructing stories that fill in the blanks. In this case, our fresco (the image on the left) invites us to fill in its missing pieces, offering subtle hints that catch our eye. For instance, we see what appear to be faint artist lines in the white area on the lower left. Was this an entrance to an ancient Roman temple? The now-decayed columns suggest that this was indeed a noteworthy building at one time. Italian Renaissance culture valued ancient Roman architecture, so a story like this fits well with what we know about the aesthetic values at that time. To help you see what I see, I added yellow lines to the image on the right.
Although I’m not a professional art conservator, I assume someone who is would take their restoration cues from extant lines or geometric features. In short, they would “read” the undamaged parts of the painting to reconstruct what was missing. A reasonable inference that most humans would make is that the arched causeway wraps around the hill on which the columns rest to the edge of the fresco, following the path of the yellow lines. The missing color on the other side of the fresco, just below the bluish hills, poses more difficulties. Small dabs of what appears to be the original green remain. But from such meager evidence, would a restoration expert infer that this entire area was originally rolling green hills? I think that’s an interesting question. Eventually, I’d like to interview a conservator and have them walk me through the restoration decision tree for this fresco. How did they proceed? And more importantly, what informed those decisions? Was it their knowledge of Renaissance aesthetic values? Was it their reading of what was left of the original fresco? Or was it a combination of things? One thing is clear: this kind of work is multi-dimensional. A professional art historian brings a wealth of practical and historical knowledge to each restoration task. My reading of this work, on the other hand, lacks that nuanced understanding.
Articulating the thinking processes of human experts, including art conservators, is especially important right now because that's what we seek in our intelligent, AI-enabled systems. Yet those systems still have a way to go before achieving this ideal. Let's look closer at the gap between expectation and reality, specifically the results I achieved when I asked the LightX Editor – a Photoshop competitor – to fill in our fresco's white (damaged) areas.
After highlighting the areas where fill was needed, I submitted this prompt to LightX: Please fill in the white space so it matches the rest of the painting. I ran this request twice and received the following two images.
In the first image, the addition of a plant with yellow flowers below the columns makes no sense and is probably a hallucinatory artifact. However, the model did an excellent job on the right, generating a complete building directly above the two humans running on the path by the river. From there, it added a line of buildings that recede into the horizon. This inference makes sense, given the first building in the sequence and the likelihood that it’s part of a small town.
I was still unhappy with what the model was doing on the left and modified the prompt as follows: Please fill in the white space so it matches the rest of the painting. Follow any existing lines as you fill in what’s missing. With this prompt, I tried to help the model find the lines I saw – the ones marked in yellow earlier – and then generate fill to match the geometric logic of the overall composition. It failed to do so. Instead, it generated a grassy river bank rather than wrapping the bridge or aqueduct to the fresco’s edge. See the image on the left.
Alright, I thought, let’s be more specific. My next prompt went like this: Please fill in the white space so it matches the rest of the painting. On the left side of the painting, add a small roof and building under the two columns. The image on the right is what the model generated. Finally, we see a line of arches matching the original two in the unrestored fresco. The razor-straight line here is neither aesthetically appealing nor in keeping with the curve evident in the original two arches. But at least it’s a start!
Overall, LightX did better than Photoshop – at least it did not hallucinate and generate a giraffe or raven. I would have liked to have experimented more, but I quickly used up the free credits LightX offers. New users can create up to five free images. After that, you must pay for the service. The larger question is whether one can tell a story of what they see so that the model sees it and reacts as a seasoned conservator would. There’s also the question of efficiency. I might be able to get the model to follow the natural lines of the composition, but the prompt engineering to achieve that might take longer than having an expert just do the work by hand.