California Dreamin'
In 1965, the Mammas and the Papas recorded California Dreamin’, a song about the narrator’s desire to be in warm, sunny Los Angeles during a cold winter’s day in New York City. The song was a hit, and it’s easy to understand why. We all dream of locations where we’d like to be – white sandy beaches, verdant tropical forests, and sun-drenched cities. California Dreamin’ works because it evokes this kind of reverie.
Interestingly, AI models can also have dreams. Computer scientists call them hallucinations. Now, I don’t particularly like the word hallucination because it implies that the machine is conscious, which it isn’t. But then again, this wasn’t my choice. This is the word everyone uses for a model that’s on a psychedelic trip to a groovy land and responds as if it lived there. My apologies for the 1960’s slang…
To illustrate this problem, let’s head off once again to my Forni Cerato dream. Jennifer, our digital artist, has been working hard, fabricating digital furnishings, making clothes for the digital humans (including me) who inhabit the estate, and working on the villa itself. In my June 18th post, I shared some images of her stucco work, but it’s not just stucco that needs to be restored. The villa also features some attractive exterior frescoes located on the walls of its enclosed front entrance. Here’s a picture of one of those frescoes as it looks today.
Considering that this work is about 480 years old, it doesn’t look too bad. Even so, large parts of the fresco have exfoliated, exposing the plaster underneath. This is not uncommon with Renaissance frescoes. Leonardo’s Last Supper, for example, is in a fragile state, having undergone extensive restoration. The elements, especially humidity, can quickly degrade these kinds of works. Physical restoration work can be expensive and time-consuming. The project to restore the Last Supper lasted 21 years, from 1978 to 1999, cost millions, and utilized expertise from many fields.
Fortunately, that is not the case at my digital Forni Cerato. After reviewing the fresco images, we decided to explore generative AI’s ability to fill in the white space and do the restoration work for us. No need to hire expensive humans to painstakingly recreate what was missing. We’ll just let generative AI handle this job – at least, that’s how we thought it would be. Photoshop touts its generative AI tools, so Jennifer decided to put it to the test. Well, as usual, things did not go as initially planned. Jennifer uploaded the original image to Photoshop and then highlighted the white areas she wanted the model to fill. Finally, she wrote a short prompt, asking the tool to add fill that matched the undamaged parts of the fresco. We won’t go through all the prompt variations Jennifer wrote. Instead, let’s look at a couple of its most dreamy (hallucinatory) outputs.
This is one of the clearest examples of model hallucination I have ever seen. In the first image, the model added what appears to be a crow walking across the river. The bird’s color matches that of what’s left of the original image’s arched bridge or aqueduct that borders the river, just below the Roman columns on the left. But other than color as a possible explanation, this is obviously a case of California Dreamin’. The second image is even stranger, featuring at least two, possibly three, giraffes. Why the model decided to add them, unprompted by anything that Jennifer wrote, is a mystery to both of us. Any thoughts or ideas would be appreciated!
We continue to experiment with Photoshop and another generative AI photo tool called the LightX Editor. In next week’s post, I’ll share some of our findings as we achieved some limited success. The thin line that separates a story from a dream or a hallucination is an interesting one. As we’ve seen today, AI cannot distinguish between the reasonable and the unreasonable, the story and the dream. Sometimes, the dream of giraffes in a bucolic Renaissance setting is helpful, typically when an artist seeks provocative ideas. Dreamy output, however, is not helpful when someone is trying to restore a work to its original state. There are compositional cues and even a line or two in our fresco that an art conservator would probably reference as they restored this work. We’ll have more to say about that in future posts.