Reading Between the Lines
Today, I’d like to return to a thought presented in my last post – the thought that the mind fills in missing details, constructs coherent stories from minimal data, and can seemingly read between the lines. In that post, we evaluated the ability of a couple of generative AI tools to recreate the missing parts of a Forni Cerato fresco. Those tools did not quite measure up. Photoshop hallucinated. The LightX Editor did better but still failed to respect the logic of the existing composition, specifically the underlying lines I saw in it. The way I filled in those blank spaces reflected what I saw, as well as my mind’s creative ability to construct a plausible narrative of what the original fresco looked like, given what was still there. This ability is not unique to me. We do this all the time when we read a novel, for example, and vicariously inhabit the life and world of a protagonist. But here’s the interesting part. We create that life and world by reading one-dimensional objects, sequences of letters formed into words.
Let’s stop and define our terms before proceeding. A one-dimensional object is an entity whose length can be measured. In fact, length is the only thing that can be measured in a one-dimensional object. The length of this string – abcdef – is six characters. A two-dimensional object has both length and width. Remember all those graphs you plotted in Algebra class? For each point, you had to specify its location on both the x and y axes. Therefore, a point on a plane needs two numbers to define its location precisely. And finally, a three-dimensional object has length, width, and height. Points in a three-dimensional space are specified using three numbers, one for each dimension. We don’t have to stop there. If we add time, we now have a 4th dimension. The number of dimensions we can add is limitless. In AI, spaces of a thousand or more dimensions are valuable and prevalent.
The act of reading is, therefore, a process by which we project a one-dimensional string of characters and spaces into a three or four-dimensional world. As noted in the last paragraph, we only need a single number to describe each point in a vector, the mathematical term for a string of sequential numbers or characters. However, we need twice as much information to locate points in a two-dimensional space, three times as much for points in a three-dimensional space, four times as much for points in a four-dimensional space, and so on.
Projection, therefore, requires additional information. For humans, that information comes from our memories, the sum of our embodied experiences in a physical world. The situation is much different for AI models. Here, the underlying data patterns are represented in a neural network composed of parameters, often called weights. These values are a byproduct of the data the model was trained on: one-dimensional text strings or two-dimensional images. This, then, is the model’s disembodied memory, a memory much different from our own.
Interestingly, a similar reading process is evident at the molecular level. In my September 10 post (Turtles All the Way Down), I pointed out that ribosomes read the 3-letter words, called codons, found in an RNA sequence as they synthesize a protein. Each codon therefore stands for a single amino acid. As the amino acid chain exits the ribosome, protein folding begins. The process consists of four steps. In step one, the primary protein structure is simply a chain of amino acids. In step two, hydrogen bonding of the peptide backbone causes the amino acids to fold into a repeating pattern. In step three, side chain interactions assume a 3-dimensional shape. And finally – in step four – the protein acquires its final shape as multiple amino acid side chains come together. Here’s an image of that four-step process.
The final shape or conformation of the protein determines its function; that is, its job in the real world. When a protein unfolds – called denaturation – it can no longer do any work. Denaturation can lead to tragic consequences. Dementia, for example, is caused by proteins unfolding in the brain.
At the molecular level, the “reading” process is surprisingly similar to what happens when humans read a text. A one-dimensional object, an RNA strand, is converted into a three-dimensional protein. Additional information, in both cases, is added to a simple, linear string of characters. Remember, we need three times the information to describe a functional protein as we need for the RNA sequence that underpins its synthesis. The process is akin to what happens when one adds water to a desiccated sponge. It expands to its full shape.
Biologist Marcello Barbieri asks where this additional information comes from. The answer he provides in his fascinating book, The Organic Codes, is that it comes from the context, the environment itself. According to him, the atoms in a sequence of amino acids obey natural law, that is, the organizing principles of physics and chemistry. That context, in turn, has its say in how the protein folds, adding extra information to the original one-dimensional strand of RNA.
I believe the same holds true for reading. Scientists still don’t know how the mind creates a world from a one-dimensional sequence of characters – how it reads between the lines. The question of how memory contributes to this process is an interesting one too. Are story laws at work as our eyes scan a line of text? It’s questions like these that will help us more fully understand the difference between biological intelligence and its machine imitator.