Engineering Serendipity
The Oxford Learner Dictionary defines serendipity as follows:
/ˌserənˈdɪpəti/
the fact of something interesting or pleasant happening by chance
Given this definition, it’s paradoxical that someone might engineer serendipity, an event that happens by chance. Yet, isn’t that what happens every time one enters an archive or library? Books, manuscripts, and documents are not strewn about haphazardly. No. The physical facility and even its classification scheme are “engineered” by humans to facilitate discovery and retrieval. Indeed, this structure is a prerequisite of serendipity. Consider the case where you go to the library with the call number for a specific book. Once at the location, you scan the books to the right or left, above or below. Something catches the eye. And in an instant, serendipity strikes. This experience didn’t happen by chance. It was designed.
With respect to libraries, the classification system underpins serendipity. It has its advantages, but it’s also inflexible. Books and journals are physically arranged by call number. Philosophy and Religion, for example, live next to each other in B of the Library of Congress system. That means that when I locate a book on ancient religions, I’ll be near works of philosophy. They might even be a shelf or two apart, depending on the library’s size. However, I’ll be physically far from books on Technology classified in T. One quickly sees the problem. Any potential serendipitous encounter between items found in these two sections is impossible. Who says I might not wish to make a connection between ancient Religion and Technology? Take, for instance, the technology Egyptians used to construct the pyramids, a structure closely linked to their religious practices and notions of pharaonic immortality. Are there any possible links between these two topics? If there are, the content, depending on the whims of a cataloger and whether they assign it to B or T, will either be within range of my location in the stacks or outside. In this case, I need an algorithm that can range across the stacks, looking for items that feature ancient Egyptian Religion even if it’s a minor theme. Even if an item cataloged in T only briefly references Egyptian religious practices, that link might still prove valuable to me.
If physical spaces can be engineered to deliver serendipitous experiences, why can’t we do the same with artificial intelligence (AI)? With AI, we’re no longer constrained by physical space. That’s both an advantage and a disadvantage. There’s something to be said for the embodied experiences of the sort I just described. Even so, wouldn’t it be nice if we could build a research system that offers up similar kinds of surprises? This would be akin to an intelligent research assistant meeting you at a specific location in the stacks or archive with a stack of relevant items for your consideration.
Now, some of the items might be closely related to the item of interest to you; others might be farther afield. But in either case, the algorithm found the item because the author of it made a connection, either strong or weak, to your topic of interest. Interestingly, distant connections are often more valuable. Research has shown, for example, that “weak ties” (acquaintances you don’t know well) can be more helpful than “strong ties” (close friends) when searching for a job. Weak ties are valuable precisely because they connect you to broader networks and new information. The same holds when searching for relevant content.
Current AI can calculate “distances” between ideas and concepts. This ability offers us a unique opportunity to create new kinds of interdisciplinary research systems, systems that surprise us with unexpected connections. Amazon already does this in a limited way. At their site, one can see related items and what others with similar interests have purchased. These connections are often interesting and unexpected. However, it’s not just the connections already made within a peer group that are of interest but also those that the group has yet to make to more distant items. Returning to our library example, it's as if a small group of like-minded colleagues cluster about me at a specific location in the stacks. They identify and pull items from the shelves in our aisle and present them to me. Serendipity is now a group effort. But as a group, we still occupy a delimited physical space. We’re all in the same aisle. Once again, we need a way to identify links to content that lie outside our collective horizon.
The discussion so far points to an interesting thought. In the age of AI, the links between knowledge artifacts are just as valuable as the items themselves. This is a key yet overlooked development. Can AI make a contribution here? I believe so.