Enhancing digital asset search with WordNet

We live in a digital world. The great majority of content we digest daily for work, entertainment, or other purposes, is displayed on screens. Huge volumes of assets like images, videos, or documents are produced daily and uploaded to various data storage systems, be it local disk drives, cloud data stores, or advanced digital asset management (DAM) systems. With the amount of data ever-growing, it becomes increasingly difficult to manage and interact with it. From the content administrator's perspective, it is easy to lose track of what is already present in the system. Such an impediment might lead to undesired consequences. For instance, when specific assets are hard to retrieve, it is easy to inadvertently add them again, leading to unnecessary duplication. By the same token, updating or removing no-longer-wanted objects becomes a daunting task.

The inconveniences are not limited to content creators only, as the end users might suffer too. Without a doubt, effective search functionality is crucial for meaningful interaction with any data source. However, if the amount of data is large, and the contents are not tidily annotated, the users might feel overwhelmed. The chaos present in the data can easily discourage people from using it.

This is where Okapi comes into play. Okapi is a software solution developed by Wunderman Thompson Technology with the purpose of facilitating content retrieval in DAM platforms. With its powerful search extension capabilities, Okapi can plug into existing systems to enhance UX and enrich your queries.

Chasing rabbits

The natural language we speak is sometimes said to be our biggest invention. It enables us to exchange complex thoughts as well as describe objects and concepts in many ways. However, the richness of the language, although bringing undisputed benefits, generates some significant challenges at the same time. Natural language is imprecise and ambiguous, especially when used across geographically and culturally distributed teams. If we imagine a content entry team like that and hundreds of thousands of assets they manage, we can quickly realize that finding what you actually look for may be difficult. Let's look at a basic example and say we need a nice picture of a dog for some new campaign. Of course, the default WoW (Ways of Working) is to type "dog" as a search query in our system:

What if there are not that many assets with "dog" in the metadata and, instead, most of them are tagged much more precisely to reflect breeds and types of dogs like "husky", "chihuahua", "watchdog", or "puppy"? How to proceed in such a case? To see all the pictures of dogs, one would need to list all the breeds and types one by one.

There are at least a few disadvantages to such an approach. First, it is time-consuming. Secondly, it requires extra attention and methodical effort. Finally, someone would need to somehow know all the types and breeds of dogs in existence... So, do you need to be a canine expert to find your assets efficiently? Or eventually an expert in all domains that may be represented in your asset collection? This doesn't seem feasible.

Furthermore, if you are not a native speaker, finding all the reasonable search terms becomes even more difficult. Impressive collections of assets that companies invest in and maintain may simply not be used efficiently or some elements might not even be used at all. Such cases motivated us to look for and implement a totally new user experience by including semantic analysis in the standard search mechanism.

Semantic database

Semantics is the key area of natural language processing. For machines, any sentence we type or say is just a collection of letters that are combined into words. Syntax ensures these sequences are grammatically correct, but it is semantics that brings the real value into the equation. Semantics is about understanding the meaning or sense of words and entire phrases. Words like nouns, verbs or adverbs can be grouped by the same meaning and constitute what we call cognitive synonyms or synsets ("synonym sets"). So, an example of a synset can be a set of {car, auto, automobile, machine, motorcar}, with the assigned meaning as follows: "a motor vehicle with four wheels; usually propelled by an internal combustion engine." Furthermore, synsets can be linked with each other with various semantic relations like:

• Hyponymy – subtype/specification relation, e.g. a watchdog is a hyponym of a dog
• Hypernymy – generalization relations, e.g. a dog is a hypernym of a watchdog
• Meronymy – part-whole relation, e.g. a wheel is a meronym of a car
• Antonymy – opposite-meaning relation, e.g. dry is an antonym of wet

Collections of synsets connected with semantic relations are known as language ontologies. They're commonly used for advanced language processing (e.g. if one says he uses a car, a system knows the car is a type of vehicle, so it can automatically conclude the person is probably moving). One of the most famous ontological structures (or semantic databases to be more precise) is Princeton's WordNet which was worked on for the last 30 years and connects almost 120 000 synsets covering most of the English vocabulary. Here is an example of the inherited hypernymy for the "dog" synset in WordNet:

Try it yourself at Princeton search.

As we can see, WordNet not only helps to understand the words, but it also explains the world around us. So, it may not be that surprising anymore that we call these structure ontologies, as ontology itself is one of the oldest and most fundamental branches of philosophy, which studies concepts of existence, being and reality.

What's Okapi?

Okapi is a custom solution boosting search functionalities of existing platforms. It started as an internal RnD project in 2020 and quickly transformed into a fully fledged project upon showing its great potential. Okapi is a RESTful API, which abstracts away contrived processing of the WordNet structure, exposing user-friendly endpoints with powerful semantic analysis capabilities. Okapi has already been successfully integrated as a plug-in feature within the Adobe Experience Manager content editing mode and in Asset Share Commons, Adobe's open-source asset share reference implementation built on AEM. Moreover, Okapi can be easily applied to other asset management tools and services with little or no extra implementation. You can imagine it as an extra layer or option available to your users while they search across their collection of digital assets. The feature can also be customized across companies to address users' specific ways of working, taxonomies and language features.

Connecting to WordNet

We have seen that WordNet has immense potential. How can we integrate with it to seize its power in our custom software? At first glance, the WordNet structure, with its innumerable synsets and relations, might seem unwieldy. To interact with it, you can implement your own API or use one of the various utilities that already exist. We decided to go with the latter option and use a library called extJWNL (Extended Java WordNet Library), which makes it possible to load and query the WordNet database from Java. Adding it to your project is as simple as specifying the appropriate dependencies. If you are using maven, add these to your pom.xml file:

<!-- main library dependency -->
<dependency>
  <groupId>net.sf.extjwnl</groupId>
  <artifactId>extjwnl</artifactId>
  <version>2.0.5</version></dependency>
<!-- Princeton WordNet 3.1 data dependency -->
<dependency>
  <groupId>net.sf.extjwnl</groupId>
  <artifactId>extjwnl-data-wn31</artifactId>
  <version>1.2</version>
</dependency>

The second dependency is not needed if you want to use your own copy of WordNet added to your project's resources as a file. However, it is easier to embed it into your project by using the dependency above.

For the complete documentation on how to use the library, check out the official Wiki.

In this post, we will show an example of getting all the meanings of a given word, in this case - "car". The code looks as follows:

package com.wblachowski.wordnet.example;

import java.util.stream.Collectors;
import net.sf.extjwnl.JWNLException;
import net.sf.extjwnl.data.POS;
import net.sf.extjwnl.data.Word;
import net.sf.extjwnl.dictionary.Dictionary;

public class Senses {
    public static void main(String[] args) throws JWNLException {
        var dict = Dictionary.getDefaultResourceInstance();
        var indexWord = dict.lookupIndexWord(POS.NOUN, "car");
        System.out.println("Senses of car:");
        for (var sense : indexWord.getSenses()) {
            var lemmas = sense.getWords().stream().map(Word::getLemma).collect(Collectors.joining(", "));
            var definition = sense.getGloss();
            System.out.println(String.format("• %s%n\t%s", lemmas, definition));
        }
    }
}

At the beginning, we load the WordNet resource. The data is then queried with the given word to get all its senses (or meanings). We iterate through them, printing every synonymous word representing a given meaning, as well as the definition. When we run the code, we get the following output:

Senses of car:
• car, auto, automobile, machine, motorcar
	a motor vehicle with four wheels; usually propelled by an internal combustion engine; "he needs a car to get to work"
• car, railcar, railway car, railroad car
	a wheeled vehicle adapted to the rails of railroad; "three cars had jumped the rails"
• car, gondola
	the compartment that is suspended from an airship and that carries personnel and the cargo and the power plant
• car, elevator car
	where passengers ride up and down; "the car was on the top floor"
• cable car, car
	a conveyance for passengers or freight on a cable railway; "they took a cable car to the top of the mountain"

We can observe that even such a simple word has five distinct meanings. While the difference between some of them might seem subtle, the creators of WordNet decided to keep them separate. As an extra bonus, some meanings include example sentences in their definitions, which makes it easier to put a particular concept into a broader context. Getting hyponyms and hypernyms for a queried word is just as simple. Let's get back to the example from the beginning of the post, where we discussed different dog breeds. The following code snippet prints the definition and direct hyponyms of "dog":

package com.wblachowski.wordnet.example;

import java.util.stream.Collectors;
import net.sf.extjwnl.JWNLException;
import net.sf.extjwnl.data.POS;
import net.sf.extjwnl.data.PointerUtils;
import net.sf.extjwnl.data.Word;
import net.sf.extjwnl.dictionary.Dictionary;

public class Hyponyms {
    public static void main(String[] args) throws JWNLException {
        var dict = Dictionary.getDefaultResourceInstance();
        var indexWord = dict.lookupIndexWord(POS.NOUN, "dog");
        var mainSense = indexWord.getSenses().get(0);
        var hyponyms = PointerUtils.getDirectHyponyms(mainSense);
        System.out.println(String.format("Definition of dog:%nt%s%n%nDirect hyponyms:", mainSense.getGloss()));

        for (var node : hyponyms) {
            var synset = node.getSynset();
            var lemmas = synset.getWords().stream().map(Word::getLemma).collect(Collectors.joining(", "));
            var definition = synset.getGloss();
            System.out.println(String.format("• %s%n\t%s", lemmas, definition));
        }
    }
}

Output:

Definition of dog:
a member of the genus Canis (probably descended from the common wolf) that has been domesticated by man since prehistoric times; occurs in many breeds; "the dog barked all night"

Direct hyponyms:
• puppy
	a young dog
• pooch, doggie, doggy, barker, bow-wow
	informal terms for dogs
• cur, mongrel, mutt
	an inferior dog or one of mixed breed
• lapdog
	a dog small and tame enough to be held in the lap
• toy dog, toy
	any of several breeds of very small dogs kept purely as pets
• hunting dog
	a dog used in hunting game
• working dog
	any of several breeds of usually large powerful dogs bred to work as draft animals and guard and guide dogs
• dalmatian, coach dog, carriage dog
	a large breed having a smooth white coat with black or brown spots; originated in Dalmatia
• basenji
	small smooth-haired breed of African origin having a tightly curled tail and the inability to bark
• pug, pug-dog
	small compact smooth-coated breed of Asiatic origin having a tightly curled tail and broad flat wrinkled muzzle
• Leonberg
	a large dog (usually with a golden coat) produced by crossing a St Bernard and a Newfoundland
• Newfoundland, Newfoundland dog
	a breed of very large heavy dogs with a thick coarse usually black coat; highly intelligent dogs and vigorous swimmers; developed in Newfoundland
• Great Pyrenees
	bred of large heavy-coated white dogs resembling the Newfoundland
• spitz
	any of various stocky heavy-coated breeds of dogs native to northern regions having pointed muzzles and erect ears with a curled furry tail
• griffon, Brussels griffon, Belgian griffon
	breed of various very small compact wiry-coated dogs of Belgian origin having a short bearded muzzle
• corgi, Welsh corgi
	either of two Welsh breeds of long-bodied short-legged dogs with erect ears and a fox-like head
• poodle, poodle dog
	an intelligent dog with a heavy curly solid-colored coat that is usually clipped; an old breed sometimes trained as sporting dogs or as performing dogs
• Mexican hairless
	any of an old breed of small nearly hairless dogs of Mexico

As we can see, WordNet is pretty dog-savvy. The result is not limited to breeds and includes phrases describing types of dogs in other ways, including "lapdog", "puppy" or "hunting dog".

High-level architecture

With the basic theory behind us, we can now think about plugging WordNet into existing systems. The overview of the Okapi's architecture is represented by the following diagram:

The core of the system is a service capable of interacting with WordNet and exposing an API, which provides endpoints with multiple functionalities, e.g. listing synonyms of a given word, getting broader or narrower terms, and more. Any number of external services like CMSs, mobile apps, web pages, and other can communicate with the service to boost their functionalities. The WordNet resource itself can be treated as a static database, which can be updated on demand if a new WordNet version is released. Such a solution decouples the local instance of WordNet from the official release, which gives multiple benefits. First, no changes to the database will happen without our knowledge, and we can still update it if we see it fit. Additionally, this allows us to maintain a full-fledged fork of the original WordNet, which may include any customization we can think of. For instance, if we want to simplify the database, we can truncate the original WordNet tree to our needs. Conversely, if our use-case would benefit from a more verbose structure, we may add new nodes.

Okapi in action

Let us come back to the example from the first section of this blog. Looking for a proper picture of a dog in a vast collection of assets may be difficult. But not if you bring the ontologies on board. The semantic analysis that we propose works for you and calculates what "dog" really means and where it is placed within the overall WordNet structure. With a single click, the Okapi features are employed to look for synonyms as well as inherited hyponyms of "dog":

Of course, one can search the other way round (with another single click in the app) and crawl the assets up in the database structure to look for near and far generalized terms related to dogs, like carnivores, mammals, or animals. A user can decide on his or her own how far to look in both directions. This is a kind of graph traversal we will tell more about in some future blogs – stay tuned! With these features implemented to extend searches, users can find appropriate assets with little or no effort, without being limited with their language skills or domain knowledge. For more examples and use-cases check What if your asset management system could read between the lines?

Summary

There's great value in maintaining large stores of data. In most cases, however, the usefulness of such sources is only as great as the ability to effectively query them for desired content. With Okapi, the pain of organizing and retrieving digital assets is mitigated. The tool redefines the search functionality by allowing the users to think of their queries as concepts, instead of precise phrases worded in a particular way. Furthermore, Okapi is a showcase of how decades of scholarly effort can be combined with an intuitive tech solution to significantly augment the asset search functionality in DAM systems and beyond.