Retrieval and Search

December 05 , 2023 by Shane Connelly

Why Result Diversity Matters

If you’ve ever done a search on an ecommerce site and endlessly scrolled through what seems like the same result, you know why result diversity matters.

Let’s search for “iron:”

Figure 1: Typical eComm image search results for “iron” search term

Did the user mean scrap iron?  A clothes iron?  A waffle iron?  A hair straightening iron?

The problem is just a fundamental problem with nearly all languages: a lot of user searches and interactions can be very ambiguous.  In most cases of search results, diversity is the enemy of relevance, but when a user wants to / needs to be exposed to different ideas to help them sift through the noise faster, increasing the diversity of results can be extremely advantageous.

Maximum Marginal Relevance

With our latest release, Vectara adds a new reranker: one that adds “Maximum Marginal Relevance” (MMR).  MMR seeks to provide more diverse results when they’re still relevant, which can help with ambiguous queries and occasions where a user really just wants to explore new and novel things.

MMR works very well to diversify results in the search use case, but it also has an advantage, specifically in retrieval augmented generation (RAG) use cases.  That is, summarization tasks like those in a RAG pipeline work best when the summarization generator has more different information.  If the top results can provide some different angles on the same question, with the right prompt, most LLMs can provide significantly more comprehensive summaries and simultaneously reduce any inherent bias in the data.

Using MMR in Vectara

To use MMR in Vectara, you need to pass a reranker ID to the query and you can optionally set the “diversity factor.”  You can do this via the API or the UI.  Full details on how to set these can be found in our documentation, but if you’re using the API playground, you’ll want to set the reranker ID to 272725718 and a non-zero diversity factor.  Values around 0.5 tends to be a good starting point and you can increase/decrease the diversity from there depending on your preferences.

We can see the result of this in looking at recent news.  First, with MMR turned off and asking an ambiguous question about what’s happening in Argentina, I get very relevant results:

Figure 2: Standard results with Vectara without any reranker

If I’m just interested in the result of the presidential election, this is a great set of results, though lacking much diversity in the types of information I’m presented.  But a lot more is going on in Argentina than just the presidential election result, and even within the recent election, some more angles a reader might be interested in.

As we turn on MMR, with the same query we can see several of the same pieces of news, but also a variety of new results:

Figure 3: Vectara with the MMR reranker enabled

We now get a bit more information about the election, including looking at it from the eyes of some specific voters as reported by NPR as well as a recent clash from a sports match and a recent story on “El Juicio.”

The result of this diversity leads to a more complete summary as well for these ambiguous questions, because the generative summarization LLM is given a broader picture of information that could answer the question.  Without the MMR reranker turned on:

Figure 4: Standard retrieval-augmented generative summary with Vectara without any reranker
Compared to with the MMR reranker turned on:

Figure 5: Retrieval-augmented generative summary with Vectara with the MMR reranker enabled

Comparing the diversity of information contained in these summaries, we can see that the MMR reranker provides a much more comprehensive summary across a wider set of subjects.  This allows the user to get a big-picture view of these ambiguous questions, and allows them to ask more pointed follow-up questions of the system.  This leads to a higher user satisfaction, as the user can go as wide or as deep as they like.

As always, we’d love to hear your feedback!  Connect with us on our forums or on our Discord.  If you’d like to see what Vectara can offer you for retrieval augmented generation on your application or website,sign up for an account.