Introduction

In August 2024, we introduced user-defined functions, allowing you to specify your own logic to rerank search results based on metadata from the retrieved documents.

Previously, Vectara had supported two other rerankers: Maximal Marginal Relevance (MMR) and Multilingual Reranker v1. These two rerankers each serve their own purpose for improving the search results returned for your RAG query, but we understand that there are other important factors that may be specific to each application or use case. This is where the UDF reranker comes in.

In this blog, we will provide some guidance about best practices for using the UDF reranker as well as some examples. We will also talk about chain reranking, which allows you to string together multiple rerankers in a sequence. 

To build up the examples for this blog post, we created a Vectara corpus that contains reviews for Airbnb listings in Barcelona. 

The complete code and examples are available in this jupyter notebook.

What is UDF Reranking?

UDF reranking in Vectara allows you to define a function that is based on custom computation of values to rerank the search results in a query.

For example, you might want to rank results higher if they are more recent, closer to a certain location (from a geo-location point of view), or any other criteria that is important to your business.  It also allows you to apply arbitrary ordering and filtering.  For example, with UDF reranking, you can introduce user experience capabilities like “sort by price” or “remove out-of-stock items from the result list” for an e-commerce application.

To use this reranker, you specify the UDF reranker in the query API, under the “search” parameters with type=userfn and then user_function specifies the actual function.

The user-defined function can take advantage of a long list of functions and operators supported by the API.

The most important function is the get() function, which allows you to retrieve document and part metadata from the retrieved search results. For example, if you want to use a document’s metadata field called date, you would write get('$.document_metadata.date') in your user-defined function. 

In nearly every UDF function, you will see get('$.score'). This function retrieves the relevance score for each of the documents based on the previous retrieval step (vector embedding or hybrid search).

Another commonly used group of functions are the mathematical functions. These include the basic mathematical operators (+, -, *, /) as well as other common functions (e.g., absolute value, power functions, and trigonometric functions). 

The group of date and time functions can be used to calculate the difference between two datetime objects. This can be helpful for biasing the search results by some recency metric. 

Finally, you can use an if-else expression to define different functions to calculate a document’s reranking score based on some document or part metadata. Check out this documentation to learn the format of this kind of expression.

A UDF can also return NULL. If your function does return NULL for some documents, those would be removed from the result set altogether.

Using UDF Reranker with Chain Reranking

Vectara’s chain-reranking provides a way to chain together multiple reranking methods to achieve better control over the reranking, and combining the strengths of various reranking methods. This of course includes the UDF reranker.

A great way to use the UDF reranker is in a chain: first the multilingual reranker, followed by the maximal marginal relevance (MMR) reranker, and then a user-defined function, as shown below:

In the above example, we can see how to structure a chain reranker in a Vectara query API call.

With chain reranking, the score for each reranking step (get('$.score')) is taken from the previous step. So for the example above, the score for the UDF step would be taken from the output of MMR, and the score for MMR would be taken from the output of “Rerank_Multilingual_v1”.

In the above example, we used the following UDF:

The get('$.score') function provides the document’s score after applying the multilingual_reranker and MMR rerankers. The output score from the multilingual_reranker will always be between 0 and 1 and changes from that after the MMR step. The overall_rating field in the metadata for an Airbnb will be between 0 and 5.

Thus, our user-defined function evenly weights the overall_rating value from the document metadata with the (previous ranking score * 5).

If you would like to read more about chain reranking, check out our release blog.

Now let’s look at some example user-defined functions that are useful for our Airbnb dataset.

Finding the Closest Airbnb Property

One important consideration when choosing the right Airbnb is the location. Suppose that you want to find an Airbnb that is close to many of the top attractions in Barcelona so that your travel time is minimized on your trip. 

We can add a user-defined function that calculates the distance between the Airbnb and another location, such as Casa Batlló, a historic landmark in Barcelona that is within close proximity to other popular attractions like Basílica de la Sagrada Família and Plaça de Catalunya.

To demonstrate this, let’s see what happens when we rerank Airbnb properties based on two criteria. The first is simply with relevance reranking and MMR, and the 2nd is with the addition of a UDF that ranks properties closer to Casa Batlló higher.

You can see the specific UDF in this notebook.

We can see that the results with the UDF included are closer to our desired location.

Giving Preference to Recent Reviews

In many applications, it may be helpful to sort the results based on some datetime data. Let’s look through an example of this with our Airbnb data.

For our example, we would likely want to see the most recent reviews because those most accurately reflect the current state of the Airbnb. However, we do not want to completely ignore the relevancy of the review’s content to our query. Thus, we use the amount of time since the review was posted as a penalty to the previous retrieval score for the query “I want an Airbnb that has a full kitchen and dining room.”

Here are the results returned by the two queries:

(without UDF)

Text: The Airbnb has a good kitchen and living area.Date: 2018-04-11

Text: Full kitchen and large living/dining room.Date: 2019-02-28

Text: I would recommend this Airbnb to everyone traveling to Barcelona – two bedrooms, two full bathrooms, full sized kitchen, dining, living room, and incredible views.Date: 2024-06-21

Text: that’s what you want in an Airbnb!Date: 2023-05-18

Text: You get hotel quality with the advantages that an airbnb has to offer like a nice full kitchen.Date: 2022-07-11

(with UDF)

Text: I would recommend this Airbnb to everyone traveling to Barcelona – two bedrooms, two full bathrooms, full-sized kitchen, dining, living room, and incredible views.Date: 2024-06-21

Text: Amenities are perfect – a full kitchen (curiously no microwave but has an oven), roomy dining and living rooms, 3 bathrooms, and good-sized bedrooms.Date: 2024-01-04

Text: The Airbnb itself was had good wifi, ac unit, and a full kitchen.Date: 2023-10-17

Text: that’s what you want in an airbnb!Date: 2023-05-18

Text: Full kitchen, large living room with dining area.Date: 2022-10-15

You can see that the reviews returned by the query with the UDF reranker are more recent than those returned by the query without the UDF reranker.

Conclusion

Ranking with user-defined functions is a powerful new feature of the Vectara platform, that allows you to augment the existing multi-lingual reranking or MMR reranking with custom logic that fits your application.

In this blog, we showcased some examples of user-defined functions in a real-life example about Airbnb reviews. You can see the full code for this blog in this jupyter notebook.

Several of these examples can work in other contexts as well, but there are infinite possibilities for defining these functions. We encourage you to use this blog as a guide as you build your own functions and AI applications.

We’d love to hear your feedback about these examples as well as the functionality of user-defined functions.

Try out all of these capabilities yourself by signing up for a Vectara account and experimenting with user-defined functions in our API playground. Share your favorite function that you create or request a new function that is not currently supported by our UDF reranker through our forums, user community, or Discord.

UDF Reranking NotebookSee full queries and comparisons of the examples presented in this blog in our Jupyter Notebook!More Examples!