Application Development

August 15 , 2024 by Shane Connelly

Today, we’re incredibly excited to announce user-defined sorting functions for Vectara! These will help you onboard more “legacy” search applications to Vectara by providing the functional requirements they need, give you much more granular control over the result ordering, and reduce LLM costs for generation as well. Before we get much further though, let’s talk about what Vectara’s user-defined ranking functions are and how they work.

There’s More to Relevance Than Text

Vectara is great at being able to find the most relevant answers to questions in your corpora and surface them to a generative response. However, sometimes what you or your business cares about is more than just the textual relevance. You may care about the recency of the information, the price or margins of a product, or perhaps the average rating. Using traditional neural retrieval methods that “just” consider how relevant a section is to the user’s question, these would be impossible to consider.

Vectara’s new UDF (user-defined function) ranking capabilities allow you to override the ranking function with your own functions. For example, you can do strict ordering by replacing the result score with e.g. the price to get “sort by price” type of behavior for e-commerce use cases, but you can also combine signals like the textual relevance with the age or price to influence the result ordering.

Seeing It in Action

Using Vectara’s UDF reranking is done by providing a function of your choosing to rank the results. In the function, you have access to the document and part metadata as well as the Vectara relevance score.

If you want to sort based on metadata like “price, increasing” you can do that by setting the score function to only consider the metadata. For example:

get('$.document_metadata.price', -999999)

This sets the result score to the price value but if there is no price defined on the result, default the score to -999999 to place it at the bottom of the list.

You can also use $.part_metadata, the result score, and a boatload of mathematical functions to influence the ranking. You can use now() and a number of date-related functions to add a “recency bias” as well like:

get('$.score') - (to_unix_timestamp(now()) - 
get('$.document_metadata.date_epochseconds'))

More “complicated” arithmetic functions are available as well, and over time, our intent is to build out a library of capabilities that are available to our users for everything from arbitrary ordering to geographic score influencing (e.g. score things lower when they get further away from a central point).

As a few other examples to serve as inspiration for your own usage, consider the following.

To place all products that have no stock at the bottom for an e-commerce use case:

if get('$.document_metadata.units_in_stock') > 0 then 
get('$.score') else -999999

To boost products that have a higher review score (average_review_score is a metadata field with values between 0…10):

get('$.score') + get('$.document_metadata.average_review_score') 
/ 10

To place “promoted” products at the top of the list:

if get('$.document_metadata.promoted') == true then 999999 + 
get('$.score') else get('$.score')

To boost certain categories of products:

if get('$.document_metadata.product_category') == 'shoes' then 1.5 
* get('$.score') else get('$.score')

If you have a specific function in mind that you need, please join our Discord community server and post a request in our “feature-requests” channel.

To use the new functions, you embed these in the “reranker” component of Vectara as in:

{
  "query": "query",
  "search": {
     "reranker": {
       "type": "userfn",
       "function": "get('$.document_metadata.price', -999999)"
     }
  }
}

If you want to try any of these examples out for your own application, have a look at our API playground.

Modifying the Generative Inputs

One last thing to note is that this reranking happens before the list of results are sent to the generative system for retrieval-augmented generation (RAG) use cases. This means that you can use these user-defined reranking functions to influence the information that the generative LLM has when performing a summarization task. This can be very useful when you don’t want the generative LLM to consider outdated information, for example, by applying a UDF reranker that penalizes older information.

Conclusion

In this blog, we introduced Vectara’s new UDF functionality and showed how to plug in custom scoring functions to Vectara for custom ordering of results. These can enable you to completely customize the functionality of Vectara to your application’s needs.

For the latest documentation on the custom scoring functions and how to use them with Vectara, have a look here. As always, we’d love to hear your feedback! Connect with us on our forums or on our Discord or on our community. If you’d like to see what Vectara can offer you for retrieval-augmented generation on your application or website, sign up for an account!