Application Development

February 20 , 2024 by Shane Connelly

Today, we’re excited to release a new capability in Vectara: Custom Retrieval Augmented Generation (RAG) Prompt Engine that can directly add the retrieved documents and their metadata into the generation.  That probably sounds a bit abstract, but that’s in part because this feature enables a wide range of applications. In this blog, you’ll find concrete examples that illustrate its effectiveness.  So before we dive into how it works, let’s cover some of my favorite example use cases to understand the power.

Example Use Cases:

RFI/RFP/questionnaire answering based on previous answers

Most enterprise software companies receive RFIs, RFPs, and other security and compliance questionnaires.  Many hours are spent by sales, security, and compliance teams toiling to fill these out, in part because the way that the questions are posed is different every time.  “What compliance certifications does your company have” and “Does your company have SOC 2 certification?” is an example where the first answer might be “SOC 2 Type 2” and the second might be “Yes.”  LLMs have become good at answering these, but they don’t know the answer to your business data.  Vectara’s RAG system can now help auto-fill these types of answers based on answers you’ve previously provided in other RFIs and RFPs.

Drafting Jira tickets based on user feedback

If you have support chats feedback already in a support or user forum system, it can be time intensive to convert those chats into actionable feedback for the developer team.  With the new custom prompts, you can now now solve this with a single click: Vectara can perform a search, find relevant documents, Vectara can then draft a Jira ticket or GitHub issue for you based on the most common problems or requests.  The beauty of custom prompts here is that it not only automates the human labor work, but also improves the quality of Jira tickets, which can further drive clarity in communications.

Custom formatting of results

Have problems editing the summary into a markdown format? Or spend late-night hours trying to add markdown formatting rules? You can save all that hassle now. With a custom prompt, the Vectara LLM can create the perfect markdown for you.

These are all things that you may be thinking “yes, those are great use cases for LLMs in general,” but marrying high quality retrieval (the R in RAG) with a prompt that can encapsulate the relevant data from retrieval has been hard for many users to execute.  At Vectara, we really believe the power in building LLM applications starts from using your data and helping you solve your use case.

Let’s walk through how you can achieve the first use case (an RFI answering bot) using custom prompts in Vectara.

Custom Prompts in Vectara

Vectara’s new release now supports Velocity Templates as a templating engine, which you can use to loop through the results and include metadata before sending to the generative LLM with an additional prompt.

This looks like the following:

[
    {
"role": "system",
"content": "You are an RFI answering assistant acting on behalf of the company Vectara. You are provided with search results from previously answered RFIs that may help answer the given question. The format of each result is the date in which it was answered and the response text.  You must summarize these results into a coherent answer. Only use information provided in this chat."},
    #foreach ($qResult in $vectaraQueryResults)
  	#if ($foreach.first)
    	{"role": "user", "content": "Search for \"$esc.java(${vectaraQuery})\", and give me the first search result."},
    	{"role": "assistant", "content": "$esc.java(${qResult.getText()})" },
  	#else
    	{"role": "user", "content": "Give me the $vectaraIdxWord[$foreach.index] search result."},
    	{"role": "assistant", "content": "qResult.docMetadata().get('answerDate') $esc.java(${qResult.getText()})" },
  	#end
    #end
    {
"role": "user",
"content": "Generate a comprehensive and informative answer for the question \"$esc.java(${vectaraQuery})\" solely based on the search results in this chat. You must only use information from the provided results. Combine search results together into a coherent answer. Do not repeat text. Only use the most relevant results that answer the question accurately. If there are 2 answers that seem in conflict, use the most recent answer according to the date.  If a result does not answer the question, do not use it. If the search results are not valid, respond with \"The returned results did not contain sufficient information to the question.\"."}
]

This is a long prompt, so let’s decompose it a bit.

First is just a bit of general prompt engineering: we’re going to tell the generative LLM to answer an RFI question and tell it how the results will come back from the query.  We’ll give it some specific instructions later on how to consider the data, such as any biases on the date of response.

Next, notice here that we can use this templating to iterate through $vectaraQueryResults, inserting the results in any order you’d like.  qResult.getText() will provide the most relevant snippet of text that answers the query from the result, so you can use this not only to tell the LLM where to focus its response, but you can even cut off or omit results based on your own business rules!  You can tell the LLM things like “give higher preference to results that show up earlier.”  You can tell the LLM what the query was specifically and reference the individual results.  And, again, you can even reference metadata like the date a question was answered.

With this templating language in place that, you reference the most relevant snippets from the result text and give you some utility functions for common RAG systems (like the word-based index – “first,” “second,” “third,” and so on that we’ve seen help LLMs understand the result).  And the entire end-to-end vectorization/retrieval/response generation can be handled in a single API request: no round trips or rewriting queries required on your part!  You can have a look at the variables and functions you have available in your prompt templates here.

Putting it to Use

Vectara becomes really powerful when you put it to use in an application or integration.  In this case, let’s use Vectara to answer our questions right out of Google Sheets.  Here is a small Google Sheet Apps Script that you can add to a questionnaire google sheet by adding it via Extensions → Apps Script:

After this, our Google Sheet has access to the knowledge base of questionnaires we’ve previously uploaded in a convenient Google Sheet function: VectaraRFIAnswer()

Give it a name, and click “Run” once to grant it the right permissions:

Let’s try it out:

Executing this, we get back the response, as provided from previous RFI answers:

Now let’s try rephrasing the question slightly to simulate an RFI from another company:

You can see here, Vectara is able to react to the second question by answering it more specifically than a copy/paste of the previous answer: by rephrasing the answer to directly address the question.

The possibilities of using this are truly endless.  You can now power completely custom workflows, integrations, and actions directly with a best-in-class retrieval system and custom prompt engine as well.

Conclusion

In this short release blog, we built an RFI question/answering bot in Google Sheets in under 70 lines of Javascript.  And while we think this alone is incredibly practical and useful, we’re even more excited to see what our users build with this new capability!  This is currently a Scale-only feature, so if you’d like to try it out, reach out to one of our GenAI experts to have a chat.

For the latest documentation on custom prompts and how to use them with Vectara, have a look at our Documentation.

As always, we’d love to hear your feedback!  Connect with us on our forums or on our Discord or on our community. If you’d like to see what Vectara can offer you for retrieval augmented generation on your application or website, sign up for an account!