Application Development

January 30 , 2024 by Ofer Mendelevitch & Adeel

Introduction

“I’m pretty sure I saw an article about this new LLM, but for the life of me I can’t find it.”

Does this sound familiar?

It certainly happened to me more than once.

Today we’re announcing a new sample application that uses Vectara to solve this problem – we call it Sankofa, which means to retrieve  in the Twi language of Ghana. It is often represented by a bird with its head turned backwards. With Sankofa, all of the text of every webpage you’ve ever visited becomes searchable with Vectara’s powerful hybrid search. 

Setting up Your Vectara Account for Sankofa

To use Sankofa, first sign-up for your free Vectara account, and create a corpus (you can call it “sankofa” or any other name you choose). This corpus is where Sankofa will store all the text of pages you want it to index for you.

Click “Create Corpus” and your corpus is ready to go.

Next you’ll create an API key for your corpus. Choose your Sankofa corpus in the Vectara console, and click “Access Control”, then the “Create API Key” button. Make sure the key includes both the QueryService and IndexService permissions:

Now you are ready to install and configure Sankofa.

Installing Sankofa

Sankofa is a browser extension, and currently available for Chrome, Firefox and Edge for direct installation on your browser. 

Sankofa is open source (Apache 2.0 license) and is available on Github. 

Configuring Sankofa

Now that Sankofa is installed, let’s configure it. When you first click on the Sankofa icon, you get this screen:

Clicking on the “Let’s Configure” button opens up Sankofa’s configuration screen.

Here you enter your Vectara customer ID, corpus ID and API key. When you click “Save”, Sankofa validates your account credentials and saves them.

You have a few other parameters you can configure to control the behavior of Sankofa:

1. By default, Sankofa is configured so that you have to manually index pages as you are visiting them. If you check the “Auto send all pages” button, then Sankofa will automatically index every page you visit in your browser; use the drop down to add a delay before a page is indexed (e.g. index after I was on this page for 5 seconds).
2. With automated indexing, you can also define domains to exclude, such as github.com or youtube.com.

Using Sankofa

Now the fun part starts. Go ahead and index a few pages. For the example below, we indexed the following pages:

1. This Wikipedia Sankofa page.
2. This RAG page from the Vectara website

In this case, we used the main indexing button (“Index Page”) but note that Sankofa has another fun feature: you can select any portion of a webpage in your browser and then index just that text with a right-click to open a context menu.

Now we can ask “What is Sankofa”?

Sankofa opens up a new tab in the browser that shows the following:

This new tab shows the answer to your question (using Vectara’s powerful RAG pipeline) as well as the citations. Since we only visited a handful of pages, in this case, the citations all come from the single Wikipedia page on this topic that we indexed.

Another option is to use the “Find Similar Pages” button: here Sankofa will review the text on the current page and find pages previously indexed that have a similar semantic meaning to the current page.

Summary

We are thrilled to share this initial release of Sankofa, a browser extension that demonstrates the power of Vectara’s generative AI platform, by providing a way to retrieve information you need from web pages you visited. 

A famous proverb associated with the word Sankofa says: “It is not wrong to go back for that which you have forgotten” – Sankofa allows you to do just that.

We will continue to add features and functionality to Sankofa in the coming weeks and months, and are eager to hear how you use Sankofa – so please leave us comments about it on the repository, on Discord and in our forums.