Large Language Models
May 16, 2023 by Ofer Mendelevitch | 8 min readRead Now
Seeking knowledge about our world is something that all of us need. It is a task that goes back to the dawn of human civilization, ever since we figured out… Read more »
December 22, 2022 by Amr Awadallah
Seeking knowledge about our world is something that all of us need. It is a task that goes back to the dawn of human civilization, ever since we figured out how to capture our knowledge and share it (via drawings on the wall of a cave, or spoken words).
With the advent of Information Technology we got a way to easily search the massive amounts of knowledge that we are accumulating. However, the main techniques we evolved over the last few decades have been focused on the concept of keyword matching. I.e. use a few words describing what you are looking for to locate the information around it. That keyword approach has a number of drawbacks, but primarily it suffers from Semantic Loss. By that we mean the keywords on their own, without comprehending them, don’t capture the true meaning of what we are looking for, the true essence of what we seek. Keyword search tried to be better at that by “faking semantic” understanding of our queries. That process involves building a sophisticated knowledge graph and thousands of language rules that try to capture:
Building and more importantly maintaining these language rules is a very laborious task that overwhelms even the most sophisticated search teams. That is significantly more complicated if you have a multi-language audience (e.g. English, Spanish, and Chinese). You would now have to update these rules for every language separately, and you have to further refine that to support mixed language queries, for example: “should the Le hors-d’œuvre portion of a meal include fromage?”
To summarize the current state of affairs, we tried to make keyword search fake semantic understanding, which worked a bit, but is extremely difficult to build and maintain, leaving the mastering of this capability to only the most well funded search organizations.
Enter the 2010s, where we had a number of advancements in natural language understanding via neural networks and large language models (LLMs) that led to computers being able to comprehend human language at the adult level, and in some domains at the graduate level. LLM-powered search or Neural Search is truly the proper way to “Search by Meaning”, as compared to legacy keyword search techniques which could hardly fake it.
There are three seminal innovations that took place, which then led to a cascading effect of exponentially accelerating advancements:
The result of these rapid advancements was that by the end of 2018, neural networks became capable of comprehending the meaning of human language at the adult level. This was demonstrated by BERT beating state-of-the-art accuracy on a number of Natural Language Processing (NLP) benchmarks, including the Stanford QnA Data Set.
So now imagine you have at your disposal a graduate student whom you can ask any sophisticated question, for example “how does James Web work?”, and not only do they understand you mean NASA’s new telescope, but they then go and read all information about this topic and give you back a few paragraphs that truly answer the key essence of your inquiry! That is what we have today. Neural Search can read your accessible knowledge, map all of that knowledge from language space into a meaning space, then when you perform the search it returns the most relevant knowledge snippets back to you.
Did you know that there are more than 7000 human languages in the world? Human knowledge, our history, and our culture is being captured across all of these languages (at different scales). But today, when we search by legacy keyword search techniques, we are only getting back information from the same language that has these keywords. In the new world of Neural Search, all of these words, regardless of language, are converted from “word space” into a shared “meaning space” (A lingua franca space, or more appropriately a significatio franca space). This means that regardless of the language your query is in, or the language(s) the target knowledge is in, the Neural Search Engine will always return the most relevant answer to your question. For example, if I search in English for “How to cook the perfect paella?” the perfect recipe for that will most likely come from a Spanish source where the Paella dish was originally invented. Another example, if you are a global phone manufacturer, and one of your technicians in South Korea documented the perfect solution to a customer issue in Korean, the corresponding technician in Brazil who searches for that in Portuguese will still find that perfect answer.
In conclusion, Neural Search is the most effective way to do Semantic Search. We tried to fake Semantic Search via laborious hacks on top of Keyword Search, and that clearly was both suboptimal and expensive to achieve. Neural Search allows us to effectively search by the true essence, the true meaning, of the concepts we seek versus the keywords used to describe those concepts. Neural Search achieves significantly better search results, removes language as a barrier to seeking knowledge, and significantly reduces the costs of building and maintaining the next generation of search systems. Our mission at Vectara is to help the word find meaning. We started by enabling that through plug-n-play Neural Search, and we will be extending that to many other semantic capabilities in the future.