Bitext mining describes the process of finding parallel (translated) sentence pairs in monolingual corpora. For example, you have an set of English sentences:
```
This is an example sentences.
Hello World!
My final third sentence in this list.
```
And a set of German sentences:
```
Hallo Welt!
Dies ist ein Beispielsatz.
Dieser Satz taucht im Englischen nicht auf.
```
Here, you want to find all translation pairs between the English set and the German set of languages.
The correct (two) translations are:
```
Hello World! Hallo Welt!
This is an example sentences. Dies ist ein Beispielsatz.
```
Usually you apply this method to large corpora, for example, you want to find all translated sentences in the English Wikipedia and the Chinese Wikipedia.
## Marging Based Mining
We follow the setup from [Artetxe and Schwenk, Section 4.3](https://arxiv.org/pdf/1812.10464.pdf) to find translated sentences in two datasets:
1) First, we encode all sentences to their respective embedding. As shown in [our paper](https://arxiv.org/abs/2004.09813) is [LaBSE](https://tfhub.dev/google/LaBSE/1) currently the best method for bitext mining. The model is integrated in Sentence-Transformers
2) Once we have all embeddings, we find the *k* nearest neighbor sentences for all sentences in both directions. Typical choices for k are between 4 and 16.
3) Then, we score all possible sentence combinations using the formula mentioned in Section 4.3.
4) The pairs with the highest scores are most likely translated sentences. Note, that the score can be larger than 1. Usually you have to find some cut-off where you ignore pairs below that threshold. For a high quality, a threshold of about 1.2 - 1.3 works quite well.
## Examples
-**[bucc2018.py](bucc2018.py)** - This script contains an example for the [BUCC 2018 shared task](https://comparable.limsi.fr/bucc2018/bucc2018-task.html) on finding parallel sentences. This dataset can be used to evaluate different strategies, as we know which sentences are parallel in the two corpora. The script mines for parallel sentences and then prints the optimal threshold that leads to the highest F1-score.
-**[bitext_mining.py](bitext_mining.py)** - This file reads in two text files (with a single sentence in each line) and outputs parallel sentences to *parallel-sentences-out.tsv.gz.
-**[In-domain Data Selection for MT](https://www.clinjournal.org/clinj/article/view/137)** - This paper also employed S-BERT to generate/select in-domain parallel data for machine translation systems – using monolingual texts.
Paraphrase mining is the task of finding paraphrases (texts with identical / similar meaning) in a large corpus of sentences. In [Semantic Textual Similarity](../../../docs/usage/semantic_textual_similarity.md) we saw a simplified version of finding paraphrases in a list of sentences. The approach presented there used a brute-force approach to score and rank all pairs.
However, as this has a quadratic runtime, it fails to scale to large (10,000 and more) collections of sentences.
For larger collections, *util* offers the *paraphrase_mining* function that can be used like this:
Instead of computing all pairwise cosine scores and ranking all possible, combinations, the approach is a bit more complex (and hence efficient). We chunk our corpus into smaller pieces, which is defined by *query_chunk_size* and *corpus_chunk_size*. For example, if we set *query_chunk_size=1000*, we search paraphrases for 1,000 sentences at a time in the remaining corpus (all other sentences). However, the remaining corpus is also chunked, for example, if we set *corpus_chunk_size=10000*, we look for paraphrases in 10k sentences at a time.
If we pass a list of 20k sentences, we will chunk it to 20x1000 sentences, and each of the query is compared first against sentences 0-10k and then 10k-20k.
This is done to reduce the memory requirement. Increasing both values improves the speed, but increases also the memory requirement.
The next critical thing is finding the pairs with the highest similarities. Instead of getting and sorting all n^2 pairwise scores, we take for each query only the *top_k* scores. So with *top_k=100*, we find at most 100 paraphrases per sentence per chunk. You can play around with *top_k* to the ensure a certain behaviour.
You will get for each sentence only the one most other relevant sentence. Note, if B is the most similar sentence for A, A must not be the most similar sentence for B. So it can happen that the returned list contains entries like (A, B) and (B, C).
The final relevant parameter is *max_pairs*, which determines the maximum number of paraphrase pairs you like to get returned. If you set it to e.g. *max_pairs=100*, you will not get more than 100 paraphrase pairs returned. Usually, you get fewer pairs returned as the list is cleaned of duplicates, e.g., if it contains (A, B) and (B, A), then only one is returned.
In [Semantic Search](../semantic-search/README.md) we have shown how to use SentenceTransformer to compute embeddings for queries, sentences, and paragraphs and how to use this for semantic search.
For complex search tasks, for example, for question answering retrieval, the search can significantly be improved by using **Retrieve & Re-Rank**.
## Retrieve & Re-Rank Pipeline
A pipeline for information retrieval / question answering retrieval that works well is the following. All components are provided and explained in this article:
Given a search query, we first use a **retrieval system** that retrieves a large list of e.g. 100 possible hits which are potentially relevant for the query. For the retrieval, we can use either lexical search, e.g. with Elasticsearch, or we can use dense retrieval with a bi-encoder.
However, the retrieval system might retrieve documents that are not that relevant for the search query. Hence, in a second stage, we use a **re-ranker** based on a **cross-encoder** that scores the relevancy of all candidates for the given search query.
The output will be a ranked list of hits we can present to the user.
## Retrieval: Bi-Encoder
For the retrieval of the candidate set, we can either use lexical search (e.g. [Elasticsearch](https://www.elastic.co/elasticsearch/)), or we can use a bi-encoder which is implemented in this repository.
Lexical search looks for literal matches of the query words in your document collection. It will not recognize synonyms, acronyms or spelling variations. In contrast, semantic search (or dense retrieval) encodes the search query into vector space and retrieves the document embeddings that are close in vector space.
Semantic search overcomes the short comings of lexical search and can recognize synonym and acronyms. Have a look at the [semantic search article](../semantic-search/README.md) for different options to implement semantic search.
## Re-Ranker: Cross-Encoder
The retriever has to be efficient for large document collections with millions of entries. However, it might return irrelevant candidates.
A re-ranker based on a Cross-Encoder can substantially improve the final results for the user. The query and a possible document is passed simultaneously to transformer network, which then outputs a single score between 0 and 1 indicating how relevant the document is for the given query.
The advantage of Cross-Encoders is the higher performance, as they perform attention across the query and the document.
Scoring thousands or millions of (query, document)-pairs would be rather slow. Hence, we use the retriever to create a set of e.g. 100 possible candidates which are then re-ranked by the Cross-Encoder.
## Example Scripts
***[retrieve_rerank_simple_wikipedia.ipynb](retrieve_rerank_simple_wikipedia.ipynb)**[ [Colab Version](https://colab.research.google.com/github/UKPLab/sentence-transformers/blob/master/examples/applications/retrieve_rerank/retrieve_rerank_simple_wikipedia.ipynb) ]: This script uses the smaller [Simple English Wikipedia](https://simple.wikipedia.org/wiki/Main_Page) as document collection to provide answers to user questions / search queries. First, we split all Wikipedia articles into paragraphs and encode them with a bi-encoder. If a new query / question is entered, it is encoded by the same bi-encoder and the paragraphs with the highest cosine-similarity are retrieved (see [semantic search](../semantic-search/README.md)). Next, the retrieved candidates are scored by a Cross-Encoder re-ranker and the 5 passages with the highest score from the Cross-Encoder are presented to the user.
-**[in_document_search_crossencoder.py](in_document_search_crossencoder.py):** If you only have a small set of paragraphs, we don't do the retrieval stage. This is for example the case if you want to perform search within a single document. In this example, we take the Wikipedia article about Europe and split it into paragraphs. Then, the search query / question and all paragraphs are scored using the Cross-Encoder re-ranker. The most relevant passages for the query are returned.
## Pre-trained Bi-Encoders (Retrieval)
The bi-encoder produces embeddings independently for your paragraphs and for your search queries. You can use it like this:
This example show how in-document search can be used with a CrossEncoder.
The document is split into passage. Here, we use three consecutive sentences as a passage. You can use shorter passage, for example, individual sentences,
or longer passages, like full paragraphs.
The CrossEncoder takes the search query and scores every passage how relevant the passage is for the given score. The five passages with the highest score are then returned.
As CrossEncoder, we use cross-encoder/ms-marco-TinyBERT-L-2, a BERT model with only 2 layers trained on the MS MARCO dataset. This is an extremely quick model able to score up to 9000 passages per second (on a V100 GPU). You can also use a larger model, which gives better results but is also slower.
Note: As we score the [query, passage]-pair for every new query, this search method
becomes at some point in-efficient if the document gets too large.
Usage: python in_document_search_crossencoder.py
Note: Requires NLTK: `pip install nltk`
"""
fromsentence_transformersimportCrossEncoder
fromnltkimportsent_tokenize
importtime
# As document, we take the first two section from the Wikipedia article about Europe
document="""Europe is a continent located entirely in the Northern Hemisphere and mostly in the Eastern Hemisphere. It comprises the westernmost part of Eurasia and is bordered by the Arctic Ocean to the north, the Atlantic Ocean to the west, the Mediterranean Sea to the south, and Asia to the east. Europe is commonly considered to be separated from Asia by the watershed of the Ural Mountains, the Ural River, the Caspian Sea, the Greater Caucasus, the Black Sea, and the waterways of the Turkish Straits. Although some of this border is over land, Europe is generally accorded the status of a full continent because of its great physical size and the weight of history and tradition.
Europe covers about 10,180,000 square kilometres (3,930,000 sq mi), or 2% of the Earth's surface (6.8% of land area), making it the second smallest continent. Politically, Europe is divided into about fifty sovereign states, of which Russia is the largest and most populous, spanning 39% of the continent and comprising 15% of its population. Europe had a total population of about 741 million (about 11% of the world population) as of 2018. The European climate is largely affected by warm Atlantic currents that temper winters and summers on much of the continent, even at latitudes along which the climate in Asia and North America is severe. Further from the sea, seasonal differences are more noticeable than close to the coast.
European culture is the root of Western civilization, which traces its lineage back to ancient Greece and ancient Rome. The fall of the Western Roman Empire in 476 AD and the subsequent Migration Period marked the end of Europe's ancient history and the beginning of the Middle Ages. Renaissance humanism, exploration, art and science led to the modern era. Since the Age of Discovery, started by Portugal and Spain, Europe played a predominant role in global affairs. Between the 16th and 20th centuries, European powers colonized at various times the Americas, almost all of Africa and Oceania, and the majority of Asia.
The Age of Enlightenment, the subsequent French Revolution and the Napoleonic Wars shaped the continent culturally, politically and economically from the end of the 17th century until the first half of the 19th century. The Industrial Revolution, which began in Great Britain at the end of the 18th century, gave rise to radical economic, cultural and social change in Western Europe and eventually the wider world. Both world wars took place for the most part in Europe, contributing to a decline in Western European dominance in world affairs by the mid-20th century as the Soviet Union and the United States took prominence. During the Cold War, Europe was divided along the Iron Curtain between NATO in the West and the Warsaw Pact in the East, until the revolutions of 1989 and fall of the Berlin Wall.
In 1949, the Council of Europe was founded with the idea of unifying Europe to achieve common goals. Further European integration by some states led to the formation of the European Union (EU), a separate political entity that lies between a confederation and a federation. The EU originated in Western Europe but has been expanding eastward since the fall of the Soviet Union in 1991. The currency of most countries of the European Union, the euro, is the most commonly used among Europeans; and the EU's Schengen Area abolishes border and immigration controls between most of its member states. There exists a political movement favoring the evolution of the European Union into a single federation encompassing much of the continent.
In classical Greek mythology, Europa (Ancient Greek: Εὐρώπη, Eurṓpē) was a Phoenician princess. One view is that her name derives from the ancient Greek elements εὐρύς (eurús), "wide, broad" and ὤψ (ōps, gen. ὠπός, ōpós) "eye, face, countenance", hence their composite Eurṓpē would mean "wide-gazing" or "broad of aspect". Broad has been an epithet of Earth herself in the reconstructed Proto-Indo-European religion and the poetry devoted to it. An alternative view is that of R.S.P. Beekes who has argued in favor of a Pre-Indo-European origin for the name, explaining that a derivation from ancient Greek eurus would yield a different toponym than Europa. Beekes has located toponyms related to that of Europa in the territory of ancient Greece and localities like that of Europos in ancient Macedonia.
There have been attempts to connect Eurṓpē to a Semitic term for "west", this being either Akkadian erebu meaning "to go down, set" (said of the sun) or Phoenician 'ereb "evening, west", which is at the origin of Arabic Maghreb and Hebrew ma'arav. Michael A. Barry finds the mention of the word Ereb on an Assyrian stele with the meaning of "night, [the country of] sunset", in opposition to Asu "[the country of] sunrise", i.e. Asia. The same naming motive according to "cartographic convention" appears in Greek Ἀνατολή (Anatolḗ "[sun] rise", "east", hence Anatolia). Martin Litchfield West stated that "phonologically, the match between Europa's name and any form of the Semitic word is very poor", while Beekes considers a connection to Semitic languages improbable. Next to these hypotheses there is also a Proto-Indo-European root *h1regʷos, meaning "darkness", which also produced Greek Erebus.
Most major world languages use words derived from Eurṓpē or Europa to refer to the continent. Chinese, for example, uses the word Ōuzhōu (歐洲/欧洲), which is an abbreviation of the transliterated name Ōuluóbā zhōu (歐羅巴洲) (zhōu means "continent"); a similar Chinese-derived term Ōshū (欧州) is also sometimes used in Japanese such as in the Japanese name of the European Union, Ōshū Rengō (欧州連合), despite the katakana Yōroppa (ヨーロッパ) being more commonly used. In some Turkic languages, the originally Persian name Frangistan ("land of the Franks") is used casually in referring to much of Europe, besides official names such as Avrupa or Evropa."""
## We split this article into paragraphs and then every paragraph into sentences
"# Retrieve & Re-Rank Demo over Simple Wikipedia\n",
"\n",
"This examples demonstrates the Retrieve & Re-Rank Setup and allows to search over [Simple Wikipedia](https://simple.wikipedia.org/wiki/Main_Page).\n",
"\n",
"You can input a query or a question. The script then uses semantic search\n",
"to find relevant passages in Simple English Wikipedia (as it is smaller and fits better in RAM).\n",
"\n",
"For semantic search, we use `SentenceTransformer('multi-qa-MiniLM-L6-cos-v1')` and retrieve\n",
"32 potentially passages that answer the input query.\n",
"\n",
"Next, we use a more powerful CrossEncoder (`cross_encoder = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')`) that\n",
"scores the query and all retrieved passages for their relevancy. The cross-encoder further boost the performance,\n",
"especially when you search over a corpus for which the bi-encoder was not trained for.\n"
"Input question: What is the capital of the United States?\n",
"Top-3 lexical search (BM25) hits\n",
"\t13.316\tCapital punishment (the death penalty) has existed in the United States since before the United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states. The federal government (including the United States military) also uses capital punishment.\n",
"\t11.434\tOhio is one of the 50 states in the United States. Its capital is Columbus. Columbus also is the largest city in Ohio.\n",
"\t11.179\tNevada is one of the United States' states. Its capital is Carson City. Other big cities are Las Vegas and Reno.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Bi-Encoder Retrieval hits\n",
"\t0.622\tCities in the United States:\n",
"\t0.597\tThe United States Capitol is the building where the United States Congress meets. It is the center of the legislative branch of the U.S. federal government. It is in Washington, D.C., on top of Capitol Hill at the east end of the National Mall.\n",
"\t0.596\tIn the United States:\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Cross-Encoder Re-ranker hits\n",
"\t8.906\tWashington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district. The President of the USA and many major national government offices are in the territory. This makes it the political center of the United States of America.\n",
"\t3.755\tA capital city (or capital town or just capital) is a city or town, specified by law or constitution, by the government of a country, or part of a country, such as a state, province or county. It usually serves as the location of the government's central meeting place and offices. Most of the country's leaders and officials work in the capital city.\n",
"\t3.681\tThe United States Capitol is the building where the United States Congress meets. It is the center of the legislative branch of the U.S. federal government. It is in Washington, D.C., on top of Capitol Hill at the east end of the National Mall.\n"
]
}
],
"source": [
"search(query = \"What is the capital of the United States?\")"
"Input question: What is the best orchestra in the world?\n",
"Top-3 lexical search (BM25) hits\n",
"\t15.328\tThe BBC Symphony Orchestra is the main orchestra of the British Broadcasting Corporation. It is one of the best orchestras in Britain.\n",
"\t15.320\tThe NHK Symphony Orchestra is a Japanese orchestra based in Tokyo, Japan. In Japanese it is written: NHK交響楽団, pronounced: Enueichikei Kōkyō Gakudan. When the orchestra was started in 1926 it was called \"New Symphony Orchestra\". It was the first large professional orchestra in Japan. Later, it changed its name to \"Japan Symphony Orchestra\". In 1951 it started to get money from the Japanese radio station NHK (Nippon Hōsō Kyōkai), so it changed its name again to the name it has now. It is thought of as the best orchestra in Japan. They have played in many parts of the world, including at the BBC Proms in London.\n",
"\t14.079\tThe Bamberger Symphoniker (Bamberg Symphony Orchestra) is a world-famous orchestra from the city of Bamberg, Germany. It was formed in 1946. Most of the musicians who formed the orchestra were Germans who had been forced to leave Czechoslovakia after the World War II. Most of them had previously been members of the German Philharmonic Orchestra of Prague.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Bi-Encoder Retrieval hits\n",
"\t0.701\tThe Vienna Philharmonic (in German: die Wiener Philharmoniker) is an orchestra based in Vienna, Austria. It is thought of as one of the greatest orchestras in the world.\n",
"\t0.641\tThe Vienna Symphony () is an orchestra in Vienna, Austria.\n",
"\t0.640\tThe Berlin Philharmonic (in German: Die Berliner Philharmoniker), is an orchestra from Berlin, Germany. It is one of the greatest orchestras in the world. The conductor of the orchestra is Sir Simon Rattle.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Cross-Encoder Re-ranker hits\n",
"\t5.952\tThe London Symphony Orchestra (LSO) is one of the most famous orchestras of the world. They are based in London's Barbican Centre, but they often tour to lots of different countries.\n",
"\t5.794\tThe Vienna Philharmonic (in German: die Wiener Philharmoniker) is an orchestra based in Vienna, Austria. It is thought of as one of the greatest orchestras in the world.\n",
"\t5.324\tThe Berlin Philharmonic (in German: Die Berliner Philharmoniker), is an orchestra from Berlin, Germany. It is one of the greatest orchestras in the world. The conductor of the orchestra is Sir Simon Rattle.\n"
]
}
],
"source": [
"search(query = \"What is the best orchestra in the world?\")"
"\t13.795\tAmy MacDonald is a Scottish singer and songwriter. She became famous in 2007 with her first album \"This Is The Life\" and her first single \"Poison Prince\". She has become even more successful in Europe since her single \"This Is The Life\" charted at number 1 in many European countries.\n",
"\t13.758\tThe Croatian language is spoken mainly throughout the countries of Croatia and Bosnia and Herzegovina and in the surrounding countries of Europe.\n",
"\t13.019\tOrganization for Security and Co-operation in Europe (OSCE) is an international organization for peace and human rights. Presently, it has 57 countries as its members. Most of the member countries of the OSCE are from Europe, the Caucasus, Central Asia and North America.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Bi-Encoder Retrieval hits\n",
"\t0.538\tThe Council of Europe (, ) is an international organization of 47 member states in the European region. One of its first successes was the European Convention on Human Rights in 1950, which serves as the basis for the European Court of Human Rights.\n",
"\t0.531\tEngland is a country in Europe. It is a country with over sixty cities in it. It is in a union with Scotland, Wales and Northern Ireland. All four countries are in the British Isles and are part of the United Kingdom (UK).\n",
"\t0.507\tEurope is a Swedish rock band. The band was started by Joey Tempest and John Norum in 1979. Their song \"The Final Countdown\" was a big hit in 1986.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Cross-Encoder Re-ranker hits\n",
"\t5.199\tThe European Union (abbreviation: EU) is a confederation of 27 member countries in Europe established by the Maastricht Treaty in 1992-1993. The EU grew out of the European Economic Community (EEC) which was established by the Treaties of Rome in 1957. It has created a common economic area with Europe-wide laws allowing the citizens of EU countries to move and trade in other EU countries almost the same as they do in their own. Nineteen of these countries also share the same type of money: the euro.\n",
"\t3.202\tA European Union member state is any one of the twenty-seven countries that have joined the European Union (EU) since it was found in 1958 as the European Economic Community (EEC). From an original membership of six states, there have been five successive enlargements. The largest happened on 1 May 2004, when ten member states joined.\n",
"\t2.798\tThe Schengen Area is an area that includes 26 European countries. All of those countries have signed the Schengen Agreement in Schengen, Luxembourg in 1985.\n"
"\t17.374\tThe Cold War was the tense relationship between the United States (and its allies), and the Soviet Union (the USSR and its allies) between the end of World War II and the fall of the Soviet Union. It is called the \"Cold\" War because the US and the USSR never actually fought each other directly. Instead, they opposed each other in conflicts known as proxy wars, where each country chose a side to support.\n",
"\t17.291\tThe Reagan Doctrine was a document by the United States under the Reagan Administration. It was about being against the global influence of the Soviet Union during the final years of the Cold War. The doctrine lasted for less than a decade, it was the most important document of United States foreign policy from the early 1980s until the end of the Cold War in 1991.\n",
"\t15.420\tCold Norton is a village and civil parish in Maldon District, Essex, England. In 2001 there were 1103 people living in Cold Norton. Cold Norton is at the south-east end of the Danbury Ridge.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Bi-Encoder Retrieval hits\n",
"\t0.613\tThe Cold War was the tense relationship between the United States (and its allies), and the Soviet Union (the USSR and its allies) between the end of World War II and the fall of the Soviet Union. It is called the \"Cold\" War because the US and the USSR never actually fought each other directly. Instead, they opposed each other in conflicts known as proxy wars, where each country chose a side to support.\n",
"\t0.557\tThe Continuation War was a war between Finland and the Soviet Union. It was fought between June 25, 1941 and September 19, 1944. It continued the Winter War. Nazi Germany helped Finland as part of the Eastern Front (World War II). The ceasefire started on September 4 at 7.00 am in the Finnish side. The Soviet Union's ceasefire started on September 5. The first peace treaty was signed on September 19 and the final one on February 10, 1947 in Paris.\n",
"\t0.548\tThe Winter War (30 November 1939 - 13 March 1940) was a conflict fought between the Soviet Union and Finland. It began when the Soviet Union tried to invade Finland soon after the Invasion of Poland. The Soviet military forces expected a victory over Finland in a few weeks, because the Soviet army had many more tanks and planes than the Finnish army.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Cross-Encoder Re-ranker hits\n",
"\t5.203\tThe Cold War was the tense relationship between the United States (and its allies), and the Soviet Union (the USSR and its allies) between the end of World War II and the fall of the Soviet Union. It is called the \"Cold\" War because the US and the USSR never actually fought each other directly. Instead, they opposed each other in conflicts known as proxy wars, where each country chose a side to support.\n",
"\t2.531\tA speech was made by American President Harry S. Truman to the U.S. Congress on 12 March 1947. In this speech he said he thought that The United States should help Greece and Turkey to stop them being 'Totalitarianists' although he meant Soviet Communism. This became known as the Truman Doctrine. Some Historians believe that this was the start of the Cold War.\n",
"\t2.079\tThe Sino-Soviet split (1960–1989) was a time when the relations between the People's Republic of China and the Soviet Union weakened during the Cold War. Eventually, China's leader, Mao Zedong, decided to break the alliance with the Soviet Union.\n"
"\t22.997\tReliable information on the lifespans of house cats is hard to find. However, research has been done to get an estimate (an educated guess) on how long cats usually live. Cats usually live for 13 to 20 years. Sometimes cats can live for 22 to 30 years but there are claims of cats dying at ages of more than 30 years old.\n",
"\t16.974\tThe sabertoothed cats or sabretooth cats are some of the best known and most popular extinct animals. They are among the most impressive carnivores that ever have lived. These cats had long canines and jaws which opened wider than modern cats. This suggests a different style of killing from modern felines.\n",
"\t16.490\tThe Cyprus cat is a breed of cat. These cats are thought to have first come from ancient Egypt or Palestine. They were brought to the island of Cyprus by St. Helen. These are now common domestic cats that live in homes or outside. Many of these cats still live all over Cyprus. But, a large number are now feral. This means they are not tame and they run wild.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Bi-Encoder Retrieval hits\n",
"\t0.765\tReliable information on the lifespans of house cats is hard to find. However, research has been done to get an estimate (an educated guess) on how long cats usually live. Cats usually live for 13 to 20 years. Sometimes cats can live for 22 to 30 years but there are claims of cats dying at ages of more than 30 years old.\n",
"\t0.531\tCreme Puff (August 3, 1967 - August 10, 2005) was a female cat who died in 2005 at the age of 38. She was the oldest cat ever recorded, according to the 2010 edition of \"Guinness World Records\".\n",
"\t0.508\tThe cat righting reflex is a cat's natural ability to turn itself around as it falls so it will land on its feet. This righting reflex starts to happen at 3–4 weeks of age. The cat has entirely learned how to do this by 6–7 weeks. Cats are able to do this because they have a flexible backbone and a clavicle that does not move. The minimum height needed for this to happen safely in most cats is about 12 inches.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Cross-Encoder Re-ranker hits\n",
"\t10.431\tReliable information on the lifespans of house cats is hard to find. However, research has been done to get an estimate (an educated guess) on how long cats usually live. Cats usually live for 13 to 20 years. Sometimes cats can live for 22 to 30 years but there are claims of cats dying at ages of more than 30 years old.\n",
"\t2.998\tThe sand cat (\"Felis margarita\") is a small wild cat in the Felinae subfamily. It is distributed over African and Asian deserts. Sometimes people call it \"desert cat,\" but that is really the name of a different animal. The sand cat does live in deserts, even the Sahara and Arabian Desert. It is also found in Iran and Pakistan. In zoos, this cat can live for up to 13 years.\n",
"\t2.348\tBobcat (\"Lynx rufus\") are fierce cats that live in forests, swamps, mountains, prairie, and deserts in much of North America. Bobcats are generally nocturnal (most active at night), but are most active at dawn and dusk. They spend the day in their den (a cave, hollow log or rock crevice). They are very good climbers and swimmers. Bobcats are eaten by cougars, coyotes, wolves, and bears. Bobcats usually live from 10 to 14 years. Bobcats and lynxes are closely related.\n"
"Input question: How many people live in Toronto?\n",
"Top-3 lexical search (BM25) hits\n",
"\t15.978\tMarkham, Ontario is a city in Regional Municipality of York, in the Greater Toronto Area of Southern Ontario, Canada. There are twice as many people there as in 1990. 261,573 people live in Markham. It is the 4th largest town in the Greater Toronto Area, after Toronto, Mississauga, and Brampton.\n",
"\t11.299\tThe Toronto Zoo is a zoo in Toronto, Ontario, Canada. With , the Toronto Zoo is the largest zoo in Canada.\n",
"\t10.679\tDenzil Minnan-Wong (; born ) is a Canadian politician. He is a Toronto city councillor. He is the person that represents Ward 16, an area of Toronto. He is a chairperson of the Employee and Labour Relations Committee in Toronto's municipal government and is also the deputy mayor of Toronto. He is also part of the board of the Toronto Transit Commission and the Toronto Hydro.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Bi-Encoder Retrieval hits\n",
"\t0.604\tVaughan is a city in Ontario, Canada, 335,000 people live there .\n",
"\t0.595\tToronto is a city in Ohio in the United States.\n",
"\t0.594\tToronto is a city in Kansas, United States.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Cross-Encoder Re-ranker hits\n",
"\t4.448\tIt has about 110,000 people living there.\n",
"\t3.632\tThe 2010 census counted 23,302 people living in the city. (2005 showed 24,709). The city was founded on January 1, 1955. Much of the city was heavily damaged by the 2011 Tōhoku earthquake and tsunami.\n",
"\t3.372\tAs of January 1, 2010, about 164,294 people lived there. That was 295.83 people per km². The total area is 555.35 km².\n"
]
}
],
"source": [
"search(query = \"How many people live in Toronto?\")"
"\t11.010\tGlafcos Ioannou Clerides (; 24 April 1919 – 15 November 2013) was a Greek-Cypriot politician. He was the fourth President of Cyprus. He was the oldest living former President of the Republic of Cyprus.\n",
"\t9.237\tJosé Celso de Mello Filho (Tatuí, November 1, 1945), is a Brazilian jurist. He is the oldest member of the Supreme Federal Court of Brazil. He was nominated by President José Sarney in 1989.\n",
"\t8.872\tUSS \"Constitution\" is a wooden, three-masted heavy frigate of the United States Navy. Named by President George Washington after the Constitution of the United States of America, she is the world's oldest commissioned naval vessel afloat.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Bi-Encoder Retrieval hits\n",
"\t0.645\tWilliam Henry Harrison (February 9, 1773 – April 4, 1841) was the 9th President of the United States. His nickname was \"Old Tippecanoe \" and he was a well-respected war veteran. Harrison served the shortest term of any United States President. His term lasted for exactly one month.\n",
"\t0.624\tRichard Arvin Overton (May 11, 1906 – December 27, 2018) was an American supercentenarian. He was the oldest verified surviving American World War II veteran, as well as the oldest American man. In 2013 he was honored by President Barack Obama. On that same Memorial Day, Overton met with Texas Governor Rick Perry. He was born in Bastrop County, Texas.\n",
"\t0.620\tAlben William Barkley (November 24, 1877 – April 30, 1956) was a Democratic member of the U.S. House of Representatives and the United States Senate from Paducah, Kentucky, majority leader of the Senate, and the thirty-fifth Vice President of the United States. He was the oldest Vice President of the United States at the age of . He ran for president in 1952, however the Democratic Party primaries had nominated Adlai Stevenson II instead and labor union leaders rejected him to run because of old age (74).\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Cross-Encoder Re-ranker hits\n",
"\t6.089\tAlben William Barkley (November 24, 1877 – April 30, 1956) was a Democratic member of the U.S. House of Representatives and the United States Senate from Paducah, Kentucky, majority leader of the Senate, and the thirty-fifth Vice President of the United States. He was the oldest Vice President of the United States at the age of . He ran for president in 1952, however the Democratic Party primaries had nominated Adlai Stevenson II instead and labor union leaders rejected him to run because of old age (74).\n",
"\t5.538\tRichard Arvin Overton (May 11, 1906 – December 27, 2018) was an American supercentenarian. He was the oldest verified surviving American World War II veteran, as well as the oldest American man. In 2013 he was honored by President Barack Obama. On that same Memorial Day, Overton met with Texas Governor Rick Perry. He was born in Bastrop County, Texas.\n",
"\t4.702\tJohn Nance Garner IV nicknamed \"Cactus Jack\" (November 22, 1868 – November 7, 1967) was the forty-fourth Speaker of the United States House of Representatives (1931-33) and the thirty-second Vice President of the United States (1933-41). Garner once described the Vice-Presidency as being \"not worth a bucket of warm spit.\" Also, he lived to be 98 years old. That made him the oldest former Vice President of the United States.\n"
"\t24.891\tEast Antarctica, also called Greater Antarctica, is the largest part (two-thirds) of the Antarctic continent. It is on the Indian Ocean side of the Transantarctic Mountains. It is the coldest, windiest, and driest part of Earth. East Antarctica holds the record as the coldest place on earth.\n",
"\t12.650\tEarth Day is a day that is supposed to inspire more awareness and appreciation for the Earth's natural environment. It takes place each year on April 22. It now takes place in more than 193 countries around the world. During Earth Day, the world encourages everyone to turn off all unwanted lights.\n",
"\t12.172\tHeinrich events occurred during the coldest point of \"Bond Cycles\" in which many icebergs were discharged into the North Atlantic and melted.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Bi-Encoder Retrieval hits\n",
"\t0.633\tEast Antarctica, also called Greater Antarctica, is the largest part (two-thirds) of the Antarctic continent. It is on the Indian Ocean side of the Transantarctic Mountains. It is the coldest, windiest, and driest part of Earth. East Antarctica holds the record as the coldest place on earth.\n",
"\t0.556\tThe North Pole is the point that is farthest north on Earth. It is the point on which axis of Earth turns. It is in the Arctic Ocean and it is cold there because the sun does not shine there for about half a year and never rises very high. The ocean around the pole is always very cold and it is covered by a thick sheet of ice.\n",
"\t0.516\tPlaces with a subarctic climate (also called boreal climate) have long, usually very cold winters, and short, warm summers. It is found on large landmasses, away from oceans, usually at latitudes from 50° to 70°N. Because there are no large landmasses at such latitudes in the Southern Hemisphere, it is only found at high \"altitudes \"(heights) in the Andes and the mountains of Australia and New Zealand's South Island. These climates are in groups \"Dfc\", \"Dwc\", \"Dfd\" and \"Dwd\" in the Köppen climate classification\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Cross-Encoder Re-ranker hits\n",
"\t6.129\tEast Antarctica, also called Greater Antarctica, is the largest part (two-thirds) of the Antarctic continent. It is on the Indian Ocean side of the Transantarctic Mountains. It is the coldest, windiest, and driest part of Earth. East Antarctica holds the record as the coldest place on earth.\n",
"\t0.594\tThe Arctic is the area around the Earth's North Pole. The Arctic includes parts of Russia, Alaska, Canada, Greenland, Lapland and Svalbard as well as the Arctic Ocean. It is an ocean, mostly covered with ice. Most scientists call the area north of the treeline Arctic. Trees will not grow when the temperatures get too cold. The forests of the continents stop when they get too far north or too high up a mountain. (Higher places are colder, too.) The place where in the trees stop is called the tree line.\n",
"\t0.215\tThe North Pole is the point that is farthest north on Earth. It is the point on which axis of Earth turns. It is in the Arctic Ocean and it is cold there because the sun does not shine there for about half a year and never rises very high. The ocean around the pole is always very cold and it is covered by a thick sheet of ice.\n"
"\t23.364\tTesla, Inc. is a company based in Palo Alto, California which makes electric cars. It was started in 2003 by Martin Eberhard, Dylan Stott, and Elon Musk (who also co-founded PayPal and SpaceX and is the CEO of SpaceX). Eberhard no longer works there. Today, Elon Musk is the Chief Executive Officer (CEO). It started selling its first car, the Roadster in 2008.\n",
"\t19.943\tThe Boring Company is a tunnel boring company founded by Elon Musk, who earlier started SpaceX. It aims to reduce traffic congestion in urban areas. It is involved in the building of the Hyperloop in Los Angeles.\n",
"\t18.392\tElon Reeve Musk (born June 28, 1971) is a businessman and philanthropist. He was born in South Africa. He moved to Canada and later became an American citizen. Musk is the current CEO & Chief Product Architect of Tesla Motors, a company that makes electric vehicles. He is also the CEO of Solar City, a company that makes solar panels, and the CEO & CTO of SpaceX, an aerospace company. In August 2020, Bloomberg ranked Musk third among the richest people on the planet with net worth to be $115.4 billion.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Bi-Encoder Retrieval hits\n",
"\t0.574\tElon Reeve Musk (born June 28, 1971) is a businessman and philanthropist. He was born in South Africa. He moved to Canada and later became an American citizen. Musk is the current CEO & Chief Product Architect of Tesla Motors, a company that makes electric vehicles. He is also the CEO of Solar City, a company that makes solar panels, and the CEO & CTO of SpaceX, an aerospace company. In August 2020, Bloomberg ranked Musk third among the richest people on the planet with net worth to be $115.4 billion.\n",
"\t0.471\tMuraoka was born on June 21, 1893, in Kofu, Yamanashi Prefecture. Her birth name was Hana Annaka. Her parents were Methodists. She was raised as a Christian. She studied at the Tokyo Eiwa Jogakuin. She began writing children's stories when she was encouraged by translator Hiroko Katayama. She graduated from school in 1913.\n",
"\t0.467\tJose Alves dos Santos Júnior (born July 29, 1969) is a former Brazilian football player.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Cross-Encoder Re-ranker hits\n",
"\t7.449\tElon Reeve Musk (born June 28, 1971) is a businessman and philanthropist. He was born in South Africa. He moved to Canada and later became an American citizen. Musk is the current CEO & Chief Product Architect of Tesla Motors, a company that makes electric vehicles. He is also the CEO of Solar City, a company that makes solar panels, and the CEO & CTO of SpaceX, an aerospace company. In August 2020, Bloomberg ranked Musk third among the richest people on the planet with net worth to be $115.4 billion.\n",
"\t-2.518\tNour El-Sherif (; 28 April 1946 – 11 August 2015) was a Egyptian actor. His birth name is Mohamad Geber Mohamad ِAbd Allah (Arabic: محمد جابر محمد عبد الله). El-Sherif was born in the working-class neighbourhood of Sayeda Zainab in Cairo (). He was known for his conspiracy theories about the Holocaust and the September 11 attacks.\n",
"\t-4.171\tOlly Murs (born 14 May 1984) is an English singer. He comes from Witham, Essex. He became famous after he finished second in The X Factor in 2009.\n"
"\t27.300\tThe Eiffel Tower (French: La Tour Eiffel, ], IPA pronunciation: \"EYE-full\" English; \"eh-FEHL\" French) is a landmark in Paris. It was built between 1887 and 1889 for the Exposition Universelle (World Fair). The Tower was the Exposition's main attraction.\n",
"\t25.263\tParis is a city in the U.S. state of Texas. It is in Lamar County, Texas. It had a population of 25,171 in 2010. It has been called the \"Second Largest Paris in the World\". It has a replica of the Eiffel Tower.\n",
"\t24.059\tParis is a city in the U.S. state of Tennessee. It had a population of 25,171 in 2010. It has been called the \"World's Biggest Fish Fry\". It has a 70-foot replica of the Eiffel Tower.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Bi-Encoder Retrieval hits\n",
"\t0.812\tThe Eiffel Tower (French: La Tour Eiffel, ], IPA pronunciation: \"EYE-full\" English; \"eh-FEHL\" French) is a landmark in Paris. It was built between 1887 and 1889 for the Exposition Universelle (World Fair). The Tower was the Exposition's main attraction.\n",
"\t0.626\tAlexandre Gustave Eiffel (December 15, 1832 – December 27, 1923; , ) was a French structural engineer and architect. He is known for designing the Eiffel Tower. He also designed the armature (supporting framework) for the Statue of Liberty, New York Harbor, United States.\n",
"\t0.538\tThe Élysée Palace (, ) is the official residence of the French president. It is in Paris, in the 8th \"arrondissement\", near the Champs-Élysées. The building is under protection of a unit of the famous Republican Guard. It was built between 1718 and 1722.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Cross-Encoder Re-ranker hits\n",
"\t10.574\tThe Eiffel Tower (French: La Tour Eiffel, ], IPA pronunciation: \"EYE-full\" English; \"eh-FEHL\" French) is a landmark in Paris. It was built between 1887 and 1889 for the Exposition Universelle (World Fair). The Tower was the Exposition's main attraction.\n",
"\t4.362\tAlexandre Gustave Eiffel (December 15, 1832 – December 27, 1923; , ) was a French structural engineer and architect. He is known for designing the Eiffel Tower. He also designed the armature (supporting framework) for the Statue of Liberty, New York Harbor, United States.\n",
"\t3.756\tThe current tower is the second tower on the site. The original tower was built in 1912. The design was based on the Eiffel Tower. The tower had an cable car that connected it to a nearby amusement park called Luna Park. The original was 64 meters tall. It was the second tallest building in Asia at that time. It quickly became one of the most popular locations in the city. Visitors came from all over.\n"
"Input question: Which US president was killed?\n",
"Top-3 lexical search (BM25) hits\n",
"\t10.179\tLyndon Baines Johnson (August 27, 1908 – January 22, 1973) was a member of the Democratic Party and the 36th president of the United States serving from 1963 to 1969. Johnson took over as president when President Kennedy was killed in November 1963. He was then re-elected in the 1964 election.\n",
"\t10.091\tLech Kaczyński, the fourth President of the Republic of Poland, died on 10 April 2010. He died in a plane crash outside of Smolensk, Russia. The plane was a Tu-154 belonging to the Polish Air Force. The crash killed all 96 on board. His wife, Maria Kaczyńska, was also among those killed.\n",
"\t9.791\tJacobo Majluta Azar (October 9, 1934 – March 2, 1996) was a Dominican politician. He was Vice President of the Dominican Republic during the Antonio Guzmán Fernández presidency between 1978 to 1982. He became President of the Dominican Republic after Guzmán Fernández killed himself in 1982. He was president for a month between July to August 1982.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Bi-Encoder Retrieval hits\n",
"\t0.686\tJohn F. Kennedy was the 35th President of the United States. He was assassinated (murdered) in Dealey Plaza, Dallas, Texas, on Friday, November 22, 1963. This happened while he was traveling in a Presidential motorcade with his wife Jacqueline, the Governor of Texas John Connally, and the governor's wife Nellie.\n",
"\t0.656\tWilliam McKinley, the 25th President of the United States, was assassinated on September 6, 1901, inside the Temple of Music on the grounds of the Pan-American Exposition in Buffalo, New York.\n",
"\t0.655\tJames Abram Garfield (November 19, 1831 - September 19, 1881) was the 20th (1881) President of the United States and the 2nd President to be assassinated (killed while in office). President Garfield was in office from March to September of 1881. He was in office for a total of six months and fifteen days. For almost half that time he was bedridden as a result of an attempt to kill him. He was shot on July 2 and finally died in September the same year he got into office.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Cross-Encoder Re-ranker hits\n",
"\t9.278\tWilliam McKinley, the 25th President of the United States, was assassinated on September 6, 1901, inside the Temple of Music on the grounds of the Pan-American Exposition in Buffalo, New York.\n",
"\t8.403\tJohn F. Kennedy was the 35th President of the United States. He was assassinated (murdered) in Dealey Plaza, Dallas, Texas, on Friday, November 22, 1963. This happened while he was traveling in a Presidential motorcade with his wife Jacqueline, the Governor of Texas John Connally, and the governor's wife Nellie.\n",
"\t7.914\tOn December 26, 2006, Gerald Ford, the 38th President of the United States, died at his home in Rancho Mirage, California at 6:45 p.m. local time (02:45, December 27, UTC).\n"
]
}
],
"source": [
"search(query = \"Which US president was killed?\")"
"\t18.743\tChinese New Year, known in China as the SpringFestival and in Singapore as the LunarNewYear, is a holiday on and around the new moon on the first day of the year in the traditional Chinese calendar. This calendar is based on the changes in the moon and is only sometimes changed to fit the seasons of the year based on how the Earth moves around the sun. Because of this, Chinese New Year is never on January1. It moves around between January21 and February20.\n",
"\t18.527\tNew Year in Japan is one of the most important festivals. Unlike the Chinese New Year, it is held on January 1.\n",
"\t15.789\tThe CCTV New Year's Gala (Simplified Chinese: 中国中央电视台春节联欢晚会; Traditional Chinese: 中國中央電視台春節聯歡晚會; Pinyin: \"Zhōngguó zhōngyāng diànshìtái chūnjié liánhuān wǎnhuì\") is a Chinese New Year special produced by China Central Television. It was presented by Zhao Zhongxiang.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Bi-Encoder Retrieval hits\n",
"\t0.782\tChinese New Year, known in China as the SpringFestival and in Singapore as the LunarNewYear, is a holiday on and around the new moon on the first day of the year in the traditional Chinese calendar. This calendar is based on the changes in the moon and is only sometimes changed to fit the seasons of the year based on how the Earth moves around the sun. Because of this, Chinese New Year is never on January1. It moves around between January21 and February20.\n",
"\t0.648\tChinese National Day is the national day of China. It may mean:\n",
"\t0.642\tNational Day is a yearly holiday in the People's Republic of China. It celebrates the beginning of its new government on October1, 1949. It is one of two Golden Weeks in the country, along with the Chinese New Year.\n",
"\n",
"-------------------------\n",
"\n",
"Top-3 Cross-Encoder Re-ranker hits\n",
"\t10.896\tChinese New Year, known in China as the SpringFestival and in Singapore as the LunarNewYear, is a holiday on and around the new moon on the first day of the year in the traditional Chinese calendar. This calendar is based on the changes in the moon and is only sometimes changed to fit the seasons of the year based on how the Earth moves around the sun. Because of this, Chinese New Year is never on January1. It moves around between January21 and February20.\n",
"\t6.147\tNew Year in Japan is one of the most important festivals. Unlike the Chinese New Year, it is held on January 1.\n",
"\t5.081\tNational Day is a yearly holiday in the People's Republic of China. It celebrates the beginning of its new government on October1, 1949. It is one of two Golden Weeks in the country, along with the Chinese New Year.\n"
Semantic search seeks to improve search accuracy by understanding the content of the search query. In contrast to traditional search engines which only find documents based on lexical matches, semantic search can also find synonyms.
## Background
The idea behind semantic search is to embed all entries in your corpus, whether they be sentences, paragraphs, or documents, into a vector space.
At search time, the query is embedded into the same vector space and the closest embeddings from your corpus are found. These entries should have a high semantic overlap with the query.
A **critical distinction** for your setup is *symmetric* vs. *asymmetric semantic search*:
- For **symmetric semantic search** your query and the entries in your corpus are of about the same length and have the same amount of content. An example would be searching for similar questions: Your query could for example be *"How to learn Python online?"* and you want to find an entry like *"How to learn Python on the web?"*. For symmetric tasks, you could potentially flip the query and the entries in your corpus.
- For **asymmetric semantic search**, you usually have a **short query** (like a question or some keywords) and you want to find a longer paragraph answering the query. An example would be a query like *"What is Python"* and you want to find the paragraph *"Python is an interpreted, high-level and general-purpose programming language. Python's design philosophy ..."*. For asymmetric tasks, flipping the query and the entries in your corpus usually does not make sense.
It is critical **that you choose the right model** for your type of task.
Suitable models for **symmetric semantic search**: [Pre-Trained Sentence Embedding Models](https://www.sbert.net/docs/pretrained_models.html#sentence-embedding-models)
Suitable models for **asymmetric semantic search**: [Pre-Trained MS MARCO Models](https://www.sbert.net/docs/pretrained-models/msmarco-v3.html)
## Python
For small corpora (up to about 1 million entries) we can compute the cosine-similarity between the query and all entries in the corpus.
In the following example, we define a small corpus with few example sentences and compute the embeddings for the corpus as well as for our query.
We then use the [util.cos_sim()](../../../docs/usage/semantic_textual_similarity.md) function to compute the cosine similarity between the query and all corpus entries.
For large corpora, sorting all scores would take too much time. Hence, we use [torch.topk](https://pytorch.org/docs/stable/generated/torch.topk.html) to only get the top k entries.
For a simple example, see [semantic_search.py](semantic_search.py):
```eval_rst
.. literalinclude:: semantic_search.py
```
## util.semantic_search
Instead of implementing semantic search by yourself, you can use the *util.semantic_search* function.
By default, up to 100 queries are processed in parallel. Further, the corpus is chunked into set of up to 500k entries. You can increase *query_chunk_size* and *corpus_chunk_size*, which leads to increased speed for large corpora, but also increases the memory requirement.
## Speed Optimization
To get the optimal speed for the `util.semantic_search` method, it is advisable to have the `query_embeddings` as well as the `corpus_embeddings` on the same GPU-device. This significantly boost the performance.
Further, we can normalize the corpus embeddings so that each corpus embeddings is of length 1. In that case, we can use dot-product for computing scores.
[Elasticsearch](https://www.elastic.co/elasticsearch/) has the possibility to [index dense vectors](https://www.elastic.co/what-is/vector-search) and to use them for document scoring. We can easily index embedding vectors, store other data alongside our vectors and, most importantly, efficiently retrieve relevant entries using [approximate nearest neighbor search](https://www.elastic.co/blog/introducing-approximate-nearest-neighbor-search-in-elasticsearch-8-0)(HNSW, see also below) on the embeddings.
For further details, see [semantic_search_quora_elasticsearch.py](semantic_search_quora_elasticsearch.py).
## Approximate Nearest Neighbor
Searching a large corpus with millions of embeddings can be time-consuming if exact nearest neighbor search is used (like it is used by *util.semantic_search*).
In that case, Approximate Nearest Neighbor (ANN) can be helpful. Here, the data is partitioned into smaller fractions of similar embeddings. This index can be searched efficiently and the embeddings with the highest similarity (the nearest neighbors) can be retrieved within milliseconds, even if you have millions of vectors.
However, the results are not necessarily exact. It is possible that some vectors with high similarity will be missed. That's the reason why it is called approximate nearest neighbor.
For all ANN methods, there are usually one or more parameters to tune that determine the recall-speed trade-off. If you want the highest speed, you have a high chance of missing hits. If you want high recall, the search speed decreases.
Three popular libraries for approximate nearest neighbor are [Annoy](https://github.com/spotify/annoy), [FAISS](https://github.com/facebookresearch/faiss), and [hnswlib](https://github.com/nmslib/hnswlib/). Personally I find hnswlib the most suitable library: It is easy to use, offers a great performance and has nice features included that are important for real applications.
For further details, see [Retrieve & Re-rank](../retrieve_rerank/README.md).
## Examples
In the following we list examples for different use-cases.
### Similar Questions Retrieval
[semantic_search_quora_pytorch.py](semantic_search_quora_pytorch.py)[ [Colab version](https://colab.research.google.com/drive/12cn5Oo0v3HfQQ8Tv6-ukgxXSmT3zl35A?usp=sharing) ] shows an example based on the [Quora duplicate questions](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs) dataset. The user can enter a question, and the code retrieves the most similar questions from the dataset using the *util.semantic_search* method. As model, we use *distilbert-multilingual-nli-stsb-quora-ranking*, which was trained to identify similar questions and supports 50+ languages. Hence, the user can input the question in any of the 50+ languages. This is a **symmetric search task**, as the search queries have the same length and content as the questions in the corpus.
### Similar Publication Retrieval
[semantic_search_publications.py](semantic_search_publications.py)[ [Colab version](https://colab.research.google.com/drive/12hfBveGHRsxhPIUMmJYrll2lFU4fOX06?usp=sharing) ] shows an example how to find similar scientific publications. As corpus, we use all publications that have been presented at the EMNLP 2016 - 2018 conferences. As search query, we input the title and abstract of more recent publications and find related publications from our copurs. We use the [SPECTER](https://arxiv.org/abs/2004.07180) model. This is a **symmetric search task**, as the paper in the corpus consists of title & abstract and we search for title & abstract.
### Question & Answer Retrieval
[semantic_search_wikipedia_qa.py](semantic_search_wikipedia_qa.py) [ [Colab Version](https://colab.research.google.com/drive/11GunvCqJuebfeTlgbJWkIMT0xJH6PWF1?usp=sharing) ]:This example uses a model that was trained on the [Natural Questions dataset](https://ai.google.com/research/NaturalQuestions/). It consists of about 100k real Google search queries, together with an annotated passage from Wikipedia that provides the answer. It is an example of an **asymmetric search task**. As corpus, we use the smaller [Simple English Wikipedia](https://simple.wikipedia.org/wiki/Main_Page) so that it fits easily into memory.
[retrieve_rerank_simple_wikipedia.ipynb](../retrieve_rerank/retrieve_rerank_simple_wikipedia.ipynb) [ [Colab Version](https://colab.research.google.com/github/UKPLab/sentence-transformers/blob/master/examples/applications/retrieve_rerank/retrieve_rerank_simple_wikipedia.ipynb) ]:This script uses the [Retrieve & Re-rank](../retrieve_rerank/README.md) strategy and is an example for an **asymmetric search task**. We split all Wikipedia articles into paragraphs and encode them with a bi-encoder. If a new query / question is entered, it is encoded by the same bi-encoder and the paragraphs with the highest cosine-similarity are retrieved (see [semantic search](../semantic-search/README.md)). Next, the retrieved candidates are scored by a Cross-Encoder re-ranker and the 5 passages with the highest score from the Cross-Encoder are presented to the user. We use models that were trained on the [MS Marco Passage Reranking](https://github.com/microsoft/MSMARCO-Passage-Ranking/) dataset, a dataset with about 500k real queries from Bing search.
title="Specializing Word Embeddings (for Parsing) by Information Bottleneck",
abstract="Pre-trained word embeddings like ELMo and BERT contain rich syntactic and semantic information, resulting in state-of-the-art performance on various tasks. We propose a very fast variational information bottleneck (VIB) method to nonlinearly compress these embeddings, keeping only the information that helps a discriminative parser. We compress each word embedding to either a discrete tag or a continuous vector. In the discrete version, our automatically compressed tags form an alternative tag set: we show experimentally that our tags capture most of the information in traditional POS tag annotations, but our tag sequences can be parsed more accurately at the same level of tag granularity. In the continuous version, we show experimentally that moderately compressing the word embeddings by our method yields a more accurate parser in 8 of 9 languages, unlike simple dimensionality reduction.",
)
# This paper was the EMNLP 2020 Best Paper
search_papers(
title="Digital Voicing of Silent Speech",
abstract="In this paper, we consider the task of digitally voicing silent speech, where silently mouthed words are converted to audible speech based on electromyography (EMG) sensor measurements that capture muscle impulses. While prior work has focused on training speech synthesis models from EMG collected during vocalized speech, we are the first to train from EMG collected during silently articulated speech. We introduce a method of training on silent EMG by transferring audio targets from vocalized to silent signals. Our method greatly improves intelligibility of audio generated from silent EMG compared to a baseline that only trains with vocalized data, decreasing transcription word error rate from 64% to 4% in one data condition and 88% to 68% in another. To spur further development on this task, we share our new dataset of silent and vocalized facial EMG measurements.",
)
# This paper was a EMNLP 2020 Honourable Mention Papers
search_papers(
title="If beam search is the answer, what was the question?",
abstract="Quite surprisingly, exact maximum a posteriori (MAP) decoding of neural language generators frequently leads to low-quality results. Rather, most state-of-the-art results on language generation tasks are attained using beam search despite its overwhelmingly high search error rate. This implies that the MAP objective alone does not express the properties we desire in text, which merits the question: if beam search is the answer, what was the question? We frame beam search as the exact solution to a different decoding objective in order to gain insights into why high probability under a model alone may not indicate adequacy. We find that beam search enforces uniform information density in text, a property motivated by cognitive science. We suggest a set of decoding objectives that explicitly enforce this property and find that exact decoding with these objectives alleviates the problems encountered when decoding poorly calibrated language generation models. Additionally, we analyze the text produced using various decoding strategies and see that, in our neural machine translation experiments, the extent to which this property is adhered to strongly correlates with BLEU.",
)
# This paper was a EMNLP 2020 Honourable Mention Papers
search_papers(
title="Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems",
abstract="The lack of time efficient and reliable evalu-ation methods is hampering the development of conversational dialogue systems (chat bots). Evaluations that require humans to converse with chat bots are time and cost intensive, put high cognitive demands on the human judges, and tend to yield low quality results. In this work, we introduce Spot The Bot, a cost-efficient and robust evaluation framework that replaces human-bot conversations with conversations between bots. Human judges then only annotate for each entity in a conversation whether they think it is human or not (assuming there are humans participants in these conversations). These annotations then allow us to rank chat bots regarding their ability to mimic conversational behaviour of humans. Since we expect that all bots are eventually recognized as such, we incorporate a metric that measures which chat bot is able to uphold human-like be-havior the longest, i.e.Survival Analysis. This metric has the ability to correlate a bot’s performance to certain of its characteristics (e.g.fluency or sensibleness), yielding interpretable results. The comparably low cost of our frame-work allows for frequent evaluations of chatbots during their evaluation cycle. We empirically validate our claims by applying Spot The Bot to three domains, evaluating several state-of-the-art chat bots, and drawing comparisonsto related work. The framework is released asa ready-to-use tool.",
)
# EMNLP 2020 paper on making Sentence-BERT multilingual
search_papers(
title="Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation",
abstract="We present an easy and efficient method to extend existing sentence embedding models to new languages. This allows to create multilingual versions from previously monolingual models. The training is based on the idea that a translated sentence should be mapped to the same location in the vector space as the original sentence. We use the original (monolingual) model to generate sentence embeddings for the source language and then train a new system on translated sentences to mimic the original model. Compared to other methods for training multilingual sentence embeddings, this approach has several advantages: It is easy to extend existing models with relatively few samples to new languages, it is easier to ensure desired properties for the vector space, and the hardware requirements for training is lower. We demonstrate the effectiveness of our approach for 50+ languages from various language families. Code to extend sentence embeddings models to more than 400 languages is publicly available.",
This example uses Approximate Nearest Neighbor Search (ANN) with Annoy (https://github.com/spotify/annoy).
Searching a large corpus with Millions of embeddings can be time-consuming. To speed this up,
ANN can index the existent vectors. For a new query vector, this index can be used to find the nearest neighbors.
This nearest neighbor search is not perfect, i.e., it might not perfectly find all top-k nearest neighbors.
In this example, we use Annoy. It learns to a tree that partitions embeddings into smaller sections. For our query embeddings,
we can efficiently check which section matches and only search that section for nearest neighbor.
Selecting the n_trees parameter is quite important. With more trees, we get a better recall, but a worse run-time.
This script will compare the result from ANN with exact nearest neighbor search and output a Recall@k value
as well as the missing results in the top-k hits list.
See the Annoy repository, how to install Annoy.
For details how Annoy works, see: https://erikbern.com/2015/10/01/nearest-neighbors-and-vector-models-part-2-how-to-search-in-high-dimensional-spaces.html
As dataset, we use the Quora Duplicate Questions dataset, which contains about 500k questions:
# Number of clusters used for faiss. Select a value 4*sqrt(N) to 16*sqrt(N) - https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index
n_clusters=1024
# We use Inner Product (dot-product) as Index. We will normalize our vectors to unit length, then is Inner Product equal to cosine similarity
