@@ -54,8 +54,7 @@ Additionally, the following method can be used to load values from a data file
...
@@ -54,8 +54,7 @@ Additionally, the following method can be used to load values from a data file
Example usage
Example usage
^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^
An example using these processors is given in the
An example using these processors is given in the `run_glue.py <https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py>`__ script.
`The Stanford Question Answering Dataset (SQuAD) <https://rajpurkar.github.io/SQuAD-explorer//>`__ is a benchmark that evaluates
the performance of models on question answering. Two versions are available, v1.1 and v2.0. The first version (v1.1) was released together with the paper
`SQuAD: 100,000+ Questions for Machine Comprehension of Text <https://arxiv.org/abs/1606.05250>`__. The second version (v2.0) was released alongside
the paper `Know What You Don't Know: Unanswerable Questions for SQuAD <https://arxiv.org/abs/1806.03822>`__.
This library hosts a processor for each of the two versions:
"question":"In what country is Normandy located?",
"id":"56ddde6b9a695914005b9628",
"answers":[{
"text":"France",
"answer_start":159
}],
"is_impossible":false
},{
"question":"When were the Normans in Normandy?",
"id":"56ddde6b9a695914005b9629",
"answers":[{
"text":"10th and 11th centuries",
"answer_start":94
}],
"is_impossible":false
},{
"question":"From which countries did the Norse originate?",
"id":"56ddde6b9a695914005b962a",
"answers":[{
"text":"Denmark, Iceland and Norway",
"answer_start":256
}],
"is_impossible":false
},{
"plausible_answers":[{
"text":"Rollo",
"answer_start":308
}],
"question":"Who did King Charles III swear fealty to?",
"id":"5ad39d53604f3c001a3fe8d3",
"answers":[],
"is_impossible":true
},{
"plausible_answers":[{
"text":"10th century",
"answer_start":671
}],
"question":"When did the Frankish identity emerge?",
"id":"5ad39d53604f3c001a3fe8d4",
"answers":[],
"is_impossible":true
}],
"context":"The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants would gradually merge with the Carolingian-based cultures of West Francia. The distinct cultural and ethnic identity of the Normans emerged initially in the first half of the 10th century, and it continued to evolve over the succeeding centuries."
},{
"qas":[{
"question":"Who was the duke in the battle of Hastings?",
"id":"56dddf4066d3e219004dad5f",
"answers":[{
"text":"William the Conqueror",
"answer_start":1022
}],
"is_impossible":false
},{
"plausible_answers":[{
"text":"Antioch",
"answer_start":1295
}],
"question":"What principality did William the conquerer found?",
"id":"5ad3a266604f3c001a3fea2b",
"answers":[],
"is_impossible":true
}],
"context":"The Norman dynasty had a major political, cultural and military impact on medieval Europe and even the Near East. The Normans were famed for their martial spirit and eventually for their Christian piety, becoming exponents of the Catholic orthodoxy into which they assimilated. They adopted the Gallo-Romance language of the Frankish land they settled, their dialect becoming known as Norman, Normaund or Norman French, an important literary language. The Duchy of Normandy, which they formed by treaty with the French crown, was a great fief of medieval France, and under Richard I of Normandy was forged into a cohesive and formidable principality in feudal tenure. The Normans are noted both for their culture, such as their unique Romanesque architecture and musical traditions, and for their significant military accomplishments and innovations. Norman adventurers founded the Kingdom of Sicily under Roger II after conquering southern Italy on the Saracens and Byzantines, and an expedition on behalf of their duke, William the Conqueror, led to the Norman conquest of England at the Battle of Hastings in 1066. Norman cultural and military influence spread from these new European centres to the Crusader states of the Near East, where their prince Bohemond I founded the Principality of Antioch in the Levant, to Scotland and Wales in Great Britain, to Ireland, and to the coasts of north Africa and the Canary Islands."
}]
},{
"title":"Computational_complexity_theory",
"paragraphs":[{
"qas":[{
"question":"What branch of theoretical computer science deals with broadly classifying computational problems by difficulty and class of relationship?",
"id":"56e16182e3433e1400422e28",
"answers":[{
"text":"Computational complexity theory",
"answer_start":0
}],
"is_impossible":false
},{
"plausible_answers":[{
"text":"algorithm",
"answer_start":472
}],
"question":"What is a manual application of mathematical steps?",
"id":"5ad5316b5b96ef001a10ab76",
"answers":[],
"is_impossible":true
}],
"context":"Computational complexity theory is a branch of the theory of computation in theoretical computer science that focuses on classifying computational problems according to their inherent difficulty, and relating those classes to each other. A computational problem is understood to be a task that is in principle amenable to being solved by a computer, which is equivalent to stating that the problem may be solved by mechanical application of mathematical steps, such as an algorithm."
},{
"qas":[{
"question":"What measure of a computational problem broadly defines the inherent difficulty of the solution?",
"id":"56e16839cd28a01900c67887",
"answers":[{
"text":"if its solution requires significant resources",
"answer_start":46
}],
"is_impossible":false
},{
"question":"What method is used to intuitively assess or quantify the amount of resources required to solve a computational problem?",
"id":"56e16839cd28a01900c67888",
"answers":[{
"text":"mathematical models of computation",
"answer_start":176
}],
"is_impossible":false
},{
"question":"What are two basic primary resources used to guage complexity?",
"id":"56e16839cd28a01900c67889",
"answers":[{
"text":"time and storage",
"answer_start":305
}],
"is_impossible":false
},{
"plausible_answers":[{
"text":"the number of gates in a circuit",
"answer_start":436
}],
"question":"What unit is measured to determine circuit simplicity?",
"id":"5ad532575b96ef001a10ab7f",
"answers":[],
"is_impossible":true
},{
"plausible_answers":[{
"text":"the number of processors",
"answer_start":502
}],
"question":"What number is used in perpendicular computing?",
"id":"5ad532575b96ef001a10ab80",
"answers":[],
"is_impossible":true
}],
"context":"A problem is regarded as inherently difficult if its solution requires significant resources, whatever the algorithm used. The theory formalizes this intuition, by introducing mathematical models of computation to study these problems and quantifying the amount of resources needed to solve them, such as time and storage. Other complexity measures are also used, such as the amount of communication (used in communication complexity), the number of gates in a circuit (used in circuit complexity) and the number of processors (used in parallel computing). One of the roles of computational complexity theory is to determine the practical limits on what computers can and cannot do."
@@ -78,6 +78,8 @@ class PreTrainedTokenizer(object):
...
@@ -78,6 +78,8 @@ class PreTrainedTokenizer(object):
"pad_token","cls_token","mask_token",
"pad_token","cls_token","mask_token",
"additional_special_tokens"]
"additional_special_tokens"]
padding_side="right"
@property
@property
defbos_token(self):
defbos_token(self):
""" Beginning of sentence token (string). Log an error if used while not having been set. """
""" Beginning of sentence token (string). Log an error if used while not having been set. """
...
@@ -191,6 +193,11 @@ class PreTrainedTokenizer(object):
...
@@ -191,6 +193,11 @@ class PreTrainedTokenizer(object):
""" Id of the padding token in the vocabulary. Log an error if used while not having been set. """
""" Id of the padding token in the vocabulary. Log an error if used while not having been set. """
returnself.convert_tokens_to_ids(self.pad_token)
returnself.convert_tokens_to_ids(self.pad_token)
@property
defpad_token_type_id(self):
""" Id of the padding token type in the vocabulary."""
returnself._pad_token_type_id
@property
@property
defcls_token_id(self):
defcls_token_id(self):
""" Id of the classification token in the vocabulary. E.g. to extract a summary of an input sequence leveraging self-attention along the full depth of the model. Log an error if used while not having been set. """
""" Id of the classification token in the vocabulary. E.g. to extract a summary of an input sequence leveraging self-attention along the full depth of the model. Log an error if used while not having been set. """
...
@@ -214,10 +221,14 @@ class PreTrainedTokenizer(object):
...
@@ -214,10 +221,14 @@ class PreTrainedTokenizer(object):
