Add tests to make sure task definitions don't change without the version also changing

tests/testdata/hendrycksTest-machine_learning-v0-loglikelihood.json

+[["Question: Statement 1| Layer Normalization is used in the original ResNet paper, not Batch Normalization. Statement 2| DCGANs use self-attention to stabilize training.\nChoices:\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer:", " True, True"], ["Question: Statement 1| Layer Normalization is used in the original ResNet paper, not Batch Normalization. Statement 2| DCGANs use self-attention to stabilize training.\nChoices:\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer:", " False, False"], ["Question: Statement 1| Layer Normalization is used in the original ResNet paper, not Batch Normalization. Statement 2| DCGANs use self-attention to stabilize training.\nChoices:\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer:", " True, False"], ["Question: Statement 1| Layer Normalization is used in the original ResNet paper, not Batch Normalization. Statement 2| DCGANs use self-attention to stabilize training.\nChoices:\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer:", " False, True"], ["Question: Another term for out-of-distribution detection is?\nChoices:\nA. anomaly detection\nB. one-class detection\nC. train-test mismatch robustness\nD. background detection\nAnswer:", " anomaly detection"], ["Question: Another term for out-of-distribution detection is?\nChoices:\nA. anomaly detection\nB. one-class detection\nC. train-test mismatch robustness\nD. background detection\nAnswer:", " one-class detection"], ["Question: Another term for out-of-distribution detection is?\nChoices:\nA. anomaly detection\nB. one-class detection\nC. train-test mismatch robustness\nD. background detection\nAnswer:", " train-test mismatch robustness"], ["Question: Another term for out-of-distribution detection is?\nChoices:\nA. anomaly detection\nB. one-class detection\nC. train-test mismatch robustness\nD. background detection\nAnswer:", " background detection"], ["Question: Neural networks:\nChoices:\nA. Optimize a convex objective function\nB. Can only be trained with stochastic gradient descent\nC. Can use a mix of different activation functions\nD. None of the above\nAnswer:", " Optimize a convex objective function"], ["Question: Neural networks:\nChoices:\nA. Optimize a convex objective function\nB. Can only be trained with stochastic gradient descent\nC. Can use a mix of different activation functions\nD. None of the above\nAnswer:", " Can only be trained with stochastic gradient descent"], ["Question: Neural networks:\nChoices:\nA. Optimize a convex objective function\nB. Can only be trained with stochastic gradient descent\nC. Can use a mix of different activation functions\nD. None of the above\nAnswer:", " Can use a mix of different activation functions"], ["Question: Neural networks:\nChoices:\nA. Optimize a convex objective function\nB. Can only be trained with stochastic gradient descent\nC. Can use a mix of different activation functions\nD. None of the above\nAnswer:", " None of the above"], ["Question: In building a linear regression model for a particular data set, you observe the coefficient of one of the features having a relatively high negative value. This suggests that\nChoices:\nA. This feature has a strong effect on the model (should be retained)\nB. This feature does not have a strong effect on the model (should be ignored)\nC. It is not possible to comment on the importance of this feature without additional information\nD. Nothing can be determined.\nAnswer:", " This feature has a strong effect on the model (should be retained)"], ["Question: In building a linear regression model for a particular data set, you observe the coefficient of one of the features having a relatively high negative value. This suggests that\nChoices:\nA. This feature has a strong effect on the model (should be retained)\nB. This feature does not have a strong effect on the model (should be ignored)\nC. It is not possible to comment on the importance of this feature without additional information\nD. Nothing can be determined.\nAnswer:", " This feature does not have a strong effect on the model (should be ignored)"], ["Question: In building a linear regression model for a particular data set, you observe the coefficient of one of the features having a relatively high negative value. This suggests that\nChoices:\nA. This feature has a strong effect on the model (should be retained)\nB. This feature does not have a strong effect on the model (should be ignored)\nC. It is not possible to comment on the importance of this feature without additional information\nD. Nothing can be determined.\nAnswer:", " It is not possible to comment on the importance of this feature without additional information"], ["Question: In building a linear regression model for a particular data set, you observe the coefficient of one of the features having a relatively high negative value. This suggests that\nChoices:\nA. This feature has a strong effect on the model (should be retained)\nB. This feature does not have a strong effect on the model (should be ignored)\nC. It is not possible to comment on the importance of this feature without additional information\nD. Nothing can be determined.\nAnswer:", " Nothing can be determined."], ["Question: Statement 1| RoBERTa pretrains on a corpus that is approximate 10x larger than the corpus BERT pretrained on. Statement 2| ResNeXts in 2018 usually used tanh activation functions.\nChoices:\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer:", " True, True"], ["Question: Statement 1| RoBERTa pretrains on a corpus that is approximate 10x larger than the corpus BERT pretrained on. Statement 2| ResNeXts in 2018 usually used tanh activation functions.\nChoices:\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer:", " False, False"], ["Question: Statement 1| RoBERTa pretrains on a corpus that is approximate 10x larger than the corpus BERT pretrained on. Statement 2| ResNeXts in 2018 usually used tanh activation functions.\nChoices:\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer:", " True, False"], ["Question: Statement 1| RoBERTa pretrains on a corpus that is approximate 10x larger than the corpus BERT pretrained on. Statement 2| ResNeXts in 2018 usually used tanh activation functions.\nChoices:\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer:", " False, True"], ["Question: As the number of training examples goes to infinity, your model trained on that data will have:\nChoices:\nA. Lower variance\nB. Higher variance\nC. Same variance\nD. None of the above\nAnswer:", " Lower variance"], ["Question: As the number of training examples goes to infinity, your model trained on that data will have:\nChoices:\nA. Lower variance\nB. Higher variance\nC. Same variance\nD. None of the above\nAnswer:", " Higher variance"], ["Question: As the number of training examples goes to infinity, your model trained on that data will have:\nChoices:\nA. Lower variance\nB. Higher variance\nC. Same variance\nD. None of the above\nAnswer:", " Same variance"], ["Question: As the number of training examples goes to infinity, your model trained on that data will have:\nChoices:\nA. Lower variance\nB. Higher variance\nC. Same variance\nD. None of the above\nAnswer:", " None of the above"], ["Question: MLE estimates are often undesirable because\nChoices:\nA. they are biased\nB. they have high variance\nC. they are not consistent estimators\nD. None of the above\nAnswer:", " they are biased"], ["Question: MLE estimates are often undesirable because\nChoices:\nA. they are biased\nB. they have high variance\nC. they are not consistent estimators\nD. None of the above\nAnswer:", " they have high variance"], ["Question: MLE estimates are often undesirable because\nChoices:\nA. they are biased\nB. they have high variance\nC. they are not consistent estimators\nD. None of the above\nAnswer:", " they are not consistent estimators"], ["Question: MLE estimates are often undesirable because\nChoices:\nA. they are biased\nB. they have high variance\nC. they are not consistent estimators\nD. None of the above\nAnswer:", " None of the above"], ["Question: For Kernel Regression, which one of these structural assumptions is the one that most affects the trade-off between underfitting and overfitting:\nChoices:\nA. Whether kernel function is Gaussian versus triangular versus box-shaped\nB. Whether we use Euclidian versus L1 versus L\u221e metrics\nC. The kernel width\nD. The maximum height of the kernel function\nAnswer:", " Whether kernel function is Gaussian versus triangular versus box-shaped"], ["Question: For Kernel Regression, which one of these structural assumptions is the one that most affects the trade-off between underfitting and overfitting:\nChoices:\nA. Whether kernel function is Gaussian versus triangular versus box-shaped\nB. Whether we use Euclidian versus L1 versus L\u221e metrics\nC. The kernel width\nD. The maximum height of the kernel function\nAnswer:", " Whether we use Euclidian versus L1 versus L\u221e metrics"], ["Question: For Kernel Regression, which one of these structural assumptions is the one that most affects the trade-off between underfitting and overfitting:\nChoices:\nA. Whether kernel function is Gaussian versus triangular versus box-shaped\nB. Whether we use Euclidian versus L1 versus L\u221e metrics\nC. The kernel width\nD. The maximum height of the kernel function\nAnswer:", " The kernel width"], ["Question: For Kernel Regression, which one of these structural assumptions is the one that most affects the trade-off between underfitting and overfitting:\nChoices:\nA. Whether kernel function is Gaussian versus triangular versus box-shaped\nB. Whether we use Euclidian versus L1 versus L\u221e metrics\nC. The kernel width\nD. The maximum height of the kernel function\nAnswer:", " The maximum height of the kernel function"], ["Question: Statement 1| RELUs are not monotonic, but sigmoids are monotonic. Statement 2| Neural networks trained with gradient descent with high probability converge to the global optimum.\nChoices:\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer:", " True, True"], ["Question: Statement 1| RELUs are not monotonic, but sigmoids are monotonic. Statement 2| Neural networks trained with gradient descent with high probability converge to the global optimum.\nChoices:\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer:", " False, False"], ["Question: Statement 1| RELUs are not monotonic, but sigmoids are monotonic. Statement 2| Neural networks trained with gradient descent with high probability converge to the global optimum.\nChoices:\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer:", " True, False"], ["Question: Statement 1| RELUs are not monotonic, but sigmoids are monotonic. Statement 2| Neural networks trained with gradient descent with high probability converge to the global optimum.\nChoices:\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer:", " False, True"], ["Question: Statement 1| The training error of 1-nearest neighbor classifier is 0. Statement 2| As the number of data points grows to infinity, the MAP estimate approaches the MLE estimate for all possible priors. In other words, given enough data, the choice of prior is irrelevant.\nChoices:\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer:", " True, True"], ["Question: Statement 1| The training error of 1-nearest neighbor classifier is 0. Statement 2| As the number of data points grows to infinity, the MAP estimate approaches the MLE estimate for all possible priors. In other words, given enough data, the choice of prior is irrelevant.\nChoices:\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer:", " False, False"], ["Question: Statement 1| The training error of 1-nearest neighbor classifier is 0. Statement 2| As the number of data points grows to infinity, the MAP estimate approaches the MLE estimate for all possible priors. In other words, given enough data, the choice of prior is irrelevant.\nChoices:\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer:", " True, False"], ["Question: Statement 1| The training error of 1-nearest neighbor classifier is 0. Statement 2| As the number of data points grows to infinity, the MAP estimate approaches the MLE estimate for all possible priors. In other words, given enough data, the choice of prior is irrelevant.\nChoices:\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer:", " False, True"]]
\ No newline at end of file

tests/testdata/hendrycksTest-machine_learning-v0-res.json

+{"results": {"hendrycksTest-machine_learning": {"acc": 0.1, "acc_stderr": 0.09999999999999999, "acc_norm": 0.1, "acc_norm_stderr": 0.09999999999999999}}, "versions": {"hendrycksTest-machine_learning": 0}}
\ No newline at end of file

tests/testdata/hendrycksTest-management-v0-loglikelihood.json

+[["Question: What is the definition of a scenario in scenario planning?\nChoices:\nA. An imagined sequence of future events\nB. An unpredictable event\nC. A planned for event\nD. An unplanned for event\nAnswer:", " An imagined sequence of future events"], ["Question: What is the definition of a scenario in scenario planning?\nChoices:\nA. An imagined sequence of future events\nB. An unpredictable event\nC. A planned for event\nD. An unplanned for event\nAnswer:", " An unpredictable event"], ["Question: What is the definition of a scenario in scenario planning?\nChoices:\nA. An imagined sequence of future events\nB. An unpredictable event\nC. A planned for event\nD. An unplanned for event\nAnswer:", " A planned for event"], ["Question: What is the definition of a scenario in scenario planning?\nChoices:\nA. An imagined sequence of future events\nB. An unpredictable event\nC. A planned for event\nD. An unplanned for event\nAnswer:", " An unplanned for event"], ["Question: What theory is built around the principle that 'people make choices regarding how to behave based on values and beliefs'?\nChoices:\nA. Expectancy\nB. Instrumental\nC. Classical\nD. Contingency\nAnswer:", " Expectancy"], ["Question: What theory is built around the principle that 'people make choices regarding how to behave based on values and beliefs'?\nChoices:\nA. Expectancy\nB. Instrumental\nC. Classical\nD. Contingency\nAnswer:", " Instrumental"], ["Question: What theory is built around the principle that 'people make choices regarding how to behave based on values and beliefs'?\nChoices:\nA. Expectancy\nB. Instrumental\nC. Classical\nD. Contingency\nAnswer:", " Classical"], ["Question: What theory is built around the principle that 'people make choices regarding how to behave based on values and beliefs'?\nChoices:\nA. Expectancy\nB. Instrumental\nC. Classical\nD. Contingency\nAnswer:", " Contingency"], ["Question: To what time-frame do strategic plans relate?\nChoices:\nA. Long-term\nB. Medium-term\nC. Short-term\nD. Unspecified time it takes to achieve an aim\nAnswer:", " Long-term"], ["Question: To what time-frame do strategic plans relate?\nChoices:\nA. Long-term\nB. Medium-term\nC. Short-term\nD. Unspecified time it takes to achieve an aim\nAnswer:", " Medium-term"], ["Question: To what time-frame do strategic plans relate?\nChoices:\nA. Long-term\nB. Medium-term\nC. Short-term\nD. Unspecified time it takes to achieve an aim\nAnswer:", " Short-term"], ["Question: To what time-frame do strategic plans relate?\nChoices:\nA. Long-term\nB. Medium-term\nC. Short-term\nD. Unspecified time it takes to achieve an aim\nAnswer:", " Unspecified time it takes to achieve an aim"], ["Question: Who of the following is the industrial philanthropist?\nChoices:\nA. Frederick Taylor\nB. Seebohm Rowntree\nC. Henry Ford\nD. Max Weber\nAnswer:", " Frederick Taylor"], ["Question: Who of the following is the industrial philanthropist?\nChoices:\nA. Frederick Taylor\nB. Seebohm Rowntree\nC. Henry Ford\nD. Max Weber\nAnswer:", " Seebohm Rowntree"], ["Question: Who of the following is the industrial philanthropist?\nChoices:\nA. Frederick Taylor\nB. Seebohm Rowntree\nC. Henry Ford\nD. Max Weber\nAnswer:", " Henry Ford"], ["Question: Who of the following is the industrial philanthropist?\nChoices:\nA. Frederick Taylor\nB. Seebohm Rowntree\nC. Henry Ford\nD. Max Weber\nAnswer:", " Max Weber"], ["Question: What is the term for the action in which managers at an organisation analyse the current situation of their organisation and then develop plans to accomplish its mission and achieve its goals?\nChoices:\nA. Synergy planning\nB. Strategy formulation\nC. Functional planning\nD. SWOT analysis\nAnswer:", " Synergy planning"], ["Question: What is the term for the action in which managers at an organisation analyse the current situation of their organisation and then develop plans to accomplish its mission and achieve its goals?\nChoices:\nA. Synergy planning\nB. Strategy formulation\nC. Functional planning\nD. SWOT analysis\nAnswer:", " Strategy formulation"], ["Question: What is the term for the action in which managers at an organisation analyse the current situation of their organisation and then develop plans to accomplish its mission and achieve its goals?\nChoices:\nA. Synergy planning\nB. Strategy formulation\nC. Functional planning\nD. SWOT analysis\nAnswer:", " Functional planning"], ["Question: What is the term for the action in which managers at an organisation analyse the current situation of their organisation and then develop plans to accomplish its mission and achieve its goals?\nChoices:\nA. Synergy planning\nB. Strategy formulation\nC. Functional planning\nD. SWOT analysis\nAnswer:", " SWOT analysis"], ["Question: Who of the following is a leading writer on contingency theory of leadership?\nChoices:\nA. Rosabeth Kanter\nB. Joan Woodward\nC. Rensis Likert\nD. Fred Fiedler\nAnswer:", " Rosabeth Kanter"], ["Question: Who of the following is a leading writer on contingency theory of leadership?\nChoices:\nA. Rosabeth Kanter\nB. Joan Woodward\nC. Rensis Likert\nD. Fred Fiedler\nAnswer:", " Joan Woodward"], ["Question: Who of the following is a leading writer on contingency theory of leadership?\nChoices:\nA. Rosabeth Kanter\nB. Joan Woodward\nC. Rensis Likert\nD. Fred Fiedler\nAnswer:", " Rensis Likert"], ["Question: Who of the following is a leading writer on contingency theory of leadership?\nChoices:\nA. Rosabeth Kanter\nB. Joan Woodward\nC. Rensis Likert\nD. Fred Fiedler\nAnswer:", " Fred Fiedler"], ["Question: What is the term for the 'rule of thumb' type of bias in decision making?\nChoices:\nA. Framing bias\nB. Hindsight bias\nC. Over-confidence bias\nD. Heuristics\nAnswer:", " Framing bias"], ["Question: What is the term for the 'rule of thumb' type of bias in decision making?\nChoices:\nA. Framing bias\nB. Hindsight bias\nC. Over-confidence bias\nD. Heuristics\nAnswer:", " Hindsight bias"], ["Question: What is the term for the 'rule of thumb' type of bias in decision making?\nChoices:\nA. Framing bias\nB. Hindsight bias\nC. Over-confidence bias\nD. Heuristics\nAnswer:", " Over-confidence bias"], ["Question: What is the term for the 'rule of thumb' type of bias in decision making?\nChoices:\nA. Framing bias\nB. Hindsight bias\nC. Over-confidence bias\nD. Heuristics\nAnswer:", " Heuristics"], ["Question: Workers' acceptance of change is characteristic of what type of culture?\nChoices:\nA. Team culture\nB. Collaborative culture\nC. Group culture\nD. Collective culture\nAnswer:", " Team culture"], ["Question: Workers' acceptance of change is characteristic of what type of culture?\nChoices:\nA. Team culture\nB. Collaborative culture\nC. Group culture\nD. Collective culture\nAnswer:", " Collaborative culture"], ["Question: Workers' acceptance of change is characteristic of what type of culture?\nChoices:\nA. Team culture\nB. Collaborative culture\nC. Group culture\nD. Collective culture\nAnswer:", " Group culture"], ["Question: Workers' acceptance of change is characteristic of what type of culture?\nChoices:\nA. Team culture\nB. Collaborative culture\nC. Group culture\nD. Collective culture\nAnswer:", " Collective culture"], ["Question: Which one of the following is not one of Drucker's five guiding principles of management?\nChoices:\nA. Making people's strengths effective and their weaknesses irrelevant.\nB. Enhancing the ability of people to contribute.\nC. To operate the organisation's status system.\nD. Integrating people in a common venture by thinking through, setting and exemplifying the organisational objectives, values and goals.\nAnswer:", " Making people's strengths effective and their weaknesses irrelevant."], ["Question: Which one of the following is not one of Drucker's five guiding principles of management?\nChoices:\nA. Making people's strengths effective and their weaknesses irrelevant.\nB. Enhancing the ability of people to contribute.\nC. To operate the organisation's status system.\nD. Integrating people in a common venture by thinking through, setting and exemplifying the organisational objectives, values and goals.\nAnswer:", " Enhancing the ability of people to contribute."], ["Question: Which one of the following is not one of Drucker's five guiding principles of management?\nChoices:\nA. Making people's strengths effective and their weaknesses irrelevant.\nB. Enhancing the ability of people to contribute.\nC. To operate the organisation's status system.\nD. Integrating people in a common venture by thinking through, setting and exemplifying the organisational objectives, values and goals.\nAnswer:", " To operate the organisation's status system."], ["Question: Which one of the following is not one of Drucker's five guiding principles of management?\nChoices:\nA. Making people's strengths effective and their weaknesses irrelevant.\nB. Enhancing the ability of people to contribute.\nC. To operate the organisation's status system.\nD. Integrating people in a common venture by thinking through, setting and exemplifying the organisational objectives, values and goals.\nAnswer:", " Integrating people in a common venture by thinking through, setting and exemplifying the organisational objectives, values and goals."], ["Question: What is a definition of an objective?\nChoices:\nA. A defined specified outcome to be achieved in the long-term\nB. A clear set of goals to be attained given a set number of resources\nC. A clearly defined and measurable outcome to be achieved over a specified timeframe\nD. A set standard of performance agreed by workers and managers\nAnswer:", " A defined specified outcome to be achieved in the long-term"], ["Question: What is a definition of an objective?\nChoices:\nA. A defined specified outcome to be achieved in the long-term\nB. A clear set of goals to be attained given a set number of resources\nC. A clearly defined and measurable outcome to be achieved over a specified timeframe\nD. A set standard of performance agreed by workers and managers\nAnswer:", " A clear set of goals to be attained given a set number of resources"], ["Question: What is a definition of an objective?\nChoices:\nA. A defined specified outcome to be achieved in the long-term\nB. A clear set of goals to be attained given a set number of resources\nC. A clearly defined and measurable outcome to be achieved over a specified timeframe\nD. A set standard of performance agreed by workers and managers\nAnswer:", " A clearly defined and measurable outcome to be achieved over a specified timeframe"], ["Question: What is a definition of an objective?\nChoices:\nA. A defined specified outcome to be achieved in the long-term\nB. A clear set of goals to be attained given a set number of resources\nC. A clearly defined and measurable outcome to be achieved over a specified timeframe\nD. A set standard of performance agreed by workers and managers\nAnswer:", " A set standard of performance agreed by workers and managers"]]
\ No newline at end of file

tests/testdata/hendrycksTest-management-v0-res.json

+{"results": {"hendrycksTest-management": {"acc": 0.3, "acc_stderr": 0.15275252316519464, "acc_norm": 0.2, "acc_norm_stderr": 0.13333333333333333}}, "versions": {"hendrycksTest-management": 0}}
\ No newline at end of file

tests/testdata/hendrycksTest-marketing-v0-loglikelihood.json

+[["Question:  The process of outsourcing a task or group of tasks to a generally large group of people is known as:\nChoices:\nA. Social media marketing.\nB. Internet advertising.\nC. Crowdsourcing.\nD. E-marketing.\nAnswer:", " Social media marketing."], ["Question:  The process of outsourcing a task or group of tasks to a generally large group of people is known as:\nChoices:\nA. Social media marketing.\nB. Internet advertising.\nC. Crowdsourcing.\nD. E-marketing.\nAnswer:", " Internet advertising."], ["Question:  The process of outsourcing a task or group of tasks to a generally large group of people is known as:\nChoices:\nA. Social media marketing.\nB. Internet advertising.\nC. Crowdsourcing.\nD. E-marketing.\nAnswer:", " Crowdsourcing."], ["Question:  The process of outsourcing a task or group of tasks to a generally large group of people is known as:\nChoices:\nA. Social media marketing.\nB. Internet advertising.\nC. Crowdsourcing.\nD. E-marketing.\nAnswer:", " E-marketing."], ["Question: When there are high levels of business failures and unemployment, the business cycle is said to be in which of the following phases?\nChoices:\nA. Expansion\nB. Peak\nC. Recovery\nD. Trough\nAnswer:", " Expansion"], ["Question: When there are high levels of business failures and unemployment, the business cycle is said to be in which of the following phases?\nChoices:\nA. Expansion\nB. Peak\nC. Recovery\nD. Trough\nAnswer:", " Peak"], ["Question: When there are high levels of business failures and unemployment, the business cycle is said to be in which of the following phases?\nChoices:\nA. Expansion\nB. Peak\nC. Recovery\nD. Trough\nAnswer:", " Recovery"], ["Question: When there are high levels of business failures and unemployment, the business cycle is said to be in which of the following phases?\nChoices:\nA. Expansion\nB. Peak\nC. Recovery\nD. Trough\nAnswer:", " Trough"], ["Question:  Providing free samples of perfumes (scent) in magazines is an example of which of the following?\nChoices:\nA. Classical conditioning.\nB. Operant conditioning.\nC. Social learning.\nD. Behavioural learning.\nAnswer:", " Classical conditioning."], ["Question:  Providing free samples of perfumes (scent) in magazines is an example of which of the following?\nChoices:\nA. Classical conditioning.\nB. Operant conditioning.\nC. Social learning.\nD. Behavioural learning.\nAnswer:", " Operant conditioning."], ["Question:  Providing free samples of perfumes (scent) in magazines is an example of which of the following?\nChoices:\nA. Classical conditioning.\nB. Operant conditioning.\nC. Social learning.\nD. Behavioural learning.\nAnswer:", " Social learning."], ["Question:  Providing free samples of perfumes (scent) in magazines is an example of which of the following?\nChoices:\nA. Classical conditioning.\nB. Operant conditioning.\nC. Social learning.\nD. Behavioural learning.\nAnswer:", " Behavioural learning."], ["Question: Robert is a marketer for a global consumer products company. He is working on the promotional campaign designed to reach a target audience in a new international market. Robert is working hard to make sure that the promotional campaign is clearly understood by the nation's consumers and doesn't offend anyone. By which of the factors in the external environment is he being influenced\nChoices:\nA. Socio-cultural environment.\nB. Competitive environment.\nC. Economic environment.\nD. Legal environment.\nAnswer:", " Socio-cultural environment."], ["Question: Robert is a marketer for a global consumer products company. He is working on the promotional campaign designed to reach a target audience in a new international market. Robert is working hard to make sure that the promotional campaign is clearly understood by the nation's consumers and doesn't offend anyone. By which of the factors in the external environment is he being influenced\nChoices:\nA. Socio-cultural environment.\nB. Competitive environment.\nC. Economic environment.\nD. Legal environment.\nAnswer:", " Competitive environment."], ["Question: Robert is a marketer for a global consumer products company. He is working on the promotional campaign designed to reach a target audience in a new international market. Robert is working hard to make sure that the promotional campaign is clearly understood by the nation's consumers and doesn't offend anyone. By which of the factors in the external environment is he being influenced\nChoices:\nA. Socio-cultural environment.\nB. Competitive environment.\nC. Economic environment.\nD. Legal environment.\nAnswer:", " Economic environment."], ["Question: Robert is a marketer for a global consumer products company. He is working on the promotional campaign designed to reach a target audience in a new international market. Robert is working hard to make sure that the promotional campaign is clearly understood by the nation's consumers and doesn't offend anyone. By which of the factors in the external environment is he being influenced\nChoices:\nA. Socio-cultural environment.\nB. Competitive environment.\nC. Economic environment.\nD. Legal environment.\nAnswer:", " Legal environment."], ["Question:  Post-purchase re-evaluation of the consumer proposition acquisition process attempts to measure the degree of:\nChoices:\nA. Selling success experienced by the vendor.\nB. Consumer satisfaction with the purchase.\nC. Follow-up effectiveness of the firm.\nD. Advertising influence on the purchase.\nAnswer:", " Selling success experienced by the vendor."], ["Question:  Post-purchase re-evaluation of the consumer proposition acquisition process attempts to measure the degree of:\nChoices:\nA. Selling success experienced by the vendor.\nB. Consumer satisfaction with the purchase.\nC. Follow-up effectiveness of the firm.\nD. Advertising influence on the purchase.\nAnswer:", " Consumer satisfaction with the purchase."], ["Question:  Post-purchase re-evaluation of the consumer proposition acquisition process attempts to measure the degree of:\nChoices:\nA. Selling success experienced by the vendor.\nB. Consumer satisfaction with the purchase.\nC. Follow-up effectiveness of the firm.\nD. Advertising influence on the purchase.\nAnswer:", " Follow-up effectiveness of the firm."], ["Question:  Post-purchase re-evaluation of the consumer proposition acquisition process attempts to measure the degree of:\nChoices:\nA. Selling success experienced by the vendor.\nB. Consumer satisfaction with the purchase.\nC. Follow-up effectiveness of the firm.\nD. Advertising influence on the purchase.\nAnswer:", " Advertising influence on the purchase."], ["Question:  The action of presenting persuasive communication and the action of audiences in interpreting that communication to assimilate it into their existing understanding is referred to as:\nChoices:\nA. Needs identifying.\nB. Innovating.\nC. Advertising.\nD. Framing.\nAnswer:", " Needs identifying."], ["Question:  The action of presenting persuasive communication and the action of audiences in interpreting that communication to assimilate it into their existing understanding is referred to as:\nChoices:\nA. Needs identifying.\nB. Innovating.\nC. Advertising.\nD. Framing.\nAnswer:", " Innovating."], ["Question:  The action of presenting persuasive communication and the action of audiences in interpreting that communication to assimilate it into their existing understanding is referred to as:\nChoices:\nA. Needs identifying.\nB. Innovating.\nC. Advertising.\nD. Framing.\nAnswer:", " Advertising."], ["Question:  The action of presenting persuasive communication and the action of audiences in interpreting that communication to assimilate it into their existing understanding is referred to as:\nChoices:\nA. Needs identifying.\nB. Innovating.\nC. Advertising.\nD. Framing.\nAnswer:", " Framing."], ["Question:  This is part of the communication process where receivers unpack the various components of the message, and begin to make sense and give the message meaning:\nChoices:\nA. Encoding.\nB. Decoding.\nC. Transfer.\nD. Noise.\nAnswer:", " Encoding."], ["Question:  This is part of the communication process where receivers unpack the various components of the message, and begin to make sense and give the message meaning:\nChoices:\nA. Encoding.\nB. Decoding.\nC. Transfer.\nD. Noise.\nAnswer:", " Decoding."], ["Question:  This is part of the communication process where receivers unpack the various components of the message, and begin to make sense and give the message meaning:\nChoices:\nA. Encoding.\nB. Decoding.\nC. Transfer.\nD. Noise.\nAnswer:", " Transfer."], ["Question:  This is part of the communication process where receivers unpack the various components of the message, and begin to make sense and give the message meaning:\nChoices:\nA. Encoding.\nB. Decoding.\nC. Transfer.\nD. Noise.\nAnswer:", " Noise."], ["Question: _____________ helps in 'problematizing hitherto uncontentious marketing areas to reveal underlying institutional and theoretical dysfunctionalities' (Saren, 2011:95).\nChoices:\nA. Production marketing\nB. Sustainable marketing\nC. Relationship marketing\nD. Critical marketing analysis\nAnswer:", " Production marketing"], ["Question: _____________ helps in 'problematizing hitherto uncontentious marketing areas to reveal underlying institutional and theoretical dysfunctionalities' (Saren, 2011:95).\nChoices:\nA. Production marketing\nB. Sustainable marketing\nC. Relationship marketing\nD. Critical marketing analysis\nAnswer:", " Sustainable marketing"], ["Question: _____________ helps in 'problematizing hitherto uncontentious marketing areas to reveal underlying institutional and theoretical dysfunctionalities' (Saren, 2011:95).\nChoices:\nA. Production marketing\nB. Sustainable marketing\nC. Relationship marketing\nD. Critical marketing analysis\nAnswer:", " Relationship marketing"], ["Question: _____________ helps in 'problematizing hitherto uncontentious marketing areas to reveal underlying institutional and theoretical dysfunctionalities' (Saren, 2011:95).\nChoices:\nA. Production marketing\nB. Sustainable marketing\nC. Relationship marketing\nD. Critical marketing analysis\nAnswer:", " Critical marketing analysis"], ["Question: This business-to-business pricing approach seeks to understand customers' needs before pricing the offering according to those needs in order to generate a long-term relationship. This is referred to as:\nChoices:\nA. Geographical pricing.\nB. Discount pricing.\nC. Relationship pricing.\nD. Value-in-use pricing.\nAnswer:", " Geographical pricing."], ["Question: This business-to-business pricing approach seeks to understand customers' needs before pricing the offering according to those needs in order to generate a long-term relationship. This is referred to as:\nChoices:\nA. Geographical pricing.\nB. Discount pricing.\nC. Relationship pricing.\nD. Value-in-use pricing.\nAnswer:", " Discount pricing."], ["Question: This business-to-business pricing approach seeks to understand customers' needs before pricing the offering according to those needs in order to generate a long-term relationship. This is referred to as:\nChoices:\nA. Geographical pricing.\nB. Discount pricing.\nC. Relationship pricing.\nD. Value-in-use pricing.\nAnswer:", " Relationship pricing."], ["Question: This business-to-business pricing approach seeks to understand customers' needs before pricing the offering according to those needs in order to generate a long-term relationship. This is referred to as:\nChoices:\nA. Geographical pricing.\nB. Discount pricing.\nC. Relationship pricing.\nD. Value-in-use pricing.\nAnswer:", " Value-in-use pricing."], ["Question:  What type of media helps advertisers demonstrate the benefits of using a particular product and can bring life and energy to an advertiser's message?\nChoices:\nA. Broadcast media.\nB. Interactive media.\nC. Print media.\nD. Support media.\nAnswer:", " Broadcast media."], ["Question:  What type of media helps advertisers demonstrate the benefits of using a particular product and can bring life and energy to an advertiser's message?\nChoices:\nA. Broadcast media.\nB. Interactive media.\nC. Print media.\nD. Support media.\nAnswer:", " Interactive media."], ["Question:  What type of media helps advertisers demonstrate the benefits of using a particular product and can bring life and energy to an advertiser's message?\nChoices:\nA. Broadcast media.\nB. Interactive media.\nC. Print media.\nD. Support media.\nAnswer:", " Print media."], ["Question:  What type of media helps advertisers demonstrate the benefits of using a particular product and can bring life and energy to an advertiser's message?\nChoices:\nA. Broadcast media.\nB. Interactive media.\nC. Print media.\nD. Support media.\nAnswer:", " Support media."]]
\ No newline at end of file

tests/testdata/hendrycksTest-marketing-v0-res.json

+{"results": {"hendrycksTest-marketing": {"acc": 0.4, "acc_stderr": 0.16329931618554522, "acc_norm": 0.5, "acc_norm_stderr": 0.16666666666666666}}, "versions": {"hendrycksTest-marketing": 0}}
\ No newline at end of file

tests/testdata/hendrycksTest-medical_genetics-v0-loglikelihood.json

+[["Question: Tay-Sachs disease is caused by deficiency of _________________?\nChoices:\nA. Alpha-L-iduronidase\nB. Glucose-6-phosphatase\nC. Hexosaminidase A\nD. Homogentisic acid oxidase\nAnswer:", " Alpha-L-iduronidase"], ["Question: Tay-Sachs disease is caused by deficiency of _________________?\nChoices:\nA. Alpha-L-iduronidase\nB. Glucose-6-phosphatase\nC. Hexosaminidase A\nD. Homogentisic acid oxidase\nAnswer:", " Glucose-6-phosphatase"], ["Question: Tay-Sachs disease is caused by deficiency of _________________?\nChoices:\nA. Alpha-L-iduronidase\nB. Glucose-6-phosphatase\nC. Hexosaminidase A\nD. Homogentisic acid oxidase\nAnswer:", " Hexosaminidase A"], ["Question: Tay-Sachs disease is caused by deficiency of _________________?\nChoices:\nA. Alpha-L-iduronidase\nB. Glucose-6-phosphatase\nC. Hexosaminidase A\nD. Homogentisic acid oxidase\nAnswer:", " Homogentisic acid oxidase"], ["Question: If an X-linked recessive disorder is in Hardy-Weinberg equilibrium and the incidence in males equals 1 in 100, then the expected incidence of affected homozygous females would be _______.\nChoices:\nA. 1 in 1000\nB. 1 in 4000\nC. 1 in 10 000\nD. 1 in 40 000\nAnswer:", " 1 in 1000"], ["Question: If an X-linked recessive disorder is in Hardy-Weinberg equilibrium and the incidence in males equals 1 in 100, then the expected incidence of affected homozygous females would be _______.\nChoices:\nA. 1 in 1000\nB. 1 in 4000\nC. 1 in 10 000\nD. 1 in 40 000\nAnswer:", " 1 in 4000"], ["Question: If an X-linked recessive disorder is in Hardy-Weinberg equilibrium and the incidence in males equals 1 in 100, then the expected incidence of affected homozygous females would be _______.\nChoices:\nA. 1 in 1000\nB. 1 in 4000\nC. 1 in 10 000\nD. 1 in 40 000\nAnswer:", " 1 in 10 000"], ["Question: If an X-linked recessive disorder is in Hardy-Weinberg equilibrium and the incidence in males equals 1 in 100, then the expected incidence of affected homozygous females would be _______.\nChoices:\nA. 1 in 1000\nB. 1 in 4000\nC. 1 in 10 000\nD. 1 in 40 000\nAnswer:", " 1 in 40 000"], ["Question: Consanguinity shows a strong association with which pattern of inheritance?\nChoices:\nA. Autosomal dominant\nB. Autosomal recessive\nC. X-linked dominant\nD. X-linked recessive\nAnswer:", " Autosomal dominant"], ["Question: Consanguinity shows a strong association with which pattern of inheritance?\nChoices:\nA. Autosomal dominant\nB. Autosomal recessive\nC. X-linked dominant\nD. X-linked recessive\nAnswer:", " Autosomal recessive"], ["Question: Consanguinity shows a strong association with which pattern of inheritance?\nChoices:\nA. Autosomal dominant\nB. Autosomal recessive\nC. X-linked dominant\nD. X-linked recessive\nAnswer:", " X-linked dominant"], ["Question: Consanguinity shows a strong association with which pattern of inheritance?\nChoices:\nA. Autosomal dominant\nB. Autosomal recessive\nC. X-linked dominant\nD. X-linked recessive\nAnswer:", " X-linked recessive"], ["Question: Exon skipping is associated with:\nChoices:\nA. nonsense mutations.\nB. regulatory mutations.\nC. RNA processing mutations.\nD. silent mutations.\nAnswer:", " nonsense mutations."], ["Question: Exon skipping is associated with:\nChoices:\nA. nonsense mutations.\nB. regulatory mutations.\nC. RNA processing mutations.\nD. silent mutations.\nAnswer:", " regulatory mutations."], ["Question: Exon skipping is associated with:\nChoices:\nA. nonsense mutations.\nB. regulatory mutations.\nC. RNA processing mutations.\nD. silent mutations.\nAnswer:", " RNA processing mutations."], ["Question: Exon skipping is associated with:\nChoices:\nA. nonsense mutations.\nB. regulatory mutations.\nC. RNA processing mutations.\nD. silent mutations.\nAnswer:", " silent mutations."], ["Question: Simple tandem repeat polymorphisms in humans are most useful for\nChoices:\nA. solving criminal and paternity cases\nB. reconstructing the relationships of humans and chimps.\nC. estimating relationships of humans and Neanderthals\nD. transferring disease resistance factors into bone marrow cells\nAnswer:", " solving criminal and paternity cases"], ["Question: Simple tandem repeat polymorphisms in humans are most useful for\nChoices:\nA. solving criminal and paternity cases\nB. reconstructing the relationships of humans and chimps.\nC. estimating relationships of humans and Neanderthals\nD. transferring disease resistance factors into bone marrow cells\nAnswer:", " reconstructing the relationships of humans and chimps."], ["Question: Simple tandem repeat polymorphisms in humans are most useful for\nChoices:\nA. solving criminal and paternity cases\nB. reconstructing the relationships of humans and chimps.\nC. estimating relationships of humans and Neanderthals\nD. transferring disease resistance factors into bone marrow cells\nAnswer:", " estimating relationships of humans and Neanderthals"], ["Question: Simple tandem repeat polymorphisms in humans are most useful for\nChoices:\nA. solving criminal and paternity cases\nB. reconstructing the relationships of humans and chimps.\nC. estimating relationships of humans and Neanderthals\nD. transferring disease resistance factors into bone marrow cells\nAnswer:", " transferring disease resistance factors into bone marrow cells"], ["Question: Which of the following is a feature of X-linked dominant inheritance?\nChoices:\nA. Parental consanguinity\nB. Male to male transmission\nC. Transmission only by females\nD. Transmitted by males only to females\nAnswer:", " Parental consanguinity"], ["Question: Which of the following is a feature of X-linked dominant inheritance?\nChoices:\nA. Parental consanguinity\nB. Male to male transmission\nC. Transmission only by females\nD. Transmitted by males only to females\nAnswer:", " Male to male transmission"], ["Question: Which of the following is a feature of X-linked dominant inheritance?\nChoices:\nA. Parental consanguinity\nB. Male to male transmission\nC. Transmission only by females\nD. Transmitted by males only to females\nAnswer:", " Transmission only by females"], ["Question: Which of the following is a feature of X-linked dominant inheritance?\nChoices:\nA. Parental consanguinity\nB. Male to male transmission\nC. Transmission only by females\nD. Transmitted by males only to females\nAnswer:", " Transmitted by males only to females"], ["Question: Zinc finger proteins and helix-turn-helix proteins are\nChoices:\nA. types of DNA-binding proteins\nB. involved in the control of translation\nC. components of ribosomes\nD. part of the hemoglobin in blood cells\nAnswer:", " types of DNA-binding proteins"], ["Question: Zinc finger proteins and helix-turn-helix proteins are\nChoices:\nA. types of DNA-binding proteins\nB. involved in the control of translation\nC. components of ribosomes\nD. part of the hemoglobin in blood cells\nAnswer:", " involved in the control of translation"], ["Question: Zinc finger proteins and helix-turn-helix proteins are\nChoices:\nA. types of DNA-binding proteins\nB. involved in the control of translation\nC. components of ribosomes\nD. part of the hemoglobin in blood cells\nAnswer:", " components of ribosomes"], ["Question: Zinc finger proteins and helix-turn-helix proteins are\nChoices:\nA. types of DNA-binding proteins\nB. involved in the control of translation\nC. components of ribosomes\nD. part of the hemoglobin in blood cells\nAnswer:", " part of the hemoglobin in blood cells"], ["Question: Male breast cancer is associated with mutations in ___.\nChoices:\nA. BRCA1\nB. BRCA2\nC. NF1\nD. RET\nAnswer:", " BRCA1"], ["Question: Male breast cancer is associated with mutations in ___.\nChoices:\nA. BRCA1\nB. BRCA2\nC. NF1\nD. RET\nAnswer:", " BRCA2"], ["Question: Male breast cancer is associated with mutations in ___.\nChoices:\nA. BRCA1\nB. BRCA2\nC. NF1\nD. RET\nAnswer:", " NF1"], ["Question: Male breast cancer is associated with mutations in ___.\nChoices:\nA. BRCA1\nB. BRCA2\nC. NF1\nD. RET\nAnswer:", " RET"], ["Question: QTL analysis is used to\nChoices:\nA. identify chromosome regions associated with a complex trait in a genetic cross\nB. determine which genes are expressed at a developmental stage\nC. map genes in bacterial viruses\nD. identify RNA polymerase binding sites\nAnswer:", " identify chromosome regions associated with a complex trait in a genetic cross"], ["Question: QTL analysis is used to\nChoices:\nA. identify chromosome regions associated with a complex trait in a genetic cross\nB. determine which genes are expressed at a developmental stage\nC. map genes in bacterial viruses\nD. identify RNA polymerase binding sites\nAnswer:", " determine which genes are expressed at a developmental stage"], ["Question: QTL analysis is used to\nChoices:\nA. identify chromosome regions associated with a complex trait in a genetic cross\nB. determine which genes are expressed at a developmental stage\nC. map genes in bacterial viruses\nD. identify RNA polymerase binding sites\nAnswer:", " map genes in bacterial viruses"], ["Question: QTL analysis is used to\nChoices:\nA. identify chromosome regions associated with a complex trait in a genetic cross\nB. determine which genes are expressed at a developmental stage\nC. map genes in bacterial viruses\nD. identify RNA polymerase binding sites\nAnswer:", " identify RNA polymerase binding sites"], ["Question: Which component of transcribed RNA in eukaryotes is present in the initial transcript but is removed before translation occurs?\nChoices:\nA. Intron\nB. 3\u2019 Poly A tail\nC. Ribosome binding site\nD. 5\u2019 cap\nAnswer:", " Intron"], ["Question: Which component of transcribed RNA in eukaryotes is present in the initial transcript but is removed before translation occurs?\nChoices:\nA. Intron\nB. 3\u2019 Poly A tail\nC. Ribosome binding site\nD. 5\u2019 cap\nAnswer:", " 3\u2019 Poly A tail"], ["Question: Which component of transcribed RNA in eukaryotes is present in the initial transcript but is removed before translation occurs?\nChoices:\nA. Intron\nB. 3\u2019 Poly A tail\nC. Ribosome binding site\nD. 5\u2019 cap\nAnswer:", " Ribosome binding site"], ["Question: Which component of transcribed RNA in eukaryotes is present in the initial transcript but is removed before translation occurs?\nChoices:\nA. Intron\nB. 3\u2019 Poly A tail\nC. Ribosome binding site\nD. 5\u2019 cap\nAnswer:", " 5\u2019 cap"]]
\ No newline at end of file

tests/testdata/hendrycksTest-medical_genetics-v0-res.json

+{"results": {"hendrycksTest-medical_genetics": {"acc": 0.3, "acc_stderr": 0.15275252316519464, "acc_norm": 0.4, "acc_norm_stderr": 0.1632993161855452}}, "versions": {"hendrycksTest-medical_genetics": 0}}
\ No newline at end of file

tests/testdata/hendrycksTest-miscellaneous-v0-loglikelihood.json

+[["Question: Which of the following must be obtained by foreigners wishing to permanently reside in the US?\nChoices:\nA. visa\nB. bill of landing\nC. driver's license\nD. carte blanche\nAnswer:", " visa"], ["Question: Which of the following must be obtained by foreigners wishing to permanently reside in the US?\nChoices:\nA. visa\nB. bill of landing\nC. driver's license\nD. carte blanche\nAnswer:", " bill of landing"], ["Question: Which of the following must be obtained by foreigners wishing to permanently reside in the US?\nChoices:\nA. visa\nB. bill of landing\nC. driver's license\nD. carte blanche\nAnswer:", " driver's license"], ["Question: Which of the following must be obtained by foreigners wishing to permanently reside in the US?\nChoices:\nA. visa\nB. bill of landing\nC. driver's license\nD. carte blanche\nAnswer:", " carte blanche"], ["Question: Who were the Know-Nothings?\nChoices:\nA. a '60's comedy troupe\nB. computer designers\nC. a political party\nD. a spy ring\nAnswer:", " a '60's comedy troupe"], ["Question: Who were the Know-Nothings?\nChoices:\nA. a '60's comedy troupe\nB. computer designers\nC. a political party\nD. a spy ring\nAnswer:", " computer designers"], ["Question: Who were the Know-Nothings?\nChoices:\nA. a '60's comedy troupe\nB. computer designers\nC. a political party\nD. a spy ring\nAnswer:", " a political party"], ["Question: Who were the Know-Nothings?\nChoices:\nA. a '60's comedy troupe\nB. computer designers\nC. a political party\nD. a spy ring\nAnswer:", " a spy ring"], ["Question: Which of the following best helps explain why some localities have normally large tidal ranges (up to 60 feet) and others have one- to two-foot tidal ranges?\nChoices:\nA. The position of the Sun is different at different localities.\nB. The Coriolis effect and rotation of Earth tend to enhance tidal flow in the higher latitudes.\nC. Ocean floor topography and the shape of the coastline serve to amplify tidal flow at specific localities.\nD. Trade winds push the water into large tidal bulges near rocky shorelines.\nAnswer:", " The position of the Sun is different at different localities."], ["Question: Which of the following best helps explain why some localities have normally large tidal ranges (up to 60 feet) and others have one- to two-foot tidal ranges?\nChoices:\nA. The position of the Sun is different at different localities.\nB. The Coriolis effect and rotation of Earth tend to enhance tidal flow in the higher latitudes.\nC. Ocean floor topography and the shape of the coastline serve to amplify tidal flow at specific localities.\nD. Trade winds push the water into large tidal bulges near rocky shorelines.\nAnswer:", " The Coriolis effect and rotation of Earth tend to enhance tidal flow in the higher latitudes."], ["Question: Which of the following best helps explain why some localities have normally large tidal ranges (up to 60 feet) and others have one- to two-foot tidal ranges?\nChoices:\nA. The position of the Sun is different at different localities.\nB. The Coriolis effect and rotation of Earth tend to enhance tidal flow in the higher latitudes.\nC. Ocean floor topography and the shape of the coastline serve to amplify tidal flow at specific localities.\nD. Trade winds push the water into large tidal bulges near rocky shorelines.\nAnswer:", " Ocean floor topography and the shape of the coastline serve to amplify tidal flow at specific localities."], ["Question: Which of the following best helps explain why some localities have normally large tidal ranges (up to 60 feet) and others have one- to two-foot tidal ranges?\nChoices:\nA. The position of the Sun is different at different localities.\nB. The Coriolis effect and rotation of Earth tend to enhance tidal flow in the higher latitudes.\nC. Ocean floor topography and the shape of the coastline serve to amplify tidal flow at specific localities.\nD. Trade winds push the water into large tidal bulges near rocky shorelines.\nAnswer:", " Trade winds push the water into large tidal bulges near rocky shorelines."], ["Question: Which of the following steps should be taken first to effectively present new information to coworkers?\nChoices:\nA. Analyzing the audience\nB. Determining the objective\nC. Creating three to five key messages\nD. Choosing a communication tool\nAnswer:", " Analyzing the audience"], ["Question: Which of the following steps should be taken first to effectively present new information to coworkers?\nChoices:\nA. Analyzing the audience\nB. Determining the objective\nC. Creating three to five key messages\nD. Choosing a communication tool\nAnswer:", " Determining the objective"], ["Question: Which of the following steps should be taken first to effectively present new information to coworkers?\nChoices:\nA. Analyzing the audience\nB. Determining the objective\nC. Creating three to five key messages\nD. Choosing a communication tool\nAnswer:", " Creating three to five key messages"], ["Question: Which of the following steps should be taken first to effectively present new information to coworkers?\nChoices:\nA. Analyzing the audience\nB. Determining the objective\nC. Creating three to five key messages\nD. Choosing a communication tool\nAnswer:", " Choosing a communication tool"], ["Question: When Corporation X pays out $300,000 in salaries, which of the following is the impact on assets and liabilities?\nChoices:\nA. Salary expense is increased by $300,000; cash is increased by $300,000\nB. Salary expense is increased by $300,000; cash is decreased by $300,000\nC. Salary expense is decreased by $300,000; cash is increased by $300,000\nD. Salary expense is decreased by $300,000; cash is decreased by $300,000\nAnswer:", " Salary expense is increased by $300,000; cash is increased by $300,000"], ["Question: When Corporation X pays out $300,000 in salaries, which of the following is the impact on assets and liabilities?\nChoices:\nA. Salary expense is increased by $300,000; cash is increased by $300,000\nB. Salary expense is increased by $300,000; cash is decreased by $300,000\nC. Salary expense is decreased by $300,000; cash is increased by $300,000\nD. Salary expense is decreased by $300,000; cash is decreased by $300,000\nAnswer:", " Salary expense is increased by $300,000; cash is decreased by $300,000"], ["Question: When Corporation X pays out $300,000 in salaries, which of the following is the impact on assets and liabilities?\nChoices:\nA. Salary expense is increased by $300,000; cash is increased by $300,000\nB. Salary expense is increased by $300,000; cash is decreased by $300,000\nC. Salary expense is decreased by $300,000; cash is increased by $300,000\nD. Salary expense is decreased by $300,000; cash is decreased by $300,000\nAnswer:", " Salary expense is decreased by $300,000; cash is increased by $300,000"], ["Question: When Corporation X pays out $300,000 in salaries, which of the following is the impact on assets and liabilities?\nChoices:\nA. Salary expense is increased by $300,000; cash is increased by $300,000\nB. Salary expense is increased by $300,000; cash is decreased by $300,000\nC. Salary expense is decreased by $300,000; cash is increased by $300,000\nD. Salary expense is decreased by $300,000; cash is decreased by $300,000\nAnswer:", " Salary expense is decreased by $300,000; cash is decreased by $300,000"], ["Question: Before he went into coaching Phil Jackson played for which of the following NBA teams?\nChoices:\nA. Boston Celtics\nB. Los Angeles Lakers\nC. New York Knicks\nD. Philadelphia 76ers\nAnswer:", " Boston Celtics"], ["Question: Before he went into coaching Phil Jackson played for which of the following NBA teams?\nChoices:\nA. Boston Celtics\nB. Los Angeles Lakers\nC. New York Knicks\nD. Philadelphia 76ers\nAnswer:", " Los Angeles Lakers"], ["Question: Before he went into coaching Phil Jackson played for which of the following NBA teams?\nChoices:\nA. Boston Celtics\nB. Los Angeles Lakers\nC. New York Knicks\nD. Philadelphia 76ers\nAnswer:", " New York Knicks"], ["Question: Before he went into coaching Phil Jackson played for which of the following NBA teams?\nChoices:\nA. Boston Celtics\nB. Los Angeles Lakers\nC. New York Knicks\nD. Philadelphia 76ers\nAnswer:", " Philadelphia 76ers"], ["Question: Which of the following components is present in both pneumatic and hydraulic systems?\nChoices:\nA. Exhaust\nB. Reservoir\nC. Compressor\nD. Control value\nAnswer:", " Exhaust"], ["Question: Which of the following components is present in both pneumatic and hydraulic systems?\nChoices:\nA. Exhaust\nB. Reservoir\nC. Compressor\nD. Control value\nAnswer:", " Reservoir"], ["Question: Which of the following components is present in both pneumatic and hydraulic systems?\nChoices:\nA. Exhaust\nB. Reservoir\nC. Compressor\nD. Control value\nAnswer:", " Compressor"], ["Question: Which of the following components is present in both pneumatic and hydraulic systems?\nChoices:\nA. Exhaust\nB. Reservoir\nC. Compressor\nD. Control value\nAnswer:", " Control value"], ["Question: Where was the chicken First domesticated?\nChoices:\nA. France\nB. India\nC. Peru\nD. Zaire\nAnswer:", " France"], ["Question: Where was the chicken First domesticated?\nChoices:\nA. France\nB. India\nC. Peru\nD. Zaire\nAnswer:", " India"], ["Question: Where was the chicken First domesticated?\nChoices:\nA. France\nB. India\nC. Peru\nD. Zaire\nAnswer:", " Peru"], ["Question: Where was the chicken First domesticated?\nChoices:\nA. France\nB. India\nC. Peru\nD. Zaire\nAnswer:", " Zaire"], ["Question: What is the loudest animal on earth?\nChoices:\nA. African elephant\nB. sperm whale\nC. blue whale\nD. Indian elephant\nAnswer:", " African elephant"], ["Question: What is the loudest animal on earth?\nChoices:\nA. African elephant\nB. sperm whale\nC. blue whale\nD. Indian elephant\nAnswer:", " sperm whale"], ["Question: What is the loudest animal on earth?\nChoices:\nA. African elephant\nB. sperm whale\nC. blue whale\nD. Indian elephant\nAnswer:", " blue whale"], ["Question: What is the loudest animal on earth?\nChoices:\nA. African elephant\nB. sperm whale\nC. blue whale\nD. Indian elephant\nAnswer:", " Indian elephant"], ["Question: When it's noon in New York during daylight-saving time what time is it in Honolulu?\nChoices:\nA. 6:00 AM\nB. 6:30 AM\nC. 7:00 AM\nD. 8:00 AM\nAnswer:", " 6:00 AM"], ["Question: When it's noon in New York during daylight-saving time what time is it in Honolulu?\nChoices:\nA. 6:00 AM\nB. 6:30 AM\nC. 7:00 AM\nD. 8:00 AM\nAnswer:", " 6:30 AM"], ["Question: When it's noon in New York during daylight-saving time what time is it in Honolulu?\nChoices:\nA. 6:00 AM\nB. 6:30 AM\nC. 7:00 AM\nD. 8:00 AM\nAnswer:", " 7:00 AM"], ["Question: When it's noon in New York during daylight-saving time what time is it in Honolulu?\nChoices:\nA. 6:00 AM\nB. 6:30 AM\nC. 7:00 AM\nD. 8:00 AM\nAnswer:", " 8:00 AM"]]
\ No newline at end of file

tests/testdata/hendrycksTest-miscellaneous-v0-res.json

+{"results": {"hendrycksTest-miscellaneous": {"acc": 0.2, "acc_stderr": 0.13333333333333333, "acc_norm": 0.2, "acc_norm_stderr": 0.13333333333333333}}, "versions": {"hendrycksTest-miscellaneous": 0}}
\ No newline at end of file

tests/testdata/hendrycksTest-moral_disputes-v0-loglikelihood.json

+[["Question:  On Pogge's view, affluent citizens are harming the poor by\nChoices:\nA. imposing on them the lasting effects of historical crimes.\nB. keeping them below welfare in the state-of-nature.\nC. both A and B\nD. None of the above\nAnswer:", " imposing on them the lasting effects of historical crimes."], ["Question:  On Pogge's view, affluent citizens are harming the poor by\nChoices:\nA. imposing on them the lasting effects of historical crimes.\nB. keeping them below welfare in the state-of-nature.\nC. both A and B\nD. None of the above\nAnswer:", " keeping them below welfare in the state-of-nature."], ["Question:  On Pogge's view, affluent citizens are harming the poor by\nChoices:\nA. imposing on them the lasting effects of historical crimes.\nB. keeping them below welfare in the state-of-nature.\nC. both A and B\nD. None of the above\nAnswer:", " both A and B"], ["Question:  On Pogge's view, affluent citizens are harming the poor by\nChoices:\nA. imposing on them the lasting effects of historical crimes.\nB. keeping them below welfare in the state-of-nature.\nC. both A and B\nD. None of the above\nAnswer:", " None of the above"], ["Question:  The basic idea of social contract theories of morality is that correct or justified moral rules or principles are the ones that result from\nChoices:\nA. a social leader's moral deliberations.\nB. an actual or hypothetical social agreement of some sort.\nC. a contract that has been signed by most of the affected parties.\nD. none of the above\nAnswer:", " a social leader's moral deliberations."], ["Question:  The basic idea of social contract theories of morality is that correct or justified moral rules or principles are the ones that result from\nChoices:\nA. a social leader's moral deliberations.\nB. an actual or hypothetical social agreement of some sort.\nC. a contract that has been signed by most of the affected parties.\nD. none of the above\nAnswer:", " an actual or hypothetical social agreement of some sort."], ["Question:  The basic idea of social contract theories of morality is that correct or justified moral rules or principles are the ones that result from\nChoices:\nA. a social leader's moral deliberations.\nB. an actual or hypothetical social agreement of some sort.\nC. a contract that has been signed by most of the affected parties.\nD. none of the above\nAnswer:", " a contract that has been signed by most of the affected parties."], ["Question:  The basic idea of social contract theories of morality is that correct or justified moral rules or principles are the ones that result from\nChoices:\nA. a social leader's moral deliberations.\nB. an actual or hypothetical social agreement of some sort.\nC. a contract that has been signed by most of the affected parties.\nD. none of the above\nAnswer:", " none of the above"], ["Question:  On the proposal that we need to establish world food banks to help those who are in need, Hardin would say that\nChoices:\nA. if the proposal were to be realized, the operation must be conducted consistently.\nB. only the richer countries have some moral obligation to make deposits in the world food banks.\nC. it would be subject to the tragedy of the commons.\nD. we need to go with the idea because we ought not to punish poor people who are caught in an emergency.\nAnswer:", " if the proposal were to be realized, the operation must be conducted consistently."], ["Question:  On the proposal that we need to establish world food banks to help those who are in need, Hardin would say that\nChoices:\nA. if the proposal were to be realized, the operation must be conducted consistently.\nB. only the richer countries have some moral obligation to make deposits in the world food banks.\nC. it would be subject to the tragedy of the commons.\nD. we need to go with the idea because we ought not to punish poor people who are caught in an emergency.\nAnswer:", " only the richer countries have some moral obligation to make deposits in the world food banks."], ["Question:  On the proposal that we need to establish world food banks to help those who are in need, Hardin would say that\nChoices:\nA. if the proposal were to be realized, the operation must be conducted consistently.\nB. only the richer countries have some moral obligation to make deposits in the world food banks.\nC. it would be subject to the tragedy of the commons.\nD. we need to go with the idea because we ought not to punish poor people who are caught in an emergency.\nAnswer:", " it would be subject to the tragedy of the commons."], ["Question:  On the proposal that we need to establish world food banks to help those who are in need, Hardin would say that\nChoices:\nA. if the proposal were to be realized, the operation must be conducted consistently.\nB. only the richer countries have some moral obligation to make deposits in the world food banks.\nC. it would be subject to the tragedy of the commons.\nD. we need to go with the idea because we ought not to punish poor people who are caught in an emergency.\nAnswer:", " we need to go with the idea because we ought not to punish poor people who are caught in an emergency."], ["Question:  What kind of consequentialist theory does Dershowitz think can justify terrorism in certain extreme particular cases?\nChoices:\nA. act-based deontology\nB. rule-based hedonism\nC. rule utilitarianism\nD. act utilitarianism\nAnswer:", " act-based deontology"], ["Question:  What kind of consequentialist theory does Dershowitz think can justify terrorism in certain extreme particular cases?\nChoices:\nA. act-based deontology\nB. rule-based hedonism\nC. rule utilitarianism\nD. act utilitarianism\nAnswer:", " rule-based hedonism"], ["Question:  What kind of consequentialist theory does Dershowitz think can justify terrorism in certain extreme particular cases?\nChoices:\nA. act-based deontology\nB. rule-based hedonism\nC. rule utilitarianism\nD. act utilitarianism\nAnswer:", " rule utilitarianism"], ["Question:  What kind of consequentialist theory does Dershowitz think can justify terrorism in certain extreme particular cases?\nChoices:\nA. act-based deontology\nB. rule-based hedonism\nC. rule utilitarianism\nD. act utilitarianism\nAnswer:", " act utilitarianism"], ["Question:  The best explanation for drug addiction, according to Shapiro, appeals to\nChoices:\nA. one's individual mindset and social setting.\nB. the pharmacological effects of drug use (e.g., withdrawal).\nC. one's genetic profile, which explains why some people have \"addictive personalities.\"\nD. specific psychological disorders such as obsessive-compulsive disorder.\nAnswer:", " one's individual mindset and social setting."], ["Question:  The best explanation for drug addiction, according to Shapiro, appeals to\nChoices:\nA. one's individual mindset and social setting.\nB. the pharmacological effects of drug use (e.g., withdrawal).\nC. one's genetic profile, which explains why some people have \"addictive personalities.\"\nD. specific psychological disorders such as obsessive-compulsive disorder.\nAnswer:", " the pharmacological effects of drug use (e.g., withdrawal)."], ["Question:  The best explanation for drug addiction, according to Shapiro, appeals to\nChoices:\nA. one's individual mindset and social setting.\nB. the pharmacological effects of drug use (e.g., withdrawal).\nC. one's genetic profile, which explains why some people have \"addictive personalities.\"\nD. specific psychological disorders such as obsessive-compulsive disorder.\nAnswer:", " one's genetic profile, which explains why some people have \"addictive personalities.\""], ["Question:  The best explanation for drug addiction, according to Shapiro, appeals to\nChoices:\nA. one's individual mindset and social setting.\nB. the pharmacological effects of drug use (e.g., withdrawal).\nC. one's genetic profile, which explains why some people have \"addictive personalities.\"\nD. specific psychological disorders such as obsessive-compulsive disorder.\nAnswer:", " specific psychological disorders such as obsessive-compulsive disorder."], ["Question:  According to Kant, all imperatives are expressed by the word\nChoices:\nA. \"want.\"\nB. \"ought.\"\nC. \"will.\"\nD. \"may.\"\nAnswer:", " \"want.\""], ["Question:  According to Kant, all imperatives are expressed by the word\nChoices:\nA. \"want.\"\nB. \"ought.\"\nC. \"will.\"\nD. \"may.\"\nAnswer:", " \"ought.\""], ["Question:  According to Kant, all imperatives are expressed by the word\nChoices:\nA. \"want.\"\nB. \"ought.\"\nC. \"will.\"\nD. \"may.\"\nAnswer:", " \"will.\""], ["Question:  According to Kant, all imperatives are expressed by the word\nChoices:\nA. \"want.\"\nB. \"ought.\"\nC. \"will.\"\nD. \"may.\"\nAnswer:", " \"may.\""], ["Question:  Van den Haag responds to the \"miscarriages of justice\" objection by claiming that\nChoices:\nA. miscarriages of justice are offset by the moral benefits and usefulness of doing justice.\nB. there have been no miscarriages of justice, in the sense used in the objection.\nC. miscarriages of justice are inevitable and so irrelevant.\nD. none of the above\nAnswer:", " miscarriages of justice are offset by the moral benefits and usefulness of doing justice."], ["Question:  Van den Haag responds to the \"miscarriages of justice\" objection by claiming that\nChoices:\nA. miscarriages of justice are offset by the moral benefits and usefulness of doing justice.\nB. there have been no miscarriages of justice, in the sense used in the objection.\nC. miscarriages of justice are inevitable and so irrelevant.\nD. none of the above\nAnswer:", " there have been no miscarriages of justice, in the sense used in the objection."], ["Question:  Van den Haag responds to the \"miscarriages of justice\" objection by claiming that\nChoices:\nA. miscarriages of justice are offset by the moral benefits and usefulness of doing justice.\nB. there have been no miscarriages of justice, in the sense used in the objection.\nC. miscarriages of justice are inevitable and so irrelevant.\nD. none of the above\nAnswer:", " miscarriages of justice are inevitable and so irrelevant."], ["Question:  Van den Haag responds to the \"miscarriages of justice\" objection by claiming that\nChoices:\nA. miscarriages of justice are offset by the moral benefits and usefulness of doing justice.\nB. there have been no miscarriages of justice, in the sense used in the objection.\nC. miscarriages of justice are inevitable and so irrelevant.\nD. none of the above\nAnswer:", " none of the above"], ["Question:  Mill thinks that if something is desirable, but not desirable as an end, then it must be\nChoices:\nA. desirable as a rule.\nB. desirable in theory.\nC. desirable as a means.\nD. none of the above\nAnswer:", " desirable as a rule."], ["Question:  Mill thinks that if something is desirable, but not desirable as an end, then it must be\nChoices:\nA. desirable as a rule.\nB. desirable in theory.\nC. desirable as a means.\nD. none of the above\nAnswer:", " desirable in theory."], ["Question:  Mill thinks that if something is desirable, but not desirable as an end, then it must be\nChoices:\nA. desirable as a rule.\nB. desirable in theory.\nC. desirable as a means.\nD. none of the above\nAnswer:", " desirable as a means."], ["Question:  Mill thinks that if something is desirable, but not desirable as an end, then it must be\nChoices:\nA. desirable as a rule.\nB. desirable in theory.\nC. desirable as a means.\nD. none of the above\nAnswer:", " none of the above"], ["Question:  Cases in which a doctor is involved to some degree in assisting an individual to commit suicide are known as\nChoices:\nA. mercy killing.\nB. physician-assisted suicide.\nC. involuntary euthanasia.\nD. all of the above\nAnswer:", " mercy killing."], ["Question:  Cases in which a doctor is involved to some degree in assisting an individual to commit suicide are known as\nChoices:\nA. mercy killing.\nB. physician-assisted suicide.\nC. involuntary euthanasia.\nD. all of the above\nAnswer:", " physician-assisted suicide."], ["Question:  Cases in which a doctor is involved to some degree in assisting an individual to commit suicide are known as\nChoices:\nA. mercy killing.\nB. physician-assisted suicide.\nC. involuntary euthanasia.\nD. all of the above\nAnswer:", " involuntary euthanasia."], ["Question:  Cases in which a doctor is involved to some degree in assisting an individual to commit suicide are known as\nChoices:\nA. mercy killing.\nB. physician-assisted suicide.\nC. involuntary euthanasia.\nD. all of the above\nAnswer:", " all of the above"], ["Question:  According to Norcross, any attempt to justify the claim that humans have a higher moral status than other animals by appealing to some version of rationality as the morally relevant difference between humans and animals will\nChoices:\nA. fail to give an adequate answer to the argument from marginal cases.\nB. fail to make the case that such a difference is morally relevant to the status of animals as moral patients as opposed to their status as moral agents.\nC. both A and B\nD. neither A nor B\nAnswer:", " fail to give an adequate answer to the argument from marginal cases."], ["Question:  According to Norcross, any attempt to justify the claim that humans have a higher moral status than other animals by appealing to some version of rationality as the morally relevant difference between humans and animals will\nChoices:\nA. fail to give an adequate answer to the argument from marginal cases.\nB. fail to make the case that such a difference is morally relevant to the status of animals as moral patients as opposed to their status as moral agents.\nC. both A and B\nD. neither A nor B\nAnswer:", " fail to make the case that such a difference is morally relevant to the status of animals as moral patients as opposed to their status as moral agents."], ["Question:  According to Norcross, any attempt to justify the claim that humans have a higher moral status than other animals by appealing to some version of rationality as the morally relevant difference between humans and animals will\nChoices:\nA. fail to give an adequate answer to the argument from marginal cases.\nB. fail to make the case that such a difference is morally relevant to the status of animals as moral patients as opposed to their status as moral agents.\nC. both A and B\nD. neither A nor B\nAnswer:", " both A and B"], ["Question:  According to Norcross, any attempt to justify the claim that humans have a higher moral status than other animals by appealing to some version of rationality as the morally relevant difference between humans and animals will\nChoices:\nA. fail to give an adequate answer to the argument from marginal cases.\nB. fail to make the case that such a difference is morally relevant to the status of animals as moral patients as opposed to their status as moral agents.\nC. both A and B\nD. neither A nor B\nAnswer:", " neither A nor B"]]
\ No newline at end of file

tests/testdata/hendrycksTest-moral_disputes-v0-res.json

+{"results": {"hendrycksTest-moral_disputes": {"acc": 0.0, "acc_stderr": 0.0, "acc_norm": 0.0, "acc_norm_stderr": 0.0}}, "versions": {"hendrycksTest-moral_disputes": 0}}
\ No newline at end of file

tests/testdata/hendrycksTest-moral_scenarios-v0-loglikelihood.json

+[["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I wore a fake moustache to the costume party. Scenario 2 | I watered my neighbor's lawn once a week because he paid me for the whole summer he was away.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I wore a fake moustache to the costume party. Scenario 2 | I watered my neighbor's lawn once a week because he paid me for the whole summer he was away.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I wore a fake moustache to the costume party. Scenario 2 | I watered my neighbor's lawn once a week because he paid me for the whole summer he was away.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I wore a fake moustache to the costume party. Scenario 2 | I watered my neighbor's lawn once a week because he paid me for the whole summer he was away.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | When I went to the bank, I covered my face with a mask to prevent catching a virus. Scenario 2 | When I went through the road block I gave the policeman my ID.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | When I went to the bank, I covered my face with a mask to prevent catching a virus. Scenario 2 | When I went through the road block I gave the policeman my ID.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | When I went to the bank, I covered my face with a mask to prevent catching a virus. Scenario 2 | When I went through the road block I gave the policeman my ID.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | When I went to the bank, I covered my face with a mask to prevent catching a virus. Scenario 2 | When I went through the road block I gave the policeman my ID.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I put the gerbil in the oven for 35 minutes. Scenario 2 | I brought a knife with me when I went to school.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I put the gerbil in the oven for 35 minutes. Scenario 2 | I brought a knife with me when I went to school.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I put the gerbil in the oven for 35 minutes. Scenario 2 | I brought a knife with me when I went to school.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I put the gerbil in the oven for 35 minutes. Scenario 2 | I brought a knife with me when I went to school.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | The hospital signs told us to turn off all electrical devices, so I turned my phone off. Scenario 2 | I had the flu so I went to my office Christmas party.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | The hospital signs told us to turn off all electrical devices, so I turned my phone off. Scenario 2 | I had the flu so I went to my office Christmas party.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | The hospital signs told us to turn off all electrical devices, so I turned my phone off. Scenario 2 | I had the flu so I went to my office Christmas party.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | The hospital signs told us to turn off all electrical devices, so I turned my phone off. Scenario 2 | I had the flu so I went to my office Christmas party.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I made her sushi from her leftover salmon. Scenario 2 | I jumped out of the window onto the ground three feet below.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I made her sushi from her leftover salmon. Scenario 2 | I jumped out of the window onto the ground three feet below.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I made her sushi from her leftover salmon. Scenario 2 | I jumped out of the window onto the ground three feet below.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I made her sushi from her leftover salmon. Scenario 2 | I jumped out of the window onto the ground three feet below.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I helped him in English by proofreading his essay.   Scenario 2 | I was very proud of the SAT test scores I got for Henry.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I helped him in English by proofreading his essay.   Scenario 2 | I was very proud of the SAT test scores I got for Henry.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I helped him in English by proofreading his essay.   Scenario 2 | I was very proud of the SAT test scores I got for Henry.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I helped him in English by proofreading his essay.   Scenario 2 | I was very proud of the SAT test scores I got for Henry.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I turned on the baseball field's sprinklers to water the grass. Scenario 2 | I spent all night studying to pass the test.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I turned on the baseball field's sprinklers to water the grass. Scenario 2 | I spent all night studying to pass the test.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I turned on the baseball field's sprinklers to water the grass. Scenario 2 | I spent all night studying to pass the test.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I turned on the baseball field's sprinklers to water the grass. Scenario 2 | I spent all night studying to pass the test.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I passionately kissed my husband in the living room. Scenario 2 | I rushed my grandfather to the hospital when he had a heart attack.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I passionately kissed my husband in the living room. Scenario 2 | I rushed my grandfather to the hospital when he had a heart attack.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I passionately kissed my husband in the living room. Scenario 2 | I rushed my grandfather to the hospital when he had a heart attack.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I passionately kissed my husband in the living room. Scenario 2 | I rushed my grandfather to the hospital when he had a heart attack.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I washed the car and was paid. Scenario 2 | During the house's demolition, I pushed over a wall as he watched.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I washed the car and was paid. Scenario 2 | During the house's demolition, I pushed over a wall as he watched.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I washed the car and was paid. Scenario 2 | During the house's demolition, I pushed over a wall as he watched.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I washed the car and was paid. Scenario 2 | During the house's demolition, I pushed over a wall as he watched.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I built a snowman outside the beautiful home. Scenario 2 | After John finished his 10th beer he drove me home.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I built a snowman outside the beautiful home. Scenario 2 | After John finished his 10th beer he drove me home.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Wrong, Not wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I built a snowman outside the beautiful home. Scenario 2 | After John finished his 10th beer he drove me home.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Wrong"], ["Question: For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I built a snowman outside the beautiful home. Scenario 2 | After John finished his 10th beer he drove me home.\nChoices:\nA. Wrong, Wrong\nB. Wrong, Not wrong\nC. Not wrong, Wrong\nD. Not wrong, Not wrong\nAnswer:", " Not wrong, Not wrong"]]
\ No newline at end of file

tests/testdata/hendrycksTest-moral_scenarios-v0-res.json

+{"results": {"hendrycksTest-moral_scenarios": {"acc": 0.2, "acc_stderr": 0.13333333333333333, "acc_norm": 0.4, "acc_norm_stderr": 0.16329931618554522}}, "versions": {"hendrycksTest-moral_scenarios": 0}}
\ No newline at end of file

tests/testdata/hendrycksTest-nutrition-v0-loglikelihood.json

+[["Question: What characteristic is not representative of a type IIb muscle fibre?\n\nChoices:\nA. Low oxidative capacity\nB. High fatigue resistance\nC. High motor unit strength\nD. Fast contractile speed\nAnswer:", " Low oxidative capacity"], ["Question: What characteristic is not representative of a type IIb muscle fibre?\n\nChoices:\nA. Low oxidative capacity\nB. High fatigue resistance\nC. High motor unit strength\nD. Fast contractile speed\nAnswer:", " High fatigue resistance"], ["Question: What characteristic is not representative of a type IIb muscle fibre?\n\nChoices:\nA. Low oxidative capacity\nB. High fatigue resistance\nC. High motor unit strength\nD. Fast contractile speed\nAnswer:", " High motor unit strength"], ["Question: What characteristic is not representative of a type IIb muscle fibre?\n\nChoices:\nA. Low oxidative capacity\nB. High fatigue resistance\nC. High motor unit strength\nD. Fast contractile speed\nAnswer:", " Fast contractile speed"], ["Question: Which of the following inborn errors of metabolism gives rise to zinc deficiency?\n\nChoices:\nA. Acrodermatitis enteropathica\nB. Wilson's disease\nC. Menkes disease\nD. Haemochromatosis\nAnswer:", " Acrodermatitis enteropathica"], ["Question: Which of the following inborn errors of metabolism gives rise to zinc deficiency?\n\nChoices:\nA. Acrodermatitis enteropathica\nB. Wilson's disease\nC. Menkes disease\nD. Haemochromatosis\nAnswer:", " Wilson's disease"], ["Question: Which of the following inborn errors of metabolism gives rise to zinc deficiency?\n\nChoices:\nA. Acrodermatitis enteropathica\nB. Wilson's disease\nC. Menkes disease\nD. Haemochromatosis\nAnswer:", " Menkes disease"], ["Question: Which of the following inborn errors of metabolism gives rise to zinc deficiency?\n\nChoices:\nA. Acrodermatitis enteropathica\nB. Wilson's disease\nC. Menkes disease\nD. Haemochromatosis\nAnswer:", " Haemochromatosis"], ["Question: After consumption of a mixed meal, a complex cascade of events takes place that integrates fat metabolism at the whole body level. Which of the following is correct?\n\nChoices:\nA. Consumption of a meal leads to suppression of lipase activity within adipose tissue leading to a decrease in plasma NEFA concentrations\nB. Adipose tissue lipoprotein lipase is activated by insulin and therefore is most active following meal consumption\nC. During the postprandial period the VLDL synthesis pathway is suppressed in favour of hydrolysis of chylomicrons\nD. All options given are correct\nAnswer:", " Consumption of a meal leads to suppression of lipase activity within adipose tissue leading to a decrease in plasma NEFA concentrations"], ["Question: After consumption of a mixed meal, a complex cascade of events takes place that integrates fat metabolism at the whole body level. Which of the following is correct?\n\nChoices:\nA. Consumption of a meal leads to suppression of lipase activity within adipose tissue leading to a decrease in plasma NEFA concentrations\nB. Adipose tissue lipoprotein lipase is activated by insulin and therefore is most active following meal consumption\nC. During the postprandial period the VLDL synthesis pathway is suppressed in favour of hydrolysis of chylomicrons\nD. All options given are correct\nAnswer:", " Adipose tissue lipoprotein lipase is activated by insulin and therefore is most active following meal consumption"], ["Question: After consumption of a mixed meal, a complex cascade of events takes place that integrates fat metabolism at the whole body level. Which of the following is correct?\n\nChoices:\nA. Consumption of a meal leads to suppression of lipase activity within adipose tissue leading to a decrease in plasma NEFA concentrations\nB. Adipose tissue lipoprotein lipase is activated by insulin and therefore is most active following meal consumption\nC. During the postprandial period the VLDL synthesis pathway is suppressed in favour of hydrolysis of chylomicrons\nD. All options given are correct\nAnswer:", " During the postprandial period the VLDL synthesis pathway is suppressed in favour of hydrolysis of chylomicrons"], ["Question: After consumption of a mixed meal, a complex cascade of events takes place that integrates fat metabolism at the whole body level. Which of the following is correct?\n\nChoices:\nA. Consumption of a meal leads to suppression of lipase activity within adipose tissue leading to a decrease in plasma NEFA concentrations\nB. Adipose tissue lipoprotein lipase is activated by insulin and therefore is most active following meal consumption\nC. During the postprandial period the VLDL synthesis pathway is suppressed in favour of hydrolysis of chylomicrons\nD. All options given are correct\nAnswer:", " All options given are correct"], ["Question: Acute binge drinking is associated with?\n\nChoices:\nA. Happy heart syndrome\nB. Home heart syndrome\nC. Beach heart syndrome\nD. Holiday heart syndrome\nAnswer:", " Happy heart syndrome"], ["Question: Acute binge drinking is associated with?\n\nChoices:\nA. Happy heart syndrome\nB. Home heart syndrome\nC. Beach heart syndrome\nD. Holiday heart syndrome\nAnswer:", " Home heart syndrome"], ["Question: Acute binge drinking is associated with?\n\nChoices:\nA. Happy heart syndrome\nB. Home heart syndrome\nC. Beach heart syndrome\nD. Holiday heart syndrome\nAnswer:", " Beach heart syndrome"], ["Question: Acute binge drinking is associated with?\n\nChoices:\nA. Happy heart syndrome\nB. Home heart syndrome\nC. Beach heart syndrome\nD. Holiday heart syndrome\nAnswer:", " Holiday heart syndrome"], ["Question: The thermic effect of food\n\nChoices:\nA. is substantially higher for carbohydrate than for protein\nB. is accompanied by a slight decrease in body core temperature.\nC. is partly related to sympathetic activity stimulation in the postprandial phase\nD. is not attenuated by food malabsorption.\nAnswer:", " is substantially higher for carbohydrate than for protein"], ["Question: The thermic effect of food\n\nChoices:\nA. is substantially higher for carbohydrate than for protein\nB. is accompanied by a slight decrease in body core temperature.\nC. is partly related to sympathetic activity stimulation in the postprandial phase\nD. is not attenuated by food malabsorption.\nAnswer:", " is accompanied by a slight decrease in body core temperature."], ["Question: The thermic effect of food\n\nChoices:\nA. is substantially higher for carbohydrate than for protein\nB. is accompanied by a slight decrease in body core temperature.\nC. is partly related to sympathetic activity stimulation in the postprandial phase\nD. is not attenuated by food malabsorption.\nAnswer:", " is partly related to sympathetic activity stimulation in the postprandial phase"], ["Question: The thermic effect of food\n\nChoices:\nA. is substantially higher for carbohydrate than for protein\nB. is accompanied by a slight decrease in body core temperature.\nC. is partly related to sympathetic activity stimulation in the postprandial phase\nD. is not attenuated by food malabsorption.\nAnswer:", " is not attenuated by food malabsorption."], ["Question: Which of the factors below is NOT considered a risk factor for Eating Disorders\n\nChoices:\nA. Perfectionism traits\nB. Female gender\nC. Dieting during adolescence\nD. Parental anxiety\nAnswer:", " Perfectionism traits"], ["Question: Which of the factors below is NOT considered a risk factor for Eating Disorders\n\nChoices:\nA. Perfectionism traits\nB. Female gender\nC. Dieting during adolescence\nD. Parental anxiety\nAnswer:", " Female gender"], ["Question: Which of the factors below is NOT considered a risk factor for Eating Disorders\n\nChoices:\nA. Perfectionism traits\nB. Female gender\nC. Dieting during adolescence\nD. Parental anxiety\nAnswer:", " Dieting during adolescence"], ["Question: Which of the factors below is NOT considered a risk factor for Eating Disorders\n\nChoices:\nA. Perfectionism traits\nB. Female gender\nC. Dieting during adolescence\nD. Parental anxiety\nAnswer:", " Parental anxiety"], ["Question: Higher dietary sodium (salt) intake is generally associated with:\n\nChoices:\nA. Decreased calcium excretion in the urine\nB. Increased risk of fracture\nC. Decreased dietary calcium absorption\nD. Negative calcium balance and bone mineral loss\nAnswer:", " Decreased calcium excretion in the urine"], ["Question: Higher dietary sodium (salt) intake is generally associated with:\n\nChoices:\nA. Decreased calcium excretion in the urine\nB. Increased risk of fracture\nC. Decreased dietary calcium absorption\nD. Negative calcium balance and bone mineral loss\nAnswer:", " Increased risk of fracture"], ["Question: Higher dietary sodium (salt) intake is generally associated with:\n\nChoices:\nA. Decreased calcium excretion in the urine\nB. Increased risk of fracture\nC. Decreased dietary calcium absorption\nD. Negative calcium balance and bone mineral loss\nAnswer:", " Decreased dietary calcium absorption"], ["Question: Higher dietary sodium (salt) intake is generally associated with:\n\nChoices:\nA. Decreased calcium excretion in the urine\nB. Increased risk of fracture\nC. Decreased dietary calcium absorption\nD. Negative calcium balance and bone mineral loss\nAnswer:", " Negative calcium balance and bone mineral loss"], ["Question: Which of the following is the main nitrogenous compound in urine?\n\nChoices:\nA. Uric acid\nB. Ammonia\nC. Urea\nD. Creatinine\nAnswer:", " Uric acid"], ["Question: Which of the following is the main nitrogenous compound in urine?\n\nChoices:\nA. Uric acid\nB. Ammonia\nC. Urea\nD. Creatinine\nAnswer:", " Ammonia"], ["Question: Which of the following is the main nitrogenous compound in urine?\n\nChoices:\nA. Uric acid\nB. Ammonia\nC. Urea\nD. Creatinine\nAnswer:", " Urea"], ["Question: Which of the following is the main nitrogenous compound in urine?\n\nChoices:\nA. Uric acid\nB. Ammonia\nC. Urea\nD. Creatinine\nAnswer:", " Creatinine"], ["Question: Risk management for chemical contaminants of food generally relies on:\n\nChoices:\nA. Banning of the contaminant from food\nB. Allowing levels of contamination based on what manufacturers believe is achievable\nC. Banning any food found to contain detectable amounts of the contaminants\nD. Prohibiting the introduction into commerce food containing contaminants at levels greater than a specified level (a risk-based MRL or tolerance)\nAnswer:", " Banning of the contaminant from food"], ["Question: Risk management for chemical contaminants of food generally relies on:\n\nChoices:\nA. Banning of the contaminant from food\nB. Allowing levels of contamination based on what manufacturers believe is achievable\nC. Banning any food found to contain detectable amounts of the contaminants\nD. Prohibiting the introduction into commerce food containing contaminants at levels greater than a specified level (a risk-based MRL or tolerance)\nAnswer:", " Allowing levels of contamination based on what manufacturers believe is achievable"], ["Question: Risk management for chemical contaminants of food generally relies on:\n\nChoices:\nA. Banning of the contaminant from food\nB. Allowing levels of contamination based on what manufacturers believe is achievable\nC. Banning any food found to contain detectable amounts of the contaminants\nD. Prohibiting the introduction into commerce food containing contaminants at levels greater than a specified level (a risk-based MRL or tolerance)\nAnswer:", " Banning any food found to contain detectable amounts of the contaminants"], ["Question: Risk management for chemical contaminants of food generally relies on:\n\nChoices:\nA. Banning of the contaminant from food\nB. Allowing levels of contamination based on what manufacturers believe is achievable\nC. Banning any food found to contain detectable amounts of the contaminants\nD. Prohibiting the introduction into commerce food containing contaminants at levels greater than a specified level (a risk-based MRL or tolerance)\nAnswer:", " Prohibiting the introduction into commerce food containing contaminants at levels greater than a specified level (a risk-based MRL or tolerance)"], ["Question: Which of the following is not a non-coding RNA?\n\nChoices:\nA. mRNA\nB. tRNA\nC. rRNA\nD. miRNA\nAnswer:", " mRNA"], ["Question: Which of the following is not a non-coding RNA?\n\nChoices:\nA. mRNA\nB. tRNA\nC. rRNA\nD. miRNA\nAnswer:", " tRNA"], ["Question: Which of the following is not a non-coding RNA?\n\nChoices:\nA. mRNA\nB. tRNA\nC. rRNA\nD. miRNA\nAnswer:", " rRNA"], ["Question: Which of the following is not a non-coding RNA?\n\nChoices:\nA. mRNA\nB. tRNA\nC. rRNA\nD. miRNA\nAnswer:", " miRNA"]]
\ No newline at end of file

tests/testdata/hendrycksTest-nutrition-v0-res.json

+{"results": {"hendrycksTest-nutrition": {"acc": 0.2, "acc_stderr": 0.13333333333333333, "acc_norm": 0.4, "acc_norm_stderr": 0.1632993161855452}}, "versions": {"hendrycksTest-nutrition": 0}}
\ No newline at end of file

tests/testdata/hendrycksTest-philosophy-v0-loglikelihood.json

+[["Question: Anscombe criticizes as absurd Kant\u2019s idea of:\nChoices:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:", " the thing in itself."], ["Question: Anscombe criticizes as absurd Kant\u2019s idea of:\nChoices:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:", " the categorical imperative."], ["Question: Anscombe criticizes as absurd Kant\u2019s idea of:\nChoices:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:", " the phenomenal self."], ["Question: Anscombe criticizes as absurd Kant\u2019s idea of:\nChoices:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:", " legislating for oneself."], ["Question: According to Gauthier, the basis of morality is:\nChoices:\nA. maximizing the utility of all sentient beings.\nB. God\u2019s commands.\nC. the agreement of rational persons choosing the terms of their interaction.\nD. the purposive order of nature.\nAnswer:", " maximizing the utility of all sentient beings."], ["Question: According to Gauthier, the basis of morality is:\nChoices:\nA. maximizing the utility of all sentient beings.\nB. God\u2019s commands.\nC. the agreement of rational persons choosing the terms of their interaction.\nD. the purposive order of nature.\nAnswer:", " God\u2019s commands."], ["Question: According to Gauthier, the basis of morality is:\nChoices:\nA. maximizing the utility of all sentient beings.\nB. God\u2019s commands.\nC. the agreement of rational persons choosing the terms of their interaction.\nD. the purposive order of nature.\nAnswer:", " the agreement of rational persons choosing the terms of their interaction."], ["Question: According to Gauthier, the basis of morality is:\nChoices:\nA. maximizing the utility of all sentient beings.\nB. God\u2019s commands.\nC. the agreement of rational persons choosing the terms of their interaction.\nD. the purposive order of nature.\nAnswer:", " the purposive order of nature."], ["Question: Socrates tells Crito that he should attempt to break out of prison if and only if doing so would be:\nChoices:\nA. to his advantage.\nB. harmful to his enemies and advantageous to his friends.\nC. pleasing to the gods.\nD. just.\nAnswer:", " to his advantage."], ["Question: Socrates tells Crito that he should attempt to break out of prison if and only if doing so would be:\nChoices:\nA. to his advantage.\nB. harmful to his enemies and advantageous to his friends.\nC. pleasing to the gods.\nD. just.\nAnswer:", " harmful to his enemies and advantageous to his friends."], ["Question: Socrates tells Crito that he should attempt to break out of prison if and only if doing so would be:\nChoices:\nA. to his advantage.\nB. harmful to his enemies and advantageous to his friends.\nC. pleasing to the gods.\nD. just.\nAnswer:", " pleasing to the gods."], ["Question: Socrates tells Crito that he should attempt to break out of prison if and only if doing so would be:\nChoices:\nA. to his advantage.\nB. harmful to his enemies and advantageous to his friends.\nC. pleasing to the gods.\nD. just.\nAnswer:", " just."], ["Question: Stevenson\u2019s primary aim in this paper is to:\nChoices:\nA. provide an account of what makes right actions right.\nB. establish which things are good in themselves.\nC. develop a theory of good moral character.\nD. make ethical questions clear.\nAnswer:", " provide an account of what makes right actions right."], ["Question: Stevenson\u2019s primary aim in this paper is to:\nChoices:\nA. provide an account of what makes right actions right.\nB. establish which things are good in themselves.\nC. develop a theory of good moral character.\nD. make ethical questions clear.\nAnswer:", " establish which things are good in themselves."], ["Question: Stevenson\u2019s primary aim in this paper is to:\nChoices:\nA. provide an account of what makes right actions right.\nB. establish which things are good in themselves.\nC. develop a theory of good moral character.\nD. make ethical questions clear.\nAnswer:", " develop a theory of good moral character."], ["Question: Stevenson\u2019s primary aim in this paper is to:\nChoices:\nA. provide an account of what makes right actions right.\nB. establish which things are good in themselves.\nC. develop a theory of good moral character.\nD. make ethical questions clear.\nAnswer:", " make ethical questions clear."], ["Question: According to Feinberg, a good moral education:\nChoices:\nA. will make no use of pleasure or pain as sanctions.\nB. will be based entirely on the use of pleasure and pain as sanctions.\nC. will produce someone who does the right thing out of deference to authority.\nD. will produce someone who does the right thing simply because it is right.\nAnswer:", " will make no use of pleasure or pain as sanctions."], ["Question: According to Feinberg, a good moral education:\nChoices:\nA. will make no use of pleasure or pain as sanctions.\nB. will be based entirely on the use of pleasure and pain as sanctions.\nC. will produce someone who does the right thing out of deference to authority.\nD. will produce someone who does the right thing simply because it is right.\nAnswer:", " will be based entirely on the use of pleasure and pain as sanctions."], ["Question: According to Feinberg, a good moral education:\nChoices:\nA. will make no use of pleasure or pain as sanctions.\nB. will be based entirely on the use of pleasure and pain as sanctions.\nC. will produce someone who does the right thing out of deference to authority.\nD. will produce someone who does the right thing simply because it is right.\nAnswer:", " will produce someone who does the right thing out of deference to authority."], ["Question: According to Feinberg, a good moral education:\nChoices:\nA. will make no use of pleasure or pain as sanctions.\nB. will be based entirely on the use of pleasure and pain as sanctions.\nC. will produce someone who does the right thing out of deference to authority.\nD. will produce someone who does the right thing simply because it is right.\nAnswer:", " will produce someone who does the right thing simply because it is right."], ["Question: According to Sartre, the first principle of existentialism is that _____.\nChoices:\nA. God is dead\nB. man is all-powerful\nC. man is nothing else but what he makes of himself\nD. man is nothing\nAnswer:", " God is dead"], ["Question: According to Sartre, the first principle of existentialism is that _____.\nChoices:\nA. God is dead\nB. man is all-powerful\nC. man is nothing else but what he makes of himself\nD. man is nothing\nAnswer:", " man is all-powerful"], ["Question: According to Sartre, the first principle of existentialism is that _____.\nChoices:\nA. God is dead\nB. man is all-powerful\nC. man is nothing else but what he makes of himself\nD. man is nothing\nAnswer:", " man is nothing else but what he makes of himself"], ["Question: According to Sartre, the first principle of existentialism is that _____.\nChoices:\nA. God is dead\nB. man is all-powerful\nC. man is nothing else but what he makes of himself\nD. man is nothing\nAnswer:", " man is nothing"], ["Question: By \u201canimal motion,\u201d Hobbes means:\nChoices:\nA. involuntary operations such as heartbeat and breathing.\nB. instinctive behavior, such as nursing young.\nC. irrational behavior.\nD. all voluntary behavior.\nAnswer:", " involuntary operations such as heartbeat and breathing."], ["Question: By \u201canimal motion,\u201d Hobbes means:\nChoices:\nA. involuntary operations such as heartbeat and breathing.\nB. instinctive behavior, such as nursing young.\nC. irrational behavior.\nD. all voluntary behavior.\nAnswer:", " instinctive behavior, such as nursing young."], ["Question: By \u201canimal motion,\u201d Hobbes means:\nChoices:\nA. involuntary operations such as heartbeat and breathing.\nB. instinctive behavior, such as nursing young.\nC. irrational behavior.\nD. all voluntary behavior.\nAnswer:", " irrational behavior."], ["Question: By \u201canimal motion,\u201d Hobbes means:\nChoices:\nA. involuntary operations such as heartbeat and breathing.\nB. instinctive behavior, such as nursing young.\nC. irrational behavior.\nD. all voluntary behavior.\nAnswer:", " all voluntary behavior."], ["Question: Anscombe claims that on Sidgwick\u2019s view, the badness of an action must be estimated in light of:\nChoices:\nA. its actual consequences.\nB. its expected consequences.\nC. whether it violates any duties.\nD. whether it violates divine law.\nAnswer:", " its actual consequences."], ["Question: Anscombe claims that on Sidgwick\u2019s view, the badness of an action must be estimated in light of:\nChoices:\nA. its actual consequences.\nB. its expected consequences.\nC. whether it violates any duties.\nD. whether it violates divine law.\nAnswer:", " its expected consequences."], ["Question: Anscombe claims that on Sidgwick\u2019s view, the badness of an action must be estimated in light of:\nChoices:\nA. its actual consequences.\nB. its expected consequences.\nC. whether it violates any duties.\nD. whether it violates divine law.\nAnswer:", " whether it violates any duties."], ["Question: Anscombe claims that on Sidgwick\u2019s view, the badness of an action must be estimated in light of:\nChoices:\nA. its actual consequences.\nB. its expected consequences.\nC. whether it violates any duties.\nD. whether it violates divine law.\nAnswer:", " whether it violates divine law."], ["Question: According to Parfit, the obligation to give priority to the welfare of one\u2019s children is:\nChoices:\nA. agent-relative.\nB. agent-neutral.\nC. absolute.\nD. none of the above.\nAnswer:", " agent-relative."], ["Question: According to Parfit, the obligation to give priority to the welfare of one\u2019s children is:\nChoices:\nA. agent-relative.\nB. agent-neutral.\nC. absolute.\nD. none of the above.\nAnswer:", " agent-neutral."], ["Question: According to Parfit, the obligation to give priority to the welfare of one\u2019s children is:\nChoices:\nA. agent-relative.\nB. agent-neutral.\nC. absolute.\nD. none of the above.\nAnswer:", " absolute."], ["Question: According to Parfit, the obligation to give priority to the welfare of one\u2019s children is:\nChoices:\nA. agent-relative.\nB. agent-neutral.\nC. absolute.\nD. none of the above.\nAnswer:", " none of the above."], ["Question: According to Moore, \u201cpleasure is good\u201d means the same thing as:\nChoices:\nA. \u201cpleasure is desired.\u201d\nB. \u201cpleasure is pleasant.\u201d\nC. \u201cwe desire to desire pleasure.\u201d\nD. none of the above.\nAnswer:", " \u201cpleasure is desired.\u201d"], ["Question: According to Moore, \u201cpleasure is good\u201d means the same thing as:\nChoices:\nA. \u201cpleasure is desired.\u201d\nB. \u201cpleasure is pleasant.\u201d\nC. \u201cwe desire to desire pleasure.\u201d\nD. none of the above.\nAnswer:", " \u201cpleasure is pleasant.\u201d"], ["Question: According to Moore, \u201cpleasure is good\u201d means the same thing as:\nChoices:\nA. \u201cpleasure is desired.\u201d\nB. \u201cpleasure is pleasant.\u201d\nC. \u201cwe desire to desire pleasure.\u201d\nD. none of the above.\nAnswer:", " \u201cwe desire to desire pleasure.\u201d"], ["Question: According to Moore, \u201cpleasure is good\u201d means the same thing as:\nChoices:\nA. \u201cpleasure is desired.\u201d\nB. \u201cpleasure is pleasant.\u201d\nC. \u201cwe desire to desire pleasure.\u201d\nD. none of the above.\nAnswer:", " none of the above."]]
\ No newline at end of file

tests/testdata/hendrycksTest-philosophy-v0-res.json

+{"results": {"hendrycksTest-philosophy": {"acc": 0.3, "acc_stderr": 0.15275252316519464, "acc_norm": 0.4, "acc_norm_stderr": 0.1632993161855452}}, "versions": {"hendrycksTest-philosophy": 0}}
\ No newline at end of file

tests/testdata/hendrycksTest-prehistory-v0-loglikelihood.json

+[["Question: Archaeological evidence indicates that cattle were first domesticated where and how long ago?\nChoices:\nA. in western Europe, about 3,500 years ago\nB. in sub-Saharan Africa, about 8,500 years ago\nC. in North America, about 9,500 years ago\nD. in the Middle East, about 10,500 years ago\nAnswer:", " in western Europe, about 3,500 years ago"], ["Question: Archaeological evidence indicates that cattle were first domesticated where and how long ago?\nChoices:\nA. in western Europe, about 3,500 years ago\nB. in sub-Saharan Africa, about 8,500 years ago\nC. in North America, about 9,500 years ago\nD. in the Middle East, about 10,500 years ago\nAnswer:", " in sub-Saharan Africa, about 8,500 years ago"], ["Question: Archaeological evidence indicates that cattle were first domesticated where and how long ago?\nChoices:\nA. in western Europe, about 3,500 years ago\nB. in sub-Saharan Africa, about 8,500 years ago\nC. in North America, about 9,500 years ago\nD. in the Middle East, about 10,500 years ago\nAnswer:", " in North America, about 9,500 years ago"], ["Question: Archaeological evidence indicates that cattle were first domesticated where and how long ago?\nChoices:\nA. in western Europe, about 3,500 years ago\nB. in sub-Saharan Africa, about 8,500 years ago\nC. in North America, about 9,500 years ago\nD. in the Middle East, about 10,500 years ago\nAnswer:", " in the Middle East, about 10,500 years ago"], ["Question: Which tool technology is associated with anatomically modern Homo sapiens?\nChoices:\nA. Aurignacian\nB. Acheulean\nC. Mousterian\nD. both b and c\nAnswer:", " Aurignacian"], ["Question: Which tool technology is associated with anatomically modern Homo sapiens?\nChoices:\nA. Aurignacian\nB. Acheulean\nC. Mousterian\nD. both b and c\nAnswer:", " Acheulean"], ["Question: Which tool technology is associated with anatomically modern Homo sapiens?\nChoices:\nA. Aurignacian\nB. Acheulean\nC. Mousterian\nD. both b and c\nAnswer:", " Mousterian"], ["Question: Which tool technology is associated with anatomically modern Homo sapiens?\nChoices:\nA. Aurignacian\nB. Acheulean\nC. Mousterian\nD. both b and c\nAnswer:", " both b and c"], ["Question: Which best describes Stonehenge?\nChoices:\nA. Megalithic monument\nB. Halafian monumental center\nC. Chavin ritual monument\nD. Mesopotamian monument\nAnswer:", " Megalithic monument"], ["Question: Which best describes Stonehenge?\nChoices:\nA. Megalithic monument\nB. Halafian monumental center\nC. Chavin ritual monument\nD. Mesopotamian monument\nAnswer:", " Halafian monumental center"], ["Question: Which best describes Stonehenge?\nChoices:\nA. Megalithic monument\nB. Halafian monumental center\nC. Chavin ritual monument\nD. Mesopotamian monument\nAnswer:", " Chavin ritual monument"], ["Question: Which best describes Stonehenge?\nChoices:\nA. Megalithic monument\nB. Halafian monumental center\nC. Chavin ritual monument\nD. Mesopotamian monument\nAnswer:", " Mesopotamian monument"], ["Question: Machu Picchu was built in the mountains as a:\nChoices:\nA. defensive stronghold.\nB. shrine to the Inca rain god.\nC. relaxing getaway for elites.\nD. astronomical observatory.\nAnswer:", " defensive stronghold."], ["Question: Machu Picchu was built in the mountains as a:\nChoices:\nA. defensive stronghold.\nB. shrine to the Inca rain god.\nC. relaxing getaway for elites.\nD. astronomical observatory.\nAnswer:", " shrine to the Inca rain god."], ["Question: Machu Picchu was built in the mountains as a:\nChoices:\nA. defensive stronghold.\nB. shrine to the Inca rain god.\nC. relaxing getaway for elites.\nD. astronomical observatory.\nAnswer:", " relaxing getaway for elites."], ["Question: Machu Picchu was built in the mountains as a:\nChoices:\nA. defensive stronghold.\nB. shrine to the Inca rain god.\nC. relaxing getaway for elites.\nD. astronomical observatory.\nAnswer:", " astronomical observatory."], ["Question: The origins of Chinese civilization can be traced to:\nChoices:\nA. the site of An-yang near the Huang ho River.\nB. chiefdoms and states in numerous regions throughout China.\nC. the beginning of the Qin Dynasty.\nD. external influences originating from the Indus Valley.\nAnswer:", " the site of An-yang near the Huang ho River."], ["Question: The origins of Chinese civilization can be traced to:\nChoices:\nA. the site of An-yang near the Huang ho River.\nB. chiefdoms and states in numerous regions throughout China.\nC. the beginning of the Qin Dynasty.\nD. external influences originating from the Indus Valley.\nAnswer:", " chiefdoms and states in numerous regions throughout China."], ["Question: The origins of Chinese civilization can be traced to:\nChoices:\nA. the site of An-yang near the Huang ho River.\nB. chiefdoms and states in numerous regions throughout China.\nC. the beginning of the Qin Dynasty.\nD. external influences originating from the Indus Valley.\nAnswer:", " the beginning of the Qin Dynasty."], ["Question: The origins of Chinese civilization can be traced to:\nChoices:\nA. the site of An-yang near the Huang ho River.\nB. chiefdoms and states in numerous regions throughout China.\nC. the beginning of the Qin Dynasty.\nD. external influences originating from the Indus Valley.\nAnswer:", " external influences originating from the Indus Valley."], ["Question: The attitude of many ancient elites toward the consumption of resources seems to have been:\nChoices:\nA. \u201ca penny saved is a penny earned.\"\nB. \"if you've got it, flaunt it.\"\nC. \"look before you leap.\"\nD. \"the last shall be first and the first shall be last.\"\nAnswer:", " \u201ca penny saved is a penny earned.\""], ["Question: The attitude of many ancient elites toward the consumption of resources seems to have been:\nChoices:\nA. \u201ca penny saved is a penny earned.\"\nB. \"if you've got it, flaunt it.\"\nC. \"look before you leap.\"\nD. \"the last shall be first and the first shall be last.\"\nAnswer:", " \"if you've got it, flaunt it.\""], ["Question: The attitude of many ancient elites toward the consumption of resources seems to have been:\nChoices:\nA. \u201ca penny saved is a penny earned.\"\nB. \"if you've got it, flaunt it.\"\nC. \"look before you leap.\"\nD. \"the last shall be first and the first shall be last.\"\nAnswer:", " \"look before you leap.\""], ["Question: The attitude of many ancient elites toward the consumption of resources seems to have been:\nChoices:\nA. \u201ca penny saved is a penny earned.\"\nB. \"if you've got it, flaunt it.\"\nC. \"look before you leap.\"\nD. \"the last shall be first and the first shall be last.\"\nAnswer:", " \"the last shall be first and the first shall be last.\""], ["Question: The construction of large-scale features, such as mounds and shell middens, is often taken by archaeologists as evidence of:\nChoices:\nA. the practice of slavery.\nB. social and political complexity.\nC. a Mesolithic tradition.\nD. the shift from Paleolithic to Neolithic.\nAnswer:", " the practice of slavery."], ["Question: The construction of large-scale features, such as mounds and shell middens, is often taken by archaeologists as evidence of:\nChoices:\nA. the practice of slavery.\nB. social and political complexity.\nC. a Mesolithic tradition.\nD. the shift from Paleolithic to Neolithic.\nAnswer:", " social and political complexity."], ["Question: The construction of large-scale features, such as mounds and shell middens, is often taken by archaeologists as evidence of:\nChoices:\nA. the practice of slavery.\nB. social and political complexity.\nC. a Mesolithic tradition.\nD. the shift from Paleolithic to Neolithic.\nAnswer:", " a Mesolithic tradition."], ["Question: The construction of large-scale features, such as mounds and shell middens, is often taken by archaeologists as evidence of:\nChoices:\nA. the practice of slavery.\nB. social and political complexity.\nC. a Mesolithic tradition.\nD. the shift from Paleolithic to Neolithic.\nAnswer:", " the shift from Paleolithic to Neolithic."], ["Question: The sophisticated stone tool making industry at the Pinnacle Point site in South Africa is evidence of:\nChoices:\nA. a 250,000-year-old tradition of hand axe production and use.\nB. a 71,000-year-old complex sequence of microblade production and use.\nC. an adaptation to hunting in colder environments.\nD. both a and c.\nAnswer:", " a 250,000-year-old tradition of hand axe production and use."], ["Question: The sophisticated stone tool making industry at the Pinnacle Point site in South Africa is evidence of:\nChoices:\nA. a 250,000-year-old tradition of hand axe production and use.\nB. a 71,000-year-old complex sequence of microblade production and use.\nC. an adaptation to hunting in colder environments.\nD. both a and c.\nAnswer:", " a 71,000-year-old complex sequence of microblade production and use."], ["Question: The sophisticated stone tool making industry at the Pinnacle Point site in South Africa is evidence of:\nChoices:\nA. a 250,000-year-old tradition of hand axe production and use.\nB. a 71,000-year-old complex sequence of microblade production and use.\nC. an adaptation to hunting in colder environments.\nD. both a and c.\nAnswer:", " an adaptation to hunting in colder environments."], ["Question: The sophisticated stone tool making industry at the Pinnacle Point site in South Africa is evidence of:\nChoices:\nA. a 250,000-year-old tradition of hand axe production and use.\nB. a 71,000-year-old complex sequence of microblade production and use.\nC. an adaptation to hunting in colder environments.\nD. both a and c.\nAnswer:", " both a and c."], ["Question: Interestingly, none of the civilizations of ancient South America appear to have developed:\nChoices:\nA. monumental works.\nB. irrigation technology.\nC. a written language.\nD. significant armies.\nAnswer:", " monumental works."], ["Question: Interestingly, none of the civilizations of ancient South America appear to have developed:\nChoices:\nA. monumental works.\nB. irrigation technology.\nC. a written language.\nD. significant armies.\nAnswer:", " irrigation technology."], ["Question: Interestingly, none of the civilizations of ancient South America appear to have developed:\nChoices:\nA. monumental works.\nB. irrigation technology.\nC. a written language.\nD. significant armies.\nAnswer:", " a written language."], ["Question: Interestingly, none of the civilizations of ancient South America appear to have developed:\nChoices:\nA. monumental works.\nB. irrigation technology.\nC. a written language.\nD. significant armies.\nAnswer:", " significant armies."], ["Question: That modern Africans do not possess any Neandertal DNA indicates that:\nChoices:\nA. Gene flow between Neandertals and anatomically modern humans must have occurred before Neandertals migrated out of Africa.\nB. Gene flow between Neandertals and anatomically modern humans occurred after anatomically modern humans migrated out of Africa.\nC. Neandertals who lived in Africa did not interbreed with anatomically modern humans, but must have followed them when they migrated out of Africa.\nD. Interbreeding between anatomically modern humans and Neandertals never occurred.\nAnswer:", " Gene flow between Neandertals and anatomically modern humans must have occurred before Neandertals migrated out of Africa."], ["Question: That modern Africans do not possess any Neandertal DNA indicates that:\nChoices:\nA. Gene flow between Neandertals and anatomically modern humans must have occurred before Neandertals migrated out of Africa.\nB. Gene flow between Neandertals and anatomically modern humans occurred after anatomically modern humans migrated out of Africa.\nC. Neandertals who lived in Africa did not interbreed with anatomically modern humans, but must have followed them when they migrated out of Africa.\nD. Interbreeding between anatomically modern humans and Neandertals never occurred.\nAnswer:", " Gene flow between Neandertals and anatomically modern humans occurred after anatomically modern humans migrated out of Africa."], ["Question: That modern Africans do not possess any Neandertal DNA indicates that:\nChoices:\nA. Gene flow between Neandertals and anatomically modern humans must have occurred before Neandertals migrated out of Africa.\nB. Gene flow between Neandertals and anatomically modern humans occurred after anatomically modern humans migrated out of Africa.\nC. Neandertals who lived in Africa did not interbreed with anatomically modern humans, but must have followed them when they migrated out of Africa.\nD. Interbreeding between anatomically modern humans and Neandertals never occurred.\nAnswer:", " Neandertals who lived in Africa did not interbreed with anatomically modern humans, but must have followed them when they migrated out of Africa."], ["Question: That modern Africans do not possess any Neandertal DNA indicates that:\nChoices:\nA. Gene flow between Neandertals and anatomically modern humans must have occurred before Neandertals migrated out of Africa.\nB. Gene flow between Neandertals and anatomically modern humans occurred after anatomically modern humans migrated out of Africa.\nC. Neandertals who lived in Africa did not interbreed with anatomically modern humans, but must have followed them when they migrated out of Africa.\nD. Interbreeding between anatomically modern humans and Neandertals never occurred.\nAnswer:", " Interbreeding between anatomically modern humans and Neandertals never occurred."]]
\ No newline at end of file

tests/testdata/hendrycksTest-prehistory-v0-res.json

+{"results": {"hendrycksTest-prehistory": {"acc": 0.2, "acc_stderr": 0.13333333333333333, "acc_norm": 0.3, "acc_norm_stderr": 0.15275252316519464}}, "versions": {"hendrycksTest-prehistory": 0}}
\ No newline at end of file