git init

calibration/BraTS/README.md

+# Calibration set selected from BraTS 2019 dataset
+
+These are 40 images randomly selected from [Fold 0](../../v0.7/medical_imaging/3d-unet/folds/fold0_validation.txt), [Fold 2](../../v0.7/medical_imaging/3d-unet/folds/fold2_validation.txt), [Fold 3](../../v0.7/medical_imaging/3d-unet/folds/fold3_validation.txt), and [Fold 4](../../v0.7/medical_imaging/3d-unet/folds/fold4_validation.txt) of [BraTS 2019](https://www.med.upenn.edu/cbica/brats2019/data.html) Training Dataset.

calibration/BraTS/brats_cal_images_list.py

+import numpy as np
+
+np.random.seed(0)
+images = []
+for i in [0, 2, 3, 4]:
+    with open(
+        "../../v0.7/medical_imaging/3d-unet/folds/fold{:d}_validation.txt".format(
+            i)
+    ) as f:
+        for line in f:
+            images.append(line.rstrip())
+indices = np.random.permutation(len(images))[:40]
+selected = sorted([images[idx] for idx in indices])
+with open("brats_cal_images_list.txt", "w") as f:
+    for img in selected:
+        print(img, file=f)

calibration/BraTS/brats_cal_images_list.txt

+HGG__BraTS19_2013_18_1
+HGG__BraTS19_2013_20_1
+HGG__BraTS19_CBICA_AAP_1
+HGG__BraTS19_CBICA_ABN_1
+HGG__BraTS19_CBICA_ABO_1
+HGG__BraTS19_CBICA_ALU_1
+HGG__BraTS19_CBICA_ANZ_1
+HGG__BraTS19_CBICA_APY_1
+HGG__BraTS19_CBICA_AQJ_1
+HGG__BraTS19_CBICA_AQZ_1
+HGG__BraTS19_CBICA_ASN_1
+HGG__BraTS19_CBICA_ASY_1
+HGG__BraTS19_CBICA_AUW_1
+HGG__BraTS19_CBICA_AXJ_1
+HGG__BraTS19_CBICA_AXM_1
+HGG__BraTS19_CBICA_AYG_1
+HGG__BraTS19_CBICA_AYU_1
+HGG__BraTS19_CBICA_AZD_1
+HGG__BraTS19_CBICA_BAX_1
+HGG__BraTS19_CBICA_BGR_1
+HGG__BraTS19_CBICA_BHV_1
+HGG__BraTS19_TCIA01_235_1
+HGG__BraTS19_TCIA02_394_1
+HGG__BraTS19_TCIA02_473_1
+HGG__BraTS19_TCIA02_606_1
+HGG__BraTS19_TCIA03_419_1
+HGG__BraTS19_TCIA04_192_1
+HGG__BraTS19_TCIA04_479_1
+HGG__BraTS19_TCIA06_372_1
+HGG__BraTS19_TCIA08_278_1
+LGG__BraTS19_2013_28_1
+LGG__BraTS19_TCIA09_462_1
+LGG__BraTS19_TCIA10_130_1
+LGG__BraTS19_TCIA10_202_1
+LGG__BraTS19_TCIA10_346_1
+LGG__BraTS19_TCIA10_387_1
+LGG__BraTS19_TCIA10_628_1
+LGG__BraTS19_TCIA12_470_1
+LGG__BraTS19_TCIA13_621_1
+LGG__BraTS19_TCIA13_653_1

calibration/CriteoTerabyte/README.md

+# Calibration set selected from Criteo Terabyte dataset
+
+For MLPerf Inference, we use the first 128000 rows (user-item pairs) of the second half of `day_23` as the calibration set. Specifically, `day_23` contains 178274637 rows in total, so we use the rows **from the 89137319-th row to the 89265318-th row (both inclusive) in `day_23`** as the calibration set (assuming 0-based indexing).
+
+Please refer to the [DLRM reference README.md](../../v0.5/recommendation/README.md) for more information.

calibration/ImageNet/README.md

+Please only use at most 1 calibration file from this folder for calibration.

calibration/KiTS/README.md

+# Calibration set selected from KiTS19 dataset
+
+These are 20 images randomly selected from [KiTS19](https://github.com/neheller/kits19) Training Dataset used in [MLPerf-Training 3D-UNet](https://github.com/mlcommons/training/blob/master/image_segmentation/pytorch).

calibration/KiTS/calibration_cases.txt

+case_00050
+case_00117
+case_00172
+case_00149
+case_00091
+case_00147
+case_00144
+case_00100
+case_00090
+case_00196
+case_00183
+case_00130
+case_00136
+case_00047
+case_00073
+case_00141
+case_00053
+case_00148
+case_00081
+case_00150

calibration/LibriSpeech/README.md

+# Calibration set for RNN-T benchmark
+
+These are 500 sequences randomly selected from the **train-clean-100** set in [LibriSpeech](http://www.openslr.org/12/) dataset and to be used as the reference calibration set for MLPerf Inference RNN-T benchmark. 
+
+## Criteria
+
+1. The calibration set and the test set should not have the same speakers.
+2. The calibration set and the test set should not have the same sentences as labels.
+3. The model should get about the same WER on the calibration set and the test set.
+4. The calibration data should represent all characters in the alphabet. The distribution of characters does not need to be uniform or an unbiased estimate of the full dataset's distribution, though.
+5. The sequences in the calibration set are all below 15 seconds long.
+

calibration/SQuAD-v1.1/README.md

+The integers in bert_calibration_features.txt correspond to 100 randomly selected indices in the list of features generated from dev-v1.1.json using [convert_examples_to_features()](https://github.com/mlcommons/inference/blob/master/language/bert/create_squad_data.py#L249) with a doc_stride of 128 and a max_seq_len of 384.
+
+The values in bert_calibration_qas_ids.txt correspond to 100 randomly selected qas ids in the dev-v1.1.json file.
+
+Please only use at most 1 calibration file from this folder for calibration.
+

calibration/doc/example writeup.adoc

+TensorRT Post-Training Quantization
+-----------------------------------
+
+TensorRT post-training quantization requires a dynamic range for each weight and activation tensor. At present TensorRT quantization is symmetric for both 
+
+Weights
+~~~~~~~
+For each weight Tensor, we find the maximum absolute value t of any element of the tensor. The dynamic range is then [-t,t]. For convolution and transpose convolution operators, dynamic range values are per-channel, otherwise they are per-tensor.
+
+Activations
+~~~~~~~~~~~
+For each activation tensor, TensorRT uses a distinct dynamic range that applies across the entire tensor. We invoke the model on a set of representative inputs in FP32 precision, and create a per-tensor histogram of absolute values. The histogram initially uses 1024 equal-range bins whose range is set by the initial image batch, but dynamically resizes by doubling the number of bins as necessary to accommodate the range of subsequent batches. Call this histogram, which has a power-of-2 bins, where all data elements are guaranteed to fall into one of the bins, the “starting histogram”. 
+
+We then apply one of two methods, as chosen by the application.
+
+* Minmax: as with weights, we compute the maximum absolute value of the tensor.
+
+* Entropy: for each bin B in the starting histogram, we compute a divergence value as follows: 
+** Create a truncated histogram where each bin has the same range and count as the original, except that all elements in bins beyond B are considered to be in B, and all bins beyond B are removed.
+** Create a coarse histogram by discretizing the truncated histogram into 127 bins of equal range between 0 and the midpoint of B, placing all elements in the final bin of the truncated histogram into the final bin of the coarse histogram.
+** Compute the KL-divergence between the distributions represented by the coarse histogram and the truncated histogram.
+
+The dynamic range chosen is the center of the bin which minimizes divergence. 
+
+Additional Details
+~~~~~~~~~~~~~~~~~~
+A number of minor modifications are applied to this basic algorithm, including
+
+* discarding the first bin in the histogram (which typically contains a huge number of noise activations) immediately after it has been built
+* how empty bins are treated when computing divergence.
+
+For some operations which are not expected to change dynamic range (e.g. max-pooling, concatenation) we propagate dynamic range from the output to the input(s).