the source code of NNI for DCU

docs/zh_CN/installation.rst

+.. c62173d7147a43a13bf2cdf945b82d07
+
+############
+安装
+############
+
+当前支持在 Linux，macOS 和 Windows 下安装。 还可使用 Docker。
+
+..  toctree::
+    :maxdepth: 2
+
+    Linux 和 macOS <Tutorial/InstallationLinux>
+    Windows <Tutorial/InstallationWin>
+    使用 Docker <Tutorial/HowToUseDocker>
\ No newline at end of file

docs/zh_CN/model_compression.rst

+.. da97b4cdd507bd8fad43d640f3d2bfef
+
+#################
+模型压缩
+#################
+
+深度神经网络（DNNs）在许多领域都取得了巨大的成功。 然而，典型的神经网络是
+计算和能源密集型的，很难将其部署在计算资源匮乏
+或具有严格延迟要求的设备上。 因此，一个自然的想法就是对模型进行压缩
+以减小模型大小并加速模型训练/推断，同时不会显着降低模型性能。 模型压缩
+技术可以分为两类：剪枝和量化。 剪枝方法探索模型权重中的冗余，
+并尝试删除/修剪冗余和非关键的权重。 量化是指通过减少
+权重表示或激活所需的比特数来压缩模型。
+
+NNI 提供了易于使用的工具包来帮助用户设计并使用剪枝和量化算法。
+其使用了统一的接口来支持 TensorFlow 和 PyTorch。
+对用户来说， 只需要添加几行代码即可压缩模型。
+NNI 中也内置了一些主流的模型压缩算法。
+用户可以进一步利用 NNI 的自动调优功能找到最佳的压缩模型，
+该功能在自动模型压缩部分有详细介绍。
+另一方面，用户可以使用 NNI 的接口自定义新的压缩算法。
+
+详细信息，参考以下教程：
+
+..  toctree::
+    :maxdepth: 2
+
+    概述 <Compression/Overview>
+    快速入门 <Compression/QuickStart>
+    教程 <Compression/Tutorial>
+    剪枝 <Compression/pruning>
+    剪枝（V2版本） <Compression/v2_pruning>
+    量化 <Compression/quantization>
+    工具 <Compression/CompressionUtils>
+    高级用法 <Compression/advanced>
+    API 参考 <Compression/CompressionReference>

docs/zh_CN/nas.rst

+.. 0b36fb7844fd9cc88c4e74ad2c6b9ece
+
+##########################
+神经网络架构搜索
+##########################
+
+自动化的神经网络架构（NAS）搜索在寻找更好的模型方面发挥着越来越重要的作用。
+最近的研究工作证明了自动化 NAS 的可行性，并发现了一些超越手动调整的模型。
+代表工作有 NASNet, ENAS, DARTS, Network Morphism, 以及 Evolution 等。 此外，新的创新不断涌现。
+
+但是，要实现 NAS 算法需要花费大量的精力，并且很难在新算法中重用现有算法的代码。
+为了促进 NAS 创新 (如, 设计实现新的 NAS 模型，比较不同的 NAS 模型)，
+易于使用且灵活的编程接口非常重要。
+
+因此，NNI 设计了 `Retiarii <https://www.usenix.org/system/files/osdi20-zhang_quanlu.pdf>`__， 它是一个深度学习框架，支持在神经网络模型空间，而不是单个神经网络模型上进行探索性训练。
+Retiarii 的探索性训练允许用户以高度灵活的方式表达 *神经网络架构搜索* 和 *超参数调整* 的各种搜索空间。
+
+本文档中的一些常用术语：
+
+* *Model search space（模型搜索空间）* ：它意味着一组模型，用于从中探索/搜索出最佳模型。 有时我们简称为 *search space（搜索空间）* 或 *model space（模型空间）* 。
+* *Exploration strategy（探索策略）*：用于探索模型搜索空间的算法。
+* *Model evaluator（模型评估器）*：用于训练模型并评估模型的性能。
+
+按照以下说明开始您的 Retiarii 之旅。
+
+..  toctree::
+    :maxdepth: 2
+
+    概述 <NAS/Overview>
+    快速入门 <NAS/QuickStart>
+    构建模型空间 <NAS/construct_space>
+    Multi-trial NAS <NAS/multi_trial_nas>
+    One-Shot NAS <NAS/one_shot_nas>
+    硬件相关 NAS <NAS/HardwareAwareNAS>
+    NAS 基准测试 <NAS/Benchmarks>
+    NAS API 参考 <NAS/ApiReference>

docs/zh_CN/reference.rst

+.. 5f17887878bae5d51cf177a1c995c003
+
+参考
+==================
+
+..  toctree::
+    :maxdepth: 2
+
+    nnictl 命令 <Tutorial/Nnictl>
+    Experiment 配置 <reference/experiment_config>
+    Experiment 配置（遗产） <Tutorial/ExperimentConfig>
+    搜索空间<Tutorial/SearchSpaceSpec>
+    NNI Annotation<Tutorial/AnnotationSpec>
+    SDK API 参考 <sdk_reference>
+    支持的框架和库 <SupportedFramework_Library>
+    从 Python 发起实验 <Tutorial/HowToLaunchFromPython>
+    共享存储 <Tutorial/HowToUseSharedStorage>
+    Tensorboard <Tutorial/Tensorboard>

docs/zh_CN/reference/experiment_config.rst

+../../en_US/reference/experiment_config.rst
\ No newline at end of file

docs/zh_CN/sdk_reference.rst

+.. 60cb924d0ec522b7709acf4f8cff3f16
+
+####################
+Python API 参考
+####################
+
+
+..  toctree::
+    :maxdepth: 1
+
+    自动调优 <autotune_ref>
+    NAS <NAS/ApiReference>
+    模型压缩 <Compression/CompressionReference>
+    Python API <Tutorial/HowToLaunchFromPython>
\ No newline at end of file

docs/zh_CN/training_services.rst

+.. 4e054d96c7d211dc514c99d673415d8e
+
+NNI 支持的训练平台介绍
+=====================================
+
+..  toctree::
+    Overview <./TrainingService/Overview>
+    本机<./TrainingService/LocalMode>
+    远程<./TrainingService/RemoteMachineMode>
+    OpenPAI<./TrainingService/PaiMode>
+    Kubeflow<./TrainingService/KubeflowMode>
+    AdaptDL<./TrainingService/AdaptDLMode>
+    FrameworkController<./TrainingService/FrameworkControllerMode>
+    DLTS<./TrainingService/DLTSMode>
+    AML<./TrainingService/AMLMode>
+    PAI-DLC<./TrainingService/DLCMode>
+    混合模式 <./TrainingService/HybridMode>

examples/assessors/README.md

+# Define your own Assessor
+
+*Assessor receive intermediate result from Trial and decide whether the Trial should be killed. Once the Trial experiment meets the early stop conditions, the assessor will kill the Trial.*
+
+So, if users want to implement a customized Assessor, they only need to:
+
+
+**1) Inherit an assessor of a base Assessor class**
+```python
+from nni.assessor import Assessor
+
+class CustomizedAssessor(Assessor):
+    def __init__(self, ...):
+        ...
+```
+
+**2) Implement assess trial function**
+```python
+from nni.assessor import Assessor, AssessResult
+
+class CustomizedAssessor(Assessor):
+    def __init__(self, ...):
+        ...
+    
+    def assess_trial(self, trial_history):
+        """
+        Determines whether a trial should be killed. Must override.
+        trial_history: a list of intermediate result objects.
+        Returns AssessResult.Good or AssessResult.Bad.
+        """
+        # you code implement here.
+        ...
+```
+**3) Write a script to run Assessor**
+```python
+import argparse
+
+import CustomizedAssessor
+
+def main():
+    parser = argparse.ArgumentParser(description='parse command line parameters.')
+    # parse your assessor arg here.
+    ...
+    FLAGS, unparsed = parser.parse_known_args()
+
+    tuner = CustomizedAssessor(...)
+    tuner.run()
+
+main()
+```
+
+Please noted in 2). The object `trial_history` are exact the object that Trial send to Assessor by using SDK `report_intermediate_result` function.
+
+Also, user could override the `run` function in Assessor to control the process logic.
+
+More detail example you could see:
+> * [Base-Assessor](https://msrasrg.visualstudio.com/NeuralNetworkIntelligenceOpenSource/_git/Default?_a=contents&path=%2Fsrc%2Fsdk%2Fpynni%2Fnni%2Fassessor.py&version=GBadd_readme)

examples/assessors/README_zh_CN.md

+# 自定义 Assessor
+
+*Assessor 从 Trial 中接收中间结果，并决定此 Trial 是否应该终止。 一旦 Trial 满足提前终止条件，Assessor 将终止此 Trial。*
+
+因此，如果要自定义 Assessor，需要：
+
+**1) 继承于 Assessor 基类，创建 Assessor 类**
+
+```python
+from nni.assessor import Assessor
+
+class CustomizedAssessor(Assessor):
+    def __init__(self, ...):
+        ...
+```
+
+**2) 实现评估 Trial 的函数**
+
+```python
+from nni.assessor import Assessor, AssessResult
+
+class CustomizedAssessor(Assessor):
+    def __init__(self, ...):
+        ...
+
+    def assess_trial(self, trial_history):
+        """
+        决定是否应该终止 Trial。 必须重载。
+        trial_history: 中间结果列表对象。
+        返回 AssessResult.Good 或 AssessResult.Bad。
+        """
+        # 代码实现于此处。
+        ...
+```
+
+**3) 实现脚本来运行 Assessor**
+
+```python
+import argparse
+
+import CustomizedAssessor
+
+def main():
+    parser = argparse.ArgumentParser(description='parse command line parameters.')
+    # 在这里解析 Assessor 的参数。
+    ...
+    FLAGS, unparsed = parser.parse_known_args()
+
+    tuner = CustomizedAssessor(...)
+    tuner.run()
+
+main()
+```
+
+注意 2) 中， 对象 `trial_history` 和 `report_intermediate_result` 函数返回给 Assessor 的完全一致。
+
+也可以重载 Assessor 的 `run` 函数来控制过程逻辑。
+
+更多示例，可参考：
+
+> - [Base-Assessor](https://msrasrg.visualstudio.com/NeuralNetworkIntelligenceOpenSource/_git/Default?_a=contents&path=%2Fsrc%2Fsdk%2Fpynni%2Fnni%2Fassessor.py&version=GBadd_readme)
\ No newline at end of file

examples/feature_engineering/auto-feature-engineering/README.md

+ **Automatic Feature Engineering in nni**	
+ ===	
+ 
+  Now we have an [example](https://github.com/SpongebBob/tabular_automl_NNI), which could automaticlly do feature engineering in nni. 
+  
+  These code come from our contributors. And thanks our lovely contributors!
+  
+ And welcome more and more people to join us!

examples/feature_engineering/auto-feature-engineering/README_zh_CN.md

+ **NNI 中的自动特征工程** ===
+
+  此[示例](https://github.com/SpongebBob/tabular_automl_NNI)在 NNI 中实现了自动特征工程。
+
+  代码来自于贡献者。 谢谢可爱的贡献者！
+
+ 欢迎越来越多的人加入我们！

examples/feature_engineering/gbdt_selector/gbdt_selector_test.py

+# Copyright (c) Microsoft Corporation
+# All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+import bz2
+import urllib.request
+import numpy as np
+
+from sklearn.datasets import load_svmlight_file
+from sklearn.model_selection import train_test_split
+
+from nni.algorithms.feature_engineering.gbdt_selector import GBDTSelector
+
+url_zip_train = 'https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/rcv1_train.binary.bz2'
+urllib.request.urlretrieve(url_zip_train, filename='train.bz2')
+
+f_svm = open('train.svm', 'wt')
+with bz2.open('train.bz2', 'rb') as f_zip:
+    data = f_zip.read()
+    f_svm.write(data.decode('utf-8'))
+f_svm.close()
+
+X, y = load_svmlight_file('train.svm')
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
+
+lgb_params = {
+        'boosting_type': 'gbdt',
+        'objective': 'regression',
+        'metric': {'l2', 'l1'},
+        'num_leaves': 20,
+        'learning_rate': 0.05,
+        'feature_fraction': 0.9,
+        'bagging_fraction': 0.8,
+        'bagging_freq': 5,
+        'verbose': 0}
+
+eval_ratio = 0.1
+early_stopping_rounds = 10
+importance_type = 'gain'
+num_boost_round = 1000
+topk = 10
+
+selector = GBDTSelector()
+selector.fit(X_train, y_train,
+             lgb_params = lgb_params,
+             eval_ratio = eval_ratio,
+             early_stopping_rounds = early_stopping_rounds,
+             importance_type = importance_type,
+             num_boost_round = num_boost_round)
+
+print("selected features\t", selector.get_selected_features(topk=topk))
+

examples/feature_engineering/gradient_feature_selector/.gitignore

+*.bz2
+*.svm
+*.log
+*memory
+*time

examples/feature_engineering/gradient_feature_selector/benchmark_test.py

+# Copyright (c) Microsoft Corporation
+# All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+import bz2
+import urllib.request
+import numpy as np
+import datetime
+
+import line_profiler
+profile = line_profiler.LineProfiler()
+
+import os
+
+from sklearn.datasets import load_svmlight_file
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import make_pipeline
+from sklearn.linear_model import LogisticRegression
+
+from sklearn.ensemble import ExtraTreesClassifier
+from sklearn.feature_selection import SelectFromModel
+
+from nni.algorithms.feature_engineering.gradient_selector import FeatureGradientSelector
+
+
+class Benchmark():
+
+    def __init__(self, files=None, test_size=0.2):
+        self.files =  files
+        self.test_size = test_size
+
+
+    def run_all_test(self, pipeline):
+        for file_name in self.files:
+            file_path = self.files[file_name]
+
+            self.run_test(pipeline, file_name, file_path)
+
+
+    def run_test(self, pipeline, name, path):
+        print("download " + name)
+        update_name = self.download(name, path)
+        X, y = load_svmlight_file(update_name)
+        X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=self.test_size, random_state=42)
+        
+        pipeline.fit(X_train, y_train)
+        print("[Benchmark "+ name + " Score]: ", pipeline.score(X_test, y_test))
+
+
+    def download(self, name, path):
+        old_name = name + '_train.bz2'
+        update_name = name + '_train.svm'
+
+        if os.path.exists(old_name) and os.path.exists(update_name):
+            return update_name
+
+        urllib.request.urlretrieve(path, filename=old_name)
+
+        f_svm = open(update_name, 'wt')
+        with bz2.open(old_name, 'rb') as f_zip:
+            data = f_zip.read()
+            f_svm.write(data.decode('utf-8'))
+        f_svm.close()
+
+        return update_name
+
+@profile
+def test_memory(pipeline_name, name, path):
+    if pipeline_name == "LR":
+        pipeline = make_pipeline(LogisticRegression())
+
+    if pipeline_name == "FGS":
+        pipeline = make_pipeline(FeatureGradientSelector(), LogisticRegression())
+
+    if pipeline_name == "Tree":
+        pipeline = make_pipeline(SelectFromModel(ExtraTreesClassifier(n_estimators=50)), LogisticRegression())
+    
+    test_benchmark = Benchmark()
+    print("Dataset:\t", name)
+    print("Pipeline:\t", pipeline_name)
+    test_benchmark.run_test(pipeline, name, path)
+    print("")
+
+
+def test_time(pipeline_name, name, path):
+    if pipeline_name == "LR":
+        pipeline = make_pipeline(LogisticRegression())
+
+    if pipeline_name == "FGS":
+        pipeline = make_pipeline(FeatureGradientSelector(), LogisticRegression())
+
+    if pipeline_name == "Tree":
+        pipeline = make_pipeline(SelectFromModel(ExtraTreesClassifier(n_estimators=50)), LogisticRegression())
+    
+    test_benchmark = Benchmark()
+    print("Dataset:\t", name)
+    print("Pipeline:\t", pipeline_name)
+    starttime = datetime.datetime.now()
+    test_benchmark.run_test(pipeline, name, path)
+    endtime = datetime.datetime.now()
+    print("Used time: ", (endtime - starttime).microseconds/1000)
+    print("")
+
+
+if __name__ == "__main__":
+    LIBSVM_DATA = {
+        "rcv1" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/rcv1_train.binary.bz2",
+        "colon-cancer" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/covtype.libsvm.binary.bz2",
+        "gisette" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/gisette_scale.bz2",
+        "news20.binary" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/news20.binary.bz2",
+        "real-sim" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/real-sim.bz2",
+        "webspam" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/webspam_wc_normalized_trigram.svm.bz2",
+        "avazu" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/avazu-app.bz2"
+    }
+
+    import argparse
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--pipeline_name', type=str, help='display pipeline_name.')
+    parser.add_argument('--name', type=str, help='display name.')
+    parser.add_argument('--object', type=str, help='display test object: time or memory.')
+
+    args = parser.parse_args()
+    pipeline_name = args.pipeline_name
+    name = args.name
+    test_object = args.object
+    path = LIBSVM_DATA[name]
+
+    if test_object == 'time':
+        test_time(pipeline_name, name, path)
+    elif test_object == 'memory':
+        test_memory(pipeline_name, name, path)
+    else:
+        print("Not support test object.\t", test_object)
+    
+    print("Done.")

examples/feature_engineering/gradient_feature_selector/sklearn_test.py

+# Copyright (c) Microsoft Corporation
+# All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+
+import bz2
+import urllib.request
+import numpy as np
+
+from sklearn.datasets import load_svmlight_file
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import make_pipeline
+from sklearn.linear_model import LogisticRegression
+
+from sklearn.ensemble import ExtraTreesClassifier
+from sklearn.feature_selection import SelectFromModel
+
+from nni.algorithms.feature_engineering.gradient_selector import FeatureGradientSelector
+
+
+def test():
+    url_zip_train = 'https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/rcv1_train.binary.bz2'
+    urllib.request.urlretrieve(url_zip_train, filename='train.bz2')
+
+    f_svm = open('train.svm', 'wt')
+    with bz2.open('train.bz2', 'rb') as f_zip:
+        data = f_zip.read()
+        f_svm.write(data.decode('utf-8'))
+    f_svm.close()
+
+
+    X, y = load_svmlight_file('train.svm')
+    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
+
+
+    pipeline = make_pipeline(FeatureGradientSelector(n_epochs=1, n_features=10), LogisticRegression())
+    # pipeline = make_pipeline(SelectFromModel(ExtraTreesClassifier(n_estimators=50)), LogisticRegression())
+
+    pipeline.fit(X_train, y_train)
+
+    print("Pipeline Score: ", pipeline.score(X_train, y_train))
+
+if __name__ == "__main__":
+    test()

examples/feature_engineering/gradient_feature_selector/test_memory.py

+import os
+
+LIBSVM_DATA = {
+    "rcv1" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/rcv1_train.binary.bz2",
+    "colon-cancer" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/covtype.libsvm.binary.bz2",
+    "gisette" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/gisette_scale.bz2",
+    "news20.binary" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/news20.binary.bz2",
+    "real-sim" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/real-sim.bz2",
+    "avazu" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/avazu-app.bz2",
+}
+
+pipeline_name = "Tree"
+device = "HIP_VISIBLE_DEVICES=0 "
+script = "setsid python -m memory_profiler benchmark_test.py "
+test_object = "memory"
+
+for name in LIBSVM_DATA:
+    log_name = "_".join([pipeline_name, name, test_object])
+    command = device + script + "--pipeline_name " + pipeline_name + " --name " + name + " --object " + test_object + " >" +log_name + " 2>&1 &"
+    print("command is\t", command)
+    os.system(command)
+    print("log is here\t", log_name)
+
+print("Done.")
+
+

examples/feature_engineering/gradient_feature_selector/test_time.py

+import os
+
+LIBSVM_DATA = {
+    "rcv1" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/rcv1_train.binary.bz2",
+    "colon-cancer" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/covtype.libsvm.binary.bz2",
+    "gisette" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/gisette_scale.bz2",
+    "news20.binary" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/news20.binary.bz2",
+    "real-sim" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/real-sim.bz2",
+    "avazu" : "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/avazu-app.bz2",
+}
+
+pipeline_name = "LR"
+device = "HIP_VISIBLE_DEVICES=0 "
+script = "setsid python benchmark_test.py "
+test_object = "time"
+
+for name in LIBSVM_DATA:
+    log_name = "_".join([pipeline_name, name, test_object])
+    command = device + script + "--pipeline_name " + pipeline_name + " --name " + name + " --object " + test_object + " >" +log_name + " 2>&1 &"
+    print("command is\t", command)
+    os.system(command)
+    print("log is here\t", log_name)
+
+print("Done.")
+
+

examples/model_compress/.gitignore

+.pth
+.tar.gz
+data/
+MNIST/
+cifar-10-batches-py/
+experiment_data/
\ No newline at end of file

examples/model_compress/auto_compress/torch/auto_compress_module.py

+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+from typing import Callable, Optional, Iterable
+
+import torch
+import torch.nn as nn
+import torch.optim as optim
+import torch.nn.functional as F
+from torchvision import datasets, transforms
+
+from nni.algorithms.compression.pytorch.auto_compress import AbstractAutoCompressionModule
+
+torch.manual_seed(1)
+
+class LeNet(nn.Module):
+    def __init__(self):
+        super(LeNet, self).__init__()
+        self.conv1 = nn.Conv2d(1, 32, 3, 1)
+        self.conv2 = nn.Conv2d(32, 64, 3, 1)
+        self.dropout1 = nn.Dropout2d(0.25)
+        self.dropout2 = nn.Dropout2d(0.5)
+        self.fc1 = nn.Linear(9216, 128)
+        self.fc2 = nn.Linear(128, 10)
+
+    def forward(self, x):
+        x = self.conv1(x)
+        x = F.relu(x)
+        x = self.conv2(x)
+        x = F.relu(x)
+        x = F.max_pool2d(x, 2)
+        x = self.dropout1(x)
+        x = torch.flatten(x, 1)
+        x = self.fc1(x)
+        x = F.relu(x)
+        x = self.dropout2(x)
+        x = self.fc2(x)
+        output = F.log_softmax(x, dim=1)
+        return output
+
+_use_cuda = torch.cuda.is_available()
+
+_train_kwargs = {'batch_size': 64}
+_test_kwargs = {'batch_size': 1000}
+if _use_cuda:
+    _cuda_kwargs = {'num_workers': 1,
+                    'pin_memory': True,
+                    'shuffle': True}
+    _train_kwargs.update(_cuda_kwargs)
+    _test_kwargs.update(_cuda_kwargs)
+
+_transform = transforms.Compose([
+    transforms.ToTensor(),
+    transforms.Normalize((0.1307,), (0.3081,))
+])
+
+_device = torch.device("cuda" if _use_cuda else "cpu")
+
+_train_loader = None
+_test_loader = None
+
+def _train(model, optimizer, criterion, epoch):
+    global _train_loader
+    if _train_loader is None:
+        dataset = datasets.MNIST('./data', train=True, download=True, transform=_transform)
+        _train_loader = torch.utils.data.DataLoader(dataset, **_train_kwargs)
+    model.train()
+    for data, target in _train_loader:
+        data, target = data.to(_device), target.to(_device)
+        optimizer.zero_grad()
+        output = model(data)
+        loss = criterion(output, target)
+        loss.backward()
+        optimizer.step()
+
+def _test(model):
+    global _test_loader
+    if _test_loader is None:
+        dataset = datasets.MNIST('./data', train=False, transform=_transform)
+        _test_loader = torch.utils.data.DataLoader(dataset, **_test_kwargs)
+    model.eval()
+    test_loss = 0
+    correct = 0
+    with torch.no_grad():
+        for data, target in _test_loader:
+            data, target = data.to(_device), target.to(_device)
+            output = model(data)
+            test_loss += F.nll_loss(output, target, reduction='sum').item()
+            pred = output.argmax(dim=1, keepdim=True)
+            correct += pred.eq(target.view_as(pred)).sum().item()
+    test_loss /= len(_test_loader.dataset)
+    acc = 100 * correct / len(_test_loader.dataset)
+    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
+        test_loss, correct, len(_test_loader.dataset), acc))
+    return acc
+
+_model = LeNet().to(_device)
+_model.load_state_dict(torch.load('mnist_pretrain_lenet.pth'))
+
+class AutoCompressionModule(AbstractAutoCompressionModule):
+    @classmethod
+    def model(cls) -> nn.Module:
+        return _model
+
+    @classmethod
+    def evaluator(cls) -> Callable[[nn.Module], float]:
+        return _test
+
+    @classmethod
+    def optimizer_factory(cls) -> Optional[Callable[[Iterable], optim.Optimizer]]:
+        def _optimizer_factory(params: Iterable):
+            return torch.optim.SGD(params, lr=0.01)
+        return _optimizer_factory
+
+    @classmethod
+    def criterion(cls) -> Optional[Callable]:
+        return F.nll_loss
+
+    @classmethod
+    def sparsifying_trainer(cls, compress_algorithm_name: str) -> Optional[Callable[[nn.Module, optim.Optimizer, Callable, int], None]]:
+        return _train
+
+    @classmethod
+    def post_compress_finetuning_trainer(cls, compress_algorithm_name: str) -> Optional[Callable[[nn.Module, optim.Optimizer, Callable, int], None]]:
+        return _train
+
+    @classmethod
+    def post_compress_finetuning_epochs(cls, compress_algorithm_name: str) -> int:
+        return 2

examples/model_compress/auto_compress/torch/auto_compress_torch.py

+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+from pathlib import Path
+
+from nni.algorithms.compression.pytorch.auto_compress import AutoCompressionExperiment, AutoCompressionSearchSpaceGenerator
+
+from auto_compress_module import AutoCompressionModule
+
+generator = AutoCompressionSearchSpaceGenerator()
+generator.add_config('level', [
+    {
+        "sparsity": {
+            "_type": "uniform",
+            "_value": [0.01, 0.99]
+        },
+        'op_types': ['default']
+    }
+])
+generator.add_config('l1', [
+    {
+        "sparsity": {
+            "_type": "uniform",
+            "_value": [0.01, 0.99]
+        },
+        'op_types': ['Conv2d']
+    }
+])
+generator.add_config('qat', [
+    {
+        'quant_types': ['weight', 'output'],
+        'quant_bits': {
+            'weight': 8,
+            'output': 8
+        },
+        'op_types': ['Conv2d', 'Linear']
+    }])
+search_space = generator.dumps()
+
+experiment = AutoCompressionExperiment(AutoCompressionModule, 'local')
+experiment.config.experiment_name = 'auto compression torch example'
+experiment.config.trial_concurrency = 1
+experiment.config.max_trial_number = 10
+experiment.config.search_space = search_space
+experiment.config.trial_code_directory = Path(__file__).parent
+experiment.config.tuner.name = 'TPE'
+experiment.config.tuner.class_args['optimize_mode'] = 'maximize'
+experiment.config.training_service.use_active_gpu = True
+
+experiment.run(8088)