[Bugbash] add default value for experiment config key & fix HPO benchmarking (#4272)

docs/en_US/reference/experiment_config.rst

@@ -114,17 +114,17 @@ ExperimentConfig
       - Description
     
     * - experimentName
-      - ``Optional[str]``
+      - ``str``, optional
       - Mnemonic name of the experiment, which will be shown in WebUI and nnictl.
 
     * - searchSpaceFile
-      - ``Optional[str]``
+      - ``str``, optional
       - Path_ to the JSON file containing the search space.
         Search space format is determined by tuner. The common format for built-in tuners is documented  `here <../Tutorial/SearchSpaceSpec.rst>`__.
         Mutually exclusive to ``searchSpace``.
 
     * - searchSpace
-      - ``Optional[JSON]``
+      - ``JSON``, optional
       - Search space object.
         The format is determined by tuner. Common format for built-in tuners is documented `here <../Tutorial/SearchSpaceSpec.rst>`__.
         Note that ``None`` means "no such field" so empty search space should be written as ``{}``.
@@ -137,9 +137,8 @@ ExperimentConfig
         Note that using ``python3`` on Linux and macOS, and using ``python`` on Windows.
 
     * - trialCodeDirectory
-      - ``str``
-      - `Path`_ to the directory containing trial source files.
-        default: ``"."``.
+      - ``str``, optional
+      - Default: ``"."``. `Path`_ to the directory containing trial source files.
         All files in this directory will be sent to the training machine, unless in the ``.nniignore`` file.
         (See :ref:`nniignore <nniignore>` for details.)
 
@@ -149,8 +148,8 @@ ExperimentConfig
         The real concurrency also depends on hardware resources and may be less than this value.
 
     * - trialGpuNumber
-      - ``Optional[int]``
-      - This field might have slightly different meanings for various training services,
+      - ``int`` or ``None``, optional
+      - Default: None. This field might have slightly different meanings for various training services,
         especially when set to ``0`` or ``None``.
         See `training service's document <../training_services.rst>`__ for details.
 
@@ -159,75 +158,72 @@ ExperimentConfig
         but they can still use all GPU resources if they want.
 
     * - maxExperimentDuration
-      - ``Optional[str]``
-      - Limit the duration of this experiment if specified.
-        format: ``number + s|m|h|d``
-        examples: ``"10m"``, ``"0.5h"``
+      - ``str``, optional
+      - Limit the duration of this experiment if specified. The duration is unlimited if not set.
+        Format: ``number + s|m|h|d``.
+        Examples: ``"10m"``, ``"0.5h"``.
         When time runs out, the experiment will stop creating trials but continue to serve WebUI.
 
     * - maxTrialNumber
-      - ``Optional[int]``
-      - Limit the number of trials to create if specified.
+      - ``int``, optional
+      - Limit the number of trials to create if specified. The trial number is unlimited if not set.
         When the budget runs out, the experiment will stop creating trials but continue to serve WebUI.
 
     * - maxTrialDuration
-      - ``Optional[str]``
-      - Limit the duration of trial job if specified.
-        format: ``number + s|m|h|d``
-        examples: ``"10m"``, ``"0.5h"``
+      - ``str``, optional
+      - Limit the duration of trial job if specified. The duration is unlimited if not set.
+        Format: ``number + s|m|h|d``.
+        Examples: ``"10m"``, ``"0.5h"``.
         When time runs out, the current trial job will stop.
 
     * - nniManagerIp
-      - ``Optional[str]``
-      - IP of the current machine, used by training machines to access NNI manager. Not used in local mode.
-        If not specified, IPv4 address of ``eth0`` will be used.
+      - ``str``, optional
+      - Default: default connection chosen by system. IP of the current machine, used by training machines to access NNI manager. Not used in local mode.
         Except for the local mode, it is highly recommended to set this field manually.
 
     * - useAnnotation
-      - ``bool``
-      - Enable `annotation <../Tutorial/AnnotationSpec.rst>`__.
-        default: ``False``.
+      - ``bool``, optional
+      - Default: ``False``. Enable `annotation <../Tutorial/AnnotationSpec.rst>`__.
         When using annotation, ``searchSpace`` and ``searchSpaceFile`` should not be specified manually.
 
     * - debug
-      - ``bool``
-      - Enable debug mode.
-        default: ``False``
+      - ``bool``, optional
+      - Default: ``False``. Enable debug mode.
         When enabled, logging will be more verbose and some internal validation will be loosened.
 
     * - logLevel
-      - ``Optional[str]``
-      - Set log level of the whole system.
+      - ``str``, optional
+      - Default: ``info`` or ``debug``, depending on ``debug`` option. Set log level of the whole system.
         values: ``"trace"``, ``"debug"``, ``"info"``, ``"warning"``, ``"error"``, ``"fatal"``
-        Defaults to "info" or "debug", depending on ``debug`` option. When debug mode is enabled, Loglevel is set to "debug", otherwise, Loglevel is set to "info".
+        When debug mode is enabled, Loglevel is set to "debug", otherwise, Loglevel is set to "info".
         Most modules of NNI will be affected by this value, including NNI manager, tuner, training service, etc.
         The exception is trial, whose logging level is directly managed by trial code.
         For Python modules, "trace" acts as logging level 0 and "fatal" acts as ``logging.CRITICAL``.
 
     * - experimentWorkingDirectory
-      - ``Optional[str]``
-      - Specify the :ref:`directory <path>` to place log, checkpoint, metadata, and other run-time stuff.
-        By default uses ``~/nni-experiments``.
+      - ``str``, optional
+      - Default: ``~/nni-experiments``.
+        Specify the :ref:`directory <path>` to place log, checkpoint, metadata, and other run-time stuff.
         NNI will create a subdirectory named by experiment ID, so it is safe to use the same directory for multiple experiments.
 
     * - tunerGpuIndices
-      - ``Optional[list[int] | str | int]``
+      - ``list[int]`` or ``str`` or ``int``, optional
       - Limit the GPUs visible to tuner, assessor, and advisor.
         This will be the ``CUDA_VISIBLE_DEVICES`` environment variable of tuner process.
         Because tuner, assessor, and advisor run in the same process, this option will affect them all.
 
     * - tuner
-      - ``Optional[AlgorithmConfig]``
+      - ``AlgorithmConfig``, optional
       - Specify the tuner.
         The built-in tuners can be found `here <../builtin_tuner.rst>`__ and you can follow `this tutorial <../Tuner/CustomizeTuner.rst>`__ to customize a new tuner.
 
     * - assessor
-      - ``Optional[AlgorithmConfig]``
+      - ``AlgorithmConfig``, optional
       - Specify the assessor.
         The built-in assessors can be found `here <../builtin_assessor.rst>`__ and you can follow `this tutorial <../Assessor/CustomizeAssessor.rst>`__ to customize a new assessor.
 
     * - advisor
-      - ``Optional[AlgorithmConfig]``
+      - ``AlgorithmConfig``, optional
       - Specify the advisor.
         NNI provides two built-in advisors: `BOHB <../Tuner/BohbAdvisor.rst>`__ and `Hyperband <../Tuner/HyperbandAdvisor.rst>`__, and you can follow `this tutorial <../Tuner/CustomizeAdvisor.rst>`__ to customize a new advisor.
 
@@ -236,7 +232,7 @@ ExperimentConfig
       - Specify the `training service <../TrainingService/Overview.rst>`__.
 
     * - sharedStorage
-      - ``Optional[SharedStorageConfig]``
+      - ``SharedStorageConfig``, optional
       - Configure the shared storage, detailed usage can be found `here <../Tutorial/HowToUseSharedStorage.rst>`__.
 
 AlgorithmConfig
@@ -259,23 +255,23 @@ For customized algorithms, there are two ways to describe them:
       - Description
     
     * - name
-      - ``Optional[str]``
-      - Name of the built-in or registered algorithm.
+      - ``str`` or ``None``, optional
+      - Default: None. Name of the built-in or registered algorithm.
         ``str`` for the built-in and registered algorithm, ``None`` for other customized algorithms.
 
     * - className
-      - ``Optional[str]``
-      - Qualified class name of not registered customized algorithm.
+      - ``str`` or ``None``, optional
+      - Default: None. Qualified class name of not registered customized algorithm.
         ``None`` for the built-in and registered algorithm, ``str`` for other customized algorithms.
         example: ``"my_tuner.MyTuner"``
 
     * - codeDirectory
-      - ``Optional[str]``
-      - `Path`_ to the directory containing the customized algorithm class.
+      - ``str`` or ``None``, optional
+      - Default: None. Path_ to the directory containing the customized algorithm class.
         ``None`` for the built-in and registered algorithm, ``str`` for other customized algorithms.
 
     * - classArgs
-      - ``Optional[dict[str, Any]]``
+      - ``dict[str, Any]``, optional
       - Keyword arguments passed to algorithm class' constructor.
         See algorithm's document for supported value.
 
@@ -311,8 +307,8 @@ Detailed usage can be found `here <../TrainingService/LocalMode.rst>`__.
       -
     
     * - useActiveGpu
-      - ``Optional[bool]``
-      - Specify whether NNI should submit trials to GPUs occupied by other tasks.
+      - ``bool``, optional
+      - Default: ``False``. Specify whether NNI should submit trials to GPUs occupied by other tasks.
         Must be set when ``trialGpuNumber`` greater than zero.
         Following processes can make GPU "active":
 
@@ -325,12 +321,11 @@ Detailed usage can be found `here <../TrainingService/LocalMode.rst>`__.
         When you create multiple NNI experiments and ``useActiveGpu`` is set to ``True``, they will submit multiple trials to the same GPU(s) simultaneously.
 
     * - maxTrialNumberPerGpu
-      - ``int``
-      - Specify how many trials can share one GPU.
-        default: ``1``
+      - ``int``, optional
+      - Default: ``1``. Specify how many trials can share one GPU.
 
     * - gpuIndices
-      - ``Optional[list[int] | str | int]``
+      - ``list[int]`` or ``str`` or ``int``, optional
       - Limit the GPUs visible to trial processes.
         If ``trialGpuNumber`` is less than the length of this value, only a subset will be visible to each trial.
         This will be used as ``CUDA_VISIBLE_DEVICES`` environment variable.
@@ -357,8 +352,8 @@ Detailed usage can be found `here <../TrainingService/RemoteMachineMode.rst>`__.
       - List of training machines.
 
     * - reuseMode
-      - ``bool``
-      - Enable `reuse mode <../TrainingService/Overview.rst#training-service-under-reuse-mode>`__.
+      - ``bool``, optional
+      - Default: ``True``. Enable `reuse mode <../TrainingService/Overview.rst#training-service-under-reuse-mode>`__.
 
 RemoteMachineConfig
 """""""""""""""""""
@@ -376,31 +371,29 @@ RemoteMachineConfig
       - IP or hostname (domain name) of the machine.
 
     * - port
-      - ``int``
-      - SSH service port.
-        default: ``22``
+      - ``int``, optional
+      - Default: ``22``. SSH service port.
 
     * - user
       - ``str``
       - Login user name.
 
     * - password
-      - ``Optional[str]``
+      - ``str``, optional
       - If not specified, ``sshKeyFile`` will be used instead.
     
     * - sshKeyFile
-      - ``Optional[str]``
+      - ``str``, optional
       - `Path`_ to ``sshKeyFile`` (identity file).
         Only used when ``password`` is not specified.
 
     * - sshPassphrase
-      - ``Optional[str]``
+      - ``str``, optional
       - Passphrase of SSH identity file.
 
     * - useActiveGpu
-      - ``bool``
-      - Specify whether NNI should submit trials to GPUs occupied by other tasks.
-        default: ``False``
+      - ``bool``, optional
+      - Default: ``False``. Specify whether NNI should submit trials to GPUs occupied by other tasks.
         Must be set when ``trialGpuNumber`` greater than zero.
         Following processes can make GPU "active":
 
@@ -413,18 +406,17 @@ RemoteMachineConfig
         When you create multiple NNI experiments and ``useActiveGpu`` is set to ``True``, they will submit multiple trials to the same GPU(s) simultaneously.
 
     * - maxTrialNumberPerGpu
-      - ``int``
-      - Specify how many trials can share one GPU.
-        default: ``1``
+      - ``int``, optional
+      - Default: ``1``. Specify how many trials can share one GPU.
 
     * - gpuIndices
-      - ``Optional[list[int] | str | int]``
+      - ``list[int]`` or ``str`` or ``int``, optional
       - Limit the GPUs visible to trial processes.
         If ``trialGpuNumber`` is less than the length of this value, only a subset will be visible to each trial.
         This will be used as ``CUDA_VISIBLE_DEVICES`` environment variable.
 
     * - pythonPath
-      - ``Optional[str]``
+      - ``str``, optional
       - Specify a Python environment.
         This path will be inserted at the front of PATH. Here are some examples: 
 
@@ -434,7 +426,7 @@ RemoteMachineConfig
         If you are working on Anaconda, there is some difference. On Windows, you also have to add ``../script`` and ``../Library/bin`` separated by ``;``. Examples are as below:
 
           - (linux anaconda) pythonPath: ``/home/yourname/anaconda3/envs/myenv/bin/``
-          - (windows anaconda) pythonPath: ``C:/Users/yourname/.conda/envs/myenv;C:/Users/yourname/.conda/envs/myenv/Scripts;C:/Users/yourname/.conda/envs/myenv/Library/bin``
+          - (windows anaconda) pythonPath: ``C:/Users/yourname/.conda/envs/myenv``; ``C:/Users/yourname/.conda/envs/myenv/Scripts``; ``C:/Users/yourname/.conda/envs/myenv/Library/bin``
 
         This is useful if preparing steps vary for different machines.
 
@@ -485,9 +477,8 @@ Detailed usage can be found `here <../TrainingService/PaiMode.rst>`__.
       - Specify the storage name used in OpenPAI.
 
     * - dockerImage
-      - ``str``
-      - Name and tag of docker image to run the trials.
-        default: ``"msranni/nni:latest"``.
+      - ``str``, optional
+      - Default: ``"msranni/nni:latest"``. Name and tag of docker image to run the trials.
 
     * - localStorageMountPoint
       - ``str``
@@ -499,16 +490,15 @@ Detailed usage can be found `here <../TrainingService/PaiMode.rst>`__.
         This must be an absolute path.
 
     * - reuseMode
-      - ``bool``
-      - Enable `reuse mode <../TrainingService/Overview.rst#training-service-under-reuse-mode>`__.
-        default: ``False``.
+      - ``bool``, optional
+      - Default: ``True``. Enable `reuse mode <../TrainingService/Overview.rst#training-service-under-reuse-mode>`__.
 
     * - openpaiConfig
-      - ``Optional[JSON]``
+      - ``JSON``, optional
       - Embedded OpenPAI config file.
 
     * - openpaiConfigFile
-      - ``Optional[str]``
+      - ``str``, optional
       - `Path`_ to OpenPAI config file.
         An example can be found `here <https://github.com/microsoft/pai/blob/master/docs/manual/cluster-user/examples/hello-world-job.yaml>`__.
 
@@ -530,9 +520,8 @@ Detailed usage can be found `here <../TrainingService/AMLMode.rst>`__.
       -
 
     * - dockerImage
-      - ``str``
-      - Name and tag of docker image to run the trials.
-        default: ``"msranni/nni:latest"``
+      - ``str``, optional
+      - Default: ``"msranni/nni:latest"``. Name and tag of docker image to run the trials.
 
     * - subscriptionId
       - ``str``
@@ -568,17 +557,16 @@ Detailed usage can be found `here <../TrainingService/DlcMode.rst>`__.
       -
     
     * - type
-      - ``str``
-      - Job spec type.
-        default: ``"worker"``.
+      - ``str``, optional
+      - Default: ``"Worker"``. Job spec type.
 
     * - image
       - ``str``
       - Name and tag of docker image to run the trials.
 
     * - jobType
-      - ``str``
-      - PAI-DLC training job type, ``"TFJob"`` or ``"PyTorchJob"``.
+      - ``str``, optional
+      - Default: ``"TFJob"``. PAI-DLC training job type, ``"TFJob"`` or ``"PyTorchJob"``.
 
     * - podCount
       - ``str``
@@ -698,7 +686,7 @@ azureBlobConfig
       - Azure storage account name.
 
     * - storageAccountKey
-      - ``Optional[str]``
+      - ``str``
       - Azure storage account key.
 
     * - containerName

examples/trials/benchmarking/automlbenchmark/setup.sh

@@ -2,7 +2,7 @@
 
 # download automlbenchmark repository
 if [ ! -d './automlbenchmark' ] ; then
-    git clone https://github.com/openml/automlbenchmark.git --branch stable --depth 1
+    git clone https://github.com/openml/automlbenchmark.git --branch v1.6 --depth 1
 fi
 
 # install dependencies