Add BOHB Advisor (#910)

add BOHB Advisor

README.md

@@ -65,7 +65,8 @@ The tool dispatches and runs trial jobs generated by tuning algorithms to search
           <li><a href="docs/en_US/Builtin_Tuner.md#NetworkMorphism">Network Morphism</a></li>
           <li><a href="examples/tuners/enas_nni/README.md">ENAS</a></li>
           <li><a href="docs/en_US/Builtin_Tuner.md#NetworkMorphism#MetisTuner">Metis Tuner</a></li>
-        </ul> 
+          <li><a href="docs/en_US/Builtin_Tuner.md#BOHB">BOHB</a></li>
+        </ul>
           <a href="docs/en_US/Builtin_Assessors.md#assessor">Assessor</a> 
         <ul>
           <li><a href="docs/en_US/Builtin_Assessors.md#Medianstop">Median Stop</a></li>

docs/en_US/Builtin_Tuner.md

@@ -18,6 +18,7 @@ Currently we support the following algorithms:
 |[__Hyperband__](#Hyperband)|Hyperband tries to use the limited resource to explore as many configurations as possible, and finds out the promising ones to get the final result. The basic idea is generating many configurations and to run them for the small number of trial budget to find out promising one, then further training those promising ones to select several more promising one.[Reference Paper](https://arxiv.org/pdf/1603.06560.pdf)|
 |[__Network Morphism__](#NetworkMorphism)|Network Morphism provides functions to automatically search for architecture of deep learning models. Every child network inherits the knowledge from its parent network and morphs into diverse types of networks, including changes of depth, width, and skip-connection. Next, it estimates the value of a child network using the historic architecture and metric pairs. Then it selects the most promising one to train. [Reference Paper](https://arxiv.org/abs/1806.10282)|
 |[__Metis Tuner__](#MetisTuner)|Metis offers the following benefits when it comes to tuning parameters: While most tools only predict the optimal configuration, Metis gives you two outputs: (a) current prediction of optimal configuration, and (b) suggestion for the next trial. No more guesswork. While most tools assume training datasets do not have noisy data, Metis actually tells you if you need to re-sample a particular hyper-parameter. [Reference Paper](https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/)|
+|[__BOHB__](#BOHB)|BOHB is a follow-up work of Hyperband. It targets the weakness of Hyperband that new configurations are generated randomly without leveraging finished trials. For the name BOHB, HB means Hyperband, BO means Byesian Optimization. BOHB leverages finished trials by building multiple TPE models, a proportion of new configurations are generated through these models. [Reference Paper](https://arxiv.org/abs/1807.01774)|
 
 <br>
 
@@ -317,3 +318,50 @@ tuner:
   classArgs:
     optimize_mode: maximize
 ```
+
+<br>
+
+<a name="BOHB"></a>
+
+![](https://placehold.it/15/1589F0/000000?text=+) `BOHB Advisor`
+
+> Builtin Tuner Name: **BOHB**
+
+**Installation**
+
+BOHB advisor requires [ConfigSpace](https://github.com/automl/ConfigSpace) package, ConfigSpace need to be installed by following command before first use.
+
+```bash
+nnictl package install --name=BOHB
+```
+
+**Suggested scenario**
+
+Similar to Hyperband, it is suggested when you have limited computation resource but have relatively large search space. It performs well in the scenario that intermediate result (e.g., accuracy) can reflect good or bad of final result (e.g., accuracy) to some extent. In this case, it may converges to a better configuration due to bayesian optimization usage.
+
+**Requirement of classArg**
+
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*) - If 'maximize', tuners will target to maximize metrics. If 'minimize', tuner will target to minimize metrics.
+* **min_budget** (*int, optional, default = 1*) - The smallest budget assign to a trial job, (budget could be the number of mini-batches or epochs). Needs to be positive.
+* **max_budget** (*int, optional, default = 3*) - The largest budget assign to a trial job, (budget could be the number of mini-batches or epochs). Needs to be larger than min_budget.
+* **eta** (*int, optional, default = 3*) - In each iteration, a complete run of sequential halving is executed. In it, after evaluating each configuration on the same subset size, only a fraction of 1/eta of them 'advances' to the next round. Must be greater or equal to 2.
+* **min_points_in_model**(*int, optional, default = None*): number of observations to start building a KDE. Default 'None' means dim+1, when the number of completed trial in this budget is equal or larger than `max{dim+1, min_points_in_model}`, BOHB will start to build a KDE model of this budget, then use KDE model to guide the configuration selection. Need to be positive.(dim means the number of hyperparameters in search space)
+* **top_n_percent**(*int, optional, default = 15*): percentage (between 1 and 99, default 15) of the observations that are considered good. Good points and bad points are used for building KDE models. For example, if you have 100 observed trials and top_n_percent is 15, then top 15 point will used for building good point models "l(x)", the remaining 85 point will used for building bad point models "g(x)".
+* **num_samples**(*int, optional, default = 64*): number of samples to optimize EI (default 64). In this case, we will sample "num_samples"(default = 64) points, and compare the result of l(x)/g(x), then return one with the maximum l(x)/g(x) value as the next configuration if the optimize_mode is maximize. Otherwise, we return the smallest one.
+* **random_fraction**(*float, optional, default = 0.33*): fraction of purely random configurations that are sampled from the prior without the model.
+* **bandwidth_factor**(*float, optional, default = 3.0*): to encourage diversity, the points proposed to optimize EI, are sampled from a 'widened' KDE where the bandwidth is multiplied by this factor. Suggest to use default value if you are not familiar with KDE.
+* **min_bandwidth**(*float, optional, default = 0.001*): to keep diversity, even when all (good) samples have the same value for one of the parameters, a minimum bandwidth (default: 1e-3) is used instead of zero. Suggest to use default value if you are not familiar with KDE.
+
+*Please note that currently float type only support decimal representation, you have to use 0.333 instead of 1/3 and 0.001 instead of 1e-3.*
+
+**Usage example**
+
+```yml
+advisor:
+  builtinAdvisorName: BOHB
+  classArgs:
+    optimize_mode: maximize
+    min_budget: 1
+    max_budget: 27
+    eta: 3
+```
\ No newline at end of file

docs/en_US/bohbAdvisor.md

+BOHB Advisor on NNI
+===
+
+## 1. Introduction
+BOHB is a robust and efficient hyperparameter tuning algorithm mentioned in [reference paper](https://arxiv.org/abs/1807.01774). BO is the abbreviation of Bayesian optimization and HB is the abbreviation of Hyperband.
+
+BOHB relies on HB(Hyperband) to determine how many configurations to evaluate with which budget, but it **replaces the random selection of configurations at the beginning of each HB iteration by a model-based search(Byesian Optimization)**. Once the desired number of configurations for the iteration is reached, the standard successive halving procedure is carried out using these configurations. We keep track of the performance of all function evaluations g(x, b) of configurations x on all budgets b to use as a basis for our models in later iterations.
+
+Below we divide introduction of the BOHB process into two parts:
+
+### HB (Hyperband)
+
+We follow Hyperband’s way of choosing the budgets and continue to use SuccessiveHalving, for more details, you can refer to the [Hyperband in NNI](hyperbandAdvisor.md) and [reference paper of Hyperband](https://arxiv.org/abs/1603.06560). This procedure is summarized by the pseudocode below.
+
+![](../img/bohb_1.png)
+
+### BO (Bayesian Optimization)
+
+The BO part of BOHB closely resembles TPE, with one major difference: we opted for a single multidimensional KDE compared to the hierarchy of one-dimensional KDEs used in TPE in order to better handle interaction effects in the input space.
+
+Tree Parzen Estimator(TPE): uses a KDE(kernel density estimator) to model the densities.
+
+![](../img/bohb_2.png)
+
+To fit useful KDEs, we require a minimum number of data points Nmin; this is set to d + 1 for our experiments, where d is the number of hyperparameters. To build a model as early as possible, we do not wait until Nb = |Db|, the number of observations for budget b, is large enough to satisfy q · Nb ≥ Nmin. Instead, after initializing with Nmin + 2 random configurations, we choose the
+
+![](../img/bohb_3.png)
+
+best and worst configurations, respectively, to model the two densities.
+
+Note that we alse sample a constant fraction named **random fraction** of the configurations uniformly at random.
+
+## 2. Workflow
+
+![](../img/bohb_6.jpg)
+
+This image shows the workflow of BOHB. Here we set max_budget = 9, min_budget = 1, eta = 3, others as default. In this case, s_max = 2, so we will continuesly run the {s=2, s=1, s=0, s=2, s=1, s=0, ...} cycle. In each stage of SuccessiveHalving (the orange box), we will pick the top 1/eta configurations and run them again with more budget, repeated SuccessiveHalving stage until the end of this iteration. At the same time, we collect the configurations, budgets and final metrics of each trial, and use this to build a multidimensional KDEmodel with the key "budget".
+ Multidimensional KDE is used to guide the selection of configurations for the next iteration.
+
+The way of sampling procedure(use Multidimensional KDE to guide the selection) is summarized by the pseudocode below.
+
+![](../img/bohb_4.png)
+
+## 3. Usage
+
+BOHB advisor requires [ConfigSpace](https://github.com/automl/ConfigSpace) package, ConfigSpace need to be installed by following command before first use.
+
+```bash
+nnictl package install --name=BOHB
+```
+
+To use BOHB, you should add the following spec in your experiment's YAML config file:
+
+```yml
+advisor:
+  builtinAdvisorName: BOHB
+  classArgs:
+    optimize_mode: maximize
+    min_budget: 1
+    max_budget: 27
+    eta: 3
+    min_points_in_model: 7
+    top_n_percent: 15
+    num_samples: 64
+    random_fraction: 0.33
+    bandwidth_factor: 3.0
+    min_bandwidth: 0.001
+```
+
+**Requirement of classArg**
+
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*) - If 'maximize', tuners will target to maximize metrics. If 'minimize', tuner will target to minimize metrics.
+* **min_budget** (*int, optional, default = 1*) - The smallest budget assign to a trial job, (budget could be the number of mini-batches or epochs). Needs to be positive.
+* **max_budget** (*int, optional, default = 3*) - The largest budget assign to a trial job, (budget could be the number of mini-batches or epochs). Needs to be larger than min_budget.
+* **eta** (*int, optional, default = 3*) - In each iteration, a complete run of sequential halving is executed. In it, after evaluating each configuration on the same subset size, only a fraction of 1/eta of them 'advances' to the next round. Must be greater or equal to 2.
+* **min_points_in_model**(*int, optional, default = None*): number of observations to start building a KDE. Default 'None' means dim+1, when the number of completed trial in this budget is equal or larger than `max{dim+1, min_points_in_model}`, BOHB will start to build a KDE model of this budget, then use KDE model to guide the configuration selection. Need to be positive.(dim means the number of hyperparameters in search space)
+* **top_n_percent**(*int, optional, default = 15*): percentage (between 1 and 99, default 15) of the observations that are considered good. Good points and bad points are used for building KDE models. For example, if you have 100 observed trials and top_n_percent is 15, then top 15 point will used for building good point models "l(x)", the remaining 85 point will used for building bad point models "g(x)".
+* **num_samples**(*int, optional, default = 64*): number of samples to optimize EI (default 64). In this case, we will sample "num_samples"(default = 64) points, and compare the result of l(x)/g(x), then return one with the maximum l(x)/g(x) value as the next configuration if the optimize_mode is maximize. Otherwise, we return the smallest one.
+* **random_fraction**(*float, optional, default = 0.33*): fraction of purely random configurations that are sampled from the prior without the model.
+* **bandwidth_factor**(*float, optional, default = 3.0*): to encourage diversity, the points proposed to optimize EI, are sampled from a 'widened' KDE where the bandwidth is multiplied by this factor. Suggest to use default value if you are not familiar with KDE.
+* **min_bandwidth**(*float, optional, default = 0.001*): to keep diversity, even when all (good) samples have the same value for one of the parameters, a minimum bandwidth (default: 1e-3) is used instead of zero. Suggest to use default value if you are not familiar with KDE.
+
+*Please note that currently float type only support decimal representation, you have to use 0.333 instead of 1/3 and 0.001 instead of 1e-3.*
+
+## 4. File Structure
+The advisor has a lot of different files, functions and classes. Here we will only give most of those files a brief introduction:
+
+* `bohb_advisor.py` Defination of BOHB, handle the interaction with the dispatcher, including generating new trial and processing results. Also includes the implementation of HB(Hyperband) part.
+* `config_generator.py` includes the implementation of BO(Bayesian Optimization) part. The function *get_config* can generate new configuration base on BO, the function *new_result* will update model with the new result.
+
+## 5. Experiment
+
+### MNIST with BOHB
+
+code implementation: [examples/trials/mnist-advisor](https://github.com/Microsoft/nni/tree/master/examples/trials/)
+
+We chose BOHB to build CNN on the MNIST dataset. The following is our experimental final results:
+
+![](../img/bohb_5.png)
+
+More experimental result can be found in the [reference paper](https://arxiv.org/abs/1807.01774), we can see that BOHB makes good use of previous results, and has a balance trade-off in exploration and exploitation.
\ No newline at end of file

docs/en_US/builtinTuner.rst

@@ -14,4 +14,5 @@ Builtin-Tuners
     Grid Search<gridsearchTuner>
     Hyperband<hyperbandAdvisor>
     Network Morphism<networkmorphismTuner>
-    Metis Tuner<metisTuner>
\ No newline at end of file
+    Metis Tuner<metisTuner>
+    BOHB<bohbAdvisor>
\ No newline at end of file

docs/en_US/sdk_reference.rst

@@ -49,4 +49,7 @@ Assessor
 Advisor
 ------------------------
 ..  autoclass:: nni.hyperband_advisor.hyperband_advisor.Hyperband
+    :members:
+
+..  autoclass:: nni.bohb_advisor.bohb_advisor.BOHB
     :members:
\ No newline at end of file

examples/trials/mnist-advisor/config_bohb.yml

+authorName: default
+experimentName: example_mnist_bohb
+trialConcurrency: 1
+maxExecDuration: 10h
+maxTrialNum: 1000
+#choice: local, remote, pai
+trainingServicePlatform: local
+searchSpacePath: search_space.json
+#choice: true, false
+useAnnotation: false
+advisor:
+  #choice: Hyperband, BOHB
+  #(BOHB should be installed through nnictl)
+  builtinAdvisorName: BOHB
+  classArgs:
+    max_budget: 27
+    min_budget: 1
+    eta: 3
+    optimize_mode: maximize
+trial:
+  command: python3 mnist.py
+  codeDir: .
+  gpuNum: 0

examples/trials/mnist-advisor/config_hyperband.yml

 authorName: default
-experimentName: example_mnist
+experimentName: example_mnist_hyperband
 trialConcurrency: 2
 maxExecDuration: 100h
 maxTrialNum: 10000

examples/trials/mnist-advisor/config_pai.yml

@@ -10,6 +10,7 @@ searchSpacePath: search_space.json
 useAnnotation: false
 advisor:
   #choice: Hyperband, BOHB
+  #(BOHB should be installed through nnictl)
   builtinAdvisorName: Hyperband
   classArgs:
     #R: the maximum trial budget

src/nni_manager/rest_server/restValidationSchemas.ts

@@ -144,7 +144,7 @@ export namespace ValidationSchemas {
             versionCheck: joi.boolean(),
             logCollection: joi.string(),
             advisor: joi.object({
-                builtinAdvisorName: joi.string().valid('Hyperband'),
+                builtinAdvisorName: joi.string().valid('Hyperband', 'BOHB'),
                 codeDir: joi.string(),
                 classFileName: joi.string(),
                 className: joi.string(),

src/sdk/pynni/nni/bohb_advisor/bohb_advisor.py

+# Copyright (c) Microsoft Corporation
+# All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge,
+# to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction,
+# including without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and
+# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+'''
+bohb_advisor.py
+'''
+
+from enum import Enum, unique
+import sys
+import math
+import logging
+import json_tricks
+
+import ConfigSpace as CS
+import ConfigSpace.hyperparameters as CSH
+
+from nni.protocol import CommandType, send
+from nni.msg_dispatcher_base import MsgDispatcherBase
+from nni.utils import extract_scalar_reward
+
+from .config_generator import CG_BOHB
+
+logger = logging.getLogger('BOHB_Advisor')
+
+_next_parameter_id = 0
+_KEY = 'TRIAL_BUDGET'
+_epsilon = 1e-6
+
+@unique
+class OptimizeMode(Enum):
+    """Optimize Mode class"""
+    Minimize = 'minimize'
+    Maximize = 'maximize'
+
+
+def create_parameter_id():
+    """Create an id
+
+    Returns
+    -------
+    int
+        parameter id
+    """
+    global _next_parameter_id  # pylint: disable=global-statement
+    _next_parameter_id += 1
+    return _next_parameter_id - 1
+
+
+def create_bracket_parameter_id(brackets_id, brackets_curr_decay, increased_id=-1):
+    """Create a full id for a specific bracket's hyperparameter configuration
+
+    Parameters
+    ----------
+    brackets_id: int
+        brackets id
+    brackets_curr_decay: int
+        brackets curr decay
+    increased_id: int
+        increased id
+    Returns
+    -------
+    int
+        params id
+    """
+    if increased_id == -1:
+        increased_id = str(create_parameter_id())
+    params_id = '_'.join([str(brackets_id),
+                          str(brackets_curr_decay),
+                          increased_id])
+    return params_id
+
+
+class Bracket():
+    """
+    A bracket in BOHB, all the information of a bracket is managed by 
+    an instance of this class.
+
+    Parameters
+    ----------
+    s: int
+        The current Successive Halving iteration index.
+    s_max: int
+        total number of Successive Halving iterations
+    eta: float
+        In each iteration, a complete run of sequential halving is executed. In it,
+		after evaluating each configuration on the same subset size, only a fraction of
+		1/eta of them 'advances' to the next round.
+	max_budget : float
+		The largest budget to consider. Needs to be larger than min_budget!
+		The budgets will be geometrically distributed
+        :math:`a^2 + b^2 = c^2 \sim \eta^k` for :math:`k\in [0, 1, ... , num\_subsets - 1]`.
+    optimize_mode: str
+        optimize mode, 'maximize' or 'minimize'
+    """
+    def __init__(self, s, s_max, eta, max_budget, optimize_mode):
+        self.s = s
+        self.s_max = s_max
+        self.eta = eta
+        self.max_budget = max_budget
+        self.optimize_mode = optimize_mode
+
+        self.n = math.ceil((s_max + 1) * eta**s / (s + 1) - _epsilon)
+        self.r = max_budget / eta**s
+        self.i = 0
+        self.hyper_configs = []         # [ {id: params}, {}, ... ]
+        self.configs_perf = []          # [ {id: [seq, acc]}, {}, ... ]
+        self.num_configs_to_run = []    # [ n, n, n, ... ]
+        self.num_finished_configs = []  # [ n, n, n, ... ]
+        self.no_more_trial = False
+
+    def is_completed(self):
+        """check whether this bracket has sent out all the hyperparameter configurations"""
+        return self.no_more_trial
+
+    def get_n_r(self):
+        """return the values of n and r for the next round"""
+        return math.floor(self.n / self.eta**self.i + _epsilon), math.floor(self.r * self.eta**self.i +_epsilon)
+
+    def increase_i(self):
+        """i means the ith round. Increase i by 1"""
+        self.i += 1
+
+    def set_config_perf(self, i, parameter_id, seq, value):
+        """update trial's latest result with its sequence number, e.g., epoch number or batch number
+
+        Parameters
+        ----------
+        i: int
+            the ith round
+        parameter_id: int
+            the id of the trial/parameter
+        seq: int
+            sequence number, e.g., epoch number or batch number
+        value: int
+            latest result with sequence number seq
+
+        Returns
+        -------
+        None
+        """
+        if parameter_id in self.configs_perf[i]:
+            if self.configs_perf[i][parameter_id][0] < seq:
+                self.configs_perf[i][parameter_id] = [seq, value]
+        else:
+            self.configs_perf[i][parameter_id] = [seq, value]
+
+    def inform_trial_end(self, i):
+        """If the trial is finished and the corresponding round (i.e., i) has all its trials finished,
+        it will choose the top k trials for the next round (i.e., i+1)
+
+        Parameters
+        ----------
+        i: int
+            the ith round
+
+        Returns
+        -------
+        new trial or None:
+            If we have generated new trials after this trial end, we will return a new trial parameters.
+            Otherwise, we will return None.
+        """
+        global _KEY  # pylint: disable=global-statement
+        self.num_finished_configs[i] += 1
+        logger.debug('bracket id: %d, round: %d %d, finished: %d, all: %d',
+                     self.s, self.i, i, self.num_finished_configs[i], self.num_configs_to_run[i])
+        if self.num_finished_configs[i] >= self.num_configs_to_run[i] and self.no_more_trial is False:
+            # choose candidate configs from finished configs to run in the next round
+            assert self.i == i + 1
+            # finish this bracket
+            if self.i > self.s:
+                self.no_more_trial = True
+                return None
+            this_round_perf = self.configs_perf[i]
+            if self.optimize_mode is OptimizeMode.Maximize:
+                sorted_perf = sorted(this_round_perf.items(
+                ), key=lambda kv: kv[1][1], reverse=True)  # reverse
+            else:
+                sorted_perf = sorted(
+                    this_round_perf.items(), key=lambda kv: kv[1][1])
+            logger.debug(
+                'bracket %s next round %s, sorted hyper configs: %s', self.s, self.i, sorted_perf)
+            next_n, next_r = self.get_n_r()
+            logger.debug('bracket %s next round %s, next_n=%d, next_r=%d',
+                         self.s, self.i, next_n, next_r)
+            hyper_configs = dict()
+            for k in range(next_n):
+                params_id = sorted_perf[k][0]
+                params = self.hyper_configs[i][params_id]
+                params[_KEY] = next_r  # modify r
+                # generate new id
+                increased_id = params_id.split('_')[-1]
+                new_id = create_bracket_parameter_id(
+                    self.s, self.i, increased_id)
+                hyper_configs[new_id] = params
+            self._record_hyper_configs(hyper_configs)
+            return [[key, value] for key, value in hyper_configs.items()]
+        return None
+
+    def get_hyperparameter_configurations(self, num, r, config_generator):
+        """generate num hyperparameter configurations from search space using Bayesian optimization
+
+        Parameters
+        ----------
+        num: int
+            the number of hyperparameter configurations
+
+        Returns
+        -------
+        list
+            a list of hyperparameter configurations. Format: [[key1, value1], [key2, value2], ...]
+        """
+        global _KEY
+        assert self.i == 0
+        hyperparameter_configs = dict()
+        for _ in range(num):
+            params_id = create_bracket_parameter_id(self.s, self.i)
+            params = config_generator.get_config(r)
+            params[_KEY] = r
+            hyperparameter_configs[params_id] = params
+        self._record_hyper_configs(hyperparameter_configs)
+        return [[key, value] for key, value in hyperparameter_configs.items()]
+
+    def _record_hyper_configs(self, hyper_configs):
+        """after generating one round of hyperconfigs, this function records the generated hyperconfigs,
+        creates a dict to record the performance when those hyperconifgs are running, set the number of finished configs
+        in this round to be 0, and increase the round number.
+
+        Parameters
+        ----------
+        hyper_configs: list
+            the generated hyperconfigs
+        """
+        self.hyper_configs.append(hyper_configs)
+        self.configs_perf.append(dict())
+        self.num_finished_configs.append(0)
+        self.num_configs_to_run.append(len(hyper_configs))
+        self.increase_i()
+
+
+class BOHB(MsgDispatcherBase):
+    """
+    BOHB performs robust and efficient hyperparameter optimization
+    at scale by combining the speed of Hyperband searches with the
+    guidance and guarantees of convergence of Bayesian Optimization.
+    Instead of sampling new configurations at random, BOHB uses 
+    kernel density estimators to select promising candidates.
+
+    Parameters
+    ----------
+    optimize_mode: str
+        optimize mode, 'maximize' or 'minimize'
+    min_budget: float
+		The smallest budget to consider. Needs to be positive!
+	max_budget: float
+		The largest budget to consider. Needs to be larger than min_budget!
+		The budgets will be geometrically distributed
+        :math:`a^2 + b^2 = c^2 \sim \eta^k` for :math:`k\in [0, 1, ... , num\_subsets - 1]`.
+    eta: int
+		In each iteration, a complete run of sequential halving is executed. In it,
+		after evaluating each configuration on the same subset size, only a fraction of
+		1/eta of them 'advances' to the next round.
+		Must be greater or equal to 2.
+    min_points_in_model: int
+		number of observations to start building a KDE. Default 'None' means
+		dim+1, the bare minimum.
+    top_n_percent: int
+		percentage ( between 1 and 99, default 15) of the observations that are considered good.
+	num_samples: int
+		number of samples to optimize EI (default 64)
+	random_fraction: float
+		fraction of purely random configurations that are sampled from the
+		prior without the model.
+	bandwidth_factor: float
+		to encourage diversity, the points proposed to optimize EI, are sampled
+		from a 'widened' KDE where the bandwidth is multiplied by this factor (default: 3)
+	min_bandwidth: float
+		to keep diversity, even when all (good) samples have the same value for one of the parameters,
+		a minimum bandwidth (Default: 1e-3) is used instead of zero.
+    """
+    def __init__(self,
+                 optimize_mode='maximize',
+                 min_budget=1,
+                 max_budget=3,
+                 eta=3,
+                 min_points_in_model=None,
+                 top_n_percent=15,
+                 num_samples=64,
+                 random_fraction=1/3,
+                 bandwidth_factor=3,
+                 min_bandwidth=1e-3):
+        super(BOHB, self).__init__()
+        self.optimize_mode = OptimizeMode(optimize_mode)
+        self.min_budget = min_budget
+        self.max_budget = max_budget
+        self.eta = eta
+        self.min_points_in_model = min_points_in_model
+        self.top_n_percent = top_n_percent
+        self.num_samples = num_samples
+        self.random_fraction = random_fraction
+        self.bandwidth_factor = bandwidth_factor
+        self.min_bandwidth = min_bandwidth
+
+        # all the configs waiting for run
+        self.generated_hyper_configs = []
+        # all the completed configs
+        self.completed_hyper_configs = []
+
+        self.s_max = math.floor(
+            math.log(self.max_budget / self.min_budget, self.eta) + _epsilon)
+        # current bracket(s) number
+        self.curr_s = self.s_max
+        # In this case, tuner increases self.credit to issue a trial config sometime later.
+        self.credit = 0
+        self.brackets = dict()
+        self.search_space = None
+        # [key, value] = [parameter_id, parameter]
+        self.parameters = dict()
+
+        # config generator
+        self.cg = None
+
+    def load_checkpoint(self):
+        pass
+
+    def save_checkpoint(self):
+        pass
+
+    def handle_initialize(self, data):
+        """Initialize Tuner, including creating Bayesian optimization-based parametric models 
+        and search space formations
+
+        Parameters
+        ----------
+        data: search space
+            search space of this experiment
+
+        Raises
+        ------
+        ValueError
+            Error: Search space is None
+        """
+        logger.info('start to handle_initialize')
+        # convert search space jason to ConfigSpace
+        self.handle_update_search_space(data)
+
+        # generate BOHB config_generator using Bayesian optimization
+        if self.search_space:
+            self.cg = CG_BOHB(configspace=self.search_space,
+                              min_points_in_model=self.min_points_in_model,
+                              top_n_percent=self.top_n_percent,
+                              num_samples=self.num_samples,
+                              random_fraction=self.random_fraction,
+                              bandwidth_factor=self.bandwidth_factor,
+                              min_bandwidth=self.min_bandwidth)
+        else:
+            raise ValueError('Error: Search space is None')
+        # generate first brackets
+        self.generate_new_bracket()
+        send(CommandType.Initialized, '')
+
+    def generate_new_bracket(self):
+        """generate a new bracket"""
+        logger.debug(
+            'start to create a new SuccessiveHalving iteration, self.curr_s=%d', self.curr_s)
+        if self.curr_s < 0:
+            logger.info("s < 0, Finish this round of Hyperband in BOHB. Generate new round")
+            self.curr_s = self.s_max
+        self.brackets[self.curr_s] = Bracket(s=self.curr_s, s_max=self.s_max, eta=self.eta,
+                                    max_budget=self.max_budget, optimize_mode=self.optimize_mode)
+        next_n, next_r = self.brackets[self.curr_s].get_n_r()
+        logger.debug(
+            'new SuccessiveHalving iteration, next_n=%d, next_r=%d', next_n, next_r)
+        # rewrite with TPE
+        generated_hyper_configs = self.brackets[self.curr_s].get_hyperparameter_configurations(
+            next_n, next_r, self.cg)
+        self.generated_hyper_configs = generated_hyper_configs.copy()
+
+    def handle_request_trial_jobs(self, data):
+        """recerive the number of request and generate trials
+
+        Parameters
+        ----------
+        data: int
+            number of trial jobs that nni manager ask to generate
+        """
+        # Receive new request
+        self.credit += data
+
+        for _ in range(self.credit):
+            self._request_one_trial_job()
+
+    def _request_one_trial_job(self):
+        """get one trial job, i.e., one hyperparameter configuration.
+
+        If this function is called, Command will be sent by BOHB:
+        a. If there is a parameter need to run, will return "NewTrialJob" with a dict:
+        { 
+            'parameter_id': id of new hyperparameter
+            'parameter_source': 'algorithm'
+            'parameters': value of new hyperparameter
+        }
+        b. If BOHB don't have parameter waiting, will return "NoMoreTrialJobs" with
+        {
+            'parameter_id': '-1_0_0',
+            'parameter_source': 'algorithm',
+            'parameters': ''
+        }
+        """
+        if not self.generated_hyper_configs:
+            ret = {
+                'parameter_id': '-1_0_0',
+                'parameter_source': 'algorithm',
+                'parameters': ''
+            }
+            send(CommandType.NoMoreTrialJobs, json_tricks.dumps(ret))
+            return
+        assert self.generated_hyper_configs
+        params = self.generated_hyper_configs.pop()
+        ret = {
+            'parameter_id': params[0],
+            'parameter_source': 'algorithm',
+            'parameters': params[1]
+        }
+        self.parameters[params[0]] = params[1]
+        send(CommandType.NewTrialJob, json_tricks.dumps(ret))
+        self.credit -= 1
+
+    def handle_update_search_space(self, data):
+        """change json format to ConfigSpace format dict<dict> -> configspace
+
+        Parameters
+        ----------
+        data: JSON object
+            search space of this experiment
+        """
+        search_space = data
+        cs = CS.ConfigurationSpace()
+        for var in search_space:
+            _type = str(search_space[var]["_type"])
+            if _type == 'choice':
+                cs.add_hyperparameter(CSH.CategoricalHyperparameter(
+                    var, choices=search_space[var]["_value"]))
+            elif _type == 'randint':
+                cs.add_hyperparameter(CSH.UniformIntegerHyperparameter(
+                    var, lower=0, upper=search_space[var]["_value"][0]))
+            elif _type == 'uniform':
+                cs.add_hyperparameter(CSH.UniformFloatHyperparameter(
+                    var, lower=search_space[var]["_value"][0], upper=search_space[var]["_value"][1]))
+            elif _type == 'quniform':
+                cs.add_hyperparameter(CSH.UniformFloatHyperparameter(
+                    var, lower=search_space[var]["_value"][0], upper=search_space[var]["_value"][1], 
+                    q=search_space[var]["_value"][2]))
+            elif _type == 'loguniform':
+                cs.add_hyperparameter(CSH.UniformFloatHyperparameter(
+                    var, lower=search_space[var]["_value"][0], upper=search_space[var]["_value"][1], 
+                    log=True))
+            elif _type == 'qloguniform':
+                cs.add_hyperparameter(CSH.UniformFloatHyperparameter(
+                    var, lower=search_space[var]["_value"][0], upper=search_space[var]["_value"][1], 
+                    q=search_space[var]["_value"][2], log=True))
+            elif _type == 'normal':
+                cs.add_hyperparameter(CSH.NormalFloatHyperparameter(
+                    var, mu=search_space[var]["_value"][1], sigma=search_space[var]["_value"][2]))
+            elif _type == 'qnormal':
+                cs.add_hyperparameter(CSH.NormalFloatHyperparameter(
+                    var, mu=search_space[var]["_value"][1], sigma=search_space[var]["_value"][2], 
+                    q=search_space[var]["_value"][3]))
+            elif _type == 'lognormal':
+                cs.add_hyperparameter(CSH.NormalFloatHyperparameter(
+                    var, mu=search_space[var]["_value"][1], sigma=search_space[var]["_value"][2], 
+                    log=True))
+            elif _type == 'qlognormal':
+                cs.add_hyperparameter(CSH.NormalFloatHyperparameter(
+                    var, mu=search_space[var]["_value"][1], sigma=search_space[var]["_value"][2], 
+                    q=search_space[var]["_value"][3], log=True))
+            else:
+                raise ValueError(
+                    'unrecognized type in search_space, type is {}'.format(_type))
+
+        self.search_space = cs
+
+    def handle_trial_end(self, data):
+        """receive the information of trial end and generate next configuaration.
+
+        Parameters
+        ----------
+        data: dict()
+            it has three keys: trial_job_id, event, hyper_params
+            trial_job_id: the id generated by training service
+            event: the job's state
+            hyper_params: the hyperparameters (a string) generated and returned by tuner
+        """
+        logger.debug('Tuner handle trial end, result is %s', data)
+
+        hyper_params = json_tricks.loads(data['hyper_params'])
+        s, i, _ = hyper_params['parameter_id'].split('_')
+        hyper_configs = self.brackets[int(s)].inform_trial_end(int(i))
+
+        if hyper_configs is not None:
+            logger.debug(
+                'bracket %s next round %s, hyper_configs: %s', s, i, hyper_configs)
+            self.generated_hyper_configs = self.generated_hyper_configs + hyper_configs
+            for _ in range(self.credit):
+                self._request_one_trial_job()
+        # Finish this bracket and generate a new bracket
+        elif self.brackets[int(s)].no_more_trial:
+            self.curr_s -= 1
+            self.generate_new_bracket()
+            for _ in range(self.credit):
+                self._request_one_trial_job()
+
+    def handle_report_metric_data(self, data):
+        """reveice the metric data and update Bayesian optimization with final result
+
+        Parameters
+        ----------
+        data:
+            it is an object which has keys 'parameter_id', 'value', 'trial_job_id', 'type', 'sequence'.
+
+        Raises
+        ------
+        ValueError
+            Data type not supported
+        """
+        logger.debug('handle report metric data = %s', data)
+
+        assert 'value' in data
+        value = extract_scalar_reward(data['value'])
+        if self.optimize_mode is OptimizeMode.Maximize:
+            reward = -value
+        else:
+            reward = value
+        assert 'parameter_id' in data
+        s, i, _ = data['parameter_id'].split('_')
+
+        logger.debug('bracket id = %s, metrics value = %s, type = %s', s, value, data['type'])
+        s = int(s)
+
+        assert 'type' in data
+        if data['type'] == 'FINAL':
+            # and PERIODICAL metric are independent, thus, not comparable.
+            assert 'sequence' in data
+            self.brackets[s].set_config_perf(
+                int(i), data['parameter_id'], sys.maxsize, value)
+            self.completed_hyper_configs.append(data)
+       
+            _parameters = self.parameters[data['parameter_id']]
+            _parameters.pop(_KEY)
+            # update BO with loss, max_s budget, hyperparameters
+            self.cg.new_result(loss=reward, budget=data['sequence'], parameters=_parameters, update_model=True)
+        elif data['type'] == 'PERIODICAL':
+            self.brackets[s].set_config_perf(
+                int(i), data['parameter_id'], data['sequence'], value)
+        else:
+            raise ValueError(
+                'Data type not supported: {}'.format(data['type']))
+
+    def handle_add_customized_trial(self, data):
+        pass

src/sdk/pynni/nni/bohb_advisor/config_generator.py

+# BSD 3-Clause License
+# Copyright (c) 2017-2018, ML4AAD
+# All rights reserved.
+
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+
+# * Redistributions of source code must retain the above copyright notice, this
+#   list of conditions and the following disclaimer.
+
+# * Redistributions in binary form must reproduce the above copyright notice,
+#   this list of conditions and the following disclaimer in the documentation
+#   and/or other materials provided with the distribution.
+
+# * Neither the name of the copyright holder nor the names of its
+#   contributors may be used to endorse or promote products derived from
+#   this software without specific prior written permission.
+
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+import logging
+import traceback
+
+import ConfigSpace
+import ConfigSpace.hyperparameters
+import ConfigSpace.util
+import numpy as np
+import scipy.stats as sps
+import statsmodels.api as sm
+
+logger = logging.getLogger('BOHB_Advisor')
+
+class CG_BOHB(object):
+    def __init__(self, configspace, min_points_in_model=None,
+                 top_n_percent=15, num_samples=64, random_fraction=1/3,
+                 bandwidth_factor=3, min_bandwidth=1e-3):
+        """Fits for each given budget a kernel density estimator on the best N percent of the
+        evaluated configurations on this budget.
+
+
+        Parameters:
+        -----------
+        configspace: ConfigSpace
+            Configuration space object
+        top_n_percent: int
+            Determines the percentile of configurations that will be used as training data
+            for the kernel density estimator, e.g if set to 10 the 10% best configurations will be considered
+            for training.
+        min_points_in_model: int
+            minimum number of datapoints needed to fit a model
+        num_samples: int
+            number of samples drawn to optimize EI via sampling
+        random_fraction: float
+            fraction of random configurations returned
+        bandwidth_factor: float
+            widens the bandwidth for contiuous parameters for proposed points to optimize EI
+        min_bandwidth: float
+            to keep diversity, even when all (good) samples have the same value for one of the parameters,
+            a minimum bandwidth (Default: 1e-3) is used instead of zero.
+        """
+        self.top_n_percent = top_n_percent
+        self.configspace = configspace
+        self.bw_factor = bandwidth_factor
+        self.min_bandwidth = min_bandwidth
+
+        self.min_points_in_model = min_points_in_model
+        if min_points_in_model is None:
+            self.min_points_in_model = len(self.configspace.get_hyperparameters())+1
+
+        if self.min_points_in_model < len(self.configspace.get_hyperparameters())+1:
+            logger.warning('Invalid min_points_in_model value. Setting it to %i'%(len(self.configspace.get_hyperparameters())+1))
+            self.min_points_in_model =len(self.configspace.get_hyperparameters())+1
+
+        self.num_samples = num_samples
+        self.random_fraction = random_fraction
+
+        hps = self.configspace.get_hyperparameters()
+
+        self.kde_vartypes = ""
+        self.vartypes = []
+
+        for h in hps:
+            if hasattr(h, 'choices'):
+                self.kde_vartypes += 'u'
+                self.vartypes += [len(h.choices)]
+            else:
+                self.kde_vartypes += 'c'
+                self.vartypes += [0]
+
+        self.vartypes = np.array(self.vartypes, dtype=int)
+
+        # store precomputed probs for the categorical parameters
+        self.cat_probs = []
+
+        self.configs = dict()
+        self.losses = dict()
+        self.good_config_rankings = dict()
+        self.kde_models = dict()
+
+    def largest_budget_with_model(self):
+        if len(self.kde_models) == 0:
+            return(-float('inf'))
+        return(max(self.kde_models.keys()))
+
+    def sample_from_largest_budget(self, info_dict):
+        """We opted for a single multidimensional KDE compared to the
+        hierarchy of one-dimensional KDEs used in TPE. The dimensional is
+        seperated by budget. This function sample a configuration from
+        largest budget. Firstly we sample "num_samples" configurations,
+        then prefer one with the largest l(x)/g(x).
+        
+        Parameters:
+        -----------
+        info_dict: dict
+            record the information of this configuration
+
+        Returns
+        -------
+        dict:
+            new configuration named sample
+        dict:
+            info_dict, record the information of this configuration
+        """
+        best = np.inf
+        best_vector = None
+
+        budget = max(self.kde_models.keys())
+
+        l = self.kde_models[budget]['good'].pdf
+        g = self.kde_models[budget]['bad'].pdf
+
+        minimize_me = lambda x: max(1e-32, g(x))/max(l(x), 1e-32)
+
+        kde_good = self.kde_models[budget]['good']
+        kde_bad = self.kde_models[budget]['bad']
+
+        for i in range(self.num_samples):
+            idx = np.random.randint(0, len(kde_good.data))
+            datum = kde_good.data[idx]
+            vector = []
+
+            for m, bw, t in zip(datum, kde_good.bw, self.vartypes):
+
+                bw = max(bw, self.min_bandwidth)
+                if t == 0:
+                    bw = self.bw_factor*bw
+                    vector.append(sps.truncnorm.rvs(-m/bw, (1-m)/bw, loc=m, scale=bw))
+                else:
+                    if np.random.rand() < (1-bw):
+                        vector.append(int(m))
+                    else:
+                        vector.append(np.random.randint(t))
+            val = minimize_me(vector)
+
+            if not np.isfinite(val):
+                logger.warning('sampled vector: %s has EI value %s'%(vector, val))
+                logger.warning("data in the KDEs:\n%s\n%s"%(kde_good.data, kde_bad.data))
+                logger.warning("bandwidth of the KDEs:\n%s\n%s"%(kde_good.bw, kde_bad.bw))
+                logger.warning("l(x) = %s"%(l(vector)))
+                logger.warning("g(x) = %s"%(g(vector)))
+
+                # right now, this happens because a KDE does not contain all values for a categorical parameter
+                # this cannot be fixed with the statsmodels KDE, so for now, we are just going to evaluate this one
+                # if the good_kde has a finite value, i.e. there is no config with that value in the bad kde,
+                # so it shouldn't be terrible.
+                if np.isfinite(l(vector)):
+                    best_vector = vector
+                    break
+
+            if val < best:
+                best = val
+                best_vector = vector
+
+        if best_vector is None:
+            logger.debug("Sampling based optimization with %i samples failed -> using random configuration"%self.num_samples)
+            sample = self.configspace.sample_configuration().get_dictionary()
+            info_dict['model_based_pick'] = False
+
+        else:
+            logger.debug('best_vector: {}, {}, {}, {}'.format(best_vector, best, l(best_vector), g(best_vector)))
+            for i, hp_value in enumerate(best_vector):
+                if isinstance(
+                    self.configspace.get_hyperparameter(
+                        self.configspace.get_hyperparameter_by_idx(i)
+                    ),
+                    ConfigSpace.hyperparameters.CategoricalHyperparameter
+                ):
+                    best_vector[i] = int(np.rint(best_vector[i]))
+            sample = ConfigSpace.Configuration(self.configspace, vector=best_vector).get_dictionary()
+
+            sample = ConfigSpace.util.deactivate_inactive_hyperparameters(
+                configuration_space=self.configspace,
+                configuration=sample)
+            info_dict['model_based_pick'] = True
+
+        return sample, info_dict
+
+    def get_config(self, budget):
+        """Function to sample a new configuration
+        This function is called inside BOHB to query a new configuration
+
+        Parameters:
+        -----------
+        budget: float
+            the budget for which this configuration is scheduled
+
+        Returns
+        -------
+        config
+            return a valid configuration with parameters and budget
+        """
+        logger.debug('start sampling a new configuration.')
+        sample = None
+        info_dict = {}
+
+        # If no model is available, sample from prior
+        # also mix in a fraction of random configs
+        if len(self.kde_models.keys()) == 0 or np.random.rand() < self.random_fraction:
+            sample = self.configspace.sample_configuration()
+            info_dict['model_based_pick'] = False
+
+        if sample is None:
+            sample, info_dict= self.sample_from_largest_budget(info_dict)
+
+        sample = ConfigSpace.util.deactivate_inactive_hyperparameters(
+            configuration_space=self.configspace,
+            configuration=sample.get_dictionary()
+        ).get_dictionary()
+
+        logger.debug('done sampling a new configuration.')
+        sample['TRIAL_BUDGET'] = budget
+        return sample
+
+    def impute_conditional_data(self, array):
+        return_array = np.empty_like(array)
+        for i in range(array.shape[0]):
+            datum = np.copy(array[i])
+            nan_indices = np.argwhere(np.isnan(datum)).flatten()
+            while(np.any(nan_indices)):
+                nan_idx = nan_indices[0]
+                valid_indices = np.argwhere(np.isfinite(array[:,nan_idx])).flatten()
+                if len(valid_indices) > 0:
+                    # pick one of them at random and overwrite all NaN values
+                    row_idx = np.random.choice(valid_indices)
+                    datum[nan_indices] = array[row_idx, nan_indices]
+                else:
+                    # no good point in the data has this value activated, so fill it with a valid but random value
+                    t = self.vartypes[nan_idx]
+                    if t == 0:
+                        datum[nan_idx] = np.random.rand()
+                    else:
+                        datum[nan_idx] = np.random.randint(t)
+                nan_indices = np.argwhere(np.isnan(datum)).flatten()
+            return_array[i,:] = datum
+        return(return_array)
+
+    def new_result(self, loss, budget, parameters, update_model=True):
+        """
+        Function to register finished runs. Every time a run has finished, this function should be called
+        to register it with the loss.
+
+        Parameters:
+        -----------
+        loss: float
+            the loss of the parameters
+        budget: float
+            the budget of the parameters
+        parameters: dict
+            the parameters of this trial
+        update_model: bool
+            whether use this parameter to update BP model
+
+        Returns
+        -------
+        None
+        """
+        if loss is None:
+            # One could skip crashed results, but we decided
+            # assign a +inf loss and count them as bad configurations
+            loss = np.inf
+
+        if budget not in self.configs.keys():
+            self.configs[budget] = []
+            self.losses[budget] = []
+
+        # skip model building if we already have a bigger model
+        if max(list(self.kde_models.keys()) + [-np.inf]) > budget:
+            return
+
+        # We want to get a numerical representation of the configuration in the original space
+        conf = ConfigSpace.Configuration(self.configspace, parameters)
+        self.configs[budget].append(conf.get_array())
+        self.losses[budget].append(loss)
+
+        # skip model building:
+        # a) if not enough points are available
+        if len(self.configs[budget]) <= self.min_points_in_model - 1:
+            logger.debug("Only %i run(s) for budget %f available, need more than %s \
+            -> can't build model!"%(len(self.configs[budget]), budget, self.min_points_in_model+1))
+            return
+        # b) during warnm starting when we feed previous results in and only update once
+        if not update_model:
+            return
+
+        train_configs = np.array(self.configs[budget])
+        train_losses = np.array(self.losses[budget])
+
+        n_good = max(self.min_points_in_model, (self.top_n_percent * train_configs.shape[0])//100)
+        n_bad = max(self.min_points_in_model, ((100-self.top_n_percent)*train_configs.shape[0])//100)
+
+        # Refit KDE for the current budget
+        idx = np.argsort(train_losses)
+
+        train_data_good = self.impute_conditional_data(train_configs[idx[:n_good]])
+        train_data_bad = self.impute_conditional_data(train_configs[idx[n_good:n_good+n_bad]])
+
+        if train_data_good.shape[0] <= train_data_good.shape[1]:
+            return
+        if train_data_bad.shape[0] <= train_data_bad.shape[1]:
+            return
+
+        #more expensive crossvalidation method
+        #bw_estimation = 'cv_ls'
+        # quick rule of thumb
+        bw_estimation = 'normal_reference'
+
+        bad_kde = sm.nonparametric.KDEMultivariate(data=train_data_bad, var_type=self.kde_vartypes, bw=bw_estimation)
+        good_kde = sm.nonparametric.KDEMultivariate(data=train_data_good, var_type=self.kde_vartypes, bw=bw_estimation)
+
+        bad_kde.bw = np.clip(bad_kde.bw, self.min_bandwidth, None)
+        good_kde.bw = np.clip(good_kde.bw, self.min_bandwidth, None)
+
+        self.kde_models[budget] = {
+            'good': good_kde,
+            'bad' : bad_kde
+        }
+
+        # update probs for the categorical parameters for later sampling
+        logger.debug('done building a new model for budget %f based on %i/%i split\nBest loss for this budget:%f\n'
+                     %(budget, n_good, n_bad, np.min(train_losses)))

src/sdk/pynni/nni/bohb_advisor/requirements.txt

+ConfigSpace==0.4.7
+statsmodels==0.9.0
\ No newline at end of file

src/sdk/pynni/nni/constants.py

@@ -60,9 +60,11 @@ ClassArgs = {
 }
 
 AdvisorModuleName = {
-    'Hyperband': 'nni.hyperband_advisor.hyperband_advisor'
+    'Hyperband': 'nni.hyperband_advisor.hyperband_advisor',
+    'BOHB': 'nni.bohb_advisor.bohb_advisor'
 }
 
 AdvisorClassName = {
-    'Hyperband': 'Hyperband'
+    'Hyperband': 'Hyperband',
+    'BOHB': 'BOHB'
 }
\ No newline at end of file