Update parallel tpe (#1373)

* update readme in ga_squad

* update readme

* fix typo

* Update README.md

* Update README.md

* Update README.md

* update readme

* update

* fix path

* update reference

* fix bug in config file

* update nni_arch_overview.png

* update

* update

* update

* update home page

* update default value of metis tuner

* fix broken link in CommunitySharings

* update docs about nested search space

* update docs

* rename cascding to nested

* fix broken link

* update

* update issue link

* fix typo

* update evaluate parameters from GMM

* refine code

* fix optimized mode bug

* update import warning

* update warning

* update optimized mode

* fix parallel_tpe bug and refine

* fix bug

* fix kwargs bug

* fix bug

* update the default value of parallel_tpe

* add docs for parallel-tpe

* fix config_schema bug

* update community doc

* fix typo

* update doc for parallel_tpe

* fix bug in receive trial result

* update by comments

* update by comments

* fix self.optimal_y is 0 when mean mode

* update

* update

* update rst

* fix typo | add docstring

docs/en_US/CommunitySharings/ParallelizingTpeSearch.md

+# Parallelizing a Sequential Algorithm TPE
+
+TPE approaches were actually run asynchronously in order to make use of multiple compute nodes and to avoid wasting time waiting for trial evaluations to complete. For the TPE approach, the so-called constant liar approach was used: each time a candidate point x∗ was proposed, a fake fitness evaluation of the y was assigned temporarily, until the evaluation completed and reported the actual loss f(x∗).
+
+## Introducion and Problems
+
+### Sequential Model-based Global Optimization
+
+Sequential Model-Based Global Optimization (SMBO) algorithms have been used in many applications where evaluation of the fitness function is expensive. In an application where the true fitness function f: X → R is costly to evaluate, model-based algorithms approximate f with a surrogate that is cheaper to evaluate. Typically the inner loop in an SMBO algorithm is the numerical optimization of this surrogate, or some transformation of the surrogate. The point x∗ that maximizes the surrogate (or its transformation) becomes the proposal for where the true function f should be evaluated. This active-learning-like algorithm template is summarized in the figure below. SMBO algorithms differ in what criterion they optimize to obtain x∗ given a model (or surrogate) of f, and in they model f via observation history H.
+
+![](../../img/parallel_tpe_search4.PNG)
+
+The algorithms in this work optimize the criterion of Expected Improvement (EI). Other criteria have been suggested, such as Probability of Improvement and Expected Improvement, minimizing the Conditional Entropy of the Minimizer, and the bandit-based criterion. We chose to use the EI criterion in TPE because it is intuitive, and has been shown to work well in a variety of settings. Expected improvement is the expectation under some model M of f : X → RN that f(x) will exceed (negatively) some threshold y∗:
+
+![](../../img/parallel_tpe_search_ei.PNG)
+
+Since calculation of p(y|x) is expensive, TPE approach modeled p(y|x) by p(x|y) and p(y).The TPE defines p(x|y) using two such densities:
+
+![](../../img/parallel_tpe_search_tpe.PNG)
+
+where l(x) is the density formed by using the observations {x(i)} such that corresponding loss
+f(x(i)) was less than y∗ and g(x) is the density formed by using the remaining observations. TPE algorithm depends on a y∗ that is larger than the best observed f(x) so that some points can be used to form l(x). The TPE algorithm chooses y∗ to be some quantile γ of the observed y values, so that p(y<`y∗`) = γ, but no specific model for p(y) is necessary. The tree-structured form of l and g makes it easy to draw manycandidates according to l and evaluate them according to g(x)/l(x). On each iteration, the algorithm returns the candidate x∗ with the greatest EI.
+
+Here is a simulation of the TPE algorithm in a two-dimensional search space. The difference of background color represents different values. It can be seen that TPE combines exploration and exploitation very well. (Black indicates the points of this round samples, and yellow indicates the points has been taken in the history.)
+
+![](../../img/parallel_tpe_search1.gif)
+
+**Since EI is a continuous function, the highest x of EI is determined at a certain status.** As shown in the figure below, the blue triangle is the point that is most likely to be sampled in this state.
+
+![](../../img/parallel_tpe_search_ei2.PNG)
+
+TPE performs well when we use it in sequential, but if we provide a larger concurrency, then **there will be a large number of points produced in the same EI state**, too concentrated points will reduce the exploration ability of the tuner, resulting in resources waste.
+
+Here is the simulation figure when we set `concurrency=60`, It can be seen that this phenomenon is obvious.
+
+![](../../img/parallel_tpe_search2.gif)
+
+## Research solution
+
+### Approximated q-EI Maximization
+
+The multi-points criterion that we have presented below can potentially be used to deliver an additional design of experiments in one step through the resolution of the optimization problem.
+
+![](../../img/parallel_tpe_search_qEI.PNG)
+
+However, the computation of q-EI becomes intensive as q increases. After our research, there are four popular greedy strategies that approach the result of problem while avoiding its numerical cost.
+
+#### Solution 1: Believing the OK Predictor: The KB(Kriging Believer) Heuristic Strategy
+
+The Kriging Believer strategy replaces the conditional knowledge about the responses at the sites chosen within the last iterations by deterministic values equal to the expectation of the Kriging predictor. Keeping the same notations as previously, the strategy can be summed up as follows:
+
+![](../../img/parallel_tpe_search_kb.PNG)
+
+This sequential strategy delivers a q-points design and is computationally affordable since it relies on the analytically known EI, optimized in d dimensions. However, there is a risk of failure, since believing an OK predictor that overshoots the observed data may lead to a sequence that gets trapped in a non-optimal region for many iterations. We now propose a second strategy that reduces this risk.
+
+#### Solution 2: The CL(Constant Liar) Heuristic Strategy
+
+Let us now consider a sequential strategy in which the metamodel is updated (still without hyperparameter re-estimation) at each iteration with a value L exogenously fixed by the user, here called a ”lie”. The strategy referred to as the Constant Liar consists in lying with the same value L at every iteration: maximize EI (i.e. find xn+1), actualize the model as if y(xn+1) = L, and so on always with the same L ∈ R:
+
+![](../../img/parallel_tpe_search_cl.PNG)
+
+L should logically be determined on the basis of the values taken by y at X. Three values, min{Y}, mean{Y}, and max{Y} are considered here. **The larger L is, the more explorative the algorithm will be, and vice versa.**
+
+We have simulated the method above. The following figure shows the result of using mean value liars to maximize q-EI. We find that the points we have taken have begun to be scattered.
+
+![](../../img/parallel_tpe_search3.gif)
+
+## Experiment
+
+### Branin-Hoo
+
+The four optimization strtigeies presented in the last section are now complared on the Branin-Hoo function which is a classical test-case in global optimization.
+
+![](../../img/parallel_tpe_search_branin.PNG)
+
+The recommended values of a, b, c, r, s and t are: a = 1, b = 5.1 ⁄ (4π2), c = 5 ⁄ π, r = 6, s = 10 and t = 1 ⁄ (8π). This function has three global minimizers(-3.14, 12.27), (3.14, 2.27), (9.42, 2.47).
+
+Next is the comparaison of the q-EI associated with the q first points (q ∈ [1,10]) given by the constant liar strategies (min and max), 2000 q-points designs uniformly drawn for every q, and 2000 q-points LHS designs taken at random for every q.
+
+![](../../img/parallel_tpe_search_result.PNG)
+
+As we can seen on figure, CL[max] and CL[min] offer very good q-EI results compared to random designs, especially for small values of q.
+
+### Gaussian Mixed Model function
+
+We also compared the case of using parallel optimization and not using parallel optimization. A two-dimensional multimodal Gaussian Mixed distribution is used to simulate, the following is our result:
+
+||concurrency=80|concurrency=60|concurrency=40|concurrency=20|concurrency=10|
+|---|---|---|---|---|---|
+|Without parallel optimization|avg =  0.4841 <br> var =  0.1953|avg =  0.5155 <br> var =  0.2219|avg =  0.5773 <br> var =  0.2570|avg =  0.4680 <br> var =  0.1994| avg = 0.2774 <br> var = 0.1217|
+|With parallel optimization|avg =  0.2132 <br> var = 0.0700|avg =  0.2177<br>var =  0.0796| avg =  0.1835 <br> var =  0.0533|avg =  0.1671 <br> var =  0.0413|avg =  0.1918 <br> var =  0.0697|
+
+Note: The total number of samples per test is 240 (ensure that the budget is equal). The trials in each form were repeated 1000 times, the value is the average and variance of the best results in 1000 trials.
+
+## References
+
+[1] James Bergstra, Remi Bardenet, Yoshua Bengio, Balazs Kegl. "Algorithms for Hyper-Parameter Optimization". [Link](https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf)
+
+[2] Meng-Hiot Lim, Yew-Soon Ong. "Computational Intelligence in Expensive Optimization Problems". [Link](https://link.springer.com/content/pdf/10.1007%2F978-3-642-10701-6.pdf)
+
+[3] M. Jordan, J. Kleinberg, B. Scho¨lkopf. "Pattern Recognition and Machine Learning". [Link](http://users.isr.ist.utl.pt/~wurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf)

docs/en_US/CommunitySharings/community_sharings.rst

@@ -10,3 +10,4 @@ In addtion to the official tutorilas and examples, we encourage community contri
     NNI in Recommenders <RecommendersSvd>
     Neural Architecture Search Comparison <NasComparision>
     Hyper-parameter Tuning Algorithm Comparsion <HpoComparision>
+    Parallelizing Optimization for TPE <ParallelizingTpeSearch>

docs/en_US/Tuner/BuiltinTuner.md

@@ -42,6 +42,8 @@ TPE, as a black-box optimization, can be used in various scenarios and shows goo
 
 * **optimize_mode** (*maximize or minimize, optional, default = maximize*) - If 'maximize', the tuner will target to maximize metrics. If 'minimize', the tuner will target to minimize metrics.
 
+Note: We have optimized the parallelism of TPE for large-scale trial-concurrency. For the principle of optimization or turn-on optimization, please refer to [TPE document](HyperoptTuner.md).
+
 **Usage example:**
 
 ```yaml

docs/en_US/Tuner/HyperoptTuner.md

@@ -5,10 +5,33 @@ TPE, Random Search, Anneal Tuners on NNI
 
 The Tree-structured Parzen Estimator (TPE) is a sequential model-based optimization (SMBO) approach. SMBO methods sequentially construct models to approximate the performance of hyperparameters based on historical measurements, and then subsequently choose new hyperparameters to test based on this model. The TPE approach models P(x|y) and P(y) where x represents hyperparameters and y the associated evaluate matric. P(x|y) is modeled by transforming the generative process of hyperparameters, replacing the distributions of the configuration prior with non-parametric densities. This optimization approach is described in detail in [Algorithms for Hyper-Parameter Optimization](https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf). ​
 
+### Parallel TPE optimization
+
+TPE approaches were actually run asynchronously in order to make use of multiple compute nodes and to avoid wasting time waiting for trial evaluations to complete. The original intention of the algorithm design is to optimize sequential. When we use TPE with a large concurrency, its performance will be bad. We have optimized this phenomenon using Constant Liar algorithm. For the principle of optimization, please refer to our [research blog](../CommunitySharings/ParallelizingTpeSearch.md).
+
+### Usage
+
+ To use TPE, you should add the following spec in your experiment's YAML config file:
+
+ ```yaml
+tuner:
+  builtinTunerName: TPE
+  classArgs:
+    optimize_mode: maximize
+    parallel_optimize: True
+    constant_liar_type: min
+```
+
+**Requirement of classArg**
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*) - If 'maximize', tuners will target to maximize metrics. If 'minimize', tuner will target to minimize metrics.
+* **parallel_optimize** (*bool, optional, default = False*) - If True, TPE will use Constant Liar algorithm to optimize parallel hyperparameter tuning. Otherwise, TPE will not discriminate between sequential or parallel situations.
+* **constant_liar_type** (*min or max or mean, optional, default = min*) - The type of constant liar to use, will logically be determined on the basis of the values taken by y at X. Corresponding to three values, min{Y}, max{Y}, and mean{Y}.
+
+
 ## Random Search
 
 In [Random Search for Hyper-Parameter Optimization](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf) show that Random Search might be surprisingly simple and effective. We suggests that we could use Random Search as baseline when we have no knowledge about the prior distribution of hyper-parameters.
 
 ## Anneal
 
-This simple annealing algorithm begins by sampling from the prior, but tends over time to sample from points closer and closer to the best ones observed. This algorithm is a simple variation on random search that leverages smoothness in the response surface. The annealing rate is not adaptive.
+This simple annealing algorithm begins by sampling from the prior, but tends over time to sample from points closer and closer to the best ones observed. This algorithm is a simple variation on random search that leverages smoothness in the response surface. The annealing rate is not adaptive.
\ No newline at end of file

src/sdk/pynni/nni/hyperopt_tuner/hyperopt_tuner.py

@@ -190,13 +190,19 @@ class HyperoptTuner(Tuner):
     HyperoptTuner is a tuner which using hyperopt algorithm.
     """
 
-    def __init__(self, algorithm_name, optimize_mode='minimize'):
+    def __init__(self, algorithm_name, optimize_mode='minimize', 
+                 parallel_optimize=False, constant_liar_type='min'):
         """
         Parameters
         ----------
         algorithm_name : str
             algorithm_name includes "tpe", "random_search" and anneal".
         optimize_mode : str
+        parallel_optimize : bool 
+            More detail could reference: docs/en_US/Tuner/HyperoptTuner.md
+        constant_liar_type : str
+            constant_liar_type including "min", "max" and "mean"
+            More detail could reference: docs/en_US/Tuner/HyperoptTuner.md
         """
         self.algorithm_name = algorithm_name
         self.optimize_mode = OptimizeMode(optimize_mode)
@@ -205,6 +211,13 @@ class HyperoptTuner(Tuner):
         self.rval = None
         self.supplement_data_num = 0
 
+        self.parallel = parallel_optimize
+        if self.parallel:
+            self.CL_rval = None
+            self.constant_liar_type = constant_liar_type
+            self.running_data = []
+            self.optimal_y = None
+
     def _choose_tuner(self, algorithm_name):
         """
         Parameters
@@ -266,6 +279,10 @@ class HyperoptTuner(Tuner):
             # but it can cause deplicate parameter rarely
             total_params = self.get_suggestion(random_search=True)
         self.total_data[parameter_id] = total_params
+
+        if self.parallel:
+            self.running_data.append(parameter_id)
+        
         params = split_index(total_params)
         return params
 
@@ -287,10 +304,39 @@ class HyperoptTuner(Tuner):
             raise RuntimeError('Received parameter_id not in total_data.')
         params = self.total_data[parameter_id]
 
+        # code for parallel
+        if self.parallel:
+            constant_liar = kwargs.get('constant_liar', False)
+
+            if constant_liar:
+                rval = self.CL_rval
+            else:
+                rval = self.rval
+                self.running_data.remove(parameter_id)
+
+                # update the reward of optimal_y
+                if self.optimal_y is None:
+                    if self.constant_liar_type == 'mean':
+                        self.optimal_y = [reward, 1]
+                    else:
+                        self.optimal_y = reward
+                else:
+                    if self.constant_liar_type == 'mean':
+                        _sum = self.optimal_y[0] + reward
+                        _number = self.optimal_y[1] + 1
+                        self.optimal_y = [_sum, _number]
+                    elif self.constant_liar_type == 'min':
+                        self.optimal_y = min(self.optimal_y, reward)
+                    elif self.constant_liar_type == 'max':
+                        self.optimal_y = max(self.optimal_y, reward)
+                logger.debug("Update optimal_y with reward, optimal_y = %s", self.optimal_y)
+        else:
+            rval = self.rval
+
+
         if self.optimize_mode is OptimizeMode.Maximize:
             reward = -reward
 
-        rval = self.rval
         domain = rval.domain
         trials = rval.trials
 
@@ -375,13 +421,26 @@ class HyperoptTuner(Tuner):
         total_params : dict
             parameter suggestion
         """
+        if self.parallel and len(self.total_data)>20 and len(self.running_data) and self.optimal_y is not None:
+            self.CL_rval = copy.deepcopy(self.rval)
+            if self.constant_liar_type == 'mean':
+                _constant_liar_y = self.optimal_y[0] / self.optimal_y[1]
+            else:
+                _constant_liar_y = self.optimal_y
+            for _parameter_id in self.running_data:
+                self.receive_trial_result(parameter_id=_parameter_id, parameters=None, value=_constant_liar_y, constant_liar=True)
+            rval = self.CL_rval
 
-        rval = self.rval
+            random_state = np.random.randint(2**31 - 1)
+        else:
+            rval = self.rval
+            random_state = rval.rstate.randint(2**31 - 1)
+    
         trials = rval.trials
         algorithm = rval.algo
         new_ids = rval.trials.new_trial_ids(1)
         rval.trials.refresh()
-        random_state = rval.rstate.randint(2**31 - 1)
+
         if random_search:
             new_trials = hp.rand.suggest(new_ids, rval.domain, trials,
                                          random_state)

tools/nni_cmd/config_schema.py

@@ -92,6 +92,16 @@ tuner_schema_dict = {
         Optional('includeIntermediateResults'): setType('includeIntermediateResults', bool),
         Optional('gpuNum'): setNumberRange('gpuNum', int, 0, 99999),
     },
+    'TPE': {
+        'builtinTunerName': 'TPE',
+        'classArgs': {
+            Optional('optimize_mode'): setChoice('optimize_mode', 'maximize', 'minimize'),
+            Optional('parallel_optimize'): setType('parallel_optimize', bool),
+            Optional('constant_liar_type'): setChoice('constant_liar_type', 'min', 'max', 'mean')
+        },
+        Optional('includeIntermediateResults'): setType('includeIntermediateResults', bool),
+        Optional('gpuNum'): setNumberRange('gpuNum', int, 0, 99999),
+    },
     'NetworkMorphism': {
         'builtinTunerName': 'NetworkMorphism',
         'classArgs': {