Parameters.md 18.3 KB
Newer Older
1
2
# Parameters

3
4
This is a page contains all parameters in LightGBM.

Guolin Ke's avatar
Guolin Ke committed
5
6
7
8
***List of other Helpful Links***
* [Python API Reference](./Python-API.md)
* [Parameters Tuning](./Parameters-tuning.md)

9
10
11
***External Links***
* [Laurae++ Interactive Documentation](https://sites.google.com/view/lauraepp/parameters)

12
13
14
15
16
17
***Update of 04/13/2017***

Default values for the following parameters have changed:

* min_data_in_leaf = 100 => 20
* min_sum_hessian_in_leaf = 10 => 1e-3
Guolin Ke's avatar
Guolin Ke committed
18
19
* num_leaves = 127 => 31
* num_iterations = 10 => 100
Guolin Ke's avatar
Guolin Ke committed
20

21
22
## Parameter format

23
The parameter format is `key1=value1 key2=value2 ... ` . And parameters can be set both in config file and command line. By using command line, parameters should not have spaces before and after `=`. By using config files, one line can only contain one parameter. you can use `#` to comment. If one parameter appears in both command line and config file, LightGBM will use the parameter in command line.
24
25
26

## Core Parameters

27
* `config`, default=`""`, type=string, alias=`config_file`
28
  * path of config file
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
* `task`, default=`train`, type=enum, options=`train`,`prediction`
  * `train` for training
  * `prediction` for prediction.
* `application`, default=`regression`, type=enum, options=`regression`,`regression_l1`,`huber`,`fair`,`poisson`,`binary`,`lambdarank`,`multiclass`, alias=`objective`,`app`
  * `regression`, regression application
    * `regression_l2`, L2 loss, alias=`mean_squared_error`,`mse`
    * `regression_l1`, L1 loss, alias=`mean_absolute_error`,`mae`
    * `huber`, [Huber loss](https://en.wikipedia.org/wiki/Huber_loss "Huber loss - Wikipedia")
    * `fair`, [Fair loss](https://www.kaggle.com/c/allstate-claims-severity/discussion/24520)
    * `poisson`, [Poisson regression](https://en.wikipedia.org/wiki/Poisson_regression "Poisson regression")
  * `binary`, binary classification application 
  * `lambdarank`, [lambdarank](https://pdfs.semanticscholar.org/fc9a/e09f9ced555558fdf1e997c0a5411fb51f15.pdf) application
  * `multiclass`, multi-class classification application, should set `num_class` as well
* `boosting`, default=`gbdt`, type=enum, options=`gbdt`,`dart`, alias=`boost`,`boosting_type`
  * `gbdt`, traditional Gradient Boosting Decision Tree 
  * `dart`, [Dropouts meet Multiple Additive Regression Trees](https://arxiv.org/abs/1505.01866)
  * `goss`, Gradient-based One-Side Sampling
* `data`, default=`""`, type=string, alias=`train`,`train_data`
47
  * training data, LightGBM will train from this data
48
* `valid`, default=`""`, type=multi-string, alias=`test`,`valid_data`,`test_data`
49
  * validation/test data, LightGBM will output metrics for these data
50
51
  * support multi validation data, separate by `,`
* `num_iterations`, default=`100`, type=int, alias=`num_iteration`,`num_tree`,`num_trees`,`num_round`,`num_rounds`
52
  * number of boosting iterations
53
  * note: `num_tree` here equal with `num_iterations`. For multi-class, it actually learns `num_class * num_iterations` trees.
54
  * note: For python/R package, cannot use this parameters to control number of iterations.
55
* `learning_rate`, default=`0.1`, type=double, alias=`shrinkage_rate`
56
  * shrinkage rate
57
58
  * in `dart`, it also affects normalization weights of dropped trees
* `num_leaves`, default=`31`, type=int, alias=`num_leaf`
59
  * number of leaves in one tree
60
61
62
63
* `tree_learner`, default=`serial`, type=enum, options=`serial`,`feature`,`data`
  * `serial`, single machine tree learner
  * `feature`, feature parallel tree learner
  * `data`, data parallel tree learner
Guolin Ke's avatar
Guolin Ke committed
64
  * Refer to [Parallel Learning Guide](./Parallel-Learning-Guide.md) to get more details.
65
* `num_threads`, default=OpenMP_default, type=int, alias=`num_thread`,`nthread`
66
67
68
  * Number of threads for LightGBM. 
  * For the best speed, set this to the number of **real CPU cores**, not the number of threads (most CPU using [hyper-threading](https://en.wikipedia.org/wiki/Hyper-threading) to generate 2 threads per CPU core).
  * For parallel learning, should not use full CPU cores since this will cause poor performance for the network.
69
* `device`, default=`cpu`, options=`cpu`,`gpu`
Guolin Ke's avatar
Guolin Ke committed
70
  * Choose device for the tree learning, can use gpu to achieve the faster learning.
71
  * Note: 1. Recommend use the smaller `max_bin`(e.g `63`) to get the better speed up. 2. For the faster speed, GPU use 32-bit float point to sum up by default, may affect the accuracy for some tasks. You can set `gpu_use_dp=true` to enable 64-bit float point, but it will slow down the training. 3. Refer to [Installation Guide](https://github.com/Microsoft/LightGBM/wiki/Installation-Guide#with-gpu-support) to build with GPU .
Guolin Ke's avatar
Guolin Ke committed
72

73
74

## Learning control parameters
75
* `max_depth`, default=`-1`, type=int
76
  * Limit the max depth for tree model. This is used to deal with overfit when #data is small. Tree still grow by leaf-wise. 
77
78
  * `< 0` means no limit 
* `min_data_in_leaf`, default=`20`, type=int, alias=`min_data_per_leaf` , `min_data`
79
  * Minimal number of data in one leaf. Can use this to deal with over-fit.
80
81
82
83
* `min_sum_hessian_in_leaf`, default=`1e-3`, type=double, alias=`min_sum_hessian_per_leaf`, `min_sum_hessian`, `min_hessian`
  * Minimal sum hessian in one leaf. Like `min_data_in_leaf`, can use this to deal with over-fit.
* `feature_fraction`, default=`1.0`, type=double, `0.0 < feature_fraction < 1.0`, alias=`sub_feature`
  * LightGBM will random select part of features on each iteration if `feature_fraction` smaller than `1.0`. For example, if set to `0.8`, will select 80% features before training each tree.
84
85
  * Can use this to speed up training
  * Can use this to deal with over-fit
86
* `feature_fraction_seed`, default=`2`, type=int
87
  * Random seed for feature fraction.
88
89
* `bagging_fraction`, default=`1.0`, type=double, , `0.0 < bagging_fraction < 1.0`, alias=`sub_row`
  * Like `feature_fraction`, but this will random select part of data
wxchan's avatar
wxchan committed
90
  * Can use this to speed up training
91
  * Can use this to deal with over-fit
92
93
94
95
96
  * Note: To enable bagging, should set `bagging_freq` to a non zero value as well
* `bagging_freq`, default=`0`, type=int
  * Frequency for bagging, `0` means disable bagging. `k` means will perform bagging at every `k` iteration.
  * Note: To enable bagging, should set `bagging_fraction` as well
* `bagging_seed` , default=`3`, type=int
97
  * Random seed for bagging.
98
99
100
* `early_stopping_round` , default=`0`, type=int, alias=`early_stopping_rounds`,`early_stopping`
  * Will stop training if one metric of one validation data doesn't improve in last `early_stopping_round` rounds.
* `lambda_l1` , default=`0`, type=double
101
  * l1 regularization 
102
* `lambda_l2` , default=`0`, type=double
103
  * l2 regularization 
104
* `min_gain_to_split` , default=`0`, type=double
105
  * The minimal gain to perform split 
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
* `drop_rate`, default=`0.1`, type=double
  * only used in `dart`
* `skip_drop`, default=`0.5`, type=double
  * only used in `dart`, probability of skipping drop
* `max_drop`, default=`50`, type=int
  * only used in `dart`, max number of dropped trees on one iteration. `<=0` means no limit.
* `uniform_drop`, default=`false`, type=bool
  * only used in `dart`, true if want to use uniform drop
* `xgboost_dart_mode`, default=`false`, type=bool
  * only used in `dart`, true if want to use xgboost dart mode
* `drop_seed`, default=`4`, type=int
  * only used in `dart`, used to random seed to choose dropping models.
* `top_rate`, default=`0.2`, type=double
  * only used in `goss`,  the retain ratio of large gradient data
* `other_rate`, default=`0.1`, type=int
  * only used in `goss`,  the retain ratio of small gradient data
122
123
124
125


## IO parameters

126
* `max_bin`, default=`255`, type=int
127
  * max number of bin that feature values will bucket in. Small bin may reduce training accuracy but may increase general power (deal with over-fit).
128
129
  * LightGBM will auto compress memory according `max_bin`. For example, LightGBM will use `uint8_t` for feature value if `max_bin=255`.
* `data_random_seed`, default=`1`, type=int
130
  * random seed for data partition in parallel learning(not include feature parallel).
131
* `output_model`, default=`LightGBM_model.txt`, type=string, alias=`model_output`,`model_out`
132
  * file name of output model in training.
133
* `input_model`, default=`""`, type=string, alias=`model_input`,`model_in`
134
135
136
  * file name of input model.
  * for prediction task, will prediction data using this model.
  * for train task, will continued train from this model.
137
* `output_result`, default=`LightGBM_predict_result.txt`, type=string, alias=`predict_result`,`prediction_result`
138
  * file name of prediction result in prediction task.
139
* `is_pre_partition`, default=`false`, type=bool
140
  * used for parallel learning(not include feature parallel).
141
142
143
144
  * `true` if training data are pre-partitioned, and different machines using different partition.
* `is_sparse`, default=`true`, type=bool, alias=`is_enable_sparse`
  * used to enable/disable sparse optimization. Set to `false` to disable sparse optimization.
* `two_round`, default=`false`, type=bool, alias=`two_round_loading`,`use_two_round_loading`
145
  * by default, LightGBM will map data file to memory and load features from memory. This will provide faster data loading speed. But it may out of memory when the data file is very big.
146
147
148
149
150
151
152
153
  * set this to `true` if data file is too big to fit in memory.
* `save_binary`, default=`false`, type=bool, alias=`is_save_binary`,`is_save_binary_file`
  * set this to `true` will save the data set(include validation data) to a binary file. Speed up the data loading speed for the next time.
* `verbosity`, default=`1`, type=int, alias=`verbose`
  * `<0` = Fatel, `=0` = Error(Warn), `>0` = Info
* `header`, default=`false`, type=bool, alias=`has_header`
  * `true` if input data has header
* `label`, default=`""`, type=string, alias=`label_column`
154
  * specific the label column
155
156
157
  * Use number for index, e.g. `label=0` means column_0 is the label
  * Add a prefix `name:` for column name, e.g. `label=name:is_click`
* `weight`, default=`""`, type=string, alias=`weight_column`
158
  * specific the weight column
159
160
161
162
  * Use number for index, e.g. `weight=0` means column_0 is the weight
  * Add a prefix `name:` for column name, e.g. `weight=name:weight`
  * Note: Index start from `0`. And it doesn't count the label column when passing type is Index. e.g. when label is  column_0, and weight is column_1, the correct parameter is `weight=0`.
* `query`, default=`""`, type=string, alias=`query_column`,`group`,`group_column`
163
  * specific the query/group id column
164
165
166
167
  * Use number for index, e.g. `query=0` means column_0 is the query id
  * Add a prefix `name:` for column name, e.g. `query=name:query_id`
  * Note: Data should group by query_id. Index start from `0`. And it doesn't count the label column when passing type is Index. e.g. when label is  column_0, and query_id is column_1, the correct parameter is `query=0`.
* `ignore_column`, default=`""`, type=string, alias=`ignore_feature`,`blacklist`
168
  * specific some ignore columns in training
169
170
171
172
  * Use number for index, e.g. `ignore_column=0,1,2` means column_0, column_1 and column_2 will be ignored.
  * Add a prefix `name:` for column name, e.g. `ignore_column=name:c1,c2,c3` means c1, c2 and c3 will be ignored.
  * Note: Index start from `0`. And it doesn't count the label column.
* `categorical_feature`, default=`""`, type=string, alias=`categorical_column`,`cat_feature`,`cat_column`
173
  * specific categorical features
174
175
176
177
  * Use number for index, e.g. `categorical_feature=0,1,2` means column_0, column_1 and column_2 are categorical features.
  * Add a prefix `name:` for column name, e.g. `categorical_feature=name:c1,c2,c3` means c1, c2 and c3 are categorical features.
  * Note: Only support categorical with `int` type. Index start from `0`. And it doesn't count the label column.
* `predict_raw_score`, default=`false`, type=bool, alias=`raw_score`,`is_predict_raw_score`
178
  * only used in prediction task
179
180
181
  * Set to `true` will only predict the raw scores.
  * Set to `false` will transformed score
* `predict_leaf_index `, default=`false`, type=bool, alias=`leaf_index `,`is_predict_leaf_index `
182
  * only used in prediction task
183
  * Set to `true` to predict with leaf index of all trees
Guolin Ke's avatar
Guolin Ke committed
184
* `bin_construct_sample_cnt`, default=`200000`, type=int
185
186
187
  * Number of data that sampled to construct histogram bins.
  * Will give better training result when set this larger. But will increase data loading time.
  * Set this to larger value if data is very sparse.
188
* `num_iteration_predict`, default=`-1`, type=int
189
  * only used in prediction task, used to how many trained iterations will be used in prediction. 
190
  * `<= 0` means no limit
191
192
193
194


## Objective parameters

195
* `sigmoid`, default=`1.0`, type=double
196
  * parameter for sigmoid function. Will be used in binary classification and lambdarank.
197
* `huber_delta`, default=`1.0`, type=double
Tsukasa OMOTO's avatar
Tsukasa OMOTO committed
198
  * parameter for [Huber loss](https://en.wikipedia.org/wiki/Huber_loss "Huber loss - Wikipedia"). Will be used in regression task.
199
* `fair_c`, default=`1.0`, type=double
wxchan's avatar
wxchan committed
200
  * parameter for [Fair loss](https://www.kaggle.com/c/allstate-claims-severity/discussion/24520). Will be used in regression task.
201
* `poission_max_delta_step`, default=`0.7`, type=double
202
  * parameter used to safeguard optimization
203
* `scale_pos_weight`, default=`1.0`, type=double
204
  * weight of positive class in binary classification task
205
* `boost_from_average`, default=`true`, type=bool
206
  * adjust initial score to the mean of labels for faster convergence, only used in Regression task.
207
208
209
* `is_unbalance`, default=`false`, type=bool
  * used in binary classification. Set this to `true` if training data are unbalance.
* `max_position`, default=`20`, type=int
210
  * used in lambdarank, will optimize NDCG at this position.
211
212
213
214
* `label_gain`, default=`0,1,3,7,15,31,63,...`, type=multi-double
  * used in lambdarank, relevant gain for labels. For example, the gain of label `2` is `3` if using default label gains.
  * Separate by `,`
* `num_class`, default=`1`, type=int, alias=`num_classes`
215
216
217
218
  * only used in multi-class classification

## Metric parameters

219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
* `metric`, default={`l2` for regression}, {`binary_logloss` for binary classification},{`ndcg` for lambdarank}, type=multi-enum, options=`l1`,`l2`,`ndcg`,`auc`,`binary_logloss`,`binary_error`...
  * `l1`, absolute loss, alias=`mean_absolute_error`, `mae`
  * `l2`, square loss, alias=`mean_squared_error`, `mse`
  * `l2_root`, root square loss, alias=`root_mean_squared_error`, `rmse`
  * `huber`, [Huber loss](https://en.wikipedia.org/wiki/Huber_loss "Huber loss - Wikipedia")
  * `fair`, [Fair loss](https://www.kaggle.com/c/allstate-claims-severity/discussion/24520)
  * `poisson`, [Poisson regression](https://en.wikipedia.org/wiki/Poisson_regression "Poisson regression")
  * `ndcg`, [NDCG](https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG)
  * `map`, [MAP](https://www.kaggle.com/wiki/MeanAveragePrecision)
  * `auc`, [AUC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve)
  * `binary_logloss`, [log loss](https://www.kaggle.com/wiki/LogLoss)
  * `binary_error`. For one sample `0` for correct classification, `1` for error classification.
  * `multi_logloss`, log loss for mulit-class classification
  * `multi_error`. error rate for mulit-class classification
  * Support multi metrics, separate by `,`
* `metric_freq`, default=`1`, type=int
235
  * frequency for metric output
236
* `is_training_metric`, default=`false`, type=bool
237
  * set this to true if need to output metric result of training
238
239
* `ndcg_at`, default=`1,2,3,4,5`, type=multi-int, alias=`ndcg_eval_at`,`eval_at`
  * NDCG evaluation position, separate by `,`
240
241
242

## Network parameters

Guolin Ke's avatar
Guolin Ke committed
243
Following parameters are used for parallel learning, and only used for base(socket) version. 
244

245
* `num_machines`, default=`1`, type=int, alias=`num_machine`
246
  * Used for parallel learning, the number of machines for parallel learning application
Guolin Ke's avatar
Guolin Ke committed
247
  * Need to set this in both socket and mpi version. 
248
* `local_listen_port`, default=`12400`, type=int, alias=`local_port`
249
  * TCP listen port for local machines.
Guolin Ke's avatar
Guolin Ke committed
250
  * Should allow this port in firewall setting before training. 
251
* `time_out`, default=`120`, type=int
252
  * Socket time-out in minutes.
253
* `machine_list_file`, default=`""`, type=string
254
  * File that list machines for this parallel learning application
255
  * Each line contains one IP and one port for one machine. The format is `ip port`, separate by space.
256

Guolin Ke's avatar
Guolin Ke committed
257
258
## GPU parameters

259
* `gpu_platform_id`, default=`-1`, type=int
Guolin Ke's avatar
Guolin Ke committed
260
261
  * OpenCL platform ID. Usually each GPU vendor exposes one OpenCL platform.
  * Default value is -1, using the system-wide default platform.
262
* `gpu_device_id`, default=`-1`, type=int
Guolin Ke's avatar
Guolin Ke committed
263
264
  * OpenCL device ID in the specified platform. Each GPU in the selected platform has a unique device ID. 
  * Default value is -1, using the default device in the selected platform. 
265
* `gpu_use_dp`, default=`false`, type=bool
Guolin Ke's avatar
Guolin Ke committed
266
267
  * Set to true to use double precision math on GPU (default using single precision).

268
269
270
271
272
273
274
275
276
277
278
279
## Others

### Continued training with input score
LightGBM support continued train with initial score. It uses an additional file to store these initial score, like the following:

```
0.5
-0.1
0.9
...
```

280
It means the initial score of first data is `0.5`, second is `-0.1`, and so on. The initial score file corresponds with data file line by line, and has per score per line. And if the name of data file is "train.txt", the initial score file should be named as "train.txt.init" and in the same folder as the data file. And LightGBM will auto load initial score file if it exists. 
281
282
283
284
285
286
287
288
289
290
291
292


### Weight data
LightGBM support weighted training. It uses an additional file to store weight data, like the following:

```
1.0
0.5
0.8
...
```

293
It means the weight of first data is `1.0`, second is `0.5`, and so on. The weight file corresponds with data file line by line, and has per weight per line. And if the name of data file is "train.txt", the weight file should be named as "train.txt.weight" and in the same folder as the data file. And LightGBM will auto load weight file if it exists.
294
295

update:
296
You can specific weight column in data file now. Please refer to parameter `weight` in above.
297
298
299
300
301
302
303
304
305
306
307
308

### Query data

For LambdaRank learning, it needs query information for training data. LightGBM use an additional file to store query data. Following is an example:

```
27
18
67
...
```

309
It means first `27` lines samples belong one query and next `18` lines belong to another, and so on.(**Note: data should order by query**) If name of data file is "train.txt", the query file should be named as "train.txt.query" and in same folder of training data. LightGBM will load the query file automatically if it exists.
310

311
You can specific query/group id in data file now. Please refer to parameter `group` in above.