Parameters.md 22.2 KB
Newer Older
1
2
# Parameters

Guolin Ke's avatar
Guolin Ke committed
3
This is a page contains all parameters in LightGBM.
4

Guolin Ke's avatar
Guolin Ke committed
5
***List of other Helpful Links***
6
* [Python API](./Python-API.rst)
Guolin Ke's avatar
Guolin Ke committed
7
8
* [Parameters Tuning](./Parameters-tuning.md)

9
10
11
***External Links***
* [Laurae++ Interactive Documentation](https://sites.google.com/view/lauraepp/parameters)

wxchan's avatar
wxchan committed
12
***Update of 08/04/2017***
13
14
15
16
17

Default values for the following parameters have changed:

* min_data_in_leaf = 100 => 20
* min_sum_hessian_in_leaf = 10 => 1e-3
Guolin Ke's avatar
Guolin Ke committed
18
19
* num_leaves = 127 => 31
* num_iterations = 10 => 100
Guolin Ke's avatar
Guolin Ke committed
20

21
## Parameter Format
22

23
The parameter format is `key1=value1 key2=value2 ... ` . And parameters can be set both in config file and command line. By using command line, parameters should not have spaces before and after `=`. By using config files, one line can only contain one parameter. you can use `#` to comment. If one parameter appears in both command line and config file, LightGBM will use the parameter in command line.
24
25
26

## Core Parameters

27
* `config`, default=`""`, type=string, alias=`config_file`
28
  * path of config file
29
30
31
* `task`, default=`train`, type=enum, options=`train`,`prediction`
  * `train` for training
  * `prediction` for prediction.
32
  * `convert_model` for converting model file into if-else format, see more information in [Convert model parameters](#convert-model-parameters)
33
34
35
36
37
38
39
* `application`, default=`regression`, type=enum, options=`regression`,`regression_l1`,`huber`,`fair`,`poisson`,`binary`,`lambdarank`,`multiclass`, alias=`objective`,`app`
  * `regression`, regression application
    * `regression_l2`, L2 loss, alias=`mean_squared_error`,`mse`
    * `regression_l1`, L1 loss, alias=`mean_absolute_error`,`mae`
    * `huber`, [Huber loss](https://en.wikipedia.org/wiki/Huber_loss "Huber loss - Wikipedia")
    * `fair`, [Fair loss](https://www.kaggle.com/c/allstate-claims-severity/discussion/24520)
    * `poisson`, [Poisson regression](https://en.wikipedia.org/wiki/Poisson_regression "Poisson regression")
40
  * `binary`, binary classification application
41
  * `lambdarank`, [lambdarank](https://papers.nips.cc/paper/2971-learning-to-rank-with-nonsmooth-cost-functions.pdf) application
42
43
    * The label should be `int` type in lambdarank tasks, and larger number represent the higher relevance (e.g. 0:bad, 1:fair, 2:good, 3:perfect).
    * `label_gain` can be used to set the gain(weight) of `int` label.
44
  * `multiclass`, multi-class classification application, should set `num_class` as well
Guolin Ke's avatar
Guolin Ke committed
45
* `boosting`, default=`gbdt`, type=enum, options=`gbdt`,`rf`,`dart`,`goss`, alias=`boost`,`boosting_type`
46
  * `gbdt`, traditional Gradient Boosting Decision Tree
Guolin Ke's avatar
Guolin Ke committed
47
  * `rf`, Random Forest
48
49
50
  * `dart`, [Dropouts meet Multiple Additive Regression Trees](https://arxiv.org/abs/1505.01866)
  * `goss`, Gradient-based One-Side Sampling
* `data`, default=`""`, type=string, alias=`train`,`train_data`
51
  * training data, LightGBM will train from this data
52
* `valid`, default=`""`, type=multi-string, alias=`test`,`valid_data`,`test_data`
53
  * validation/test data, LightGBM will output metrics for these data
54
55
  * support multi validation data, separate by `,`
* `num_iterations`, default=`100`, type=int, alias=`num_iteration`,`num_tree`,`num_trees`,`num_round`,`num_rounds`
56
  * number of boosting iterations
i3v's avatar
i3v committed
57
58
  * note: For python/R package, **this parameter is ignored**, use `num_boost_round` (Python) or `nrounds` (R) input arguments of `train` and `cv` methods instead
  * note: internally, LightGBM constructs `num_class * num_iterations` trees for `multiclass` problems
59
* `learning_rate`, default=`0.1`, type=double, alias=`shrinkage_rate`
60
  * shrinkage rate
61
62
  * in `dart`, it also affects normalization weights of dropped trees
* `num_leaves`, default=`31`, type=int, alias=`num_leaf`
63
  * number of leaves in one tree
64
65
66
67
* `tree_learner`, default=`serial`, type=enum, options=`serial`,`feature`,`data`
  * `serial`, single machine tree learner
  * `feature`, feature parallel tree learner
  * `data`, data parallel tree learner
68
  * Refer to [Parallel Learning Guide](./Parallel-Learning-Guide.rst) to get more details.
69
* `num_threads`, default=OpenMP_default, type=int, alias=`num_thread`,`nthread`
70
  * Number of threads for LightGBM.
71
  * For the best speed, set this to the number of **real CPU cores**, not the number of threads (most CPU using [hyper-threading](https://en.wikipedia.org/wiki/Hyper-threading) to generate 2 threads per CPU core).
Laurae's avatar
Laurae committed
72
73
  * Do not set it too large if your dataset is small (do not use 64 threads for a dataset with 10,000 for instance).
  * Be aware a task manager or any similar CPU monitoring tool might report cores not being fully utilized. This is normal.
74
  * For parallel learning, should not use full CPU cores since this will cause poor performance for the network.
75
* `device`, default=`cpu`, options=`cpu`,`gpu`
Guolin Ke's avatar
Guolin Ke committed
76
  * Choose device for the tree learning, can use gpu to achieve the faster learning.
77
  * Note: 1. Recommend use the smaller `max_bin`(e.g `63`) to get the better speed up. 2. For the faster speed, GPU use 32-bit float point to sum up by default, may affect the accuracy for some tasks. You can set `gpu_use_dp=true` to enable 64-bit float point, but it will slow down the training. 3. Refer to [Installation Guide](./Installation-Guide.rst) to build with GPU .
Guolin Ke's avatar
Guolin Ke committed
78

79

80
81
## Learning Control Parameters

82
* `max_depth`, default=`-1`, type=int
83
84
  * Limit the max depth for tree model. This is used to deal with overfit when #data is small. Tree still grow by leaf-wise.
  * `< 0` means no limit
85
* `min_data_in_leaf`, default=`20`, type=int, alias=`min_data_per_leaf` , `min_data`
86
  * Minimal number of data in one leaf. Can use this to deal with over-fit.
87
88
89
90
* `min_sum_hessian_in_leaf`, default=`1e-3`, type=double, alias=`min_sum_hessian_per_leaf`, `min_sum_hessian`, `min_hessian`
  * Minimal sum hessian in one leaf. Like `min_data_in_leaf`, can use this to deal with over-fit.
* `feature_fraction`, default=`1.0`, type=double, `0.0 < feature_fraction < 1.0`, alias=`sub_feature`
  * LightGBM will random select part of features on each iteration if `feature_fraction` smaller than `1.0`. For example, if set to `0.8`, will select 80% features before training each tree.
91
92
  * Can use this to speed up training
  * Can use this to deal with over-fit
93
* `feature_fraction_seed`, default=`2`, type=int
94
  * Random seed for feature fraction.
95
* `bagging_fraction`, default=`1.0`, type=double, , `0.0 < bagging_fraction < 1.0`, alias=`sub_row`
96
  * Like `feature_fraction`, but this will random select part of data without resampling
wxchan's avatar
wxchan committed
97
  * Can use this to speed up training
98
  * Can use this to deal with over-fit
99
100
101
102
103
  * Note: To enable bagging, should set `bagging_freq` to a non zero value as well
* `bagging_freq`, default=`0`, type=int
  * Frequency for bagging, `0` means disable bagging. `k` means will perform bagging at every `k` iteration.
  * Note: To enable bagging, should set `bagging_fraction` as well
* `bagging_seed` , default=`3`, type=int
104
  * Random seed for bagging.
105
106
107
* `early_stopping_round` , default=`0`, type=int, alias=`early_stopping_rounds`,`early_stopping`
  * Will stop training if one metric of one validation data doesn't improve in last `early_stopping_round` rounds.
* `lambda_l1` , default=`0`, type=double
108
  * l1 regularization
109
* `lambda_l2` , default=`0`, type=double
110
  * l2 regularization
111
* `min_gain_to_split` , default=`0`, type=double
112
  * The minimal gain to perform split
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
* `drop_rate`, default=`0.1`, type=double
  * only used in `dart`
* `skip_drop`, default=`0.5`, type=double
  * only used in `dart`, probability of skipping drop
* `max_drop`, default=`50`, type=int
  * only used in `dart`, max number of dropped trees on one iteration. `<=0` means no limit.
* `uniform_drop`, default=`false`, type=bool
  * only used in `dart`, true if want to use uniform drop
* `xgboost_dart_mode`, default=`false`, type=bool
  * only used in `dart`, true if want to use xgboost dart mode
* `drop_seed`, default=`4`, type=int
  * only used in `dart`, used to random seed to choose dropping models.
* `top_rate`, default=`0.2`, type=double
  * only used in `goss`,  the retain ratio of large gradient data
* `other_rate`, default=`0.1`, type=int
  * only used in `goss`,  the retain ratio of small gradient data
Guolin Ke's avatar
Guolin Ke committed
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
* `max_cat_group`, default=`64`, type=int
  * use for the categorical features.
  * When #catogory is large, finding the split point on it is easily over-fitting. So LightGBM merges them into `max_cat_group` groups, and finds the split points on the group boundaries.
* `min_data_per_group`, default=`10`, type=int
  * Min number of data per categorical group.
* `max_cat_threshold`, default=`256`, type=int
  * use for the categorical features. Limit the max threshold points in categorical features.
* `min_cat_smooth`, default=`5`, type=double
  * use for the categorical features. Refer to the descrption in paramater `cat_smooth_ratio`.
* `max_cat_smooth`, default=`100`, type=double
  * use for the categorical features. Refer to the descrption in paramater `cat_smooth_ratio`.
* `cat_smooth_ratio`, default=`0.01`, type=double
  * use for the categorical features. This can reduce the effect of noises in categorical features, especially for categories with few data.
  * The smooth denominator is `a = min(max_cat_smooth, max(min_cat_smooth, num_data/num_category*cat_smooth_ratio))`.
  * The smooth numerator  is `b = a * sum_gradient / sum_hessian`.
144
145


146
## IO Parameters
147

148
* `max_bin`, default=`255`, type=int
149
  * max number of bin that feature values will bucket in. Small bin may reduce training accuracy but may increase general power (deal with over-fit).
150
  * LightGBM will auto compress memory according `max_bin`. For example, LightGBM will use `uint8_t` for feature value if `max_bin=255`.
151
152
* `min_data_in_bin`, default=`5`, type=int
  * min number of data inside one bin, use this to avoid one-data-one-bin (may over-fitting).
153
* `data_random_seed`, default=`1`, type=int
154
  * random seed for data partition in parallel learning(not include feature parallel).
155
* `output_model`, default=`LightGBM_model.txt`, type=string, alias=`model_output`,`model_out`
156
  * file name of output model in training.
157
* `input_model`, default=`""`, type=string, alias=`model_input`,`model_in`
158
159
160
  * file name of input model.
  * for prediction task, will prediction data using this model.
  * for train task, will continued train from this model.
161
* `output_result`, default=`LightGBM_predict_result.txt`, type=string, alias=`predict_result`,`prediction_result`
162
  * file name of prediction result in prediction task.
163
* `is_pre_partition`, default=`false`, type=bool
164
  * used for parallel learning(not include feature parallel).
165
166
167
168
  * `true` if training data are pre-partitioned, and different machines using different partition.
* `is_sparse`, default=`true`, type=bool, alias=`is_enable_sparse`
  * used to enable/disable sparse optimization. Set to `false` to disable sparse optimization.
* `two_round`, default=`false`, type=bool, alias=`two_round_loading`,`use_two_round_loading`
169
  * by default, LightGBM will map data file to memory and load features from memory. This will provide faster data loading speed. But it may out of memory when the data file is very big.
170
171
172
173
174
175
176
177
  * set this to `true` if data file is too big to fit in memory.
* `save_binary`, default=`false`, type=bool, alias=`is_save_binary`,`is_save_binary_file`
  * set this to `true` will save the data set(include validation data) to a binary file. Speed up the data loading speed for the next time.
* `verbosity`, default=`1`, type=int, alias=`verbose`
  * `<0` = Fatel, `=0` = Error(Warn), `>0` = Info
* `header`, default=`false`, type=bool, alias=`has_header`
  * `true` if input data has header
* `label`, default=`""`, type=string, alias=`label_column`
178
  * specific the label column
179
180
181
  * Use number for index, e.g. `label=0` means column_0 is the label
  * Add a prefix `name:` for column name, e.g. `label=name:is_click`
* `weight`, default=`""`, type=string, alias=`weight_column`
182
  * specific the weight column
183
184
185
186
  * Use number for index, e.g. `weight=0` means column_0 is the weight
  * Add a prefix `name:` for column name, e.g. `weight=name:weight`
  * Note: Index start from `0`. And it doesn't count the label column when passing type is Index. e.g. when label is  column_0, and weight is column_1, the correct parameter is `weight=0`.
* `query`, default=`""`, type=string, alias=`query_column`,`group`,`group_column`
187
  * specific the query/group id column
188
189
190
191
  * Use number for index, e.g. `query=0` means column_0 is the query id
  * Add a prefix `name:` for column name, e.g. `query=name:query_id`
  * Note: Data should group by query_id. Index start from `0`. And it doesn't count the label column when passing type is Index. e.g. when label is  column_0, and query_id is column_1, the correct parameter is `query=0`.
* `ignore_column`, default=`""`, type=string, alias=`ignore_feature`,`blacklist`
192
  * specific some ignore columns in training
193
194
195
196
  * Use number for index, e.g. `ignore_column=0,1,2` means column_0, column_1 and column_2 will be ignored.
  * Add a prefix `name:` for column name, e.g. `ignore_column=name:c1,c2,c3` means c1, c2 and c3 will be ignored.
  * Note: Index start from `0`. And it doesn't count the label column.
* `categorical_feature`, default=`""`, type=string, alias=`categorical_column`,`cat_feature`,`cat_column`
197
  * specific categorical features
198
199
  * Use number for index, e.g. `categorical_feature=0,1,2` means column_0, column_1 and column_2 are categorical features.
  * Add a prefix `name:` for column name, e.g. `categorical_feature=name:c1,c2,c3` means c1, c2 and c3 are categorical features.
Guolin Ke's avatar
Guolin Ke committed
200
  * Note: Only support categorical with `int` type (Note: the negative values will be treated as Missing values). Index start from `0`. And it doesn't count the label column.
201
* `predict_raw_score`, default=`false`, type=bool, alias=`raw_score`,`is_predict_raw_score`
202
  * only used in prediction task
203
204
  * Set to `true` will only predict the raw scores.
  * Set to `false` will transformed score
205
* `predict_leaf_index`, default=`false`, type=bool, alias=`leaf_index`,`is_predict_leaf_index`
206
  * only used in prediction task
207
  * Set to `true` to predict with leaf index of all trees
208
209
210
* `predict_contrib`, default=`false`, type=bool, alias=`contrib`,`is_predict_contrib`
  * only used in prediction task
  * Set to `true` to estimate [SHAP values](https://arxiv.org/abs/1706.06060), which represent how each feature contributed to each prediction. Produces number of features + 1 values where the last value is the expected value of the model output over the training data.
Guolin Ke's avatar
Guolin Ke committed
211
* `bin_construct_sample_cnt`, default=`200000`, type=int
212
213
214
  * Number of data that sampled to construct histogram bins.
  * Will give better training result when set this larger. But will increase data loading time.
  * Set this to larger value if data is very sparse.
215
* `num_iteration_predict`, default=`-1`, type=int
216
  * only used in prediction task, used to how many trained iterations will be used in prediction.
217
  * `<= 0` means no limit
218
219
220
221
222
* `pred_early_stop`, default=`false`, type=bool
  * Set to `true` will use early-stopping to speed up the prediction. May affect the accuracy.
* `pred_early_stop_freq`, default=`10`, type=int
  * The frequency of checking early-stopping prediction.
* `pred_early_stop_margin`, default=`10.0`, type=double
223
  * The Threshold of margin in early-stopping prediction.
224
* `use_missing`, default=`true`, type=bool
Seong-Jin Kim's avatar
Seong-Jin Kim committed
225
  * Set to `false` will disable the special handle of missing value.
Guolin Ke's avatar
Guolin Ke committed
226
227
228
* `zero_as_missing`, default=`false`, type=bool
  * Set to `true` will treat all zero as missing values (including the unshown values in libsvm/sparse matrics).
  * Set to `false` will use `na` to represent missing values.
229
230
231
232
233
* `init_score_file`, default=`""`, type=string
  * Path of training initial score file, `""` will use `train_data_file+".init"` (if exists).
* `valid_init_score_file`, default=`""`, type=multi-string
  * Path of validation initial score file, `""` will use `valid_data_file+".init"` (if exists).
  * separate by `,` for multi-validation data
234

235
236

## Objective Parameters
237

238
* `sigmoid`, default=`1.0`, type=double
239
  * parameter for sigmoid function. Will be used in binary classification and lambdarank.
240
* `huber_delta`, default=`1.0`, type=double
Tsukasa OMOTO's avatar
Tsukasa OMOTO committed
241
  * parameter for [Huber loss](https://en.wikipedia.org/wiki/Huber_loss "Huber loss - Wikipedia"). Will be used in regression task.
242
* `fair_c`, default=`1.0`, type=double
wxchan's avatar
wxchan committed
243
  * parameter for [Fair loss](https://www.kaggle.com/c/allstate-claims-severity/discussion/24520). Will be used in regression task.
244
245
* `gaussian_eta`, default=`1.0`, type=double
  * parameter to control the width of Gaussian function. Will be used in l1 and huber regression loss.
246
* `poission_max_delta_step`, default=`0.7`, type=double
247
  * parameter used to safeguard optimization
248
* `scale_pos_weight`, default=`1.0`, type=double
249
  * weight of positive class in binary classification task
250
* `boost_from_average`, default=`true`, type=bool
251
  * adjust initial score to the mean of labels for faster convergence, only used in Regression task.
252
253
254
* `is_unbalance`, default=`false`, type=bool
  * used in binary classification. Set this to `true` if training data are unbalance.
* `max_position`, default=`20`, type=int
255
  * used in lambdarank, will optimize NDCG at this position.
256
257
258
259
* `label_gain`, default=`0,1,3,7,15,31,63,...`, type=multi-double
  * used in lambdarank, relevant gain for labels. For example, the gain of label `2` is `3` if using default label gains.
  * Separate by `,`
* `num_class`, default=`1`, type=int, alias=`num_classes`
260
261
  * only used in multi-class classification

262
263

## Metric Parameters
264

265
266
267
268
269
270
271
272
* `metric`, default={`l2` for regression}, {`binary_logloss` for binary classification},{`ndcg` for lambdarank}, type=multi-enum, options=`l1`,`l2`,`ndcg`,`auc`,`binary_logloss`,`binary_error`...
  * `l1`, absolute loss, alias=`mean_absolute_error`, `mae`
  * `l2`, square loss, alias=`mean_squared_error`, `mse`
  * `l2_root`, root square loss, alias=`root_mean_squared_error`, `rmse`
  * `huber`, [Huber loss](https://en.wikipedia.org/wiki/Huber_loss "Huber loss - Wikipedia")
  * `fair`, [Fair loss](https://www.kaggle.com/c/allstate-claims-severity/discussion/24520)
  * `poisson`, [Poisson regression](https://en.wikipedia.org/wiki/Poisson_regression "Poisson regression")
  * `ndcg`, [NDCG](https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG)
273
  * `map`, [MAP](https://en.wikipedia.org/wiki/Information_retrieval#Mean_average_precision)
274
275
276
277
278
279
280
  * `auc`, [AUC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve)
  * `binary_logloss`, [log loss](https://www.kaggle.com/wiki/LogLoss)
  * `binary_error`. For one sample `0` for correct classification, `1` for error classification.
  * `multi_logloss`, log loss for mulit-class classification
  * `multi_error`. error rate for mulit-class classification
  * Support multi metrics, separate by `,`
* `metric_freq`, default=`1`, type=int
281
  * frequency for metric output
282
* `is_training_metric`, default=`false`, type=bool
283
  * set this to true if need to output metric result of training
284
285
* `ndcg_at`, default=`1,2,3,4,5`, type=multi-int, alias=`ndcg_eval_at`,`eval_at`
  * NDCG evaluation position, separate by `,`
286

287
288

## Network Parameters
289

290
Following parameters are used for parallel learning, and only used for base(socket) version.
291

292
* `num_machines`, default=`1`, type=int, alias=`num_machine`
293
  * Used for parallel learning, the number of machines for parallel learning application
294
  * Need to set this in both socket and mpi version.
295
* `local_listen_port`, default=`12400`, type=int, alias=`local_port`
296
  * TCP listen port for local machines.
297
  * Should allow this port in firewall setting before training.
298
* `time_out`, default=`120`, type=int
299
  * Socket time-out in minutes.
300
* `machine_list_file`, default=`""`, type=string
301
  * File that list machines for this parallel learning application
302
  * Each line contains one IP and one port for one machine. The format is `ip port`, separate by space.
303

304
305

## GPU Parameters
Guolin Ke's avatar
Guolin Ke committed
306

307
* `gpu_platform_id`, default=`-1`, type=int
Guolin Ke's avatar
Guolin Ke committed
308
309
  * OpenCL platform ID. Usually each GPU vendor exposes one OpenCL platform.
  * Default value is -1, using the system-wide default platform.
310
* `gpu_device_id`, default=`-1`, type=int
311
312
  * OpenCL device ID in the specified platform. Each GPU in the selected platform has a unique device ID.
  * Default value is -1, using the default device in the selected platform.
313
* `gpu_use_dp`, default=`false`, type=bool
Guolin Ke's avatar
Guolin Ke committed
314
315
  * Set to true to use double precision math on GPU (default using single precision).

316
317

## Convert Model Parameters
wxchan's avatar
wxchan committed
318
319
320
321
322
323
324
325
326
327

This feature is only supported in command line version yet.

* `convert_model_language`, default=`""`, type=string
  * only `cpp` is supported yet.
  * if `convert_model_language` is set when `task` is set to `train`, the model will also be converted.

* `convert_model`, default=`"gbdt_prediction.cpp"`, type=string
  * output file name of converted model.

328
329
## Others

330
331
### Continued Training with Input Score

332
333
334
335
336
337
338
339
340
LightGBM support continued train with initial score. It uses an additional file to store these initial score, like the following:

```
0.5
-0.1
0.9
...
```

341
It means the initial score of first data is `0.5`, second is `-0.1`, and so on. The initial score file corresponds with data file line by line, and has per score per line. And if the name of data file is "train.txt", the initial score file should be named as "train.txt.init" and in the same folder as the data file. And LightGBM will auto load initial score file if it exists.
342
343


344
345
### Weight Data

346
347
348
349
350
351
352
353
354
LightGBM support weighted training. It uses an additional file to store weight data, like the following:

```
1.0
0.5
0.8
...
```

355
It means the weight of first data is `1.0`, second is `0.5`, and so on. The weight file corresponds with data file line by line, and has per weight per line. And if the name of data file is "train.txt", the weight file should be named as "train.txt.weight" and in the same folder as the data file. And LightGBM will auto load weight file if it exists.
356
357

update:
358
You can specific weight column in data file now. Please refer to parameter `weight` in above.
359

360
361

### Query Data
362
363
364
365
366
367
368
369
370
371

For LambdaRank learning, it needs query information for training data. LightGBM use an additional file to store query data. Following is an example:

```
27
18
67
...
```

372
It means first `27` lines samples belong one query and next `18` lines belong to another, and so on.(**Note: data should order by query**) If name of data file is "train.txt", the query file should be named as "train.txt.query" and in same folder of training data. LightGBM will load the query file automatically if it exists.
373

374
You can specific query/group id in data file now. Please refer to parameter `group` in above.