GitLab

Fix access violation exception that can occur during invocation of loop lambda function when inner_start >= inner_end in 'For' template (#3936)

* Fix index out-of-range exception generated by BaggingHelper on small datasets.

Prior to this change, the line "score_t threshold = tmp_gradients[top_k - 1];" would generate an exception, since tmp_gradients would be empty when the cnt input value to the function is zero.

* Update goss.hpp

* Update goss.hpp

* Add API method LGBM_BoosterPredictForMats which runs prediction on a data set given as of array of pointers to rows (as opposed to existing method LGBM_BoosterPredictForMat which requires data given as contiguous array)

* Fix incorrect upstream merge

* Add link to LightGBM.NET

* Fix indenting to 2 spaces

* Dummy edit to trigger CI

* Dummy edit to trigger CI

* remove duplicate functions from merge

* Fix access violation exception that can occur during invocation of loop lambda function when inner_start >= inner_end in 'For' template.

In particular, this can occur in Tree::AddPredictionToScore on line 291 where the loop lambda function body (created by the PredictionFun macro) dereferences used_data_indices[start].

For reference, the particular case which triggered this exception in my case was:
  * start = 0
  * end = 93,203
  * n_block = 56
  * min_block_size = 512
for which the BlockInfo method gave:
  * n_block = 56
  * num_inner = 1,696
and so, for the case i=55 (i.e., the last case in the loop), we get
  * inner_start = start + num_inner * i
                = 93,280
which is greater than 'end' and hence triggers the exception.

* Change formatting of proposed modification
Co-authored-by: matthew-peacock <matthew.peacock@whiteoakam.com>
Co-authored-by: Guolin Ke <guolin.ke@outlook.com>

* remove max_block_size_ in train states (fix #3570)

* avoid zero elements per row

* add min constraint for min_block_size_

* store without offset in multi_val_dense_bin

* fix offset bug

* add comment for offset

* add comment for bin type selection

* faster operations for offset

* keep most freq bin in histogram for multi val dense

* use original feature iterators

* consider 9 cases (3 x 3) for multi val bin construction

* fix dense bin setting

* fix bin data in multi val group

* fix offset of the first feature histogram

* use float hist buf

* avx in histogram construction

* use avx for hist construction without prefetch

* vectorize bin extraction

* use only 128 vec

* use avx2

* use vectorization for sparse row wise

* add bit size for multi val dense bin

* float with no vectorization

* change multithreading strategy to dynamic

* remove intrinsic header

* fix dense multi val col copy

* remove bit size

* use large enough block size when the bin number is large

* calc min block size by sparsity

* rescale gradients

* rollback gradients scaling

* single precision histogram buffer as an option

* add float hist buffer with thread buffer

* fix setting zero in hist data

* fix hist begin pointer in tree learners

* remove debug logs

* remove omp simd

* update Makevars of R-package

* fix feature group binary storing

* two row wise for double hist buffer

* add subfeature for two row wise

* remove useless code and fix two row wise

* refactor code

* grouping the dense feature groups can get sparse multi val bin

* clean format problems

* one thread for two blocks in sep row wise

* use ordered gradients for sep row wise

* fix grad ptr

* ordered grad with combined block for sep row wise

* fix block threading

* use the same min block size

* rollback share min block size

* remove logs

* Update src/io/dataset.cpp
Co-authored-by: Guolin Ke <guolin.ke@outlook.com>

* fix parameter description

* remove sep_row_wise

* remove check codes

* add check for empty multi val bin

* fix lint error

* rollback changes in config.h

* Apply suggestions from code review
Co-authored-by: Ubuntu <shiyu@gbdt-04.ren3kv4wanvufliwrpy4k03lsf.xx.internal.cloudapp.net>
Co-authored-by: Guolin Ke <guolin.ke@outlook.com>

* rollback omp sum

* remove sum reduction

* speed up multi-threading sum

* Apply suggestions from code review

* Update serial_tree_learner.cpp

* Update src/treelearner/serial_tree_learner.cpp

* stable multi-threading reduction

* Update src/treelearner/serial_tree_learner.cpp

* more fixes

* Apply suggestions from code review

* Apply suggestions from code review

* Update src/boosting/gbdt.cpp

This reverts commit 656d2676.

* only one fix

* add more

* add more

* don't cache `num_thread`, to avoid change outside

* rename

* update document

* Update docs/Parameters.rst

* Update include/LightGBM/config.h

* Apply suggestions from code review
Co-Authored-By: Nikita Titov <nekit94-08@mail.ru>

* Apply suggestions from code review
Co-Authored-By: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* fix bagging

* fixed cpplint issues

* updated docs
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* some refines

* more omp refactoring

* format define

* fix merge bug

* some fixes

* fix some warnings

* Apply suggestions from code review

* Apply suggestions from code review

* remove dup codes

* added all necessary includes - fixed build/include_what_you_use error

* fixed the order of includes (build/include_order)

* Added possibility to compile without openmp (useful for clang on Mac)

* Added CMake option to create dylib on MacOS