GitLab

* some refines

* more omp refactoring

* format define

* fix merge bug

* some fixes

* fix some warnings

* Apply suggestions from code review

* Apply suggestions from code review

* remove dup codes

* reset

* fix a bug

* fix test

* Update c_api.h

* support to no filter features by min_data

* add warning in reset config

* refine warnings for override dataset's parameter

* some cleans

* clean code

* clean code

* refine C API function doxygen comments

* refined new param description

* refined doxygen comments for R API function

* removed stuff related to int8

* break long line in warning message

* removed tests which results cannot be validated anymore

* added test for warnings about unchangeable params

* write parameter from dataset to booster

* consider free_raw_data.

* fix params

* fix bug

* implementing R

* fix typo

* filter params in R

* fix R

* not min_data

* refined tests

* fixed linting

* refine

* pilint

* add docstring

* fix docstring

* R lint

* updated description for C API function

* use param aliases in Python

* fixed typo

* fixed typo

* added more params to test

* removed debug print

* fix dataset construct place

* fix merge bug

* Update feature_histogram.hpp

* add is_sparse back

* remove unused parameters

* fix lint

* add data random seed

* update

* [R-package] centrallized Dataset parameter aliases and added tests on Dataset parameter updating (#2767)
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* commit

* refactoring

* Update src/io/bin.cpp

* Apply suggestions from code review

* bug

* code clean

* remove warning

* commit

* update parameter

* fix subset bug

* typo

* add fixme tag

* bin mapper

* fix test
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* various minor style, docs and cpplint improvements

* fixed typo in warning

* fix recently added cpplint errors

* move note for params upper in description for consistency

* warning for testing col-wise and row-wise

* fix timetag

* slightly reduce the cost of multi-val bin construct.

* Update multi_val_sparse_bin.hpp

* commit

* fix a bug

* fix bug

* reset to track changes

* refine the auto choose logic

* sort the time stats output

* fix include

* change  multi_val_bin_sparse_threshold

* add cmake

* add _mm_malloc and _mm_free for cross platform

* fix cmake bug

* timer for split

* try to fix cmake

* fix tests

* refactor DataPartition::Split

* fix test

* typo

* formating

* Revert "formating"

This reverts commit 5b8de4f7fb9d975ee23701d276a66d40ee6d4222.

* add document

* [R-package] Added tests on use of force_col_wise and force_row_wise in training (#2719)

* naming

* fix gpu code

* Update include/LightGBM/bin.h
Co-Authored-By: James Lamb <jaylamb20@gmail.com>

* Update src/treelearner/ocl/histogram16.cl

* test: swap compilers for CI

* fix omp

* not avx2

* no aligned for feature histogram

* Revert "refactor DataPartition::Split"

This reverts commit 256e6d9641ade966a1f54da1752e998a1149b6f8.

* slightly refactor data partition

* reduce the memory cost
Co-authored-by: James Lamb <jaylamb20@gmail.com>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* implement

* fix warning

* fix bug

* fix a bug

* remove unneed function

* fix data push bug

* fix valid data push

* fix bug for missing_type=zero

* refine split

* renames

* typo

* add prefetch for dense bin

* prefetch for ordered bin

* Update meta.h

* Update meta.h

* Update dense_bin.hpp

* reduce the buffer when using high dimensional data in distributed mode.

* Update dataset_loader.cpp

* refix

* typo

* fix number of bin accumulation.

* avoid overflow

* fix warning

* efficient solution.

* Update dataset.h

* fix bin count output

* fix warning

* bug in dist number of feature check

* fix possible edge case

* Update dataset.cpp

* possible bug fix

* fix

* check the shape for mat, csr and csc

* guess from csr

* support file checking

* better error msg

* grammar

* clean code

* code clean

* check range for CSR

* Update test_.py

* Update test_.py

* added tests

* Fix bug where small values of max_bin cause crash.

* Revert "Fix bug where small values of max_bin cause crash."

This reverts commit fe5c8e2547057c1fa5750bcddd359dd7708fab4b.

* Add functionality to force bin thresholds.

* Fix style issues.

* Use stable sort.

* Minor style and doc fixes.

* Add functionality to force bin thresholds.

* Fix style issues.

* Use stable sort.

* Minor style and doc fixes.

* Change binning behavior to be same as PR #2342.

* Add functionality to force bin thresholds.

* Fix style issues.

* Use stable sort.

* Minor style and doc fixes.

* Add functionality to force bin thresholds.

* Fix style issues.

* Use stable sort.

* Minor style and doc fixes.

* Change binning behavior to be same as PR #2342.

* Add functionality to force bin thresholds.

* Fix style issues.

* Minor style and doc fixes.

* Add functionality to force bin thresholds.

* Fix style issues.

* Minor style and doc fixes.

* Change binning behavior to be same as PR #2342.

* Add functionality to force bin thresholds.

* Fix style issues.

* Use stable sort.

* Minor style and doc fixes.

* Add functionality to force bin thresholds.

* Fix style issues.

* Use stable sort.

* Minor style and doc fixes.

* Change binning behavior to be same as PR #2342.

* Use different bin finding function for predefined bounds.

* Fix style issues.

* Minor refactoring, overload FindBinWithZeroAsOneBin.

* Fix style issues.

* Fix bug and add new test.

* Add warning when using categorical features with forced bins.

* Pass forced_upper_bounds by reference.

* Pass container types by const reference.

* Get categorical features using FeatureBinMapper.

* Fix bug for small max_bin.

* Move GetForcedBins to DatasetLoader.

* Find forced bins in dataset_loader.

* Minor fixes.

* fix many cpp lint errors

* indent

* fix bug

* fix more

* fix gpu

* more fixes

* Add parameter max_bin_by_feature.

* Fix minor bug.

* Fix minor bug.

* Fix calculation of header size for writing binary file.

* Fix style issues.

* Fix python style issue.

* Fix test and python style issue.

* added all necessary includes - fixed build/include_what_you_use error

* fixed the order of includes (build/include_order)

* Initial attempt to implement appending features in-memory to another data set

The intent is for this to enable munging files together easily, without needing to round-trip via numpy or write multiple copies to disk.
In turn, that enables working more efficiently with data sets that were written separately.

* Implement Dataset.dump_text, and fix small bug in appending of group bin boundaries.

Dumping to text enables us to compare results, without having to worry about issues like features being reordered.

* Add basic tests for validation logic for add_features_from.

* Remove various internal mapping items from dataset text dumps

These are too sensitive to the exact feature order chosen, which is not visible to the user.
Including them in tests appears unnecessary, as the data dumping code should provide enough coverage.

* Add test that add_features_from results in identical data sets according to dump_text.

* Add test that booster behaviour after using add_features_from matches that of training on the full data

This checks:
- That training after add_features_from works at all
- That add_features_from does not cause training to misbehave

* Expose feature_penalty and monotone_types/constraints via get_field

These getters allow us to check that add_features_from does the right thing with these vectors.

* Add tests that add_features correctly handles feature_penalty and monotone_constraints.

* Ensure add_features_from properly frees the added dataset and add unit test for this

Since add_features_from moves the feature group pointers from the added dataset to the dataset being added to, the added dataset is invalid after the call.
We must ensure we do not try and access this handle.

* Remove some obsolete TODOs

* Tidy up DumpTextFile by using a single iterator for each feature

This iterators were also passed around as raw pointers without being freed, which is now fixed.

* Factor out offsetting logic in AddFeaturesFrom

* Remove obsolete TODO

* Remove another TODO

This one is debatable, test code can be a bit messy and duplicate-heavy, factoring it out tends to end badly.
Leaving this for now, will revisit if adding more tests later on becomes a mess.

* Add documentation for newly-added methods.

* Fix whitespace issues identified by pylint.

* Fix a few more whitespace issues.

* Fix doc comments

* Implement deep copying for feature groups.

* Replace awkward std::move usage by emplace_back, and reduce vector size to num_features rather than num_total_features.

* Copy feature groups in addFeaturesFrom, rather than moving them.

* Fix bugs in FeatureGroup copy constructor and ensure source dataset remains usable

* Add reserve to PushVector and PushOffset

* Move definition of Clone into class body

* Fix PR review issues

* Fix for loop increment style.

* Fix test failure

* Some more docstring fixes.

* Remove blank line

* add warnings for override parameters of Dataset

* fix pep8

* add feature_penalty

* refactor

* add R's code

* Update basic.py

* Update basic.py

* fix parameter bug

* Update lgb.Dataset.R

* fix a bug

* removed comparison warning

* fixed spacing

File: [LightGBM//src/io/dataset.cpp]
Function: [138:FastFeatureBundling(...)]

Reserving vectors where we already know the size to save on reallocation costs.

Also removed a variable that was unnecessary.

* fix ndcg consistency.

* more stable sorts

* Update gbdt_model_text.cpp

* Update dataset.cpp

* Update gbdt_model_text.cpp

* average predictions for constant features

* fix possible numerical issues in std::log.

* fix pylint

* fix bugs in c_api

* fix styles

* clean code for multi class

* rewrite test

* fix pylint

* skip test_constant_features

* refine test

* fix tests

* fix tests

* update FAQ

* fix test

* Update FAQ.rst

* add per-feature-penalites

* fix comment

* [WIP] refine config

* [wip] ready for the auto code generate

* auto generate config codes

* use with to open file

* fix bug

* fix pylint

* fix bug

* fix pylint

* fix bugs.

* tmp for failed test.

* fix tests.

* added nthreads alias

* added new aliases from new config.h

* fixed duplicated alias

* refactored parameter_generator.py

* added new aliases from config.h and removed remaining old names

* fix bugs & some miss alias

* added aliases

* add more descriptions.

* add comment.

* decode error description

* added break line char in log massages

* Shut up warnings

- warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct LightGBM::HistogramBinEntry'; use assignment or value-initialization instead [-Wclass-memaccess]
- warning: 'void* memcpy(void*, const void*, size_t)' writing to an object of type 'class std::tuple<int, double, double>' with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]

* void*

* Read and write datsets from hdfs.
* Only enabled when cmake is run with -DUSE_HDFS:BOOL=TRUE
* Introduces VirtualFile(Reader|Writer) to asbtract VFS differences