GitLab

* Initial attempt to implement appending features in-memory to another data set

The intent is for this to enable munging files together easily, without needing to round-trip via numpy or write multiple copies to disk.
In turn, that enables working more efficiently with data sets that were written separately.

* Implement Dataset.dump_text, and fix small bug in appending of group bin boundaries.

Dumping to text enables us to compare results, without having to worry about issues like features being reordered.

* Add basic tests for validation logic for add_features_from.

* Remove various internal mapping items from dataset text dumps

These are too sensitive to the exact feature order chosen, which is not visible to the user.
Including them in tests appears unnecessary, as the data dumping code should provide enough coverage.

* Add test that add_features_from results in identical data sets according to dump_text.

* Add test that booster behaviour after using add_features_from matches that of training on the full data

This checks:
- That training after add_features_from works at all
- That add_features_from does not cause training to misbehave

* Expose feature_penalty and monotone_types/constraints via get_field

These getters allow us to check that add_features_from does the right thing with these vectors.

* Add tests that add_features correctly handles feature_penalty and monotone_constraints.

* Ensure add_features_from properly frees the added dataset and add unit test for this

Since add_features_from moves the feature group pointers from the added dataset to the dataset being added to, the added dataset is invalid after the call.
We must ensure we do not try and access this handle.

* Remove some obsolete TODOs

* Tidy up DumpTextFile by using a single iterator for each feature

This iterators were also passed around as raw pointers without being freed, which is now fixed.

* Factor out offsetting logic in AddFeaturesFrom

* Remove obsolete TODO

* Remove another TODO

This one is debatable, test code can be a bit messy and duplicate-heavy, factoring it out tends to end badly.
Leaving this for now, will revisit if adding more tests later on becomes a mess.

* Add documentation for newly-added methods.

* Fix whitespace issues identified by pylint.

* Fix a few more whitespace issues.

* Fix doc comments

* Implement deep copying for feature groups.

* Replace awkward std::move usage by emplace_back, and reduce vector size to num_features rather than num_total_features.

* Copy feature groups in addFeaturesFrom, rather than moving them.

* Fix bugs in FeatureGroup copy constructor and ensure source dataset remains usable

* Add reserve to PushVector and PushOffset

* Move definition of Clone into class body

* Fix PR review issues

* Fix for loop increment style.

* Fix test failure

* Some more docstring fixes.

* Remove blank line

* convert datatable to numpy directly

* fix according to comments

* updated more docstrings

* simplified isinstance check

* Update compat.py

* added OpenMP options for python-package installation

* fixed grammar typo

* always save the score of the first round in early stopping

fix #1971

* avoid using std::log on non-positive numbers

* remove unnecessary changes

* add tests

* Update test_sklearn.py

* enhanced tests

[tests][python] added tests for metrics' behavior and fixed case for multiclass task with custom objective (#1954)

* added metrics test for standard interface

* simplified code

* less trees

* less trees

* use dummy custom objective and metric

* added tests for multiclass metrics aliases

* fixed bug in case of custom obj and num_class > 1

* added metric test for sklearn wrapper

* Fix build on macOS Mojave

Fixed #1898

- https://iscinumpy.gitlab.io/post/omp-on-high-sierra/
- https://cliutils.gitlab.io/modern-cmake/chapters/packages/OpenMP.html
- https://github.com/Homebrew/homebrew-core/pull/20589

* update setup.py

* update docs

* fix setup.py

* update docs

* update docs

* update setup.py

* update docs

* add warnings for override parameters of Dataset

* fix pep8

* add feature_penalty

* refactor

* add R's code

* Update basic.py

* Update basic.py

* fix parameter bug

* Update lgb.Dataset.R

* fix a bug

* added get_data method

* hotfix

* added warning for other data types

* reworked according to review comments

* minor addition to FAQ

* added test

* [python] fix creating train_set in fit

https://github.com/Microsoft/LightGBM/blob/cc99f0d36ae929eb02b22a072823ab7c6d3155ab/python-package/lightgbm/sklearn.py#L519
may False even if valid_data[0] is X and valid_data[1] is y actually, because `check_X_y` might return copy of X and y.
https://scikit-learn.org/0.20/modules/generated/sklearn.utils.check_X_y.html

cf. https://github.com/Microsoft/LightGBM/pull/451

* use assertIn

* disabled early stopping in dart mode

* check dart mode during the init phase of early_stop callback

* add option to disable openmp in python package build

* fix pylint

* add pandas_categorical to returned json

* save pandas_categorical to model file

* added regression test

* removed excess conversion to list

* removed deprecated line

* hotfix

* speed up the build

* remove gpu docker

* add boost ppa

* remove duplicate installs

* Update test.sh

* conda install mkl qt

* addd windows

* Update .vsts-ci.yml

* fix gpu boost

* removed misleading note about best_iteration

* Update engine.py

* Update Python-Intro.rst

* Updated Engine.py

* Updated Python-Intro.rst

* add article 'the best', break huge line and remove excess empty line

* added docstring style test and fixed errors in existing docstrings

* hotfix

* hotfix

* fix grammar

* hotfix

* downgrade Xcode version on Azure

* fixed docs and warning

* Fixed some typos in Python API Docs

* FixTypo changed validation set -> sets

* Update README.rst

* Update README.rst

* separate list item for <2.1.2 version and future -> present time

* copy old Booster's attributes to new one in refit method

* fixed according to review comment

* raise error in case of null objective

* fix indices type in csr and csc

* fix

* add indexs in shuffle model.

* fix pep

* fix bug

* fixed FutureWarning about cv default value

* fixed according to new check_estimator API

* fixed joblib warning

* break extremely large lines in basic.py

* break extremely large lines in callback.py

* break extremely large lines in engine.py

* break extremely large lines in sklearn.py

* hotfixes

* added sklearn splitter classes in cv function

* added tests