GitLab

* fix Dask docstrings and mimic sklearn importing way

* Update .vsts-ci.yml

* revert CI checks

* use import aliases for Dask classes

* check Dask is installed in _predict() func

* fix lint issues introduced during resolving merge conflicts

* Update dask.py

* [dask] allow parameter aliases for tree_learner and local_listen_port (fixes #3671)

* num_thread too

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* empty commit

* add _choose_param_value

* revert param order change

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update python-package/lightgbm/dask.py
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* just import deepcopy

* remove machines aliases

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* run artifacts packaging job on Ubuntu

* Update .vsts-ci.yml

* TST Migrates test_sklearn.py to pytest

* STY Fixes linting

* FIX Adds reason

* ENH Address comments

* store data parts in dict instead of list

* simplify weight/group parts dict assignment

* Merge test_<est>_local_predict and test_<est> tests for Dask module ##3833

* Merge test_<est>_local_predict to test_<est> tests in dask module - refactor #3833

* Update test_classifier and rename variables resolves #3833

* rename variables resolves #3833

* manage precision error #3833
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* [dask] factor dask-ml out of tests (fixes #3796)

* Update tests/python_package_test/test_dask.py
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update test_dask.py

* Update test_dask.py

* [dask] warn if attempting to use tree_learn other than data parallel

* add param

* initial Dask docs

* fix MRO

* address review comments

* Update dask.py

* Update basic.py

* hotfix pop

* centralize Python-package logging in one place

* continue

* fix test name

* removed unused import

* enhance test

* fix lint

* hotfix test

* workaround for GPU test

* remove custom logger from Dask-package

* replace one log func with flags by multiple funcs

* move make_ranking into utils

* do not cache

* TST Migrates test_plotting.py to pytest

* STY Fixes linting

* [dask] use keyword args for internal function calls

* add missing comma

* Update python-package/lightgbm/dask.py
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* revert whitespace changes

* test style
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update test_dual.py

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update test_dual.py

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* move all submodules to external_libs folder

* Update .Rbuildignore

* Update MANIFEST.in

* Update .appveyor.yml

* Update CMakeLists.txt

* Update build_r.R

* Update test.sh

* Update setup.py

* Update CMakeLists.txt

* Update test.sh

* Update setup.py

* Update conf.py

* Update MANIFEST.in

* Update LightGBM.vcxproj

* continue

* test

* test

* Update setup.py

* hotfix

* revert CI tests

* adding pred_contrib support

* add tests

* linting

* remove raw_score

* add pred kwargs

* faster tests

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* changes to tests

* Update tests/python_package_test/test_dask.py
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* ranker support wip

* fix ranker tests

* fix _make_ranking rnd gen bug, add sleep to help w stoch binding port failed exceptions

* add wait_for_workers to prevent Binding port exception

* another attempt to stabilize test_dask.py

* requested changes: docstrings, dask_ml, tuples for list_of_parts

* fix lint bug, add group param to test_ranker_local_predict

* decorator to skip tests with errors on fixture teardown

* remove gpu ranker tests, reduce make_ranking data complexity

* another attempt to
silence client, decorator does not silence fixture errors

* address requested changes on 1/20/20

* skip test_dask for all GPU tasks

* address changes requested on 1/21/21

* issubclass instead of __qualname__
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* parity in group docstr with sklearn
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* _make_ranking docstr cleanup
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* TST Migrates tset_engine.py to pytest

* ENH Apply suggestions

* ENH Uses temp path

* ENH Fixes typos

By using a unique lock instead of the shared lock the timings are very similar,
but predictions are correct.

Even so, by designing a small C++ benchmark with a very simple LGBM model,more threads on a simple model are slower than
the single-thread case. This is probably due to very small work units,
the lock contention overhead increases.

We should in the future benchmark with more complex models to see if supporting
threading on these calls is worth it in performance gains.

If not, then we could choose to not to provide thread-safety and remove
the locks altogether for maximal throughput.

See https://github.com/microsoft/LightGBM/issues/3751 for timings.

See gist for benchmark code:

  https://gist.github.com/AlbertoEAF/5972db15a27c294bab65b97e1bc4c315

[dask] allow parameter aliases for local_listen_port, num_threads, tree_learner (fixes #3671) (#3789)

* [dask] allow parameter aliases for tree_learner and local_listen_port (fixes #3671)

* num_thread too

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* empty commit
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>