GitLab

[python] improving the syntax of the fstring in the file : tests/python_package_test/test_dask.py (#4358)

* updated the old syntax with fstrings

* Updated the strings with + catenation to fstrings

* Updated the strings with + catenation to fstrings

* Update tests/python_package_test/test_dask.py
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* run Dask tests on aarch64 architecture

* make random Dask test to fail

* Revert "make random Dask test to fail"

This reverts commit c43c98507f818994bb08b4f7d289ecad3b3449eb.

* empty commit

* empty commit

* empty commit

* empty commit
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* replace client fixture with cluster fixture

* wait on persist before rebalance

* include voting_parallel tree_learner in test_regressor, test_classifier and test_ranker

* remove test for warnings and test for error when using feature_parallel

* use real names for tree_learner intest and include test for aliases. use the error message in the test for error in feature parallel

* split all tests with rf in test_classifier

* remove task parametrization for tree_learner aliases test. smaller input data from feature_parallel error

* define task for tree_learner aliases

* [dask] make random port search more resilient to random collisions

* linting

* more reliable ports check

* address review comments

* add error message

* test all boosting types

* lint

* bring scores comparison back and set y as second argument in assert_eq

* Update test_dask.py

* Update test_dask.py

* include test for prediction with raw_score

* close client

* initial comments

* update data creation and include ranking task

* linting

* update _create_data

* compare unique raw_predictions with values in leaves_df

* make one categorical variable informative. increase n_samples. reduce n_features for regression

* adjust tolerances in checks

* simplify Dask tests code

* enable CI

* disable CI

* [dask] raise more informative error for duplicates in 'machines'

* uncomment

* avoid test failure

* Revert "avoid test failure"

This reverts commit 9442bdf00f193a19a923dc0deb46b7822cb6f601.

* include multiclass-classification task and task_to_model_factory dicts

* define centers coordinates. flatten init_scores within each partition for multiclass-classification

* include issue comment and fix linting error

* include support for init_score

* use dataframe from init_score and test difference with and without init_score in local model

* revert refactoring

* initial docs. test between distributed models with and without init_score

* remove ranker from tests

* test value for root node and change docs

* comma

* re-include parametrize

* fix incorrect merge

* use single init_score and the booster_ attribute

* use np.float64 instead of float

* [dask] [ci] add support for scikit-learn 0.24+ in tests (fixes #4031)

* Update tests/python_package_test/test_dask.py
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* try upgrading mixtexsetup

* they changed the executable name UGH

* more changes for executable name

* another path change

* changing package mirrors

* undo experiments
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* simplify fit calls in Dask tests

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* include support for column array as label

* remove nested ifs

* fix linting errors

* include tests for sklearn regressors

* include docstring for numpy_1d_array_to_dtype

* include . at end of docstring

* remove pandas import and test for regression, classification and ranking

* check predictions of sklearn models as well

* test training only in dask. drop pandas series tests

* use PANDAS_INSTALLED and pd_Series

* inline imports

* use col array in fit for test_dask

* include review comments

* use socket.bind with port 0 and client.run to find random open ports

* include test for found ports

* find random open ports as default

* parametrize local_listen_port. type hint to _find_random_open_port. fid open ports only on workers with data.

* make indentation consistent and pass list of workers to client.run

* remove socket import

* change random port implementation

* fix test

* [dask] allow tight control over ports

* getting there, getting there

* fix params maybe

* fixing params

* remove unnecessary stuff

* fix tests

* fixes

* some minor changes

* fix flaky test

* linting

* more linting

* clarify parameter description

* add warning

* revert docs change

* Update python-package/lightgbm/dask.py

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* trying to fix stuff

* this is working

* update tests

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* indent
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* add test_dask.py

* Update tests/python_package_test/test_dask.py
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* clients

* remove ports

* safe sklearn checks

* safe sklearn checks

* fix whitespace

* fix whitespace-try 2

* fix whitespace-try 3

* isort

* isort

* sklearn_checks_to_learn
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* [dask] make find-open-port test more reliable

* use listen_port fixture

* Apply suggestions from code review

* Update setup.sh

* Update test.sh

* Update test_dask.py

* Update test_engine.py

* Update .vsts-ci.yml

* got fit() working

* add predict()

* predict_proba()

* remove custom objective docs

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* fix capitalization

* Update tests/python_package_test/test_dask.py
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* include test for training when a worker has no data

* test single partition against local model for all tasks and outputs

* remove futures_of

* include james' comments

* remove product import

* fix tests

* fix tests

* fix test comments

* simplify tests

* Apply suggestions from code review

[dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators  (#3911)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* [dask] Add unit tests that signatures are the same between Dask and scikit-learn estimators (fixes microsoft#3907)

* add support for pandas categorical columns

* remove commented code

* quotes

* syntax error

* fix shape for ranker test

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update tests/python_package_test/test_dask.py

* trying

* fix tests

* remove unnecessary debugging stuff

* skip accuracy checks on categorical

* use category columns as categorical features
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update test_dask.py

* Update dask.py

* Update .vsts-ci.yml

* Revert "Update .vsts-ci.yml"

This reverts commit 98422be5b5095f0585de333b5b5545356776ef88.

* starting on Dask client

* more docs stuff

* fix pickling

* just copy docstrings

* fit docs

* switch test order

* linting

* use client kwarg

* remove inner set_params()

* add type hints

* fix type hints

* remove commented code

* reorder

* fix tests, add client_ property

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* fix tests

* linting

* simplify
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* [dask] reduce teardown erros in Dask tests

* azure

* show logs

* try again

* more

* submodules

* try a bunch of sdist tests

* empty commit

* empty commit

* empty commit

* use sh-ubuntu

* 10 sdist tasks

* stuff

* empty commit

* empty commit

* empty commit

* empty commit

* try bdist

* empty commit

* empty commit

* empty commit

* empty commit

* py37

* empty commit

* empty commit

* empty commit

* empty commit

* empty commit

* empty commit

* empty commit

* empty commit

* empty commit

* python 3.8

* empty commit

* empty commit

* empty commit

* empty commit

* empty commit

* cuda config

* comment out cuda again

* setting timeout

* put client close in the right place

* uncomment CI, make timeout 60

* [dask] add tests on warnings, fix incorrect variable in log

* Update tests/python_package_test/test_dask.py
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Merge test_<est>_local_predict and test_<est> tests for Dask module ##3833

* Merge test_<est>_local_predict to test_<est> tests in dask module - refactor #3833

* Update test_classifier and rename variables resolves #3833

* rename variables resolves #3833

* manage precision error #3833
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* [dask] factor dask-ml out of tests (fixes #3796)

* Update tests/python_package_test/test_dask.py
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>