GitLab

* [ci] use conda-forge in CI jobs (fixes #4948)

* comment out more jobs

* try reverting graphviz patch, running more cuda jobs

* get graphviz from PyPI and try removing some patches for r-lintr

* start running appveyor again

* use conda-forge if using conda

* fix commands

* conda install graphviz

* try newer openmp

* pin below openmp 11.x

* focus on gpu task

* trying to narrow down error

* maybe gcc11 is the issue

* start adding other tests back

* pin openmp too

* maybe need to pin to gcc less than 10.x

* pin libgfortran and libstdcxx as well

* pin to gcc 9.3.0

* move constraints up to initial environment

* add all CI jobs back

* try installing python-graphviz separately

* try new lightgbm/vsts-agent image

* fix typo

* test if pinning gcc for linux gpu_source build is still necessary

* ok yes, pinning gcc is necessary

* test if Linux gpu_source works with Python 3.9.6

* no special exception for Linux gpu_source job

* pin to Python 3.9.6 in Linux gpu_source

* try explicitly asking for libstdcxx-ng for every linux build

* swap compilers

* switch compilers back

* revert accidental whitespace change

* comment out CI

* try Linux gpu_source with different Python versions

* Revert "try Linux gpu_source with different Python versions"

This reverts commit f6f63cbb9b4a9cf138f3580ae4223a8acdd0e94a.

* Revert "comment out CI"

This reverts commit ece191f01e3650c2f325e80ff86bfc8c485fb7bc.

* remove libxml2 install, change CONDA path

* avoid installing conda in rchk job

* empty commit 1

* empty commit 2

* empty commit 3

* empty commit 4

* add more verbose logging around installation of python-graphviz

* empty commit 1

* get mamba info

* get more conda info

* add another mamba info call

* allow for other macOS environments in GHA configuration

* Revert "allow for other macOS environments in GHA configuration"

This reverts commit a3c7a19926be94e3719f5ae9100fbe30e87b35da.

* get more logs from mamba

* get Build.ArtifactsStagingDirectory

* get more logs and try to force re-installing everything

* clean cache after every step

* remove --update-all and make logs less verbose

* remove more print statements and uncomment jobs

* test if conda-clean issue fixes segfaults for gpu_source

* pin python version for gpu_source

* empty commit 1

* use miniforge instead

* empty commit 1

* Apply suggestions from code review

* bring workarounds back

* remove duplicated graphviz system-wide installation (reverts #4095, #4097, #4238)

* empty commit 1

* empty commit 2

* empty commit 3

* empty commit 4

* empty commit 5

* empty commit 6

* empty commit 7

* empty commit 8

* empty commit 9

* empty commit 10

* empty commit 10

* empty commit 10

* empty commit 10

* empty commit 11

* one more try

* try to downgrade Python version for Linux GPU job

* swap compilers

* Revert "swap compilers"

This reverts commit f04dc27b17920a69cbcba1254a8e109ce9791154.
Co-authored-by: Nikita Titov <nekit94-12@hotmail.com>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update .appveyor.yml

* Update .vsts-ci.yml

* Update python_package.yml

* Update setup.py

* Update test.sh

* Update .vsts-ci.yml

* Update setup.sh

* Update .vsts-ci.yml

* Update test.sh

* Update README.rst

* Update test.sh

* Update test.sh

* Update test.sh

* Update FindLibR.cmake

* Update r_package.yml

* Update FindLibR.cmake

* Update r_package.yml

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update static_analysis.yml

* Update CMakeLists.txt

* Update test.sh

* Update CMakeLists.txt

* Update static_analysis.yml

* pin Dask version at CI

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* workaround for Python 3.6

* Update test.sh

* indicate support of Monterey and drop Catalina

* Update test.sh

* restore Mojave

* Use the latest gcc version in macOS CI jobs

* test: swap compilers

* Revert "test: swap compilers"

* Revert "test: swap compilers"

* [ci] add CI job running rchk

* try commenting out more stuff

* ignore R internal error

* pipes

* remove PROTECT()

* try removing testthat

* revert temporary testing changes

* call predict on one row of data to determine output shape

* make DaskLGBMRanker predict method equal to the others

* remove extra drop_axis

* [python-package] create Dataset from sampled data.

* [python-package] create Dataset from List[Sequence].

1. Use random access for data sampling
2. Support read data from multiple input files
3. Read data in batch so no need to hold all data in memory

* [python-package] example: create Dataset from multiple HDF5 file.

* fix: revert is_class implementation for seq

* fix: unwanted memory view reference for seq

* fix: seq is_class accepts sklearn matrices

* fix: requirements for example

* fix: pycode

* feat: print static code linting stage

* fix: linting: avoid shell str regex conversion

* code style: doc style

* code style: isort

* fix ci dependency: h5py on windows

* [py] remove rm files in test seq
https://github.com/microsoft/LightGBM/pull/4089#discussion_r612929623

* docs(python): init_from_sample summary

https://github.com/microsoft/LightGBM/pull/4089#discussion_r612903389



* remove dataset dump sample data debugging code.

* remove typo fix.

Create separate PR for this.

* fix typo in src/c_api.cpp
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* style(linting): py3 type hint for seq

* test(basic): os.path style path handling

* Revert "feat: print static code linting stage"

This reverts commit 10bd79f7f8258bea8e61c3abb8c9c7e4456a916d.

* feat(python): sequence on validation set

* minor(python): comment

* minor(python): test option hint

* style(python): fix code linting

* style(python): add pydoc for ref_dataset

* doc(python): sequence
Co-authored-by: shiyu1994 <shiyu_k1994@qq.com>

* revert(python): sequence class abc

* chore(python): remove rm_files

* Remove useless static_assert.

* refactor: test_basic test for sequence.

* fix lint complaint.

* remove dataset._dump_text in sequence test.

* Fix reverting typo fix.

* Apply suggestions from code review
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Fix type hint, code and doc style.

* fix failing test_basic.

* Remove TODO about keep constant in sync with cpp.

* Install h5py only when running python-examples.

* Fix lint complaint.

* Apply suggestions from code review
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Doc fixes, remove unused params_str in __init_from_seqs.

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Remove unnecessary conda install in windows ci script.

* Keep param as example in dataset_from_multi_hdf5.py

* Add _get_sample_count function to remove code duplication.

* Use batch_size parameter in generate_hdf.

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Fix after applying suggestions.

* Fix test, check idx is instance of numbers.Integral.

* Update python-package/lightgbm/basic.py
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Expose Sequence class in Python-API doc.

* Handle Sequence object not having batch_size.

* Fix isort lint complaint.

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update docstring to mention Sequence as data input.

* Remove get_one_line in test_basic.py

* Make Sequence an abstract class.

* Reduce number of tests for test_sequence.

* Add c_api: LGBM_SampleCount, fix potential bug in LGBMSampleIndices.

* empty commit to trigger ci

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Rename to LGBM_GetSampleCount, change LGBM_SampleIndices out_len to int32_t.

Also rename total_nrow to num_total_row in c_api.h for consistency.

* Doc about Sequence in docs/Python-Intro.rst.

* Fix: basic.py change LGBM_SampleIndices out_len to int32.

* Add create_valid test case with Dataset from Sequence.

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Apply suggestions from code review
Co-authored-by: shiyu1994 <shiyu_k1994@qq.com>

* Remove no longer used DEFAULT_BIN_CONSTRUCT_SAMPLE_CNT.

* Update python-package/lightgbm/basic.py
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Willian Zhang <willian@willian.email>
Co-authored-by: Willian Z <Willian@Willian-Zhang.com>
Co-authored-by: James Lamb <jaylamb20@gmail.com>
Co-authored-by: shiyu1994 <shiyu_k1994@qq.com>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* run cpp tests with sanitizers

* re-trigger CI

* continue

* small cleanup

* restore cpp test

* [ci] pin dask and distributed in CI jobs

* fix for Python 3.6

* fix control flow mistake

* Update .ci/test.sh
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Install graphviz from default conda channel

* Update test.sh

* Update setup.sh

* Update test.sh

* Update setup.sh

* Update setup.sh

* Update setup.sh

* Update setup.sh

* Update setup.sh

* Update setup.sh

* run cpp tests at CI

* Update docs/Installation-Guide.rst
Co-authored-by: James Lamb <jaylamb20@gmail.com>
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Update chunked_array.hpp

* Update ChunkedArray_API_extensions.i

* Update StringArray.i

* apply cpplint to cpp tests

* Update test_chunked_array to please cpplint (#4121)

* Update test_chunked_array to please cpplint

* Simplify var name

* Add comment
Co-authored-by: Alberto Ferreira <AlbertoEAF@users.noreply.github.com>

* Update test.sh

* Update setup.sh

* Update .vsts-ci.yml

* Update test.sh

* Update setup.sh

* Update .vsts-ci.yml

* Update setup.sh

* Update setup.sh

* [feature] Add ChunkedArray to SWIG

* Add ChunkedArray
* Add ChunkedArray_API_extensions.i
* Add SWIG class wrappers

* Address some review comments

* Fix linting issues

* Move test to tests/test_ChunkedArray_manually.cpp

* Add test note

* Move ChunkedArray to include/LightGBM/utils/

* Declare more explicit types of ChunkedArray in the SWIG API.

* Port ChunkedArray tests to googletest

* Please C++ linter

* Address StrikerRUS' review comments

* Update SWIG doc & disable ChunkedArray<int64_t>

* Use CHECK_EQ instead of assert

* Change include order (linting)

* Rename ChunkedArray -> chunked_array files

* Change header guards

* Address last comments from StrikerRUS

This reverts commit 4e9c9768.

* Update Parallel-Learning-Guide.rst

* Update test.sh

* fix path

* address review comments

* [ci] prevent getting incompatible dask and distributed versions

* Update .ci/test.sh
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* empty commit
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update setup.sh

* Update test.sh

* Update test_dask.py

* Update test_engine.py

* Update .vsts-ci.yml

* run isort in CI linting job

* workaround conda compatibility issues

* starting on Dask client

* more docs stuff

* fix pickling

* just copy docstrings

* fit docs

* switch test order

* linting

* use client kwarg

* remove inner set_params()

* add type hints

* fix type hints

* remove commented code

* reorder

* fix tests, add client_ property

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* fix tests

* linting

* simplify
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update linkchecker.yml

* Update test.sh

* [ci] move link checking to a cron job

* run once to register

* set PYTHON_VERSION

* actually build docs

* uncomment

* remove stuff

* Update docs/.linkcheckerrc

* condense

* remove PR triggers again

* Update .github/workflows/linkchecker.yml

* move TASK

* [ci] fix R linting job (fixes #3873)

* need to run static analysis

* uncomment CI configs

* Update .ci/test.sh

* [dask] factor dask-ml out of tests (fixes #3796)

* Update tests/python_package_test/test_dask.py
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* move all submodules to external_libs folder

* Update .Rbuildignore

* Update MANIFEST.in

* Update .appveyor.yml

* Update CMakeLists.txt

* Update build_r.R

* Update test.sh

* Update setup.py

* Update CMakeLists.txt

* Update test.sh

* Update setup.py

* Update conf.py

* Update MANIFEST.in

* Update LightGBM.vcxproj

* continue

* test

* test

* Update setup.py

* hotfix

* revert CI tests

* Update test.sh

* Update test.sh

* Update setup.sh

* Update test_windows.ps1

* Update test_windows.ps1

* Update test_windows.ps1

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update test_windows.ps1

* Update .vsts-ci.yml

* [ci] move CI jobs from Travis to Azure DevOps (fixes #3519)

* comment out other CIs to avoid wasting cycles

* try without docker

* add container back

* stop using --user in pip install

* run check-docs and lint without container

* job names

* move more jobs to Azure-hosted Linux pool

* fix PATH for check-docs

* uncomment other CI

* uncomment windows

* remove uses of maxParallel

* try moving macos-latest jobs to GitHub Actions

* fix config

* fix missing conda env

* set Python version

* remove commented-out code

* add more to GitHub Actions

* try to fix GPU

* remove static_analysis to prevent conflicts with #3726

* change workflow name

* try using ubuntu:latest docker

* fix conda

* trying to find where permissions first break

* add workaround for sudo

* please azure please

* image syntax

* more sudo

* noninteractive

* LC_ALL

* more  sudo

* more stuff

* CONDA dir

* paths

* get path

* missing CONDA

* fix path stuff

* more tests

* fix graphviz

* stuff

* more graphviz

* install xorg-libxau

* graphviz works, run more jobs

* stuff

* enable more tests

* uncomment GitHub Actions

* uncomment all other CIs

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* add travis.yml to Rbuildignore

* add Rbuildignore rule for fmt

* add libomp for clang builds

* changes from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* replace file extensions in docs during Sphinx build

* Update .travis.yml

* Update .travis.yml

* Update conf.py

* fix lint

* fix lint

* [ci] Move check-docs and lint jobs off Travis

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* try moving check-docs into a containerized job

* add install flag and README badge

* trying to find the error

* ignore pipefail

* remove -e

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* more changes

* additional changes

* Update .github/workflows/static_analysis.yml

* Update README.md
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* move constants
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>