GitLab

[ci] Skip Arrow tests on AppVeyor, use Intel macOS runners, upgrade to XCode 14.3 on macOS jobs, disable MacOS MPI jobs (#6425)

[ci] [python-package] enforce 'twine check' and 'check-wheel-contents' on Python distributions (#5707)

[ci] prefer CPython in Windows test environment and use safer approach for cleaning up network (fixes #5509) (#5510)

* Update test_windows.ps1

* Update .appveyor.yml

* Update test_windows.ps1

* Update test_windows.ps1

* Update test_windows.ps1

* Update test_windows.ps1

* Update test_windows.ps1

* Update .appveyor.yml

* Update test_windows.ps1

* Update .appveyor.yml

* Update .appveyor.yml

* Update test_windows.ps1

* Update .appveyor.yml

* Update .appveyor.yml

* Turn `early_stopping` into a Callable class

* Fix

* Lint

* Remove print

* Fix order

* Revert "Lint"

This reverts commit 7ca8b557572446888cf793c0082d9a7efd1e29a7.

* Apply suggestion from code review

* Nit

* Lint

* Move callable class outside the func for pickling

* Move _pickle and _unpickle to tests utils

* Add early stopping callback picklability test

* Nit

* Fix

* Lint

* Improve type hint

* Lint

* Lint

* Add cloudpickle to test_windows

* Update tests/python_package_test/test_engine.py

* Fix

* Apply suggestions from code review

* Update test_windows.ps1

* Update .appveyor.yml

* Update test_windows.ps1

* Update .appveyor.yml

* Update dockerfile-python

* Update README.md

* Update dockerfile.gpu

* Update dockerfile.gpu

* Update .vsts-ci.yml

* Update .appveyor.yml

* Update test_windows.ps1

This reverts commit d62378b4.

* Update test_windows.ps1

* Update .appveyor.yml

* Update .appveyor.yml

* [python-package] create Dataset from sampled data.

* [python-package] create Dataset from List[Sequence].

1. Use random access for data sampling
2. Support read data from multiple input files
3. Read data in batch so no need to hold all data in memory

* [python-package] example: create Dataset from multiple HDF5 file.

* fix: revert is_class implementation for seq

* fix: unwanted memory view reference for seq

* fix: seq is_class accepts sklearn matrices

* fix: requirements for example

* fix: pycode

* feat: print static code linting stage

* fix: linting: avoid shell str regex conversion

* code style: doc style

* code style: isort

* fix ci dependency: h5py on windows

* [py] remove rm files in test seq
https://github.com/microsoft/LightGBM/pull/4089#discussion_r612929623

* docs(python): init_from_sample summary

https://github.com/microsoft/LightGBM/pull/4089#discussion_r612903389



* remove dataset dump sample data debugging code.

* remove typo fix.

Create separate PR for this.

* fix typo in src/c_api.cpp
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* style(linting): py3 type hint for seq

* test(basic): os.path style path handling

* Revert "feat: print static code linting stage"

This reverts commit 10bd79f7f8258bea8e61c3abb8c9c7e4456a916d.

* feat(python): sequence on validation set

* minor(python): comment

* minor(python): test option hint

* style(python): fix code linting

* style(python): add pydoc for ref_dataset

* doc(python): sequence
Co-authored-by: shiyu1994 <shiyu_k1994@qq.com>

* revert(python): sequence class abc

* chore(python): remove rm_files

* Remove useless static_assert.

* refactor: test_basic test for sequence.

* fix lint complaint.

* remove dataset._dump_text in sequence test.

* Fix reverting typo fix.

* Apply suggestions from code review
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Fix type hint, code and doc style.

* fix failing test_basic.

* Remove TODO about keep constant in sync with cpp.

* Install h5py only when running python-examples.

* Fix lint complaint.

* Apply suggestions from code review
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Doc fixes, remove unused params_str in __init_from_seqs.

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Remove unnecessary conda install in windows ci script.

* Keep param as example in dataset_from_multi_hdf5.py

* Add _get_sample_count function to remove code duplication.

* Use batch_size parameter in generate_hdf.

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Fix after applying suggestions.

* Fix test, check idx is instance of numbers.Integral.

* Update python-package/lightgbm/basic.py
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Expose Sequence class in Python-API doc.

* Handle Sequence object not having batch_size.

* Fix isort lint complaint.

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update docstring to mention Sequence as data input.

* Remove get_one_line in test_basic.py

* Make Sequence an abstract class.

* Reduce number of tests for test_sequence.

* Add c_api: LGBM_SampleCount, fix potential bug in LGBMSampleIndices.

* empty commit to trigger ci

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Rename to LGBM_GetSampleCount, change LGBM_SampleIndices out_len to int32_t.

Also rename total_nrow to num_total_row in c_api.h for consistency.

* Doc about Sequence in docs/Python-Intro.rst.

* Fix: basic.py change LGBM_SampleIndices out_len to int32.

* Add create_valid test case with Dataset from Sequence.

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Apply suggestions from code review
Co-authored-by: shiyu1994 <shiyu_k1994@qq.com>

* Remove no longer used DEFAULT_BIN_CONSTRUCT_SAMPLE_CNT.

* Update python-package/lightgbm/basic.py
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Willian Zhang <willian@willian.email>
Co-authored-by: Willian Z <Willian@Willian-Zhang.com>
Co-authored-by: James Lamb <jaylamb20@gmail.com>
Co-authored-by: shiyu1994 <shiyu_k1994@qq.com>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* run cpp tests with sanitizers

* re-trigger CI

* continue

* small cleanup

* restore cpp test

* Correct spelling

Most changes were in comments, and there were a few changes to literals for log output.

There were no changes to variable names, function names, IDs, or functionality.

* Clarify a phrase in a comment
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Clarify a phrase in a comment
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Clarify a phrase in a comment
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Correct spelling

Most are code comments, but one case is a literal in a logging message.

There are a few grammar fixes too.
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* run cpp tests at CI

* Update docs/Installation-Guide.rst
Co-authored-by: James Lamb <jaylamb20@gmail.com>
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Update test.sh

* Update test.sh

* Update setup.sh

* Update test_windows.ps1

* Update test_windows.ps1

* Update test_windows.ps1

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update test_windows.ps1

* Update .vsts-ci.yml

* speed up files downloading for Windows CI

* run CI

* revert changing CI branch

[docs][ci] added docs about GPU support out of the box for Windows wheels and small refactoring for dual test (#3660)

* added docs about GPU support out of the box for Windows and small refactoring for dual test

* test

* Revert "test"

This reverts commit 45188103c2c9b3773bc62d97fbe865a2e5dd9120.

* fix docs

* fix docs

* hotfix config

* Apply suggestions from code review
Co-authored-by: TP Boudreau <tpboudreau@gmail.com>
Co-authored-by: TP Boudreau <tpboudreau@gmail.com>

* Add specialized OpenCL/Python package build path

* Refer to upstream OpenCL repository

* Reset build job count in setup.py

* Run test for dual CPU/GPU code in CI

* TEMPORARY: refer to OpenCL fork to ensure Linux CI builds succeed

* Try switching to OpenCL 2.0

* Add --opencl-python-pacakge option to wheel

* Fix some typos.

* Remove intermediate cmake target

* Restrict OpenCL headers to documented API version

* Refresh the environment after installation

* Try testing with two processes

* Use command line definition to activate integrated build

* Try newer OpenCL runtime

* If we hit OpenCL code path that's good enough

* Make tests run correctly

* Correct directory

* Flag reference to unofficial repo with FIXME

* TEMPORARY: update private repo tag for dependency

* Further post-merge fixes.

* Fix attribute name.

* Reformat with black.

* Rewrite config.h before library compilation for Windows wheel

* Run dual cpu/gpu test once only for Windows bdist CI

* Avoid C API tests when building gpu library for wheel

* TEMPORARY -- add clinfo tool to vm for debugging

* Increase compilation logging

* Rename OpenCL registry parameter after driver installation

* Install legacy OpenCL runtime

* ...

* Remove config.h change, clinfo installation, latest opencl runtime reference

* Add explicit script exit

* revert to 18.1 intel driver

* Run test_dual.py in two different modes

* Revert "Run test_dual.py in two different modes"

This reverts commit cbfec02659816f0cb321e1184994a9c2f37d2a77.

* Remove unused expected failure test

* Install OpenCL in separate powershell script

* fix setup.py imports

* activate conda env after session update

* separate update functions; exit opencl installation on error

* nothing; start pipeline

* download from original source

* specify tls version for installer download

* eliminate variable from command

* download opencl installer in parts

* remove duplicate setup.py command class entry

* download smaller parts

* remove redundant bdist wheel class

* Import chocolatey modules and use RefreshEnv

* insert pipeline cache for opencl installer

* rerun to ensure pipeline cache hit

* add comments, test another pipeline cache hit

* remove cache task; download full installer file from new location

* try another installer download timing

* reinsert pipeline cache

* retain cache task; download full file on cache miss

* test cache hit again

* remove todo
Co-authored-by: TP Boudreau <tpboudreau@gmail.com>

* Update setup.py

* Update .appveyor.yml

* Update .travis.yml

* Update .vsts-ci.yml

* Update __init__.py

* Update test.sh

* Update test_windows.ps1

* Update advanced_example.py

* Update requirements_base.txt

* Update conf.py

* Update conf.py

* Update test_engine.py

* Update utils.py

* Update dockerfile-r

* Update README.md

* Update dockerfile.gpu

* Update test_consistency.py

* Update basic.py

* Update compat.py

* Update engine.py

* Update sklearn.py

* Update sklearn.py

* Update callback.py

* Update setup.py

* Update __init__.py

* Update plotting.py

* Update sklearn.py

* Update engine.py

* Update compat.py

* Update callback.py

* Update basic.py

* Update compat.py

* Update basic.py

* Update basic.py

* Update compat.py

* Update compat.py

* Update plotting.py

* Update engine.py

* Update basic.py

* Update sklearn.py

* Update compat.py

* Update engine.py

* Update engine.py

* Update callback.py

* Update basic.py

* Update basic.py

* Update basic.py

* Update basic.py

* Update basic.py

* Update sklearn.py

* Update sklearn.py

* Update plotting.py

* Update sklearn.py

* Update compat.py

* Update compat.py

* Update engine.py

* Update plotting.py

* Update sklearn.py

* Update basic.py

* Update basic.py

* Update basic.py

* Update basic.py

* Update compat.py

* Update compat.py

* Update compat.py

* Update engine.py

* Update basic.py

* Update compat.py

* Update basic.py

* Update basic.py

* Update basic.py

* Update compat.py

* Update compat.py

* Update basic.py

* Update basic.py

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update conf.py

* Revert "Update dockerfile-r"

This reverts commit 4ff6ffc7e3eeda24cc6a59a3bb0c973f02d9d71c.

* [ci] host some artifacts needed by R Windows CI jobs

* remove link

* host SWIG
Co-authored-by: Guolin Ke <guolin.ke@outlook.com>

* first test of appveyor

* strings are strings

* lil bit of build script

* fixing paths

* removed unnecessary file

* updated CRAN URL

* added a lot more printing

* fixing paths

* more stuff

* fixed paths

* more stuff

* more path guessing

* even more paths

* more stuff

* moar logz

* and now for something totally different

* please work

* ok could be a thing

* changing directories

* we might be in business

* fixed install syntax

* tryinv mingw

* more mingw

* ignore Suggests check

* trying Azure DevOps

* just run bare minimum for Azure DevOps

* fixed  build dir thing

* trying to set libPaths

* more testing

* trying R 3.6.3

* R 3.6.3

* this feels right

* still messing around with libraries

* more paths

* removed duplication in Windows testing code

* simpler

* fixed conda stuff

* more conda stuff

* more fixes

* fixed testing script

* moved AppVeyor setup to the top

* commenting

* ch-ch-ch-ch-chaaaanges

* paths

* plz work

* fixed conda stuff in Windows CI

* uncommented stuff to test a full build

* fixed quotes and removed some unnecessary stuff

* added install.libs.R change

* quotes are impoortant

* added commented-out stuff back in

* added Windows script, download retries, and MSVC linking

* minor fixes

* cleaned up debugging code in FindLibR

* cleaned up debugging code and moved R first in CI

* fixed vsts-ci indentation

* cut documentation stuff out of MSVC build

* fix R CMD check for Azure

* misc whitespace changes

* Added echoing of build logs from R CMD check

* cut out more documentation tests

* fixed NOTE about imports from Matrix

* moved some changes out of this PR and into #2963

* fixed whitespace stuff

* added check on number of NOTES

* adding better checks

* fixing check on NOTEs

* removing unnecessary variable

* Update .ci/test_r_package_windows.ps1
Co-Authored-By: Nikita Titov <nekit94-08@mail.ru>

* some changes

* fix quoting

* trying MINGW on Azure DevOps

* fixing paths

* more paths

* fixing paths

* testing paths

* fixing slashes

* pinned CTAN mirror

* get better logs

* made sure Azure finds MinGW, fixed search for LIBR_CORE_LIBRARY, stopped building R docs on Azure

* Apply suggestions from code review
Co-Authored-By: Nikita Titov <nekit94-08@mail.ru>

* added CXX, CC for Windows builds and changed back to building docs on all MINGW builds

* stored LIBR_CORE_LIBRARY hints in one variable

* Apply suggestions from code review
Co-Authored-By: Nikita Titov <nekit94-08@mail.ru>

* changes from code review

* increased parallel builds for Azure CI

* Apply suggestions from code review
Co-Authored-By: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Revert "specify the last supported version of scikit-learn (#2637)"

This reverts commit d1002776.

* ban scikit-learn 0.22.0 and skip broken test

* fix updated test

* fix lint test

* Revert "fix lint test"

This reverts commit 8b4db0805fe7a9e7f7eb0be3eac231f85026d196.

* [ci] reduced duplication in Python Windows CI

* fixed typo

* reorganized

* fixed task name

* Update .ci/test_windows.ps1
Co-Authored-By: Nikita Titov <nekit94-08@mail.ru>

* changes from code review

* added conda init powershell back

* Update .ci/test_windows.ps1
Co-Authored-By: Nikita Titov <nekit94-08@mail.ru>

* Switched to absolute paths

* fixed env variables
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>