Commits · 222a7368f93dd0b2b541477ac115e1b299518e4c · tianlh / LightGBM-DCU

19 Dec, 2020 1 commit

[python] Windows wheels that support GPUs via OpenCL (#3403) · 222a7368

Itamar Turner-Trauring authored Dec 19, 2020



* Add specialized OpenCL/Python package build path

* Refer to upstream OpenCL repository

* Reset build job count in setup.py

* Run test for dual CPU/GPU code in CI

* TEMPORARY: refer to OpenCL fork to ensure Linux CI builds succeed

* Try switching to OpenCL 2.0

* Add --opencl-python-pacakge option to wheel

* Fix some typos.

* Remove intermediate cmake target

* Restrict OpenCL headers to documented API version

* Refresh the environment after installation

* Try testing with two processes

* Use command line definition to activate integrated build

* Try newer OpenCL runtime

* If we hit OpenCL code path that's good enough

* Make tests run correctly

* Correct directory

* Flag reference to unofficial repo with FIXME

* TEMPORARY: update private repo tag for dependency

* Further post-merge fixes.

* Fix attribute name.

* Reformat with black.

* Rewrite config.h before library compilation for Windows wheel

* Run dual cpu/gpu test once only for Windows bdist CI

* Avoid C API tests when building gpu library for wheel

* TEMPORARY -- add clinfo tool to vm for debugging

* Increase compilation logging

* Rename OpenCL registry parameter after driver installation

* Install legacy OpenCL runtime

* ...

* Remove config.h change, clinfo installation, latest opencl runtime reference

* Add explicit script exit

* revert to 18.1 intel driver

* Run test_dual.py in two different modes

* Revert "Run test_dual.py in two different modes"

This reverts commit cbfec02659816f0cb321e1184994a9c2f37d2a77.

* Remove unused expected failure test

* Install OpenCL in separate powershell script

* fix setup.py imports

* activate conda env after session update

* separate update functions; exit opencl installation on error

* nothing; start pipeline

* download from original source

* specify tls version for installer download

* eliminate variable from command

* download opencl installer in parts

* remove duplicate setup.py command class entry

* download smaller parts

* remove redundant bdist wheel class

* Import chocolatey modules and use RefreshEnv

* insert pipeline cache for opencl installer

* rerun to ensure pipeline cache hit

* add comments, test another pipeline cache hit

* remove cache task; download full installer file from new location

* try another installer download timing

* reinsert pipeline cache

* retain cache task; download full file on cache miss

* test cache hit again

* remove todo
Co-authored-by: TP Boudreau <tpboudreau@gmail.com>

222a7368

09 Dec, 2020 1 commit

[python] Drop Python 2 support (#3581) · 44a6fb7f

Nikita Titov authored Dec 09, 2020

* Update setup.py

* Update .appveyor.yml

* Update .travis.yml

* Update .vsts-ci.yml

* Update __init__.py

* Update test.sh

* Update test_windows.ps1

* Update advanced_example.py

* Update requirements_base.txt

* Update conf.py

* Update conf.py

* Update test_engine.py

* Update utils.py

* Update dockerfile-r

* Update README.md

* Update dockerfile.gpu

* Update test_consistency.py

* Update basic.py

* Update compat.py

* Update engine.py

* Update sklearn.py

* Update sklearn.py

* Update callback.py

* Update setup.py

* Update __init__.py

* Update plotting.py

* Update sklearn.py

* Update engine.py

* Update compat.py

* Update callback.py

* Update basic.py

* Update compat.py

* Update basic.py

* Update basic.py

* Update compat.py

* Update compat.py

* Update plotting.py

* Update engine.py

* Update basic.py

* Update sklearn.py

* Update compat.py

* Update engine.py

* Update engine.py

* Update callback.py

* Update basic.py

* Update basic.py

* Update basic.py

* Update basic.py

* Update basic.py

* Update sklearn.py

* Update sklearn.py

* Update plotting.py

* Update sklearn.py

* Update compat.py

* Update compat.py

* Update engine.py

* Update plotting.py

* Update sklearn.py

* Update basic.py

* Update basic.py

* Update basic.py

* Update basic.py

* Update compat.py

* Update compat.py

* Update compat.py

* Update engine.py

* Update basic.py

* Update compat.py

* Update basic.py

* Update basic.py

* Update basic.py

* Update compat.py

* Update compat.py

* Update basic.py

* Update basic.py

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update conf.py

* Revert "Update dockerfile-r"

This reverts commit 4ff6ffc7e3eeda24cc6a59a3bb0c973f02d9d71c.

44a6fb7f

08 Dec, 2020 1 commit

Fix model locale issue and improve model R/W performance. (#3405) · 792c9303

Alberto Ferreira authored Dec 08, 2020

* Fix LightGBM models locale sensitivity and improve R/W performance.

When Java is used, the default C++ locale is broken. This is true for
Java providers that use the C API or even Python models that require JEP.

This patch solves that issue making the model reads/writes insensitive
to such settings.
To achieve it, within the model read/write codebase:
 - C++ streams are imbued with the classic locale
 - Calls to functions that are dependent on the locale are replaced
 - The default locale is not changed!

This approach means:
 - The user's locale is never tampered with, avoiding issues such as
    https://github.com/microsoft/LightGBM/issues/2979 with the previous
    approach https://github.com/microsoft/LightGBM/pull/2891
 - Datasets can still be read according the user's locale
 - The model file has a single format independent of locale

Changes:
 - Add CommonC namespace which provides faster locale-independent versions of Common's methods
 - Model code makes conversions through CommonC
 - Cleanup unused Common methods
 - Performance improvements. Use fast libraries for locale-agnostic conversion:
   - value->string: https://github.com/fmtlib/fmt
   - string->double: https://github.com/lemire/fast_double_parser (10x
      faster double parsing according to their benchmark)

Bugfixes:
 - https://github.com/microsoft/LightGBM/issues/2500
 - https://github.com/microsoft/LightGBM/issues/2890
 - https://github.com/ninia/jep/issues/205

 (as it is related to LGBM as well)

* Align CommonC namespace

* Add new external_libs/ to python setup

* Try fast_double_parser fix #1

Testing commit e09e5aad828bcb16bea7ed0ed8322e019112fdbe

If it works it should fix more LGBM builds

* CMake: Attempt to link fmt without explicit PUBLIC tag

* Exclude external_libs from linting

* Add exernal_libs to MANIFEST.in

* Set dynamic linking option for fmt.

* linting issues

* Try to fix lint includes

* Try to pass fPIC with static fmt lib

* Try CMake P_I_C option with fmt library

* [R-package] Add CMake support for R and CRAN

* Cleanup CMakeLists

* Try fmt hack to remove stdout

* Switch to header-only mode

* Add PRIVATE argument to target_link_libraries

* use fmt in header-only mode

* Remove CMakeLists comment

* Change OpenMP to PUBLIC linking in Mac

* Update fmt submodule to 7.1.2

* Use fmt in header-only-mode

* Remove fmt from CMakeLists.txt

* Upgrade fast_double_parser to v0.2.0

* Revert "Add PRIVATE argument to target_link_libraries"

This reverts commit 3dd45dde7b92531b2530ab54522bb843c56227a7.

* Address James Lamb's comments

* Update R-package/.Rbuildignore
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Upgrade to fast_double_parser v0.3.0 - Solaris support

* Use legacy code only in Solaris

* Fix lint issues

* Fix comment

* Address StrikerRUS's comments (solaris ifdef).

* Change header guards
Co-authored-by: James Lamb <jaylamb20@gmail.com>

792c9303

26 Nov, 2020 1 commit
- [python] Allow to build Python wheel package (#3594) · 918ccaec
  Martynov Maxim authored Nov 26, 2020
```
Co-authored-by: Мартынов Максим Сергеевич <msmarty5@mts.ru>
```
  918ccaec
21 Sep, 2020 1 commit

Add option to build with integrated OpenCL (#3144) · 3454698e

TP Boudreau authored Sep 21, 2020



* Add specialized OpenCL/Python package build path

* Refer to upstream OpenCL repository

* Reset build job count in setup.py

* TEMPORARY: refer to OpenCL fork to ensure Linux CI builds succeed

* Remove intermediate cmake target

* Restrict OpenCL headers to documented API version

* Use command line definition to activate integrated build

* Flag reference to unofficial repo with FIXME

* TEMPORARY: update private repo tag for dependency

* Remove integrated build for non-Win32 and related cleanup

* Remove commented code

* Rename integrated OpenCL build option and other cleanups

* Small cleanups

* Update CMakeIntegratedOpenCL.cmake
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update CMakeIntegratedOpenCL.cmake
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update CMakeIntegratedOpenCL.cmake
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update CMakeIntegratedOpenCL.cmake
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update CMakeLists.txt
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update CMakeLists.txt
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update CMakeLists.txt
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update CMakeIntegratedOpenCL.cmake

Targeted download of Boost submodules.
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update CMakeLists.txt
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

3454698e

20 Sep, 2020 1 commit

[GPU] Add support for CUDA-based GPU build (#3160) · f7ad9457

Chip Kerchner authored Sep 20, 2020

* Initial CUDA work

* redirect log to python console (#3090)

* redir log to python console

* fix pylint

* Apply suggestions from code review

* Update basic.py

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update c_api.h

* Apply suggestions from code review

* super-minor: better wording
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: StrikerRUS <nekit94-12@hotmail.com>

* re-order includes (fixes #3132) (#3133)

* Revert "re-order includes (fixes #3132) (#3133)" (#3153)

This reverts commit 656d2676

* Missing change from previous rebase

* Minor cleanup and removal of development scripts.

* Only set gpu_use_dp on by default for CUDA. Other minor change.

* Fix python lint indentation problem.

* More python lint issues.

* Big lint cleanup - more to come.

* Another large lint cleanup - more to come.

* Even more lint cleanup.

* Minor cleanup so less differences in code.

* Revert is_use_subset changes

* Another rebase from master to fix recent conflicts.

* More lint.

* Simple code cleanup - add & remove blank lines, revert unneccessary format changes, remove added dead code.

* Removed parameters added for CUDA and various bug fix.

* Yet more lint and unneccessary changes.

* Revert another change.

* Removal of unneccessary code.

* temporary appveyor.yml for building and testing

* Remove return value in ReSize

* Removal of unused variables.

* Code cleanup from reviewers suggestions.

* Removal of FIXME comments and unused defines.

* More reviewers comments cleanup.

* Fix config variables.

* Attempt to fix check-docs failure

* Update Paramster.rst for num_gpu

* Removing test appveyor.yml

* Add CUDA_RESOLVE_DEVICE_SYMBOLS to libraries to fix linking issue.

* Fixed handling of data elements less than 2K.

* More reviewers comments cleanup.

* Removal of TODO and fix printing of int64_t

* Add cuda change for CI testing and remove cuda from device_type in python.

* Missed one change form previous check-in

* Removal AdditionConfig and fix settings.

* Limit number of GPUs to one for now in CUDA.

* Update Parameters.rst for previous check-in

* Whitespace removal.

* Cleanup unused code.

* Changed uint/ushort/ulong to unsigned int/short/long to help Windows based CUDA compiler work.

* Lint change from previous check-in.

* Changes based on reviewers comments.

* More reviewer comment changes.

* Adding warning for is_sparse. Revert tmp_subset code. Only return FeatureGroupData if not is_multi_val_

* Fix so that CUDA code will compile even if you enable the SCORE_T_USE_DOUBLE define.

* Reviewer comment cleanup.

* Replace warning with Log message. Removal of some of the USE_CUDA. Fix typo and removal of pragma once.

* Remove PRINT debug for CUDA code.

* Allow to use of multiple GPUs for CUDA.

* More multi-GPUs enablement for CUDA.

* More code cleanup based on reviews comments.

* Update docs with latest config changes.
Co-authored-by: Gordon Fossum <fossum@us.ibm.com>
Co-authored-by: ChipKerchner <ckerchne@linux.vnet.ibm.com>
Co-authored-by: Guolin Ke <guolin.ke@outlook.com>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: StrikerRUS <nekit94-12@hotmail.com>
Co-authored-by: James Lamb <jaylamb20@gmail.com>

f7ad9457

15 Sep, 2020 1 commit

[python] Drop Python 3.5 support (#3395) · a3b9dae7

Nikita Titov authored Sep 15, 2020

* Update .appveyor.yml

* Update .travis.yml

* Update .vsts-ci.yml

* Update main.yml

* Update setup.py

a3b9dae7

11 Aug, 2020 1 commit
- [python] try to use bundled distutils to setuptools during setup (#3294) · 97d5758f
  Nikita Titov authored Aug 11, 2020
  
  97d5758f
09 May, 2020 1 commit
- [python-package] Use set for OS check (#3059) · 1b4cddfb
  James Lamb authored May 09, 2020
  
  1b4cddfb
13 Apr, 2020 1 commit
- [python][ci] start to support Python 3.8 (#2713) · 6c19539e
  Nikita Titov authored Apr 14, 2020
```
* start to support Python 3.8

* update configs

* hotfix
```
  6c19539e
10 Apr, 2020 1 commit

[python] Re-enable scikit-learn 0.22+ support (#2949) · c633c6c2

Nikita Titov authored Apr 10, 2020

* Revert "specify the last supported version of scikit-learn (#2637)"

This reverts commit d1002776.

* ban scikit-learn 0.22.0 and skip broken test

* fix updated test

* fix lint test

* Revert "fix lint test"

This reverts commit 8b4db0805fe7a9e7f7eb0be3eac231f85026d196.

c633c6c2

12 Jan, 2020 1 commit

removed OpenMP ugly fix for Mojave (#2674) · 8944b5e6

Nikita Titov authored Jan 12, 2020

* removed OpenMP ugly fix for Mojave as it's fixed in latest CMake

* test: switch compilers

* switch compilers back

8944b5e6

19 Dec, 2019 1 commit
- specify the last supported version of scikit-learn (#2637) · d1002776
  Nikita Titov authored Dec 19, 2019
  
  d1002776
27 Oct, 2019 1 commit
- [python] removed unused pylint directives (#2466) · 00d1e693
  Nikita Titov authored Oct 27, 2019
  
  00d1e693
28 Aug, 2019 1 commit
- [python] correctly handle full path to compiler in CC and CXX env vars (#2345) · 5f5dcff6
  Nikita Titov authored Aug 28, 2019
  
  5f5dcff6
17 Aug, 2019 1 commit
- [docs][python] allow usage and compilation of 32-bit library (#2188) · 686b0cc2
  Nikita Titov authored Aug 17, 2019
```
* allow usage and compilation of 32-bit library

* added docs
```
  686b0cc2
08 May, 2019 1 commit
- [docs] updated Microsoft GitHub URL (#2152) · 94fbe5bb
  Guolin Ke authored May 08, 2019
```
* fix travis badge

* updated GitHub Microsoft URL
```
  94fbe5bb
29 Apr, 2019 1 commit

[python][R][docs][ci] better compatibility with Visual Studio 2019 (#2083) · 8d2ec69f

Nikita Titov authored Apr 29, 2019

* set platform via A option

* style hotfix

* updated R installation script

* updated Python installation script

* updated CI test script

* provide VS version-ingependent link for redistributables download

* added link to VS 2019 redistributables

* added VS 2019 match for Boost binaries

8d2ec69f

20 Mar, 2019 1 commit
- removed Python 3.4 support (#2051) · 7a89d005
  Nikita Titov authored Mar 20, 2019
  
  7a89d005
02 Feb, 2019 1 commit
- [python] added OpenMP options for python-package installation (#1975) · 40486b6c
  Nikita Titov authored Feb 02, 2019
```
* added OpenMP options for python-package installation

* fixed grammar typo
```
  40486b6c
27 Jan, 2019 1 commit
- [python][R][docs] added possibility to install with Visual Studio 2019 Preview (#1956) · c306c7b9
  Nikita Titov authored Jan 27, 2019
  
  c306c7b9
26 Jan, 2019 1 commit

Fix build on macOS Mojave (#1923) · b9684991

Tsukasa OMOTO authored Jan 26, 2019

* Fix build on macOS Mojave

Fixed #1898

- https://iscinumpy.gitlab.io/post/omp-on-high-sierra/
- https://cliutils.gitlab.io/modern-cmake/chapters/packages/OpenMP.html
- https://github.com/Homebrew/homebrew-core/pull/20589

* update setup.py

* update docs

* fix setup.py

* update docs

* update docs

* update setup.py

* update docs

b9684991

07 Nov, 2018 1 commit
- [python] fixed logging during installation (#1828) · d8d01e39
  Nikita Titov authored Nov 07, 2018
  
  d8d01e39
05 Nov, 2018 1 commit
- [docs] documented version without OpenMP support (#1811) · 5d46771a
  Nikita Titov authored Nov 05, 2018
  
  5d46771a
25 Oct, 2018 1 commit
- [python] add option to disable openmp in python package build (#1780) · 96cb6984
  Guolin Ke authored Oct 26, 2018
```
* add option to disable openmp in python package build

* fix pylint
```
  96cb6984
17 Oct, 2018 1 commit

[ci] speed up the linux builds (#1745) · 1a1255ac

Guolin Ke authored Oct 17, 2018

* speed up the build

* remove gpu docker

* add boost ppa

* remove duplicate installs

* Update test.sh

* conda install mkl qt

* addd windows

* Update .vsts-ci.yml

* fix gpu boost

1a1255ac

26 Aug, 2018 2 commits
- [python] copy files silently during the installation (#1617) · e8916685
  Nikita Titov authored Aug 26, 2018
  
  e8916685
- [python] fixed MPI version installation (#1616) · c5156fe0
  Nikita Titov authored Aug 26, 2018
  
  c5156fe0
19 Aug, 2018 1 commit

[python][ci] start to support Python 3.7 (#1584) · 1b53fa07

Nikita Titov authored Aug 19, 2018

* added python 3.7 to the list of supporting versions

* reorganized python versions at Travis

* addressed review comments

1b53fa07

29 Jul, 2018 1 commit
- [python] open files with utf8 encoding during installation (#1548) · 225eb5ed
  Nikita Titov authored Jul 29, 2018
  
  225eb5ed
15 Jul, 2018 1 commit

[ci][python] improved paths in setup.py and small CI refactoring (#1513) · c6cdea75

Nikita Titov authored Jul 15, 2018

* fixed paths in python-package installation

* less cd commands at CI

* hotfix

* added copying missed file from windows directory

* not copy filters file

* refined paths in nuget creation script

* removed filters file from MANIFEST.in

c6cdea75

03 Jun, 2018 1 commit

[docs][python] made OS detection more reliable and little docs improvements (#1414) · a39c848e

Nikita Titov authored Jun 03, 2018

* added missed description of plot_example in python_guide folder and fixed consistency for packages naming

* more reliable OS detection

* fixed grammar

* made pylint happy

a39c848e

28 Feb, 2018 1 commit
- added info and possibility to install Python-package HDFS version (#1252) · 5f2d5618
  Nikita Titov authored Feb 28, 2018
  
  5f2d5618
31 Dec, 2017 1 commit

[python] added option to install python-package with mpi version and... · c97abff4

Nikita Titov authored Dec 31, 2017

[python] added option to install python-package with mpi version and possibility to pass CMake options (#1034)

* fixed hierarchy of installation guide sections; made installation with proto support more verbose

* added option to install with protobuf support to python package

* fixed pylint

* added option to install MPI version to python package

* added tests for python-package installation with options --proto and --mpi

* hotfix

* removed unnecessary note from docs

* removed unnecessary line from travis script

* reverted proto

* Revert "reverted proto"

This reverts commit 5203448bb90a13844cf330527ca971eabb45a4a8.

* reverted protobuf from python-package

* added options for cmake in python-package

* fixed typos

* fixed options readability

* reworked installation logging

* fixed docs according to review comments

* fixed pylint

* fixed space in log path

c97abff4

01 Nov, 2017 1 commit
- changed minimum version of VS to 2015 (#1027) · b3c20f7a
  Nikita Titov authored Nov 01, 2017
  
  b3c20f7a
18 Sep, 2017 1 commit
- [python] avoid copying full windows folder. · 7689a4d0
  Guolin Ke authored Sep 18, 2017
  
  7689a4d0
10 Sep, 2017 1 commit

[python] [setup] removing source files (#898) · 9eab7ec8

Nikita Titov authored Sep 10, 2017

* travis cleanup

* removed precompiled files in windows folder from sdist command

* removed rubbish from install folder

* added compute folder

9eab7ec8

08 Sep, 2017 1 commit

[python] [setup] improving installation (#880) · 8984111f

Nikita Titov authored Sep 08, 2017

* disabled logs from compilers; fixed #874

* fixed safe clear_fplder

* added windows folder to manifest.in

* added windows folder to build

* added library path

* added compilation with MSBuild from .sln-file

* fixed unknown PlatformToolset returns exitcode 0

* hotfix

* updated Readme

* removed return

* added installation with mingw test to appveyor

* let's test appveyor with both VS 2015 and VS 2017; but MinGW isn't installed on VS 2017 image

* fixed built-in name 'file'

* simplified appveyor

* removed excess data_files

* fixed unreadable paths

* separated exceptions for cmake and mingw

* refactored silent_call

* don't create artifacts with VS 2015 and mingw

* be more precise with python versioning in Travis

* removed unnecessary if statement

* added classifiers for PyPI and python versions badge

* changed python version in travis

* added support of scikit-learn 0.18.x

* added more python versions to Travis

* added more python versions to Appveyor

* reduced number of tests in Travis

* Travis trick is not needed anymore

* attempt to fix according to https://github.com/Microsoft/LightGBM/pull/880#discussion_r137438856

8984111f

01 Sep, 2017 1 commit
- [pip] fix long_description (#875) · ac0c9cec
  Guolin Ke authored Sep 01, 2017
  
  ac0c9cec
18 Aug, 2017 1 commit
- python-package: fix version (#832) · aa99c40a
  Tsukasa OMOTO authored Aug 16, 2017
```
* python-package: fix version

* remove version variable
```
  aa99c40a