Commits · 792c930305a818db1463911c2cbee92d462eaab9 · tianlh / LightGBM-DCU

08 Dec, 2020 1 commit

Fix model locale issue and improve model R/W performance. (#3405) · 792c9303

Alberto Ferreira authored Dec 08, 2020

* Fix LightGBM models locale sensitivity and improve R/W performance.

When Java is used, the default C++ locale is broken. This is true for
Java providers that use the C API or even Python models that require JEP.

This patch solves that issue making the model reads/writes insensitive
to such settings.
To achieve it, within the model read/write codebase:
 - C++ streams are imbued with the classic locale
 - Calls to functions that are dependent on the locale are replaced
 - The default locale is not changed!

This approach means:
 - The user's locale is never tampered with, avoiding issues such as
    https://github.com/microsoft/LightGBM/issues/2979 with the previous
    approach https://github.com/microsoft/LightGBM/pull/2891
 - Datasets can still be read according the user's locale
 - The model file has a single format independent of locale

Changes:
 - Add CommonC namespace which provides faster locale-independent versions of Common's methods
 - Model code makes conversions through CommonC
 - Cleanup unused Common methods
 - Performance improvements. Use fast libraries for locale-agnostic conversion:
   - value->string: https://github.com/fmtlib/fmt
   - string->double: https://github.com/lemire/fast_double_parser (10x
      faster double parsing according to their benchmark)

Bugfixes:
 - https://github.com/microsoft/LightGBM/issues/2500
 - https://github.com/microsoft/LightGBM/issues/2890
 - https://github.com/ninia/jep/issues/205

 (as it is related to LGBM as well)

* Align CommonC namespace

* Add new external_libs/ to python setup

* Try fast_double_parser fix #1

Testing commit e09e5aad828bcb16bea7ed0ed8322e019112fdbe

If it works it should fix more LGBM builds

* CMake: Attempt to link fmt without explicit PUBLIC tag

* Exclude external_libs from linting

* Add exernal_libs to MANIFEST.in

* Set dynamic linking option for fmt.

* linting issues

* Try to fix lint includes

* Try to pass fPIC with static fmt lib

* Try CMake P_I_C option with fmt library

* [R-package] Add CMake support for R and CRAN

* Cleanup CMakeLists

* Try fmt hack to remove stdout

* Switch to header-only mode

* Add PRIVATE argument to target_link_libraries

* use fmt in header-only mode

* Remove CMakeLists comment

* Change OpenMP to PUBLIC linking in Mac

* Update fmt submodule to 7.1.2

* Use fmt in header-only-mode

* Remove fmt from CMakeLists.txt

* Upgrade fast_double_parser to v0.2.0

* Revert "Add PRIVATE argument to target_link_libraries"

This reverts commit 3dd45dde7b92531b2530ab54522bb843c56227a7.

* Address James Lamb's comments

* Update R-package/.Rbuildignore
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Upgrade to fast_double_parser v0.3.0 - Solaris support

* Use legacy code only in Solaris

* Fix lint issues

* Fix comment

* Address StrikerRUS's comments (solaris ifdef).

* Change header guards
Co-authored-by: James Lamb <jaylamb20@gmail.com>

792c9303

26 Nov, 2020 1 commit
- [python] Allow to build Python wheel package (#3594) · 918ccaec
  Martynov Maxim authored Nov 26, 2020
```
Co-authored-by: Мартынов Максим Сергеевич <msmarty5@mts.ru>
```
  918ccaec
17 Nov, 2020 1 commit

[ci] remove conda from R CI jobs (#3573) · 34187a86

James Lamb authored Nov 17, 2020

* [ci] always use latest conda version on GitHub Actions

* try new use-conda version

* just remove conda

* skip conda even more

34187a86

20 Sep, 2020 1 commit

[GPU] Add support for CUDA-based GPU build (#3160) · f7ad9457

Chip Kerchner authored Sep 20, 2020

* Initial CUDA work

* redirect log to python console (#3090)

* redir log to python console

* fix pylint

* Apply suggestions from code review

* Update basic.py

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update c_api.h

* Apply suggestions from code review

* super-minor: better wording
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: StrikerRUS <nekit94-12@hotmail.com>

* re-order includes (fixes #3132) (#3133)

* Revert "re-order includes (fixes #3132) (#3133)" (#3153)

This reverts commit 656d2676

* Missing change from previous rebase

* Minor cleanup and removal of development scripts.

* Only set gpu_use_dp on by default for CUDA. Other minor change.

* Fix python lint indentation problem.

* More python lint issues.

* Big lint cleanup - more to come.

* Another large lint cleanup - more to come.

* Even more lint cleanup.

* Minor cleanup so less differences in code.

* Revert is_use_subset changes

* Another rebase from master to fix recent conflicts.

* More lint.

* Simple code cleanup - add & remove blank lines, revert unneccessary format changes, remove added dead code.

* Removed parameters added for CUDA and various bug fix.

* Yet more lint and unneccessary changes.

* Revert another change.

* Removal of unneccessary code.

* temporary appveyor.yml for building and testing

* Remove return value in ReSize

* Removal of unused variables.

* Code cleanup from reviewers suggestions.

* Removal of FIXME comments and unused defines.

* More reviewers comments cleanup.

* Fix config variables.

* Attempt to fix check-docs failure

* Update Paramster.rst for num_gpu

* Removing test appveyor.yml

* Add CUDA_RESOLVE_DEVICE_SYMBOLS to libraries to fix linking issue.

* Fixed handling of data elements less than 2K.

* More reviewers comments cleanup.

* Removal of TODO and fix printing of int64_t

* Add cuda change for CI testing and remove cuda from device_type in python.

* Missed one change form previous check-in

* Removal AdditionConfig and fix settings.

* Limit number of GPUs to one for now in CUDA.

* Update Parameters.rst for previous check-in

* Whitespace removal.

* Cleanup unused code.

* Changed uint/ushort/ulong to unsigned int/short/long to help Windows based CUDA compiler work.

* Lint change from previous check-in.

* Changes based on reviewers comments.

* More reviewer comment changes.

* Adding warning for is_sparse. Revert tmp_subset code. Only return FeatureGroupData if not is_multi_val_

* Fix so that CUDA code will compile even if you enable the SCORE_T_USE_DOUBLE define.

* Reviewer comment cleanup.

* Replace warning with Log message. Removal of some of the USE_CUDA. Fix typo and removal of pragma once.

* Remove PRINT debug for CUDA code.

* Allow to use of multiple GPUs for CUDA.

* More multi-GPUs enablement for CUDA.

* More code cleanup based on reviews comments.

* Update docs with latest config changes.
Co-authored-by: Gordon Fossum <fossum@us.ibm.com>
Co-authored-by: ChipKerchner <ckerchne@linux.vnet.ibm.com>
Co-authored-by: Guolin Ke <guolin.ke@outlook.com>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: StrikerRUS <nekit94-12@hotmail.com>
Co-authored-by: James Lamb <jaylamb20@gmail.com>

f7ad9457

14 Sep, 2020 1 commit

[ci] Use Ubuntu Focal image at Travis (#3393) · 164818b0

Nikita Titov authored Sep 14, 2020

* Use Ubuntu Focal image at Travis

* test gcc

* test Bionic and gcc

* test Focal with another Python version for GPU test

* revert testing changes

* install linkchecker via pip

* simplify linkchecker config

164818b0

05 Jul, 2020 2 commits

[ci][docs][R-package] catch builds that have not updated docs (#3205) · d8bc0fc7

James Lamb authored Jul 05, 2020



* [ci] [R-package] catch builds that have not updated docs

* drop reliance on .Renviron

* remove docs changes

* Update .ci/test_r_package.sh
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* make all R tasks start with r-package
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

d8bc0fc7

[ci] bump gcc version for macOS (#3208) · bf01f53a
Nikita Titov authored Jul 05, 2020
```
* update gcc version

* Update setup.sh

* fix GitHub Actions
```
bf01f53a

25 May, 2020 1 commit
- [R-package] move R source files into R-package, reduce duplication in build_r.R (#3087) · 4d43e96b
  James Lamb authored May 25, 2020
```
* [R-package] move R source files into R-package

* fix linting warning

* stuff
```
  4d43e96b
16 Apr, 2020 1 commit
- [ci] Changed use of strcpy to snprintf (fixes #1990) (#2973) · 9843506e
  James Lamb authored Apr 16, 2020
```
* [ci] Changed use of strcpy to snprintf

* fix

* fully enable cpplint
```
  9843506e
13 Apr, 2020 1 commit
- [ci] fixed r-lintr version constraint (#2990) · be591c84
  Nikita Titov authored Apr 14, 2020
```
* Update test.sh

* update brew to pull new URL for basictex
```
  be591c84
10 Apr, 2020 1 commit

[python] Re-enable scikit-learn 0.22+ support (#2949) · c633c6c2

Nikita Titov authored Apr 10, 2020

* Revert "specify the last supported version of scikit-learn (#2637)"

This reverts commit d1002776.

* ban scikit-learn 0.22.0 and skip broken test

* fix updated test

* fix lint test

* Revert "fix lint test"

This reverts commit 8b4db0805fe7a9e7f7eb0be3eac231f85026d196.

c633c6c2

28 Mar, 2020 1 commit
- [ci] changed CRAN mirror to fix failing installs (#2954) · 03ce02aa
  James Lamb authored Mar 27, 2020
  
  03ce02aa
01 Mar, 2020 1 commit

Debug flags (#2825) · 8d90bbe3

Guolin Ke authored Mar 01, 2020



* add debug mode in camke

* add debug dll

* Apply suggestions from code review
Co-Authored-By: Nikita Titov <nekit94-08@mail.ru>

* fix naming

* Apply suggestions from code review

* Apply suggestions from code review

* Update LightGBM.sln

* refine

* run MPI job in debug mode

* document USE_DEBUG and USE_TIMETAG
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

8d90bbe3

16 Jan, 2020 1 commit
- [ci] indicate support only for the last 3 macOS versions in wheel name (#2691) · b588cae2
  Nikita Titov authored Jan 16, 2020
```
* indicate support only for the last 3 macOS versions in wheel name

* document support only for 3 newest macOS versions
```
  b588cae2
12 Jan, 2020 1 commit

removed OpenMP ugly fix for Mojave (#2674) · 8944b5e6

Nikita Titov authored Jan 12, 2020

* removed OpenMP ugly fix for Mojave as it's fixed in latest CMake

* test: switch compilers

* switch compilers back

8944b5e6

19 Dec, 2019 1 commit
- specify the last supported version of scikit-learn (#2637) · d1002776
  Nikita Titov authored Dec 19, 2019
  
  d1002776
15 Dec, 2019 1 commit
- [R-package][ci] added CI stage for R package (fixes #2335, fixes #2569) (#2530) · 86ca4845
  James Lamb authored Dec 15, 2019
  
  86ca4845
16 Nov, 2019 1 commit

[R-package][ci] Added more R linters (fixes #2477) (#2533) · 1f7b06b9

James Lamb authored Nov 16, 2019



* Added more linters on R code

* started working on implicit integers

* finished style changes to handle implicit integers

* regenned documentation and added concatenation linter

* changed channel for r-lintr

* try building stringi before lintr

* trying to get libicui18n

* trying another thing

* trying conda-forge again

* added re-install of stringi

* uncommented other stages

* Update .ci/test.sh
Co-Authored-By: Nikita Titov <nekit94-08@mail.ru>

* removed apt update and changed lintr version floor

* get lintr from CRAN

* R needs to come before C++ linting

* testing lintr install from CRAN

* trying one more thing

* more verbose

* order might matter

* removed commented code

* cleaned up linting block in test.sh

* grouped conda install calls and fixed a few integer array things

1f7b06b9

10 Nov, 2019 1 commit
- support Catalina (#2556) · 76bda1e2
  Nikita Titov authored Nov 11, 2019
  
  76bda1e2
24 Oct, 2019 1 commit
- [R-package] added R linting and changed R code to comma-first (fixes #2373) (#2437) · fc991c9d
  James Lamb authored Oct 24, 2019
  
  fc991c9d
02 Oct, 2019 1 commit
- [ci] added build/c++11 to cpplint filter (#2487) · 67099ff8
  Nikita Titov authored Oct 02, 2019
  
  67099ff8
21 Sep, 2019 1 commit
- [ci] added build/include_subdir check to filter for cpplint (#2427) · 4f89cc10
  Nikita Titov authored Sep 21, 2019
  
  4f89cc10
19 Sep, 2019 1 commit
- [ci] added cpplint check (#2416) · 306c6dbf
  Nikita Titov authored Sep 19, 2019
```
* added cpplint heck

* removed TODO from code

* change order of checks
```
  306c6dbf
03 Sep, 2019 1 commit
- [ci][tests] install joblib for test directly (#2374) · df26b65d
  Nikita Titov authored Sep 03, 2019
  
  df26b65d
14 May, 2019 1 commit

[ci] added swig compilation on macOS and Windows with artifacts (#2170) · 2a369170

Nikita Titov authored May 14, 2019

* compile SWIG artifact on Windows

* compile SWIG artifact on Windows

* combined swig and sdist jobs for Linux

* added conditions for artifacts publishing

* added SWIG artifact compilation on macOS

* hotfix

* test: switch compilers

* harsh workaround for OpenMP library conflict

* switch compilers back

* consistent order of tasks in condition

* dummy commit to trigger CI and fix CLA

2a369170

12 May, 2019 1 commit

[ci][docs] updated fix for OpenMP conflicts (#2166) · 76170788

Nikita Titov authored May 12, 2019

* test: AppleClang on Travis

* updated fix for OpenMP multiple instances

* updated docs

* switch compiler back

76170788

09 May, 2019 1 commit

[ci] CI fixes (#2161) · abbbbd75

Nikita Titov authored May 09, 2019

* Update setup.sh

* Update test.sh

* Update setup.sh

* test: gcc on Azure

* Update test.sh

* Update setup.sh

* Update test.sh

* switch compiler back on Azure

abbbbd75

06 May, 2019 1 commit
- [ci] nose is not needed anymore (#2151) · 2c41d15e
  Nikita Titov authored May 06, 2019
  
  2c41d15e
05 May, 2019 1 commit

[ci][docs] generate docs for C API (#2059) · cfcc020e

Nikita Titov authored May 05, 2019

* use file to install deps for docs

* added C_API docs

* use breathe without exhale

* added missed params descriptions and make Doxygen fail for warnings

* escape char hotfix

* ignore unknown directive for rstcheck

* better handle env variable

* hotfix for 'Unknown directive type' error with C_API=NO

* Update .gitignore

* fixed pylint

* use already defined constants in conf.py

* do not suppress Doxygen's output

* addressed review comments

* removed unneeded import

cfcc020e

30 Apr, 2019 1 commit

[ci] added job for swig compilation on Linux with artifacts (#2125) · 739c2635

Nikita Titov authored Apr 30, 2019

* added job for swig compilation on Linux with artifacts

* hotfix

* test: SWIG with Clang compiler

* switch compiler back to gcc

* switch to main docker

739c2635

10 Apr, 2019 1 commit

[ci] update macOS on Travis to Mojave (#2086) · 0e025e9c

Nikita Titov authored Apr 10, 2019

* added fix for OpenMP on macOS into test script

* test: AppleClang on Travis

* use Mojave on Travis

* bash hotfix

* get back to gcc compiler on Travis macOS

0e025e9c

18 Mar, 2019 1 commit
- removed hotfix (#2054) · babe8b31
  Nikita Titov authored Mar 19, 2019
  
  babe8b31
14 Mar, 2019 1 commit
- [ci] compatibility hotfix for notebook execution (#2048) · b020a25d
  Nikita Titov authored Mar 14, 2019
```
* ci fix

* ci fix for Appveyor

* actually firx Appveyor
```
  b020a25d
18 Feb, 2019 1 commit
- fixed conda (#2016) · b447a7bc
  Nikita Titov authored Feb 18, 2019
  
  b447a7bc
30 Jan, 2019 1 commit

[tests][python] added test for huge string model (#1964) · fe77eac5

Nikita Titov authored Jan 30, 2019

* added test for huge string model

* fixed tree sizes field

* simplified model structure

* fixed test and added try/except

fe77eac5

13 Nov, 2018 1 commit
- [ci][python] fall back to producing one wheel for macOS and drop support of... · a694712b
  Nikita Titov authored Nov 13, 2018
```
[ci][python] fall back to producing one wheel for macOS and drop support of OSX 10.6 and 10.7 (#1826)
```
  a694712b
11 Nov, 2018 1 commit
- use latest Sphinx with rstcheck (#1836) · 71bd9ed5
  Nikita Titov authored Nov 11, 2018
  
  71bd9ed5
06 Nov, 2018 1 commit
- slightly refactored ci tests script (#1827) · 28e47aae
  Nikita Titov authored Nov 06, 2018
  
  28e47aae
05 Nov, 2018 1 commit

[ci] run example jupyter notebooks as part of tests at the CI side (#1807) · 5c8e365f

Nikita Titov authored Nov 05, 2018

* execute notebooks on CIs

* run notebooks in interactive mode and increase timeout for MinGW

* hotfix alphabetically sort packages installation

5c8e365f

01 Nov, 2018 1 commit
- [ci] check dynamic symbol versions at CI side (#1812) · dfe0fae4
  Nikita Titov authored Nov 01, 2018
```
* renamed helper folder to helpers

* added library dependencies check
```
  dfe0fae4