Commits · 792c930305a818db1463911c2cbee92d462eaab9 · tianlh / LightGBM-DCU

08 Dec, 2020 1 commit

Fix model locale issue and improve model R/W performance. (#3405) · 792c9303

Alberto Ferreira authored Dec 08, 2020

* Fix LightGBM models locale sensitivity and improve R/W performance.

When Java is used, the default C++ locale is broken. This is true for
Java providers that use the C API or even Python models that require JEP.

This patch solves that issue making the model reads/writes insensitive
to such settings.
To achieve it, within the model read/write codebase:
 - C++ streams are imbued with the classic locale
 - Calls to functions that are dependent on the locale are replaced
 - The default locale is not changed!

This approach means:
 - The user's locale is never tampered with, avoiding issues such as
    https://github.com/microsoft/LightGBM/issues/2979 with the previous
    approach https://github.com/microsoft/LightGBM/pull/2891
 - Datasets can still be read according the user's locale
 - The model file has a single format independent of locale

Changes:
 - Add CommonC namespace which provides faster locale-independent versions of Common's methods
 - Model code makes conversions through CommonC
 - Cleanup unused Common methods
 - Performance improvements. Use fast libraries for locale-agnostic conversion:
   - value->string: https://github.com/fmtlib/fmt
   - string->double: https://github.com/lemire/fast_double_parser (10x
      faster double parsing according to their benchmark)

Bugfixes:
 - https://github.com/microsoft/LightGBM/issues/2500
 - https://github.com/microsoft/LightGBM/issues/2890
 - https://github.com/ninia/jep/issues/205

 (as it is related to LGBM as well)

* Align CommonC namespace

* Add new external_libs/ to python setup

* Try fast_double_parser fix #1

Testing commit e09e5aad828bcb16bea7ed0ed8322e019112fdbe

If it works it should fix more LGBM builds

* CMake: Attempt to link fmt without explicit PUBLIC tag

* Exclude external_libs from linting

* Add exernal_libs to MANIFEST.in

* Set dynamic linking option for fmt.

* linting issues

* Try to fix lint includes

* Try to pass fPIC with static fmt lib

* Try CMake P_I_C option with fmt library

* [R-package] Add CMake support for R and CRAN

* Cleanup CMakeLists

* Try fmt hack to remove stdout

* Switch to header-only mode

* Add PRIVATE argument to target_link_libraries

* use fmt in header-only mode

* Remove CMakeLists comment

* Change OpenMP to PUBLIC linking in Mac

* Update fmt submodule to 7.1.2

* Use fmt in header-only-mode

* Remove fmt from CMakeLists.txt

* Upgrade fast_double_parser to v0.2.0

* Revert "Add PRIVATE argument to target_link_libraries"

This reverts commit 3dd45dde7b92531b2530ab54522bb843c56227a7.

* Address James Lamb's comments

* Update R-package/.Rbuildignore
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Upgrade to fast_double_parser v0.3.0 - Solaris support

* Use legacy code only in Solaris

* Fix lint issues

* Fix comment

* Address StrikerRUS's comments (solaris ifdef).

* Change header guards
Co-authored-by: James Lamb <jaylamb20@gmail.com>

792c9303

07 Dec, 2020 1 commit

[R-package] fix partial matching of keyword arguments in lgb.cv() (fixes #3629) (#3630) · f38f118c

James Lamb authored Dec 07, 2020

* [ci] [R-package] fix issue with partial argument name matches

* fix partial name matches

* simpler NOTE check

* change conditions

* simplify

f38f118c

29 Nov, 2020 1 commit

[R-package] Use command-line args to customize CMake builds (fixes #2441) (#3574) · 1b8cf2d2

James Lamb authored Nov 29, 2020



* [R-package] Use command-line args to customize CMake builds (fixes #2441)

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

1b8cf2d2

27 Nov, 2020 1 commit
- [R-package][tests] update values in valgrind test (#3600) · 6be3e57d
  Nikita Titov authored Nov 27, 2020
  
  6be3e57d
26 Nov, 2020 1 commit
- [python] Allow to build Python wheel package (#3594) · 918ccaec
  Martynov Maxim authored Nov 26, 2020
```
Co-authored-by: Мартынов Максим Сергеевич <msmarty5@mts.ru>
```
  918ccaec
23 Nov, 2020 1 commit
- [ci] test 32-bit R in CI (#3588) · c5d9d243
  James Lamb authored Nov 23, 2020
```
* [ci] test 32-bit R in CI

* add R 3.6 Windows CRAN job

* add tests
```
  c5d9d243
17 Nov, 2020 1 commit

[ci] remove conda from R CI jobs (#3573) · 34187a86

James Lamb authored Nov 17, 2020

* [ci] always use latest conda version on GitHub Actions

* try new use-conda version

* just remove conda

* skip conda even more

34187a86

13 Nov, 2020 1 commit

[ci] Use different PATH based on Rtools version (#3558) · b80240ce

James Lamb authored Nov 13, 2020

* [ci] Use different PATH based on Rtools version

* better paths

* comment out CIs

* empty commit

* empty commit

* empty commit

* empty commit

* empty commit

* empty commit

* uncomment CIs

* reorganize PATH components

b80240ce

08 Nov, 2020 1 commit
- [ci] simplify R CI scripts (#3535) · e93fd333
  James Lamb authored Nov 08, 2020
```
* [ci] simplify R CI scripts

* update GitHub Actions workflow
```
  e93fd333
07 Nov, 2020 1 commit
- [ci] Make valgrind check stricter (#3536) · c0ffb71e
  James Lamb authored Nov 07, 2020
  
  c0ffb71e
06 Nov, 2020 1 commit
- [R-package] [ci] add correct path for mingw32-make in CI (#3523) · 35d6ba6f
  James Lamb authored Nov 06, 2020
  
  35d6ba6f
26 Oct, 2020 1 commit
- setup CUDA CI job (#3424) · 12e220b7
  Nikita Titov authored Oct 26, 2020
```
* setup CUDA CI job

* minor updates
```
  12e220b7
22 Oct, 2020 1 commit

[ci] update SWIG installation on macOS (#3477) · a53abd96

Nikita Titov authored Oct 23, 2020

* Update .vsts-ci.yml

* Update setup.sh

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* try not to hardcode path

a53abd96

20 Oct, 2020 1 commit

[R-package] Improves logging in lint_r_code.R (fixes #3471) (#3472) · aab8cdc5

Ashish Sinha authored Oct 20, 2020



* Update lint_r_code.R

* improves logging in lint_r_code

* Update .ci/lint_r_code.R
Co-authored-by: James Lamb <jaylamb20@gmail.com>
Co-authored-by: James Lamb <jaylamb20@gmail.com>

aab8cdc5

18 Oct, 2020 2 commits

[ci] [R-package] Fix memory leaks found by valgrind (#3443) · 81d76113

James Lamb authored Oct 18, 2020



* fix int64 write error

* attempt

* [WIP] [ci] [R-package] Add CI job that runs valgrind tests

* update all-successful

* install

* executable

* fix redirect stuff

* Apply suggestions from code review
Co-authored-by: Guolin Ke <guolin.ke@outlook.com>

* more flags

* add mc to msvc proj

* fix memory leak in mc

* Update monotone_constraints.hpp

* Update r_package.yml

* remove R_INT64_PTR

* disable openmp

* Update gbdt_model_text.cpp

* Update gbdt_model_text.cpp

* Apply suggestions from code review

* try to free vector

* free more memories.

* Update src/boosting/gbdt_model_text.cpp

* fix using

* try the UNPROTECT(1);

* fix a const pointer

* fix Common

* reduce UNPROTECT

* remove UNPROTECT(1);

* fix null handle

* fix predictor

* use NULL after free

* fix a leaking in test

* try more fixes

* test the effect of tests

* throw exception in Fatal

* add test back

* Apply suggestions from code review

* commet some tests

* Apply suggestions from code review

* Apply suggestions from code review

* trying to comment out tests

* Update openmp_wrapper.h

* Apply suggestions from code review

* Update configure

* Update configure.ac

* trying to uncomment

* more comments

* more uncommenting

* more uncommenting

* fix comment

* more uncommenting

* uncomment fully-commented out stuff

* try uncommenting more dataset tests

* uncommenting more tests

* ok getting closer

* more uncommenting

* free dataset

* skipping a test, more uncommenting

* more skipping

* re-enable OpenMP

* allow on OpenMP thing

* move valgrind to comment-only job

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* changes from code review

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* linting

* issue comments too

* remove issue_comment
Co-authored-by: Guolin Ke <guolin.ke@outlook.com>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

81d76113

[ci] ignore R CMD CHECK warnings on new R version (#3468) · c182555d
James Lamb authored Oct 18, 2020
```
* [ci] ignore R CMD CHECK warnings on new R version

* just update version
```
c182555d

16 Oct, 2020 1 commit

[R-package] Suppresses unknown pragma warnings during CRAN build (#3460) · 2e1b39ba

Josh Maxwell authored Oct 16, 2020



* Modified script to also remove pragma warnings.

This also includes modifying the scope of the pragma removal.
Previously this script only searched inside the ./src/include/LightGMB directory.
This was not inclusive enough to remove the warnings in files shown in the ticket.

* Adds CI test that exits if unknown pragma warnings are present.

* Expanding pragma removal to cpp and hpp files.

* Update .ci/test_r_package.sh

Removing unneeded conditions since this script will only run on Linux and Mac builds anyway.
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Update .ci/test_r_package.sh

Fixes typo
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* replacing double quotes with single quotes

* Using a more portable find syntax so it works on macOS and Linux
Co-authored-by: James Lamb <jaylamb20@gmail.com>

2e1b39ba

08 Oct, 2020 1 commit

[R-package] miscellaneous changes to comply with CRAN requirements (#3338) · 186711de

James Lamb authored Oct 08, 2020



* [R-package] update DESCRIPTION per CRAN comments

* newlines

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* more fixes

* update Rbuildignore

* more changes

* more changes per CRAN response

* add email

* run examples in CI

* add newest CRAN response

* add Solaris patch

* update patch

* another attempt at ifaddrs patch

* fix unnecessary comment

* update configure

* comments

* bump version

* tabs

* fix address alignment, required by cran (#3415)

* fix dataset binary file alignment

* many fixes

* fix warnings

* fix bug

* Update file_io.cpp

* Update file_io.cpp

* simplify code

* Apply suggestions from code review

* general

* remove unneeded alignment

* Update file_io.h

* int32 to byte8 alignment

* Apply suggestions from code review

* Apply suggestions from code review

* [R-package] add new copyright holder in DESCRIPTION (#3409)

* [R-package] add new copyright holder in DESCRIPTION

* fix role

* fixing conflicts

* [R-package] add new copyright holder in DESCRIPTION (#3409)

* [R-package] add new copyright holder in DESCRIPTION

* fix role

* trying to fix conflicts

* more fixes

* this will work

* update cran-comments

* simplify solaris, add more testing docs

* stuff

* remove rchck docs

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* remove extra use of cat()

* change solaris check

* update docs

* remove testing code

* fix warning about cleanup not having execute permissions

* fix cmake builds

* remove blank line
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Guolin Ke <guolin.ke@outlook.com>

186711de

06 Oct, 2020 1 commit

[R-package][ci] Make Windows CI fail if R CMD check fails (#3435) · 967a7814

James Lamb authored Oct 06, 2020

* [R-package] [ci] Make Windows CI fail if R CMD check fails

* restore fix

* catch more errors

* remove testing

* fix processx calls

* try to get status code

* more logs

* trying something

* stuff

* previous changes

* uncomment

967a7814

04 Oct, 2020 1 commit
- [R-package] add 'cleanup' script to handle left-behind Makevars (#3434) · 2903ca0c
  James Lamb authored Oct 04, 2020
```
* [R-package] add 'cleanup' script to handle left-behind Makevars

* always install library for checkbashisms
```
  2903ca0c
20 Sep, 2020 1 commit

[GPU] Add support for CUDA-based GPU build (#3160) · f7ad9457

Chip Kerchner authored Sep 20, 2020

* Initial CUDA work

* redirect log to python console (#3090)

* redir log to python console

* fix pylint

* Apply suggestions from code review

* Update basic.py

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update c_api.h

* Apply suggestions from code review

* super-minor: better wording
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: StrikerRUS <nekit94-12@hotmail.com>

* re-order includes (fixes #3132) (#3133)

* Revert "re-order includes (fixes #3132) (#3133)" (#3153)

This reverts commit 656d2676

* Missing change from previous rebase

* Minor cleanup and removal of development scripts.

* Only set gpu_use_dp on by default for CUDA. Other minor change.

* Fix python lint indentation problem.

* More python lint issues.

* Big lint cleanup - more to come.

* Another large lint cleanup - more to come.

* Even more lint cleanup.

* Minor cleanup so less differences in code.

* Revert is_use_subset changes

* Another rebase from master to fix recent conflicts.

* More lint.

* Simple code cleanup - add & remove blank lines, revert unneccessary format changes, remove added dead code.

* Removed parameters added for CUDA and various bug fix.

* Yet more lint and unneccessary changes.

* Revert another change.

* Removal of unneccessary code.

* temporary appveyor.yml for building and testing

* Remove return value in ReSize

* Removal of unused variables.

* Code cleanup from reviewers suggestions.

* Removal of FIXME comments and unused defines.

* More reviewers comments cleanup.

* Fix config variables.

* Attempt to fix check-docs failure

* Update Paramster.rst for num_gpu

* Removing test appveyor.yml

* Add CUDA_RESOLVE_DEVICE_SYMBOLS to libraries to fix linking issue.

* Fixed handling of data elements less than 2K.

* More reviewers comments cleanup.

* Removal of TODO and fix printing of int64_t

* Add cuda change for CI testing and remove cuda from device_type in python.

* Missed one change form previous check-in

* Removal AdditionConfig and fix settings.

* Limit number of GPUs to one for now in CUDA.

* Update Parameters.rst for previous check-in

* Whitespace removal.

* Cleanup unused code.

* Changed uint/ushort/ulong to unsigned int/short/long to help Windows based CUDA compiler work.

* Lint change from previous check-in.

* Changes based on reviewers comments.

* More reviewer comment changes.

* Adding warning for is_sparse. Revert tmp_subset code. Only return FeatureGroupData if not is_multi_val_

* Fix so that CUDA code will compile even if you enable the SCORE_T_USE_DOUBLE define.

* Reviewer comment cleanup.

* Replace warning with Log message. Removal of some of the USE_CUDA. Fix typo and removal of pragma once.

* Remove PRINT debug for CUDA code.

* Allow to use of multiple GPUs for CUDA.

* More multi-GPUs enablement for CUDA.

* More code cleanup based on reviews comments.

* Update docs with latest config changes.
Co-authored-by: Gordon Fossum <fossum@us.ibm.com>
Co-authored-by: ChipKerchner <ckerchne@linux.vnet.ibm.com>
Co-authored-by: Guolin Ke <guolin.ke@outlook.com>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: StrikerRUS <nekit94-12@hotmail.com>
Co-authored-by: James Lamb <jaylamb20@gmail.com>

f7ad9457

14 Sep, 2020 1 commit

[ci] Use Ubuntu Focal image at Travis (#3393) · 164818b0

Nikita Titov authored Sep 14, 2020

* Use Ubuntu Focal image at Travis

* test gcc

* test Bionic and gcc

* test Focal with another Python version for GPU test

* revert testing changes

* install linkchecker via pip

* simplify linkchecker config

164818b0

13 Sep, 2020 1 commit
- [ci] update Boost version (#3389) · 4f0f822b
  Nikita Titov authored Sep 13, 2020
  
  4f0f822b
09 Sep, 2020 1 commit

[ci] host some artifacts needed by R Windows CI jobs (#3359) · f7f4e602

James Lamb authored Sep 09, 2020



* [ci] host some artifacts needed by R Windows CI jobs

* remove link

* host SWIG
Co-authored-by: Guolin Ke <guolin.ke@outlook.com>

f7f4e602

06 Sep, 2020 1 commit

[ci] remove Azure checks from R CI (#3355) · 0faf8741

James Lamb authored Sep 06, 2020

* [ci] remove texlive-fonts-extra and Azure checks R from CI

* put texlive-fonts-extra back

0faf8741

29 Aug, 2020 1 commit
- [ci] fix R CMD check issue 'unable to verify current time' (#3341) · 0038a662
  James Lamb authored Aug 29, 2020
  
  0038a662
25 Aug, 2020 1 commit

[R-package] [ci] Reduce manual effort for version updates (#3335) · 5538eb8d

James Lamb authored Aug 25, 2020



* [R-package] [ci] Reduce manual effort for version updates

* fix configure

* remove useless comment

* Update build_r.R
Co-authored-by: Guolin Ke <guolin.ke@outlook.com>

* fix recreate-configure check in CI
Co-authored-by: Guolin Ke <guolin.ke@outlook.com>

5538eb8d

19 Aug, 2020 1 commit
- [ci] bump Xcode version at Travis (#3323) · 0aa48690
  Nikita Titov authored Aug 19, 2020
```
* bump Xcode version at Travis

* remove temp fix for gcc at Travis
```
  0aa48690
18 Aug, 2020 1 commit
- [ci] Test 32-bit builds for R package (#3311) · 8662e36d
  James Lamb authored Aug 18, 2020
```
* [ci] Test 32-bit builds for R package

* update README

* empty commit
```
  8662e36d
15 Aug, 2020 1 commit
- [ci] fix flaky R CMD check test about library size (#3308) · 27c9aa88
  James Lamb authored Aug 15, 2020
  
  27c9aa88
09 Aug, 2020 1 commit

[ci][R-package][python] suppress warning about empty 'compute' director, fix... · 7be57d77

James Lamb authored Aug 10, 2020


[ci][R-package][python] suppress warning about empty 'compute' director, fix Rtools installation and temporary pin Sphinx version (#3277)

* [R-package] suppress warning about empty 'compute' directory

* get more logs from Rtools install

* empty commit

* add a timeout

* trying Invoke-WebRequest instead

* switch to CRAN link

* do not use Start-Process

* relative paths

* trying suppressmessageboxes

* switch strategy for running Rtools

* Update requirements_rtd.txt

* Update requirements_base.txt
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

7be57d77

07 Aug, 2020 1 commit

v3.0.0rc1 (#3071) · 63c82879

Guolin Ke authored Aug 07, 2020



* Update VERSION.txt

* Update VERSION.txt

* Update setup.py

* Update VERSION.txt

* Update setup.py

* Update VERSION.txt

* Update .appveyor.yml

* Update DESCRIPTION

* Apply suggestions from code review

* Apply suggestions from code review

* Update DESCRIPTION

* Apply suggestions from code review

* Create VERSION.txt

* Apply suggestions from code review

* Update .gitignore

* Update recreate-configure.sh

* Update test_r_package.sh

* Update build-cran-package.sh

* Delete VERSION.txt

* Create VERSION.txt

* Update main.yml

* Update test_r_package_windows.ps1

* Update configure

* Update test_r_package_windows.ps1

* Update test_r_package.sh

* Delete VERSION.txt

* Apply suggestions from code review

* update configure.ac
Co-authored-by: James Lamb <jaylamb20@gmail.com>

63c82879

06 Aug, 2020 2 commits
- [CI] fix R's installation (#3279) · 53c64bac
  Guolin Ke authored Aug 06, 2020
  
  53c64bac
- [R-package] move all examples to dontrun() to fix R CMD CHECK notes (#3270) · c454d5f8
  James Lamb authored Aug 05, 2020
```
* Move all examples to dontrun

* update docs

* fix nested dontrun

* remove :: in examples

* run_dontrun in pkgdown
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
```
  c454d5f8
29 Jul, 2020 1 commit

[R-package] make package installable with CRAN toolchain (fixes #2960) (#3188) · aa933eb4

James Lamb authored Jul 28, 2020



* [R-package] make package installable with CRAN toolchain (fixes #2960)

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* remove GPU stuff

* use wildcard to find objects to build

* use -lomp

* build configure before moving files

* using wildcard for objects

* Update .github/workflows/main.yml
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* add explicit objects back

* reduce allowed R CMD check NOTEs and catch stderr from build-cran-package on Windows

* fixing things

* pin autoconf version

* show diff

* add automake back

* run less checks

* command was in the wrong place

* fix autoconf version

* change strategy for handling configure

* fix Rbuildignore

* fix NOTEs

* fix notes about unrecognized files

* fixing extra files

* remove USE_R35

* add OpenMP check for Mac CRAN build

* run all checks

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* suggestions from code review

* undo indenting

* remove 03 from Makevars.win.in

* update language about OpenMP in configure script

* checking if configure.ac check works

* add autoconf back

* remove testing code in configure.ac

* more fixes for CI on configure script

* print git diff

* add VERSION.txt when checking configure

* fix relative paths

* remove git diff
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

aa933eb4

24 Jul, 2020 1 commit
- [ci] partly remove macOS temp CI fix (#3251) · 2c742d57
  Nikita Titov authored Jul 24, 2020
  
  2c742d57
20 Jul, 2020 2 commits
- [ci] remove unnecessary R_CHECK_FORCE_SUGGESTS in CI scripts (#3240) · f257636d
  James Lamb authored Jul 20, 2020
```
* [ci] remove unnecessary R_CHECK_FORCE_SUGGESTS in CI scripts

* remove unused test data
```
  f257636d
- [R-package] factor out {ggplot2} (#3224) · 9f52282d
  James Lamb authored Jul 20, 2020
```
* more changes

* factor out ggplot2

* update CI

* remove library()

* linting

* reduce NOTEs on Windows
```
  9f52282d
13 Jul, 2020 1 commit
- [ci] bump R 4 version (#3225) · 4b74139a
  Nikita Titov authored Jul 13, 2020
  
  4b74139a
05 Jul, 2020 1 commit

[ci][docs][R-package] catch builds that have not updated docs (#3205) · d8bc0fc7

James Lamb authored Jul 05, 2020



* [ci] [R-package] catch builds that have not updated docs

* drop reliance on .Renviron

* remove docs changes

* Update .ci/test_r_package.sh
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* make all R tasks start with r-package
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

d8bc0fc7