GitLab

[R-package] [c++] add tighter multithreading control, avoid global OpenMP side effects (fixes #4705, fixes #5102) (#6226)

* add predcontrib for sparse inputs

* register newly-added function

* comments

* correct wrong types in test

* forcibly take transpose function from Matrix

* keep row names, test comparison to dense inputs

* workaround for passing test while PR for row names is not merged

* Update R-package/R/lgb.Predictor.R
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Update R-package/R/lgb.Predictor.R
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Update R-package/R/lgb.Predictor.R
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* proper handling of integer overflow

* add test for CSR contrib row names

* add more tests for predict(<sparse>, predcontrib=TRUE)

* make linter happy

* linter

* linter

* check error messages for bad input shapes

* fix regex

* hard-coded number of columns in regex for tests
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* expose FeatureNumBin in C api

* parametrize min_data_in_bin and add test with max_bin_by_feature

* include feature_num_bin in R package

* add suggestion from review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* update error message and lint

* lint

* add call method

* minor improvements in tests

* add suggestions from review

* lint

* rename argument to feature in python and r packages
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

[R-package] enable saving Booster with saveRDS() and loading it with readRDS() (fixes #4296) (#4685)

* idiomatic serialization

* linter

* linter, namespace

* comments, linter, fix failing test

* standardize error messages for null handles

* auto-restore handle in more functions

* linter

* missing declaration

* correct wrong signature

* fix docs

* Update R-package/R/lgb.train.R
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Update R-package/R/lgb.drop_serialized.R
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Update R-package/R/lgb.restore_handle.R
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Update R-package/R/lgb.restore_handle.R
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Update R-package/R/lgb.make_serializable.R
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* move 'restore_handle' from feature importance to dump method

* missing header

* move arguments order, update docs

* linter

* avoid leaving files in working directory

* add test for save_model=NULL

* missing comma

* Update R-package/R/lgb.restore_handle.R
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update R-package/src/lightgbm_R.cpp
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* change name of error function

* update comment

* restore old serialization functions but set as deprecated

* Update R-package/R/readRDS.lgb.Booster.R
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update R-package/R/saveRDS.lgb.Booster.R
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* update docs

* Update R-package/R/readRDS.lgb.Booster.R
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Update R-package/R/saveRDS.lgb.Booster.R
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Update R-package/tests/testthat/test_basic.R
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Update R-package/R/readRDS.lgb.Booster.R
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* comments

* fix variable name

* restore serialization test for linear models

* Update R-package/R/lightgbm.R
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* update docs

* fix issues with null terminator
Co-authored-by: James Lamb <jaylamb20@gmail.com>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* add C API function that returns all param names with aliases

* add C API function that returns all param names with aliases

* add R code

* test R code

* remove debug CI

* fix R lint

* refactor

* run CI

* fix R

* fix

* revert CI checks

* revert changes in docs

* Try to make function `const`
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* add `const` in cpp file

* address review comments and sync with `master`
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* add print and summary S3 method

* correct wrong signature

* attempt at bypassing linter

* Update R-package/R/lgb.Booster.R
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Update R-package/src/lightgbm_R.h
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* Update include/LightGBM/c_api.h
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* add more tests

* linter

* don't pluralize single tree

* remove duplicated function

* update changed function name

* missing declaration

* Update lightgbm_R.h

* Update R-package/tests/testthat/test_lgb.Booster.R
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* accommodate custom objectives in print

* linter

* linter
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* [R-package] fix segfaults caused by missing Booster and Dataset handles (fixes #4208)

* fix test errors

* fixes for cpplint

* Update R-package/tests/testthat/test_dataset.R
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* fix tests

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* move asserts inside try-catch
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* fix R memory leaks

* attempt at solving linter complaints

* fix compilation on windows

* move R_API_BEGIN to correct place

* make sure exception objects reach out of scope

* better way to solve rchk complaints

* remove goto statement

* [R-package] fix protection stack imbalance and unprotected objects issues

* [R-package] fix minor linting issues

* [ci][R-package] change timeout-minutes in valgrind test

* [R-package] remove extra space
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* [R-package] remove counter for number of protected objects

* Update .github/workflows/r_valgrind.yml
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* [R-package] move more finalizer logic intoo C++ side

* add C finalizers

* use gc()

* put skip() back

* Update R-package/tests/testthat/test_lgb.Booster.R
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* Update lightgbm_R.cpp

* fix typos

* started converting handles

* more changes

* sort of working for Dataset

* yay all the tests are passing for Dataset handle changes

* working for other handle types

* remove debugging logging

* remove unnecessary spaces

* fix null logic

* more NULL

* updates

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* consolidate steps
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* [R-package] move creation of character vectors in some methods to C++ side

* convert LGBM_BoosterGetEvalNames_R

* convert LGBM_BoosterDumpModel_R and LGBM_BoosterSaveModelToString_R

* remove debugging code

* update docs

* remove comment

* add handling for larger model strings

* handle large strings in feature and eval names

* got long feature names working

* more fixes

* linting

* resize

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* stricter test
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* converted LGBM_BoosterSaveModel_R

* switch all other non-params cases

* handle params

* real pointer for matrix

* remove R_REAL_PTR

* remove R_INT_PTR

* add test

* [R-package] replace R_AS_INT with R built-in

* update header

* more changes

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* [R-package] move some functions over to SEXP objects

* clarify comment

* remove paste0()

* fix registration table

* fix cmake builds

* [R-package] prevent symbol lookup conflicts

* add unit tests

* only run test on Windows

* move to .Call() calls

* fix references

* testing registration for CMake builds

* revert NAMESPACE changes

* revert testing changes

* [R-package] raise errors from C++ side

* working but a lot of warnings

* more changes

* simplify

* cleanup

* linting

* fix valgrind issues

* revert unnecessary change

* fix int64 write error

* attempt

* [WIP] [ci] [R-package] Add CI job that runs valgrind tests

* update all-successful

* install

* executable

* fix redirect stuff

* Apply suggestions from code review
Co-authored-by: Guolin Ke <guolin.ke@outlook.com>

* more flags

* add mc to msvc proj

* fix memory leak in mc

* Update monotone_constraints.hpp

* Update r_package.yml

* remove R_INT64_PTR

* disable openmp

* Update gbdt_model_text.cpp

* Update gbdt_model_text.cpp

* Apply suggestions from code review

* try to free vector

* free more memories.

* Update src/boosting/gbdt_model_text.cpp

* fix using

* try the UNPROTECT(1);

* fix a const pointer

* fix Common

* reduce UNPROTECT

* remove UNPROTECT(1);

* fix null handle

* fix predictor

* use NULL after free

* fix a leaking in test

* try more fixes

* test the effect of tests

* throw exception in Fatal

* add test back

* Apply suggestions from code review

* commet some tests

* Apply suggestions from code review

* Apply suggestions from code review

* trying to comment out tests

* Update openmp_wrapper.h

* Apply suggestions from code review

* Update configure

* Update configure.ac

* trying to uncomment

* more comments

* more uncommenting

* more uncommenting

* fix comment

* more uncommenting

* uncomment fully-commented out stuff

* try uncommenting more dataset tests

* uncommenting more tests

* ok getting closer

* more uncommenting

* free dataset

* skipping a test, more uncommenting

* more skipping

* re-enable OpenMP

* allow on OpenMP thing

* move valgrind to comment-only job

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* changes from code review

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* linting

* issue comments too

* remove issue_comment
Co-authored-by: Guolin Ke <guolin.ke@outlook.com>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* [python] add start_iteration to python predict interface (#3058)

* Apply suggestions from code review

* Update lightgbm_R.h

* Apply suggestions from code review

* Apply suggestions from code review

* fix R interface

* update R documentation
Co-authored-by: Guolin Ke <guolin.ke@outlook.com>

* feature importance type in saved model file

* fix nullptr

* fixed formatting

* fix python/R

* Update src/c_api.cpp

* Apply suggestions from code review
Co-authored-by: James Lamb <jaylamb20@gmail.com>

* fix c_api test

* fix swig

* minor docs improvements and added defines for importance types
Co-authored-by: StrikerRUS <nekit94-12@hotmail.com>
Co-authored-by: James Lamb <jaylamb20@gmail.com>

This reverts commit 656d2676.

* [R-package] move R source files into R-package

* fix linting warning

* stuff

* [swig] Fix SWIG methods that return char** with StringArray.

+ [new] Add StringArray class to manage and manipulate arrays of fixed-length strings:

  This class is now used to wrap any char** parameters, manage memory and
  manipulate the strings.

  Such class is defined at swig/StringArray.hpp and wrapped in StringArray.i.

+ [API+fix] Wrap LGBM_BoosterGetFeatureNames it resulted in segfault before:

  Added wrapper LGBM_BoosterGetFeatureNamesSWIG(BoosterHandle) that
  only receives the booster handle and figures how much memory to allocate
  for strings and returns a StringArray which can be easily converted to String[].

+ [API+safety] For consistency, LGBM_BoosterGetEvalNamesSWIG was wrapped as well:

  * Refactor to detect any kind of errors and removed all the parameters
    besides the BoosterHandle (much simpler API to use in Java).
  * No assumptions are made about the required string space necessary (128 before).
  * The amount of required string memory is computed internally

+ [safety] No possibility of undefined behaviour

  The two methods wrapped above now compute the necessary string storage space
  prior to allocation, as the low-level C API calls would crash the process
  irreversibly if they write more memory than which is passed to them.

* Changes to C API and wrappers support char**

To support the latest SWIG changes that enable proper char**
return support that is safe, the C API was changed.

The respecive wrappers in R and Python were changed too.

* Cleanup indentation in new lightgbm_R.cpp code

* Adress review code-style comments.

* Update swig/StringArray.hpp
Co-Authored-By: Nikita Titov <nekit94-08@mail.ru>

* Update python-package/lightgbm/basic.py
Co-Authored-By: Nikita Titov <nekit94-08@mail.ru>

* Update src/lightgbm_R.cpp
Co-Authored-By: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: alberto.ferreira <alberto.ferreira@feedzai.com>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>