[ci] introduce CI jobs that mimic CRAN gcc-ASAN and clang-ASAN tests (fixes #4674) (#4678)

* add jobs mimicking CRAN gcc-ASAN and clang-ASAN

* comment out CI

* fix redirection

* remove unnecessary echo

* Revert "comment out CI"

This reverts commit 899fbb4574b99a8125b28761425e3e821bfdfef1.

* remove redundant env variables and update README

* remove inaccurate comment

* change test title

* Revert "Fix ASAN issues with `std::function` usage (#4673)"

This reverts commit 13ed38ca

.

* Revert "Revert "Fix ASAN issues with `std::function` usage (#4673)""

This reverts commit 24c275ba84f1f182275f74ff4ad2e510bb18f4bd.

* revert unnecessary change in config order

* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

.github/workflows/r_package.yml

@@ -173,16 +173,19 @@ jobs:
           $env:TASK = "${{ matrix.task }}"
           & "$env:GITHUB_WORKSPACE/.ci/test_windows.ps1"
   test-r-sanitizers:
-    name: r-package (ubuntu-latest, R-devel, GCC ASAN/UBSAN)
+    name: r-sanitizers (ubuntu-latest, R-devel, ${{ matrix.compiler }} ASAN/UBSAN)
     timeout-minutes: 60
     runs-on: ubuntu-latest
-    container: rhub/rocker-gcc-san
+    container: wch1/r-debug
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - r_customization: san
+            compiler: gcc
+          - r_customization: csan
+            compiler: clang
     steps:
-      - name: Install Git before checkout
-        shell: bash
-        run: |
-          apt-get update --allow-releaseinfo-change
-          apt-get install --no-install-recommends -y git
       - name: Checkout repository
         uses: actions/checkout@v2.3.4
         with:
@@ -191,16 +194,17 @@ jobs:
       - name: Install packages
         shell: bash
         run: |
-          Rscript -e "install.packages(c('R6', 'data.table', 'jsonlite', 'Matrix', 'testthat'), repos = 'https://cran.r-project.org', Ncpus = parallel::detectCores())"
+          RDscript${{ matrix.r_customization }} -e "install.packages(c('R6', 'data.table', 'jsonlite', 'Matrix', 'testthat'), repos = 'https://cran.r-project.org', Ncpus = parallel::detectCores())"
           sh build-cran-package.sh
-          Rdevel CMD INSTALL lightgbm_*.tar.gz || exit -1
+          RD${{ matrix.r_customization }} CMD INSTALL lightgbm_*.tar.gz || exit -1
       - name: Run tests with sanitizers
         shell: bash
         run: |
           cd R-package/tests
-          Rscriptdevel testthat.R 2>&1 > ubsan-tests.log
-          cat ubsan-tests.log
-          exit $(cat ubsan-tests.log | grep --count "runtime error")
+          exit_code=0
+          RDscript${{ matrix.r_customization }} testthat.R >> tests.log 2>&1 || exit_code=1
+          cat ./tests.log
+          exit ${exit_code}
   test-r-debian-clang:
     name: r-package (debian, R-devel, clang)
     timeout-minutes: 60

R-package/README.md

@@ -361,26 +361,51 @@ Alternatively, GitHub Actions can run code above for you. On a pull request, cre
 
 **NOTE:** Please do this only once you see that other R tests on a pull request are passing. R Hub is a free resource with limited capacity, and we want to be respectful community members.
 
-#### UBSAN
+#### <a id="UBSAN"></a>ASAN and UBSAN
 
-All packages uploaded to CRAN must pass a build using `gcc` instrumented with two sanitizers: the Address Sanitizer (ASAN) and the Undefined Behavior Sanitizer (UBSAN). For more background, see [this blog post](http://dirk.eddelbuettel.com/code/sanitizers.html).
+All packages uploaded to CRAN must pass builds using `gcc` and `clang`, instrumented with two sanitizers: the Address Sanitizer (ASAN) and the Undefined Behavior Sanitizer (UBSAN).
+
+For more background, see
+
+* [this blog post](https://dirk.eddelbuettel.com/code/sanitizers.html)
+* [top-level CRAN documentation on these checks](https://cran.r-project.org/web/checks/check_issue_kinds.html)
+* [CRAN's configuration of these checks](https://www.stats.ox.ac.uk/pub/bdr/memtests/README.txt)
 
 You can replicate these checks locally using Docker.
+For more information on the image used for testing, see https://github.com/wch/r-debug.
 
-```shell
-docker run \
-    -v $(pwd):/opt/LightGBM \
-    -w /opt/LightGBM \
-    -it rhub/rocker-gcc-san \
-    /bin/bash
+In the code below, environment variable `R_CUSTOMIZATION` should be set to one of two values.
 
-Rscript -e "install.packages(c('R6', 'data.table', 'jsonlite', 'Matrix', 'testthat'), repos = 'https://cran.rstudio.com', Ncpus = parallel::detectCores())"
+* `"san"` = replicates CRAN's `gcc-ASAN` and `gcc-UBSAN` checks
+* `"csan"` = replicates CRAN's `clang-ASAN` and `clang-UBSAN` checks
 
+```shell
+docker run \
+  --rm \
+  -it \
+  -v $(pwd):/opt/LightGBM \
+  -w /opt/LightGBM \
+  --env R_CUSTOMIZATION=san \
+  wch1/r-debug:latest \
+  /bin/bash
+
+# install dependencies
+RDscript${R_CUSTOMIZATION} \
+  -e "install.packages(c('R6', 'data.table', 'jsonlite', 'Matrix', 'testthat'), repos = 'https://cran.r-project.org', Ncpus = parallel::detectCores())"
+
+# install lightgbm
 sh build-cran-package.sh
+RD${R_CUSTOMIZATION} \
+  CMD INSTALL lightgbm_*.tar.gz
 
-Rdevel CMD install lightgbm_*.tar.gz
+# run tests
 cd R-package/tests
-Rscriptdevel testthat.R
+rm -f ./tests.log
+RDscript${R_CUSTOMIZATION} testthat.R >> tests.log 2>&1
+
+# check that tests passed
+echo "test exit code: $?"
+tail -300 ./tests.log
 ```
 
 #### Valgrind