[R-package] Fallback to MinGW when VS build fails (#664)

* Change VS -> MinGW for CRAN

* Documentation to switch to MinGW by default

* Force cmake to run twice

* Try again dual build for Rtools

* Switch to cmake for building twice

* Try with Visual Studio as default, MinGW as fallback

* Try to remove VS appropriately when failing

* Attempt to get rid of build folder first

* Switch to unlink from rm rf

* Change lib_folder correctly when VS fails

* Add README updates from Microsoft/LightGBM@1c8355c

* Update default compiler doc and add GPU online install doc

* Better GPU doc

R-package/README.md

@@ -16,16 +16,20 @@ Installing [Rtools](https://cran.r-project.org/bin/windows/Rtools/) is mandatory
 
 [cmake](https://cmake.org/) must be version 3.8 or higher.
 
-The default compiler is Visual Studio (or [MS Build](https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017)) in Windows. You also can use [MinGW64](https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/) (x86_64-posix-seh) to compile by setting `use_mingw` to `TRUE` in `R-package/src/install.libs.R`. For MinGW users who wants to install online, please check the end of this document for installation using a helper package ([Laurae2/lgbdl](https://github.com/Laurae2/lgbdl/)).
+The default compiler is Visual Studio (or [MS Build](https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017)) in Windows, with an automatic fallback to Rtools or any [MinGW64](https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/) (x86_64-posix-seh) available (this means if you have only Rtools and cmake, it will compile fine).
 
-It is recommended to use *Visual Studio* for its better multi-threading efficency in Windows for many core systems. For very simple systems (dual core computers or worse), MinGW64 is recommended for maximum performance. If you do not know what to choose, it is recommended to use [Visual Studio](https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017).
+To force the usage of Rtools / MinGW, you can set `use_mingw` to `TRUE` in `R-package/src/install.libs.R`.
 
-#### Mac OS X Preparation
+For users who wants to install online with GPU or want to choose a specific compiler, please check the end of this document for installation using a helper package ([Laurae2/lgbdl](https://github.com/Laurae2/lgbdl/)).
+
+It is recommended to use *Visual Studio* for its better multi-threading efficency in Windows for many core systems. For very simple systems (dual core computers or worse), MinGW64 is recommended for maximum performance. If you do not know what to choose, it is recommended to use [Visual Studio](https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017), the default compiler.
 
-Gcc with OpenMP support must be installed first. Refer to [wiki](https://github.com/Microsoft/LightGBM/wiki/Installation-Guide#osx) for installing gcc with OpenMP support.
+#### Mac OS X Preparation
 
+gcc with OpenMP support must be installed first. Refer to [wiki](https://github.com/Microsoft/LightGBM/wiki/Installation-Guide#osx) for installing gcc with OpenMP support.
 
 ### Install
+
 Install LightGBM R-package with the following command:
 
 ```sh
@@ -53,7 +57,7 @@ You can also install directly from R using the repository with `devtools`:
 
 ```r
 library(devtools)
-options(devtools.install.args = "--no-multiarch")   
+options(devtools.install.args = "--no-multiarch") # if you have 64-bit R only, you can skip this
 install_github("Microsoft/LightGBM", subdir = "R-package")
 ```
 
@@ -93,13 +97,12 @@ In addition, if you are using a Visual Studio precompiled DLL, assuming you do n
 
 Once you have all this setup, you can use `lgb.dl` from `lgbdl` package to install LightGBM from repository.
 
-For instance, you can install the R package from LightGBM master commit of GitHub using the following from R:
+For instance, you can install the R package from LightGBM master commit of GitHub with Visual Studio using the following from R:
 
 ```r
 lgb.dl(commit = "master",
-       compiler = "gcc",
-       repo = "https://github.com/Microsoft/LightGBM",
-       cores = 4)
+       compiler = "vs",
+       repo = "https://github.com/Microsoft/LightGBM")
 ```
 
 You may also install using a precompiled dll/lib using the following from R:
@@ -110,6 +113,15 @@ lgb.dl(commit = "master",
        repo = "https://github.com/Microsoft/LightGBM")
 ```
 
+You may also install online using a LightGBM with proper GPU support using Visual Studio (as an example here) using the following from R:
+
+```r
+lgb.dl(commit = "master",
+       compiler = "vs", # Remove this for MinGW + GPU installation
+       repo = "https://github.com/Microsoft/LightGBM",
+       use_gpu = TRUE)
+```
+
 For more details about options, please check [Laurae2/lgbdl](https://github.com/Laurae2/lgbdl/) R-package.
 
 Examples

R-package/src/install.libs.R

@@ -42,21 +42,30 @@ if (!use_precompile) {
   setwd(build_dir)
   
   # Prepare installation steps
-  cmake_cmd <- "cmake "
-  build_cmd <- "make _lightgbm -j4"
+  cmake_base <- "cmake "
+  build_cmd <- "make _lightgbm -j"
   lib_folder <- file.path(R_PACKAGE_SOURCE, "src", fsep = "/")
   
   # Check if Windows installation (for gcc vs Visual Studio)
   if (WINDOWS) {
     if (use_mingw) {
-      cmake_cmd <- paste0(cmake_cmd, " -G \"MinGW Makefiles\" ")
-      build_cmd <- "mingw32-make.exe _lightgbm -j4"
+      cmake_cmd <- paste0(cmake_base, " -G \"MinGW Makefiles\" ")
+      build_cmd <- "mingw32-make.exe _lightgbm -j"
+      system(paste0(cmake_cmd, " ..")) # Must build twice for Windows due sh.exe in Rtools
+    } else {
+      cmake_cmd <- paste0(cmake_base, " -DCMAKE_GENERATOR_PLATFORM=x64 ")
+      tryVS <- system(paste0(cmake_cmd, " .."))
+      if (tryVS == 1) {
+        unlink("./*", recursive = TRUE) # Clean up build directory
+        cmake_cmd <- paste0(cmake_base, " -G \"MinGW Makefiles\" ") # Switch to MinGW on failure, try build once
+        system(paste0(cmake_cmd, " ..")) # Must build twice for Windows due sh.exe in Rtools
+        build_cmd <- "mingw32-make.exe _lightgbm -j"
       } else {
-      cmake_cmd <- paste0(cmake_cmd, " -DCMAKE_GENERATOR_PLATFORM=x64 ")
         build_cmd <- "cmake --build . --target _lightgbm  --config Release"
         lib_folder <- file.path(R_PACKAGE_SOURCE, "src/Release", fsep = "/")
       }
     }
+  }
   
   if (use_gpu) {
     cmake_cmd <- paste0(cmake_cmd, " -DUSE_GPU=ON ")