Even the newest CUDA runtime has a buggy cudaStreamSynchronize.

5557577c · Davis King · 3d5a3d7b · 5557577c
Commit 5557577c authored Jan 05, 2020 by Davis King
Show whitespace changes
Inline Side-by-side

Showing with 4 additions and 1 deletion

dlib/cuda/gpu_data.cpp dlib/cuda/gpu_data.cpp +4 -1

No files found.
--- a/dlib/cuda/gpu_data.cpp
+++ b/dlib/cuda/gpu_data.cpp
@@ -83,7 +83,10 @@ namespace dlib
    {
 #if !defined CUDA_VERSION
 #error CUDA_VERSION not defined
-#elif CUDA_VERSION >= 9020 && CUDA_VERSION <= 10010
+#elif CUDA_VERSION >= 9020 && CUDA_VERSION <= 11000
+        // We will stop using this alternative version with cuda V11, hopefully the bug in
+        // cudaStreamSynchronize is fixed by then.
+        //
        // This should be pretty much the same as cudaStreamSynchronize, which for some
        // reason makes training freeze in some cases.
        // (see https://github.com/davisking/dlib/issues/1513)