Ignore TempDir clean up error (#2379)

Summary:
On CircleCI, Windows unittests are failing for Python 3.7 with
`PermissionError` at the end of test when it cleans up temporary
directory.

According to the discussion https://github.com/python/cpython/issues/74168,
this is caused by a known issue with `shutil.rmtree`.

In the above thread it is advised to simply ignore the error as it
is not guaranteed that temp directories are cleaned up.

This commit follows the same path and simply ignore the error
so that our CI gets back to green.

Pull Request resolved: https://github.com/pytorch/audio/pull/2379

Reviewed By: carolineechen

Differential Revision: D36305595

Pulled By: mthrok

fbshipit-source-id: d9049c2ee3447712119786311f639a1f9f8911c5

test/torchaudio_unittest/common_utils/case_utils.py

@@ -36,8 +36,20 @@ class TempDirMixin:
     def tearDownClass(cls):
         super().tearDownClass()
         if cls.temp_dir_ is not None:
-            cls.temp_dir_.cleanup()
-            cls.temp_dir_ = None
+            try:
+                cls.temp_dir_.cleanup()
+                cls.temp_dir_ = None
+            except PermissionError:
+                # On Windows there is a know issue with `shutil.rmtree`,
+                # which fails intermittenly.
+                #
+                # https://github.com/python/cpython/issues/74168
+                #
+                # We observed this on CircleCI, where Windows job raises
+                # PermissionError.
+                #
+                # Following the above thread, we ignore it.
+                pass
 
     def get_temp_path(self, *paths):
         temp_dir = os.path.join(self.get_base_temp_dir(), self.id())