only select pth files with prefix "model" as model checkpoint file

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/605

D2GO workflow async validation monitor the model checkpoint files *.pth in **e2e_train** folder (such as **model_0004999.pth**, **model_final.pth**) and launch async val operator as needed.
All model files actually have prefix **"model"**.  In some cases, there are non-model-checkpoint files also with pth file extension.
To exclude them, add a filtering to check if the file prefix is "model".

Reviewed By: ayushidalmia

Differential Revision: D48021972

fbshipit-source-id: 54d9c14117192809ea76d812ebd4240b44166637

d2go/utils/validation_monitor.py

@@ -55,6 +55,13 @@ def fetch_checkpoints_till_final(checkpoint_dir):
         checkpoint_paths = DetectionCheckpointer(
             None, save_dir=checkpoint_dir
         ).get_all_checkpoint_files()
+
+        checkpoint_paths = [
+            cpt_path
+            for cpt_path in checkpoint_paths
+            if os.path.basename(cpt_path).startswith("model")
+        ]
+
         checkpoint_paths.extend(_get_lightning_checkpoints(checkpoint_dir))
 
         final_model_path = None