• Ayush Chaurasia's avatar
    Add W&B backend for hyperparameter sweep (#14582) · c74f3d4c
    Ayush Chaurasia authored
    # Add support for W&B hyperparameter sweep
    This PR:
    * allows using wandb for running hyperparameter search.
    * The runs are visualized on W&B sweeps dashboard
    * This supports runnning sweeps on parallel devices, all reporting to the same central dashboard.
    
    ### Usage
    **To run new a hyperparameter search:**
    ```
    trainer.hyperparameter_search(
        backend="wandb", 
        project="transformers_sweep", # name of the project
        n_trials=5,
        metric="eval/loss", # metric to be optimized, default 'eval/loss'. A warning is raised if the passed metric is not found
    )
    ```
    This outputs a sweep id. Eg. `my_project/sweep_id`
    
    **To run sweeps on parallel devices:**
    Just pass sweep id which you want to run parallel
    ```
    trainer.hyperparameter_search(
        backend="wandb", 
        sweep_id = "my_project/sweep_id"
    )
    ```
    c74f3d4c
self-scheduled.yml 17.6 KB