Skip to content

GitLab

Switch branch/tag
  1. 02 Jun, 2023 1 commit
  3. 31 May, 2023 2 commits
  5. 26 May, 2023 1 commit
  7. 25 May, 2023 3 commits
  9. 24 May, 2023 1 commit
  11. 23 May, 2023 4 commits
  13. 22 May, 2023 2 commits
  15. 18 May, 2023 1 commit
  17. 16 May, 2023 1 commit
  19. 11 May, 2023 2 commits
  21. 10 May, 2023 1 commit
  23. 09 May, 2023 3 commits
  25. 08 May, 2023 1 commit
  27. 05 May, 2023 1 commit
    • Ashwin Mathur's avatar
      Add `no_trainer` scripts to pre-train Vision Transformers (#23156) · fc6c8b0e
      Ashwin Mathur authored 
      

* Add run_mim_no_trainer.py draft from #20412

Add parse_args method and copy over other dependencies

Add Method call for sending telemetry

Initialize Accelerator

Make one log on every process

Set seed and Handle repository creation

Initialize dataset and Set validation split

Create Config

Adapt Config

Update Config

Create Feature Extractor

Create model

Set column names

Create transforms

Create mask generator

Create method to preprocess images

Shuffle datasets if needed and set transforms

Create Dataloaders

Add optimizer

Add learning rate scheduler

Prepare everything with our accelerator

Tie weights for TPU training

Recalculate training steps and training epochs

Set accelerator checkpointing steps

Initialize trackers and store configuration

Set total batch size

Fix typo: mlm -> mim

Log info at the start of training

Load in the weights and states from previous save

update the progress_bar if load from checkpoint

Define train loop

Add evaluation loop to training

Add to parse_args method

Push repo to hub

Save accelerator state

End training and save model and feature extractor

Remove unused imports

Fix trailing whitespace

* Update code based on comments, Rename feature_extractor to image_processor

* Fix linting

* Add argument for learning rate

* Add argument for setting number of training epochs

* Remove incorrect logger argument

* Convert max_train_steps to int for tqdm

---------
Co-authored-by: default avatarSaad Mahmud <shuvro.mahmud79@gmail.com>
      fc6c8b0e
  29. 03 May, 2023 1 commit
  31. 02 May, 2023 3 commits
  33. 25 Apr, 2023 1 commit
  35. 21 Apr, 2023 1 commit
  37. 20 Apr, 2023 1 commit
  39. 17 Apr, 2023 2 commits
  41. 14 Apr, 2023 1 commit
    • Sayak Paul's avatar
      [Examples] TPU-based training of a language model using TensorFlow (#21657) · 390e121f
      Sayak Paul authored 
      

* add: tokenizer training script for TF TPU LM training.

* add: script for preparing the TFRecord shards.

* add: sequence of execution to readme.

* remove limit from the tfrecord shard name.

* Add initial train_model.py

* Add basic training arguments and model init

* Get up to the point of writing the data collator

* Pushing progress so far!

* Complete first draft of model training code

* feat: grouping of texts efficiently.
Co-authored-by: default avatarMatt <rocketknight1@gmail.com>

* Add proper masking collator and get training loop working

* fix: things.

* Read sample counts from filenames

* Read sample counts from filenames

* Draft README

* Improve TPU warning

* Use distribute instead of distribute.experimental

* Apply suggestions from code review
Co-authored-by: default avatarMatt <Rocketknight1@users.noreply.github.com>

* Modularize loading and add MLM probability as arg

* minor refactoring to better use the cli args.

* readme fillup.

* include tpu and inference sections in the readme.

* table of contents.

* parallelize maps.

* polish readme.

* change script name to run_mlm.py

* address PR feedback (round I).

---------
Co-authored-by: default avatarMatt <rocketknight1@gmail.com>
Co-authored-by: default avatarMatt <Rocketknight1@users.noreply.github.com>
      390e121f
  43. 13 Apr, 2023 1 commit
  45. 11 Apr, 2023 1 commit
  47. 05 Apr, 2023 1 commit
  49. 04 Apr, 2023 1 commit
  51. 31 Mar, 2023 1 commit
  53. 29 Mar, 2023 1 commit