Commits · d413bd5f98795296bc31625fbef4c22975964510 · OpenDAS / Megatron-LM

02 Apr, 2021 1 commit
- More features added · d413bd5f
  Mostofa Patwary authored Apr 01, 2021
  
  d413bd5f
01 Apr, 2021 1 commit
- updated filter_ngrams.py · f559787d
  Mostofa Patwary authored Mar 31, 2021
  
  f559787d
30 Mar, 2021 1 commit
- fix one more issue · 0c01c2fe
  Mostofa Patwary authored Mar 29, 2021
  
  0c01c2fe
26 Mar, 2021 1 commit
- added parallelism for computing jaccard similaity · 43d307d5
  mpatwary authored Mar 25, 2021
  
  43d307d5
19 Mar, 2021 1 commit
- Added more feature in train data deduplication · 882683dc
  Mostofa Patwary authored Mar 19, 2021
  
  882683dc
08 Mar, 2021 1 commit
- Added code to cleanup and fix code · 47e7eb80
  Mostofa Patwary authored Mar 08, 2021
  
  47e7eb80
05 Mar, 2021 3 commits
- Fixed a bug · 9c2eb971
  Mostofa Patwary authored Mar 05, 2021
  
  9c2eb971
- Many more features added · b35e15f2
  Mostofa Patwary authored Mar 04, 2021
  
  b35e15f2
- Many more features added · c44f7622
  Mostofa Patwary authored Mar 04, 2021
  
  c44f7622
02 Mar, 2021 1 commit
- Dedup for other tasks added · 6013e23c
  Mostofa Patwary authored Mar 02, 2021
  
  6013e23c
26 Feb, 2021 3 commits
- Updated readme · b08b5edc
  Mostofa Patwary authored Feb 26, 2021
  
  b08b5edc
- added threads · a29890c3
  Mostofa Patwary authored Feb 26, 2021
  
  a29890c3
- Other tasks dedup added · 90c9e3f2
  Mostofa Patwary authored Feb 25, 2021
  
  90c9e3f2
22 Feb, 2021 1 commit
- addressed the comments · ebc4a408
  Mostofa Patwary authored Feb 22, 2021
  
  ebc4a408
19 Feb, 2021 4 commits
- updated the readme · 5a867963
  Mostofa Patwary authored Feb 19, 2021
  
  5a867963
- now multiple pickle files can be added · 4b821fb5
  Mostofa Patwary authored Feb 19, 2021
  
  4b821fb5
- Added an example · f6ddeca0
  Mostofa Patwary authored Feb 18, 2021
  
  f6ddeca0
- added storing and loading of pickle file of hash · 2f6e46bd
  Mostofa Patwary authored Feb 18, 2021
  
  2f6e46bd
18 Feb, 2021 1 commit
- added readme · a449d312
  Mostofa Patwary authored Feb 18, 2021
  
  a449d312
17 Feb, 2021 2 commits
- filtering n-grams · 59372322
  Mostofa Patwary authored Feb 17, 2021
  
  59372322
- Fixed a minor bug · 0a158128
  Mostofa Patwary authored Feb 17, 2021
  
  0a158128
16 Apr, 2020 1 commit
- changed licence 2019 to 2020 · 463d1257
  Mohammad authored Apr 16, 2020
  
  463d1257
14 Apr, 2020 1 commit
- Move openwebtext scripts to tools directory. · 0dc10d09
  Jared Casper authored Apr 14, 2020
  
  0dc10d09