Commits · 43d307d5cf2f313e1a53122eb2a448d579111d21 · OpenDAS / Megatron-LM

26 Mar, 2021 1 commit
- added parallelism for computing jaccard similaity · 43d307d5
  mpatwary authored Mar 25, 2021
  
  43d307d5
19 Mar, 2021 1 commit
- Added more feature in train data deduplication · 882683dc
  Mostofa Patwary authored Mar 19, 2021
  
  882683dc
08 Mar, 2021 1 commit
- Added code to cleanup and fix code · 47e7eb80
  Mostofa Patwary authored Mar 08, 2021
  
  47e7eb80
05 Mar, 2021 3 commits
- Fixed a bug · 9c2eb971
  Mostofa Patwary authored Mar 05, 2021
  
  9c2eb971
- Many more features added · b35e15f2
  Mostofa Patwary authored Mar 04, 2021
  
  b35e15f2
- Many more features added · c44f7622
  Mostofa Patwary authored Mar 04, 2021
  
  c44f7622
02 Mar, 2021 1 commit
- Dedup for other tasks added · 6013e23c
  Mostofa Patwary authored Mar 02, 2021
  
  6013e23c
26 Feb, 2021 3 commits
- Updated readme · b08b5edc
  Mostofa Patwary authored Feb 26, 2021
  
  b08b5edc
- added threads · a29890c3
  Mostofa Patwary authored Feb 26, 2021
  
  a29890c3
- Other tasks dedup added · 90c9e3f2
  Mostofa Patwary authored Feb 25, 2021
  
  90c9e3f2
22 Feb, 2021 1 commit
- addressed the comments · ebc4a408
  Mostofa Patwary authored Feb 22, 2021
  
  ebc4a408
19 Feb, 2021 4 commits
- updated the readme · 5a867963
  Mostofa Patwary authored Feb 19, 2021
  
  5a867963
- now multiple pickle files can be added · 4b821fb5
  Mostofa Patwary authored Feb 19, 2021
  
  4b821fb5
- Added an example · f6ddeca0
  Mostofa Patwary authored Feb 18, 2021
  
  f6ddeca0
- added storing and loading of pickle file of hash · 2f6e46bd
  Mostofa Patwary authored Feb 18, 2021
  
  2f6e46bd
18 Feb, 2021 1 commit
- added readme · a449d312
  Mostofa Patwary authored Feb 18, 2021
  
  a449d312
17 Feb, 2021 2 commits
- filtering n-grams · 59372322
  Mostofa Patwary authored Feb 17, 2021
  
  59372322
- Fixed a minor bug · 0a158128
  Mostofa Patwary authored Feb 17, 2021
  
  0a158128
16 Apr, 2020 1 commit
- changed licence 2019 to 2020 · 463d1257
  Mohammad authored Apr 16, 2020
  
  463d1257
14 Apr, 2020 1 commit
- Move openwebtext scripts to tools directory. · 0dc10d09
  Jared Casper authored Apr 14, 2020
  
  0dc10d09