1. 08 Jul, 2023 1 commit
    • Volodymyr Kyrylov's avatar
      rotary: update cos/sin cache when switching from inference mode · 70ab266a
      Volodymyr Kyrylov authored
      This resolves RuntimeErrors after running evaluation in inference mode:
      
      ```
        File "/home/proger/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
          return forward_call(*args, **kwargs)
        File "/home/proger/.local/lib/python3.10/site-packages/flash_attn/modules/mha.py", line 492, in forward
          qkv = self.rotary_emb(qkv)
        File "/home/proger/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
          return forward_call(*args, **kwargs)
        File "/home/proger/.local/lib/python3.10/site-packages/flash_attn/layers/rotary.py", line 229, in forward
          return apply_rotary_emb_qkv_(
        File "/home/proger/.local/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
          return super().apply(*args, **kwargs)  # type: ignore[misc]
      RuntimeError: Inference tensors cannot be saved for backward. To work around you can make a clone to get a normal tensor and use it in autograd.
      ```
      70ab266a
  2. 06 Jul, 2023 1 commit
  3. 04 Jul, 2023 1 commit
  4. 03 Jul, 2023 4 commits
  5. 02 Jul, 2023 1 commit
  6. 02 Jun, 2023 2 commits
  7. 30 May, 2023 3 commits
  8. 27 May, 2023 2 commits
  9. 25 May, 2023 5 commits
  10. 19 May, 2023 3 commits
  11. 17 May, 2023 1 commit
  12. 12 May, 2023 2 commits
  13. 06 May, 2023 3 commits
  14. 05 May, 2023 1 commit
  15. 27 Apr, 2023 1 commit
  16. 26 Apr, 2023 1 commit
  17. 21 Apr, 2023 6 commits
  18. 19 Apr, 2023 1 commit
  19. 18 Apr, 2023 1 commit