Skip to content

GitLab

"cacheflow/vscode:/vscode.git/clone" did not exist on "8274ca23ac9f2ea0ccf758d1883794643aecb2e0"
Switch branch/tag
Find file Normal viewHistoryPermalink
tensor_parallel.rst 1.49 KB
Newer Older
xingjinliang's avatar
升级megatron_v0.10  
xingjinliang committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67 
tensor\_parallel package
========================

This package contains an implementation for tensor parallelism in transformer
models (see `Megatron-LM: Training Multi-Billion Parameter Language Models
Using Model Parallelism <https://arxiv.org/abs/1909.08053>`_ and `Reducing
Activation Recomputation in Large Transformer Models <https://arxiv.org/abs/2205.05198>`_
for details).

Submodules
----------

tensor\_parallel.cross\_entropy module
--------------------------------------

.. automodule:: core.tensor_parallel.cross_entropy
   :members:
   :undoc-members:
   :show-inheritance:

tensor\_parallel.data module
----------------------------

.. automodule:: core.tensor_parallel.data
   :members:
   :undoc-members:
   :show-inheritance:

tensor\_parallel.layers module
------------------------------

.. automodule:: core.tensor_parallel.layers
   :members:
   :undoc-members:
   :show-inheritance:

tensor\_parallel.mappings module
--------------------------------

.. automodule:: core.tensor_parallel.mappings
   :members:
   :undoc-members:
   :show-inheritance:

tensor\_parallel.random module
------------------------------

.. automodule:: core.tensor_parallel.random
   :members:
   :undoc-members:
   :show-inheritance:

tensor\_parallel.utils module
-----------------------------

.. automodule:: core.tensor_parallel.utils
   :members:
   :undoc-members:
   :show-inheritance:

Module contents
---------------

.. automodule:: core.tensor_parallel
   :members:
   :undoc-members:
   :show-inheritance: