Skip to content

GitLab

Commit d444a97a authored by yangzhong's avatar yangzhong
Browse files

首次上传

parents
Pipeline #3020 canceled with stages
Hide whitespace changes
Inline Side-by-side
Showing with 3641 additions and 0 deletions
CHANGELOG.md 0 → 100644
View file @ d444a97a
# Changelog
## NVIDIA Megatron Core 0.9.0
- Uneven pipeline parallelism
- Enable pipeline parallelism where first and last ranks have fewer transformer layers than the intermediate ranks
- Per layer CUDAGraph support for GPT training with Transformer Engine modules
- Enable different TP sizes for the vision encoder
- Enable pipeline parallelism for T5 & Llava models
- Support multi-tile multi-image input in Llava models
- MoE
- FP8 support
- Runtime upcycling support
- Dispatcher implementation optimizations
- Shared expert support with overlapping optimizations
- Qwen Model support
- Known Issues
- When using sequence parallel, during the transformer block forward pass, dropout is not using the appropriate rng context.
## NVIDIA Megatron Core 0.8.0
- Multimodal
- Added initial support for training vision language models using the LLaVA architecture
- Added initial support for inference with multimodal inputs
- End-to-end multimodal example from data collection to training to evaluation is provided in examples/multimodal
- MoE
- Context Parallel support.
- Distributed checkpoint support for grouped GEMM.
- Mamba
## NVIDIA Megatron Core 0.7.0
- MoE
- Token drop support
- Several efficiency optimizations
- Improved model parallelism
- Memory optimizations
- Distributed checkpointing
- Enabled for Retro
- Asynchronous checkpoint saving
- Several minor bug fixes, speed improvements, and memory optimizations
## NVIDIA Megatron Core 0.6.0
- MoE (Mixture of Experts)
- Performance optimization
- Communication optimization for multi GPU and Single GPU
- 23% improvement (323 TFLOPS/GPU) over MCore 0.5.0 on Mixtral with Hopper BF16
- GroupedMLP enhancement for Hopper
- DP Overlapping. Support overlapping computation with gradient reduction and parameter gathering.
- All-to-All based Token Dispatcher
- Layer-wise logging for load balancing loss.
- Improved expert parallel support including distributed optimizer.
- Distributed optimizer
- RETRO
- Data processing
- BERT
- Distributed checkpointing
- Dist checkpointing
- PyTorch native distributed backend
- Improved saving/loading speed
- TensorRT-LLM Export
- Integration with TensorRT Model Optimizer Post-training quantization (PTQ)
- Text generation driver to perform PTQ in Megatron-LM
- Llama2 and Nemotron3-8b examples to use TensorRT-LLM unified build API to build engine after training.
- Several minor enhancements, bug fixes, and documentation updates
## NVIDIA Megatron Core 0.5.0
### Key Features and Enhancements
Megatron core documentation is now [live!](https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/index.html#quick-start)
### Model Features
- MoE (Mixture of Experts)
- Support for Z-loss, Load balancing and Sinkhorn
- Layer and communications refactor
- Richer parallelism mappings and EP can be combined with other model parallel techniques for larger MoE variants, e.g. EP + TP + DP + SP + PP
- Token dropless architecture with Top-K routing
- Performance optimization with with GroupedGEMM when number of local experts is > 1
- Distributed checkpointing
- Interleaved rotary embedding
### Datasets
- Masked WordPiece datasets for BERT and T5
- Raw and mock datasets
### Parallelism
### Performance
- Activation offloading to CPU
- Rope and Swiglu fusion
- Sliding window attention (via Transformer Engine)
### General Improvements
- Timers
## NVIDIA Megatron Core 0.4.0
### Key Features and Enhancements
#### Models
- BERT
- RETRO
- T5
#### Parallelism
- Mixture of Experts support for GPT
- Model parallel efficient Distributed Data Parallel (DDP)
- Context Parallel (2D Tensor Parallel) support
#### Datasets
- GPT Dataset
- Blended Dataset
CODEOWNERS 0 → 100644
View file @ d444a97a
[Core-ADLR] @mcore-reviewers/core-adlr
megatron/core/
[Core-NeMo] @mcore-reviewers/core-nemo
megatron/core/
^[Core-MLPerf] @mcore-reviewers/mlperf
megatron/core/
[MoE-ADLR] @mcore-reviewers/moe-adlr
megatron/core/transformer/moe/
[MoE-Moe] @mcore-reviewers/moe-moe
megatron/core/transformer/moe/
[Datasets] @mcore-reviewers/datasets
megatron/core/datasets/
[BERT] @mcore-reviewers/bert
megatron/core/models/bert/
[GPT] @mcore-reviewers/gpt
megatron/core/models/gpt/
[Retro] @mcore-reviewers/retro
megatron/core/models/retro/
[Distributed Checkpointing] @mcore-reviewers/dist-checkpointing
megatron/core/dist_checkpointing/
[Distributed Optimizer] @mcore-reviewers/dist-optimizer
megatron/core/optimizer/distrib_optimizer/
[Inference] @mcore-reviewers/inference
megatron/core/inference/
^[Quantization and Inference (QAT)] @mcore-reviewers/quantization-and-inference
megatron/core/inference/
; [Context Parallelism] @mcore-reviewers/context-parallelism
;
[CI] @mcore-reviewers/ci
.gitlab/
.github/
.gitlab-ci.yml
Dockerfile.ci.lts
Dockerfile.ci.dev
tests/
CONTRIBUTING.md 0 → 100644
View file @ d444a97a
# Contributing to Megatron-LM
This document outlines the processes and policies for issues and pull requests by non-NVIDIA contributors to the Megatron-LM github repository.
Everyone is welcome to contribute to the project but development of Megatron-LM continues internally at NVIDIA. When contributing it important to ensure that changes are in line with the project direction. Small changes to fix bugs are welcomed and appreciated. If proposing large architectural changes or changes for stylistic reasons open an issue first so we can discuss it.
PRs will first be pulled into NVIDIA's internal Megatron-LM repo and then pushed back out to the open github repo with proper credit given to the committers.
## Issue policy
Please do file any bugs you find, keeping the following in mind:
- If filing a bug, i.e. you have found something that doesn't work as expected, use the BUG template.
- If you've found a regression in speed or accuracy use the REGRESSION template.
- If you are requesting a new feature or modification of an existing feature use the ENHANCEMENT template.
- If opening an issue to ask a question no template is needed but please make your question as clear and concise as possible.
- One issue per bug. Putting multiple things in the same issue makes both discussion and completion unnecessarily complicated.
- Your bug is mostly likely to get attention from the development team quickly if we can easily reproduce it.
- Use proper spelling, grammar, and punctuation.
- Write in an authoritative and technical tone.
## Code submission policy
Here are some dos & don'ts to try and stick to:
### Do:
- Format new code in a style that is consistent with the file being changed. Megatron-LM doesn't (yet) have a style guide or enforced formatting.
- Split your changes into separate, atomic commits i.e. A commit per feature or fix.
- Make sure your commits are rebased on the master branch.
- Write the commit message subject line in the imperative mood ("Change the default argument for X", not "Changed the default argument for X").
- Write your commit messages in proper English, with care and punctuation.
- Check the spelling of your code, comments and commit messages.
### Don't:
- Submit code that's incompatible with the project licence.
- Touch anything outside the stated scope of the PR. This includes formatting changes to code not relevant to the PR.
- Iterate excessively on your design across multiple commits.
- Include commented-out code.
- Attempt large architectural changes without first opening an issue to discuss.
## Issue and Pull Request Q&A (Updated Jul 2023)
### I've submitted an issue and PR. When can I expect to get some feedback?
Megatron-LM is developed and maintained by a small team of researchers. We will endeavour to read and acknowledge all new issues and PRs within a week. A few rules of thumb:
- Reproducible bugs/regressions and bug/regression fixes are likely to get the attention of maintainers the quickest.
- Issues requesting an enhancement may only recieve acknowlegement that they've been read and may be closed with a "wontfix" label if they're not inline with the project direction. If they are acknowledged and remain open you can assume the maintainers agree they're a desirable feature.
- Support requests, i.e. requests for help running the code, have the lowest priority and will be responded to as maintainer time permits.
### If my issue or PR isn't getting attention, how long should I wait before pinging one of the project maintainers?
One week if there is no acknowledgement of the intial request.
### Who are the project maintainers I should ping?
The corresponding maintainers at this time are @jaredcasper and @jon-barker.
### Is there a policy for issues and PRs that haven't been touched in X days? Should they be closed?
Yes, starting in July 2023 we have a bot that will mark untouched PRs as "stale" after 60 days.
We have a long backlog of issues and PRs dating back 3.5 years. We are trying to triage these now by working backwards. Older issues we believe may still be relevant may recieve a request to re-test them with the latest code. If there's no response they may be closed. Again, if you they should be re-opened then just respond with a comment to that effect.
Thank-you!
\ No newline at end of file
Dockerfile.ci.dev 0 → 100644
View file @ d444a97a
# syntax=docker/dockerfile:1.3-labs
ARG FROM_IMAGE_NAME
FROM $FROM_IMAGE_NAME as build_causal_conv1d
WORKDIR /opt
RUN CAUSAL_CONV1D_FORCE_BUILD=TRUE pip3 wheel -v git+https://github.com/Dao-AILab/causal-conv1d.git@v1.2.2.post1
FROM $FROM_IMAGE_NAME as build_grouped_gemm
WORKDIR /opt
RUN pip3 wheel -v git+https://github.com/fanshiqing/grouped_gemm@v1.1.2
FROM $FROM_IMAGE_NAME as build_mamba_ssm
WORKDIR /opt
RUN MAMBA_FORCE_BUILD=TRUE pip3 wheel -v git+https://github.com/state-spaces/mamba.git@v2.2.0
FROM $FROM_IMAGE_NAME as main
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get install -y --no-install-recommends gettext python3-venv && \
apt-get clean && \
python -m venv /opt/jet && \
wget https://github.com/mikefarah/yq/releases/download/v4.44.1/yq_linux_amd64 -O /usr/local/bin/yq && \
chmod a+x /usr/local/bin/yq
COPY --from=build_causal_conv1d /opt/causal_conv1d-*.whl ./
COPY --from=build_grouped_gemm /opt/grouped_gemm-*.whl ./
COPY --from=build_mamba_ssm /opt/mamba_ssm-*.whl ./
RUN \
--mount=type=bind,source=requirements,target=requirements \
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \
--mount=type=bind,source=setup.py,target=setup.py \
--mount=type=bind,source=megatron/core/package_info.py,target=megatron/core/package_info.py \
--mount=type=bind,source=megatron/core/README.md,target=megatron/core/README.md \
--mount=type=bind,source=megatron/core/__init__.py,target=megatron/core/__init__.py <<"EOF" bash -ex
pip install causal_conv1d-*.whl mamba_ssm-*.whl grouped_gemm-*.whl
PY_ENV=pytorch:24.07 pip install .
EOF
# Since megatron does not have any dependencies (and isn't a dependency to any other package), we can install it separately to make everything a bit quicker
ARG MCORE_REPO
ARG MCORE_REF
ARG MCORE_BACKWARDS_REF
RUN <<"EOF" bash -exu
# Checkout latest
cd /opt
rm -rf /opt/megatron-lm; mkdir megatron-lm; cd megatron-lm
git init
git remote add origin ${MCORE_REPO}
git fetch origin '+refs/merge-requests/*:refs/remotes/merge-requests/*'
git fetch origin $MCORE_REF
git checkout $MCORE_REF
# Checkout backwards-ref
cd /opt
rm -rf /opt/megatron-lm-legacy; mkdir megatron-lm-legacy; cd megatron-lm-legacy
git init
git remote add origin ${MCORE_REPO}
git fetch origin $MCORE_BACKWARDS_REF
git checkout $MCORE_BACKWARDS_REF
rm -rf megatron; cp -a /opt/megatron-lm/megatron ./
EOF
RUN PY_ENV=pytorch:24.07 pip install -e /opt/megatron-lm
ENV PYTHONPATH="/opt/megatron-lm:$PYTHONPATH"
##### For NVIDIANS only #####
FROM main as jet
ARG CACHEBUST=0
RUN --mount=type=secret,id=JET_INDEX_URLS \
JET_INDEX_URLS=$(cat /run/secrets/JET_INDEX_URLS) && \
pip install jet-client jet-api --upgrade $JET_INDEX_URLS
ENV PATH="$PATH:/opt/jet/bin"
###
\ No newline at end of file
Dockerfile.ci.lts 0 → 100644
View file @ d444a97a
# syntax=docker/dockerfile:1.3-labs
ARG FROM_IMAGE_NAME
FROM $FROM_IMAGE_NAME as build_causal_conv1d
WORKDIR /opt
RUN CAUSAL_CONV1D_FORCE_BUILD=TRUE pip3 wheel -v git+https://github.com/Dao-AILab/causal-conv1d.git@v1.2.2.post1
FROM $FROM_IMAGE_NAME as build_grouped_gemm
WORKDIR /opt
RUN pip3 wheel -v git+https://github.com/fanshiqing/grouped_gemm@v1.1.2
FROM $FROM_IMAGE_NAME as build_mamba_ssm
WORKDIR /opt
RUN MAMBA_FORCE_BUILD=TRUE pip3 wheel -v git+https://github.com/state-spaces/mamba.git@v2.0.3
ARG FROM_IMAGE_NAME
FROM $FROM_IMAGE_NAME as main
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get install -y --no-install-recommends gettext python3-venv && \
apt-get clean && \
python -m venv /opt/jet && \
wget https://github.com/mikefarah/yq/releases/download/v4.44.1/yq_linux_amd64 -O /usr/local/bin/yq && \
chmod a+x /usr/local/bin/yq
COPY --from=build_causal_conv1d /opt/causal_conv1d-*.whl ./
COPY --from=build_grouped_gemm /opt/grouped_gemm-*.whl ./
COPY --from=build_mamba_ssm /opt/mamba_ssm-*.whl ./
RUN \
--mount=type=bind,source=requirements,target=requirements \
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \
--mount=type=bind,source=setup.py,target=setup.py \
--mount=type=bind,source=megatron/core/package_info.py,target=megatron/core/package_info.py \
--mount=type=bind,source=megatron/core/README.md,target=megatron/core/README.md \
--mount=type=bind,source=megatron/core/__init__.py,target=megatron/core/__init__.py <<"EOF" bash -ex
pip install causal_conv1d-*.whl mamba_ssm-*.whl grouped_gemm-*.whl
PY_ENV=pytorch:24.07 pip install .
EOF
# Since megatron does not have any dependencies (and isn't a dependency to any other package), we can install it separately to make everything a bit quicker
ARG MCORE_REPO
ARG MCORE_REF
ARG MCORE_BACKWARDS_REF
RUN <<"EOF" bash -exu
# Checkout latest
cd /opt
rm -rf /opt/megatron-lm; mkdir megatron-lm; cd megatron-lm
git init
git remote add origin ${MCORE_REPO}
git fetch origin '+refs/merge-requests/*:refs/remotes/merge-requests/*'
git fetch origin $MCORE_REF
git checkout $MCORE_REF
# Checkout backwards-ref
cd /opt
rm -rf /opt/megatron-lm-legacy; mkdir megatron-lm-legacy; cd megatron-lm-legacy
git init
git remote add origin ${MCORE_REPO}
git fetch origin $MCORE_BACKWARDS_REF
git checkout $MCORE_BACKWARDS_REF
rm -rf megatron; cp -a /opt/megatron-lm/megatron ./
EOF
RUN PY_ENV=pytorch:24.01 pip install -e /opt/megatron-lm
ENV PYTHONPATH="/opt/megatron-lm:$PYTHONPATH"
##### For NVIDIANS only #####
FROM main as jet
ARG CACHEBUST=0
RUN --mount=type=secret,id=JET_INDEX_URLS \
JET_INDEX_URLS=$(cat /run/secrets/JET_INDEX_URLS) && \
pip install jet-api jet-client --upgrade $JET_INDEX_URLS
ENV PATH="$PATH:/opt/jet/bin"
###
\ No newline at end of file
Dockerfile.linting 0 → 100644
View file @ d444a97a
# syntax=docker/dockerfile:experimental
ARG FROM_IMAGE_NAME
FROM $FROM_IMAGE_NAME as main
ENV DEBIAN_FRONTEND=noninteractive
RUN sed -i -e 's/^APT/# APT/' -e 's/^DPkg/# DPkg/' \
/etc/apt/apt.conf.d/docker-clean
RUN apt-get update && \
apt-get install -y python3-venv && \
apt-get clean && \
python -m venv /opt/jet
RUN pip3 install --no-cache-dir \
black==24.4.2 \
isort==5.13.2 \
flake8==7.1.0 \
pylint==3.2.6 \
mypy
COPY . /opt/megatron-lm
WORKDIR /opt/megatron-lm
##### For NVIDIANS only #####
FROM main as jet
ARG CACHEBUST=0
RUN --mount=type=secret,id=JET_INDEX_URLS \
JET_INDEX_URLS=$(cat /run/secrets/JET_INDEX_URLS) && \
pip install jet-client jet-api --upgrade $JET_INDEX_URLS
ENV PATH="$PATH:/opt/jet/bin"
###
\ No newline at end of file
GPT_pretraining.sh 0 → 100644
View file @ d444a97a
#!/bin/bash
# Runs the "7B" parameter model
export HSA_FORCE_FINE_GRAIN_PCIE=1
export OMP_NUM_THREADS=1
export NCCL_P2P_LEVEL=SYS
export NCCL_ALGO=Ring
export NCCL_NCHANNELS_PER_PEER=16
export NCCL_MIN_NCHANNELS=20
export NCCL_IB_TIMEOUT=22
export CUDA_DEVICE_MAX_CONNECTIONS=1
export NCCL_NET_GDR_LEVEL=SYS
export NCCL_NET_GDR_READ=0
CHECKPOINT_PATH=./tmp #$1 #<Specify path>
TENSORBOARD_LOGS_PATH=./tmp #$2 #<Specify path>
DATA_PATH="/datasets/oscar-1GB-gpt_text_document" #<Specify path and file prefix>_text_document
VOCAB_PATH=./gpt2-vocab.json
MERGE_PATH=./gpt2-merges.txt
GPT_MODEL_ARGS=(
--num-layers 12
--hidden-size 768
--num-attention-heads 12
--ffn-hidden-size 3072
--seq-length 1024
--max-position-embeddings 1024
)
# export NVTE_FLASH_ATTN=1 # 走autlass
# export NVTE_FLASH_ATTN_TRITON=1 # 走triton_fa
# --transformer-impl transformer_engine
# --use-mcore-models
TRAINING_ARGS=(
--transformer-impl local
--use-legacy-models
--micro-batch-size 1
--global-batch-size 60 #240 #512 #64
--train-iters 100
--weight-decay 0.1
--adam-beta1 0.9
--adam-beta2 0.95
--init-method-std 0.006
--clip-grad 1.0
--bf16
--use-distributed-optimizer
--ckpt-format torch
--disable-bias-linear
--overlap-grad-reduce
--attention-dropout 0
--hidden-dropout 0
--ddp-average-in-collective
--recompute-granularity full
--recompute-num-layers 5
--recompute-method block
--no-gradient-accumulation-fusion
--swiglu
--lr 3.0e-5
--lr-decay-style cosine
--min-lr 3.0e-6
--lr-warmup-iters 1
)
MODEL_PARALLEL_ARGS=(
--sequence-parallel
--tensor-model-parallel-size 2
--pipeline-model-parallel-size 2
)
DATA_ARGS=(
--data-path $DATA_PATH
--split 949,50,1
--untie-embeddings-and-output-weights
--use-rotary-position-embeddings
--normalization RMSNorm
--no-position-embedding
--vocab-file $VOCAB_PATH
--merge-file $MERGE_PATH
--tokenizer-type GPT2BPETokenizer
)
EVAL_AND_LOGGING_ARGS=(
--log-interval 1
--save-interval 10000
--eval-interval 1000
--save $CHECKPOINT_PATH
--load $CHECKPOINT_PATH
--eval-iters 10
--tensorboard-dir $TENSORBOARD_LOGS_PATH
)
RANK=$OMPI_COMM_WORLD_RANK
LOCAL_RANK=$OMPI_COMM_WORLD_LOCAL_RANK
WORLD_SIZE=$OMPI_COMM_WORLD_SIZE
DIST_URL=${1}
DIST_PORT=34566
DISTRIBUTED_ARGS=(
--rank ${RANK}
--world-size ${WORLD_SIZE}
--local-rank ${LOCAL_RANK}
--dist-url tcp://${DIST_URL}:${DIST_PORT}
)
APP="python -u pretrain_gpt.py \
${GPT_MODEL_ARGS[@]} \
${TRAINING_ARGS[@]} \
${MODEL_PARALLEL_ARGS[@]} \
${DATA_ARGS[@]} \
${EVAL_AND_LOGGING_ARGS[@]} \
${DISTRIBUTED_ARGS[@]} \
"
case ${LOCAL_RANK} in
[0])
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
# ${APP}
numactl --cpunodebind=0 --membind=0 ${APP}
;;
[1])
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
# ${APP}
numactl --cpunodebind=0 --membind=0 ${APP}
;;
[2])
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
# ${APP}
numactl --cpunodebind=0 --membind=0 ${APP}
;;
[3])
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
# ${APP}
numactl --cpunodebind=0 --membind=0 ${APP}
;;
[4])
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
# ${APP}
numactl --cpunodebind=0 --membind=0 ${APP}
;;
[5])
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
# ${APP}
numactl --cpunodebind=0 --membind=0 ${APP}
;;
[6])
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
# ${APP}
numactl --cpunodebind=0 --membind=0 ${APP}
;;
[7])
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
# ${APP}
numactl --cpunodebind=0 --membind=0 ${APP}
;;
esac
LICENSE 0 → 100644
View file @ d444a97a
The following applies to all files unless otherwise noted:
# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
--
This repository also contains code from Hugging Face Inc., Google Research,
Facebook (from their Fairseq, Dino, and ParlAI projects), Microsoft (from their
Swin-Transformer project), Philip Popien, the Mamba project (Tri Dao and
Albert Gu), and the Triton language and compiler project (Philippe Tillet and
OpenAI). Files from these organizations have notices at the top of each file.
Below are licenses used in those files, as indicated.
--------------------------------------------------------------------------------
-- LICENSE FOR Facebook, huggingface, Google Research, LLaVA, and Mamba code --
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--------------------------------------------------------------------------------
LICENSE FOR
Facebook, Inc. and its affiliates,
Meta Platforms, Inc. and its affiliates,
Microsoft Corporation,
OpenGVLab/InternVL, and
Triton language and compiler.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Llama_pretraining.sh 0 → 100755
View file @ d444a97a
#!/bin/bash
set -eux
#export FLASH_ATTENTION_PRINT_PARAM=1
# Runs the "7B" parameter model
export HSA_FORCE_FINE_GRAIN_PCIE=1
export OMP_NUM_THREADS=1
export NCCL_P2P_LEVEL=PXB # SYS
#export HIP_ALLOC_INITIALIZE=0
# export GPU_MAX_HW_QUEUES=10
export NCCL_ALGO=Ring
export NCCL_NCHANNELS_PER_PEER=16
export NCCL_MIN_NCHANNELS=32 # 20
export NCCL_MAX_NCHANNELS=32 # 20
export NCCL_IB_TIMEOUT=22
export CUDA_DEVICE_MAX_CONNECTIONS=1
export NCCL_IB_HCA=mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1
export NCCL_NET_GDR_LEVEL=7
export NCCL_NET_GDR_READ=1
export RCCL_SDMA_COPY_ENABLE=0
export GLOG_minloglevel=3 # 打印error级别的nccl日志
export ALLREDUCE_STREAM_WITH_COMPUTE=1
export SENDRECV_STREAM_WITH_COMPUTE=1
export cache_size_limit=64
SAVE_PATH=./tmp_7b
TENSORBOARD_LOGS_PATH=./tmp_7b #$2 #<Specify path>
DATA_PATH="/models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document" #<Specify path and file prefix>_text_document
GPT_MODEL_ARGS=(
--num-layers 32
--hidden-size 4096
--ffn-hidden-size 11008
--num-attention-heads 32
--max-position-embeddings 4096
--normalization RMSNorm
--position-embedding-type rope # none #
--untie-embeddings-and-output-weights # 分开处理embed和输出权重, 增加灵活性
)
export NVTE_FLASH_ATTN=1 # 走cutlass
# export NVTE_FLASH_ATTN_TRITON=1 # 走triton_fa
# --transformer-impl transformer_engine # 走core用这两组参数
# --use-mcore-models
# --transformer-impl local # 走legacy用这两组参数
# --use-legacy-models
TRAINING_ARGS=(
--transformer-impl local # 走legacy用这两组参数
--use-legacy-models
--micro-batch-size 1
--global-batch-size 64 #32 #240 #60 #512 #64
--train-iters 50
--weight-decay 0.1
--adam-beta1 0.9
--adam-beta2 0.95
--init-method-std 0.006
--clip-grad 1.0
--bf16
# --fp16 # 开启fp16需要指定loss-scale
# --loss-scale 1024
--use-distributed-optimizer
--disable-bias-linear
--attention-dropout 0
--hidden-dropout 0
# --no-gradient-accumulation-fusion
--swiglu
--lr 3.0e-5
--lr-decay-style cosine
--min-lr 3.0e-6
--lr-warmup-iters 1
--ckpt-format torch
--ddp-average-in-collective # 在dp阶段通信中, 梯度或参数将被直接平均, 而不是先求和(到一个设备)再平均
# --recompute-granularity full # 开启重计算降低显存增加耗时
# --recompute-num-layers 5 #0 #
# --recompute-method block
--overlap-grad-reduce # 重叠ddp grad reduce
# --tp-comm-overlap # tensor parallel comm和gemm重叠, 优化项未适配
# --tp-comm-overlap-rs-dgrad # reduce-scatter和dgrad gemm重叠
--use-flash-attn-cutlass
)
# 使用torch fa的环境变量
# export TORCHINDUCTOR_COORDINATE_DESCENT_TUNING=1
# export TORCHINDUCTOR_BENCHMARK_FUSION=1
# export TORCHINDUCTOR_BENCHMARK_MULTI_TEMPLATES=1
# export TORCHINDUCTOR_MAX_AUTOTUNE=1
# export TORCHINDUCTOR_CACHE_DIR=./cache
# --use-flash-attn-cutlass # cutlass fa
# --use-flash-attn-triton # triton fa
# --use-flash-attn-torch # torch fa
MODEL_PARALLEL_ARGS=(
--sequence-parallel
--tensor-model-parallel-size 1
--pipeline-model-parallel-size 2
# --context-parallel-size 2
# --num-layers-per-virtual-pipeline-stage 4
# --microbatch-group-size-per-virtual-pipeline-stage 1
# --no-overlap-p2p-communication # 开启后
)
DATA_ARGS=(
--data-path $DATA_PATH
--seq-length 4096 #4096
--split 949,50,1
--tokenizer-type Llama2Tokenizer
--tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model
# --tokenizer-model /data/model_weights/llama2_7b_hf/tokenizer.model
)
EVAL_AND_LOGGING_ARGS=(
--log-interval 1
--log-throughput
--save-interval 1000
--eval-interval 1000
#--save $SAVE_PATH
#--load $SAVE_PATH
--eval-iters 3
--tensorboard-dir $TENSORBOARD_LOGS_PATH
)
# FINETUNE_ARGS=(
# # --finetune
# # --pretrained-checkpoint $CHECKPOINT_PATH
# --load $CHECKPOINT_PATH
# --no-load-optim
# --no-load-rng
# )
PROFILE_ARGS=(
--profile
--profile-step-start 4
--profile-step-end 5
--use-pytorch-profiler
--profile-ranks 0 1 2 3 4 5 6 7
--profile-dir prof_data
)
RANK=$OMPI_COMM_WORLD_RANK
LOCAL_RANK=$OMPI_COMM_WORLD_LOCAL_RANK
WORLD_SIZE=$OMPI_COMM_WORLD_SIZE
DIST_URL=${1}
DIST_PORT=34577
DISTRIBUTED_ARGS=(
--rank ${RANK}
--world-size ${WORLD_SIZE}
--local-rank ${LOCAL_RANK}
--dist-url tcp://${DIST_URL}:${DIST_PORT}
)
APP="python -u pretrain_gpt.py \
${GPT_MODEL_ARGS[@]} \
${TRAINING_ARGS[@]} \
${MODEL_PARALLEL_ARGS[@]} \
${DATA_ARGS[@]} \
${EVAL_AND_LOGGING_ARGS[@]} \
${DISTRIBUTED_ARGS[@]} \
"
# 开启profile
# ${PROFILE_ARGS[@]} \
# export HIP_VISIBLE_DEVICES=0,7 # # 4,5,6,7 #,
export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 # # 4,5,6,7 #,
# export CUDA_VISIBLE_DEVICES=4,5,6,7 # 0,1,2,3,
# ${APP}
case ${LOCAL_RANK} in
[0])
export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
# hipprof --hip-trace --trace-off numactl --cpunodebind=0 --membind=0 ${APP}
numactl --cpunodebind=0 --membind=0 ${APP}
;;
[1])
export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
# hipprof --hip-trace --trace-off numactl --cpunodebind=0 --membind=0 ${APP}
numactl --cpunodebind=1 --membind=1 ${APP}
;;
[2])
export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
# hipprof --hip-trace --trace-off numactl --cpunodebind=0 --membind=0 ${APP}
numactl --cpunodebind=2 --membind=2 ${APP}
;;
[3])
export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
numactl --cpunodebind=3 --membind=3 ${APP}
# hipprof --hip-trace --trace-off numactl --cpunodebind=0 --membind=0 ${APP}
;;
[4])
export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
numactl --cpunodebind=4 --membind=4 ${APP}
# hipprof --hip-trace --trace-off numactl --cpunodebind=0 --membind=0 ${APP}
;;
[5])
export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
numactl --cpunodebind=5 --membind=5 ${APP}
# hipprof --hip-trace --trace-off numactl --cpunodebind=0 --membind=0 ${APP}
;;
[6])
export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
numactl --cpunodebind=6 --membind=6 ${APP}
# hipprof --hip-trace --trace-off numactl --cpunodebind=0 --membind=0 ${APP}
;;
[7])
export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
numactl --cpunodebind=7 --membind=7 ${APP}
# hipprof --hip-trace --trace-off numactl --cpunodebind=0 --membind=0 ${APP}
;;
esac
Llama_pretraining_8ka.log 0 → 100644
View file @ d444a97a
+ export HSA_FORCE_FINE_GRAIN_PCIE=1
+ HSA_FORCE_FINE_GRAIN_PCIE=1
+ export OMP_NUM_THREADS=1
+ OMP_NUM_THREADS=1
+ export HSA_FORCE_FINE_GRAIN_PCIE=1
+ HSA_FORCE_FINE_GRAIN_PCIE=1
+ export OMP_NUM_THREADS=1
+ OMP_NUM_THREADS=1
+ export NCCL_P2P_LEVEL=PXB
+ NCCL_P2P_LEVEL=PXB
+ export NCCL_ALGO=Ring
+ NCCL_ALGO=Ring
+ export NCCL_NCHANNELS_PER_PEER=16
+ NCCL_NCHANNELS_PER_PEER=16
+ export NCCL_P2P_LEVEL=PXB
+ NCCL_P2P_LEVEL=PXB
+ export NCCL_ALGO=Ring
+ NCCL_ALGO=Ring
+ export NCCL_NCHANNELS_PER_PEER=16
+ NCCL_NCHANNELS_PER_PEER=16
+ export NCCL_MIN_NCHANNELS=32
+ NCCL_MIN_NCHANNELS=32
+ export NCCL_MIN_NCHANNELS=32
+ NCCL_MIN_NCHANNELS=32
+ export NCCL_MAX_NCHANNELS=32
+ NCCL_MAX_NCHANNELS=32
+ export NCCL_MAX_NCHANNELS=32
+ NCCL_MAX_NCHANNELS=32
+ export NCCL_IB_TIMEOUT=22
+ NCCL_IB_TIMEOUT=22
+ export CUDA_DEVICE_MAX_CONNECTIONS=1
+ export NCCL_IB_TIMEOUT=22
+ NCCL_IB_TIMEOUT=22
+ export CUDA_DEVICE_MAX_CONNECTIONS=1
+ CUDA_DEVICE_MAX_CONNECTIONS=1
+ export NCCL_IB_HCA=mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1
+ NCCL_IB_HCA=mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1
+ CUDA_DEVICE_MAX_CONNECTIONS=1
+ export NCCL_IB_HCA=mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1
+ NCCL_IB_HCA=mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1
+ export NCCL_NET_GDR_LEVEL=7
+ NCCL_NET_GDR_LEVEL=7
+ export NCCL_NET_GDR_LEVEL=7
+ NCCL_NET_GDR_LEVEL=7
+ export NCCL_NET_GDR_READ=1
+ export NCCL_NET_GDR_READ=1
+ NCCL_NET_GDR_READ=1
+ NCCL_NET_GDR_READ=1
+ export RCCL_SDMA_COPY_ENABLE=0
+ RCCL_SDMA_COPY_ENABLE=0
+ export RCCL_SDMA_COPY_ENABLE=0
+ RCCL_SDMA_COPY_ENABLE=0
+ export GLOG_minloglevel=3
+ GLOG_minloglevel=3
+ export GLOG_minloglevel=3
+ GLOG_minloglevel=3
+ export ALLREDUCE_STREAM_WITH_COMPUTE=1
+ ALLREDUCE_STREAM_WITH_COMPUTE=1
+ export SENDRECV_STREAM_WITH_COMPUTE=1
+ export ALLREDUCE_STREAM_WITH_COMPUTE=1
+ ALLREDUCE_STREAM_WITH_COMPUTE=1
+ export SENDRECV_STREAM_WITH_COMPUTE=1
+ SENDRECV_STREAM_WITH_COMPUTE=1
+ SENDRECV_STREAM_WITH_COMPUTE=1
+ export cache_size_limit=64
+ cache_size_limit=64
+ export cache_size_limit=64
+ cache_size_limit=64
+ SAVE_PATH=./tmp_7b
+ TENSORBOARD_LOGS_PATH=./tmp_7b
+ SAVE_PATH=./tmp_7b
+ TENSORBOARD_LOGS_PATH=./tmp_7b
+ DATA_PATH=/models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document
+ DATA_PATH=/models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document
+ GPT_MODEL_ARGS=(--num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights)
+ export HSA_FORCE_FINE_GRAIN_PCIE=1
+ HSA_FORCE_FINE_GRAIN_PCIE=1
+ export OMP_NUM_THREADS=1
+ OMP_NUM_THREADS=1
+ export NCCL_P2P_LEVEL=PXB
+ GPT_MODEL_ARGS=(--num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights)
+ NCCL_P2P_LEVEL=PXB
+ export NCCL_ALGO=Ring
+ NCCL_ALGO=Ring
+ export NCCL_NCHANNELS_PER_PEER=16
+ NCCL_NCHANNELS_PER_PEER=16
+ export NCCL_MIN_NCHANNELS=32
+ NCCL_MIN_NCHANNELS=32
+ export NVTE_FLASH_ATTN=1
+ NVTE_FLASH_ATTN=1
+ export NVTE_FLASH_ATTN=1
+ NVTE_FLASH_ATTN=1
+ export NCCL_MAX_NCHANNELS=32
+ NCCL_MAX_NCHANNELS=32
+ export NCCL_IB_TIMEOUT=22
+ NCCL_IB_TIMEOUT=22
+ TRAINING_ARGS=(--transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass)
+ TRAINING_ARGS=(--transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass)
+ export CUDA_DEVICE_MAX_CONNECTIONS=1
+ CUDA_DEVICE_MAX_CONNECTIONS=1
+ export NCCL_IB_HCA=mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1
+ NCCL_IB_HCA=mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1
+ export NCCL_NET_GDR_LEVEL=7
+ NCCL_NET_GDR_LEVEL=7
+ MODEL_PARALLEL_ARGS=(--sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2)
+ DATA_ARGS=(--data-path $DATA_PATH --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model)
+ MODEL_PARALLEL_ARGS=(--sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2)
+ DATA_ARGS=(--data-path $DATA_PATH --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model)
+ export NCCL_NET_GDR_READ=1
+ NCCL_NET_GDR_READ=1
+ export RCCL_SDMA_COPY_ENABLE=0
+ RCCL_SDMA_COPY_ENABLE=0
+ export GLOG_minloglevel=3
+ GLOG_minloglevel=3
+ EVAL_AND_LOGGING_ARGS=(--log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir $TENSORBOARD_LOGS_PATH)
+ EVAL_AND_LOGGING_ARGS=(--log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir $TENSORBOARD_LOGS_PATH)
+ export ALLREDUCE_STREAM_WITH_COMPUTE=1
+ ALLREDUCE_STREAM_WITH_COMPUTE=1
+ export SENDRECV_STREAM_WITH_COMPUTE=1
+ SENDRECV_STREAM_WITH_COMPUTE=1
+ export cache_size_limit=64
+ PROFILE_ARGS=(--profile --profile-step-start 4 --profile-step-end 5 --use-pytorch-profiler --profile-ranks 0 1 2 3 4 5 6 7 --profile-dir prof_data)
+ PROFILE_ARGS=(--profile --profile-step-start 4 --profile-step-end 5 --use-pytorch-profiler --profile-ranks 0 1 2 3 4 5 6 7 --profile-dir prof_data)
+ cache_size_limit=64
+ SAVE_PATH=./tmp_7b
+ TENSORBOARD_LOGS_PATH=./tmp_7b
+ RANK=1
+ LOCAL_RANK=1
+ WORLD_SIZE=8
+ RANK=0
+ LOCAL_RANK=0
+ WORLD_SIZE=8
+ DATA_PATH=/models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document
+ GPT_MODEL_ARGS=(--num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights)
+ DIST_URL=localhost
+ DIST_PORT=34577
+ DIST_URL=localhost
+ DIST_PORT=34577
+ DISTRIBUTED_ARGS=(--rank ${RANK} --world-size ${WORLD_SIZE} --local-rank ${LOCAL_RANK} --dist-url tcp://${DIST_URL}:${DIST_PORT})
+ DISTRIBUTED_ARGS=(--rank ${RANK} --world-size ${WORLD_SIZE} --local-rank ${LOCAL_RANK} --dist-url tcp://${DIST_URL}:${DIST_PORT})
+ export NVTE_FLASH_ATTN=1
+ NVTE_FLASH_ATTN=1
+ APP='python -u pretrain_gpt.py --num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights --transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass --sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2 --data-path /models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model --log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir ./tmp_7b --rank 0 --world-size 8 --local-rank 0 --dist-url tcp://localhost:34577 '
+ export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ APP='python -u pretrain_gpt.py --num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights --transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass --sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2 --data-path /models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model --log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir ./tmp_7b --rank 1 --world-size 8 --local-rank 1 --dist-url tcp://localhost:34577 '
+ export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ TRAINING_ARGS=(--transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass)
+ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ case ${LOCAL_RANK} in
+ export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ case ${LOCAL_RANK} in
+ export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ MODEL_PARALLEL_ARGS=(--sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2)
+ DATA_ARGS=(--data-path $DATA_PATH --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model)
+ EVAL_AND_LOGGING_ARGS=(--log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir $TENSORBOARD_LOGS_PATH)
+ numactl --cpunodebind=1 --membind=1 python -u pretrain_gpt.py --num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights --transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass --sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2 --data-path /models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model --log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir ./tmp_7b --rank 1 --world-size 8 --local-rank 1 --dist-url tcp://localhost:34577
+ numactl --cpunodebind=0 --membind=0 python -u pretrain_gpt.py --num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights --transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass --sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2 --data-path /models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model --log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir ./tmp_7b --rank 0 --world-size 8 --local-rank 0 --dist-url tcp://localhost:34577
+ PROFILE_ARGS=(--profile --profile-step-start 4 --profile-step-end 5 --use-pytorch-profiler --profile-ranks 0 1 2 3 4 5 6 7 --profile-dir prof_data)
+ RANK=3
+ LOCAL_RANK=3
+ WORLD_SIZE=8
+ DIST_URL=localhost
+ DIST_PORT=34577
+ DISTRIBUTED_ARGS=(--rank ${RANK} --world-size ${WORLD_SIZE} --local-rank ${LOCAL_RANK} --dist-url tcp://${DIST_URL}:${DIST_PORT})
+ APP='python -u pretrain_gpt.py --num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights --transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass --sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2 --data-path /models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model --log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir ./tmp_7b --rank 3 --world-size 8 --local-rank 3 --dist-url tcp://localhost:34577 '
+ export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ case ${LOCAL_RANK} in
+ export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ numactl --cpunodebind=3 --membind=3 python -u pretrain_gpt.py --num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights --transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass --sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2 --data-path /models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model --log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir ./tmp_7b --rank 3 --world-size 8 --local-rank 3 --dist-url tcp://localhost:34577
+ export HSA_FORCE_FINE_GRAIN_PCIE=1
+ HSA_FORCE_FINE_GRAIN_PCIE=1
+ export HSA_FORCE_FINE_GRAIN_PCIE=1
+ HSA_FORCE_FINE_GRAIN_PCIE=1
+ export OMP_NUM_THREADS=1
+ OMP_NUM_THREADS=1
+ export OMP_NUM_THREADS=1
+ OMP_NUM_THREADS=1
+ export NCCL_P2P_LEVEL=PXB
+ NCCL_P2P_LEVEL=PXB
+ export NCCL_P2P_LEVEL=PXB
+ NCCL_P2P_LEVEL=PXB
+ export NCCL_ALGO=Ring
+ NCCL_ALGO=Ring
+ export NCCL_ALGO=Ring
+ NCCL_ALGO=Ring
+ export NCCL_NCHANNELS_PER_PEER=16
+ export NCCL_NCHANNELS_PER_PEER=16
+ NCCL_NCHANNELS_PER_PEER=16
+ export NCCL_MIN_NCHANNELS=32
+ NCCL_MIN_NCHANNELS=32
+ NCCL_NCHANNELS_PER_PEER=16
+ export NCCL_MIN_NCHANNELS=32
+ NCCL_MIN_NCHANNELS=32
+ export NCCL_MAX_NCHANNELS=32
+ NCCL_MAX_NCHANNELS=32
+ export NCCL_IB_TIMEOUT=22
+ NCCL_IB_TIMEOUT=22
+ export NCCL_MAX_NCHANNELS=32
+ NCCL_MAX_NCHANNELS=32
+ export NCCL_IB_TIMEOUT=22
+ NCCL_IB_TIMEOUT=22
+ export CUDA_DEVICE_MAX_CONNECTIONS=1
+ export CUDA_DEVICE_MAX_CONNECTIONS=1
+ CUDA_DEVICE_MAX_CONNECTIONS=1
+ CUDA_DEVICE_MAX_CONNECTIONS=1
+ export NCCL_IB_HCA=mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1
+ NCCL_IB_HCA=mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1
+ export NCCL_NET_GDR_LEVEL=7
+ NCCL_NET_GDR_LEVEL=7
+ export NCCL_IB_HCA=mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1
+ NCCL_IB_HCA=mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1
+ export NCCL_NET_GDR_LEVEL=7
+ NCCL_NET_GDR_LEVEL=7
+ export NCCL_NET_GDR_READ=1
+ export NCCL_NET_GDR_READ=1
+ NCCL_NET_GDR_READ=1
+ export RCCL_SDMA_COPY_ENABLE=0
+ NCCL_NET_GDR_READ=1
+ export RCCL_SDMA_COPY_ENABLE=0
+ RCCL_SDMA_COPY_ENABLE=0
+ RCCL_SDMA_COPY_ENABLE=0
+ export GLOG_minloglevel=3
+ GLOG_minloglevel=3
+ export GLOG_minloglevel=3
+ GLOG_minloglevel=3
+ export ALLREDUCE_STREAM_WITH_COMPUTE=1
+ ALLREDUCE_STREAM_WITH_COMPUTE=1
+ export SENDRECV_STREAM_WITH_COMPUTE=1
+ export ALLREDUCE_STREAM_WITH_COMPUTE=1
+ ALLREDUCE_STREAM_WITH_COMPUTE=1
+ export SENDRECV_STREAM_WITH_COMPUTE=1
+ SENDRECV_STREAM_WITH_COMPUTE=1
+ SENDRECV_STREAM_WITH_COMPUTE=1
+ export cache_size_limit=64
+ cache_size_limit=64
+ SAVE_PATH=./tmp_7b
+ export cache_size_limit=64
+ cache_size_limit=64
+ SAVE_PATH=./tmp_7b
+ TENSORBOARD_LOGS_PATH=./tmp_7b
+ TENSORBOARD_LOGS_PATH=./tmp_7b
+ DATA_PATH=/models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document
+ DATA_PATH=/models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document
+ GPT_MODEL_ARGS=(--num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights)
+ GPT_MODEL_ARGS=(--num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights)
+ export NVTE_FLASH_ATTN=1
+ NVTE_FLASH_ATTN=1
+ export NVTE_FLASH_ATTN=1
+ NVTE_FLASH_ATTN=1
+ TRAINING_ARGS=(--transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass)
+ TRAINING_ARGS=(--transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass)
+ MODEL_PARALLEL_ARGS=(--sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2)
+ MODEL_PARALLEL_ARGS=(--sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2)
+ DATA_ARGS=(--data-path $DATA_PATH --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model)
+ DATA_ARGS=(--data-path $DATA_PATH --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model)
+ EVAL_AND_LOGGING_ARGS=(--log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir $TENSORBOARD_LOGS_PATH)
+ EVAL_AND_LOGGING_ARGS=(--log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir $TENSORBOARD_LOGS_PATH)
+ PROFILE_ARGS=(--profile --profile-step-start 4 --profile-step-end 5 --use-pytorch-profiler --profile-ranks 0 1 2 3 4 5 6 7 --profile-dir prof_data)
+ PROFILE_ARGS=(--profile --profile-step-start 4 --profile-step-end 5 --use-pytorch-profiler --profile-ranks 0 1 2 3 4 5 6 7 --profile-dir prof_data)
+ RANK=4
+ LOCAL_RANK=4
+ RANK=6
+ LOCAL_RANK=6
+ WORLD_SIZE=8
+ WORLD_SIZE=8
+ DIST_URL=localhost
+ DIST_URL=localhost
+ DIST_PORT=34577
+ DIST_PORT=34577
+ DISTRIBUTED_ARGS=(--rank ${RANK} --world-size ${WORLD_SIZE} --local-rank ${LOCAL_RANK} --dist-url tcp://${DIST_URL}:${DIST_PORT})
+ DISTRIBUTED_ARGS=(--rank ${RANK} --world-size ${WORLD_SIZE} --local-rank ${LOCAL_RANK} --dist-url tcp://${DIST_URL}:${DIST_PORT})
+ APP='python -u pretrain_gpt.py --num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights --transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass --sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2 --data-path /models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model --log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir ./tmp_7b --rank 6 --world-size 8 --local-rank 6 --dist-url tcp://localhost:34577 '
+ APP='python -u pretrain_gpt.py --num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights --transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass --sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2 --data-path /models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model --log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir ./tmp_7b --rank 4 --world-size 8 --local-rank 4 --dist-url tcp://localhost:34577 '
+ export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ case ${LOCAL_RANK} in
+ export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ case ${LOCAL_RANK} in
+ export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ numactl --cpunodebind=6 --membind=6 python -u pretrain_gpt.py --num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights --transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass --sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2 --data-path /models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model --log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir ./tmp_7b --rank 6 --world-size 8 --local-rank 6 --dist-url tcp://localhost:34577
+ numactl --cpunodebind=4 --membind=4 python -u pretrain_gpt.py --num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights --transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass --sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2 --data-path /models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model --log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir ./tmp_7b --rank 4 --world-size 8 --local-rank 4 --dist-url tcp://localhost:34577
+ export HSA_FORCE_FINE_GRAIN_PCIE=1
+ HSA_FORCE_FINE_GRAIN_PCIE=1
+ export OMP_NUM_THREADS=1
+ OMP_NUM_THREADS=1
+ export NCCL_P2P_LEVEL=PXB
+ NCCL_P2P_LEVEL=PXB
+ export NCCL_ALGO=Ring
+ NCCL_ALGO=Ring
+ export NCCL_NCHANNELS_PER_PEER=16
+ NCCL_NCHANNELS_PER_PEER=16
+ export NCCL_MIN_NCHANNELS=32
+ NCCL_MIN_NCHANNELS=32
+ export NCCL_MAX_NCHANNELS=32
+ NCCL_MAX_NCHANNELS=32
+ export NCCL_IB_TIMEOUT=22
+ NCCL_IB_TIMEOUT=22
+ export CUDA_DEVICE_MAX_CONNECTIONS=1
+ CUDA_DEVICE_MAX_CONNECTIONS=1
+ export NCCL_IB_HCA=mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1
+ NCCL_IB_HCA=mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1
+ export NCCL_NET_GDR_LEVEL=7
+ NCCL_NET_GDR_LEVEL=7
+ export NCCL_NET_GDR_READ=1
+ NCCL_NET_GDR_READ=1
+ export RCCL_SDMA_COPY_ENABLE=0
+ RCCL_SDMA_COPY_ENABLE=0
+ export GLOG_minloglevel=3
+ GLOG_minloglevel=3
+ export ALLREDUCE_STREAM_WITH_COMPUTE=1
+ ALLREDUCE_STREAM_WITH_COMPUTE=1
+ export SENDRECV_STREAM_WITH_COMPUTE=1
+ SENDRECV_STREAM_WITH_COMPUTE=1
+ export cache_size_limit=64
+ cache_size_limit=64
+ SAVE_PATH=./tmp_7b
+ TENSORBOARD_LOGS_PATH=./tmp_7b
+ DATA_PATH=/models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document
+ GPT_MODEL_ARGS=(--num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights)
+ export NVTE_FLASH_ATTN=1
+ NVTE_FLASH_ATTN=1
+ TRAINING_ARGS=(--transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass)
+ MODEL_PARALLEL_ARGS=(--sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2)
+ DATA_ARGS=(--data-path $DATA_PATH --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model)
+ EVAL_AND_LOGGING_ARGS=(--log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir $TENSORBOARD_LOGS_PATH)
+ PROFILE_ARGS=(--profile --profile-step-start 4 --profile-step-end 5 --use-pytorch-profiler --profile-ranks 0 1 2 3 4 5 6 7 --profile-dir prof_data)
+ RANK=2
+ LOCAL_RANK=2
+ WORLD_SIZE=8
+ export HSA_FORCE_FINE_GRAIN_PCIE=1
+ HSA_FORCE_FINE_GRAIN_PCIE=1
+ DIST_URL=localhost
+ DIST_PORT=34577
+ export OMP_NUM_THREADS=1
+ OMP_NUM_THREADS=1
+ DISTRIBUTED_ARGS=(--rank ${RANK} --world-size ${WORLD_SIZE} --local-rank ${LOCAL_RANK} --dist-url tcp://${DIST_URL}:${DIST_PORT})
+ export NCCL_P2P_LEVEL=PXB
+ NCCL_P2P_LEVEL=PXB
+ export NCCL_ALGO=Ring
+ NCCL_ALGO=Ring
+ export NCCL_NCHANNELS_PER_PEER=16
+ NCCL_NCHANNELS_PER_PEER=16
+ export NCCL_MIN_NCHANNELS=32
+ NCCL_MIN_NCHANNELS=32
+ export NCCL_MAX_NCHANNELS=32
+ NCCL_MAX_NCHANNELS=32
+ export NCCL_IB_TIMEOUT=22
+ NCCL_IB_TIMEOUT=22
+ APP='python -u pretrain_gpt.py --num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights --transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass --sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2 --data-path /models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model --log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir ./tmp_7b --rank 2 --world-size 8 --local-rank 2 --dist-url tcp://localhost:34577 '
+ export CUDA_DEVICE_MAX_CONNECTIONS=1
+ CUDA_DEVICE_MAX_CONNECTIONS=1
+ export NCCL_IB_HCA=mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1
+ export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ NCCL_IB_HCA=mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1
+ export NCCL_NET_GDR_LEVEL=7
+ NCCL_NET_GDR_LEVEL=7
+ export NCCL_NET_GDR_READ=1
+ NCCL_NET_GDR_READ=1
+ export RCCL_SDMA_COPY_ENABLE=0
+ RCCL_SDMA_COPY_ENABLE=0
+ export GLOG_minloglevel=3
+ GLOG_minloglevel=3
+ case ${LOCAL_RANK} in
+ export ALLREDUCE_STREAM_WITH_COMPUTE=1
+ ALLREDUCE_STREAM_WITH_COMPUTE=1
+ export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ export SENDRECV_STREAM_WITH_COMPUTE=1
+ SENDRECV_STREAM_WITH_COMPUTE=1
+ export cache_size_limit=64
+ cache_size_limit=64
+ SAVE_PATH=./tmp_7b
+ TENSORBOARD_LOGS_PATH=./tmp_7b
+ DATA_PATH=/models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document
+ numactl --cpunodebind=2 --membind=2 python -u pretrain_gpt.py --num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights --transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass --sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2 --data-path /models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model --log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir ./tmp_7b --rank 2 --world-size 8 --local-rank 2 --dist-url tcp://localhost:34577
+ GPT_MODEL_ARGS=(--num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights)
+ export NVTE_FLASH_ATTN=1
+ NVTE_FLASH_ATTN=1
+ TRAINING_ARGS=(--transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass)
+ MODEL_PARALLEL_ARGS=(--sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2)
+ DATA_ARGS=(--data-path $DATA_PATH --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model)
+ EVAL_AND_LOGGING_ARGS=(--log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir $TENSORBOARD_LOGS_PATH)
+ PROFILE_ARGS=(--profile --profile-step-start 4 --profile-step-end 5 --use-pytorch-profiler --profile-ranks 0 1 2 3 4 5 6 7 --profile-dir prof_data)
+ RANK=5
+ LOCAL_RANK=5
+ WORLD_SIZE=8
+ DIST_URL=localhost
+ DIST_PORT=34577
+ DISTRIBUTED_ARGS=(--rank ${RANK} --world-size ${WORLD_SIZE} --local-rank ${LOCAL_RANK} --dist-url tcp://${DIST_URL}:${DIST_PORT})
+ APP='python -u pretrain_gpt.py --num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights --transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass --sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2 --data-path /models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model --log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir ./tmp_7b --rank 5 --world-size 8 --local-rank 5 --dist-url tcp://localhost:34577 '
+ export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ case ${LOCAL_RANK} in
+ export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ numactl --cpunodebind=5 --membind=5 python -u pretrain_gpt.py --num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights --transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass --sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2 --data-path /models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model --log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir ./tmp_7b --rank 5 --world-size 8 --local-rank 5 --dist-url tcp://localhost:34577
+ export HSA_FORCE_FINE_GRAIN_PCIE=1
+ HSA_FORCE_FINE_GRAIN_PCIE=1
+ export OMP_NUM_THREADS=1
+ OMP_NUM_THREADS=1
+ export NCCL_P2P_LEVEL=PXB
+ NCCL_P2P_LEVEL=PXB
+ export NCCL_ALGO=Ring
+ NCCL_ALGO=Ring
+ export NCCL_NCHANNELS_PER_PEER=16
+ NCCL_NCHANNELS_PER_PEER=16
+ export NCCL_MIN_NCHANNELS=32
+ NCCL_MIN_NCHANNELS=32
+ export NCCL_MAX_NCHANNELS=32
+ NCCL_MAX_NCHANNELS=32
+ export NCCL_IB_TIMEOUT=22
+ NCCL_IB_TIMEOUT=22
+ export CUDA_DEVICE_MAX_CONNECTIONS=1
+ CUDA_DEVICE_MAX_CONNECTIONS=1
+ export NCCL_IB_HCA=mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1
+ NCCL_IB_HCA=mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1,mlx5_9:1
+ export NCCL_NET_GDR_LEVEL=7
+ NCCL_NET_GDR_LEVEL=7
+ export NCCL_NET_GDR_READ=1
+ NCCL_NET_GDR_READ=1
+ export RCCL_SDMA_COPY_ENABLE=0
+ RCCL_SDMA_COPY_ENABLE=0
+ export GLOG_minloglevel=3
+ GLOG_minloglevel=3
+ export ALLREDUCE_STREAM_WITH_COMPUTE=1
+ ALLREDUCE_STREAM_WITH_COMPUTE=1
+ export SENDRECV_STREAM_WITH_COMPUTE=1
+ SENDRECV_STREAM_WITH_COMPUTE=1
+ export cache_size_limit=64
+ cache_size_limit=64
+ SAVE_PATH=./tmp_7b
+ TENSORBOARD_LOGS_PATH=./tmp_7b
+ DATA_PATH=/models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document
+ GPT_MODEL_ARGS=(--num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights)
+ export NVTE_FLASH_ATTN=1
+ NVTE_FLASH_ATTN=1
+ TRAINING_ARGS=(--transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass)
+ MODEL_PARALLEL_ARGS=(--sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2)
+ DATA_ARGS=(--data-path $DATA_PATH --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model)
+ EVAL_AND_LOGGING_ARGS=(--log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir $TENSORBOARD_LOGS_PATH)
+ PROFILE_ARGS=(--profile --profile-step-start 4 --profile-step-end 5 --use-pytorch-profiler --profile-ranks 0 1 2 3 4 5 6 7 --profile-dir prof_data)
+ RANK=7
+ LOCAL_RANK=7
+ WORLD_SIZE=8
+ DIST_URL=localhost
+ DIST_PORT=34577
+ DISTRIBUTED_ARGS=(--rank ${RANK} --world-size ${WORLD_SIZE} --local-rank ${LOCAL_RANK} --dist-url tcp://${DIST_URL}:${DIST_PORT})
+ APP='python -u pretrain_gpt.py --num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights --transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass --sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2 --data-path /models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model --log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir ./tmp_7b --rank 7 --world-size 8 --local-rank 7 --dist-url tcp://localhost:34577 '
+ export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ case ${LOCAL_RANK} in
+ export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+ numactl --cpunodebind=7 --membind=7 python -u pretrain_gpt.py --num-layers 32 --hidden-size 4096 --ffn-hidden-size 11008 --num-attention-heads 32 --max-position-embeddings 4096 --normalization RMSNorm --position-embedding-type rope --untie-embeddings-and-output-weights --transformer-impl local --use-legacy-models --micro-batch-size 1 --global-batch-size 64 --train-iters 50 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --bf16 --use-distributed-optimizer --disable-bias-linear --attention-dropout 0 --hidden-dropout 0 --swiglu --lr 3.0e-5 --lr-decay-style cosine --min-lr 3.0e-6 --lr-warmup-iters 1 --ckpt-format torch --ddp-average-in-collective --overlap-grad-reduce --use-flash-attn-cutlass --sequence-parallel --tensor-model-parallel-size 1 --pipeline-model-parallel-size 2 --data-path /models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document --seq-length 4096 --split 949,50,1 --tokenizer-type Llama2Tokenizer --tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model --log-interval 1 --log-throughput --save-interval 1000 --eval-interval 1000 --eval-iters 3 --tensorboard-dir ./tmp_7b --rank 7 --world-size 8 --local-rank 7 --dist-url tcp://localhost:34577
/megatron-lm/megatron/training/arguments.py:601: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
[WARNING | megatron.core.rerun_state_machine]: RerunStateMachine initialized in mode disabled
/megatron-lm/megatron/training/arguments.py:601: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
[WARNING | megatron.core.rerun_state_machine]: RerunStateMachine initialized in mode disabled
/megatron-lm/megatron/training/arguments.py:601: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
[WARNING | megatron.core.rerun_state_machine]: RerunStateMachine initialized in mode disabled
/megatron-lm/megatron/training/arguments.py:601: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
/megatron-lm/megatron/training/arguments.py:601: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
/megatron-lm/megatron/training/arguments.py:601: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
[WARNING | megatron.core.rerun_state_machine]: RerunStateMachine initialized in mode disabled
/megatron-lm/megatron/training/arguments.py:601: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
using world size: 8, data-parallel size: 4, context-parallel size: 1, hierarchical context-parallel sizes: Nonetensor-model-parallel size: 1, encoder-tensor-model-parallel size: 0, pipeline-model-parallel size: 2, encoder-pipeline-model-parallel size: 0
WARNING: overriding default arguments for tokenizer_type:GPT2BPETokenizer with tokenizer_type:Llama2Tokenizer
WARNING: Setting args.overlap_p2p_comm and args.align_param_gather to False since non-interleaved schedule does not support overlapping p2p communication and aligned param AG
accumulate and all-reduce gradients in fp32 for bfloat16 data type.
using torch.bfloat16 for parameters ...
/megatron-lm/megatron/training/arguments.py:601: UserWarning: Disabling sequence parallelism because tensor model parallelism is disabled
warnings.warn("Disabling sequence parallelism because tensor model parallelism is disabled")
------------------------ arguments ------------------------
accumulate_allreduce_grads_in_fp32 .............. True
adam_beta1 ...................................... 0.9
adam_beta2 ...................................... 0.95
adam_eps ........................................ 1e-08
add_bias_linear ................................. False
add_position_embedding .......................... True
add_qkv_bias .................................... False
adlr_autoresume ................................. False
adlr_autoresume_interval ........................ 1000
align_grad_reduce ............................... True
align_param_gather .............................. False
app_tag_run_name ................................ None
app_tag_run_version ............................. 0.0.0
apply_layernorm_1p .............................. False
apply_query_key_layer_scaling ................... False
apply_residual_connection_post_layernorm ........ False
apply_rope_fusion ............................... True
async_save ...................................... None
async_tensor_model_parallel_allreduce ........... True
attention_backend ............................... AttnBackend.auto
attention_dropout ............................... 0.0
attention_softmax_in_fp32 ....................... False
auto_detect_ckpt_format ......................... False
barrier_with_L1_time ............................ True
bert_binary_head ................................ True
bert_embedder_type .............................. megatron
bert_load ....................................... None
bf16 ............................................ True
bias_dropout_fusion ............................. True
bias_gelu_fusion ................................ False
bias_swiglu_fusion .............................. True
biencoder_projection_dim ........................ 0
biencoder_shared_query_context_model ............ False
block_data_path ................................. None
calculate_per_token_loss ........................ False
check_for_nan_in_loss_and_grad .................. True
check_for_spiky_loss ............................ False
check_weight_hash_across_dp_replicas_interval ... None
ckpt_assume_constant_structure .................. False
ckpt_convert_format ............................. None
ckpt_convert_save ............................... None
ckpt_convert_update_legacy_dist_opt_format ...... False
ckpt_format ..................................... torch
ckpt_fully_parallel_load ........................ False
ckpt_fully_parallel_save ........................ True
ckpt_fully_parallel_save_deprecated ............. False
ckpt_step ....................................... None
classes_fraction ................................ 1.0
clip_grad ....................................... 1.0
clone_scatter_output_in_embedding ............... True
config_logger_dir ...............................
consumed_train_samples .......................... 0
consumed_valid_samples .......................... 0
context_parallel_size ........................... 1
cp_comm_type .................................... ['p2p']
create_attention_mask_in_dataloader ............. True
cross_entropy_loss_fusion ....................... False
data_args_path .................................. None
data_cache_path ................................. None
data_parallel_random_init ....................... False
data_parallel_size .............................. 4
data_path ....................................... ['/models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document']
data_per_class_fraction ......................... 1.0
data_sharding ................................... True
dataloader_type ................................. single
ddp_average_in_collective ....................... True
ddp_bucket_size ................................. None
decoder_first_pipeline_num_layers ............... None
decoder_last_pipeline_num_layers ................ None
decoder_num_layers .............................. None
decoder_seq_length .............................. None
decoupled_lr .................................... None
decoupled_min_lr ................................ None
decrease_batch_size_if_needed ................... False
defer_embedding_wgrad_compute ................... False
deprecated_use_mcore_models ..................... False
deterministic_mode .............................. False
dino_bottleneck_size ............................ 256
dino_freeze_last_layer .......................... 1
dino_head_hidden_size ........................... 2048
dino_local_crops_number ......................... 10
dino_local_img_size ............................. 96
dino_norm_last_layer ............................ False
dino_teacher_temp ............................... 0.07
dino_warmup_teacher_temp ........................ 0.04
dino_warmup_teacher_temp_epochs ................. 30
disable_straggler_on_startup .................... False
dist_ckpt_format_deprecated ..................... None
dist_ckpt_strictness ............................ assume_ok_unexpected
dist_url ........................................ tcp://localhost:34577
distribute_saved_activations .................... False
distributed_backend ............................. nccl
distributed_timeout_minutes ..................... 10
embedding_path .................................. None
empty_unused_memory_level ....................... 0
enable_ft_package ............................... False
enable_one_logger ............................... True
encoder_num_layers .............................. 32
encoder_pipeline_model_parallel_size ............ 0
encoder_seq_length .............................. 4096
encoder_tensor_model_parallel_size .............. 0
end_weight_decay ................................ 0.1
eod_mask_loss ................................... False
error_injection_rate ............................ 0
error_injection_type ............................ transient_error
eval_interval ................................... 1000
eval_iters ...................................... 3
evidence_data_path .............................. None
exit_duration_in_mins ........................... None
exit_interval ................................... None
exit_on_missing_checkpoint ...................... False
exit_signal_handler ............................. False
exp_avg_dtype ................................... torch.float32
exp_avg_sq_dtype ................................ torch.float32
expert_model_parallel_size ...................... 1
expert_tensor_parallel_size ..................... 1
ffn_hidden_size ................................. 11008
finetune ........................................ False
flash_decode .................................... False
fp16 ............................................ False
fp16_lm_cross_entropy ........................... False
fp32_residual_connection ........................ False
fp8 ............................................. None
fp8_amax_compute_algo ........................... most_recent
fp8_amax_history_len ............................ 1
fp8_interval .................................... 1
fp8_margin ...................................... 0
fp8_param_gather ................................ False
fp8_wgrad ....................................... True
global_batch_size ............................... 64
gradient_accumulation_fusion .................... True
group_query_attention ........................... False
head_lr_mult .................................... 1.0
hidden_dropout .................................. 0.0
hidden_size ..................................... 4096
hierarchical_context_parallel_sizes ............. None
hybrid_attention_ratio .......................... 0.0
hybrid_mlp_ratio ................................ 0.0
hybrid_override_pattern ......................... None
hysteresis ...................................... 2
ict_head_size ................................... None
ict_load ........................................ None
img_h ........................................... 224
img_w ........................................... 224
indexer_batch_size .............................. 128
indexer_log_interval ............................ 1000
inference_batch_times_seqlen_threshold .......... -1
inference_max_seq_length ........................ 2560
init_method_std ................................. 0.006
init_method_xavier_uniform ...................... False
initial_loss_scale .............................. 4294967296
iter_per_epoch .................................. 1250
kv_channels ..................................... 128
kv_lora_rank .................................... 32
lazy_mpu_init ................................... None
load ............................................ None
local_rank ...................................... 0
log_interval .................................... 1
log_loss_scale_to_tensorboard ................... True
log_memory_to_tensorboard ....................... False
log_num_zeros_in_grad ........................... False
log_params_norm ................................. False
log_progress .................................... False
log_straggler ................................... False
log_throughput .................................. True
log_timers_to_tensorboard ....................... False
log_validation_ppl_to_tensorboard ............... False
log_world_size_to_tensorboard ................... False
logging_level ................................... None
loss_scale ...................................... None
loss_scale_window ............................... 1000
lr .............................................. 3e-05
lr_decay_iters .................................. None
lr_decay_samples ................................ None
lr_decay_style .................................. cosine
lr_warmup_fraction .............................. None
lr_warmup_init .................................. 0.0
lr_warmup_iters ................................. 1
lr_warmup_samples ............................... 0
lr_wsd_decay_iters .............................. None
lr_wsd_decay_samples ............................ None
lr_wsd_decay_style .............................. exponential
main_grads_dtype ................................ torch.float32
main_params_dtype ............................... torch.float32
make_vocab_size_divisible_by .................... 128
manual_gc ....................................... False
manual_gc_eval .................................. True
manual_gc_interval .............................. 0
mask_factor ..................................... 1.0
mask_prob ....................................... 0.15
mask_type ....................................... random
masked_softmax_fusion ........................... True
max_position_embeddings ......................... 4096
max_tokens_to_oom ............................... 12000
memory_snapshot_path ............................ snapshot.pickle
merge_file ...................................... None
micro_batch_size ................................ 1
microbatch_group_size_per_vp_stage .............. None
min_loss_scale .................................. 1.0
min_lr .......................................... 3e-06
mmap_bin_files .................................. True
mock_data ....................................... False
moe_aux_loss_coeff .............................. 0.0
moe_expert_capacity_factor ...................... None
moe_extended_tp ................................. False
moe_ffn_hidden_size ............................. 11008
moe_grouped_gemm ................................ False
moe_input_jitter_eps ............................ None
moe_layer_freq .................................. 1
moe_layer_recompute ............................. False
moe_pad_expert_input_to_capacity ................ False
moe_per_layer_logging ........................... False
moe_router_load_balancing_type .................. aux_loss
moe_router_pre_softmax .......................... False
moe_router_topk ................................. 2
moe_router_topk_limited_devices ................. None
moe_router_topk_scaling_factor .................. None
moe_shared_expert_intermediate_size ............. None
moe_shared_expert_overlap ....................... False
moe_token_dispatcher_type ....................... allgather
moe_token_drop_policy ........................... probs
moe_use_legacy_grouped_gemm ..................... False
moe_use_upcycling ............................... False
moe_z_loss_coeff ................................ None
multi_latent_attention .......................... False
nccl_communicator_config_path ................... None
no_load_optim ................................... None
no_load_rng ..................................... None
no_persist_layer_norm ........................... False
no_save_optim ................................... None
no_save_rng ..................................... None
non_persistent_ckpt_type ........................ None
non_persistent_global_ckpt_dir .................. None
non_persistent_local_ckpt_algo .................. fully_parallel
non_persistent_local_ckpt_dir ................... None
non_persistent_save_interval .................... None
norm_epsilon .................................... 1e-05
normalization ................................... RMSNorm
num_attention_heads ............................. 32
num_channels .................................... 3
num_classes ..................................... 1000
num_dataset_builder_threads ..................... 1
num_distributed_optimizer_instances ............. 1
num_experts ..................................... None
num_layers ...................................... 32
num_layers_per_virtual_pipeline_stage ........... None
num_query_groups ................................ 1
num_workers ..................................... 2
one_logger_async ................................ False
one_logger_project .............................. megatron-lm
one_logger_run_name ............................. None
onnx_safe ....................................... None
openai_gelu ..................................... False
optimizer ....................................... adam
output_bert_embeddings .......................... False
overlap_grad_reduce ............................. True
overlap_p2p_comm ................................ False
overlap_p2p_comm_warmup_flush ................... False
overlap_param_gather ............................ False
overlap_param_gather_with_optimizer_step ........ False
override_opt_param_scheduler .................... False
params_dtype .................................... torch.bfloat16
patch_dim ....................................... 16
per_split_data_args_path ........................ None
perform_initialization .......................... True
pipeline_model_parallel_size .................... 2
pipeline_model_parallel_split_rank .............. None
position_embedding_type ......................... rope
pretrained_checkpoint ........................... None
profile ......................................... False
profile_dir ..................................... ./
profile_ranks ................................... [0]
profile_step_end ................................ 12
profile_step_start .............................. 10
q_lora_rank ..................................... None
qk_head_dim ..................................... 128
qk_layernorm .................................... False
qk_pos_emb_head_dim ............................. 64
query_in_block_prob ............................. 0.1
rampup_batch_size ............................... None
rank ............................................ 0
recompute_granularity ........................... None
recompute_method ................................ None
recompute_num_layers ............................ None
record_memory_history ........................... False
renormalize_blend_weights ....................... False
rerun_mode ...................................... disabled
reset_attention_mask ............................ False
reset_position_ids .............................. False
retriever_report_topk_accuracies ................ []
retriever_score_scaling ......................... False
retriever_seq_length ............................ 256
retro_add_retriever ............................. False
retro_attention_gate ............................ 1
retro_cyclic_train_iters ........................ None
retro_encoder_attention_dropout ................. 0.1
retro_encoder_hidden_dropout .................... 0.1
retro_encoder_layers ............................ 2
retro_num_neighbors ............................. 2
retro_num_retrieved_chunks ...................... 2
retro_project_dir ............................... None
retro_verify_neighbor_count ..................... True
rotary_base ..................................... 10000
rotary_interleaved .............................. False
rotary_percent .................................. 1.0
rotary_scaling_factor ........................... 1.0
rotary_seq_len_interpolation_factor ............. None
s3_cache_path ................................... None
sample_rate ..................................... 1.0
save ............................................ None
save_interval ................................... 1000
scatter_gather_tensors_in_pipeline .............. True
seed ............................................ 1234
seq_length ...................................... 4096
sequence_parallel ............................... False
sgd_momentum .................................... 0.9
short_seq_prob .................................. 0.1
skip_train ...................................... False
skipped_train_samples ........................... 0
spec ............................................ None
split ........................................... 949,50,1
squared_relu .................................... False
standalone_embedding_stage ...................... False
start_weight_decay .............................. 0.1
straggler_ctrlr_port ............................ 65535
straggler_minmax_count .......................... 1
swiglu .......................................... True
swin_backbone_type .............................. tiny
tensor_model_parallel_size ...................... 1
tensorboard_dir ................................. ./tmp_7b
tensorboard_log_interval ........................ 1
tensorboard_queue_size .......................... 1000
test_data_path .................................. None
test_mode ....................................... False
tiktoken_num_special_tokens ..................... 1000
tiktoken_pattern ................................ None
tiktoken_special_tokens ......................... None
timing_log_level ................................ 0
timing_log_option ............................... minmax
titles_data_path ................................ None
tokenizer_model ................................. /models1/Llama-2-7b-chat-hf/tokenizer.model
tokenizer_type .................................. Llama2Tokenizer
tp_comm_bootstrap_backend ....................... nccl
tp_comm_bulk_dgrad .............................. True
tp_comm_bulk_wgrad .............................. True
tp_comm_overlap ................................. False
tp_comm_overlap_ag .............................. True
tp_comm_overlap_cfg ............................. None
tp_comm_overlap_rs .............................. True
tp_comm_overlap_rs_dgrad ........................ False
tp_comm_split_ag ................................ True
tp_comm_split_rs ................................ True
train_data_path ................................. None
train_iters ..................................... 50
train_samples ................................... None
train_sync_interval ............................. None
transformer_impl ................................ local
transformer_pipeline_model_parallel_size ........ 2
untie_embeddings_and_output_weights ............. True
use_checkpoint_args ............................. False
use_checkpoint_opt_param_scheduler .............. False
use_cpu_initialization .......................... None
use_dist_ckpt ................................... False
use_dist_ckpt_deprecated ........................ False
use_distributed_optimizer ....................... True
use_flash_attn .................................. True
use_flash_attn_cutlass .......................... True
use_flash_attn_torch ............................ False
use_flash_attn_triton ........................... False
use_hip_profiler ................................ False
use_legacy_models ............................... True
use_mp_args_from_checkpoint_args ................ False
use_one_sent_docs ............................... False
use_precision_aware_optimizer ................... False
use_pytorch_profiler ............................ False
use_ring_exchange_p2p ........................... False
use_rope_scaling ................................ False
use_rotary_position_embeddings .................. False
use_tokenizer_model_from_checkpoint_args ........ True
use_torch_fsdp2 ................................. False
use_tp_pp_dp_mapping ............................ False
v_head_dim ...................................... 128
valid_data_path ................................. None
variable_seq_lengths ............................ False
virtual_pipeline_model_parallel_size ............ None
vision_backbone_type ............................ vit
vision_pretraining .............................. False
vision_pretraining_type ......................... classify
vocab_extra_ids ................................. 0
vocab_file ...................................... None
vocab_size ...................................... None
wandb_exp_name ..................................
wandb_project ...................................
wandb_save_dir ..................................
weight_decay .................................... 0.1
weight_decay_incr_style ......................... constant
wgrad_deferral_limit ............................ 0
world_size ...................................... 8
yaml_cfg ........................................ None
-------------------- end of arguments ---------------------
> building Llama2Tokenizer tokenizer ...
[WARNING | megatron.core.rerun_state_machine]: RerunStateMachine initialized in mode disabled
[WARNING | megatron.core.rerun_state_machine]: RerunStateMachine initialized in mode disabled
> padded vocab (size: 32000) with 0 dummy tokens (new size: 32000)
[WARNING | megatron.core.rerun_state_machine]: RerunStateMachine initialized in mode disabled
> initializing torch distributed ...
2025-10-30 15:48:22.868274: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/tensorboard/compat/__init__.py", line 42, in tf
from tensorboard.compat import notf # noqa: F401
ImportError: cannot import name 'notf' from 'tensorboard.compat' (/usr/local/lib/python3.10/dist-packages/tensorboard/compat/__init__.py)
During handling of the above exception, another exception occurred:
AttributeError: 'MessageFactory' object has no attribute 'GetPrototype'
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/tensorboard/compat/__init__.py", line 42, in tf
from tensorboard.compat import notf # noqa: F401
ImportError: cannot import name 'notf' from 'tensorboard.compat' (/usr/local/lib/python3.10/dist-packages/tensorboard/compat/__init__.py)
During handling of the above exception, another exception occurred:
AttributeError: 'MessageFactory' object has no attribute 'GetPrototype'
> setting tensorboard ...
WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
[WARNING | megatron.core.rerun_state_machine]: RerunStateMachine initialized in mode disabled
> initialized tensor model parallel with size 1
> initialized pipeline model parallel with size 2
> setting random seeds to 1234 ...
> compiling dataset index builder ...
make: Entering directory '/megatron-lm/megatron/core/datasets'
make: Nothing to be done for 'default'.
make: Leaving directory '/megatron-lm/megatron/core/datasets'
>>> done with dataset index builder. Compilation time: 0.032 seconds
> compiling and loading fused kernels ...
>>> done with compiling and loading fused kernels. Compilation time: 1.511 seconds
time to initialize megatron (seconds): 5.644
[after megatron is initialized] datetime: 2025-10-30 15:48:27
building GPT model ...
> number of parameters on (tensor, pipeline) model parallel rank (0, 1): 3369209856
GPTModel(
(language_model): TransformerLanguageModel(
(embedding): Embedding(
(word_embeddings): VocabParallelEmbedding()
(embedding_dropout): Dropout(p=0.0, inplace=False)
)
(rotary_pos_emb): RotaryEmbedding()
(encoder): ParallelTransformer(
(layers): ModuleList(
(0-15): 16 x ParallelTransformerLayer(
(input_norm): RMSNorm()
(self_attention): ParallelAttention(
(query_key_value): ColumnParallelLinear()
(core_attention): CoreAttention(
(scale_mask_softmax): FusedScaleMaskSoftmax()
(attention_dropout): Dropout(p=0.0, inplace=False)
)
(core_attention_flash): FlashSelfAttention()
(dense): RowParallelLinear()
)
(post_attention_norm): RMSNorm()
(mlp): ParallelMLP(
(dense_h_to_4h): ColumnParallelLinear()
(dense_4h_to_h): RowParallelLinear()
)
)
)
)
)
)
> number of parameters on (tensor, pipeline) model parallel rank (0, 0): 3369205760
[after model, optimizer, and learning rate scheduler are built] datetime: 2025-10-30 15:48:27
> building train, validation, and test datasets ...
> datasets target sizes (minimum size):
train: 3200
validation: 192
test: 192
> building train, validation, and test datasets for GPT ...
> finished creating GPT datasets ...
[after dataloaders are built] datetime: 2025-10-30 15:48:27
done with setup ...
training ...
(min, max) time across ranks (ms):
model-and-optimizer-setup ......................: (199.88, 204.83)
train/valid/test-data-iterators-setup ..........: (360.81, 415.69)
[before the start of training step] datetime: 2025-10-30 15:48:27
Number of parameters in transformer layers in billions: 6.48
Number of parameters in embedding layers in billions: 0.26
Total number of parameters in billions: 6.74
Number of parameters in most loaded shard in billions: 3.3693
[2025-10-30 15:49:58] iteration 1/ 50 | consumed samples: 64 | elapsed time per iteration (ms): 90286.8 | throughput per GPU (TFLOP/s/GPU): 16.7 | learning rate: 3.000000E-05 | global batch size: 64 | lm loss: 1.045589E+01 | loss scale: 1.0 | grad norm: 270.234 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
Number of parameters in other shards in billions: 3.2383
Theoretical memory footprints: weight and optimizer=28919.29 MB
[Rank 4] (after 1 iterations) memory (MB) | allocated: 28987.10498046875 | max allocated: 32813.6396484375 | reserved: 36168.0 | max reserved: 36168.0
[Rank 0] (after 1 iterations) memory (MB) | allocated: 28920.36181640625 | max allocated: 39419.125 | reserved: 40814.0 | max reserved: 40814.0
Could not open /var/log/hylog/.
[2025-10-30 15:50:38] iteration 2/ 50 | consumed samples: 128 | elapsed time per iteration (ms): 40295.7 | throughput per GPU (TFLOP/s/GPU): 37.5 | learning rate: 2.997226E-05 | global batch size: 64 | lm loss: 1.046932E+01 | loss scale: 1.0 | grad norm: 242.090 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 15:51:18] iteration 3/ 50 | consumed samples: 192 | elapsed time per iteration (ms): 39819.8 | throughput per GPU (TFLOP/s/GPU): 37.9 | learning rate: 2.988916E-05 | global batch size: 64 | lm loss: 8.424404E+00 | loss scale: 1.0 | grad norm: 502.275 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 15:51:57] iteration 4/ 50 | consumed samples: 256 | elapsed time per iteration (ms): 39600.0 | throughput per GPU (TFLOP/s/GPU): 38.1 | learning rate: 2.975105E-05 | global batch size: 64 | lm loss: 1.290415E+01 | loss scale: 1.0 | grad norm: 174.010 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 15:52:37] iteration 5/ 50 | consumed samples: 320 | elapsed time per iteration (ms): 39826.9 | throughput per GPU (TFLOP/s/GPU): 37.9 | learning rate: 2.955848E-05 | global batch size: 64 | lm loss: 9.753544E+00 | loss scale: 1.0 | grad norm: 47.956 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 15:53:17] iteration 6/ 50 | consumed samples: 384 | elapsed time per iteration (ms): 39826.8 | throughput per GPU (TFLOP/s/GPU): 37.9 | learning rate: 2.931225E-05 | global batch size: 64 | lm loss: 9.120786E+00 | loss scale: 1.0 | grad norm: 165.685 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 15:53:57] iteration 7/ 50 | consumed samples: 448 | elapsed time per iteration (ms): 39536.7 | throughput per GPU (TFLOP/s/GPU): 38.2 | learning rate: 2.901338E-05 | global batch size: 64 | lm loss: 8.215652E+00 | loss scale: 1.0 | grad norm: 40.534 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 15:54:37] iteration 8/ 50 | consumed samples: 512 | elapsed time per iteration (ms): 40029.6 | throughput per GPU (TFLOP/s/GPU): 37.7 | learning rate: 2.866308E-05 | global batch size: 64 | lm loss: 7.065186E+00 | loss scale: 1.0 | grad norm: 10.479 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 15:55:16] iteration 9/ 50 | consumed samples: 576 | elapsed time per iteration (ms): 39653.0 | throughput per GPU (TFLOP/s/GPU): 38.1 | learning rate: 2.826280E-05 | global batch size: 64 | lm loss: 7.098128E+00 | loss scale: 1.0 | grad norm: 8.814 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 15:55:56] iteration 10/ 50 | consumed samples: 640 | elapsed time per iteration (ms): 39674.7 | throughput per GPU (TFLOP/s/GPU): 38.1 | learning rate: 2.781419E-05 | global batch size: 64 | lm loss: 6.366463E+00 | loss scale: 1.0 | grad norm: 6.373 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 15:56:36] iteration 11/ 50 | consumed samples: 704 | elapsed time per iteration (ms): 40139.3 | throughput per GPU (TFLOP/s/GPU): 37.6 | learning rate: 2.731908E-05 | global batch size: 64 | lm loss: 6.430417E+00 | loss scale: 1.0 | grad norm: 6.818 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 15:57:16] iteration 12/ 50 | consumed samples: 768 | elapsed time per iteration (ms): 39490.8 | throughput per GPU (TFLOP/s/GPU): 38.2 | learning rate: 2.677952E-05 | global batch size: 64 | lm loss: 6.327631E+00 | loss scale: 1.0 | grad norm: 3.020 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 15:57:55] iteration 13/ 50 | consumed samples: 832 | elapsed time per iteration (ms): 39506.9 | throughput per GPU (TFLOP/s/GPU): 38.2 | learning rate: 2.619772E-05 | global batch size: 64 | lm loss: 6.092177E+00 | loss scale: 1.0 | grad norm: 2.616 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 15:58:35] iteration 14/ 50 | consumed samples: 896 | elapsed time per iteration (ms): 39718.0 | throughput per GPU (TFLOP/s/GPU): 38.0 | learning rate: 2.557606E-05 | global batch size: 64 | lm loss: 6.129852E+00 | loss scale: 1.0 | grad norm: 4.508 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 15:59:14] iteration 15/ 50 | consumed samples: 960 | elapsed time per iteration (ms): 39658.4 | throughput per GPU (TFLOP/s/GPU): 38.1 | learning rate: 2.491711E-05 | global batch size: 64 | lm loss: 6.379290E+00 | loss scale: 1.0 | grad norm: 15.828 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 15:59:54] iteration 16/ 50 | consumed samples: 1024 | elapsed time per iteration (ms): 39884.8 | throughput per GPU (TFLOP/s/GPU): 37.9 | learning rate: 2.422357E-05 | global batch size: 64 | lm loss: 6.207567E+00 | loss scale: 1.0 | grad norm: 3.418 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:00:34] iteration 17/ 50 | consumed samples: 1088 | elapsed time per iteration (ms): 39599.3 | throughput per GPU (TFLOP/s/GPU): 38.1 | learning rate: 2.349830E-05 | global batch size: 64 | lm loss: 6.430919E+00 | loss scale: 1.0 | grad norm: 18.031 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:01:14] iteration 18/ 50 | consumed samples: 1152 | elapsed time per iteration (ms): 39926.3 | throughput per GPU (TFLOP/s/GPU): 37.8 | learning rate: 2.274427E-05 | global batch size: 64 | lm loss: 6.162337E+00 | loss scale: 1.0 | grad norm: 14.185 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:01:54] iteration 19/ 50 | consumed samples: 1216 | elapsed time per iteration (ms): 39747.6 | throughput per GPU (TFLOP/s/GPU): 38.0 | learning rate: 2.196458E-05 | global batch size: 64 | lm loss: 5.844732E+00 | loss scale: 1.0 | grad norm: 3.080 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:02:34] iteration 20/ 50 | consumed samples: 1280 | elapsed time per iteration (ms): 39922.5 | throughput per GPU (TFLOP/s/GPU): 37.8 | learning rate: 2.116243E-05 | global batch size: 64 | lm loss: 5.706470E+00 | loss scale: 1.0 | grad norm: 7.213 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:03:13] iteration 21/ 50 | consumed samples: 1344 | elapsed time per iteration (ms): 39735.6 | throughput per GPU (TFLOP/s/GPU): 38.0 | learning rate: 2.034112E-05 | global batch size: 64 | lm loss: 5.828917E+00 | loss scale: 1.0 | grad norm: 5.298 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:03:53] iteration 22/ 50 | consumed samples: 1408 | elapsed time per iteration (ms): 39615.3 | throughput per GPU (TFLOP/s/GPU): 38.1 | learning rate: 1.950403E-05 | global batch size: 64 | lm loss: 5.981213E+00 | loss scale: 1.0 | grad norm: 3.407 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:04:38] iteration 23/ 50 | consumed samples: 1472 | elapsed time per iteration (ms): 44924.4 | throughput per GPU (TFLOP/s/GPU): 33.6 | learning rate: 1.865460E-05 | global batch size: 64 | lm loss: 5.460212E+00 | loss scale: 1.0 | grad norm: 4.123 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:05:44] iteration 24/ 50 | consumed samples: 1536 | elapsed time per iteration (ms): 65925.0 | throughput per GPU (TFLOP/s/GPU): 22.9 | learning rate: 1.779631E-05 | global batch size: 64 | lm loss: 5.226260E+00 | loss scale: 1.0 | grad norm: 5.918 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:06:24] iteration 25/ 50 | consumed samples: 1600 | elapsed time per iteration (ms): 40182.0 | throughput per GPU (TFLOP/s/GPU): 37.6 | learning rate: 1.693270E-05 | global batch size: 64 | lm loss: 5.305700E+00 | loss scale: 1.0 | grad norm: 1.561 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:07:04] iteration 26/ 50 | consumed samples: 1664 | elapsed time per iteration (ms): 39762.6 | throughput per GPU (TFLOP/s/GPU): 38.0 | learning rate: 1.606730E-05 | global batch size: 64 | lm loss: 5.153278E+00 | loss scale: 1.0 | grad norm: 2.196 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:07:44] iteration 27/ 50 | consumed samples: 1728 | elapsed time per iteration (ms): 39921.2 | throughput per GPU (TFLOP/s/GPU): 37.8 | learning rate: 1.520369E-05 | global batch size: 64 | lm loss: 5.105300E+00 | loss scale: 1.0 | grad norm: 1.382 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:08:23] iteration 28/ 50 | consumed samples: 1792 | elapsed time per iteration (ms): 39815.2 | throughput per GPU (TFLOP/s/GPU): 37.9 | learning rate: 1.434540E-05 | global batch size: 64 | lm loss: 4.925309E+00 | loss scale: 1.0 | grad norm: 1.777 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:09:03] iteration 29/ 50 | consumed samples: 1856 | elapsed time per iteration (ms): 39752.5 | throughput per GPU (TFLOP/s/GPU): 38.0 | learning rate: 1.349597E-05 | global batch size: 64 | lm loss: 5.181439E+00 | loss scale: 1.0 | grad norm: 1.845 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:09:43] iteration 30/ 50 | consumed samples: 1920 | elapsed time per iteration (ms): 39643.3 | throughput per GPU (TFLOP/s/GPU): 38.1 | learning rate: 1.265888E-05 | global batch size: 64 | lm loss: 5.208538E+00 | loss scale: 1.0 | grad norm: 2.059 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:10:23] iteration 31/ 50 | consumed samples: 1984 | elapsed time per iteration (ms): 39988.1 | throughput per GPU (TFLOP/s/GPU): 37.8 | learning rate: 1.183757E-05 | global batch size: 64 | lm loss: 4.890507E+00 | loss scale: 1.0 | grad norm: 1.327 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:11:03] iteration 32/ 50 | consumed samples: 2048 | elapsed time per iteration (ms): 40034.3 | throughput per GPU (TFLOP/s/GPU): 37.7 | learning rate: 1.103542E-05 | global batch size: 64 | lm loss: 4.993505E+00 | loss scale: 1.0 | grad norm: 1.328 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:11:43] iteration 33/ 50 | consumed samples: 2112 | elapsed time per iteration (ms): 39777.8 | throughput per GPU (TFLOP/s/GPU): 38.0 | learning rate: 1.025573E-05 | global batch size: 64 | lm loss: 5.216469E+00 | loss scale: 1.0 | grad norm: 1.112 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:12:22] iteration 34/ 50 | consumed samples: 2176 | elapsed time per iteration (ms): 39744.7 | throughput per GPU (TFLOP/s/GPU): 38.0 | learning rate: 9.501700E-06 | global batch size: 64 | lm loss: 5.064697E+00 | loss scale: 1.0 | grad norm: 1.134 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:13:02] iteration 35/ 50 | consumed samples: 2240 | elapsed time per iteration (ms): 39795.0 | throughput per GPU (TFLOP/s/GPU): 37.9 | learning rate: 8.776425E-06 | global batch size: 64 | lm loss: 4.957899E+00 | loss scale: 1.0 | grad norm: 0.992 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:13:42] iteration 36/ 50 | consumed samples: 2304 | elapsed time per iteration (ms): 39734.5 | throughput per GPU (TFLOP/s/GPU): 38.0 | learning rate: 8.082888E-06 | global batch size: 64 | lm loss: 5.244042E+00 | loss scale: 1.0 | grad norm: 1.144 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:14:22] iteration 37/ 50 | consumed samples: 2368 | elapsed time per iteration (ms): 39786.1 | throughput per GPU (TFLOP/s/GPU): 38.0 | learning rate: 7.423938E-06 | global batch size: 64 | lm loss: 5.032987E+00 | loss scale: 1.0 | grad norm: 0.960 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:15:01] iteration 38/ 50 | consumed samples: 2432 | elapsed time per iteration (ms): 39439.9 | throughput per GPU (TFLOP/s/GPU): 38.3 | learning rate: 6.802284E-06 | global batch size: 64 | lm loss: 4.990102E+00 | loss scale: 1.0 | grad norm: 0.900 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:15:41] iteration 39/ 50 | consumed samples: 2496 | elapsed time per iteration (ms): 39764.0 | throughput per GPU (TFLOP/s/GPU): 38.0 | learning rate: 6.220479E-06 | global batch size: 64 | lm loss: 4.856393E+00 | loss scale: 1.0 | grad norm: 1.125 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:16:20] iteration 40/ 50 | consumed samples: 2560 | elapsed time per iteration (ms): 39466.0 | throughput per GPU (TFLOP/s/GPU): 38.3 | learning rate: 5.680916E-06 | global batch size: 64 | lm loss: 5.073430E+00 | loss scale: 1.0 | grad norm: 1.028 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:17:00] iteration 41/ 50 | consumed samples: 2624 | elapsed time per iteration (ms): 39639.2 | throughput per GPU (TFLOP/s/GPU): 38.1 | learning rate: 5.185811E-06 | global batch size: 64 | lm loss: 5.006877E+00 | loss scale: 1.0 | grad norm: 0.856 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:17:40] iteration 42/ 50 | consumed samples: 2688 | elapsed time per iteration (ms): 39736.3 | throughput per GPU (TFLOP/s/GPU): 38.0 | learning rate: 4.737197E-06 | global batch size: 64 | lm loss: 4.772885E+00 | loss scale: 1.0 | grad norm: 0.983 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:18:19] iteration 43/ 50 | consumed samples: 2752 | elapsed time per iteration (ms): 39481.7 | throughput per GPU (TFLOP/s/GPU): 38.2 | learning rate: 4.336920E-06 | global batch size: 64 | lm loss: 4.907492E+00 | loss scale: 1.0 | grad norm: 0.823 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:18:59] iteration 44/ 50 | consumed samples: 2816 | elapsed time per iteration (ms): 39949.9 | throughput per GPU (TFLOP/s/GPU): 37.8 | learning rate: 3.986624E-06 | global batch size: 64 | lm loss: 4.758832E+00 | loss scale: 1.0 | grad norm: 1.009 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:19:39] iteration 45/ 50 | consumed samples: 2880 | elapsed time per iteration (ms): 39817.1 | throughput per GPU (TFLOP/s/GPU): 37.9 | learning rate: 3.687747E-06 | global batch size: 64 | lm loss: 4.631381E+00 | loss scale: 1.0 | grad norm: 0.807 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:20:19] iteration 46/ 50 | consumed samples: 2944 | elapsed time per iteration (ms): 39566.5 | throughput per GPU (TFLOP/s/GPU): 38.2 | learning rate: 3.441519E-06 | global batch size: 64 | lm loss: 4.772638E+00 | loss scale: 1.0 | grad norm: 1.379 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:20:58] iteration 47/ 50 | consumed samples: 3008 | elapsed time per iteration (ms): 39582.4 | throughput per GPU (TFLOP/s/GPU): 38.2 | learning rate: 3.248951E-06 | global batch size: 64 | lm loss: 4.898998E+00 | loss scale: 1.0 | grad norm: 0.859 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:21:38] iteration 48/ 50 | consumed samples: 3072 | elapsed time per iteration (ms): 39651.1 | throughput per GPU (TFLOP/s/GPU): 38.1 | learning rate: 3.110835E-06 | global batch size: 64 | lm loss: 5.095502E+00 | loss scale: 1.0 | grad norm: 0.887 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:22:17] iteration 49/ 50 | consumed samples: 3136 | elapsed time per iteration (ms): 39759.3 | throughput per GPU (TFLOP/s/GPU): 38.0 | learning rate: 3.027737E-06 | global batch size: 64 | lm loss: 4.849247E+00 | loss scale: 1.0 | grad norm: 0.706 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-10-30 16:22:57] iteration 50/ 50 | consumed samples: 3200 | elapsed time per iteration (ms): 39629.9 | throughput per GPU (TFLOP/s/GPU): 38.1 | learning rate: 3.000000E-06 | global batch size: 64 | lm loss: 5.294223E+00 | loss scale: 1.0 | grad norm: 0.789 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[after training is done] datetime: 2025-10-30 16:22:57
[WARNING | megatron.core.rerun_state_machine]: Setting RerunStateMachine mode disabled
Evaluating on 192 samples
Evaluating iter 1/3
Evaluating iter 2/3
Evaluating iter 3/3
(min, max) time across ranks (ms):
evaluate .......................................: (16322.27, 16446.91)
----------------------------------------------------------------------------------------------------------------
[WARNING | megatron.core.rerun_state_machine]: Setting RerunStateMachine mode disabled
[WARNING | megatron.core.rerun_state_machine]: Setting RerunStateMachine mode disabled
validation loss at iteration 50 on validation set | lm loss value: 4.873671E+00 | lm loss PPL: 1.308002E+02 |
----------------------------------------------------------------------------------------------------------------
[WARNING | megatron.core.rerun_state_machine]: Setting RerunStateMachine mode disabled
Evaluating on 192 samples
Evaluating iter 1/3
Evaluating iter 2/3
Evaluating iter 3/3
(min, max) time across ranks (ms):
evaluate .......................................: (5993.56, 6118.05)
----------------------------------------------------------------------------------------------------------
validation loss at iteration 50 on test set | lm loss value: 5.035580E+00 | lm loss PPL: 1.537888E+02 |
----------------------------------------------------------------------------------------------------------
[WARNING | megatron.core.rerun_state_machine]: Setting RerunStateMachine mode disabled
[WARNING | megatron.core.rerun_state_machine]: Setting RerunStateMachine mode disabled
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1030 16:23:21.049338 936590 ProcessGroupNCCL.cpp:1197] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1030 16:23:21.139221 936594 ProcessGroupNCCL.cpp:1197] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1030 16:23:21.723956 936591 ProcessGroupNCCL.cpp:1197] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1030 16:23:21.764384 936596 ProcessGroupNCCL.cpp:1197] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1030 16:23:21.794720 936589 ProcessGroupNCCL.cpp:1197] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1030 16:23:22.382212 936597 ProcessGroupNCCL.cpp:1197] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1030 16:23:22.453176 936593 ProcessGroupNCCL.cpp:1197] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1030 16:23:52.474910 936599 ProcessGroupNCCL.cpp:1197] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
MANIFEST.in 0 → 100644
View file @ d444a97a
include megatron/core/requirements.txt
include megatron/core/README.md
recursive-include requirements *
README.md 0 → 100644
View file @ d444a97a
### 环境配置
1. 拉取镜像：docker pull image.sourcefind.cn:5000/dcu/admin/base/pytorch:2.4.1-ubuntu22.04-dtk25.04.1-py3.10
2. 安装基础依赖包
<pre>
pip install -r requirements.txt
</pre>
若使用 pip install 下载安装过慢，可添加源：-i https://pypi.tuna.tsinghua.edu.cn/simple/
### 下载openwebtext训练数据
https://huggingface.co/datasets/Skylion007/openwebtext
### 数据预处理
#### 1.将原始 tar 包转换为 openwebtext.jsonl
配置路径（根据你的实际路径修改）
SUBSETS_DIR = "/models/datasets/openwebtext/subsets" # 存放tar包的目录
OUTPUT_JSONL = "/models/datasets/openwebtext/openwebtext.jsonl" # 输出的jsonl文件
```
python convert_openwebtext_jsonl.py
```
输出openwebtext.jsonl文件，大小68G
#### 2.预处理为Llama-2 格式的数据
```
python tools/preprocess_data.py \
--input openwebtext.jsonl \
--output-prefix /models/datasets/openwebtext/openwebtext-llama-7b \
--tokenizer-type Llama2Tokenizer \
--tokenizer-model /path/to/llama2_7b_hf/tokenizer.model \
--workers 16 \
--append-eod
```
### 下载tokenizer文件
链接: https://www.modelscope.cn/models/shakechen/Llama-2-7b-hf/files
下载其中的tokenizer*文件
### Llama预训练
脚本: `Llama_pretraining.sh`
修改数据集与tokenizer路径
```shell
DATA_PATH="/models/datasets/openwebtext/openwebtext-llama-7b/openwebtext-llama-7b_text_document"
--tokenizer-model /models1/Llama-2-7b-chat-hf/tokenizer.model
```
- 单机8卡训练
```shell
mpirun --allow-run-as-root -np 8 Llama_pretraining.sh localhost >& Llama_pretraining.log
```
在`Llama_pretraining.log`中查看训练日志
README.md.origin 0 → 100644
View file @ d444a97a
<div align="center">
Megatron-LM & Megatron-Core
===========================
<h4>GPU optimized techniques for training transformer models at-scale</h4>
[![Documentation](https://img.shields.io/badge/docs-latest-brightgreen.svg?style=flat)](https://docs.nvidia.com/megatron-core/developer-guide/latest/index.html)
[![version](https://img.shields.io/badge/release-0.5.0-green)](./setup.py)
[![license](https://img.shields.io/badge/license-OpenBSD-blue)](./LICENSE)
<div align="left">
# Latest News
- **[2024/7]** Megatron-Core v0.7 improves scalability and training resiliency and adds support for multimodal training ([blog](https://developer.nvidia.com/blog/train-generative-ai-models-more-efficiently-with-new-nvidia-megatron-core-functionalities/)).
- **[2024/6]** Megatron-Core added supports for Mamba-based models. Check out our paper [An Empirical Study of Mamba-based Language Models](https://arxiv.org/pdf/2406.07887) and [code example](https://github.com/NVIDIA/Megatron-LM/tree/ssm/examples/mamba).
- **[2024/1 Announcement]** NVIDIA has released the core capabilities in **Megatron-LM** into [**Megatron-Core**](https://github.com/NVIDIA/Megatron-LM/tree/main/megatron/core) in this repository. Megatron-Core expands upon Megatron-LM's GPU-optimized techniques with more cutting-edge innovations on system-level optimizations, featuring composable and modular APIs. Explore the [Megatron-Core intro](#megatron-core) for more details.
# Table of Contents
- [Megatron-LM \& Megatron-Core](#megatron-lm--megatron-core)
- [Latest News](#latest-news)
- [Table of Contents](#table-of-contents)
- [Megatron Overview](#megatron-overview)
- [Megatron-LM](#megatron-lm)
- [Megatron-Core](#megatron-core)
- [Training Speed and Scalability](#training-speed-and-scalability)
- [Setup](#setup)
- [Downloading Checkpoints](#downloading-checkpoints)
- [Usage](#usage)
- [Training](#training)
- [Data Preprocessing](#data-preprocessing)
- [BERT Pretraining](#bert-pretraining)
- [GPT Pretraining](#gpt-pretraining)
- [T5 Pretraining](#t5-pretraining)
- [Distributed Pretraining](#distributed-pretraining)
- [Activation Checkpointing and Recomputation](#activation-checkpointing-and-recomputation)
- [Distributed Optimizer](#distributed-optimizer)
- [FlashAttention](#flashattention)
- [GPT-3 Example](#gpt-3-example)
- [Retro and InstructRetro](#retro-and-instructretro)
- [Mamba-based Language Models](#mamba-based-language-models)
- [Mixture of Experts](#mixture-of-experts)
- [Key Features of MoE](#key-features-of-moe)
- [Evaluation and Tasks](#evaluation-and-tasks)
- [GPT Text Generation](#gpt-text-generation)
- [Detoxify GPT via Self-generation](#detoxify-gpt-via-self-generation)
- [GPT Evaluation](#gpt-evaluation)
- [WikiText Perplexity Evaluation](#wikitext-perplexity-evaluation)
- [LAMBADA Cloze Accuracy](#lambada-cloze-accuracy)
- [BERT Task Evaluation](#bert-task-evaluation)
- [RACE Evaluation](#race-evaluation)
- [MNLI Evaluation](#mnli-evaluation)
- [Llama-2 Inference and Finetuning](#llama-2-inference-and-finetuning)
- [Model Optimization and Deployment](#model-optimization-and-deployment)
- [Quantization and TensorRT-LLM Deployment](#quantization-and-tensorrt-llm-deployment)
- [Datasets](#datasets)
- [Collecting Wikipedia Training Data](#collecting-wikipedia-training-data)
- [Collecting GPT Webtext Data](#collecting-gpt-webtext-data)
- [Reproducibility](#reproducibility)
- [Projects Using Megatron](#projects-using-megatron)
# Megatron Overview
This repository comprises two essential components: **Megatron-LM** and **Megatron-Core**. Megatron-LM serves as a research-oriented framework leveraging Megatron-Core for large language model (LLM) training. Megatron-Core, on the other hand, is a library of GPU optimized training techniques that comes with formal product support including versioned APIs and regular releases. You can use Megatron-Core alongside Megatron-LM or [Nvidia NeMo Framework](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/nemo_megatron/mcore_customization.html) for an end-to-end and cloud-native solution. Alternatively, you can integrate Megatron-Core's building blocks into your preferred training framework.
## Megatron-LM
First introduced in 2019, Megatron ([1](https://arxiv.org/pdf/1909.08053.pdf), [2](https://arxiv.org/pdf/2104.04473.pdf), and [3](https://arxiv.org/pdf/2205.05198)) sparked a wave of innovation in the AI community, enabling researchers and developers to utilize the underpinnings of this library to further LLM advancements. Today, many of the most popular LLM developer frameworks have been inspired by and built directly leveraging the open-source Megatron-LM library, spurring a wave of foundation models and AI startups. Some of the most popular LLM frameworks built on top of Megatron-LM include [Colossal-AI](https://github.com/hpcaitech/ColossalAI), [HuggingFace Accelerate](https://github.com/huggingface/accelerate), and [NVIDIA NeMo Framework](https://www.nvidia.com/en-us/ai-data-science/generative-ai/nemo-framework/). A list of projects that have directly used Megatron can be found [here](#projects-using-megatron).
## Megatron-Core
Megatron-Core is an open-source PyTorch-based library that contains GPU-optimized techniques and cutting-edge system-level optimizations. It abstracts them into composable and modular APIs, allowing full flexibility for developers and model researchers to train custom transformers at-scale on NVIDIA accelerated computing infrastructure. This library is compatible with all NVIDIA Tensor Core GPUs, including FP8 acceleration support for [NVIDIA Hopper architectures](https://www.nvidia.com/en-us/data-center/technologies/hopper-architecture/).
Megatron-Core offers core building blocks such as attention mechanisms, transformer blocks and layers, normalization layers, and embedding techniques. Additional functionality like activation recomputation, distributed checkpointing is also natively built-in to the library. The building blocks and functionality are all GPU optimized, and can be built with advanced parallelization strategies for optimal training speed and stability on NVIDIA Accelerated Computing Infrastructure. Another key component of the Megatron-Core library includes advanced model parallelism techniques (tensor, sequence, pipeline, context, and MoE expert parallelism).
Megatron-Core can be used with [NVIDIA NeMo](https://www.nvidia.com/en-us/ai-data-science/products/nemo/), an enterprise-grade AI platform. Alternatively, you can explore Megatron-Core with the native PyTorch training loop [here](https://github.com/NVIDIA/Megatron-LM/tree/main/examples). Visit [Megatron-Core documentation](https://docs.nvidia.com/megatron-core/developer-guide/latest/index.html) to learn more.
# Training Speed and Scalability
Our codebase is capable of efficiently training large language models (i.e., models with hundreds of billions of parameters) with both model and data parallelism. To demonstrate how our software scales with multiple GPUs and model sizes, we consider GPT models ranging from 2 billion parameters to 462 billion parameters. All models use a vocabulary size of 131,072 and a sequence length of 4096. We vary hidden size, number of attention heads, and number of layers to arrive at a specific model size. As the model size increases, we also modestly increase batch size. Our experiments use up to 6144 [H100](https://www.nvidia.com/en-us/data-center/h100/) GPUs. We perform fine-grained overlapping of data-parallel (`--overlap-grad-reduce --overlap-param-gather`), tensor-parallel (`--tp-comm-overlap`) and pipeline-parallel communication (enabled by default) with computation to improve scalability. The reported throughputs are measured for end-to-end training and include all operations including data loading, optimizer steps, communication, and even logging. Note that we did not train these models to convergence.
![Model table](images/model_table.png)
Our weak scaled results show superlinear scaling (MFU increases from 41% for the smallest model considered to 47-48% for the largest models); this is because larger GEMMs have higher arithmetic intensity and are consequently more efficient to execute.
![Weak scaling](images/weak_scaling.png)
We also strong scaled the standard GPT-3 model (our version has slightly more than 175 billion parameters due to larger vocabulary size) from 96 H100 GPUs to 4608 GPUs, using the same batch size of 1152 sequences throughout. Communication becomes more exposed at larger scale, leading to a reduction in MFU from 47% to 42%.
![Strong scaling](images/strong_scaling.png)
# Setup
We strongly recommend using the latest release of [NGC's PyTorch container](https://ngc.nvidia.com/catalog/containers/nvidia:pytorch) with DGX nodes. If you can't use this for some reason, use the latest pytorch, cuda, nccl, and NVIDIA [APEX](https://github.com/NVIDIA/apex#quick-start) releases. Data preprocessing requires [NLTK](https://www.nltk.org/install.html), though this is not required for training, evaluation, or downstream tasks.
You can launch an instance of the PyTorch container and mount Megatron, your dataset, and checkpoints with the following Docker commands:
```
docker pull nvcr.io/nvidia/pytorch:xx.xx-py3
docker run --gpus all -it --rm -v /path/to/megatron:/workspace/megatron -v /path/to/dataset:/workspace/dataset -v /path/to/checkpoints:/workspace/checkpoints nvcr.io/nvidia/pytorch:xx.xx-py3
```
## Downloading Checkpoints
We have provided pretrained [BERT-345M](https://ngc.nvidia.com/catalog/models/nvidia:megatron_bert_345m) and [GPT-345M](https://ngc.nvidia.com/catalog/models/nvidia:megatron_lm_345m) checkpoints to evaluate or for finetuning downstream tasks. To access these checkpoints, first [sign up](https://ngc.nvidia.com/signup) for and [setup](https://ngc.nvidia.com/setup/installers/cli) the NVIDIA GPU Cloud (NGC) Registry CLI. Further documentation for downloading models can be found in the [NGC documentation](https://docs.nvidia.com/dgx/ngc-registry-cli-user-guide/index.html#topic_6_4_1).
Alternatively, you can directly download the checkpoints using:
<pre>
BERT-345M-uncased: wget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/megatron_bert_345m/versions/v0.1_uncased/zip -O megatron_bert_345m_v0.1_uncased.zip
BERT-345M-cased: wget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/megatron_bert_345m/versions/v0.1_cased/zip -O megatron_bert_345m_v0.1_cased.zip
GPT-345M: wget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/megatron_lm_345m/versions/v0.0/zip -O megatron_lm_345m_v0.0.zip
</pre>
The models require vocabulary files to run. The BERT WordPiece vocab file can be extracted from Google's pretrained BERT models: [uncased](https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt), [cased](https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-vocab.txt). The GPT [vocab file](https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json) and [merge table](https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt) can be downloaded directly.
# Usage
After installation, there are several possible workflows. The most comprehensive is:
1. Data preprocessing
2. Pretraining
3. Finetuning (Optional for zero-shot tasks)
4. Downstream task evaluation or text generation
However, steps 1 and 2 can be replaced by using one of the pretrained models mentioned above.
We've provided several scripts for pretraining both BERT and GPT in the [`examples`](./examples) directory, as well as scripts for both zero-shot and fine-tuned downstream tasks including MNLI, RACE, WikiText103, and LAMBADA evaluation. There is also a script for GPT interactive text generation.
# Training
## Data Preprocessing
The training data requires preprocessing. First, place your training data in a loose json format, with one json containing a text sample per line. For example:
<pre>
{"src": "www.nvidia.com", "text": "The quick brown fox", "type": "Eng", "id": "0", "title": "First Part"}
{"src": "The Internet", "text": "jumps over the lazy dog", "type": "Eng", "id": "42", "title": "Second Part"}
</pre>
The name of the `text` field of the json can be changed by using the `--json-key` flag in [`preprocess_data.py`](./tools/preprocess_data.py) The other metadata are optional and are not used in training.
The loose json is then processed into a binary format for training. To convert the json into mmap format use `preprocess_data.py`. An example script to prepare data for BERT training is:
<pre>
python tools/preprocess_data.py \
--input my-corpus.json \
--output-prefix my-bert \
--vocab-file bert-vocab.txt \
--tokenizer-type BertWordPieceLowerCase \
--split-sentences
</pre>
The output will be two files named, in this case, `my-bert_text_sentence.bin` and `my-bert_text_sentence.idx`. The `--data-path` specified in later BERT training is the full path and new filename, but without the file extension.
For T5 use the same preprocessing as BERT, perhaps renaming it to:
<pre>
--output-prefix my-t5 \
</pre>
Some minor modifications are required for GPT data preprocessing, namely, the addition of a merge table, an end-of-document token, removal of sentence splitting, and a change to the tokenizer type:
<pre>
python tools/preprocess_data.py \
--input my-corpus.json \
--output-prefix my-gpt2 \
--vocab-file gpt2-vocab.json \
--tokenizer-type GPT2BPETokenizer \
--merge-file gpt2-merges.txt \
--append-eod
</pre>
Here the output files are named `my-gpt2_text_document.bin` and `my-gpt2_text_document.idx`. As before, in GPT training, use the longer name without the extension as `--data-path`.
Further command line arguments are described in the source file [`preprocess_data.py`](./tools/preprocess_data.py).
## BERT Pretraining
The [`examples/bert/train_bert_340m_distributed.sh`](examples/bert/train_bert_340m_distributed.sh) script runs single GPU 345M parameter BERT pretraining. Debugging is the primary use for single GPU training, as the code base and command line arguments are optimized for highly distributed training. Most of the arguments are fairly self-explanatory. By default, the learning rate decays linearly over the training iterations starting at `--lr` to a minimum set by `--min-lr` over `--lr-decay-iters` iterations. The fraction of training iterations used for warmup is set by `--lr-warmup-fraction`. While this is single GPU training, the batch size specified by `--micro-batch-size` is a single forward-backward path batch-size and the code will perform gradient accumulation steps until it reaches `global-batch-size` which is the batch size per iteration. The data is partitioned into a 949:50:1 ratio for training/validation/test sets (default is 969:30:1). This partitioning happens on the fly, but is consistent across runs with the same random seed (1234 by default, or specified manually with `--seed`). We use `train-iters` as the training iterations requested. Alternatively, one can provide `--train-samples` which is total number of samples to train on. If this option is present, then instead of providing `--lr-decay-iters`, one will need to provide `--lr-decay-samples`.
The logging, checkpoint-saving, and evaluation interval options are specified. Note that the `--data-path` now includes the additional `_text_sentence` suffix added in preprocessing, but does not include the file extensions.
Further command line arguments are described in the source file [`arguments.py`](./megatron/training/arguments.py).
To run `train_bert_340m_distributed.sh`, make any desired modifications including setting the environment variables for `CHECKPOINT_PATH`, `VOCAB_FILE`, and `DATA_PATH`. Make sure to set these variables to their paths in the container. Then launch the container with Megatron and necessary paths mounted (as explained in [Setup](#setup)) and run the example script.
## GPT Pretraining
The `examples/gpt3/train_gpt3_175b_distributed.sh` script runs single GPU 345M parameter GPT pretraining. As mentioned above, single GPU training is primarily intended for debugging purposes, as the code is optimized for distributed training.
It follows largely the same format as the previous BERT script with a few notable differences: the tokenization scheme used is BPE (which requires a merge table and a `json` vocabulary file) instead of WordPiece, the model architecture allows for longer sequences (note that the max position embedding must be greater than or equal to the maximum sequence length), and the `--lr-decay-style` has been set to cosine decay. Note that the `--data-path` now includes the additional `_text_document` suffix added in preprocessing, but does not include the file extensions.
Further command line arguments are described in the source file [`arguments.py`](./megatron/training/arguments.py).
`train_gpt3_175b_distributed.sh` can be launched the same way as described for BERT. Set the env vars and make any other modifications, launch the container with appropriate mounts, and run the script.
More details in [`examples/gpt3/README.md`](./examples/gpt3/README.md)
## T5 Pretraining
Very similar to BERT and GPT, the `examples/t5/train_t5_220m_distributed.sh` script runs single GPU "base" (~220M parameter) T5 pretraining. The primary difference from BERT and GPT is the addition of the following arguments to accommodate the T5 architecture:
* `--kv-channels` sets the inner dimension of the "key" and "value" matrices of all attention mechanisms in the model. For BERT and GPT this defaults to the hidden size divided by the number of attention heads, but can be configured for T5.
* `--ffn-hidden-size` sets the hidden size in the feed-forward networks within a transformer layer. For BERT and GPT this defaults to 4 times the transformer hidden size, but can be configured for T5.
* `--encoder-seq-length` and `--decoder-seq-length` set the sequence length for the encoder and decoder separately.
All of the other arguments remain as they were for BERT and GPT pretraining. Run this example with the same steps described above for the other scripts.
More details in [`examples/t5/README.md`](./examples/t5/README.md)
## Distributed Pretraining
The `pretrain_{bert,gpt,t5}_distributed.sh` scripts use the PyTorch distributed launcher for distributed training. As such, multi-node training can be achieved by properly setting environment variables. See the official PyTorch [documentation](https://pytorch.org/docs/stable/elastic/run.html#launcher-api) for further description of these [environment variables](https://pytorch.org/docs/stable/distributed.html#environment-variable-initialization). By default, multi-node training uses the [nccl](https://developer.nvidia.com/nccl) distributed backend. A simple set of additional arguments and the use of the PyTorch distributed module with the `torchrun` elastic launcher (equivalent to `python -m torch.distributed.run`) are the only additional requirements to adopt distributed training. See any of `pretrain_{bert,gpt,t5}_distributed.sh` for more details.
We use two types of parallelism: data and model parallelism. Our data parallelism implementation is in `megatron/core/distributed`, and supports overlapping of the gradient reduction with the backward pass when the `--overlap-grad-reduce` command-line option is used.
Second, we developed a simple and efficient two-dimensional model-parallel approach. To use the first dimension, tensor model parallelism (splitting execution of a single transformer module over multiple GPUs, see Section 3 of [our paper](https://arxiv.org/pdf/1909.08053.pdf)), add the `--tensor-model-parallel-size` flag to specify the number of GPUs among which to split the model, along with the arguments passed to the distributed launcher as mentioned above. To use the second dimension, sequence parallelism, specify `--sequence-parallel`, which also requires tensor model parallelism to be enabled because it splits across the same GPUs (more details in Section 4.2.2 of [our paper](https://arxiv.org/pdf/2205.05198.pdf)).
To use pipeline model parallelism (sharding the transformer modules into stages with an equal number of transformer modules on each stage, and then pipelining execution by breaking the batch into smaller microbatches, see Section 2.2 of [our paper](https://arxiv.org/pdf/2104.04473.pdf)), use the `--pipeline-model-parallel-size` flag to specify the number of stages to split the model into (e.g., splitting a model with 24 transformer layers across 4 stages would mean each stage gets 6 transformer layers each).
We have examples of how to use these two different forms of model parallelism the example scripts ending in `distributed_with_mp.sh`.
Other than these minor changes, the distributed training is identical to the training on a single GPU.
The interleaved pipelining schedule (more details in Section 2.2.2 of [our paper](https://arxiv.org/pdf/2104.04473.pdf)) can be enabled using the `--num-layers-per-virtual-pipeline-stage` argument, which controls the number of transformer layers in a virtual stage (by default with the non-interleaved schedule, each GPU will execute a single virtual stage with `NUM_LAYERS / PIPELINE_MP_SIZE` transformer layers). The total number of layers in the transformer model should be divisible by this argument value. Additionally, the number of microbatches in the pipeline (computed as `GLOBAL_BATCH_SIZE / (DATA_PARALLEL_SIZE * MICRO_BATCH_SIZE)`) should be divisible by the `PIPELINE_MP_SIZE` when using this schedule (this condition is checked in an assertion in the code). The interleaved schedule is not supported for pipelines with 2 stages (`PIPELINE_MP_SIZE=2`).
## Activation Checkpointing and Recomputation
To reduce GPU memory usage when training a large model, we support various forms of activation checkpointing and recomputation. Instead of all activations being stored in memory to be used during backprop, as was traditionally the case in deep learning models, only activations at certain "checkpoints" in the model are retained (or stored) in memory, and the other activations are recomputed on-the-fly when needed for backprop. Note that this kind of checkpointing, *activation* checkpointing, is very different from the checkpointing of model parameters and optimizer state, which is mentioned elsewhere.
We support two levels of recompute granularity: `selective` and `full`. Selective recomputation is the default and is recommended in almost all cases. This mode retains in memory the activations that take less memory storage space and are more expensive to recompute and recomputes the activations that take more memory storage space but are relatively inexpensive to recompute. See [our paper](https://arxiv.org/pdf/2205.05198) for details. You should find that this mode maximizes performance while minimizing the memory required to store activations. To enable selective activation recompute simply use `--recompute-activations`.
For cases where memory is very limited, `full` recompute saves just the inputs to a transformer layer, or a group, or block, of transformer layers, and recomputes everything else. To enable full activation recompute use `--recompute-granularity full`. When using `full` activation recompute, there are two methods: `uniform` and `block`, chosen using the `--recompute-method` argument.
* The `uniform` method uniformly divides the transformer layers into groups of layers (each group of size `--recompute-num-layers`) and stores the input activations of each group in memory. The baseline group size is 1 and, in this case, the input activation of each transformer layer is stored. When the GPU memory is insufficient, increasing the number of layers per group reduces the memory usage, enabling a bigger model to be trained. For example, when `--recompute-num-layers` is set to 4, only the input activation of each group of 4 transformer layers is stored.
* The `block` method recomputes the input activations of a specific number (given by `--recompute-num-layers`) of individual transformer layers per pipeline stage and stores the input activations of the remaining layers in the pipeline stage. Reducing `--recompute-num-layers` results in storing the input activations to more transformer layers, which reduces the activation recomputation required in the backprop, thus improving training performance while increasing memory usage. For example, when we specify 5 layers to recompute of 8 layers per pipeline stage, the input activations of only the first 5 transformer layers are recomputed in the backprop step while the input activations for the final 3 layers are stored. `--recompute-num-layers` can be incrementally increased until the amount of memory storage space required is just small enough to fit in the available memory, thereby both maximally utilizing memory and maximizing performance.
## Distributed Optimizer
Usage: `--use-distributed-optimizer`. Compatible with all model and data types.
The distributed optimizer is a memory savings technique, whereby the optimizer state is evenly distributed across data parallel ranks (versus the traditional method of replicating the optimizer state across data parallel ranks). As described in [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://arxiv.org/abs/1910.02054), our implementation distributes all optimizer state that does not overlap with the model state. For example, when using fp16 model params, the distributed optimizer maintains its own separate copy of fp32 main params & grads, which are distributed across DP ranks. When using bf16 model params, however, the distributed optimizer's fp32 main grads are the same as the model's fp32 grads, and so the grads in this case are not distributed (although the fp32 main params are still distributed, as they are separate from the bf16 model params).
Theoretical memory savings vary depending on the combination of the model's param dtype and grad dtype. In our implementation, the theoretical number of bytes per parameter is (where 'd' is the data parallel size):
| | Non-distributed optim | Distributed optim |
|-|-|-|
| fp16 param, fp16 grads | 20 | 4 + 16/d |
| bf16 param, fp32 grads | 18 | 6 + 12/d |
| fp32 param, fp32 grads | 16 | 8 + 8/d |
As with regular data parallelism, overlapping of the gradient reduction (in this case, a reduce-scatter) with the backward pass can be facilitated using the `--overlap-grad-reduce` flag. Additionally, overlapping of the parameter all-gather can be overlapped with the forward pass using `--overlap-param-gather`.
## FlashAttention
Usage: `--use-flash-attn`. Support attention head dimensions at most 128.
[FlashAttention](https://github.com/HazyResearch/flash-attention) is a fast and
memory-efficient algorithm to compute exact attention. It speeds up model
training and reduces memory requirement.
To install FlashAttention:
```sh
pip install flash-attn
```
## GPT-3 Example
In `examples/gpt3/train_gpt3_175b_distributed.sh` we have provided an example of how to configure Megatron to train [GPT-3](https://arxiv.org/abs/2005.14165) with 175 billion parameters on 1024 GPUs. The script is designed for [slurm](https://slurm.schedmd.com/documentation.html) with [pyxis](https://github.com/NVIDIA/pyxis) plugin but can be easily adopted to any other scheduler. It uses 8-way tensor parallelism and 16-way pipeline parallelism. With options `global-batch-size 1536` and `rampup-batch-size 16 16 5859375`, the training will start with global batch size 16 and linearly increase the global batch size to 1536 over 5,859,375 samples with incremental steps 16. The training dataset can be either a single set or a multiple datasets combined with a set of weights.
With full global batch size of 1536 on 1024 A100 GPUs, each iteration takes around 32 seconds resulting in 138 teraFLOPs per GPU which is 44% of the theoretical peak FLOPs.
## Retro and InstructRetro
Retro [(Borgeaud et al., 2022)](https://arxiv.org/abs/2112.04426) is an autoregressive decoder-only language model (LM) pretrained with retrieval-augmentation.
Retro features practical scalability to support large-scale pretraining from scratch by retrieving from trillions of tokens.
Pretraining with retrieval provides a more efficient storage mechanism of factual knowledge, when compared to storing factual knowledge implicitly within the network's parameters, thus largely reducing model parameters while achieving lower perplexity than standard GPT.
Retro also provides the flexibility to update the
knowledge stored in LMs [(Wang et al., 2023a)](https://arxiv.org/abs/2304.06762)
by updating the retrieval database without training LMs again.
InstructRetro [(Wang et al., 2023b)](https://arxiv.org/abs/2310.07713) further scales up the size of Retro to 48B, featuring the largest LLM pretrained with retrieval (as of December 2023).
The obtained foundation model, Retro 48B, largely outperforms the GPT counterpart in terms of perplexity.
With instruction tuning on Retro, InstructRetro demonstrates significant improvement over the instruction tuned GPT on downstream tasks in the zero-shot setting. Specifically, the average improvement of InstructRetro is 7% over its GPT counterpart across 8 short-form QA tasks, and 10% over GPT across 4 challenging long-form QA tasks. We also find that one can ablate the encoder from InstructRetro architecture and directly use the InstructRetro decoder backbone as GPT, while achieving comparable results.
In this repo, we provide an end-to-end reproduction guide to implement Retro and InstructRetro, covering
- **Retrieval database construction**, which supports billions or even trillions of tokens as a large-scale retrieval database.
- **Pretraining with retrieval**, which supports pretraining from scratch and pretraining from a pretrained GPT model (Retro-fitting).
- **Instruction tuning**, where we provide an open-source instruction tuning dataset and the training recipe for instruction tuning on Retro.
- **Downstream task evaluation**, where we provide the text generation and evaluation scripts for zero-shot question answering tasks.
See [tools/retro/README.md](tools/retro/README.md) for a detailed overview.
## Mamba-based Language Models
See [examples/mamba](./examples/mamba) for details.
<!--
## REALM Pipeline
We are working on implementing the [REALM](https://arxiv.org/pdf/2002.08909.pdf) system. The following sections (will) reflect the three stages of training it. For now it's just the ICT code.
Loosely, they are pretraining the retriever modules, then jointly training the language model and the retriever, and then finetuning a question answering head on the language model with fixed retriever.
### Inverse Cloze Task (ICT) Pretraining
1. Have a corpus in loose JSON format with the intention of creating a collection of fixed-size blocks of text as the fundamental units of data. For a corpus like Wikipedia, this will mean multiple sentences per block but also multiple blocks per document.
Run `tools/preprocess_data.py` to construct one or more indexed datasets with the `--split-sentences` argument to make sentences the basic unit. For the original REALM system, we construct two datasets, one with the title of every document, and another with the body.
Refer to the following script
<pre>
python preprocess_data.py \
--input /path/to/corpus.json \
--json-keys text title \
--split-sentences \
--tokenizer-type BertWordPieceLowerCase \
--vocab-file /path/to/vocab.txt \
--output-prefix corpus_indexed \
--workers 5 # works well for 10 CPU cores. Scale up accordingly.
</pre>
2. Use a custom samples mapping function in place of `megatron/legacy/data/realm_dataset_utils.get_block_samples_mapping` if required. To do this, you will need to implement a new function in C++ inside of `megatron/core/datasets/helpers.cpp`. The samples mapping data structure is used to select the data that will constitute every training sample in advance of the training loop.
The samples mapping is responsible for holding all of the required metadata needed to construct the sample from one or more indexed datasets. In REALM, the samples mapping contains the start and end sentence indices, as well as the document index (to find the correct title for a body) and a unique ID for every block.
3. Pretrain a BERT language model using `pretrain_bert.py`, with the sequence length equal to the block size in token ids. This model should be trained on the same indexed dataset that is used to supply the blocks for the information retrieval task.
In REALM, this is an uncased bert base model trained with the standard hyperparameters.
4. Use `pretrain_ict.py` to train an `ICTBertModel` which uses two BERT-based encoders to encode queries and blocks to perform retrieval with.
The script below trains the ICT model from REALM. It references a pretrained BERT model (step 3) in the `--bert-load` argument. The batch size used in the paper is 4096, so this would need to be run with data parallel world size 32.
<pre>
python pretrain_ict.py \
--num-layers 12 \
--num-attention-heads 12 \
--hidden-size 768 \
--batch-size 128 \
--seq-length 256 \
--max-position-embeddings 256 \
--ict-head-size 128 \
--train-iters 100000 \
--bert-load /path/to/pretrained_bert \
--load checkpoints \
--save checkpoints \
--data-path /path/to/indexed_dataset \
--titles-data-path /path/to/titles_indexed_dataset \
--vocab-file /path/to/vocab.txt \
--lr 0.0001 \
--num-workers 2 \
--lr-decay-style linear \
--weight-decay 1e-2 \
--clip-grad 1.0 \
--lr-warmup-fraction .01 \
--save-interval 3000 \
--query-in-block-prob 0.1 \
--fp16
</pre>
### Building an Index of Block Embeddings
After having trained an ICT model, you can now embed an entire dataset of blocks by creating a `BlockData` structure. After that has been saved, you can load it
and wrap it with a `FaissMIPSIndex` to do fast similarity search which is key in the learned information retrieval pipeline. The initial index can be built with the following script, meant to be run in an interactive session. It can leverage multiple GPUs on multiple nodes to index large datasets much more quickly.
<pre>
python tools/create_doc_index.py \
--num-layers 12 \
--hidden-size 768 \
--ict-head-size 128 \
--num-attention-heads 12 \
--batch-size 128 \
--seq-length 256 \
--max-position-embeddings 256 \
--ict-load /path/to/pretrained_ict \
--data-path /path/to/indexed_dataset \
--titles-data-path /path/to/titles_indexed_dataset \
--block-data-path embedded_blocks.pkl \
--indexer-log-interval 1000 \
--indexer-batch-size 128 \
--vocab-file /path/to/vocab.txt \
--num-workers 2 \
--fp16
</pre>
-->
## Mixture of Experts
MoE (Mixture of Experts) is a powerful LLM architecture implemented in the Megatron-Core framework, designed to enhance the efficiency and scalability of large language models. It leverages **Expert Parallelism**, allowing multiple experts to be distributed across different workers, where each worker processes distinct batches of training samples. This method significantly increases computational throughput, enabling models to achieve high performance metrics, such as 47% MFU during BF16 training for 8x7B on H100.
Key Features of MoE:
- **Parallelism Techniques**: MoE combines various parallelism strategies, including Expert Parallelism, Data Parallelism, Tensor Parallelism, Sequence Paralleism, Pipeline Parallelism, and Context Parallelism. This combination allows for handling larger model variants effectively.
- **Router and Load Balancing**: The system employs advanced routing mechanisms like the Top-K router and utilizes load balancing algorithms to optimize token distribution among experts.
- **Performance Optimizations**: Techniques such as GroupedGEMM and FP8 training enhance the efficiency of MoE models, particularly when multiple experts are involved.
- **Token Dispatch Mechanism**: MoE supports both dropless and token drop strategies to manage token distribution effectively across experts.
For a comprehensive overview of MoE training configurations and optimizations, please refer to the detailed README located at [megatron/core/transformer/moe/README.md](./megatron/core/transformer/moe/README.md).
# Evaluation and Tasks
We provide several command line arguments, detailed in the scripts listed below, to handle various zero-shot and fine-tuned downstream tasks. However, you can also finetune your model from a pretrained checkpoint on other corpora as desired. To do so, simply add the `--finetune` flag and adjust the input files and training parameters within the original training script. The iteration count will be reset to zero, and the optimizer and internal state will be reinitialized. If the fine-tuning is interrupted for any reason, be sure to remove the `--finetune` flag before continuing, otherwise the training will start again from the beginning.
Because evaluation requires substantially less memory than training, it may be advantageous to merge a model trained in parallel for use on fewer GPUs in downstream tasks. The following script accomplishes this. This example reads in a GPT model with 4-way tensor and 4-way pipeline model parallelism and writes out a model with 2-way tensor and 2-way pipeline model parallelism.
<pre>
python tools/checkpoint/convert.py \
--model-type GPT \
--load-dir checkpoints/gpt3_tp4_pp4 \
--save-dir checkpoints/gpt3_tp2_pp2 \
--target-tensor-parallel-size 2 \
--target-pipeline-parallel-size 2
</pre>
Several downstream tasks are described for both GPT and BERT models below. They can be run in distributed and model parallel modes with the same changes used in the training scripts.
## GPT Text Generation
We have included a simple REST server to use for text generation in `tools/run_text_generation_server.py`. You run it much like you would start a pretraining job, specifying an appropriate pretrained checkpoint. There are also few optional parameters: `temperature`, `top-k`and `top-p`. See `--help` or the source file for more information. See [examples/inference/run_text_generation_server_345M.sh](examples/inference/run_text_generation_server_345M.sh) for an example of how to run the server.
Once the server is running you can use `tools/text_generation_cli.py` to query it, it takes one argument which is the host the server is running on.
<pre>
tools/text_generation_cli.py localhost:5000
</pre>
You can also use CURL or any other tools to query the server directly:
<pre>
curl 'http://localhost:5000/api' -X 'PUT' -H 'Content-Type: application/json; charset=UTF-8' -d '{"prompts":["Hello world"], "tokens_to_generate":1}'
</pre>
See [megatron/inference/text_generation_server.py](megatron/inference/text_generation_server.py) for more API options.
### Detoxify GPT via Self-generation
We include an example in `examples/academic_paper_scripts/detxoify_lm/` to detoxify language models by leveraging the generative power of language models.
See [examples/academic_paper_scripts/detxoify_lm/README.md](examples/academic_paper_scripts/detxoify_lm/README.md) for step-by-step tutorials on how to perform domain-adaptive training and detoxify LM using self-generated corpus.
## GPT Evaluation
We include example scripts for GPT evaluation on WikiText perplexity evaluation and LAMBADA Cloze accuracy.
### WikiText Perplexity Evaluation
For even comparison with prior works, we evaluate perplexity on the word-level [WikiText-103 test dataset](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-v1.zip), and appropriately compute perplexity given the change in tokens when using our subword tokenizer.
We use the following command to run WikiText-103 evaluation on a 345M parameter model.
<pre>
TASK="WIKITEXT103"
VALID_DATA=&#60;wikitext path&#62;.txt
VOCAB_FILE=gpt2-vocab.json
MERGE_FILE=gpt2-merges.txt
CHECKPOINT_PATH=checkpoints/gpt2_345m
COMMON_TASK_ARGS="--num-layers 24 \
--hidden-size 1024 \
--num-attention-heads 16 \
--seq-length 1024 \
--max-position-embeddings 1024 \
--fp16 \
--vocab-file $VOCAB_FILE"
python tasks/main.py \
--task $TASK \
$COMMON_TASK_ARGS \
--valid-data $VALID_DATA \
--tokenizer-type GPT2BPETokenizer \
--merge-file $MERGE_FILE \
--load $CHECKPOINT_PATH \
--micro-batch-size 8 \
--log-interval 10 \
--no-load-optim \
--no-load-rng
</pre>
### LAMBADA Cloze Accuracy
To compute LAMBADA cloze accuracy (the accuracy of predicting the last token given the preceding tokens) we utilize a detokenized, processed version of the [LAMBADA dataset](https://github.com/cybertronai/bflm/blob/master/lambada_test.jsonl).
We use the following command to run LAMBADA evaluation on a 345M parameter model. Note that the `--strict-lambada` flag should be used to require whole word matching. Ensure that `lambada` is part of the file path.
<pre>
TASK="LAMBADA"
VALID_DATA=&#60;lambada path&#62;.json
VOCAB_FILE=gpt2-vocab.json
MERGE_FILE=gpt2-merges.txt
CHECKPOINT_PATH=checkpoints/gpt2_345m
COMMON_TASK_ARGS=&#60;same as those in <a href="#wikitext-perplexity-evaluation">WikiText Perplexity Evaluation</a> above&#62;
python tasks/main.py \
--task $TASK \
$COMMON_TASK_ARGS \
--valid-data $VALID_DATA \
--tokenizer-type GPT2BPETokenizer \
--strict-lambada \
--merge-file $MERGE_FILE \
--load $CHECKPOINT_PATH \
--micro-batch-size 8 \
--log-interval 10 \
--no-load-optim \
--no-load-rng
</pre>
Further command line arguments are described in the source file [`main.py`](./tasks/main.py)
## BERT Task Evaluation
### RACE Evaluation
The following script finetunes the BERT model for evaluation on the [RACE dataset](http://www.cs.cmu.edu/~glai1/data/race/). The `TRAIN_DATA` and `VALID_DATA` directory contain the RACE dataset as separate `.txt` files. Note that for RACE, the batch size is the number of RACE query's to evaluate. Since each RACE query has four samples, the effective batch size passed through the model will be four times the batch size specified on the command line.
<pre>
TRAIN_DATA="data/RACE/train/middle"
VALID_DATA="data/RACE/dev/middle \
data/RACE/dev/high"
VOCAB_FILE=bert-vocab.txt
PRETRAINED_CHECKPOINT=checkpoints/bert_345m
CHECKPOINT_PATH=checkpoints/bert_345m_race
COMMON_TASK_ARGS="--num-layers 24 \
--hidden-size 1024 \
--num-attention-heads 16 \
--seq-length 512 \
--max-position-embeddings 512 \
--fp16 \
--vocab-file $VOCAB_FILE"
COMMON_TASK_ARGS_EXT="--train-data $TRAIN_DATA \
--valid-data $VALID_DATA \
--pretrained-checkpoint $PRETRAINED_CHECKPOINT \
--save-interval 10000 \
--save $CHECKPOINT_PATH \
--log-interval 100 \
--eval-interval 1000 \
--eval-iters 10 \
--weight-decay 1.0e-1"
python tasks/main.py \
--task RACE \
$COMMON_TASK_ARGS \
$COMMON_TASK_ARGS_EXT \
--tokenizer-type BertWordPieceLowerCase \
--epochs 3 \
--micro-batch-size 4 \
--lr 1.0e-5 \
--lr-warmup-fraction 0.06
</pre>
### MNLI Evaluation
The following script finetunes the BERT model for evaluation with the [MultiNLI sentence pair corpus](https://www.nyu.edu/projects/bowman/multinli/). Because the matching tasks are quite similar, the script can be quickly tweaked to work with the [Quora Question Pairs](https://www.kaggle.com/quora/question-pairs-dataset) (QQP) dataset as well.
<pre>
TRAIN_DATA="data/glue_data/MNLI/train.tsv"
VALID_DATA="data/glue_data/MNLI/dev_matched.tsv \
data/glue_data/MNLI/dev_mismatched.tsv"
PRETRAINED_CHECKPOINT=checkpoints/bert_345m
VOCAB_FILE=bert-vocab.txt
CHECKPOINT_PATH=checkpoints/bert_345m_mnli
COMMON_TASK_ARGS=&#60;same as those in <a href="#race-evaluation">RACE Evaluation</a> above&#62;
COMMON_TASK_ARGS_EXT=&#60;same as those in <a href="#race-evaluation">RACE Evaluation</a> above&#62;
python tasks/main.py \
--task MNLI \
$COMMON_TASK_ARGS \
$COMMON_TASK_ARGS_EXT \
--tokenizer-type BertWordPieceLowerCase \
--epochs 5 \
--micro-batch-size 8 \
--lr 5.0e-5 \
--lr-warmup-fraction 0.065
</pre>
## Llama-2 Inference and Finetuning
The Llama-2 [family of models](https://ai.meta.com/llama/) are an open-source set of pretrained & finetuned (for chat) models that have achieved strong results across a wide set of benchmarks. At the time of release, Llama-2 models achieved among the best results for open-source models, and were competitive with the closed-source GPT-3.5 model (see https://arxiv.org/pdf/2307.09288.pdf).
The Llama-2 checkpoints can be loaded into Megatron for inference and finetuning. See documentation [here](docs/llama_mistral.md).
# Model Optimization and Deployment
Megatron-Core (MCore) `GPTModel` family supports advanced quantization algorithms and high-performance inference through TensorRT-LLM.
## Quantization and TensorRT-LLM Deployment
See [Megatron Model Optimization and Deployment](examples/inference/quantization/README.md) for `llama2` and `nemotron3` examples.
# Datasets
We do not host any datasets for GPT or BERT training, however, we detail their collection so that our results may be reproduced.
## Collecting Wikipedia Training Data
We recommend following the Wikipedia data extraction process specified by Google research: "the recommended pre-processing is to download [the latest dump](https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2), extract the text with [WikiExtractor.py](https://github.com/attardi/wikiextractor), and then apply any necessary cleanup to convert it into plain text."
We recommend using the `--json` argument when using WikiExtractor, which will dump the Wikipedia data into loose json format (one json object per line), making it more manageable on the file system and also readily consumable by our codebase. We recommend further preprocessing this json dataset with nltk punctuation standardization. For BERT training, use the `--split-sentences` flag to `preprocess_data.py` as described [above](#data-preprocessing) to include sentence breaks in the produced index. If you'd like to use Wikipedia data for GPT training you should still clean it with nltk/spacy/ftfy, but do not use the `--split-sentences` flag.
## Collecting GPT Webtext Data
We utilize the publicly available [OpenWebText](https://github.com/eukaryote31/openwebtext) library from [jcpeterson](https://github.com/jcpeterson/openwebtext) and [eukaryote31's](https://github.com/eukaryote31/openwebtext) work to download urls. We then filter, clean, and deduplicate all downloaded content according to the procedure described in our [openwebtext](./tools/openwebtext) directory. For reddit URLs corresponding to content up to October 2018 we arrived at approximately 37GB of content.
# Reproducibility
Megatron training can be bitwise reproducible; to enable this mode use `--deterministic-mode`. This means that the same training config run twice in the same HW and SW environment should produce identical model checkpoints, losses and accuracy metric values (iteration time metrics may vary).
There are currently three known Megatron optimizations that break reproducibility whilst still producing almost identical training runs:
1. The specific NCCL algorithm that is used during an all-reduce (as specified by the environment variable `NCCL_ALGO`) is important. We have tested the following: `^NVLS`, `Tree`, `Ring`, `CollnetDirect`, `CollnetChain`. The code admits the use of `^NVLS`, which allows NCCL the choice of non-NVLS algorithms; its choice seems to be stable.
2. Flash attention is non-deterministic; do not use `--use-flash-attn`.
3. If using Transformer Engine, you must also set the environment variable `NVTE_ALLOW_NONDETERMINISTIC_ALGO=0`.
In addition, determinisim has only been verified in NGC PyTorch containers up to and newer than 23.12. If you observe nondeterminism in Megatron training under other circumstances please open an issue.
## Projects Using Megatron
Below are some of the projects where we have directly used Megatron:
* [BERT and GPT Studies Using Megatron](https://arxiv.org/pdf/1909.08053.pdf)
* [BioMegatron: Larger Biomedical Domain Language Model](https://www.aclweb.org/anthology/2020.emnlp-main.379.pdf)
* [End-to-End Training of Neural Retrievers for Open-Domain Question Answering](https://arxiv.org/abs/2101.00408)
* [Large Scale Multi-Actor Generative Dialog Modeling](https://www.aclweb.org/anthology/2020.acl-main.8.pdf)
* [Local Knowledge Powered Conversational Agents](https://arxiv.org/abs/2010.10150)
* [MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models](https://www.aclweb.org/anthology/2020.emnlp-main.226.pdf)
* [RACE Reading Comprehension Dataset Leaderboard](http://www.qizhexie.com/data/RACE_leaderboard.html)
* [Training Question Answering Models From Synthetic Data](https://www.aclweb.org/anthology/2020.emnlp-main.468.pdf)
* [Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases](https://arxiv.org/abs/2112.07868)
* [Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models](https://arxiv.org/abs/2202.04173)
* [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://arxiv.org/abs/2201.11990)
* [Multi-Stage Prompting for Knowledgeable Dialogue Generation](https://arxiv.org/abs/2203.08745)
* [Evaluating Parameter Efficient Learning for Generation](https://aclanthology.org/2022.emnlp-main.319.pdf)
* [Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models](https://arxiv.org/abs/2202.04173)
* [Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study](https://arxiv.org/abs/2304.06762)
* [InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining](https://arxiv.org/abs/2310.07713)
* [An Empirical Study of Mamba-based Language Models](https://arxiv.org/abs/2406.07887)
docs/llama_mistral.md 0 → 100644
View file @ d444a97a
# Llama, Mistral and other Llama-like model support in Megatron-LM
NOTE: In order to simplify code we now only support converting llama-3.x and mistral checkpoints downloaded from Huggingface.
The [Llama-2](https://ai.meta.com/llama/) and [Llama-3](https://llama.meta.com/) family of models are an open-source set of pretrained & finetuned (for chat) models that have achieved strong results across a wide set of benchmarks. At their times of release, both Llama-2 and Llama-3 models achieved among the best results for open-source models, and were competitive with leading closed-source models (see https://arxiv.org/pdf/2307.09288.pdf and https://ai.meta.com/blog/meta-llama-3/).
Similarly, [Mistral-7b](https://mistral.ai/news/announcing-mistral-7b/) is an open-source model with pretrained and finetuned (for chat) variants that achieve strong benchmark results.
Architecturally Llama-2, Llama-3 and Mistral-7b are very similar. As such Megatron can support loading checkpoints from all three for inference and finetuning. Converting the checkpoints and loading them is slightly different for each model and is detailed for each below.
# Llama-2
Llama-2 checkpoints can be loaded into Megatron for inference and for finetuning. Loading these checkpoints consists of three steps:
1. Get access to download the checkpoints.
2. Convert the checkpoints from Meta/Huggingface format to Megatron format.
3. Setup arguments for launching the model.
The following sections detail these steps. The final section lists benchmark result comparisons between: 1) Llama-2 inference code running the Meta-format checkpoints, and 2) Megatron inference code running the converted checkpoints.
## Contents
* [Download Meta or Huggingface checkpoints](#download-meta-or-huggingface-checkpoints)
* [Convert checkpoint format](#convert-checkpoint-format)
* [Meta format](#meta-format)
* [Huggingface format](#huggingface-format)
* [Launch model](#launch-model)
* [Megatron](#launch-megatron)
* [Meta](#launch-meta)
* [Huggingface](#launch-hf)
* [Benchmark results](#benchmark-results)
## Download Meta or Huggingface checkpoints
Users must first apply for access to download the Llama-2 checkpoints either directly from [Meta](https://ai.meta.com/resources/models-and-libraries/llama-downloads/) or through [Huggingface](https://huggingface.co/docs/transformers/main/model_doc/llama2) (HF). The checkpoints are available in two formats, Meta's native format (available from both the Meta and HF links), and HF's format (available only from HF). Either format can be converted to Megatron, as detailed next.
## Convert checkpoint format
We recommend passing `--dtype bf16` for training or finetuning. Inference can be done in bfloat16 or float16.
### Meta format
The Meta format checkpoints are converted to HF format as an intermediate step before converting to Megatron format. The `transformers` package is required, and must have version >=4.31.0 (e.g., `pip install transformers>=4.31.0`). (**Note**: we have specifically tested with versions `4.31.0` and `4.32.0`; your experience may vary with newer versions.) Assuming the downloaded checkpoints are in `$CHECKPOINT_DIR` (with separate sub-directories for 7B, 13B, 70B, etc.), the following example command can be used to convert from Llama-2 format to HF format in bfloat16:
```
python tools/checkpoint/convert.py --model-type GPT \
> --loader llama_mistral \
> --saver megatron \
> --checkpoint-type meta \
> --model-size llama2-7B \
> --load-dir $LLAMA_META_FORMAT_DIR \
> --save-dir ${MEGATRON_FORMAT_DIR} \
> --tokenizer-model ${TOKENIZER_MODEL} \
> --target-tensor-parallel-size ${TP} \
> --target-pipeline-parallel-size ${PP} \
> --bf16
```
Valid values for `--model-size` are `llama2-7B`, `llama2-13B`, and `llama2-70B` (for pretrained-only models), and `llama2-7Bf`, `llama2-13Bf`, and `llama2-70Bf` (for chat-finetuned models).
### Huggingface format
The HF checkpoints can be converted to Megatron format by using Megatron's own Llama-2 checkpoint converter for HF format (see script `tools/checkpoint/loader_llama_mistral.py`). One important argument that must be set correctly is the tensor parallel size (`TP`) for each model. The following table shows these values:
| Model size | Tensor parallel size (`TP`) |
| ---------- | --------------------------- |
| 7B | 1 |
| 13B | 2 |
| 70B | 8 |
Using these values for `TP`, along with the path to the Llama-2 tokenizer model (automatically downloaded with original checkpoint download; see `${TOKENIZER_MODEL}` below), run the following command from the root of your Megatron source code to convert from HF format to Megatron format:
```
$>: python tools/checkpoint/convert.py \
> --model-type GPT \
> --loader llama_mistral \
> --saver megatron \
> --target-tensor-parallel-size ${TP} \
> --checkpoint-type hf
> --load-dir ${HF_FORMAT_DIR} \
> --save-dir ${MEGATRON_FORMAT_DIR} \
> --tokenizer-model ${TOKENIZER_MODEL}
```
After this conversion, we are ready to load the checkpoints into a Megatron GPT model.
## Launch model
### Launch Megatron
If loading for either inference or finetuning, use the following arguments:
```
--tensor-model-parallel-size ${TP} \
--pipeline-model-parallel-size 1 \
--seq-length 4096 \
--max-position-embeddings 4096 \
--tokenizer-type Llama2Tokenizer \
--tokenizer-model ${TOKENIZER_MODEL} \
--load ${CHECKPOINT_DIR} \
--exit-on-missing-checkpoint \
--use-checkpoint-args \
--no-load-optim \
--no-load-rng \
--untie-embeddings-and-output-weights \
--use-rotary-position-embeddings \
--normalization RMSNorm \
--no-position-embedding \
--no-masked-softmax-fusion \
--attention-softmax-in-fp32
```
### Launch Meta
Meta checkpoints can be launched with: https://github.com/facebookresearch/llama
### Launch Huggingface
Huggingface checkpoints can be launched with: https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py
## Benchmark results
The tables below list the benchmark comparisons between native Llama-2 (using Meta's checkpoint and Meta's inference code) and Megatron (using a converted HF checkpoint and Megatron's inference code).
The values are the percent error between Megatron and Llama-2, calculated using the formula: `|<llama_score> - <megatron_score>| / <llama_score>`, where the type of score is detailed before each table. Across all tests (80 total per model size), the mean error is 0.15%. The small difference in benchmark scores between the two models is due to minor arithmetic differences in implementation that alter the numerics slightly. Some of the factors that influence this difference include:
- Megatron performs batch matrix multiplications in a couple places, such as within self attention and in SwiGLU, that Llama performs separately.
- Megatron uses `torch.baddbmm` within self attention, versus Llama using `torch.matmul`.
- Megatron uses a `sin`/`cos` implementation for rotary position embeddings, versus Llama using a `polar`/`complex` implementation.
- Llama calls `torch.set_default_dtype(torch.float16)` during initialization, which Megatron does not.
### Big Bench
Score type: multiple choice grade.
| bigbench / standard | 7b | 13b | 70b |
| -- | -- | -- | -- |
| date_understanding | 0.29% | 0.13% | 0.12% |
| general_knowledge | 0.00% | 0.00% | 0.00% |
| human_organs_senses | 0.00% | 0.00% | 0.00% |
| intent_recognition | 0.00% | 0.11% | 0.00% |
| riddle_sense | 0.00% | 0.00% | 0.00% |
| similarities_abstraction | 0.00% | 0.58% | 0.00% |
| simple_arithmetic_json_multiple_choice | 0.00% | 0.00% | 0.00% |
| undo_permutation | 0.19% | 0.19% | 0.18% |
### Multilingual
Score type: multiple choice grade.
| multilingual / xcopa | 7b | 13b | 70b |
| -- | -- | -- | -- |
| en-template-mGPT-remove-punctuation | 0.08% | 0.00% | 0.00% |
| et-template-mGPT-remove-punctuation | 0.00% | 0.13% | 0.25% |
| ht-template-mGPT-remove-punctuation | 0.26% | 0.13% | 0.26% |
| id-template-mGPT-remove-punctuation | 0.11% | 0.00% | 0.19% |
| it-template-mGPT-remove-punctuation | 0.00% | 0.10% | 0.09% |
| qu-template-mGPT-remove-punctuation | 0.00% | 0.00% | 0.27% |
| sw-template-mGPT-remove-punctuation | 0.14% | 0.13% | 0.13% |
| th-template-mGPT-remove-punctuation | 0.25% | 0.13% | 0.13% |
| tr-template-mGPT-remove-punctuation | 0.26% | 0.00% | 0.34% |
| vi-template-mGPT-remove-punctuation | 0.00% | 0.11% | 0.00% |
| zh-template-mGPT-remove-punctuation | 0.00% | 0.10% | 0.09% |
### LM Evaluation Harness
Score type: multiple choice grade.
| lm-eval | 7b | 13b | 70b |
| -- | -- | -- | -- |
| boolq | 0.04% | 0.04% | 0.07% |
| hellaswag | 0.02% | 0.03% | 0.03% |
| piqa | 0.00% | 0.00% | 0.07% |
| winogrande | 0.00% | 0.11% | 0.20% |
### MMLU
Score type: multiple choice grade.
Note: the number in brackets is the number of sub-tasks for each supercategory.
| mmlu | 7b | 13b | 70b |
| -- | -- | -- | -- |
| stem [18] | 0.79% | 0.05% | 0.01% |
| humanities [13] | 0.19% | 0.01% | 0.02% |
| other (business, health, misc.) [14] | 0.08% | 0.06% | 0.12% |
| social sciences [12] | 0.37% | 0.21% | 0.01% |
# Llama-3
Llama-3 checkpoints can be loaded into Megatron for inference and for finetuning. Loading these checkpoints consists of several steps:
1. Get access to download the checkpoints (weights and tokenizer).
2. Convert the checkpoints from Huggingface format to Megatron format.
3. (Optional) Validate converted checkpoints
4. Setup arguments for launching the model.
The following sections detail these steps.
## Contents
* [Download Huggingface checkpoints](#download-huggingface-checkpoints)
* [Convert checkpoint format](#convert-checkpoint-format)
* [Huggingface format](#huggingface-format)
* [Validate checkpoint](#optional-validate-checkpoint)
* [Launch model](#launch-model)
## Download Huggingface checkpoints
Users must first apply for access to download the Llama-3 checkpoints from [Huggingface](https://huggingface.co/meta-llama).
## Convert checkpoint format
We recommend passing `--dtype bf16` for training or finetuning. Inference can be done in bfloat16 or float16.
### Huggingface format
The HF checkpoints can be converted to Megatron format by using Megatron's own Llama-3 checkpoint converter for HF format (see script `tools/checkpoint/loader_llama_mistral.py`). One important argument that must be set correctly is the tensor parallel size (`TP`) for each model. The following table shows these values:
| Model size | Tensor parallel size (`TP`) |
| ---------- | --------------------------- |
| 8B | 1 |
| 70B | 8 |
Using these values for `TP`, along with the path to the Llama-3 tokenizer model (automatically downloaded with original checkpoint download; see `${TOKENIZER_MODEL}` below), run the following command from the root of your Megatron source code to convert from HF format to Megatron format:
```
$>: python tools/checkpoint/convert.py \
> --bf16 \
> --model-type GPT \
> --loader llama_mistral \
> --saver mcore \
> --target-tensor-parallel-size ${TP} \
> --checkpoint-type hf
> --load-dir ${HF_FORMAT_DIR} \
> --save-dir ${MEGATRON_FORMAT_DIR} \
> --tokenizer-model ${TOKENIZER_MODEL}
> --model-size llama3-8B \
```
Valid values for `--model-size` are `llama3-8B` and `llama3-70B` (for pretrained-only models), and `llama3-8Bf` and `llama3-70Bf` (for chat-finetuned models).
After this conversion, we are ready to load the checkpoints into a Megatron GPT model.
## (Optional) Validate checkpoints
A Megatron-LM text generation server for Llama3 can be launched using the script `examples/llama_mistral/run_text_generation_llama3.sh <PATH_TO_CONVERTED_MCORE_CHECKPOINT> <PATH_TO_DOWNLOADED_HUGGINGFACE_CHECKPOINT>`.
Once running, query the server with `curl 'http://<TEXT_GENERATION_SERVER_IP>:5000/api' -X 'PUT' -H 'Content-Type: application/json; charset=UTF-8' -d '{"prompts":["<SOME_PROMPT>"], "tokens_to_generate":100, "top_k":1}'`.
A reference generation for comparison can be obtained from the Huggingface transformers library by running `python examples/llama_mistral/huggingface_reference.py --model_path <PATH_TO_DOWNLOADED_HUGGINGFACE_CHECKPOINT> --prompt <SOME_PROMPT>`.
## Launch model
If loading for either inference or finetuning, use the following arguments:
```
--tensor-model-parallel-size ${TP} \
--pipeline-model-parallel-size 1 \
--seq-length 8192 \
--max-position-embeddings 8192 \
--tokenizer-type HuggingFaceTokenizer \
--tokenizer-model ${TOKENIZER_MODEL} \
--load ${CHECKPOINT_DIR} \
--exit-on-missing-checkpoint \
--use-checkpoint-args \
--no-load-optim \
--no-load-rng \
--untie-embeddings-and-output-weights \
--normalization RMSNorm \
--position-embedding-type rope \
--no-masked-softmax-fusion \
--attention-softmax-in-fp32 \
--disable-bias-linear \
--transformer-impl transformer_engine \
--group-query-attention 8 \
--attention-dropout 0.0 \
--hidden-dropout 0.0 \
--rotary-base 500000 \
--rotary-percent 1.0 \
--ffn-hidden-size 14336 \
--num-attention-heads 32 \
--swiglu \
--bf16 \
```
# Llama-3.1
Llama-3 checkpoints can be loaded into Megatron for inference and for finetuning. Loading these checkpoints consists of several steps:
1. Get access to download the checkpoints (weights and tokenizer).
2. Convert the checkpoints from Huggingface format to Megatron format.
3. (Optional) Validate converted checkpoints
4. Setup arguments for launching the model.
The following sections detail these steps.
## Contents
* [Download Huggingface checkpoints](#download-huggingface-checkpoints)
* [Convert checkpoint format](#convert-checkpoint-format)
* [Huggingface format](#huggingface-format)
* [Validate checkpoint](#optional-validate-checkpoint)
* [Launch model](#launch-model)
## Download Huggingface checkpoints
Users must first apply for access to download the Llama-3 checkpoints from [Huggingface](https://huggingface.co/meta-llama).
## Convert checkpoint format
We recommend passing `--dtype bf16` for training or finetuning. Inference can be done in bfloat16 or float16.
### Huggingface format
The HF checkpoints can be converted to Megatron format by using Megatron's own Llama-3 checkpoint converter for HF format (see script `tools/checkpoint/loader_llama_mistral.py`). One important argument that must be set correctly is the tensor parallel size (`TP`) for each model. The following table shows these values:
| Model size | Tensor parallel size (`TP`) |
| ---------- | --------------------------- |
| 8B | 1 |
| 70B | 8 |
Using these values for `TP`, along with the path to the Llama-3 tokenizer model (automatically downloaded with original checkpoint download; see `${TOKENIZER_MODEL}` below), run the following command from the root of your Megatron source code to convert from HF format to Megatron format:
```
$>: python tools/checkpoint/convert.py \
> --bf16 \
> --model-type GPT \
> --loader llama_mistral \
> --saver mcore \
> --target-tensor-parallel-size ${TP} \
> --checkpoint-type hf
> --load-dir ${HF_FORMAT_DIR} \
> --save-dir ${MEGATRON_FORMAT_DIR} \
> --tokenizer-model ${TOKENIZER_MODEL}
> --model-size llama3-8B \
```
Valid values for `--model-size` are `llama3.1-8B` and `llama3.1-70B` (for pretrained-only models), and `llama3.1-8Bf` and `llama3.1-70Bf` (for chat-finetuned models).
After this conversion, we are ready to load the checkpoints into a Megatron GPT model.
## (Optional) Validate checkpoints
A Megatron-LM text generation server for Llama3.1 can be launched using the script `examples/llama_mistral/run_text_generation_llama3.1.sh <PATH_TO_CONVERTED_MCORE_CHECKPOINT> <PATH_TO_DOWNLOADED_HUGGINGFACE_CHECKPOINT>`.
Once running, query the server with `curl 'http://<TEXT_GENERATION_SERVER_IP>:5000/api' -X 'PUT' -H 'Content-Type: application/json; charset=UTF-8' -d '{"prompts":["<SOME_PROMPT>"], "tokens_to_generate":100, "top_k":1}'`.
A reference generation for comparison can be obtained from the Huggingface transformers library by running `python examples/llama_mistral/huggingface_reference.py --model_path <PATH_TO_DOWNLOADED_HUGGINGFACE_CHECKPOINT> --prompt <SOME_PROMPT>`.
## Launch model
If loading for either inference or finetuning, use the following arguments:
```
--tensor-model-parallel-size ${TP} \
--pipeline-model-parallel-size 1 \
--seq-length 8192 \
--max-position-embeddings 131072 \
--tokenizer-type HuggingFaceTokenizer \
--tokenizer-model ${TOKENIZER_MODEL} \
--load ${CHECKPOINT_DIR} \
--exit-on-missing-checkpoint \
--use-checkpoint-args \
--no-load-optim \
--no-load-rng \
--untie-embeddings-and-output-weights \
--normalization RMSNorm \
--position-embedding-type rope \
--no-masked-softmax-fusion \
--attention-softmax-in-fp32 \
--disable-bias-linear \
--transformer-impl transformer_engine \
--group-query-attention 8 \
--attention-dropout 0.0 \
--hidden-dropout 0.0 \
--rotary-base 500000 \
--rotary-percent 1.0 \
--use-rope-scaling \
--ffn-hidden-size 14336 \
--num-attention-heads 32 \
--swiglu \
--bf16 \
```
# Mistral-7b
Megatron currently supports loading the v0.3 release of Mistral-7b (which does not use sliding window attention and offers a larger 32768 vocabulary) for inference and finetuning. Loading these checkpoints consists of several steps:
1. Get access to download the checkpoints (weights and tokenizer).
2. Convert the checkpoints from HuggingFace format to Megatron format.
3. (Optional) Validate converted checkpoints
4. Setup arguments for launching the model.
The following sections detail these steps.
## Contents
* [Download Huggingface checkpoints](#download-huggingface-checkpoints)
* [Convert checkpoint format](#convert-checkpoint-format)
* [(Optional) Validate checkpoint](#optional-validate-checkpoint)
* [Launch model](#launch-model)
## Download Huggingface checkpoints
Users must first apply for access to download the Mistral-7b checkpoints through [Huggingface](https://huggingface.co/mistralai/Mistral-7B-v0.3) (HF).
## Convert checkpoint format
The HF checkpoints can be converted to Megatron format by using Megatron's own Mistral checkpoint converter for HF format (see script `tools/checkpoint/loader_llama_mistral.py`).
Using the path to the Mistral tokenizer model (downloaded alongside the HF checkpoint), run the following command from the root of your Megatron source code to convert from HF format to mcore format:
```
$>: python tools/checkpoint/convert.py \
> --bf16 \
> --model-type GPT \
> --loader llama_mistral \
> --saver mcore \
> --target-tensor-parallel-size ${TP} \
> --checkpoint-type hf \
> --load-dir ${HF_FORMAT_DIR} \
> --save-dir ${MEGATRON_FORMAT_DIR} \
> --tokenizer-model ${TOKENIZER_MODEL} \
> --model-size mistral-7B \
```
Valid values for `--model-size` are mistral-7B for the pretrained model or mistral-7Bf for the chat fine-tuned model.
After this conversion, we are ready to load the checkpoints into an mcore GPT model.
## (Optional) Validate checkpoints
A Megatron-LM text generation server for Mistral-7B can be launched using the script `examples/llama_mistral/run_text_generation_mistral.sh <PATH_TO_CONVERTED_MCORE_CHECKPOINT> <PATH_TO_DOWNLOADED_HUGGINGFACE_CHECKPOINT>`.
Once running, query the server with `curl 'http://<TEXT_GENERATION_SERVER_IP>:5000/api' -X 'PUT' -H 'Content-Type: application/json; charset=UTF-8' -d '{"prompts":["<SOME_PROMPT>"], "tokens_to_generate":100, "top_k":1}'`.
A reference generation for comparison can be obtained from the Huggingface transformers library by running `python examples/llama_mistral/huggingface_reference.py --model_path <PATH_TO_DOWNLOADED_HUGGINGFACE_CHECKPOINT> --prompt <SOME_PROMPT>`.
## Launch model
If loading for either inference or finetuning, use the following arguments:
```
--tensor-model-parallel-size ${TP} \
--pipeline-model-parallel-size 1 \
--seq-length 4096 \
--max-position-embeddings 4096 \
--tokenizer-type HuggingFaceTokenizer \
--tokenizer-model ${TOKENIZER_MODEL} \
--load ${CHECKPOINT_DIR} \
--exit-on-missing-checkpoint \
--use-checkpoint-args \
--no-load-optim \
--no-load-rng \
--untie-embeddings-and-output-weights \
--normalization RMSNorm \
--position-embedding-type rope \
--no-masked-softmax-fusion \
--attention-softmax-in-fp32
--apply-layernorm-1p \
--transformer-impl transformer_engine \
--group-query-attention 8 \
--disable-bia-linear \
--rotary-base 1000000 \
--rotary-percent 1.0 \
--swiglu \
--ffn-hidden-size 14336 \
--num-attention-heads 32
```
# Other Llama-like model support
*Note: Experimental*
Many models such as Yi-34B use the Llama architecture and may be converted from HuggingFace to Megatron using the commands in [Llama3](#llama-3).
# Known numerical differences
It is not expected that the megatron and Huggingface implementations of llama3.x and mistral models will produce numerically identical results. There are multiple points where small numerical differences are expected. This is a non-exhaustive list:
1. TransformerEngine (TE) uses the model params_dtype inside RMSNorm whereas the Huggingface implementation uses fp32. See for details: https://github.com/NVIDIA/TransformerEngine/issues/1132
2. Huggingface `transformers` implements the q, k and v projections in self-attention as separate GEMMs whereas mcore combines them into a single GEMM for efficiency. This leads to small numerical differences.
docs/source/api-guide/context_parallel.rst 0 → 100644
View file @ d444a97a
context\_parallel package
=========================
Context parallelism overview
----------------------------
.. figure:: ../images/context_parallel/CP_overview.png
:alt: cp_overview
:align: center
Figure 1: A transformer layer running with TP2CP2. Communications next to Attention are for CP, others are for TP. (AG/RS: all-gather in forward and reduce-scatter in backward, RS/AG: reduce-scatter in forward and all-gather in backward, /AG: no-op in forward and all-gather in backward).
Context Parallelism ("CP") is a parallelization scheme on the dimension of sequence length. Unlike prior SP (sequence parallelism) which only splits the sequence of Dropout and LayerNorm activations, CP partitions the network inputs and all activations along sequence dimension. With CP, all modules except attention (e.g., Linear, LayerNorm, etc.) can work as usual without any changes, because they do not have inter-token operations. As for attention, the Q (query) of each token needs to compute with the KV (key and value) of all tokens in the same sequence. Hence, CP requires additional all-gather across GPUs to collect the full sequence of KV. Correspondingly, reduce-scatter should be applied to the activation gradients of KV in backward propagation. To reduce activation memory footprint, each GPU only stores the KV of a sequence chunk in forward and gathers KV again in backward. KV communication happens between a GPU and its counterparts in other TP groups. The all-gather and reduce-scatter are transformed to point-to-point communications in ring topology under the hood. Exchanging KV also can leverage MQA/GQA to reduce communication volumes, as they only have one or few attention heads for KV.
For example, in Figure 1, assuming sequence length is 8K, each GPU processes 4K tokens. GPU0 and GPU2 compose a CP group, they exchange KV with each other. Same thing also happens between GPU1 and GPU3. CP is similar to `Ring Attention <https://arxiv.org/abs/2310.01889>`_ but provides better performance by (1) leveraging the latest OSS and cuDNN flash attention kernels; (2) removing unnecessary computation resulted from low-triangle causal masking and achieving optimal load balance among GPUs.
Context parallelism benefits
----------------------------
.. figure:: ../images/context_parallel/CP_results.png
:alt: cp_results
:align: center
Figure 2: Speedup of 175B GPT with various TP+CP combinations vs. full recompute (i.e., TP8CP1).
LLM encounters OOM (out of memory) issue with long context (i.e., long sequence length) because of linearly increasing memory footprint of activations. Recomputing activations in backward can avoid OOM but also introduce significant overheads (~30% with full recompute). Enlarging TP (tensor model parallelism) can fix the OOM issue as well, but it potentially makes compute (e.g., Linear) too short to overlap communication latencies. To be clear, scaling out to more GPUs with bigger TP can hit the overlapping problem no matter if OOM happens.
CP can better address the issues. With CP, each GPU only computes on a part of the sequence, which reduces both computation and communication by CP times. Therefore, there are no concerns about the overlapping between them. The activation memory footprint per GPU is also CP times smaller, hence no OOM issue anymore. As Figure 2 shows, the combinations of TP and CP can achieve optimal performance by eliminating recompute overheads and making the best tradeoff between computation and communications.
Enabling context parallelism
----------------------------
CP support has been added to GPT. All models that share GPT code path also should be able to benefit from CP, such as Llama. CP can work with TP (tensor model parallelism), PP (pipeline model parallelism), and DP (data parallelism), where the total number of GPUs equals TPxCPxPPxDP. CP also can work with different attention variants, including MHA/MQA/GQA, uni-directional and bi-directional masking.
CP is enabled by simply setting context_parallel_size=<CP_SIZE> in command line. Default context_parallel_size is 1, which means CP is disabled. Running with CP requires Megatron-Core (>=0.5.0) and Transformer Engine (>=1.1).
docs/source/api-guide/datasets.rst 0 → 100644
View file @ d444a97a
datasets package
================
.. mdinclude :: ../../../megatron/core/datasets/readme.md
Submodules
----------
datasets.blended\_megatron\_dataset\_config module
---------------------------------------------------
.. automodule:: core.datasets.blended_megatron_dataset_config
:members:
:undoc-members:
:show-inheritance:
datasets.blended\_megatron\_dataset\_builder module
---------------------------------------------------
.. automodule:: core.datasets.blended_megatron_dataset_builder
:members:
:undoc-members:
:show-inheritance:
datasets.megatron\_tokenizer module
-----------------------------------
.. automodule:: core.datasets.megatron_tokenizer
:members:
:undoc-members:
:show-inheritance:
datasets.indexed\_dataset module
--------------------------------
.. automodule:: core.datasets.indexed_dataset
:members:
:undoc-members:
:show-inheritance:
datasets.megatron\_dataset module
---------------------------------
.. automodule:: core.datasets.megatron_dataset
:members:
:undoc-members:
:show-inheritance:
datasets.gpt\_dataset module
----------------------------
.. automodule:: core.datasets.gpt_dataset
:members:
:undoc-members:
:show-inheritance:
datasets.masked\_dataset module
-------------------------------
.. automodule:: core.datasets.masked_dataset
:members:
:undoc-members:
:show-inheritance:
datasets.bert\_dataset module
-----------------------------
.. automodule:: core.datasets.bert_dataset
:members:
:undoc-members:
:show-inheritance:
datasets.t5\_dataset module
---------------------------
.. automodule:: core.datasets.t5_dataset
:members:
:undoc-members:
:show-inheritance:
datasets.blended\_dataset module
----------------------------------
.. automodule:: core.datasets.blended_dataset
:members:
:undoc-members:
:show-inheritance:
datasets.utils module
---------------------
.. automodule:: core.datasets.utils
:members:
:undoc-members:
:show-inheritance:
Module contents
---------------
.. automodule:: core.datasets
:members:
:undoc-members:
:show-inheritance:
docs/source/api-guide/dist_checkpointing.rst 0 → 100644
View file @ d444a97a
dist\_checkpointing package
===========================
A library for saving and loading the distributed checkpoints.
A "distributed checkpoint" can have various underlying formats (current default format is based on Zarr)
but has a distinctive property - the checkpoint saved in one parallel configuration (tensor/pipeline/data parallelism)
can be loaded in a different parallel configuration.
Using the library requires defining sharded state_dict dictionaries with functions from *mapping* and *optimizer* modules.
Those state dicts can be saved or loaded with a *serialization* module using strategies from *strategies* module.
Subpackages
-----------
.. toctree::
:maxdepth: 4
dist_checkpointing.strategies
Submodules
----------
dist\_checkpointing.serialization module
----------------------------------------
.. automodule:: core.dist_checkpointing.serialization
:members:
:undoc-members:
:show-inheritance:
dist\_checkpointing.mapping module
----------------------------------
.. automodule:: core.dist_checkpointing.mapping
:members:
:undoc-members:
:show-inheritance:
dist\_checkpointing.optimizer module
------------------------------------
.. automodule:: core.dist_checkpointing.optimizer
:members:
:undoc-members:
:show-inheritance:
dist\_checkpointing.core module
-------------------------------
.. automodule:: core.dist_checkpointing.core
:members:
:undoc-members:
:show-inheritance:
dist\_checkpointing.dict\_utils module
--------------------------------------
.. automodule:: core.dist_checkpointing.dict_utils
:members:
:undoc-members:
:show-inheritance:
dist\_checkpointing.utils module
--------------------------------
.. automodule:: core.dist_checkpointing.utils
:members:
:undoc-members:
:show-inheritance:
Module contents
---------------
.. automodule:: core.dist_checkpointing
:members:
:undoc-members:
:show-inheritance:
docs/source/api-guide/dist_checkpointing.strategies.rst 0 → 100644
View file @ d444a97a
dist\_checkpointing.strategies package
======================================
Package defining different checkpoint formats (backends) and saving/loading algorithms (strategies).
Strategies can be used for implementing new checkpoint formats or implementing new (more optimal for a given use case) ways of saving/loading of existing formats.
Strategies are passed to `dist_checkpointing.load` and `dist_checkpointing.save` functions and control the actual saving/loading procedure.
Submodules
----------
dist\_checkpointing.strategies.base module
------------------------------------------
.. automodule:: core.dist_checkpointing.strategies.base
:members:
:undoc-members:
:show-inheritance:
dist\_checkpointing.strategies.tensorstore module
-------------------------------------------------
.. automodule:: core.dist_checkpointing.strategies.tensorstore
:members:
:undoc-members:
:show-inheritance:
dist\_checkpointing.strategies.two\_stage module
------------------------------------------------
.. automodule:: core.dist_checkpointing.strategies.two_stage
:members:
:undoc-members:
:show-inheritance:
dist\_checkpointing.strategies.zarr module
------------------------------------------
.. automodule:: core.dist_checkpointing.strategies.zarr
:members:
:undoc-members:
:show-inheritance:
Module contents
---------------
.. automodule:: core.dist_checkpointing.strategies
:members:
:undoc-members:
:show-inheritance:
docs/source/api-guide/dist_optimizer.md 0 → 100644
View file @ d444a97a
# Distributed Optimizer
The motivation for the distributed optimizer is to save memory by distributing the optimizer state evenly across data parallel ranks (https://arxiv.org/abs/1910.02054), versus the naive method of replicating the optimizer state across data parallel ranks.
Theoretical memory savings vary depending on the combination of the datatype of the model's parameters (`param_dtype`) and main gradients accumulated across data-parallel replicas (`grad_dtype`). We always use `fp32` main parameters for optimizer steps. In the current implementation, the theoretical number of bytes per parameter is (where d is the data parallel size):
| | Non-distributed optim | Distributed optim |
| ------ | ------ | ------ |
| `fp16` parameters, `fp16` gradients | 20 | 4 + 16/d |
| `bf16` parameters, `fp32` gradients | 18 | 6 + 12/d |
| `fp32` parameters, `fp32` gradients | 16 | 8 + 8/d |
Our implementation of the distributed optimizer uses contiguous buffers for parameters and main gradients; model gradients are copied over to the main gradients as soon as they are fully computed.
The figures below illustrate the distributed optimizer's sharding scheme, and the key steps of the distributed optimizer's parameter update:
## Data flow
![Data flow](../images/distrib_optimizer/data_flow.png)
## Sharding scheme
![Sharding scheme](../images/distrib_optimizer/sharding_scheme.png)
## Key steps
_(note: using illustrations above, assuming `bf16` model weights, `bf16` model gradients that are computed by the backward pass and `fp32` main gradients that are also used for optimizer steps; we always use `fp32` main weights for optimizer steps)_
- Backward pass finishes (gradient buffer holds 16 `fp32` gradient elements).
- Call reduce-scatter on each DP rank.
- Each DP rank now has 4 elements within the gradient buffer that are fully reduced (remaining 12 elements are garbage).
- DP rank 0 has gradient values for elements [0:4].
- DP rank 1 has gradient values for elements [4:8].
- DP rank 2 has gradient values for elements [8:12].
- DP rank 3 has gradient values for elements [12:16].
- Optimizer.step().
- Each DP rank copies its 4 `fp32` main parameter elements into the corresponding `bf16` parameter buffer (each element is cast from fp32 to fp16).
- Call all-gather on each DP rank.
- The parameter buffer now contains all 16, fully updated, `bf16` model parameter elements. Parameters in PyTorch modules already point to the appropriate locations in this parameter buffer, and thus forward passes are ready to run after the all-gather completes.
- At this point, the gradient buffer is also ready to be zero'd for the next iteration.
docs/source/api-guide/distributed.rst 0 → 100644
View file @ d444a97a
distributed package
===================
This package contains various utilities to finalize model weight gradients
on each rank before the optimizer step. This includes a distributed data
parallelism wrapper to all-reduce or reduce-scatter the gradients across
data-parallel replicas, and a `finalize\_model\_grads` method to
synchronize gradients across different parallelism modes (e.g., 'tied'
layers on different pipeline stages, or gradients for experts in a MoE on
different ranks due to expert parallelism).
Submodules
----------
distributed.distributed\_data\_parallel
---------------------------------------
Model wrapper for distributed data parallelism. Stores gradients in a
contiguous buffer, and supports the option of overlapping communication
(all-reduce or reduce-scatter) with backprop computation by breaking up
full model's gradients into smaller buckets and running all-reduce /
reduce-scatter on each bucket asynchronously.
.. automodule:: core.distributed.distributed_data_parallel
:members:
:undoc-members:
:show-inheritance:
distributed.finalize\_model\_grads
----------------------------------
Finalize model gradients for optimizer step across all used parallelism modes.
Synchronizes the all-reduce / reduce-scatter of model gradients across DP replicas,
all-reduces the layernorm gradients for sequence parallelism, embedding gradients
across first and last pipeline stages (if not tied), and expert gradients for expert
parallelism.
.. automodule:: core.distributed.finalize_model_grads
:members:
:undoc-members:
:show-inheritance:
Module contents
---------------
Contains functionality to synchronize gradients across different ranks before
optimizer step.
.. automodule:: core.distributed
:members:
:undoc-members:
:show-inheritance:
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment