v1.0.8

.gitignore

+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+.vscode
+
+checkpoints/
+wandb/

.pre-commit-config-check.yaml

+repos:
+  - repo: https://github.com/psf/black
+    rev: 22.12.0
+    hooks:
+      - id: black
+        language_version: python3
+        args:
+          - --line-length=119
+          - --check
+  - repo: https://github.com/charliermarsh/ruff-pre-commit
+    # Ruff version.
+    rev: 'v0.0.271'
+    hooks:
+      - id: ruff
+        args:
+          - --no-fix
+  - repo: local
+    hooks:
+      - id: pylint-nanotron
+        name: pylint nanotron core
+        entry: pylint --init-hook='import sys; sys.path.append(".")'
+        exclude: ^examples/.*$ # ignore examples as for each example we need to go in and look
+        language: system
+        types: [ python ]
+        args:
+          - --errors-only
+      - id: pylint-example-dataloading
+        name: pylint example dataloading
+        entry: pylint --init-hook='import sys; sys.path.append(".")'
+        files: ^examples/dataloading/.*$
+        language: system
+        types: [ python ]
+        args:
+          - --errors-only
+      - id: pylint-example-gpt2-mqa
+        name: pylint example gpt2_mqa
+        entry: pylint --init-hook='import sys; sys.path.append(".")'
+        files: ^examples/gpt2_mqa/.*$
+        language: system
+        types: [ python ]
+        args:
+          - --errors-only
+      - id: pylint-example-gpt2
+        name: pylint example gpt2
+        entry: pylint --init-hook='import sys; sys.path.append(".")'
+        files: ^examples/gpt2/.*$
+        language: system
+        types: [ python ]
+        args:
+          - --errors-only
+      - id: pylint-example-llama
+        name: pylint example llama
+        entry: pylint --init-hook='import sys; sys.path.append(".")'
+        files: ^examples/llama/.*$
+        language: system
+        types: [ python ]
+        args:
+          - --errors-only
+      - id: pylint-example-p2p
+        name: pylint example p2p
+        entry: pylint --init-hook='import sys; sys.path.append(".")'
+        files: ^examples/p2p/.*$
+        language: system
+        types: [ python ]
+        args:
+          - --errors-only
+      - id: pylint-example-t5
+        name: pylint example t5
+        entry: pylint --init-hook='import sys; sys.path.append(".")'
+        files: ^examples/t5/.*$
+        language: system
+        types: [ python ]
+        args:
+          - --errors-only

.pre-commit-config.yaml

+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.4.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+  - repo: https://github.com/psf/black
+    rev: 22.12.0
+    hooks:
+      - id: black
+        language_version: python3
+        args:
+          - --line-length=119
+  - repo: https://github.com/charliermarsh/ruff-pre-commit
+    # Ruff version.
+    rev: 'v0.0.271'
+    hooks:
+      - id: ruff
+        args:
+          - --fix
+          - --exit-non-zero-on-fix
+  # - repo: https://github.com/PyCQA/isort
+  #   rev: 5.12.0
+  #   hooks:
+  #     - id: isort
+  #       args:
+  #         - --profile=black
+  #         - --skip-glob=wandb/**/*
+  #         - --thirdparty=wandb
+  - repo: https://github.com/codespell-project/codespell
+    rev: v2.1.0
+    hooks:
+      - id: codespell
+        args:
+          - -w
+          - --ignore-words-list=nd,reacher,thist,ths,magent,ba,fo,doesnt

.pylintrc

+[MASTER]
+# Use multiprocessing for pylint
+jobs=0
+
+# List of members which are set dynamically and missed by Pylint inference
+# system, and so shouldn't trigger E1101 when accessed.
+ignore-paths=
+
+load-plugins=linter.pylint.ban_rank,
+
+[MESSAGES CONTROL]
+# Disable list of rules
+disable=
+    no-member,                  # E1101: Module 'torch' has no 'allclose' member (no-member)
+    no-name-in-module,          # E0611: No name 'HFTensorBoardLogger' in module 'huggingface_hub' (no-name-in-module)
+    import-error,               # E0401: Unable to import 'tensorboardX' (import-error)
+    relative-beyond-top-level   # E0402: Attempted relative import beyond top-level package (relative-beyond-top-level)

CODE_OF_CONDUCT.md

+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, religion, or sexual identity
+and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the
+  overall community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or
+  advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email
+  address, without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official e-mail address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+feedback@huggingface.co.
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series
+of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or
+permanent ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior,  harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within
+the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.0, available at
+https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
+
+Community Impact Guidelines were inspired by [Mozilla's code of conduct
+enforcement ladder](https://github.com/mozilla/diversity).
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see the FAQ at
+https://www.contributor-covenant.org/faq. Translations are available at
+https://www.contributor-covenant.org/translations.

CONTRIBUTING.md

+<!---
+Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# How to contribute to 🤗 Nanotron?
+
+Everyone is welcome to contribute, and we value everybody's contribution. Code
+is thus not the only way to help the community. Answering questions, helping
+others, reaching out and improving the documentations are immensely valuable to
+the community.
+
+It also helps us if you spread the word: reference the library from blog posts
+on the awesome projects it made possible, shout out on Twitter every time it has
+helped you, or simply star the repo to say "thank you".
+
+Whichever way you choose to contribute, please be mindful to respect our
+[code of conduct](CODE_OF_CONDUCT.md).
+
+## You can contribute in so many ways!
+
+Some of the ways you can contribute to nanotron:
+* Fixing outstanding issues with the existing code;
+* Contributing to the examples or to the documentation;
+* Submitting issues related to bugs or desired new features.
+
+## Submitting a new issue or feature request
+
+Do your best to follow these guidelines when submitting an issue or a feature
+request. It will make it easier for us to come back to you quickly and with good
+feedback.
+
+### Did you find a bug?
+
+The 🤗 Nanotron library is robust and reliable thanks to the users who notify us of
+the problems they encounter. So thank you for reporting an issue.
+
+First, we would really appreciate it if you could **make sure the bug was not
+already reported** (use the search bar on Github under Issues).
+
+Did not find it? :( So we can act quickly on it, please follow these steps:
+
+* Include your **OS type and version**, the versions of **Python** and **PyTorch**.
+* A short, self-contained, code snippet that allows us to reproduce the bug in
+  less than 30s;
+* Provide your Nanotron configuration used for the run;
+* Describe the expected behavior and the actual behavior;
+
+### Do you want a new feature?
+
+A good feature request addresses the following points:
+
+1. Motivation first:
+* Is it related to a problem/frustration with the library? If so, please explain
+  why. Providing a code snippet that demonstrates the problem is best.
+* Is it related to something you would need for a project? We'd love to hear
+  about it!
+* Is it something you worked on and think could benefit the community?
+  Awesome! Tell us what problem it solved for you.
+2. Write a *full paragraph* describing the feature;
+3. Provide a **code snippet** that demonstrates its future use;
+4. In case this is related to a paper, please attach a link;
+5. Attach any additional information (drawings, screenshots, etc.) you think may help.
+
+If your issue is well written we're already 80% of the way there by the time you
+post it.
+
+## Submitting a pull request (PR)
+
+Before writing code, we strongly advise you to search through the existing PRs or
+issues to make sure that nobody is already working on the same thing. If you are
+unsure, it is always a good idea to open an issue to get some feedback.
+
+You will need basic `git` proficiency to be able to contribute to
+🤗 Nanotron. `git` is not the easiest tool to use but it has the greatest
+manual. Type `git --help` in a shell and enjoy. If you prefer books, [Pro
+Git](https://git-scm.com/book/en/v2) is a very good reference.
+
+Follow these steps to start contributing:
+
+1. Fork the [repository](https://github.com/huggingface/nanotron) by
+   clicking on the 'Fork' button on the repository's page. This creates a copy of the code
+   under your GitHub user account.
+
+2. Clone your fork to your local disk, and add the base repository as a remote. The following command
+   assumes you have your public SSH key uploaded to GitHub. See the following guide for more
+   [information](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository).
+
+   ```bash
+   $ git clone git@github.com:<your Github handle>/nanotron.git
+   $ cd nanotron
+   $ git remote add upstream https://github.com/huggingface/nanotron.git
+   ```
+
+3. Create a new branch to hold your development changes, and do this for every new PR you work on.
+
+   Start by synchronizing your `main` branch with the `upstream/main` branch (ore details in the [GitHub Docs](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/syncing-a-fork)):
+
+   ```bash
+   $ git checkout main
+   $ git fetch upstream
+   $ git merge upstream/main
+   ```
+
+   Once your `main` branch is synchronized, create a new branch from it:
+
+   ```bash
+   $ git checkout -b a-descriptive-name-for-my-changes
+   ```
+
+   **Do not** work on the `main` branch.
+
+4. Set up a development environment by running the following command in a conda or a virtual environment you've created for working on this library:
+
+   ```bash
+   $ pip install -e ".[dev]"
+   $ pip install -e ".[test]"
+   $ pre-commit install
+   ```
+
+   (If nanotron was already installed in the virtual environment, remove
+   it with `pip uninstall nanotron` before reinstalling it in editable
+   mode with the `-e` flag.)
+
+   Alternatively, if you are using [Visual Studio Code](https://code.visualstudio.com/Download), the fastest way to get set up is by using
+   the provided Dev Container. Documentation on how to get started with dev containers is available [here](https://code.visualstudio.com/docs/remote/containers).
+
+5. Develop the features on your branch.
+
+   As you work on the features, you should make sure that the test suite
+   passes. You should run the tests impacted by your changes like this (see
+   below an explanation regarding the environment variable):
+
+   ```bash
+   $ pytest tests/<TEST_TO_RUN>.py
+   ```
+
+   `nanotron` relies on `ruff` to format its source code
+   consistently. After you make changes, apply automatic style corrections and code verifications
+   that can't be automated in one go with:
+
+   This target is also optimized to only work with files modified by the PR you're working on.
+
+   If you prefer to run the checks one after the other, the following command apply the
+   style corrections:
+
+   ```bash
+   $ pre-commit run --all-files
+   ```
+
+   Once you're happy with your changes, add changed files using `git add` and
+   make a commit with `git commit` to record your changes locally:
+
+   ```bash
+   $ git add modified_file.py
+   $ git commit
+   ```
+
+   Please write [good commit messages](https://chris.beams.io/posts/git-commit/).
+
+   It is a good idea to sync your copy of the code with the original
+   repository regularly. This way you can quickly account for changes:
+
+   ```bash
+   $ git fetch upstream
+   $ git rebase upstream/main
+   ```
+
+   Push the changes to your account using:
+
+   ```bash
+   $ git push -u origin a-descriptive-name-for-my-changes
+   ```
+
+6. Once you are satisfied (**and the checklist below is happy too**), go to the
+   webpage of your fork on GitHub. Click on 'Pull request' to send your changes
+   to the project maintainers for review.
+
+7. It's ok if maintainers ask you for changes. It happens to core contributors
+   too! So everyone can see the changes in the Pull request, work in your local
+   branch and push the changes to your fork. They will automatically appear in
+   the pull request.
+
+
+### Checklist
+
+1. The title of your pull request should be a summary of its contribution;
+2. If your pull request addresses an issue, please mention the issue number in
+   the pull request description to make sure they are linked (and people
+   consulting the issue know you are working on it);
+3. To indicate a work in progress please prefix the title with `[WIP]`, or mark
+   the PR as a draft PR. These are useful to avoid duplicated work, and to differentiate
+   it from PRs ready to be merged;
+4. Make sure existing tests pass;
+5. Add high-coverage tests. No quality testing = no merge.
+
+See an example of a good PR here: https://github.com/huggingface/nanotron/pull/155
+
+### Tests
+
+An extensive test suite is included to test the library behavior and several examples. Library tests can be found in
+the [tests folder](https://github.com/huggingface/nanotron/tree/main/tests).
+
+We use `pytest` in order to run the tests. From the root of the
+repository, here's how to run tests with `pytest` for the library:
+
+```bash
+# Runs all tests (where 12 of which run in parallel)
+$ pytest -n 12 tests
+```
+
+You can specify a smaller set of tests in order to test only the feature
+you're working on.

HuggingFaceTB/cosmo2-tokenizer/README.md

+---
+library_name: transformers
+datasets:
+- HuggingFaceTB/cosmo2_training_data_subset_1M
+---
+
+# cosmo2-tokenizer
+ Tokenizer for the training of cosmo2. This tokenizer was trained on 1M samples from:
+ - FineWeb-Edu 70%
+ - Cosmopedia v2 15%
+ - StarCoderData  8%
+ - OpenWebMath 5%
+ - StackOverFlow 2%
\ No newline at end of file

HuggingFaceTB/cosmo2-tokenizer/special_tokens_map.json

+{
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<|im_start|>",
+    "<|im_end|>",
+    "<repo_name>",
+    "<reponame>",
+    "<file_sep>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<jupyter_script>",
+    "<empty_output>"
+  ],
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

HuggingFaceTB/cosmo2-tokenizer/tokenizer_config.json

+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<repo_name>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<reponame>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<file_sep>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<filename>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<gh_stars>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<issue_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<issue_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<issue_closed>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<jupyter_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<jupyter_text>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<jupyter_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<jupyter_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "<jupyter_script>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "<empty_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<|im_start|>",
+    "<|im_end|>",
+    "<repo_name>",
+    "<reponame>",
+    "<file_sep>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<jupyter_script>",
+    "<empty_output>"
+  ],
+  "bos_token": "<|endoftext|>",
+  "chat_template": "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 1000000000000000019884624838656,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}

LICENSE

+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

Llama-3.2-3B/README.md

+---
+language:
+- en
+- de
+- fr
+- it
+- pt
+- hi
+- es
+- th
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- facebook
+- meta
+- pytorch
+- llama
+- llama-3
+license: llama3.2
+extra_gated_prompt: >-
+  ### LLAMA 3.2 COMMUNITY LICENSE AGREEMENT
+
+
+  Llama 3.2 Version Release Date: September 25, 2024
+
+  
+  “Agreement” means the terms and conditions for use, reproduction, distribution 
+  and modification of the Llama Materials set forth herein.
+
+  
+  “Documentation” means the specifications, manuals and documentation accompanying Llama 3.2
+  distributed by Meta at https://llama.meta.com/doc/overview.
+
+  
+  “Licensee” or “you” means you, or your employer or any other person or entity (if you are 
+  entering into this Agreement on such person or entity’s behalf), of the age required under
+  applicable laws, rules or regulations to provide legal consent and that has legal authority
+  to bind your employer or such other person or entity if you are entering in this Agreement
+  on their behalf.
+
+  
+  “Llama 3.2” means the foundational large language models and software and algorithms, including
+  machine-learning model code, trained model weights, inference-enabling code, training-enabling code,
+  fine-tuning enabling code and other elements of the foregoing distributed by Meta at 
+  https://www.llama.com/llama-downloads.
+
+  
+  “Llama Materials” means, collectively, Meta’s proprietary Llama 3.2 and Documentation (and 
+  any portion thereof) made available under this Agreement.
+
+  
+  “Meta” or “we” means Meta Platforms Ireland Limited (if you are located in or, 
+  if you are an entity, your principal place of business is in the EEA or Switzerland) 
+  and Meta Platforms, Inc. (if you are located outside of the EEA or Switzerland). 
+
+
+  By clicking “I Accept” below or by using or distributing any portion or element of the Llama Materials,
+  you agree to be bound by this Agreement.
+
+  
+  1. License Rights and Redistribution.
+  
+  a. Grant of Rights. You are granted a non-exclusive, worldwide, 
+  non-transferable and royalty-free limited license under Meta’s intellectual property or other rights 
+  owned by Meta embodied in the Llama Materials to use, reproduce, distribute, copy, create derivative works 
+  of, and make modifications to the Llama Materials.  
+
+  b. Redistribution and Use.  
+
+  i. If you distribute or make available the Llama Materials (or any derivative works thereof), 
+  or a product or service (including another AI model) that contains any of them, you shall (A) provide
+  a copy of this Agreement with any such Llama Materials; and (B) prominently display “Built with Llama”
+  on a related website, user interface, blogpost, about page, or product documentation. If you use the
+  Llama Materials or any outputs or results of the Llama Materials to create, train, fine tune, or
+  otherwise improve an AI model, which is distributed or made available, you shall also include “Llama”
+  at the beginning of any such AI model name.
+
+  ii. If you receive Llama Materials, or any derivative works thereof, from a Licensee as part
+  of an integrated end user product, then Section 2 of this Agreement will not apply to you. 
+
+  iii. You must retain in all copies of the Llama Materials that you distribute the 
+  following attribution notice within a “Notice” text file distributed as a part of such copies: 
+  “Llama 3.2 is licensed under the Llama 3.2 Community License, Copyright © Meta Platforms,
+  Inc. All Rights Reserved.”
+
+  iv. Your use of the Llama Materials must comply with applicable laws and regulations
+  (including trade compliance laws and regulations) and adhere to the Acceptable Use Policy for
+  the Llama Materials (available at https://www.llama.com/llama3_2/use-policy), which is hereby 
+  incorporated by reference into this Agreement.
+    
+  2. Additional Commercial Terms. If, on the Llama 3.2 version release date, the monthly active users
+  of the products or services made available by or for Licensee, or Licensee’s affiliates, 
+  is greater than 700 million monthly active users in the preceding calendar month, you must request 
+  a license from Meta, which Meta may grant to you in its sole discretion, and you are not authorized to
+  exercise any of the rights under this Agreement unless or until Meta otherwise expressly grants you such rights.
+  
+  3. Disclaimer of Warranty. UNLESS REQUIRED BY APPLICABLE LAW, THE LLAMA MATERIALS AND ANY OUTPUT AND 
+  RESULTS THEREFROM ARE PROVIDED ON AN “AS IS” BASIS, WITHOUT WARRANTIES OF ANY KIND, AND META DISCLAIMS
+  ALL WARRANTIES OF ANY KIND, BOTH EXPRESS AND IMPLIED, INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES
+  OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU ARE SOLELY RESPONSIBLE
+  FOR DETERMINING THE APPROPRIATENESS OF USING OR REDISTRIBUTING THE LLAMA MATERIALS AND ASSUME ANY RISKS ASSOCIATED
+  WITH YOUR USE OF THE LLAMA MATERIALS AND ANY OUTPUT AND RESULTS.
+  
+  4. Limitation of Liability. IN NO EVENT WILL META OR ITS AFFILIATES BE LIABLE UNDER ANY THEORY OF LIABILITY, 
+  WHETHER IN CONTRACT, TORT, NEGLIGENCE, PRODUCTS LIABILITY, OR OTHERWISE, ARISING OUT OF THIS AGREEMENT, 
+  FOR ANY LOST PROFITS OR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, INCIDENTAL, EXEMPLARY OR PUNITIVE DAMAGES, EVEN 
+  IF META OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF ANY OF THE FOREGOING.
+  
+  5. Intellectual Property.
+  
+  a. No trademark licenses are granted under this Agreement, and in connection with the Llama Materials, 
+  neither Meta nor Licensee may use any name or mark owned by or associated with the other or any of its affiliates, 
+  except as required for reasonable and customary use in describing and redistributing the Llama Materials or as 
+  set forth in this Section 5(a). Meta hereby grants you a license to use “Llama” (the “Mark”) solely as required 
+  to comply with the last sentence of Section 1.b.i. You will comply with Meta’s brand guidelines (currently accessible 
+  at https://about.meta.com/brand/resources/meta/company-brand/). All goodwill arising out of your use of the Mark 
+  will inure to the benefit of Meta.
+  
+  b. Subject to Meta’s ownership of Llama Materials and derivatives made by or for Meta, with respect to any
+  derivative works and modifications of the Llama Materials that are made by you, as between you and Meta,
+  you are and will be the owner of such derivative works and modifications.
+
+  c. If you institute litigation or other proceedings against Meta or any entity (including a cross-claim or
+  counterclaim in a lawsuit) alleging that the Llama Materials or Llama 3.2 outputs or results, or any portion
+  of any of the foregoing, constitutes infringement of intellectual property or other rights owned or licensable
+  by you, then any licenses granted to you under this Agreement shall terminate as of the date such litigation or
+  claim is filed or instituted. You will indemnify and hold harmless Meta from and against any claim by any third
+  party arising out of or related to your use or distribution of the Llama Materials.
+  
+  6. Term and Termination. The term of this Agreement will commence upon your acceptance of this Agreement or access
+  to the Llama Materials and will continue in full force and effect until terminated in accordance with the terms
+  and conditions herein. Meta may terminate this Agreement if you are in breach of any term or condition of this
+  Agreement. Upon termination of this Agreement, you shall delete and cease use of the Llama Materials. Sections 3,
+  4 and 7 shall survive the termination of this Agreement. 
+  
+  7. Governing Law and Jurisdiction. This Agreement will be governed and construed under the laws of the State of 
+  California without regard to choice of law principles, and the UN Convention on Contracts for the International
+  Sale of Goods does not apply to this Agreement. The courts of California shall have exclusive jurisdiction of
+  any dispute arising out of this Agreement. 
+  
+  ### Llama 3.2 Acceptable Use Policy
+  
+  Meta is committed to promoting safe and fair use of its tools and features, including Llama 3.2. 
+  If you access or use Llama 3.2, you agree to this Acceptable Use Policy (“**Policy**”). 
+  The most recent copy of this policy can be found at
+  [https://www.llama.com/llama3_2/use-policy](https://www.llama.com/llama3_2/use-policy).
+  
+  #### Prohibited Uses
+  
+  We want everyone to use Llama 3.2 safely and responsibly. You agree you will not use, or allow others to use, Llama 3.2 to:
+  
+  1. Violate the law or others’ rights, including to:
+      1. Engage in, promote, generate, contribute to, encourage, plan, incite, or further illegal or unlawful activity or content, such as:
+          1. Violence or terrorism
+          2. Exploitation or harm to children, including the solicitation, creation, acquisition, or dissemination of child exploitative content or failure to report Child Sexual Abuse Material
+          3. Human trafficking, exploitation, and sexual violence
+          4. The illegal distribution of information or materials to minors, including obscene materials, or failure to employ legally required age-gating in connection with such information or materials.
+          5. Sexual solicitation
+          6. Any other criminal activity
+      1. Engage in, promote, incite, or facilitate the harassment, abuse, threatening, or bullying of individuals or groups of individuals
+      2. Engage in, promote, incite, or facilitate discrimination or other unlawful or harmful conduct in the provision of employment, employment benefits, credit, housing, other economic benefits, or other essential goods and services
+      3. Engage in the unauthorized or unlicensed practice of any profession including, but not limited to, financial, legal, medical/health, or related professional practices
+      4. Collect, process, disclose, generate, or infer private or sensitive information about individuals, including information about individuals’ identity, health, or demographic information, unless you have obtained the right to do so in accordance with applicable law
+      5. Engage in or facilitate any action or generate any content that infringes, misappropriates, or otherwise violates any third-party rights, including the outputs or results of any products or services using the Llama Materials
+      6. Create, generate, or facilitate the creation of malicious code, malware, computer viruses or do anything else that could disable, overburden, interfere with or impair the proper working, integrity, operation or appearance of a website or computer system
+      7. Engage in any action, or facilitate any action, to intentionally circumvent or remove usage restrictions or other safety measures, or to enable functionality disabled by Meta 
+  2. Engage in, promote, incite, facilitate, or assist in the planning or development of activities that present a risk of death or bodily harm to individuals, including use of Llama 3.2 related to the following:
+      8. Military, warfare, nuclear industries or applications, espionage, use for materials or activities that are subject to the International Traffic Arms Regulations (ITAR) maintained by the United States Department of State or to the U.S. Biological Weapons Anti-Terrorism Act of 1989 or the Chemical Weapons Convention Implementation Act of 1997
+      9. Guns and illegal weapons (including weapon development)
+      10. Illegal drugs and regulated/controlled substances
+      11. Operation of critical infrastructure, transportation technologies, or heavy machinery
+      12. Self-harm or harm to others, including suicide, cutting, and eating disorders
+      13. Any content intended to incite or promote violence, abuse, or any infliction of bodily harm to an individual
+  3. Intentionally deceive or mislead others, including use of Llama 3.2 related to the following:
+      14. Generating, promoting, or furthering fraud or the creation or promotion of disinformation
+      15. Generating, promoting, or furthering defamatory content, including the creation of defamatory statements, images, or other content
+      16. Generating, promoting, or further distributing spam
+      17. Impersonating another individual without consent, authorization, or legal right
+      18. Representing that the use of Llama 3.2 or outputs are human-generated
+      19. Generating or facilitating false online engagement, including fake reviews and other means of fake online engagement 
+  4. Fail to appropriately disclose to end users any known dangers of your AI system
+  5. Interact with third party tools, models, or software designed to generate unlawful content or engage in unlawful or harmful conduct and/or represent that the outputs of such tools, models, or software are associated with Meta or Llama 3.2
+
+
+  With respect to any multimodal models included in Llama 3.2, the rights granted under Section 1(a) of the Llama 3.2 Community License Agreement are not being granted to you if you are an individual domiciled in, or a company with a principal place of business in, the European Union. This restriction does not apply to end users of a product or service that incorporates any such multimodal models.
+
+
+  Please report any violation of this Policy, software “bug,” or other problems that could lead to a violation of this Policy through one of the following means:
+
+
+  * Reporting issues with the model: [https://github.com/meta-llama/llama-models/issues](https://l.workplace.com/l.php?u=https%3A%2F%2Fgithub.com%2Fmeta-llama%2Fllama-models%2Fissues&h=AT0qV8W9BFT6NwihiOHRuKYQM_UnkzN_NmHMy91OT55gkLpgi4kQupHUl0ssR4dQsIQ8n3tfd0vtkobvsEvt1l4Ic6GXI2EeuHV8N08OG2WnbAmm0FL4ObkazC6G_256vN0lN9DsykCvCqGZ)
+  
+  * Reporting risky content generated by the model: [developers.facebook.com/llama_output_feedback](http://developers.facebook.com/llama_output_feedback)
+  
+  * Reporting bugs and security concerns: [facebook.com/whitehat/info](http://facebook.com/whitehat/info)
+  
+  * Reporting violations of the Acceptable Use Policy or unlicensed uses of Llama 3.2: LlamaUseReport@meta.com
+extra_gated_fields:
+  First Name: text
+  Last Name: text
+  Date of birth: date_picker
+  Country: country
+  Affiliation: text
+  Job title:
+    type: select
+    options:
+    - Student
+    - Research Graduate
+    - AI researcher
+    - AI developer/engineer
+    - Reporter
+    - Other
+  geo: ip_location
+  By clicking Submit below I accept the terms of the license and acknowledge that the information I provide will be collected stored processed and shared in accordance with the Meta Privacy Policy: checkbox
+extra_gated_description: >-
+  The information you provide will be collected, stored, processed and shared in
+  accordance with the [Meta Privacy
+  Policy](https://www.facebook.com/privacy/policy/).
+extra_gated_button_content: Submit
+---
+
+## Model Information
+
+The Meta Llama 3.2 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction-tuned generative models in 1B and 3B sizes (text in/text out). The Llama 3.2 instruction-tuned text only models are optimized for multilingual dialogue use cases, including agentic retrieval and summarization tasks. They outperform many of the available open source and closed chat models on common industry benchmarks.
+
+**Model Developer:** Meta
+
+**Model Architecture:** Llama 3.2 is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety. 
+
+|  | Training Data | Params | Input modalities | Output modalities | Context Length | GQA | Shared Embeddings | Token count | Knowledge cutoff |
+| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |
+| Llama 3.2 (text only)  | A new mix of publicly available online data. | 1B (1.23B) | Multilingual Text | Multilingual Text and code  | 128k | Yes | Yes | Up to 9T tokens | December 2023 |
+|  |  | 3B (3.21B) | Multilingual Text | Multilingual Text and code  |  |  |  |  |  |
+
+**Supported Languages:** English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai are officially supported. Llama 3.2 has been trained on a broader collection of languages than these 8 supported languages. Developers may fine-tune Llama 3.2 models for languages beyond these supported languages, provided they comply with the Llama 3.2 Community License and the Acceptable Use Policy. Developers are always expected to ensure that their deployments, including those that involve additional languages, are completed safely and responsibly.
+
+**Llama 3.2 Model Family:** Token counts refer to pretraining data only. All model versions use Grouped-Query Attention (GQA) for improved inference scalability.
+
+**Model Release Date:** Sept 25, 2024
+
+**Status:** This is a static model trained on an offline dataset. Future versions may be released that improve model capabilities and safety. 
+
+**License:** Use of Llama 3.2 is governed by the [Llama 3.2 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE) (a custom, commercial license agreement).
+
+**Feedback:** Where to send questions or comments about the model Instructions on how to provide feedback or comments on the model can be found in the model [README](https://github.com/meta-llama/llama-models/tree/main/models/llama3_2). For more technical information about generation parameters and recipes for how to use Llama 3.2 in applications, please go [here](https://github.com/meta-llama/llama-recipes). 
+
+## Intended Use
+
+**Intended Use Cases:** Llama 3.2 is intended for commercial and research use in multiple languages. Instruction tuned text only models are intended for assistant-like chat and agentic applications like knowledge retrieval and summarization, mobile AI powered writing assistants and query and prompt rewriting. Pretrained models can be adapted for a variety of additional natural language generation tasks. 
+
+**Out of Scope:** Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in any other way that is prohibited by the Acceptable Use Policy and Llama 3.2 Community License. Use in languages beyond those explicitly referenced as supported in this model card.
+
+## How to use
+
+This repository contains two versions of Llama-3.2-3B, for use with transformers and with the original `llama` codebase.
+
+### Use with transformers
+
+Starting with transformers >= 4.43.0 onward, you can run conversational inference using the Transformers pipeline abstraction or by leveraging the Auto classes with the generate() function.
+
+Make sure to update your transformers installation via pip install --upgrade transformers.
+
+```python
+import torch
+from transformers import pipeline
+
+model_id = "meta-llama/Llama-3.2-3B"
+
+pipe = pipeline(
+    "text-generation", 
+    model=model_id, 
+    torch_dtype=torch.bfloat16, 
+    device_map="auto"
+)
+
+pipe("The key to life is")
+```
+
+### Use with `llama`
+
+Please, follow the instructions in the [repository](https://github.com/meta-llama/llama).
+
+To download Original checkpoints, see the example command below leveraging `huggingface-cli`:
+
+```
+huggingface-cli download meta-llama/Llama-3.2-3B --include "original/*" --local-dir Llama-3.2-3B
+```
+
+## Hardware and Software
+
+**Training Factors:** We used custom training libraries, Meta's custom built GPU cluster, and production infrastructure for pretraining. Fine-tuning, annotation, and evaluation were also performed on production infrastructure.
+
+**Training Energy Use:** Training utilized a cumulative of **916k** GPU hours of computation on H100-80GB (TDP of 700W) type hardware, per the table below. Training time is the total GPU time required for training each model and power consumption is the peak power capacity per GPU device used, adjusted for power usage efficiency. 
+
+## 
+
+**Training Greenhouse Gas Emissions:** Estimated total location-based greenhouse gas emissions were **240** tons CO2eq for training. Since 2020, Meta has maintained net zero greenhouse gas emissions in its global operations and matched 100% of its electricity use with renewable energy; therefore, the total market-based greenhouse gas emissions for training were 0 tons CO2eq.
+
+|  | Training Time (GPU hours) | Logit Generation Time (GPU Hours) | Training Power Consumption (W) | Training Location-Based Greenhouse Gas Emissions (tons CO2eq) | Training Market-Based Greenhouse Gas Emissions (tons CO2eq) |
+| :---- | :---: | ----- | :---: | :---: | :---: |
+| Llama 3.2 1B | 370k | \- | 700 | 107 | 0 |
+| Llama 3.2 3B | 460k | \- | 700 | 133 | 0 |
+| Total | 830k |         86k |  | 240 | 0 |
+
+The methodology used to determine training energy use and greenhouse gas emissions can be found [here](https://arxiv.org/pdf/2204.05149). Since Meta is openly releasing these models, the training energy use and greenhouse gas emissions will not be incurred by others.
+
+## Training Data
+
+**Overview:** Llama 3.2 was pretrained on up to 9 trillion tokens of data from publicly available sources. For the 1B and 3B Llama 3.2 models, we incorporated logits from the Llama 3.1 8B and 70B models into the pretraining stage of the model development, where outputs (logits) from these larger models were used as token-level targets. Knowledge distillation was used after pruning to recover performance. In post-training we used a similar recipe as Llama 3.1 and produced final chat models by doing several rounds of alignment on top of the pre-trained model. Each round involved Supervised Fine-Tuning (SFT), Rejection Sampling (RS), and Direct Preference Optimization (DPO).
+
+**Data Freshness:** The pretraining data has a cutoff of December 2023\.
+
+## Benchmarks \- English Text
+
+In this section, we report the results for Llama 3.2 models on standard automatic benchmarks. For all these evaluations, we used our internal evaluations library. 
+
+### Base Pretrained Models 
+
+| Category | Benchmark | \# Shots | Metric | Llama 3.2 1B | Llama 3.2 3B | Llama 3.1 8B |
+| ----- | ----- | :---: | :---: | :---: | :---: | :---: |
+| General | MMLU | 5 | macro\_avg/acc\_char | 32.2 | 58 | 66.7 |
+|  | AGIEval English | 3-5 | average/acc\_char | 23.3 | 39.2 | 47.8 |
+|  | ARC-Challenge | 25 | acc\_char | 32.8 | 69.1 | 79.7 |
+| Reading comprehension | SQuAD | 1 | em | 49.2 | 67.7 | 77 |
+|  | QuAC (F1) | 1 | f1 | 37.9 | 42.9 | 44.9 |
+|  | DROP (F1) | 3 | f1 | 28.0 | 45.2 | 59.5 |
+| Long Context | Needle in Haystack | 0 | em | 96.8 | 1 | 1 |
+
+### Instruction Tuned Models
+
+| Capability |  | Benchmark | \# Shots | Metric | Llama 3.2 1B | Llama 3.2 3B | Llama 3.1 8B |
+| :---: | ----- | :---: | :---: | :---: | :---: | :---: | :---: |
+| General |  | MMLU | 5 | macro\_avg/acc | 49.3 | 63.4 | 69.4 |
+| Re-writing |  | Open-rewrite eval | 0 | micro\_avg/rougeL | 41.6 | 40.1 | 40.9 |
+| Summarization |  | TLDR9+ (test) | 1 | rougeL | 16.8 | 19.0 | 17.2 |
+| Instruction following |  | IFEval | 0 | avg(prompt/instruction acc loose/strict) | 59.5 | 77.4 | 80.4 |
+| Math |  | GSM8K (CoT) | 8 | em\_maj1@1 | 44.4 | 77.7 | 84.5 |
+|  |  | MATH (CoT) | 0 | final\_em | 30.6 | 47.3 | 51.9 |
+| Reasoning |  | ARC-C | 0 | acc | 59.4 | 78.6 | 83.4 |
+|  |  | GPQA | 0 | acc | 27.2 | 32.8 | 32.8 |
+|  |  | Hellaswag | 0 | acc | 41.2 | 69.8 | 78.7 |
+| Tool Use |  | BFCL V2 | 0 | acc | 25.7 | 67.0 | 70.9 |
+|  |  | Nexus | 0 | macro\_avg/acc | 13.5 | 34.3 | 38.5 |
+| Long Context |  | InfiniteBench/En.QA | 0 | longbook\_qa/f1 | 20.3 | 19.8 | 27.3 |
+|  |  | InfiniteBench/En.MC | 0 | longbook\_choice/acc | 38.0 | 63.3 | 72.2 |
+|  |  | NIH/Multi-needle | 0 | recall | 75.0 | 84.7 | 98.8 |
+| Multilingual |  | MGSM (CoT) | 0 | em | 24.5 | 58.2 | 68.9 |
+
+### Multilingual Benchmarks
+
+| Category | Benchmark | Language | Llama 3.2 1B | Llama 3.2 3B | Llama 3.1 8B |
+| :---: | :---: | :---: | :---: | :---: | :---: |
+| General | MMLU (5-shot, macro\_avg/acc) | Portuguese | 39.82 | 54.48 | 62.12 |
+|  |  | Spanish | 41.5 | 55.1 | 62.5 |
+|  |  | Italian | 39.8 | 53.8 | 61.6 |
+|  |  | German | 39.2 | 53.3 | 60.6 |
+|  |  | French | 40.5 | 54.6 | 62.3 |
+|  |  | Hindi | 33.5 | 43.3 | 50.9 |
+|  |  | Thai | 34.7 | 44.5 | 50.3 |
+
+## Responsibility & Safety
+
+As part of our Responsible release approach, we followed a three-pronged strategy to managing trust & safety risks:
+
+1. Enable developers to deploy helpful, safe and flexible experiences for their target audience and for the use cases supported by Llama   
+2. Protect developers against adversarial users aiming to exploit Llama capabilities to potentially cause harm  
+3. Provide protections for the community to help prevent the misuse of our models
+
+### Responsible Deployment 
+
+**Approach:** Llama is a foundational technology designed to be used in a variety of use cases. Examples on how Meta’s Llama models have been responsibly deployed can be found in our [Community Stories webpage](https://llama.meta.com/community-stories/). Our approach is to build the most helpful models, enabling the world to benefit from the technology power, by aligning our model safety for generic use cases and addressing a standard set of harms. Developers are then in the driver’s seat to tailor safety for their use cases, defining their own policies and deploying the models with the necessary safeguards in their Llama systems. Llama 3.2 was developed following the best practices outlined in our [Responsible Use Guide](https://llama.meta.com/responsible-use-guide/). 
+
+#### Llama 3.2 Instruct 
+
+**Objective:** Our main objectives for conducting safety fine-tuning are to provide the research community with a valuable resource for studying the robustness of safety fine-tuning, as well as to offer developers a readily available, safe, and powerful model for various applications to reduce the developer workload to deploy safe AI systems. We implemented the same set of safety mitigations as in Llama 3, and you can learn more about these in the Llama 3 [paper](https://ai.meta.com/research/publications/the-llama-3-herd-of-models/). 
+
+**Fine-Tuning Data:** We employ a multi-faceted approach to data collection, combining human-generated data from our vendors with synthetic data to mitigate potential safety risks. We’ve developed many large language model (LLM)-based classifiers that enable us to thoughtfully select high-quality prompts and responses, enhancing data quality control. 
+
+**Refusals and Tone:** Building on the work we started with Llama 3, we put a great emphasis on model refusals to benign prompts as well as refusal tone. We included both borderline and adversarial prompts in our safety data strategy, and modified our safety data responses to follow tone guidelines. 
+
+#### Llama 3.2 Systems
+
+**Safety as a System:** Large language models, including Llama 3.2, **are not designed to be deployed in isolation** but instead should be deployed as part of an overall AI system with additional safety guardrails as required. Developers are expected to deploy system safeguards when building agentic systems. Safeguards are key to achieve the right helpfulness-safety alignment as well as mitigating safety and security risks inherent to the system and any integration of the model or system with external tools. As part of our responsible release approach, we provide the community with [safeguards](https://llama.meta.com/trust-and-safety/) that developers should deploy with Llama models or other LLMs, including Llama Guard, Prompt Guard and Code Shield. All our [reference implementations](https://github.com/meta-llama/llama-agentic-system) demos contain these safeguards by default so developers can benefit from system-level safety out-of-the-box. 
+
+### New Capabilities and Use Cases
+
+**Technological Advancement:** Llama releases usually introduce new capabilities that require specific considerations in addition to the best practices that generally apply across all Generative AI use cases. For prior release capabilities also supported by Llama 3.2, see [Llama 3.1 Model Card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md), as the same considerations apply here as well.
+
+**Constrained Environments:** Llama 3.2 1B and 3B models are expected to be deployed in highly constrained environments, such as mobile devices. LLM Systems using smaller models will have a different alignment profile and safety/helpfulness tradeoff than more complex, larger systems. Developers should ensure the safety of their system meets the requirements of their use case. We recommend using lighter system safeguards for such use cases, like Llama Guard 3-1B or its mobile-optimized version. 
+
+### Evaluations
+
+**Scaled Evaluations:** We built dedicated, adversarial evaluation datasets and evaluated systems composed of Llama models and Purple Llama safeguards to filter input prompt and output response. It is important to evaluate applications in context, and we recommend building dedicated evaluation dataset for your use case.
+
+**Red Teaming:** We conducted recurring red teaming exercises with the goal of discovering risks via adversarial prompting and we used the learnings to improve our benchmarks and safety tuning datasets. We partnered early with subject-matter experts in critical risk areas to understand the nature of these real-world harms and how such models may lead to unintended harm for society. Based on these conversations, we derived a set of adversarial goals for the red team to attempt to achieve, such as extracting harmful information or reprogramming the model to act in a potentially harmful capacity. The red team consisted of experts in cybersecurity, adversarial machine learning, responsible AI, and integrity in addition to multilingual content specialists with background in integrity issues in specific geographic markets.
+
+### Critical Risks 
+
+In addition to our safety work above, we took extra care on measuring and/or mitigating the following critical risk areas:
+
+**1\. CBRNE (Chemical, Biological, Radiological, Nuclear, and Explosive Weapons):** Llama 3.2 1B and 3B models are smaller and less capable derivatives of Llama 3.1. For Llama 3.1 70B and 405B, to assess risks related to proliferation of chemical and biological weapons, we performed uplift testing designed to assess whether use of Llama 3.1 models could meaningfully increase the capabilities of malicious actors to plan or carry out attacks using these types of weapons and have determined that such testing also applies to the smaller 1B and 3B models. 
+
+**2\. Child Safety:** Child Safety risk assessments were conducted using a team of experts, to assess the model’s capability to produce outputs that could result in Child Safety risks and inform on any necessary and appropriate risk mitigations via fine tuning. We leveraged those expert red teaming sessions to expand the coverage of our evaluation benchmarks through Llama 3 model development. For Llama 3, we conducted new in-depth sessions using objective based methodologies to assess the model risks along multiple attack vectors including the additional languages Llama 3 is trained on. We also partnered with content specialists to perform red teaming exercises assessing potentially violating content while taking account of market specific nuances or experiences. 
+
+**3\. Cyber Attacks:** For Llama 3.1 405B, our cyber attack uplift study investigated whether LLMs can enhance human capabilities in hacking tasks, both in terms of skill level and speed.
+Our attack automation study focused on evaluating the capabilities of LLMs when used as autonomous agents in cyber offensive operations, specifically in the context of ransomware attacks. This evaluation was distinct from previous studies that considered LLMs as interactive assistants. The primary objective was to assess whether these models could effectively function as independent agents in executing complex cyber-attacks without human intervention. Because Llama 3.2’s 1B and 3B models are smaller and less capable models than Llama 3.1 405B, we broadly believe that the testing conducted for the 405B model also applies to Llama 3.2 models.
+
+### Community 
+
+**Industry Partnerships:** Generative AI safety requires expertise and tooling, and we believe in the strength of the open community to accelerate its progress. We are active members of open consortiums, including the AI Alliance, Partnership on AI and MLCommons, actively contributing to safety standardization and transparency. We encourage the community to adopt taxonomies like the MLCommons Proof of Concept evaluation to facilitate collaboration and transparency on safety and content evaluations. Our Purple Llama tools are open sourced for the community to use and widely distributed across ecosystem partners including cloud service providers. We encourage community contributions to our [Github repository](https://github.com/meta-llama/PurpleLlama).
+
+**Grants:** We also set up the [Llama Impact Grants](https://llama.meta.com/llama-impact-grants/) program to identify and support the most compelling applications of Meta’s Llama model for societal benefit across three categories: education, climate and open innovation. The 20 finalists from the hundreds of applications can be found [here](https://llama.meta.com/llama-impact-grants/#finalists). 
+
+**Reporting:** Finally, we put in place a set of resources including an [output reporting mechanism](https://developers.facebook.com/llama_output_feedback) and [bug bounty program](https://www.facebook.com/whitehat) to continuously improve the Llama technology with the help of the community.
+
+## Ethical Considerations and Limitations
+
+**Values:** The core values of Llama 3.2 are openness, inclusivity and helpfulness. It is meant to serve everyone, and to work for a wide range of use cases. It is thus designed to be accessible to people across many different backgrounds, experiences and perspectives. Llama 3.2 addresses users and their needs as they are, without insertion unnecessary judgment or normativity, while reflecting the understanding that even content that may appear problematic in some cases can serve valuable purposes in others. It respects the dignity and autonomy of all users, especially in terms of the values of free thought and expression that power innovation and progress. 
+
+**Testing:** Llama 3.2 is a new technology, and like any new technology, there are risks associated with its use. Testing conducted to date has not covered, nor could it cover, all scenarios. For these reasons, as with all LLMs, Llama 3.2’s potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying any applications of Llama 3.2 models, developers should perform safety testing and tuning tailored to their specific applications of the model. Please refer to available resources including our [Responsible Use Guide](https://llama.meta.com/responsible-use-guide), [Trust and Safety](https://llama.meta.com/trust-and-safety/) solutions, and other [resources](https://llama.meta.com/docs/get-started/) to learn more about responsible development.   

Makefile

+# Run nanotron's tests and examples's tests
+test:
+	pytest \
+        --color=yes \
+        --durations=0 \
+        --ignore tests/fp8 \
+        --verbose \
+        tests/
+
+	pip install -r examples/doremi/requirements.txt
+	pytest \
+        --color=yes \
+        --durations=0 \
+        --ignore tests/fp8 \
+        --verbose \
+        examples/doremi/tests/
+
+	pip install -r examples/llama/requirements.txt
+	pytest \
+        --color=yes \
+        --verbose \
+        examples/llama/tests/

Meta-Llama-3.1-8B/README.md

+---
+language:
+- en
+- de
+- fr
+- it
+- pt
+- hi
+- es
+- th
+pipeline_tag: text-generation
+tags:
+- facebook
+- meta
+- pytorch
+- llama
+- llama-3
+license: llama3.1
+extra_gated_prompt: >-
+  ### LLAMA 3.1 COMMUNITY LICENSE AGREEMENT
+
+  Llama 3.1 Version Release Date: July 23, 2024
+
+  "Agreement" means the terms and conditions for use, reproduction, distribution
+  and modification of the  Llama Materials set forth herein.
+
+  "Documentation" means the specifications, manuals and documentation
+  accompanying Llama 3.1 distributed by Meta at
+  https://llama.meta.com/doc/overview.
+
+  "Licensee" or "you" means you, or your employer or any other person or entity
+  (if you are entering into this Agreement on such person or entity’s behalf),
+  of the age required under applicable laws, rules or regulations to provide
+  legal consent and that has legal authority to bind your employer or such other
+  person or entity if you are entering in this Agreement on their behalf.
+
+  "Llama 3.1" means the foundational large language models and software and
+  algorithms, including machine-learning model code, trained model weights,
+  inference-enabling code, training-enabling code, fine-tuning enabling code and
+  other elements of the foregoing distributed by Meta at
+  https://llama.meta.com/llama-downloads.
+
+  "Llama Materials" means, collectively, Meta’s proprietary Llama 3.1 and
+  Documentation (and any portion thereof) made available under this Agreement.
+
+  "Meta" or "we" means Meta Platforms Ireland Limited (if you are located in or,
+  if you are an entity, your principal place of business is in the EEA or
+  Switzerland) and Meta Platforms, Inc. (if you are located outside of the EEA
+  or Switzerland).
+     
+  1. License Rights and Redistribution.
+
+  a. Grant of Rights. You are granted a non-exclusive, worldwide,
+  non-transferable and royalty-free limited license under Meta’s intellectual
+  property or other rights owned by Meta embodied in the Llama Materials to use,
+  reproduce, distribute, copy, create derivative works of, and make
+  modifications to the Llama Materials.
+
+  b. Redistribution and Use.
+
+  i. If you distribute or make available the Llama Materials (or any derivative
+  works thereof), or a product or service (including another AI model) that
+  contains any of them, you shall (A) provide a copy of this Agreement with any
+  such Llama Materials; and (B) prominently display “Built with Llama” on a
+  related website, user interface, blogpost, about page, or product
+  documentation. If you use the Llama Materials or any outputs or results of the
+  Llama Materials to create, train, fine tune, or otherwise improve an AI model,
+  which is distributed or made available, you shall also include “Llama” at the
+  beginning of any such AI model name.
+
+  ii. If you receive Llama Materials, or any derivative works thereof, from a
+  Licensee as part  of an integrated end user product, then Section 2 of this
+  Agreement will not apply to you.
+
+  iii. You must retain in all copies of the Llama Materials that you distribute
+  the following attribution notice within a “Notice” text file distributed as a
+  part of such copies: “Llama 3.1 is licensed under the Llama 3.1 Community
+  License, Copyright © Meta Platforms, Inc. All Rights Reserved.”
+
+  iv. Your use of the Llama Materials must comply with applicable laws and
+  regulations (including trade compliance laws and regulations) and adhere to
+  the Acceptable Use Policy for the Llama Materials (available at
+  https://llama.meta.com/llama3_1/use-policy), which is hereby incorporated by
+  reference into this Agreement.
+
+  2. Additional Commercial Terms. If, on the Llama 3.1 version release date, the
+  monthly active users of the products or services made available by or for
+  Licensee, or Licensee’s affiliates, is greater than 700 million monthly active
+  users in the preceding calendar month, you must request a license from Meta,
+  which Meta may grant to you in its sole discretion, and you are not authorized
+  to exercise any of the rights under this Agreement unless or until Meta
+  otherwise expressly grants you such rights.
+
+  3. Disclaimer of Warranty. UNLESS REQUIRED BY APPLICABLE LAW, THE LLAMA
+  MATERIALS AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED ON AN “AS IS”
+  BASIS, WITHOUT WARRANTIES OF ANY KIND, AND META DISCLAIMS ALL WARRANTIES OF
+  ANY KIND, BOTH EXPRESS AND IMPLIED, INCLUDING, WITHOUT LIMITATION, ANY
+  WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A
+  PARTICULAR PURPOSE. YOU ARE SOLELY RESPONSIBLE FOR DETERMINING THE
+  APPROPRIATENESS OF USING OR REDISTRIBUTING THE LLAMA MATERIALS AND ASSUME ANY
+  RISKS ASSOCIATED WITH YOUR USE OF THE LLAMA MATERIALS AND ANY OUTPUT AND
+  RESULTS.
+
+  4. Limitation of Liability. IN NO EVENT WILL META OR ITS AFFILIATES BE LIABLE
+  UNDER ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, TORT, NEGLIGENCE, PRODUCTS
+  LIABILITY, OR OTHERWISE, ARISING OUT OF THIS AGREEMENT, FOR ANY LOST PROFITS
+  OR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, INCIDENTAL, EXEMPLARY OR PUNITIVE
+  DAMAGES, EVEN IF META OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY
+  OF ANY OF THE FOREGOING.
+
+  5. Intellectual Property.
+
+  a. No trademark licenses are granted under this Agreement, and in connection
+  with the Llama Materials, neither Meta nor Licensee may use any name or mark
+  owned by or associated with the other or any of its affiliates, except as
+  required for reasonable and customary use in describing and redistributing the
+  Llama Materials or as set forth in this Section 5(a). Meta hereby grants you a
+  license to use “Llama” (the “Mark”) solely as required to comply with the last
+  sentence of Section 1.b.i. You will comply with Meta’s brand guidelines
+  (currently accessible at
+  https://about.meta.com/brand/resources/meta/company-brand/ ). All goodwill
+  arising out of your use of the Mark will inure to the benefit of Meta.
+
+  b. Subject to Meta’s ownership of Llama Materials and derivatives made by or
+  for Meta, with respect to any derivative works and modifications of the Llama
+  Materials that are made by you, as between you and Meta, you are and will be
+  the owner of such derivative works and modifications.
+
+  c. If you institute litigation or other proceedings against Meta or any entity
+  (including a cross-claim or counterclaim in a lawsuit) alleging that the Llama
+  Materials or Llama 3.1 outputs or results, or any portion of any of the
+  foregoing, constitutes infringement of intellectual property or other rights
+  owned or licensable by you, then any licenses granted to you under this
+  Agreement shall terminate as of the date such litigation or claim is filed or
+  instituted. You will indemnify and hold harmless Meta from and against any
+  claim by any third party arising out of or related to your use or distribution
+  of the Llama Materials.
+
+  6. Term and Termination. The term of this Agreement will commence upon your
+  acceptance of this Agreement or access to the Llama Materials and will
+  continue in full force and effect until terminated in accordance with the
+  terms and conditions herein. Meta may terminate this Agreement if you are in
+  breach of any term or condition of this Agreement. Upon termination of this
+  Agreement, you shall delete and cease use of the Llama Materials. Sections 3,
+  4 and 7 shall survive the termination of this Agreement.
+
+  7. Governing Law and Jurisdiction. This Agreement will be governed and
+  construed under the laws of the State of California without regard to choice
+  of law principles, and the UN Convention on Contracts for the International
+  Sale of Goods does not apply to this Agreement. The courts of California shall
+  have exclusive jurisdiction of any dispute arising out of this Agreement.
+
+  ### Llama 3.1 Acceptable Use Policy
+
+  Meta is committed to promoting safe and fair use of its tools and features,
+  including Llama 3.1. If you access or use Llama 3.1, you agree to this
+  Acceptable Use Policy (“Policy”). The most recent copy of this policy can be
+  found at
+  [https://llama.meta.com/llama3_1/use-policy](https://llama.meta.com/llama3_1/use-policy)
+
+  #### Prohibited Uses
+
+  We want everyone to use Llama 3.1 safely and responsibly. You agree you will
+  not use, or allow others to use, Llama 3.1 to:
+   1. Violate the law or others’ rights, including to:
+      1. Engage in, promote, generate, contribute to, encourage, plan, incite, or further illegal or unlawful activity or content, such as:
+          1. Violence or terrorism
+          2. Exploitation or harm to children, including the solicitation, creation, acquisition, or dissemination of child exploitative content or failure to report Child Sexual Abuse Material
+          3. Human trafficking, exploitation, and sexual violence
+          4. The illegal distribution of information or materials to minors, including obscene materials, or failure to employ legally required age-gating in connection with such information or materials.
+          5. Sexual solicitation
+          6. Any other criminal activity
+      3. Engage in, promote, incite, or facilitate the harassment, abuse, threatening, or bullying of individuals or groups of individuals
+      4. Engage in, promote, incite, or facilitate discrimination or other unlawful or harmful conduct in the provision of employment, employment benefits, credit, housing, other economic benefits, or other essential goods and services
+      5. Engage in the unauthorized or unlicensed practice of any profession including, but not limited to, financial, legal, medical/health, or related professional practices
+      6. Collect, process, disclose, generate, or infer health, demographic, or other sensitive personal or private information about individuals without rights and consents required by applicable laws
+      7. Engage in or facilitate any action or generate any content that infringes, misappropriates, or otherwise violates any third-party rights, including the outputs or results of any products or services using the Llama Materials
+      8. Create, generate, or facilitate the creation of malicious code, malware, computer viruses or do anything else that could disable, overburden, interfere with or impair the proper working, integrity, operation or appearance of a website or computer system
+  2. Engage in, promote, incite, facilitate, or assist in the planning or
+  development of activities that present a risk of death or bodily harm to
+  individuals, including use of Llama 3.1 related to the following:
+      1. Military, warfare, nuclear industries or applications, espionage, use for materials or activities that are subject to the International Traffic Arms Regulations (ITAR) maintained by the United States Department of State
+      2. Guns and illegal weapons (including weapon development)
+      3. Illegal drugs and regulated/controlled substances
+      4. Operation of critical infrastructure, transportation technologies, or heavy machinery
+      5. Self-harm or harm to others, including suicide, cutting, and eating disorders
+      6. Any content intended to incite or promote violence, abuse, or any infliction of bodily harm to an individual
+  3. Intentionally deceive or mislead others, including use of Llama 3.1 related
+  to the following:
+      1. Generating, promoting, or furthering fraud or the creation or promotion of disinformation
+      2. Generating, promoting, or furthering defamatory content, including the creation of defamatory statements, images, or other content
+      3. Generating, promoting, or further distributing spam
+      4. Impersonating another individual without consent, authorization, or legal right
+      5. Representing that the use of Llama 3.1 or outputs are human-generated
+      6. Generating or facilitating false online engagement, including fake reviews and other means of fake online engagement
+  4. Fail to appropriately disclose to end users any known dangers of your AI
+  system
+
+  Please report any violation of this Policy, software “bug,” or other problems
+  that could lead to a violation of this Policy through one of the following
+  means:
+      * Reporting issues with the model: [https://github.com/meta-llama/llama-models/issues](https://github.com/meta-llama/llama-models/issues)
+      * Reporting risky content generated by the model:
+      developers.facebook.com/llama_output_feedback
+      * Reporting bugs and security concerns: facebook.com/whitehat/info
+      * Reporting violations of the Acceptable Use Policy or unlicensed uses of Meta Llama 3: LlamaUseReport@meta.com
+extra_gated_fields:
+  First Name: text
+  Last Name: text
+  Date of birth: date_picker
+  Country: country
+  Affiliation: text
+  Job title:
+    type: select
+    options:
+    - Student
+    - Research Graduate
+    - AI researcher
+    - AI developer/engineer
+    - Reporter
+    - Other
+  geo: ip_location
+  By clicking Submit below I accept the terms of the license and acknowledge that the information I provide will be collected stored processed and shared in accordance with the Meta Privacy Policy: checkbox
+extra_gated_description: >-
+  The information you provide will be collected, stored, processed and shared in
+  accordance with the [Meta Privacy
+  Policy](https://www.facebook.com/privacy/policy/).
+extra_gated_button_content: Submit
+library_name: transformers
+---
+
+## Model Information
+
+The Meta Llama 3.1 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction tuned generative models in 8B, 70B and 405B sizes (text in/text out). The Llama 3.1 instruction tuned text only models (8B, 70B, 405B) are optimized for multilingual dialogue use cases and outperform many of the available open source and closed chat models on common industry benchmarks.
+
+**Model developer**: Meta
+
+**Model Architecture:** Llama 3.1 is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety. 
+
+
+<table>
+  <tr>
+   <td>
+   </td>
+   <td><strong>Training Data</strong>
+   </td>
+   <td><strong>Params</strong>
+   </td>
+   <td><strong>Input modalities</strong>
+   </td>
+   <td><strong>Output modalities</strong>
+   </td>
+   <td><strong>Context length</strong>
+   </td>
+   <td><strong>GQA</strong>
+   </td>
+   <td><strong>Token count</strong>
+   </td>
+   <td><strong>Knowledge cutoff</strong>
+   </td>
+  </tr>
+  <tr>
+   <td rowspan="3" >Llama 3.1 (text only)
+   </td>
+   <td rowspan="3" >A new mix of publicly available online data.
+   </td>
+   <td>8B
+   </td>
+   <td>Multilingual Text
+   </td>
+   <td>Multilingual Text and code
+   </td>
+   <td>128k
+   </td>
+   <td>Yes
+   </td>
+   <td rowspan="3" >15T+
+   </td>
+   <td rowspan="3" >December 2023
+   </td>
+  </tr>
+  <tr>
+   <td>70B
+   </td>
+   <td>Multilingual Text
+   </td>
+   <td>Multilingual Text and code
+   </td>
+   <td>128k
+   </td>
+   <td>Yes
+   </td>
+  </tr>
+  <tr>
+   <td>405B
+   </td>
+   <td>Multilingual Text
+   </td>
+   <td>Multilingual Text and code
+   </td>
+   <td>128k
+   </td>
+   <td>Yes
+   </td>
+  </tr>
+</table>
+
+
+**Supported languages:** English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.
+
+**Llama 3.1 family of models**. Token counts refer to pretraining data only. All model versions use Grouped-Query Attention (GQA) for improved inference scalability.
+
+**Model Release Date:** July 23, 2024.
+
+**Status:** This is a static model trained on an offline dataset. Future versions of the tuned models will be released as we improve model safety with community feedback.
+
+**License:** A custom commercial license, the Llama 3.1 Community License, is available at: [https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE)
+
+Where to send questions or comments about the model Instructions on how to provide feedback or comments on the model can be found in the model [README](https://github.com/meta-llama/llama3). For more technical information about generation parameters and recipes for how to use Llama 3.1 in applications, please go [here](https://github.com/meta-llama/llama-recipes). 
+
+
+## Intended Use
+
+**Intended Use Cases** Llama 3.1 is intended for commercial and research use in multiple languages. Instruction tuned text only models are intended for assistant-like chat, whereas pretrained models can be adapted for a variety of natural language generation tasks. The Llama 3.1 model collection also supports the ability to leverage the outputs of its models to improve other models including synthetic data generation and distillation. The Llama 3.1 Community License allows for these use cases. 
+
+**Out-of-scope** Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in any other way that is prohibited by the Acceptable Use Policy and Llama 3.1 Community License. Use in languages beyond those explicitly referenced as supported in this model card**.
+
+**<span style="text-decoration:underline;">Note</span>: Llama 3.1 has been trained on a broader collection of languages than the 8 supported languages. Developers may fine-tune Llama 3.1 models for languages beyond the 8 supported languages provided they comply with the Llama 3.1 Community License and the Acceptable Use Policy and in such cases are responsible for ensuring that any uses of Llama 3.1 in additional languages is done in a safe and responsible manner.
+
+## How to use
+
+This repository contains two versions of Meta-Llama-3.1-8B, for use with transformers and with the original `llama` codebase.
+
+### Use with transformers
+
+Starting with transformers >= 4.43.0 onward, you can run conversational inference using the Transformers pipeline abstraction or by leveraging the Auto classes with the generate() function.
+
+Make sure to update your transformers installation via pip install --upgrade transformers.
+
+```python
+import transformers
+import torch
+
+model_id = "meta-llama/Meta-Llama-3.1-8B"
+
+pipeline = transformers.pipeline(
+    "text-generation", model=model_id, model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto"
+)
+
+pipeline("Hey how are you doing today?")
+```
+
+### Use with `llama`
+
+Please, follow the instructions in the [repository](https://github.com/meta-llama/llama).
+
+To download Original checkpoints, see the example command below leveraging `huggingface-cli`:
+
+```
+huggingface-cli download meta-llama/Meta-Llama-3.1-8B --include "original/*" --local-dir Meta-Llama-3.1-8B
+```
+
+## Hardware and Software
+
+**Training Factors** We used custom training libraries, Meta's custom built GPU cluster, and production infrastructure for pretraining. Fine-tuning, annotation, and evaluation were also performed on production infrastructure.
+
+**Training utilized a cumulative of** 39.3M GPU hours of computation on H100-80GB (TDP of 700W) type hardware, per the table below. Training time is the total GPU time required for training each model and power consumption is the peak power capacity per GPU device used, adjusted for power usage efficiency. 
+
+
+**Training Greenhouse Gas Emissions** Estimated total location-based greenhouse gas emissions were **11,390** tons CO2eq for training. Since 2020, Meta has maintained net zero greenhouse gas emissions in its global operations and matched 100% of its electricity use with renewable energy, therefore the total market-based greenhouse gas emissions for training were 0 tons CO2eq.
+
+
+<table>
+  <tr>
+   <td>
+   </td>
+   <td><strong>Training Time (GPU hours)</strong>
+   </td>
+   <td><strong>Training Power Consumption (W)</strong>
+   </td>
+   <td><strong>Training Location-Based Greenhouse Gas Emissions</strong>
+<p>
+<strong>(tons CO2eq)</strong>
+   </td>
+   <td><strong>Training Market-Based Greenhouse Gas Emissions</strong>
+<p>
+<strong>(tons CO2eq)</strong>
+   </td>
+  </tr>
+  <tr>
+   <td>Llama 3.1 8B
+   </td>
+   <td>1.46M
+   </td>
+   <td>700
+   </td>
+   <td>420
+   </td>
+   <td>0
+   </td>
+  </tr>
+  <tr>
+   <td>Llama 3.1 70B
+   </td>
+   <td>7.0M
+   </td>
+   <td>700
+   </td>
+   <td>2,040
+   </td>
+   <td>0
+   </td>
+  </tr>
+  <tr>
+   <td>Llama 3.1 405B
+   </td>
+   <td>30.84M
+   </td>
+   <td>700
+   </td>
+   <td>8,930
+   </td>
+   <td>0
+   </td>
+  </tr>
+  <tr>
+   <td>Total
+   </td>
+   <td>39.3M
+   <td>
+<ul>
+
+</ul>
+   </td>
+   <td>11,390
+   </td>
+   <td>0
+   </td>
+  </tr>
+</table>
+
+
+
+The methodology used to determine training energy use and greenhouse gas emissions can be found [here](https://arxiv.org/pdf/2204.05149).  Since Meta is openly releasing these models, the training energy use and greenhouse gas emissions  will not be incurred by others.
+
+
+## Training Data
+
+**Overview:** Llama 3.1 was pretrained on ~15 trillion tokens of data from publicly available sources. The fine-tuning data includes publicly available instruction datasets, as well as over 25M synthetically generated examples. 
+
+**Data Freshness:** The pretraining data has a cutoff of December 2023.
+
+
+## Benchmark scores
+
+In this section, we report the results for Llama 3.1 models on standard automatic benchmarks. For all the evaluations, we use our internal evaluations library. 
+
+### Base pretrained models
+
+
+<table>
+  <tr>
+   <td><strong>Category</strong>
+   </td>
+   <td><strong>Benchmark</strong>
+   </td>
+   <td><strong># Shots</strong>
+   </td>
+   <td><strong>Metric</strong>
+   </td>
+   <td><strong>Llama 3 8B</strong>
+   </td>
+   <td><strong>Llama 3.1 8B</strong>
+   </td>
+   <td><strong>Llama 3 70B</strong>
+   </td>
+   <td><strong>Llama 3.1 70B</strong>
+   </td>
+   <td><strong>Llama 3.1 405B</strong>
+   </td>
+  </tr>
+  <tr>
+   <td rowspan="7" >General
+   </td>
+   <td>MMLU
+   </td>
+   <td>5
+   </td>
+   <td>macro_avg/acc_char
+   </td>
+   <td>66.7
+   </td>
+   <td>66.7
+   </td>
+   <td>79.5
+   </td>
+   <td>79.3
+   </td>
+   <td>85.2
+   </td>
+  </tr>
+  <tr>
+   <td>MMLU-Pro (CoT)
+   </td>
+   <td>5
+   </td>
+   <td>macro_avg/acc_char
+   </td>
+   <td>36.2
+   </td>
+   <td>37.1
+   </td>
+   <td>55.0
+   </td>
+   <td>53.8
+   </td>
+   <td>61.6
+   </td>
+  </tr>
+  <tr>
+   <td>AGIEval English
+   </td>
+   <td>3-5
+   </td>
+   <td>average/acc_char
+   </td>
+   <td>47.1
+   </td>
+   <td>47.8
+   </td>
+   <td>63.0
+   </td>
+   <td>64.6
+   </td>
+   <td>71.6
+   </td>
+  </tr>
+  <tr>
+   <td>CommonSenseQA
+   </td>
+   <td>7
+   </td>
+   <td>acc_char
+   </td>
+   <td>72.6
+   </td>
+   <td>75.0
+   </td>
+   <td>83.8
+   </td>
+   <td>84.1
+   </td>
+   <td>85.8
+   </td>
+  </tr>
+  <tr>
+   <td>Winogrande
+   </td>
+   <td>5
+   </td>
+   <td>acc_char
+   </td>
+   <td>-
+   </td>
+   <td>60.5
+   </td>
+   <td>-
+   </td>
+   <td>83.3
+   </td>
+   <td>86.7
+   </td>
+  </tr>
+  <tr>
+   <td>BIG-Bench Hard (CoT)
+   </td>
+   <td>3
+   </td>
+   <td>average/em
+   </td>
+   <td>61.1
+   </td>
+   <td>64.2
+   </td>
+   <td>81.3
+   </td>
+   <td>81.6
+   </td>
+   <td>85.9
+   </td>
+  </tr>
+  <tr>
+   <td>ARC-Challenge
+   </td>
+   <td>25
+   </td>
+   <td>acc_char
+   </td>
+   <td>79.4
+   </td>
+   <td>79.7
+   </td>
+   <td>93.1
+   </td>
+   <td>92.9
+   </td>
+   <td>96.1
+   </td>
+  </tr>
+  <tr>
+   <td>Knowledge reasoning
+   </td>
+   <td>TriviaQA-Wiki
+   </td>
+   <td>5
+   </td>
+   <td>em
+   </td>
+   <td>78.5
+   </td>
+   <td>77.6
+   </td>
+   <td>89.7
+   </td>
+   <td>89.8
+   </td>
+   <td>91.8
+   </td>
+  </tr>
+  <tr>
+   <td rowspan="4" >Reading comprehension
+   </td>
+   <td>SQuAD
+   </td>
+   <td>1
+   </td>
+   <td>em
+   </td>
+   <td>76.4
+   </td>
+   <td>77.0
+   </td>
+   <td>85.6
+   </td>
+   <td>81.8
+   </td>
+   <td>89.3
+   </td>
+  </tr>
+  <tr>
+   <td>QuAC (F1)
+   </td>
+   <td>1
+   </td>
+   <td>f1
+   </td>
+   <td>44.4
+   </td>
+   <td>44.9
+   </td>
+   <td>51.1
+   </td>
+   <td>51.1
+   </td>
+   <td>53.6
+   </td>
+  </tr>
+  <tr>
+   <td>BoolQ
+   </td>
+   <td>0
+   </td>
+   <td>acc_char
+   </td>
+   <td>75.7
+   </td>
+   <td>75.0
+   </td>
+   <td>79.0
+   </td>
+   <td>79.4
+   </td>
+   <td>80.0
+   </td>
+  </tr>
+  <tr>
+   <td>DROP (F1)
+   </td>
+   <td>3
+   </td>
+   <td>f1
+   </td>
+   <td>58.4
+   </td>
+   <td>59.5
+   </td>
+   <td>79.7
+   </td>
+   <td>79.6
+   </td>
+   <td>84.8
+   </td>
+  </tr>
+</table>
+
+
+
+### Instruction tuned models
+
+
+<table>
+  <tr>
+   <td><strong>Category</strong>
+   </td>
+   <td><strong>Benchmark</strong>
+   </td>
+   <td><strong># Shots</strong>
+   </td>
+   <td><strong>Metric</strong>
+   </td>
+   <td><strong>Llama 3 8B Instruct</strong>
+   </td>
+   <td><strong>Llama 3.1 8B Instruct</strong>
+   </td>
+   <td><strong>Llama 3 70B Instruct</strong>
+   </td>
+   <td><strong>Llama 3.1 70B Instruct</strong>
+   </td>
+   <td><strong>Llama 3.1 405B Instruct</strong>
+   </td>
+  </tr>
+  <tr>
+   <td rowspan="4" >General
+   </td>
+   <td>MMLU
+   </td>
+   <td>5
+   </td>
+   <td>macro_avg/acc
+   </td>
+   <td>68.5
+   </td>
+   <td>69.4
+   </td>
+   <td>82.0
+   </td>
+   <td>83.6
+   </td>
+   <td>87.3
+   </td>
+  </tr>
+  <tr>
+   <td>MMLU (CoT)
+   </td>
+   <td>0
+   </td>
+   <td>macro_avg/acc
+   </td>
+   <td>65.3
+   </td>
+   <td>73.0
+   </td>
+   <td>80.9
+   </td>
+   <td>86.0
+   </td>
+   <td>88.6
+   </td>
+  </tr>
+  <tr>
+   <td>MMLU-Pro (CoT)
+   </td>
+   <td>5
+   </td>
+   <td>micro_avg/acc_char
+   </td>
+   <td>45.5
+   </td>
+   <td>48.3
+   </td>
+   <td>63.4
+   </td>
+   <td>66.4
+   </td>
+   <td>73.3
+   </td>
+  </tr>
+  <tr>
+   <td>IFEval
+   </td>
+   <td>
+   </td>
+   <td>
+   </td>
+   <td>76.8
+   </td>
+   <td>80.4
+   </td>
+   <td>82.9
+   </td>
+   <td>87.5
+   </td>
+   <td>88.6
+   </td>
+  </tr>
+  <tr>
+   <td rowspan="2" >Reasoning
+   </td>
+   <td>ARC-C
+   </td>
+   <td>0
+   </td>
+   <td>acc
+   </td>
+   <td>82.4
+   </td>
+   <td>83.4
+   </td>
+   <td>94.4
+   </td>
+   <td>94.8
+   </td>
+   <td>96.9
+   </td>
+  </tr>
+  <tr>
+   <td>GPQA
+   </td>
+   <td>0
+   </td>
+   <td>em
+   </td>
+   <td>34.6
+   </td>
+   <td>30.4
+   </td>
+   <td>39.5
+   </td>
+   <td>41.7
+   </td>
+   <td>50.7
+   </td>
+  </tr>
+  <tr>
+   <td rowspan="4" >Code
+   </td>
+   <td>HumanEval
+   </td>
+   <td>0
+   </td>
+   <td>pass@1
+   </td>
+   <td>60.4
+   </td>
+   <td>72.6
+   </td>
+   <td>81.7
+   </td>
+   <td>80.5
+   </td>
+   <td>89.0
+   </td>
+  </tr>
+  <tr>
+   <td>MBPP ++ base version
+   </td>
+   <td>0
+   </td>
+   <td>pass@1
+   </td>
+   <td>70.6
+   </td>
+   <td>72.8
+   </td>
+   <td>82.5
+   </td>
+   <td>86.0
+   </td>
+   <td>88.6
+   </td>
+  </tr>
+  <tr>
+   <td>Multipl-E HumanEval
+   </td>
+   <td>0
+   </td>
+   <td>pass@1
+   </td>
+   <td>-
+   </td>
+   <td>50.8
+   </td>
+   <td>-
+   </td>
+   <td>65.5
+   </td>
+   <td>75.2
+   </td>
+  </tr>
+  <tr>
+   <td>Multipl-E MBPP
+   </td>
+   <td>0
+   </td>
+   <td>pass@1
+   </td>
+   <td>-
+   </td>
+   <td>52.4
+   </td>
+   <td>-
+   </td>
+   <td>62.0
+   </td>
+   <td>65.7
+   </td>
+  </tr>
+  <tr>
+   <td rowspan="2" >Math
+   </td>
+   <td>GSM-8K (CoT)
+   </td>
+   <td>8
+   </td>
+   <td>em_maj1@1
+   </td>
+   <td>80.6
+   </td>
+   <td>84.5
+   </td>
+   <td>93.0
+   </td>
+   <td>95.1
+   </td>
+   <td>96.8
+   </td>
+  </tr>
+  <tr>
+   <td>MATH (CoT)
+   </td>
+   <td>0
+   </td>
+   <td>final_em
+   </td>
+   <td>29.1
+   </td>
+   <td>51.9
+   </td>
+   <td>51.0
+   </td>
+   <td>68.0
+   </td>
+   <td>73.8
+   </td>
+  </tr>
+  <tr>
+   <td rowspan="4" >Tool Use
+   </td>
+   <td>API-Bank
+   </td>
+   <td>0
+   </td>
+   <td>acc
+   </td>
+   <td>48.3
+   </td>
+   <td>82.6
+   </td>
+   <td>85.1
+   </td>
+   <td>90.0
+   </td>
+   <td>92.0
+   </td>
+  </tr>
+  <tr>
+   <td>BFCL
+   </td>
+   <td>0
+   </td>
+   <td>acc
+   </td>
+   <td>60.3
+   </td>
+   <td>76.1
+   </td>
+   <td>83.0
+   </td>
+   <td>84.8
+   </td>
+   <td>88.5
+   </td>
+  </tr>
+  <tr>
+   <td>Gorilla Benchmark API Bench
+   </td>
+   <td>0
+   </td>
+   <td>acc
+   </td>
+   <td>1.7
+   </td>
+   <td>8.2
+   </td>
+   <td>14.7
+   </td>
+   <td>29.7
+   </td>
+   <td>35.3
+   </td>
+  </tr>
+  <tr>
+   <td>Nexus (0-shot)
+   </td>
+   <td>0
+   </td>
+   <td>macro_avg/acc
+   </td>
+   <td>18.1
+   </td>
+   <td>38.5
+   </td>
+   <td>47.8
+   </td>
+   <td>56.7
+   </td>
+   <td>58.7
+   </td>
+  </tr>
+  <tr>
+   <td>Multilingual
+   </td>
+   <td>Multilingual MGSM (CoT)
+   </td>
+   <td>0
+   </td>
+   <td>em
+   </td>
+   <td>-
+   </td>
+   <td>68.9
+   </td>
+   <td>-
+   </td>
+   <td>86.9
+   </td>
+   <td>91.6
+   </td>
+  </tr>
+</table>
+
+#### Multilingual benchmarks
+
+<table>
+  <tr>
+   <td><strong>Category</strong>
+   </td>
+   <td><strong>Benchmark</strong>
+   </td>
+   <td><strong>Language</strong>
+   </td>
+   <td><strong>Llama 3.1 8B</strong>
+   </td>
+   <td><strong>Llama 3.1 70B</strong>
+   </td>
+   <td><strong>Llama 3.1 405B</strong>
+   </td>
+  </tr>
+  <tr>
+   <td rowspan="9" ><strong>General</strong>
+   </td>
+   <td rowspan="9" ><strong>MMLU (5-shot, macro_avg/acc)</strong>
+   </td>
+   <td>Portuguese
+   </td>
+   <td>62.12
+   </td>
+   <td>80.13
+   </td>
+   <td>84.95
+   </td>
+  </tr>
+  <tr>
+   <td>Spanish
+   </td>
+   <td>62.45
+   </td>
+   <td>80.05
+   </td>
+   <td>85.08
+   </td>
+  </tr>
+  <tr>
+   <td>Italian
+   </td>
+   <td>61.63
+   </td>
+   <td>80.4
+   </td>
+   <td>85.04
+   </td>
+  </tr>
+  <tr>
+   <td>German
+   </td>
+   <td>60.59
+   </td>
+   <td>79.27
+   </td>
+   <td>84.36
+   </td>
+  </tr>
+  <tr>
+   <td>French
+   </td>
+   <td>62.34
+   </td>
+   <td>79.82
+   </td>
+   <td>84.66
+   </td>
+  </tr>
+  <tr>
+   <td>Hindi
+   </td>
+   <td>50.88
+   </td>
+   <td>74.52
+   </td>
+   <td>80.31
+   </td>
+  </tr>
+  <tr>
+   <td>Thai
+   </td>
+   <td>50.32
+   </td>
+   <td>72.95
+   </td>
+   <td>78.21
+   </td>
+  </tr>
+</table>
+
+
+
+## Responsibility & Safety
+
+As part of our Responsible release approach, we followed a three-pronged strategy to managing trust & safety risks:
+
+
+
+* Enable developers to deploy helpful, safe and flexible experiences for their target audience and for the use cases supported by Llama. 
+* Protect developers against adversarial users aiming to exploit Llama capabilities to potentially cause harm.
+* Provide protections for the community to help prevent the misuse of our models.
+
+
+### Responsible deployment 
+
+Llama is a foundational technology designed to be used in a variety of use cases, examples on how Meta’s Llama models have been responsibly deployed can be found in our [Community Stories webpage](https://llama.meta.com/community-stories/). Our approach is to build the most helpful models enabling the world to benefit from the technology power, by aligning our model safety for the generic use cases addressing a standard set of harms. Developers are then in the driver seat to tailor safety for their use case, defining their own policy and deploying the models with the necessary safeguards in their Llama systems. Llama 3.1 was developed following the best practices outlined in our Responsible Use Guide, you can refer to the [Responsible Use Guide](https://llama.meta.com/responsible-use-guide/) to learn more. 
+
+
+#### Llama 3.1 instruct 
+
+Our main objectives for conducting safety fine-tuning are to provide the research community with a valuable resource for studying the robustness of safety fine-tuning, as well as to offer developers a readily available, safe, and powerful model for various applications to reduce the developer workload to deploy safe AI systems. For more details on the safety mitigations implemented please read the Llama 3 paper. 
+
+**Fine-tuning data**
+
+We employ a multi-faceted approach to data collection, combining human-generated data from our vendors with synthetic data to mitigate potential safety risks. We’ve developed many large language model (LLM)-based classifiers that enable us to thoughtfully select high-quality prompts and responses, enhancing data quality control. 
+
+**Refusals and Tone**
+
+Building on the work we started with Llama 3, we put a great emphasis on model refusals to benign prompts as well as refusal tone. We included both borderline and adversarial prompts in our safety data strategy, and modified our safety data responses to follow  tone guidelines. 
+
+
+#### Llama 3.1 systems
+
+**Large language models, including Llama 3.1, are not designed to be deployed in isolation but instead should be deployed as part of an overall AI system with additional safety guardrails as required.** Developers are expected to deploy system safeguards when building agentic systems. Safeguards are key to achieve the right helpfulness-safety alignment as well as mitigating safety and security risks inherent to the system and any integration of the model or system with external tools. 
+
+As part of our responsible release approach, we provide the community with [safeguards](https://llama.meta.com/trust-and-safety/) that developers should deploy with Llama models or other LLMs, including Llama Guard 3, Prompt Guard and Code Shield. All our [reference implementations](https://github.com/meta-llama/llama-agentic-system) demos contain these safeguards by default so developers can benefit from system-level safety out-of-the-box. 
+
+
+#### New capabilities 
+
+Note that this release introduces new capabilities, including a longer context window, multilingual inputs and outputs and possible integrations by developers with third party tools. Building with these new capabilities requires specific considerations in addition to the best practices that generally apply across all Generative AI use cases.
+
+**Tool-use**: Just like in standard software development, developers are responsible for the integration of the LLM with the tools and services of their choice. They should define a clear policy for their use case and assess the integrity of the third party services they use to be aware of the safety and security limitations when using this capability. Refer to the Responsible Use Guide for best practices on the safe deployment of the third party safeguards. 
+
+**Multilinguality**: Llama 3.1 supports 7 languages in addition to English: French, German, Hindi, Italian, Portuguese, Spanish, and Thai. Llama may be able to output text in other languages than those that meet performance thresholds for safety and helpfulness. We strongly discourage developers from using this model to converse in non-supported languages without implementing finetuning and system controls in alignment with their policies and the best practices shared in the Responsible Use Guide. 
+
+
+### Evaluations
+
+We evaluated Llama models for common use cases as well as specific capabilities. Common use cases evaluations measure safety risks of systems for most commonly built applications including chat bot, coding assistant, tool calls. We built dedicated, adversarial evaluation datasets and evaluated systems composed of Llama models and Llama Guard 3 to filter input prompt and output response. It is important to evaluate applications in context, and we recommend building dedicated evaluation dataset for your use case. Prompt Guard and Code Shield are also available if relevant to the application. 
+
+Capability evaluations measure vulnerabilities of Llama models inherent to specific capabilities, for which were crafted dedicated benchmarks including long context, multilingual, tools calls, coding or memorization.
+
+**Red teaming**
+
+For both scenarios, we conducted recurring red teaming exercises with the goal of discovering risks via adversarial prompting and we used the learnings to improve our benchmarks and safety tuning datasets. 
+
+We partnered early with subject-matter experts in critical risk areas to understand the nature of these real-world harms and how such models may lead to unintended harm for society. Based on these conversations, we derived a set of adversarial goals for the red team to attempt to achieve, such as extracting harmful information or reprogramming the model to act in a potentially harmful capacity.  The red team consisted of experts in cybersecurity, adversarial machine learning, responsible AI, and integrity in addition to multilingual content specialists with background in integrity issues in specific geographic markets.
+
+
+### Critical and other risks 
+
+We specifically focused our efforts on mitigating the following critical risk areas:
+
+**1- CBRNE (Chemical, Biological, Radiological, Nuclear, and Explosive materials) helpfulness**
+
+To assess risks related to proliferation of chemical and biological weapons, we performed uplift testing designed to assess whether use of Llama 3.1 models could meaningfully increase the capabilities of malicious actors to plan or carry out attacks using these types of weapons. 
+
+
+**2. Child Safety**
+
+Child Safety risk assessments were conducted using a team of experts, to assess the model’s capability to produce outputs that could result in Child Safety risks and inform on any necessary and appropriate risk mitigations via fine tuning. We leveraged those expert red teaming sessions to expand the coverage of our evaluation benchmarks through Llama 3 model development.  For Llama 3, we conducted new in-depth sessions using objective based methodologies to assess the model risks along multiple attack vectors including the additional languages Llama 3 is trained on. We also partnered with content specialists to perform red teaming exercises assessing potentially violating content while taking account of market specific nuances or experiences. 
+
+**3. Cyber attack enablement**
+
+Our cyber attack uplift study investigated whether LLMs can enhance human capabilities in hacking tasks, both in terms of skill level and speed.
+
+Our attack automation study focused on evaluating the capabilities of LLMs when used as autonomous agents in cyber offensive operations, specifically in the context of ransomware attacks. This evaluation was distinct from previous studies that considered LLMs as interactive assistants. The primary objective was to assess whether these models could effectively function as independent agents in executing complex cyber-attacks without human intervention.
+
+Our study of Llama-3.1-405B’s social engineering uplift for cyber attackers was conducted to assess the effectiveness of AI models in aiding cyber threat actors in spear phishing campaigns. Please read our Llama 3.1 Cyber security whitepaper to learn more.
+
+
+### Community 
+
+Generative AI safety requires expertise and tooling, and we believe in the strength of the open community to accelerate its progress. We are active members of open consortiums, including the AI Alliance, Partnership on AI and MLCommons, actively contributing to safety standardization and transparency. We encourage the community to adopt taxonomies like the MLCommons Proof of Concept evaluation to facilitate collaboration and transparency on safety and content evaluations. Our Purple Llama tools are open sourced for the community to use and widely distributed across ecosystem partners including cloud service providers. We encourage community contributions to our [Github repository](https://github.com/meta-llama/PurpleLlama). 
+
+We also set up the [Llama Impact Grants](https://llama.meta.com/llama-impact-grants/) program to identify and support the most compelling applications of Meta’s Llama model for societal benefit across three categories: education, climate and open innovation. The 20 finalists from the hundreds of applications can be found [here](https://llama.meta.com/llama-impact-grants/#finalists). 
+
+Finally, we put in place a set of resources including an [output reporting mechanism](https://developers.facebook.com/llama_output_feedback) and [bug bounty program](https://www.facebook.com/whitehat) to continuously improve the Llama technology with the help of the community.
+
+
+## Ethical Considerations and Limitations
+
+The core values of Llama 3.1 are openness, inclusivity and helpfulness. It is meant to serve everyone, and to work for a wide range of use cases. It is thus designed to be accessible to people across many different backgrounds, experiences and perspectives. Llama 3.1 addresses users and their needs as they are, without insertion unnecessary judgment or normativity, while reflecting the understanding that even content that may appear problematic in some cases can serve valuable purposes in others. It respects the dignity and autonomy of all users, especially in terms of the values of free thought and expression that power innovation and progress. 
+
+But Llama 3.1 is a new technology, and like any new technology, there are risks associated with its use. Testing conducted to date has not covered, nor could it cover, all scenarios. For these reasons, as with all LLMs, Llama 3.1’s potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying any applications of Llama 3.1 models, developers should perform safety testing and tuning tailored to their specific applications of the model. Please refer to available resources including our [Responsible Use Guide](https://llama.meta.com/responsible-use-guide), [Trust and Safety](https://llama.meta.com/trust-and-safety/) solutions, and other [resources](https://llama.meta.com/docs/get-started/) to learn more about responsible development.
\ No newline at end of file

README.md

+# Llama
+彻底开源预训练大模型，本项目能够预训练出超出qwen2.5、llama3效果的大语言模型，为一些人工智能大厂的训练代码。
+
+目前各种SOTA NLP大模型算法都与Llama高度相似，故Llama适合作为算法研发的蓝底，本项目首次从数据集、预训练到调优完全开源大模型算法代码，帮助全世界所有算法研究人员共同研究以促进人类文明进步。
+<div align=center>
+    <img src="./doc/llm.png"/>
+</div>
+
+## 论文
+`Open and Efficient Foundation Language Models`
+- https://arxiv.org/pdf/2302.13971
+
+## 模型结构
+Llama系列采用极简Decoder-only结构，Llama源自基本的transformer结构，主体为attention(QKV自点积)+ffn(全连接)，最后外加一个softmax进行概率转换输出即可，为了使数据分布归一化方便训练收敛，在attention、ffn、softmax前分别再加一个RMS Norm。
+
+总体而言，Llama系列模型结构高度相似，llama1在GPT基础上引入旋转矩阵解决之前绝对位置编码复杂的问题，引入RMSNorm解决LayerNorm计算量大的问题，llama2在llama1的基础上引入GQA进一步减小计算量， llama3在llama2的基础上引入蒸馏、剪枝、量化等再进一步减小计算量，模型中其它模块（如flash-attn2、KV cache）只是增加训练推理效率的模块，本项目兼容Llama系列，以下分别为读者提供Llama的简图和详图帮助读者全方位理解。
+<div align=center>
+    <img src="./doc/llama3.png"/>
+</div>
+
+<div align=center>
+    <img src="./doc/llama3_detail.png"/>
+</div>
+
+Facebook官网最原始的llama3请参考代码：[`Llama3`](https://github.com/meta-llama/llama3/blob/main/llama/model.py)，本项目中的llama结构在ffn等层上略有修改，其它不同点只是模型规模参数和实现方式，读者若需要纯原版llama3可自行修改。
+
+## 算法原理
+整个Llama算法都体现出大道至简的思想。
+
+原理采用极简的纯矩阵计算，llama将输入embedding（将语句根据词汇量和词的位置、属性转换成数字化矩阵）后放入attention+ffn等提取特征，最后利用Softmax将解码器最后一层产生的未经归一化的分数向量（logits）转换为概率分布，其中每个元素表示生成对应词汇的概率，这使得模型可以生成一个分布，并从中选择最可能的词作为预测结果，然后一个字一个预测出来就是咱们看到的对话生成效果。
+
+损失函数采用最简单方便的CE(cross entropy) loss便可。
+<div align=center>
+    <img src="./doc/algorithm.png"/>
+</div>
+
+## 环境配置
+```
+mv nanotron_pytorch nanotron # 去框架名后缀
+```
+
+### Docker（方法一）
+```
+docker pull image.sourcefind.cn:5000/dcu/admin/base/pytorch:2.3.0-py3.10-dtk24.04.3-ubuntu20.04
+# <your IMAGE ID>为以上拉取的docker的镜像ID替换，本镜像为：b272aae8ec72
+docker run -it --shm-size=64G -v $PWD/nanotron:/home/nanotron -v /opt/hyhal:/opt/hyhal:ro --privileged=true --device=/dev/kfd --device=/dev/dri/ --group-add video --name llama <your IMAGE ID> bash
+cd /home/nanotron
+pip install -r requirements.txt
+pip install -e . #安装nanotron==0.4库
+pip install whl/rotary_emb-0.1.0+das.opt2.dtk24043-cp310-cp310-manylinux_2_28_x86_64.whl # 安装rotary_emb==0.1.0
+```
+### Dockerfile（方法二）
+```
+cd cd /home/nanotron/docker
+docker build --no-cache -t llama:latest .
+docker run --shm-size=64G --name llama -v /opt/hyhal:/opt/hyhal:ro --privileged=true --device=/dev/kfd --device=/dev/dri/ --group-add video -v $PWD/../../nanotron:/home/nanotron -it llama bash
+# 若遇到Dockerfile启动的方式安装环境需要长时间等待，可注释掉里面的pip安装，启动容器后再安装python库：pip install -r requirements.txt。
+cd /home/nanotron
+pip install -e . #安装nanotron==0.4库
+pip install whl/rotary_emb-0.1.0+das.opt2.dtk24043-cp310-cp310-manylinux_2_28_x86_64.whl # 安装rotary_emb==0.1.0
+```
+### Anaconda（方法三）
+1、关于本项目DCU显卡所需的特殊深度学习库可从光合开发者社区下载安装：
+- https://developer.hpccube.com/tool/
+```
+DTK驱动:dtk24.04.3
+python:python3.10
+torch:2.3.0
+torchvision:0.18.1
+torchaudio:2.1.2
+triton:2.1.0
+flash-attn:2.6.1
+deepspeed:0.14.2
+apex:1.3.0
+xformers:0.0.25
+```
+
+`Tips：以上dtk驱动、python、torch等DCU相关工具版本需要严格一一对应。`
+
+2、其它非特殊库参照requirements.txt安装
+```
+cd /home/nanotron
+pip install -r requirements.txt
+pip install -e . #安装nanotron==0.4库
+pip install whl/rotary_emb-0.1.0+das.opt2.dtk24043-cp310-cp310-manylinux_2_28_x86_64.whl # 安装rotary_emb==0.1.0
+```
+
+## 数据集
+实验性的迷你数据集[`openwebtext-10k`](./openwebtext-10k.tar.xz)源于openwebtext，仅供试验，实际训练中读者可在HF下载`*.parquet`开源数据集使用，或将自己的数据集按HF的官方说明制作成此类格式使用。
+
+数据集在训练之前需要用tokenlizer处理成NLP模型的输入tokens，Facebook官方采用tiktoken库制作tockens便可训练出SOTA模型：[`llama3 tokenizer`](https://github.com/meta-llama/llama3/blob/main/llama/tokenizer.py)，本项目可根据读者需求自由选择各种HF的开源tokenlizer，将其填写在`config`的`.yaml`中便可自动被项目调用。
+
+`openwebtext-10k`用于tiny llama预训练示例，[`fineweb-edu-dedup`](http://113.200.138.88:18080/aidatasets/argilla-warehouse/fineweb-edu-dedup-filtered.git) 用于smollm预训练示例（HF公司自研人工智能模型），从SCNet快速下载通道下载后重命名即可，原始`fineweb-edu-dedup`数据(`*.parquet`)可通过以下命令转换成`fineweb-edu-dedup-ds`数据(`*.ds`)，`datatrove`制作`*.ds`数据参考[`Nanosets`](./docs/nanoset.md):
+```
+sh convert_data_to_ds.sh
+```
+```
+datatrove库Tokenizer bug solve: Exception: Is a directory (os error 21)
+
+vim /usr/local/lib/python3.10/site-packages/datatrove/utils/tokenization.py , line 19
+modify:
+# return Tokenizer.from_file(name_or_path)
+return Tokenizer.from_file(name_or_path + "/tokenizer.json")
+```
+
+```
+datasets库load_dataset bug solve: OSError: Not enough disk space.
+
+vim /usr/local/lib/python3.10/site-packages/datatrove/pipeline/readers/huggingface.py , line 95
+modify:
+def run(self, data: DocumentsPipeline = None, rank: int = 0, world_size: int = 1) -> DocumentsPipeline:
+        from datasets import load_dataset  # type: ignore
+to:
+def run(self, data: DocumentsPipeline = None, rank: int = 0, world_size: int = 1) -> DocumentsPipeline:
+        from datasets import load_dataset  # type: ignore
+        import datasets
+        datasets.builder.has_sufficient_disk_space = lambda needed_bytes, directory='.': True
+```
+
+项目中已包含`tokenlizer`：[`dummy`](./robot-test/dummy-tokenizer-wordlevel) 、[`cosmo2`](./HuggingFaceTB/cosmo2-tokenizer) ，其它tokenizer（如：llama3）根据读者需求可自行下载。
+
+预训练数据的完整目录结构如下：
+```
+/home/nanotron
+    ├── openwebtext-10k.tar.xz
+    ├── stas/openwebtext-10k
+        ├── dataset_infos.json
+        ├── openwebtext-10k.py
+        ├── process.txt
+        └── README.md
+    ├── datasets/fineweb-edu-dedup
+        ├── train-00000-of-00002.parquet
+        ├── train-00001-of-00002.parquet
+    └── datasets/fineweb-edu-dedup-ds
+        ├── 00000_unshuffled.ds
+        ├── 00000_unshuffled.ds.index
+        ├── 00000_unshuffled.ds.metadata
+        ...
+```
+`备注：`本项目灵活度大，仅适于算法基础较好的研究人员使用，对算法基础和代码基础有一定的需求，其它人员可能存在一定的上手门槛，可参考光源上预训练项目[`allamo_pytorch`](http://developer.sourcefind.cn/codes/modelzoo/allamo_pytorch.git)中的简单预训练代码llama3__scratch进行上手学习。
+
+## 训练
+### 单机多卡
+本项目的最大特点是完全开源、营造自由科研环境，项目中的算法、模型读者可自由修改、研发以提出自己的算法来为社会做贡献，在[`llama`](./src/nanotron/models/llama.py)修改模型文件，但为了方便介绍，本步骤说明以小规模模型tiny llama作为示例：
+```
+cd /home/nanotron
+sh train.sh # 不同卡数的训练方式参照train.sh中的说明，完整规模llama3的训练方式可参考train.sh中的说明。
+# 遇到Do you wish to run the custom code? [y/N]，填y。
+
+# 其它功能正在优化中，欢迎共同优化和拓展。
+```
+
+```
+datasets库load_dataset bug solve: OSError: Not enough disk space.
+
+vim /home/nanotron/src/nanotron/dataloader.py , line 22
+modify:
+try:
+    import datasets
+to:
+try:
+    import datasets
+    datasets.builder.has_sufficient_disk_space = lambda needed_bytes, directory='.': True
+```
+
+Facebook原版llama3的模型参数可参考[`Llama-3.1-8B`](./checkpoints/Nanotron-Llama-3.1-8B/model_config.json)、[`Llama-3.2-3B`](./checkpoints/Nanotron-Llama-3.2-3B/model_config.json)，这两个参数文件根据以下命令可获取：
+```
+sh convert_hf_to_nanotron.sh # Llama系列的基础模型皆支持转换
+
+# 若已预训练完成某个模型，可转换成HF格式权重进行发布，以及用其它开源库继续微调，不同模型请读者根据具体参数修改此文件中的相应参数进行转换。
+# sh convert_nanotron_to_hf.sh
+```
+
+为了方便读者借鉴HF官方的预训练方式，项目中还提供了`smollm`的预训练示例，参考文档[`pre-training`](https://github.com/huggingface/smollm/tree/main/pre-training)：
+```
+sh train_smollm1_135M_demo.sh # Demo仅供试用，细节请自行研究，若读者具备超算集群，可参照launch.slurm编写自己具体的slurm脚本。
+```
+
+
+`Tips ：`通过本项目获得自主研发模型的预训练权重后，后续微调模型放在[`LLaMA-Factory`](https://github.com/hiyouga/LLaMA-Factory.git)、[`ollama`](https://github.com/ollama/ollama.git)、公开「后训练」一切的[`open-instruct`](https://github.com/allenai/open-instruct.git)等工具中进行更方便。
+
+更多资料可参考源项目的[`README_origin`](./README_origin.md)
+
+## 推理
+```
+sh infer.sh # 以checkpoints/10中的权重作为示例，其他权重请参照此示例修改权重路径。
+```
+
+更多资料可参考源项目的[`README_origin`](./README_origin.md)
+
+## result
+由于示例训练数据较少、模型为简化模型且训练时间短，此推理仅供参考显示效果，以方便读者了解项目使用方法，根据官方示例所得：
+
+`输入: `
+```
+input: [CLS] the [UNK] [UNK] [UNK] is [SEP]
+```
+
+`输出:`
+```
+generation: [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP]
+```
+上述重复字符为示例Demo的tokenlizer过于简单导致的正常现象，更换一个复杂的tokenlizer便可输出正常结果，例如使用`cosmo2-tokenizer`预训练：
+```
+CUDA_DEVICE_MAX_CONNECTIONS=1 torchrun --nproc_per_node=4 run_train.py --config-file examples/config_tiny_llama_cosmo2tokenizer.yaml
+```
+由此可见，不同tokenlizer会对训练结果造成明显差别，故建议实际训练中选择设计更好的tokenlizer。
+
+### 精度
+DCU与GPU精度一致，推理框架：pytorch。
+
+## 应用场景
+### 算法类别
+`对话问答`
+### 热点应用行业
+`制造,广媒,金融,能源,医疗,家居,教育`
+## 预训练权重
+预训练权重快速下载中心：[SCNet AIModels](http://113.200.138.88:18080/aimodels) ，项目中的预训练权重可从快速下载通道下载：[Llama-3.1-8B](http://113.200.138.88:18080/aimodels/meta-llama/Meta-Llama-3.1-8B.git) 、[Llama-3.2-3B](http://113.200.138.88:18080/aimodels/meta-llama/Llama-3.2-3B.git) 。
+
+Hugging Face下载地址为：[meta-llama/Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B) 、[meta-llama/Llama-3.2-3B](https://huggingface.co/meta-llama/Llama-3.2-3B) 
+## 源码仓库及问题反馈
+- http://developer.sourcefind.cn/codes/modelzoo/nanotron_pytorch.git
+## 参考资料
+- https://github.com/huggingface/nanotron.git
+- https://github.com/meta-llama/llama3.git
+- https://github.com/hiyouga/LLaMA-Factory.git
+- https://github.com/ollama/ollama.git
+- https://github.com/allenai/open-instruct.git
+

README_origin.md

+<h1 align="center">⚡️ Nanotron</h1>
+
+<p align="center">
+    <a href="https://github.com/huggingface/nanotron/releases">
+        <img alt="GitHub release" src="https://img.shields.io/github/release/huggingface/nanotron.svg">
+    </a>
+    <a href="https://github.com/huggingface/nanotron/blob/master/LICENSE">
+        <img alt="License" src="https://img.shields.io/github/license/huggingface/nanotron.svg?color=green">
+    </a>
+</p>
+
+<h4 align="center">
+    <p>
+        <a href="#installation">Installation</a> •
+        <a href="#quick-start">Quick Start</a> •
+        <a href="#features">Features</a> •
+        <a href="CONTRIBUTING.md">Contributing</a>
+    <p>
+</h4>
+
+<h3 align="center">
+    <a href="https://huggingface.co/nanotron"><img style="float: middle; padding: 10px 10px 10px 10px;" width="60" height="55" src="https://huggingface.co/datasets/huggingface/brand-assets/resolve/main/hf-logo.png" /></a>
+</h3>
+<h3 align="center">
+<p>Pretraining models made easy
+</h3>
+
+
+Nanotron is a library for pretraining transformer models. It provides a simple and flexible API to pretrain models on custom datasets. Nanotron is designed to be easy to use, fast, and scalable. It is built with the following principles in mind:
+
+- **Simplicity**: Nanotron is designed to be easy to use. It provides a simple and flexible API to pretrain models on custom datasets.
+- **Performance**: Optimized for speed and scalability, Nanotron uses the latest techniques to train models faster and more efficiently.
+
+## Installation
+
+```bash
+# Requirements: Python>=3.10
+git clone https://github.com/huggingface/nanotron
+cd nanotron
+pip install --upgrade pip
+pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu121
+pip install -e .
+
+# Install dependencies if you want to use the example scripts
+pip install datasets transformers
+pip install triton "flash-attn>=2.5.0" --no-build-isolation
+```
+> [!NOTE]
+> If you get `undefined symbol: ncclCommRegister` error you should install torch 2.1.2 instead: `pip install torch==2.1.2 --index-url https://download.pytorch.org/whl/cu121`
+
+> [!TIP]
+> We log to wandb automatically if it's installed. For that you can use `pip install wandb`. If you don't want to use wandb, you can run `wandb disabled`.
+
+## Quick Start
+### Training a tiny Llama model
+The following command will train a tiny Llama model on a single node with 8 GPUs. The model will be saved in the `checkpoints` directory as specified in the config file.
+```bash
+CUDA_DEVICE_MAX_CONNECTIONS=1 torchrun --nproc_per_node=8 run_train.py --config-file examples/config_tiny_llama.yaml
+```
+
+### Run generation from your checkpoint
+```bash
+torchrun --nproc_per_node=1 run_generate.py --ckpt-path checkpoints/10/ --tp 1 --pp 1
+# We could set a larger TP for faster generation, and a larger PP in case of very large models.
+```
+
+### Custom examples
+You can find more examples in the [`/examples`](/examples) directory:
+<!-- Make a table of the examples we support -->
+| Example | Description |
+| --- | --- |
+| `custom-dataloader` | Plug a custom dataloader to nanotron |
+| `datatrove` | Use the datatrove library to load data |
+| `doremi` | Use DoReMi to speed up training |
+| `mamba` | Train an example Mamba model |
+| `moe` | Train an example Mixture-of-Experts (MoE) model |
+| `mup` | Use spectral µTransfer to scale up your model |
+| `examples/config_tiny_llama_with_s3_upload.yaml` | For automatically uploading checkpoints to S3 |
+
+We're working on adding more examples soon! Feel free to add a PR to add your own example. 🚀
+
+
+## Features
+We currently support the following features:
+- [x] 3D parallelism (DP+TP+PP)
+- [x] Expert parallelism for MoEs
+- [x] AFAB and 1F1B schedules for PP
+- [x] Explicit APIs for TP and PP which enables easy debugging
+- [x] ZeRO-1 optimizer
+- [x] FP32 gradient accumulation
+- [x] Parameter tying/sharding
+- [x] Custom module checkpointing for large models
+- [x] Spectral µTransfer parametrization for scaling up neural networks
+- [x] Mamba example
+
+And we have on our roadmap:
+- [ ] FP8 training
+- [ ] ZeRO-3 optimizer (a.k.a FSDP)
+- [ ] `torch.compile` support
+- [ ] Ring attention
+- [ ] Interleaved 1f1b schedule
+
+## Credits
+We would like to thank everyone working on LLMs, especially those sharing their work openly from which we took great inspiration: Nvidia for `Megatron-LM/apex`, Microsoft for `DeepSpeed`, HazyResearch for `flash-attn`..

convert_data_to_ds.sh

+python3 tools/preprocess_data.py \
+       --tokenizer-name-or-path HuggingFaceTB/cosmo2-tokenizer \
+       --output-folder datasets/fineweb-edu-dedup-ds \
+       --n-tasks 16 \
+       hf \
+       --dataset datasets/fineweb-edu-dedup \

convert_hf_to_nanotron.sh

+torchrun --nproc-per-node 1 examples/llama/convert_hf_to_nanotron.py --checkpoint_path Meta-Llama-3.1-8B --save_path checkpoints/Nanotron-Llama-3.1-8B
+# torchrun --nproc-per-node 1 examples/llama/convert_hf_to_nanotron.py --checkpoint_path Llama-3.2-3B --save_path checkpoints/Nanotron-Llama-3.2-3B