Skip to content

GitLab

Commit e75bc9be authored by chenzk's avatar chenzk
Browse files

v1.0

parents
Hide whitespace changes
Inline Side-by-side
Showing with 1774 additions and 0 deletions
finetuning/README.md 0 → 100644
View file @ e75bc9be
# Fine-tuning
## SmolLM2 Instruct
We build the SmolLM2 Instruct family by finetuning the base 1.7B on [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) and the base 360M and 135M models on [Smol-smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk) using `TRL` and the alignement handbook and then doing DPO on [UltraFeedBack](https://huggingface.co/datasets/openbmb/UltraFeedback). You can find the scipts and instructions for dohere: https://github.com/huggingface/alignment-handbook/tree/main/recipes/smollm2#instructions-to-train-smollm2-17b-instruct
## Custom script
Here, we provide a simple script for finetuning SmolLM2. In this case, we fine-tune the base 1.7B on python data.
### Setup
Install `pytorch` [see documentation](https://pytorch.org/), and then install the requirements
```bash
pip install -r requirements.txt
```
Before you run any of the scripts make sure you are logged in `wandb` and HuggingFace Hub to push the checkpoints, and you have `accelerate` configured:
```bash
wandb login
huggingface-cli login
accelerate config
```
Now that everything is done, you can clone the repository and get into the corresponding directory.
```bash
git clone https://github.com/huggingface/smollm
cd smollm/finetune
```
### Training
To fine-tune efficiently with a low cost, we use [PEFT](https://github.com/huggingface/peft) library for Low-Rank Adaptation (LoRA) training. We also use the `SFTTrainer` from [TRL](https://github.com/huggingface/trl).
For this example, we will fine-tune SmolLM1-1.7B on the `Python` subset of [the-stack-smol](https://huggingface.co/datasets/bigcode/the-stack-smol). This is just for illustration purposes.
To launch the training:
```bash
accelerate launch train.py \
--model_id "HuggingFaceTB/SmolLM2-1.7B" \
--dataset_name "bigcode/the-stack-smol" \
--subset "data/python" \
--dataset_text_field "content" \
--split "train" \
--max_seq_length 2048 \
--max_steps 5000 \
--micro_batch_size 1 \
--gradient_accumulation_steps 8 \
--learning_rate 3e-4 \
--warmup_steps 100 \
--num_proc "$(nproc)"
```
If you want to fine-tune on other text datasets, you need to change `dataset_text_field` argument to the name of the column containing the code/text you want to train on.
finetuning/Smol_VLM_FT.ipynb 0 → 100644
View file @ e75bc9be
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/github/merveenoyan/smollm/blob/main/finetuning/Smol_VLM_FT.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nc0g2NLpUSGr"
},
"source": [
"# Fine-tune SmolVLM on Visual Question Answering using Consumer GPU with QLoRA\n",
"\n",
"In this notebook we will fine-tune SmolVLM VQAv2 dataset. With this notebook you can also fine-tune Idefics3, since both models have the same model class/architecture.\n",
"\n",
"We will use some techniques in this notebook that will let you fine-tune the model on L4 with batch size of 4 only using around 16.4 GB of VRAM. We ran this notebook in that setup to test, but because we were able to afford A100 this notebook was last ran on an A100."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WIhA1lQ7j0kw"
},
"outputs": [],
"source": [
"!pip install -q accelerate datasets peft bitsandbytes tensorboard"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XyJaqZZ3uYYl"
},
"outputs": [],
"source": [
"!pip install -q flash-attn --no-build-isolation"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wAeMA0heVBjT"
},
"source": [
"We will push out model to Hub so we need to authenticate ourselves."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "yKd5xtSGj7cm"
},
"outputs": [],
"source": [
"from huggingface_hub import notebook_login\n",
"\n",
"notebook_login()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WRq8ve-LVAzU"
},
"source": [
"In this notebook we will not do full fine-tuning but use QLoRA method, which loads an adapter to the quantized version of the model, saving space. If you want to do full fine-tuning, set `USE_LORA` and `USE_QLORA` to False. If you want to do LoRA, set `USE_QLORA` to False and `USE_LORA` to True."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "zpra68tNEtGe"
},
"outputs": [],
"source": [
"import os\n",
"os.environ[\"CUDA_DEVICE_ORDER\"] = \"PCI_BUS_ID\"\n",
"os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"1, 2\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"referenced_widgets": [
"23d3d175e6e642c7abc2bce09b73cf4d",
"db6ca8f47f274464b135909c907c946a",
"d05822c6293c424fbf9df6ec0f6b532b",
"05582fca18f443d6965776a721e30e9f",
"3d8974fd1ba9415c8070c1eab8ad75cb",
"648257c1b1c24e25a26355bddf75aa41",
"afa9a31c6b7f45e082ae07dea4a2600e",
"92232af543a4446cac53e4fcf3f4b6e1",
"a5f06e59634f4edf9f3d9409846a2b31",
"7ddfa8718bc24882ba2b50a899656107",
"5983728a1c1e43edb4d16bee6ad40171",
"dff574197f1f4466abb0eb46d36b8378"
]
},
"id": "b9CDMq0duYYn",
"outputId": "65a4a5fa-fe4d-4243-b2d7-405a8aa81c04"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "23d3d175e6e642c7abc2bce09b73cf4d",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"processor_config.json: 0%| | 0.00/68.0 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "db6ca8f47f274464b135909c907c946a",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"chat_template.json: 0%| | 0.00/434 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "d05822c6293c424fbf9df6ec0f6b532b",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"preprocessor_config.json: 0%| | 0.00/486 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "05582fca18f443d6965776a721e30e9f",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer_config.json: 0%| | 0.00/4.04k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "3d8974fd1ba9415c8070c1eab8ad75cb",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"vocab.json: 0%| | 0.00/801k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "648257c1b1c24e25a26355bddf75aa41",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"merges.txt: 0%| | 0.00/466k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "afa9a31c6b7f45e082ae07dea4a2600e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer.json: 0%| | 0.00/3.52M [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "92232af543a4446cac53e4fcf3f4b6e1",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"added_tokens.json: 0%| | 0.00/92.0 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a5f06e59634f4edf9f3d9409846a2b31",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"special_tokens_map.json: 0%| | 0.00/1.07k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some kwargs in processor config are unused and will not have any effect: image_seq_len. \n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "7ddfa8718bc24882ba2b50a899656107",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"config.json: 0%| | 0.00/7.08k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "5983728a1c1e43edb4d16bee6ad40171",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"model.safetensors: 0%| | 0.00/4.49G [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "dff574197f1f4466abb0eb46d36b8378",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"generation_config.json: 0%| | 0.00/132 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"(10536960, 2256809840)\n"
]
}
],
"source": [
"import torch\n",
"from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model\n",
"from transformers import AutoProcessor, BitsAndBytesConfig, Idefics3ForConditionalGeneration\n",
"\n",
"USE_LORA = False\n",
"USE_QLORA = True\n",
"SMOL = True\n",
"\n",
"model_id = \"HuggingFaceTB/SmolVLM-Base\" if SMOL else \"HuggingFaceM4/Idefics3-8B-Llama3\"\n",
"\n",
"processor = AutoProcessor.from_pretrained(\n",
" model_id\n",
")\n",
"\n",
"if USE_QLORA or USE_LORA:\n",
" lora_config = LoraConfig(\n",
" r=8,\n",
" lora_alpha=8,\n",
" lora_dropout=0.1,\n",
" target_modules=['down_proj','o_proj','k_proj','q_proj','gate_proj','up_proj','v_proj'],\n",
" use_dora=False if USE_QLORA else True,\n",
" init_lora_weights=\"gaussian\"\n",
" )\n",
" lora_config.inference_mode = False\n",
" if USE_QLORA:\n",
" bnb_config = BitsAndBytesConfig(\n",
" load_in_4bit=True,\n",
" bnb_4bit_use_double_quant=True,\n",
" bnb_4bit_quant_type=\"nf4\",\n",
" bnb_4bit_compute_dtype=torch.bfloat16\n",
" )\n",
"\n",
" model = Idefics3ForConditionalGeneration.from_pretrained(\n",
" model_id,\n",
" quantization_config=bnb_config if USE_QLORA else None,\n",
" _attn_implementation=\"flash_attention_2\",\n",
" device_map=\"auto\"\n",
" )\n",
" model.add_adapter(lora_config)\n",
" model.enable_adapters()\n",
" model = prepare_model_for_kbit_training(model)\n",
" model = get_peft_model(model, lora_config)\n",
" print(model.get_nb_trainable_parameters())\n",
"else:\n",
" model = Idefics3ForConditionalGeneration.from_pretrained(\n",
" model_id,\n",
" torch_dtype=torch.bfloat16,\n",
" _attn_implementation=\"flash_attention_2\",\n",
" ).to(DEVICE)\n",
"\n",
" # if you'd like to only fine-tune LLM\n",
" for param in model.model.vision_model.parameters():\n",
" param.requires_grad = False"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WIVhpp0EyZO2"
},
"source": [
"The model as is is holding 2.7 GB of GPU RAM 💗"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LMTtg3dl3NX2"
},
"source": [
"## Loading the dataset and Preprocessing"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pWHMWTSZ3Pyr"
},
"source": [
"We will load a small portion of the VQAv2 dataset. We are loading a small portion of the model for education purposes."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "POOqKqYRka5O"
},
"outputs": [],
"source": [
"from datasets import load_dataset\n",
"ds = load_dataset('merve/vqav2-small', trust_remote_code=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Znf9vMo5rnSd"
},
"outputs": [],
"source": [
"split_ds = ds[\"validation\"].train_test_split(test_size=0.5)\n",
"train_ds = split_ds[\"train\"]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "FIDioFlRuYYn",
"outputId": "79b697a7-d245-4fdc-b0e8-d9ffa8627953"
},
"outputs": [
{
"data": {
"text/plain": [
"Dataset({\n",
" features: ['multiple_choice_answer', 'question', 'image'],\n",
" num_rows: 10717\n",
"})"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"train_ds"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5nwMO3n0X7Hv"
},
"source": [
"Let's write our data collating function. We will apply prompt template to have questions and answers together so model can learn to answer. Then we pass the formatted prompts and images to the processor which processes both."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e0krVLZ-wNMl"
},
"outputs": [],
"source": [
"image_token_id = processor.tokenizer.additional_special_tokens_ids[\n",
" processor.tokenizer.additional_special_tokens.index(\"<image>\")]\n",
"\n",
"def collate_fn(examples):\n",
" texts = []\n",
" images = []\n",
" for example in examples:\n",
" image = example[\"image\"]\n",
" if image.mode != 'RGB':\n",
" image = image.convert('RGB')\n",
" question = example[\"question\"]\n",
" answer = example[\"multiple_choice_answer\"]\n",
" messages = [\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": [\n",
" {\"type\": \"text\", \"text\": \"Answer briefly.\"},\n",
" {\"type\": \"image\"},\n",
" {\"type\": \"text\", \"text\": question}\n",
" ]\n",
" },\n",
" {\n",
" \"role\": \"assistant\",\n",
" \"content\": [\n",
" {\"type\": \"text\", \"text\": answer}\n",
" ]\n",
" }\n",
" ]\n",
" text = processor.apply_chat_template(messages, add_generation_prompt=False)\n",
" texts.append(text.strip())\n",
" images.append([image])\n",
"\n",
" batch = processor(text=texts, images=images, return_tensors=\"pt\", padding=True)\n",
" labels = batch[\"input_ids\"].clone()\n",
" labels[labels == processor.tokenizer.pad_token_id] = -100\n",
" labels[labels == image_token_id] = -100\n",
" batch[\"labels\"] = labels\n",
"\n",
" return batch"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kEYDjWpE3LD5"
},
"source": [
"## Training"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QvAs896cdwg8"
},
"source": [
"We can now initialize `Trainer` and initialize `TrainingArguments` to pass to `Trainer`.\n",
"\n",
"Some notes:\n",
"- If you use 8-bit QLoRA with the below setup it uses around 16.4 GB VRAM (beautiful, fits comfortably inside L4, Colab free tier)\n",
"- We use gradient accumulation to simulate a larger batch size.\n",
"- We also save up on memory from intermediate activations by using gradient checkpointing.\n",
"\n",
"**Disclaimer:**\n",
"The techniques here aren't free lunch. The latter two will add additional compute to the training, thus slow down a bit (for reference on two A100s with bsz of 16, we were able to train for 2 hrs 43 mins with the gradient accumulation steps of 4, disabling it reduced it with 2 hr 35 mins).\n",
"If you want to speed-up, you might play around, reduce to 4-bit precision and have a higher batch size. Note that 4-bit might result in model learning less."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "QNE2yWAYrAhD"
},
"outputs": [],
"source": [
"from transformers import TrainingArguments, Trainer\n",
"\n",
"model_name = model_id.split(\"/\")[-1]\n",
"\n",
"training_args = TrainingArguments(\n",
" num_train_epochs=1,\n",
" per_device_train_batch_size=4,\n",
" gradient_accumulation_steps=4,\n",
" warmup_steps=50,\n",
" learning_rate=1e-4,\n",
" weight_decay=0.01,\n",
" logging_steps=25,\n",
" save_strategy=\"steps\",\n",
" save_steps=250,\n",
" save_total_limit=1,\n",
" optim=\"paged_adamw_8bit\", # for 8-bit, keep this, else adamw_hf\n",
" bf16=True, # underlying precision for 8bit\n",
" output_dir=f\"./{model_name}-vqav2\",\n",
" hub_model_id=f\"{model_name}-vqav2\",\n",
" report_to=\"tensorboard\",\n",
" remove_unused_columns=False,\n",
" gradient_checkpointing=True\n",
")\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "oBBSDpBhreJd",
"outputId": "071ed677-1d9f-4f98-9d19-64834440c9c4"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n"
]
}
],
"source": [
"trainer = Trainer(\n",
" model=model,\n",
" args=training_args,\n",
" data_collator=collate_fn,\n",
" train_dataset=train_ds,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_QOCpw_-uYYo",
"outputId": "7abb6937-c072-435a-c3f5-6dbb5b0b9eea"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='9' max='670' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
" [ 9/670 01:41 < 2:39:41, 0.07 it/s, Epoch 0.01/1]\n",
" </div>\n",
" <table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>Step</th>\n",
" <th>Training Loss</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" </tbody>\n",
"</table><p>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"trainer.train()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0hN0QD9_uYYo"
},
"outputs": [],
"source": [
"trainer.push_to_hub()"
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"gpuType": "A100",
"provenance": [],
"include_colab_link": true
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.4"
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {}
}
},
"nbformat": 4,
"nbformat_minor": 0
}
\ No newline at end of file
finetuning/bigcode/the-stack-smol/README.md 0 → 100644
View file @ e75bc9be
---
annotations_creators: []
language_creators:
- crowdsourced
language: ["code"]
multilinguality:
- multilingual
size_categories:
- unknown
source_datasets: []
task_categories:
- text-generation
task_ids:
- language-modeling
extra_gated_prompt: |-
## Terms of Use for The Stack
The Stack dataset is a collection of 3.1 TB of source code in 30 programming languages. We ask that you read and acknowledge the following points before using the dataset:
1. The Stack is a collection of source code from repositories with various licenses. Any use of all or part of the code gathered in The Stack must abide by the terms of the original licenses, including attribution clauses when relevant. We facilitate this by providing provenance information for each data point.
2. The Stack is regularly updated to enact validated data removal requests. By clicking on "Access repository", you agree to update your own version of The Stack to the most recent usable version specified by the maintainers in [the following thread](https://huggingface.co/datasets/bigcode/the-stack/discussions/7). If you have questions about dataset versions and allowed uses, please also ask them in the dataset’s [community discussions](https://huggingface.co/datasets/bigcode/the-stack/discussions/new). We will also notify users via email when the latest usable version changes.
3. To host, share, or otherwise provide access to The Stack dataset, you must include [these Terms of Use](https://huggingface.co/datasets/bigcode/the-stack#terms-of-use-for-the-stack) and require users to agree to it.
By clicking on "Access repository" below, you accept that your contact information (email address and username) can be shared with the dataset maintainers as well.
extra_gated_fields:
Email: text
I have read the License and agree with its terms: checkbox
---
## Dataset Description
![Smol](https://huggingface.co/datasets/bigcode/admin/resolve/main/smol.png)
A small subset (~0.1%) of [the-stack](https://huggingface.co/datasets/bigcode/the-stack) dataset, each programming language has 10,000 random samples from the original dataset. The dataset has 2.6GB of text (code).
## Languages
The dataset contains 30 programming languages:
````
"assembly", "batchfile", "c++", "c", "c-sharp", "cmake", "css", "dockerfile", "fortran", "go", "haskell", "html", "java",
"javascript", "julia", "lua", "makefile", "markdown", "perl", "php", "powershell", "python", "ruby", "rust",
"scala", "shell", "sql", "tex", "typescript", "visual-basic"
`````
## Dataset Structure
```python
from datasets import load_dataset
load_dataset("bigcode/the-stack-smol")
DatasetDict({
train: Dataset({
features: ['content', 'avg_line_length', 'max_line_length', 'alphanum_fraction', 'licenses', 'repository_name', 'path', 'size', 'lang'],
num_rows: 300000
})
})
```
### How to use it
You can either load the whole dataset like above, or load a specific language such as python by specifying the folder directory:
```python
load_dataset("bigcode/the-stack-smol", data_dir="data/python")
DatasetDict({
train: Dataset({
features: ['content', 'avg_line_length', 'max_line_length', 'alphanum_fraction', 'licenses', 'repository_name', 'path', 'size', 'lang'],
num_rows: 10000
})
})
```
finetuning/bigcode/the-stack-smol/data/python/data.json 0 → 100644
View file @ e75bc9be
This source diff could not be displayed because it is too large. You can view the blob instead.
finetuning/finetune_smollm2_python/final_checkpoint/adapter_config.json 0 → 100644
View file @ e75bc9be
{
"alpha_pattern": {},
"auto_mapping": null,
"base_model_name_or_path": "HuggingFaceTB/SmolLM2-1.7B-Instruct",
"bias": "none",
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 32,
"lora_dropout": 0.05,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": null,
"peft_type": "LORA",
"r": 16,
"rank_pattern": {},
"revision": null,
"target_modules": [
"v_proj",
"q_proj"
],
"task_type": "CAUSAL_LM",
"use_dora": false,
"use_rslora": false
}
\ No newline at end of file
finetuning/finetune_smollm2_python/final_checkpoint/config.json 0 → 100644
View file @ e75bc9be
{
"_name_or_path": "HuggingFaceTB/SmolLM2-1.7B-Instruct",
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.1,
"bos_token_id": 1,
"eos_token_id": 2,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 2048,
"initializer_range": 0.02,
"intermediate_size": 8192,
"max_position_embeddings": 8192,
"mlp_bias": false,
"model_type": "llama",
"num_attention_heads": 32,
"num_hidden_layers": 24,
"num_key_value_heads": 32,
"pad_token_id": 2,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_scaling": null,
"rope_theta": 130000,
"tie_word_embeddings": true,
"torch_dtype": "float32",
"transformers.js_config": {
"kv_cache_dtype": {
"fp16": "float16",
"q4f16": "float16"
}
},
"transformers_version": "4.46.2",
"use_cache": true,
"vocab_size": 49152
}
finetuning/finetune_smollm2_python/final_checkpoint/generation_config.json 0 → 100644
View file @ e75bc9be
{
"_from_model_config": true,
"bos_token_id": 1,
"eos_token_id": 2,
"pad_token_id": 2,
"transformers_version": "4.46.2"
}
finetuning/finetune_smollm2_python/final_checkpoint/model.safetensors.index.json 0 → 100644
View file @ e75bc9be
{
"metadata": {
"total_size": 6858088448
},
"weight_map": {
"model.embed_tokens.weight": "model-00001-of-00002.safetensors",
"model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.q_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.v_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.q_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.v_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.q_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.v_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.11.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.q_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.v_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.q_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.v_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.q_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.v_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.q_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.v_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.q_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.v_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.16.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.q_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.v_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.17.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.17.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.17.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.17.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.17.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.q_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.17.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.17.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.17.self_attn.v_proj.base_layer.weight": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00002.safetensors",
"model.layers.17.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00002.safetensors",
"model.layers.18.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.18.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.18.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.18.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.18.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.q_proj.base_layer.weight": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.v_proj.base_layer.weight": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00002.safetensors",
"model.layers.18.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00002.safetensors",
"model.layers.19.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.19.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.19.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.19.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.19.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.q_proj.base_layer.weight": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.v_proj.base_layer.weight": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00002.safetensors",
"model.layers.19.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00002.safetensors",
"model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.q_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.v_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.20.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.20.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.q_proj.base_layer.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.v_proj.base_layer.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00002.safetensors",
"model.layers.21.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.21.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.q_proj.base_layer.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.v_proj.base_layer.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00002.safetensors",
"model.layers.22.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.22.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.q_proj.base_layer.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.v_proj.base_layer.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00002.safetensors",
"model.layers.23.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.23.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.q_proj.base_layer.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.q_proj.lora_A.default.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.q_proj.lora_B.default.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.v_proj.base_layer.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.v_proj.lora_A.default.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.v_proj.lora_B.default.weight": "model-00002-of-00002.safetensors",
"model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.q_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.v_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.q_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.v_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.q_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.v_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.6.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.q_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.v_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.q_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.v_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.8.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.q_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.v_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.9.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.q_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.q_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.q_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.v_proj.base_layer.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.v_proj.lora_A.default.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.v_proj.lora_B.default.weight": "model-00001-of-00002.safetensors",
"model.norm.weight": "model-00002-of-00002.safetensors"
}
}
finetuning/finetune_smollm2_python/final_checkpoint/tokenizer_config.json 0 → 100644
View file @ e75bc9be
{
"add_prefix_space": false,
"added_tokens_decoder": {
"0": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"1": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"2": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"3": {
"content": "<repo_name>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"4": {
"content": "<reponame>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"5": {
"content": "<file_sep>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"6": {
"content": "<filename>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"7": {
"content": "<gh_stars>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"8": {
"content": "<issue_start>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"9": {
"content": "<issue_comment>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"10": {
"content": "<issue_closed>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"11": {
"content": "<jupyter_start>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"12": {
"content": "<jupyter_text>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"13": {
"content": "<jupyter_code>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"14": {
"content": "<jupyter_output>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"15": {
"content": "<jupyter_script>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"16": {
"content": "<empty_output>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>"
],
"bos_token": "<|im_start|>",
"chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
"clean_up_tokenization_spaces": false,
"eos_token": "<|im_end|>",
"model_max_length": 2048,
"pad_token": "<|im_end|>",
"tokenizer_class": "GPT2Tokenizer",
"unk_token": "<|endoftext|>",
"vocab_size": 49152
}
finetuning/finetune_smollm2_python/final_merged_checkpoint/config.json 0 → 100644
View file @ e75bc9be
{
"_name_or_path": "HuggingFaceTB/SmolLM2-1.7B-Instruct",
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 1,
"eos_token_id": 2,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 2048,
"initializer_range": 0.02,
"intermediate_size": 8192,
"max_position_embeddings": 8192,
"mlp_bias": false,
"model_type": "llama",
"num_attention_heads": 32,
"num_hidden_layers": 24,
"num_key_value_heads": 32,
"pad_token_id": 2,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_scaling": null,
"rope_theta": 130000,
"tie_word_embeddings": true,
"torch_dtype": "bfloat16",
"transformers.js_config": {
"kv_cache_dtype": {
"fp16": "float16",
"q4f16": "float16"
}
},
"transformers_version": "4.46.2",
"use_cache": true,
"vocab_size": 49152
}
finetuning/finetune_smollm2_python/final_merged_checkpoint/generation_config.json 0 → 100644
View file @ e75bc9be
{
"_from_model_config": true,
"bos_token_id": 1,
"eos_token_id": 2,
"pad_token_id": 2,
"transformers_version": "4.46.2"
}
finetuning/infer.py 0 → 100644
View file @ e75bc9be
from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "HuggingFaceTB/SmolLM2-1.7B-Instruct"
device = "cuda" # for GPU usage or "cpu" for CPU usage
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
# for multiple GPUs install accelerate and do `model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto")`
model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
messages = [{"role": "user", "content": "Write a 100-word article on 'Benefits of Open-Source in AI research"}]
input_text=tokenizer.apply_chat_template(messages, tokenize=False)
inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)
outputs = model.generate(inputs, max_new_tokens=50, temperature=0.2, top_p=0.9, do_sample=True)
print(tokenizer.decode(outputs[0]))
finetuning/requirements.txt 0 → 100644
View file @ e75bc9be
transformers
trl
peft
accelerate
datasets
scipy
wandb # wandb offline & wandb disabled
bitsandbytes
\ No newline at end of file
finetuning/train.py 0 → 100644
View file @ e75bc9be
# Code adapted from https://github.com/huggingface/trl/blob/main/examples/research_projects/stack_llama/scripts/supervised_finetuning.py
# and https://huggingface.co/blog/gemma-peft
import argparse
import multiprocessing
import os
import torch
import transformers
from accelerate import PartialState
from datasets import load_dataset
from peft import AutoPeftModelForCausalLM, LoraConfig
from transformers import (
AutoModelForCausalLM,
BitsAndBytesConfig,
is_torch_npu_available,
is_torch_xpu_available,
logging,
set_seed,
)
from trl import SFTTrainer
def get_args():
parser = argparse.ArgumentParser()
parser.add_argument("--model_id", type=str, default="HuggingFaceTB/SmolLM2-1.7B")
parser.add_argument("--dataset_name", type=str, default="bigcode/the-stack-smol")
parser.add_argument("--subset", type=str, default="data/python")
parser.add_argument("--split", type=str, default="train")
parser.add_argument("--dataset_text_field", type=str, default="content")
parser.add_argument("--max_seq_length", type=int, default=2048)
parser.add_argument("--max_steps", type=int, default=1000)
parser.add_argument("--micro_batch_size", type=int, default=1)
parser.add_argument("--gradient_accumulation_steps", type=int, default=4)
parser.add_argument("--weight_decay", type=float, default=0.01)
parser.add_argument("--bf16", type=bool, default=True)
parser.add_argument("--use_bnb", type=bool, default=False)
parser.add_argument("--attention_dropout", type=float, default=0.1)
parser.add_argument("--learning_rate", type=float, default=2e-4)
parser.add_argument("--lr_scheduler_type", type=str, default="cosine")
parser.add_argument("--warmup_steps", type=int, default=100)
parser.add_argument("--seed", type=int, default=0)
parser.add_argument("--output_dir", type=str, default="finetune_smollm2_python")
parser.add_argument("--num_proc", type=int, default=None)
parser.add_argument("--save_merged_model", type=bool, default=True)
parser.add_argument("--push_to_hub", type=bool, default=True)
parser.add_argument("--repo_id", type=str, default="SmolLM2-1.7B-finetune")
return parser.parse_args()
def main(args):
# config
lora_config = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
target_modules=["q_proj", "v_proj"],
bias="none",
task_type="CAUSAL_LM",
)
bnb_config = None
if args.use_bnb:
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
)
# load model and dataset
token = os.environ.get("HF_TOKEN", None)
model = AutoModelForCausalLM.from_pretrained(
args.model_id,
quantization_config=bnb_config,
device_map={"": PartialState().process_index},
attention_dropout=args.attention_dropout,
)
data = load_dataset(
args.dataset_name,
data_dir=args.subset,
split=args.split,
token=token,
num_proc=args.num_proc if args.num_proc else multiprocessing.cpu_count(),
)
# setup the trainer
trainer = SFTTrainer(
model=model,
train_dataset=data,
max_seq_length=args.max_seq_length,
args=transformers.TrainingArguments(
per_device_train_batch_size=args.micro_batch_size,
gradient_accumulation_steps=args.gradient_accumulation_steps,
warmup_steps=args.warmup_steps,
max_steps=args.max_steps,
learning_rate=args.learning_rate,
lr_scheduler_type=args.lr_scheduler_type,
weight_decay=args.weight_decay,
bf16=args.bf16,
logging_strategy="steps",
logging_steps=10,
output_dir=args.output_dir,
optim="paged_adamw_8bit",
seed=args.seed,
run_name=f"train-{args.model_id.split('/')[-1]}",
report_to="wandb",
),
peft_config=lora_config,
dataset_text_field=args.dataset_text_field,
)
# launch
print("Training...")
trainer.train()
print("Saving the last checkpoint of the model")
model.save_pretrained(os.path.join(args.output_dir, "final_checkpoint/"))
if args.save_merged_model:
# Free memory for merging weights
del model
if is_torch_xpu_available():
torch.xpu.empty_cache()
elif is_torch_npu_available():
torch.npu.empty_cache()
else:
torch.cuda.empty_cache()
model = AutoPeftModelForCausalLM.from_pretrained(args.output_dir, device_map="auto", torch_dtype=torch.bfloat16)
model = model.merge_and_unload()
output_merged_dir = os.path.join(args.output_dir, "final_merged_checkpoint")
model.save_pretrained(output_merged_dir, safe_serialization=True)
if args.push_to_hub:
model.push_to_hub(args.repo_id, "Upload model")
print("Training Done! 💥")
if __name__ == "__main__":
args = get_args()
set_seed(args.seed)
os.makedirs(args.output_dir, exist_ok=True)
logging.set_verbosity_error()
main(args)
\ No newline at end of file
finetuning/train.sh 0 → 100644
View file @ e75bc9be
accelerate launch train.py \
--model_id "HuggingFaceTB/SmolLM2-1.7B-Instruct" \
--dataset_name "bigcode/the-stack-smol" \
--subset "data/python" \
--dataset_text_field "content" \
--split "train" \
--max_seq_length 2048 \
--max_steps 5000 \
--micro_batch_size 1 \
--gradient_accumulation_steps 8 \
--learning_rate 3e-4 \
--warmup_steps 100 \
--num_proc "$(nproc)"
icon.png 0 → 100644
View file @ e75bc9be
icon.png

53.8 KB

inference/smolvlm/SmolVLM_video_inference.py 0 → 100644
View file @ e75bc9be
import torch
from transformers import AutoProcessor, Idefics3ForConditionalGeneration
from PIL import Image
import cv2
import numpy as np
from typing import List
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
class VideoFrameExtractor:
def __init__(self, max_frames: int = 50):
self.max_frames = max_frames
def resize_and_center_crop(self, image: Image.Image, target_size: int) -> Image.Image:
# Get current dimensions
width, height = image.size
# Calculate new dimensions keeping aspect ratio
if width < height:
new_width = target_size
new_height = int(height * (target_size / width))
else:
new_height = target_size
new_width = int(width * (target_size / height))
# Resize
image = image.resize((new_width, new_height), Image.Resampling.LANCZOS)
# Center crop
left = (new_width - target_size) // 2
top = (new_height - target_size) // 2
right = left + target_size
bottom = top + target_size
return image.crop((left, top, right, bottom))
def extract_frames(self, video_path: str) -> List[Image.Image]:
cap = cv2.VideoCapture(video_path)
if not cap.isOpened():
raise ValueError(f"Could not open video: {video_path}")
# Get video properties
total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
fps = int(cap.get(cv2.CAP_PROP_FPS))
# Calculate frame indices to extract (1fps)
frame_indices = list(range(0, total_frames, fps))
# If we have more frames than max_frames, sample evenly
if len(frame_indices) > self.max_frames:
indices = np.linspace(0, len(frame_indices) - 1, self.max_frames, dtype=int)
frame_indices = [frame_indices[i] for i in indices]
frames = []
for frame_idx in frame_indices:
cap.set(cv2.CAP_PROP_POS_FRAMES, frame_idx)
ret, frame = cap.read()
if ret:
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
pil_image = Image.fromarray(frame)
pil_image = self.resize_and_center_crop(pil_image, 384)
frames.append(pil_image)
cap.release()
return frames
def load_model(checkpoint_path: str, base_model_id: str = "HuggingFaceTB/SmolVLM-Instruct", device: str = "cuda"):
# Load processor from original model
processor = AutoProcessor.from_pretrained(base_model_id)
if checkpoint_path:
# Load fine-tuned model from checkpoint
model = Idefics3ForConditionalGeneration.from_pretrained(
checkpoint_path,
torch_dtype=torch.bfloat16,
device_map=device
)
else:
model = Idefics3ForConditionalGeneration.from_pretrained(
base_model_id,
torch_dtype=torch.bfloat16,
device_map=device
)
# Configure processor for video frames
processor.image_processor.size = (384, 384)
processor.image_processor.do_resize = False
processor.image_processor.do_image_splitting = False
return model, processor
def generate_response(model, processor, video_path: str, question: str, max_frames: int = 50):
# Extract frames
frame_extractor = VideoFrameExtractor(max_frames)
frames = frame_extractor.extract_frames(video_path)
logger.info(f"Extracted {len(frames)} frames from video")
# Create prompt with frames
image_tokens = [{"type": "image"} for _ in range(len(frames))]
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "Answer briefly."},
*image_tokens,
{"type": "text", "text": question}
]
}
]
# Process inputs
inputs = processor(
text=processor.apply_chat_template(messages, add_generation_prompt=True),
images=[img for img in frames],
return_tensors="pt"
).to(model.device)
# Generate response
outputs = model.generate(
**inputs,
max_new_tokens=100,
num_beams=5,
temperature=0.7,
do_sample=True,
use_cache=True
)
# Decode response
response = processor.decode(outputs[0], skip_special_tokens=True)
return response
def main():
# Configuration
#checkpoint_path = "/path/to/your/checkpoint"
checkpoint_path = None
base_model_id = "HuggingFaceTB/SmolVLM-Instruct"
video_path = "/path/to/video.mp4"
question = "Describe the video"
device = "cuda" if torch.cuda.is_available() else "cpu"
# Load model
logger.info("Loading model...")
model, processor = load_model(checkpoint_path, base_model_id, device)
# Generate response
logger.info("Generating response...")
response = generate_response(model, processor, video_path, question)
# Print results
print("Question:", question)
print("Response:", response)
if __name__ == "__main__":
main()
\ No newline at end of file
local_inference/README.md 0 → 100644
View file @ e75bc9be
# Local inference
You can use SmolLM2 models locally with frameworks like Transformers.js, llama.cpp, MLX and MLC.
Here you can find the code for running SmolLM locally using each of these libraries. You can also find the conversions of SmolLM & SmolLM2 in these collections: [SmolLM1](https://huggingface.co/collections/HuggingFaceTB/local-smollms-66c0f3b2a15b4eed7fb198d0) and [SmolLM2](https://huggingface.co/collections/HuggingFaceTB/smollm2-6723884218bcda64b34d7db9).
Please first install each library by following its documentation:
- [Transformers.js](https://github.com/huggingface/transformers.js)
- [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)
- [MLX](https://github.com/ml-explore/mlx)
- [MLC](https://github.com/mlc-ai/web-llm)
## Demos
Below are some demos we built for running SmolLM models in the browser:
- [WebGPU demo](https://huggingface.co/spaces/HuggingFaceTB/SmolLM2-1.7B-Instruct-WebGPU ) of SmolLM2 1.7B Instruct powered by Transformers.js and ONNX Runtime Web:
- [Bunny B1](https://github.com/dottxt-ai/demos/tree/main/its-a-smol-world) mapping natural language requests to local aplication calls using function calling and structured generation by [outlines](https://github.com/dottxt-ai/outlines).
- [Instant SmolLM](https://huggingface.co/spaces/HuggingFaceTB/instant-smollm) powered by MLC for real-time generations of SmolLM-360M-Instruct.
The models are also available on [Ollama](https://ollama.com/library/smollm2) and [PocketPal-AI](https://github.com/a-ghorbani/pocketpal-ai).
local_inference/llama-cpp-python.py 0 → 100644
View file @ e75bc9be
from llama_cpp import Llama
llm = Llama.from_pretrained(
repo_id="HuggingFaceTB/SmolLM2-1.7B-Instruct-GGUF",
filename="*q4_k_m.gguf",
verbose=False
)
output = llm(
"Q: Name the planets in the solar system? A: ", # Prompt
max_tokens=32, # Generate up to 32 tokens, set to None to generate up to the end of the context window
stop=["Q:", "\n"], # Stop generating just before the model would generate a new question
echo=True # Echo the prompt back in the output
) # Generate a completion, can also call create_completion
print(output)
\ No newline at end of file
local_inference/mlc.py 0 → 100644
View file @ e75bc9be
from mlc_llm import MLCEngine
# Create engine
model = "HF://mlc-ai/SmolLM2-1.7B-Instruct-q0f16-MLC"
engine = MLCEngine(model)
# Run chat completion in OpenAI API.
for response in engine.chat.completions.create(
messages=[{"role": "user", "content": "What is the meaning of life?"}],
model=model,
stream=True,
):
for choice in response.choices:
print(choice.delta.content, end="", flush=True)
print("\n")
engine.terminate()
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment