{ "cells": [ { "cell_type": "markdown", "source": [ "# 使用微调的方法让模型对新闻分类更加准确\n", "\n", "在本实操手册中，开发者将使用ChatGLM3-6B base模型，对新闻分类数据集进行微调，并使用微调后的模型进行推理。\n", "本操作手册使用公开数据集，数据集中包含了新闻标题和新闻关键词，开发者需要根据这些信息，将新闻分类到15个类别中的一个。\n", "为了体现模型高效的学习能力，以及让用户更快的学习本手册，我们只使用了数据集中的一小部分数据，实际上，数据集中包含了超过40万条新闻数据。\n", "\n", "## 硬件要求\n", "本实践手册需要使用 FP16 精度的模型进行推理，因此，我们推荐使用至少 16GB 显存的英伟达 GPU 来完成本实践手册。\n" ], "metadata": { "collapsed": false } }, { "cell_type": "markdown", "source": [ "首先，我们将原始的数据集格式转换为用于微调的`jsonl`格式，以方便进行微调。" ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 1, "outputs": [], "source": [ "import json\n", "\n", "# 路径可以根据实际情况修改\n", "input_file_path = 'data/toutiao_cat_data_example.txt'\n", "output_file_path = 'data/toutiao_cat_data_example.jsonl'\n", "\n", "# 提示词\n", "prompt_prefix = \"\"\"\n", "你是一个专业的新闻专家，请根据我提供的新闻信息，包括新闻标题，新闻关键词等信息，你要对每一行新闻类别进行分类并告诉我结果，不要返回其他信息和多于的文字，这些类别是:\n", "news_story\n", "news_culture\n", "news_sports\n", "news_finance\n", "news_house\n", "news_car\n", "news_edu\n", "news_tech\n", "news_military\n", "news_travel\n", "news_world\n", "stock\n", "news_agriculture\n", "news_game\n", "请选择其中一个类别并返回，你只要返回类别的名称，不要返回其他信息。让我们开始吧:\n", "\"\"\"\n", "\n", "# 分类代码和名称的映射\n", "category_map = {\n", " \"100\": \"news_story\",\n", " \"101\": \"news_culture\",\n", " \"102\": \"news_entertainment\",\n", " \"103\": \"news_sports\",\n", " \"104\": \"news_finance\",\n", " \"106\": \"news_house\",\n", " \"107\": \"news_car\",\n", " \"108\": \"news_edu\",\n", " \"109\": \"news_tech\",\n", " \"110\": \"news_military\",\n", " \"112\": \"news_travel\",\n", " \"113\": \"news_world\",\n", " \"114\": \"stock\",\n", " \"115\": \"news_agriculture\",\n", " \"116\": \"news_game\"\n", "}\n", "\n", "def process_line(line):\n", " # 分割每行数据\n", " parts = line.strip().split('_!_')\n", " if len(parts) != 5:\n", " return None\n", "\n", " # 提取所需字段\n", " _, category_code, _, news_title, news_keywords = parts\n", "\n", " # 构造 JSON 对象\n", " news_title = news_title if news_title else \"无\"\n", " news_keywords = news_keywords if news_keywords else \"无\"\n", " json_obj = {\n", " \"context\": prompt_prefix + f\"新闻标题: {news_title}\\n 新闻关键词: {news_keywords}\\n\",\n", " \"target\": category_map.get(category_code, \"无\")\n", " }\n", " return json_obj\n", "\n", "def convert_to_jsonl(input_path, output_path):\n", " with open(input_path, 'r', encoding='utf-8') as infile, \\\n", " open(output_path, 'w', encoding='utf-8') as outfile:\n", " for line in infile:\n", " json_obj = process_line(line)\n", " if json_obj:\n", " json_line = json.dumps(json_obj, ensure_ascii=False)\n", " outfile.write(json_line + '\\n')\n", "\n", "# 运行转换函数\n", "convert_to_jsonl(input_file_path, output_file_path)" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-11-24T13:43:59.281779Z", "end_time": "2023-11-24T13:43:59.330679Z" } } }, { "cell_type": "markdown", "source": [ "## 使用没有微调的模型进行推理\n", "首先，我们先试用原本的模基座模型进行推理，并查看效果。" ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 2, "outputs": [], "source": [ "PROMPT = \"\"\"\n", "你是一个专业的新闻专家，请根据我提供的新闻信息，包括新闻标题，新闻关键词等信息，你要对每一行新闻类别进行分类并告诉我结果，不要返回其他信息和多于的文字，这些类别是:\n", "news_story\n", "news_culture\n", "news_sports\n", "news_finance\n", "news_house\n", "news_car\n", "news_edu\n", "news_tech\n", "news_military\n", "news_travel\n", "news_world\n", "stock\n", "news_agriculture\n", "news_game\n", "请选择其中一个类别并返回，你只要返回类别的名称，不要返回其他信息。让我们开始吧:\n", "新闻标题：华为手机扛下敌人子弹，是什么技术让其在战争中大放异彩？\n", "新闻关键词: 华为手机\n", "\"\"\"" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-11-24T13:43:59.377486Z", "end_time": "2023-11-24T13:43:59.392276Z" } } }, { "cell_type": "code", "execution_count": 3, "outputs": [ { "data": { "text/plain": "Loading checkpoint shards: 0%| | 0/7 [00:00,\r\n", "ignore_data_skip=False,\r\n", "include_inputs_for_metrics=False,\r\n", "include_tokens_per_second=False,\r\n", "jit_mode_eval=False,\r\n", "label_names=None,\r\n", "label_smoothing_factor=0.0,\r\n", "learning_rate=2e-05,\r\n", "length_column_name=length,\r\n", "load_best_model_at_end=False,\r\n", "local_rank=0,\r\n", "log_level=passive,\r\n", "log_level_replica=warning,\r\n", "log_on_each_node=True,\r\n", "logging_dir=output/news-20231124-134412-2e-05/runs/Nov24_13-44-15_iZwz90pbe3r8jeoaaobbf0Z,\r\n", "logging_first_step=False,\r\n", "logging_nan_inf_filter=True,\r\n", "logging_steps=1.0,\r\n", "logging_strategy=steps,\r\n", "lr_scheduler_type=linear,\r\n", "max_grad_norm=1.0,\r\n", "max_steps=300,\r\n", "metric_for_best_model=None,\r\n", "mp_parameters=,\r\n", "neftune_noise_alpha=None,\r\n", "no_cuda=False,\r\n", "num_train_epochs=3.0,\r\n", "optim=adamw_torch,\r\n", "optim_args=None,\r\n", "output_dir=output/news-20231124-134412-2e-05,\r\n", "overwrite_output_dir=False,\r\n", "past_index=-1,\r\n", "per_device_eval_batch_size=8,\r\n", "per_device_train_batch_size=1,\r\n", "predict_with_generate=False,\r\n", "prediction_loss_only=False,\r\n", "push_to_hub=False,\r\n", "push_to_hub_model_id=None,\r\n", "push_to_hub_organization=None,\r\n", "push_to_hub_token=,\r\n", "ray_scope=last,\r\n", "remove_unused_columns=True,\r\n", "report_to=[],\r\n", "resume_from_checkpoint=None,\r\n", "run_name=output/news-20231124-134412-2e-05,\r\n", "save_on_each_node=False,\r\n", "save_safetensors=True,\r\n", "save_steps=100,\r\n", "save_strategy=steps,\r\n", "save_total_limit=None,\r\n", "seed=42,\r\n", "skip_memory_metrics=True,\r\n", "sortish_sampler=False,\r\n", "split_batches=False,\r\n", "tf32=None,\r\n", "torch_compile=False,\r\n", "torch_compile_backend=None,\r\n", "torch_compile_mode=None,\r\n", "torchdynamo=None,\r\n", "tpu_metrics_debug=False,\r\n", "tpu_num_cores=None,\r\n", "use_cpu=False,\r\n", "use_ipex=False,\r\n", "use_legacy_prediction_loop=False,\r\n", "use_mps_device=False,\r\n", "warmup_ratio=0.0,\r\n", "warmup_steps=0,\r\n", "weight_decay=0.0,\r\n", ")\r\n", "[INFO|tokenization_utils_base.py:2020] 2023-11-24 13:44:15,353 >> loading file tokenizer.model\r\n", "[INFO|tokenization_utils_base.py:2020] 2023-11-24 13:44:15,353 >> loading file added_tokens.json\r\n", "[INFO|tokenization_utils_base.py:2020] 2023-11-24 13:44:15,353 >> loading file special_tokens_map.json\r\n", "[INFO|tokenization_utils_base.py:2020] 2023-11-24 13:44:15,353 >> loading file tokenizer_config.json\r\n", "[INFO|tokenization_utils_base.py:2020] 2023-11-24 13:44:15,353 >> loading file tokenizer.json\r\n", "[INFO|configuration_utils.py:715] 2023-11-24 13:44:15,453 >> loading configuration file /Models/chatglm3-6b-base/config.json\r\n", "[INFO|configuration_utils.py:715] 2023-11-24 13:44:15,454 >> loading configuration file /Models/chatglm3-6b-base/config.json\r\n", "[INFO|configuration_utils.py:777] 2023-11-24 13:44:15,454 >> Model config ChatGLMConfig {\r\n", " \"_name_or_path\": \"/Models/chatglm3-6b-base\",\r\n", " \"add_bias_linear\": false,\r\n", " \"add_qkv_bias\": true,\r\n", " \"apply_query_key_layer_scaling\": true,\r\n", " \"apply_residual_connection_post_layernorm\": false,\r\n", " \"architectures\": [\r\n", " \"ChatGLMModel\"\r\n", " ],\r\n", " \"attention_dropout\": 0.0,\r\n", " \"attention_softmax_in_fp32\": true,\r\n", " \"auto_map\": {\r\n", " \"AutoConfig\": \"configuration_chatglm.ChatGLMConfig\",\r\n", " \"AutoModel\": \"modeling_chatglm.ChatGLMForConditionalGeneration\",\r\n", " \"AutoModelForCausalLM\": \"modeling_chatglm.ChatGLMForConditionalGeneration\",\r\n", " \"AutoModelForSeq2SeqLM\": \"modeling_chatglm.ChatGLMForConditionalGeneration\",\r\n", " \"AutoModelForSequenceClassification\": \"modeling_chatglm.ChatGLMForSequenceClassification\"\r\n", " },\r\n", " \"bias_dropout_fusion\": true,\r\n", " \"classifier_dropout\": null,\r\n", " \"eos_token_id\": 2,\r\n", " \"ffn_hidden_size\": 13696,\r\n", " \"fp32_residual_connection\": false,\r\n", " \"hidden_dropout\": 0.0,\r\n", " \"hidden_size\": 4096,\r\n", " \"kv_channels\": 128,\r\n", " \"layernorm_epsilon\": 1e-05,\r\n", " \"model_type\": \"chatglm\",\r\n", " \"multi_query_attention\": true,\r\n", " \"multi_query_group_num\": 2,\r\n", " \"num_attention_heads\": 32,\r\n", " \"num_layers\": 28,\r\n", " \"original_rope\": true,\r\n", " \"pad_token_id\": 0,\r\n", " \"padded_vocab_size\": 65024,\r\n", " \"post_layer_norm\": true,\r\n", " \"pre_seq_len\": null,\r\n", " \"prefix_projection\": false,\r\n", " \"quantization_bit\": 0,\r\n", " \"rmsnorm\": true,\r\n", " \"seq_length\": 32768,\r\n", " \"tie_word_embeddings\": false,\r\n", " \"torch_dtype\": \"float16\",\r\n", " \"transformers_version\": \"4.35.2\",\r\n", " \"use_cache\": true,\r\n", " \"vocab_size\": 65024\r\n", "}\r\n", "\r\n", "[INFO|modeling_utils.py:3118] 2023-11-24 13:44:15,517 >> loading weights file /Models/chatglm3-6b-base/pytorch_model.bin.index.json\r\n", "[INFO|configuration_utils.py:791] 2023-11-24 13:44:15,517 >> Generate config GenerationConfig {\r\n", " \"eos_token_id\": 2,\r\n", " \"pad_token_id\": 0\r\n", "}\r\n", "\r\n", "Loading checkpoint shards: 100%|██████████████████| 7/7 [00:07<00:00, 1.02s/it]\r\n", "[INFO|modeling_utils.py:3950] 2023-11-24 13:44:22,704 >> All model checkpoint weights were used when initializing ChatGLMForConditionalGeneration.\r\n", "\r\n", "[INFO|modeling_utils.py:3958] 2023-11-24 13:44:22,704 >> All the weights of ChatGLMForConditionalGeneration were initialized from the model checkpoint at /Models/chatglm3-6b-base.\r\n", "If your task is similar to the task the model of the checkpoint was trained on, you can already use ChatGLMForConditionalGeneration for predictions without further training.\r\n", "[INFO|modeling_utils.py:3525] 2023-11-24 13:44:22,706 >> Generation config file not found, using a generation config created from the model config.\r\n", "Train dataset size: 4999\r\n", "Sanity Check >>>>>>>>>>>>>\r\n", " '[gMASK]': 64790 -> -100\r\n", " 'sop': 64792 -> -100\r\n", " '': 30910 -> -100\r\n", " '\\n': 13 -> -100\r\n", " '你': 54622 -> -100\r\n", " '是一个': 32103 -> -100\r\n", " '专业的': 34917 -> -100\r\n", " '新闻': 31935 -> -100\r\n", " '专家': 32114 -> -100\r\n", " '，': 31123 -> -100\r\n", " '请': 55073 -> -100\r\n", " '根据': 31793 -> -100\r\n", " '我': 54546 -> -100\r\n", " '提供的': 35100 -> -100\r\n", " '新闻': 31935 -> -100\r\n", " '信息': 31707 -> -100\r\n", " '，': 31123 -> -100\r\n", " '包括': 31779 -> -100\r\n", " '新闻': 31935 -> -100\r\n", " '标题': 34490 -> -100\r\n", " '，': 31123 -> -100\r\n", " '新闻': 31935 -> -100\r\n", " '关键词': 35075 -> -100\r\n", " '等信息': 46172 -> -100\r\n", " '，': 31123 -> -100\r\n", " '你要': 34526 -> -100\r\n", " '对': 54570 -> -100\r\n", " '每一': 32467 -> -100\r\n", " '行': 54560 -> -100\r\n", " '新闻': 31935 -> -100\r\n", " '类别': 38724 -> -100\r\n", " '进行': 31636 -> -100\r\n", " '分类': 33328 -> -100\r\n", " '并': 54724 -> -100\r\n", " '告诉我': 38953 -> -100\r\n", " '结果': 31951 -> -100\r\n", " '，': 31123 -> -100\r\n", " '不要': 31844 -> -100\r\n", " '返回': 34891 -> -100\r\n", " '其他': 31722 -> -100\r\n", " '信息和': 52701 -> -100\r\n", " '多': 54573 -> -100\r\n", " '于': 54579 -> -100\r\n", " '的文字': 48746 -> -100\r\n", " '，': 31123 -> -100\r\n", " '这些': 31704 -> -100\r\n", " '类别': 38724 -> -100\r\n", " '是': 54532 -> -100\r\n", " ':': 30954 -> -100\r\n", " '\\n': 13 -> -100\r\n", " 'news': 8480 -> -100\r\n", " '_': 30962 -> -100\r\n", " 'story': 12553 -> -100\r\n", " '\\n': 13 -> -100\r\n", " 'news': 8480 -> -100\r\n", " '_': 30962 -> -100\r\n", " 'culture': 27458 -> -100\r\n", " '\\n': 13 -> -100\r\n", " 'news': 8480 -> -100\r\n", " '_': 30962 -> -100\r\n", " 's': 30917 -> -100\r\n", " 'ports': 3915 -> -100\r\n", " '\\n': 13 -> -100\r\n", " 'news': 8480 -> -100\r\n", " '_': 30962 -> -100\r\n", " 'fin': 6242 -> -100\r\n", " 'ance': 562 -> -100\r\n", " '\\n': 13 -> -100\r\n", " 'news': 8480 -> -100\r\n", " '_': 30962 -> -100\r\n", " 'house': 4199 -> -100\r\n", " '\\n': 13 -> -100\r\n", " 'news': 8480 -> -100\r\n", " '_': 30962 -> -100\r\n", " 'car': 6747 -> -100\r\n", " '\\n': 13 -> -100\r\n", " 'news': 8480 -> -100\r\n", " '_': 30962 -> -100\r\n", " 'edu': 7473 -> -100\r\n", " '\\n': 13 -> -100\r\n", " 'news': 8480 -> -100\r\n", " '_': 30962 -> -100\r\n", " 'tech': 12232 -> -100\r\n", " '\\n': 13 -> -100\r\n", " 'news': 8480 -> -100\r\n", " '_': 30962 -> -100\r\n", " 'mil': 20477 -> -100\r\n", " 'itary': 2733 -> -100\r\n", " '\\n': 13 -> -100\r\n", " 'news': 8480 -> -100\r\n", " '_': 30962 -> -100\r\n", " 'tra': 7441 -> -100\r\n", " 'vel': 609 -> -100\r\n", " '\\n': 13 -> -100\r\n", " 'news': 8480 -> -100\r\n", " '_': 30962 -> -100\r\n", " 'world': 8515 -> -100\r\n", " '\\n': 13 -> -100\r\n", " 'stock': 14148 -> -100\r\n", " '\\n': 13 -> -100\r\n", " 'news': 8480 -> -100\r\n", " '_': 30962 -> -100\r\n", " 'ag': 369 -> -100\r\n", " 'ric': 995 -> -100\r\n", " 'ulture': 4768 -> -100\r\n", " '\\n': 13 -> -100\r\n", " 'news': 8480 -> -100\r\n", " '_': 30962 -> -100\r\n", " 'game': 8947 -> -100\r\n", " '\\n': 13 -> -100\r\n", " '请': 55073 -> -100\r\n", " '选择': 31768 -> -100\r\n", " '其中一个': 46753 -> -100\r\n", " '类别': 38724 -> -100\r\n", " '并': 54724 -> -100\r\n", " '返回': 34891 -> -100\r\n", " '，': 31123 -> -100\r\n", " '你': 54622 -> -100\r\n", " '只要': 32100 -> -100\r\n", " '返回': 34891 -> -100\r\n", " '类': 54931 -> -100\r\n", " '别的': 34752 -> -100\r\n", " '名称': 33624 -> -100\r\n", " '，': 31123 -> -100\r\n", " '不要': 31844 -> -100\r\n", " '返回': 34891 -> -100\r\n", " '其他': 31722 -> -100\r\n", " '信息': 31707 -> -100\r\n", " '。': 31155 -> -100\r\n", " '让我们': 32817 -> -100\r\n", " '开始': 31699 -> -100\r\n", " '吧': 55370 -> -100\r\n", " ':': 30954 -> -100\r\n", " '\\n': 13 -> -100\r\n", " '新闻': 31935 -> -100\r\n", " '标题': 34490 -> -100\r\n", " ':': 30954 -> -100\r\n", " '': 30910 -> -100\r\n", " '京城': 46921 -> -100\r\n", " '最': 54628 -> -100\r\n", " '值得': 32421 -> -100\r\n", " '你来': 52586 -> -100\r\n", " '场': 54686 -> -100\r\n", " '文化': 31653 -> -100\r\n", " '之旅': 35383 -> -100\r\n", " '的': 54530 -> -100\r\n", " '博物馆': 32964 -> -100\r\n", " '\\n': 13 -> -100\r\n", " '新闻': 45302 -> -100\r\n", " '关键词': 35075 -> -100\r\n", " ':': 30954 -> -100\r\n", " '': 30910 -> -100\r\n", " '保利': 46340 -> -100\r\n", " '集团': 31839 -> -100\r\n", " ',': 30932 -> -100\r\n", " '马': 54988 -> -100\r\n", " '未': 54933 -> -100\r\n", " '都': 54606 -> -100\r\n", " ',': 30932 -> -100\r\n", " '中国': 31626 -> -100\r\n", " '科学技术': 35587 -> -100\r\n", " '馆': 55294 -> -100\r\n", " ',': 30932 -> -100\r\n", " '博物馆': 32964 -> -100\r\n", " ',': 30932 -> -100\r\n", " '新中国': 35873 -> -100\r\n", " '\\n': 13 -> -100\r\n", " 'news': 2374 -> 2374\r\n", " '_': 30962 -> 30962\r\n", " 'culture': 27458 -> 27458\r\n", " '': 2 -> 2\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", " '': 0 -> -100\r\n", "<<<<<<<<<<<<< Sanity Check\r\n", "[INFO|trainer.py:544] 2023-11-24 13:44:24,838 >> max_steps is given, it will override any value given in num_train_epochs\r\n", "[INFO|trainer.py:1723] 2023-11-24 13:44:25,787 >> ***** Running training *****\r\n", "[INFO|trainer.py:1724] 2023-11-24 13:44:25,787 >> Num examples = 4,999\r\n", "[INFO|trainer.py:1725] 2023-11-24 13:44:25,787 >> Num Epochs = 1\r\n", "[INFO|trainer.py:1726] 2023-11-24 13:44:25,787 >> Instantaneous batch size per device = 1\r\n", "[INFO|trainer.py:1729] 2023-11-24 13:44:25,787 >> Total train batch size (w. parallel, distributed & accumulation) = 2\r\n", "[INFO|trainer.py:1730] 2023-11-24 13:44:25,787 >> Gradient Accumulation steps = 2\r\n", "[INFO|trainer.py:1731] 2023-11-24 13:44:25,788 >> Total optimization steps = 300\r\n", "[INFO|trainer.py:1732] 2023-11-24 13:44:25,788 >> Number of trainable parameters = 1,949,696\r\n", " 0%| | 0/300 [00:00> tokenizer config file saved in output/news-20231124-134412-2e-05/checkpoint-100/tokenizer_config.json\r\n", "[INFO|tokenization_utils_base.py:2437] 2023-11-24 13:46:55,413 >> Special tokens file saved in output/news-20231124-134412-2e-05/checkpoint-100/special_tokens_map.json\r\n", "{'loss': 0.0081, 'learning_rate': 1.3266666666666668e-05, 'epoch': 0.04} \r\n", "{'loss': 0.0147, 'learning_rate': 1.3200000000000002e-05, 'epoch': 0.04} \r\n", "{'loss': 0.0156, 'learning_rate': 1.3133333333333334e-05, 'epoch': 0.04} \r\n", "{'loss': 0.0188, 'learning_rate': 1.3066666666666668e-05, 'epoch': 0.04} \r\n", "{'loss': 0.1504, 'learning_rate': 1.3000000000000001e-05, 'epoch': 0.04} \r\n", "{'loss': 0.1548, 'learning_rate': 1.2933333333333334e-05, 'epoch': 0.04} \r\n", "{'loss': 0.1723, 'learning_rate': 1.2866666666666667e-05, 'epoch': 0.04} \r\n", "{'loss': 0.0258, 'learning_rate': 1.2800000000000001e-05, 'epoch': 0.04} \r\n", "{'loss': 0.0329, 'learning_rate': 1.2733333333333336e-05, 'epoch': 0.04} \r\n", "{'loss': 0.0042, 'learning_rate': 1.2666666666666667e-05, 'epoch': 0.04} \r\n", "{'loss': 0.2594, 'learning_rate': 1.2600000000000001e-05, 'epoch': 0.04} \r\n", "{'loss': 0.3348, 'learning_rate': 1.2533333333333336e-05, 'epoch': 0.04} \r\n", "{'loss': 0.1135, 'learning_rate': 1.2466666666666667e-05, 'epoch': 0.05} \r\n", "{'loss': 0.0225, 'learning_rate': 1.2400000000000002e-05, 'epoch': 0.05} \r\n", "{'loss': 0.0036, 'learning_rate': 1.2333333333333334e-05, 'epoch': 0.05} \r\n", "{'loss': 0.0105, 'learning_rate': 1.2266666666666667e-05, 'epoch': 0.05} \r\n", "{'loss': 0.0565, 'learning_rate': 1.22e-05, 'epoch': 0.05} \r\n", "{'loss': 0.0286, 'learning_rate': 1.2133333333333335e-05, 'epoch': 0.05} \r\n", "{'loss': 0.5644, 'learning_rate': 1.206666666666667e-05, 'epoch': 0.05} \r\n", "{'loss': 0.3605, 'learning_rate': 1.2e-05, 'epoch': 0.05} \r\n", "{'loss': 0.6441, 'learning_rate': 1.1933333333333335e-05, 'epoch': 0.05} \r\n", "{'loss': 0.0111, 'learning_rate': 1.186666666666667e-05, 'epoch': 0.05} \r\n", "{'loss': 0.0098, 'learning_rate': 1.18e-05, 'epoch': 0.05} \r\n", "{'loss': 0.9559, 'learning_rate': 1.1733333333333335e-05, 'epoch': 0.05} \r\n", "{'loss': 0.0042, 'learning_rate': 1.1666666666666668e-05, 'epoch': 0.05} \r\n", "{'loss': 0.4136, 'learning_rate': 1.16e-05, 'epoch': 0.05} \r\n", "{'loss': 0.5268, 'learning_rate': 1.1533333333333334e-05, 'epoch': 0.05} \r\n", "{'loss': 0.1325, 'learning_rate': 1.1466666666666668e-05, 'epoch': 0.05} \r\n", "{'loss': 0.0051, 'learning_rate': 1.14e-05, 'epoch': 0.05} \r\n", "{'loss': 0.0006, 'learning_rate': 1.1333333333333334e-05, 'epoch': 0.05} \r\n", "{'loss': 0.007, 'learning_rate': 1.1266666666666668e-05, 'epoch': 0.05} \r\n", "{'loss': 0.441, 'learning_rate': 1.1200000000000001e-05, 'epoch': 0.05} \r\n", "{'loss': 0.0102, 'learning_rate': 1.1133333333333334e-05, 'epoch': 0.05} \r\n", "{'loss': 0.0415, 'learning_rate': 1.1066666666666669e-05, 'epoch': 0.05} \r\n", "{'loss': 0.0206, 'learning_rate': 1.1000000000000001e-05, 'epoch': 0.05} \r\n", "{'loss': 0.1697, 'learning_rate': 1.0933333333333334e-05, 'epoch': 0.05} \r\n", "{'loss': 0.2054, 'learning_rate': 1.0866666666666667e-05, 'epoch': 0.05} \r\n", "{'loss': 0.005, 'learning_rate': 1.0800000000000002e-05, 'epoch': 0.06} \r\n", "{'loss': 0.9604, 'learning_rate': 1.0733333333333333e-05, 'epoch': 0.06} \r\n", "{'loss': 0.0043, 'learning_rate': 1.0666666666666667e-05, 'epoch': 0.06} \r\n", "{'loss': 0.4815, 'learning_rate': 1.0600000000000002e-05, 'epoch': 0.06} \r\n", "{'loss': 0.4817, 'learning_rate': 1.0533333333333333e-05, 'epoch': 0.06} \r\n", "{'loss': 1.0441, 'learning_rate': 1.0466666666666668e-05, 'epoch': 0.06} \r\n", "{'loss': 1.2198, 'learning_rate': 1.04e-05, 'epoch': 0.06} \r\n", "{'loss': 0.019, 'learning_rate': 1.0333333333333335e-05, 'epoch': 0.06} \r\n", "{'loss': 0.0127, 'learning_rate': 1.0266666666666668e-05, 'epoch': 0.06} \r\n", "{'loss': 0.0606, 'learning_rate': 1.02e-05, 'epoch': 0.06} \r\n", "{'loss': 0.1299, 'learning_rate': 1.0133333333333335e-05, 'epoch': 0.06} \r\n", "{'loss': 0.0678, 'learning_rate': 1.0066666666666666e-05, 'epoch': 0.06} \r\n", "{'loss': 0.0037, 'learning_rate': 1e-05, 'epoch': 0.06} \r\n", "{'loss': 0.4082, 'learning_rate': 9.933333333333334e-06, 'epoch': 0.06} \r\n", "{'loss': 0.1784, 'learning_rate': 9.866666666666668e-06, 'epoch': 0.06} \r\n", "{'loss': 0.0074, 'learning_rate': 9.800000000000001e-06, 'epoch': 0.06} \r\n", "{'loss': 0.148, 'learning_rate': 9.733333333333334e-06, 'epoch': 0.06} \r\n", "{'loss': 0.0005, 'learning_rate': 9.666666666666667e-06, 'epoch': 0.06} \r\n", "{'loss': 0.3039, 'learning_rate': 9.600000000000001e-06, 'epoch': 0.06} \r\n", "{'loss': 0.0024, 'learning_rate': 9.533333333333334e-06, 'epoch': 0.06} \r\n", "{'loss': 0.2657, 'learning_rate': 9.466666666666667e-06, 'epoch': 0.06} \r\n", "{'loss': 0.0117, 'learning_rate': 9.4e-06, 'epoch': 0.06} \r\n", "{'loss': 0.001, 'learning_rate': 9.333333333333334e-06, 'epoch': 0.06} \r\n", "{'loss': 0.0045, 'learning_rate': 9.266666666666667e-06, 'epoch': 0.06} \r\n", "{'loss': 0.0013, 'learning_rate': 9.200000000000002e-06, 'epoch': 0.06} \r\n", "{'loss': 0.4838, 'learning_rate': 9.133333333333335e-06, 'epoch': 0.07} \r\n", "{'loss': 0.164, 'learning_rate': 9.066666666666667e-06, 'epoch': 0.07} \r\n", "{'loss': 0.0084, 'learning_rate': 9e-06, 'epoch': 0.07} \r\n", "{'loss': 0.0207, 'learning_rate': 8.933333333333333e-06, 'epoch': 0.07} \r\n", "{'loss': 0.3102, 'learning_rate': 8.866666666666668e-06, 'epoch': 0.07} \r\n", "{'loss': 0.0723, 'learning_rate': 8.8e-06, 'epoch': 0.07} \r\n", "{'loss': 0.0009, 'learning_rate': 8.733333333333333e-06, 'epoch': 0.07} \r\n", "{'loss': 0.1778, 'learning_rate': 8.666666666666668e-06, 'epoch': 0.07} \r\n", "{'loss': 1.0444, 'learning_rate': 8.6e-06, 'epoch': 0.07} \r\n", "{'loss': 0.1915, 'learning_rate': 8.533333333333335e-06, 'epoch': 0.07} \r\n", "{'loss': 0.0191, 'learning_rate': 8.466666666666668e-06, 'epoch': 0.07} \r\n", "{'loss': 0.0044, 'learning_rate': 8.400000000000001e-06, 'epoch': 0.07} \r\n", "{'loss': 0.5965, 'learning_rate': 8.333333333333334e-06, 'epoch': 0.07} \r\n", "{'loss': 0.0201, 'learning_rate': 8.266666666666667e-06, 'epoch': 0.07} \r\n", "{'loss': 1.1224, 'learning_rate': 8.2e-06, 'epoch': 0.07} \r\n", "{'loss': 0.0041, 'learning_rate': 8.133333333333334e-06, 'epoch': 0.07} \r\n", "{'loss': 0.0071, 'learning_rate': 8.066666666666667e-06, 'epoch': 0.07} \r\n", "{'loss': 0.0885, 'learning_rate': 8.000000000000001e-06, 'epoch': 0.07} \r\n", "{'loss': 0.394, 'learning_rate': 7.933333333333334e-06, 'epoch': 0.07} \r\n", "{'loss': 0.0002, 'learning_rate': 7.866666666666667e-06, 'epoch': 0.07} \r\n", "{'loss': 0.2249, 'learning_rate': 7.800000000000002e-06, 'epoch': 0.07} \r\n", "{'loss': 0.2116, 'learning_rate': 7.733333333333334e-06, 'epoch': 0.07} \r\n", "{'loss': 0.0159, 'learning_rate': 7.666666666666667e-06, 'epoch': 0.07} \r\n", "{'loss': 0.0027, 'learning_rate': 7.600000000000001e-06, 'epoch': 0.07} \r\n", "{'loss': 0.1615, 'learning_rate': 7.533333333333334e-06, 'epoch': 0.07} \r\n", "{'loss': 0.0401, 'learning_rate': 7.4666666666666675e-06, 'epoch': 0.08} \r\n", "{'loss': 0.0089, 'learning_rate': 7.4e-06, 'epoch': 0.08} \r\n", "{'loss': 0.0712, 'learning_rate': 7.333333333333333e-06, 'epoch': 0.08} \r\n", "{'loss': 0.5022, 'learning_rate': 7.266666666666668e-06, 'epoch': 0.08} \r\n", "{'loss': 0.1132, 'learning_rate': 7.2000000000000005e-06, 'epoch': 0.08} \r\n", "{'loss': 0.3318, 'learning_rate': 7.133333333333334e-06, 'epoch': 0.08} \r\n", "{'loss': 0.0071, 'learning_rate': 7.066666666666667e-06, 'epoch': 0.08} \r\n", "{'loss': 0.6094, 'learning_rate': 7e-06, 'epoch': 0.08} \r\n", "{'loss': 0.0022, 'learning_rate': 6.9333333333333344e-06, 'epoch': 0.08} \r\n", "{'loss': 0.4046, 'learning_rate': 6.866666666666667e-06, 'epoch': 0.08} \r\n", "{'loss': 0.3455, 'learning_rate': 6.800000000000001e-06, 'epoch': 0.08} \r\n", "{'loss': 0.5033, 'learning_rate': 6.733333333333334e-06, 'epoch': 0.08} \r\n", "{'loss': 0.1898, 'learning_rate': 6.666666666666667e-06, 'epoch': 0.08} \r\n", " 67%|███████████████████████████▎ | 200/300 [05:00<02:30, 1.51s/it][INFO|tokenization_utils_base.py:2428] 2023-11-24 13:49:25,887 >> tokenizer config file saved in output/news-20231124-134412-2e-05/checkpoint-200/tokenizer_config.json\r\n", "[INFO|tokenization_utils_base.py:2437] 2023-11-24 13:49:25,887 >> Special tokens file saved in output/news-20231124-134412-2e-05/checkpoint-200/special_tokens_map.json\r\n", "{'loss': 0.2219, 'learning_rate': 6.600000000000001e-06, 'epoch': 0.08} \r\n", "{'loss': 0.0261, 'learning_rate': 6.533333333333334e-06, 'epoch': 0.08} \r\n", "{'loss': 0.0268, 'learning_rate': 6.466666666666667e-06, 'epoch': 0.08} \r\n", "{'loss': 0.2255, 'learning_rate': 6.4000000000000006e-06, 'epoch': 0.08} \r\n", "{'loss': 0.0023, 'learning_rate': 6.333333333333333e-06, 'epoch': 0.08} \r\n", "{'loss': 0.003, 'learning_rate': 6.266666666666668e-06, 'epoch': 0.08} \r\n", "{'loss': 0.0272, 'learning_rate': 6.200000000000001e-06, 'epoch': 0.08} \r\n", "{'loss': 0.0585, 'learning_rate': 6.133333333333334e-06, 'epoch': 0.08} \r\n", "{'loss': 0.0094, 'learning_rate': 6.066666666666667e-06, 'epoch': 0.08} \r\n", "{'loss': 0.163, 'learning_rate': 6e-06, 'epoch': 0.08} \r\n", "{'loss': 0.0036, 'learning_rate': 5.933333333333335e-06, 'epoch': 0.08} \r\n", "{'loss': 0.0781, 'learning_rate': 5.8666666666666675e-06, 'epoch': 0.08} \r\n", "{'loss': 0.0089, 'learning_rate': 5.8e-06, 'epoch': 0.09} \r\n", "{'loss': 0.5827, 'learning_rate': 5.733333333333334e-06, 'epoch': 0.09} \r\n", "{'loss': 0.0053, 'learning_rate': 5.666666666666667e-06, 'epoch': 0.09} \r\n", "{'loss': 0.0179, 'learning_rate': 5.600000000000001e-06, 'epoch': 0.09} \r\n", "{'loss': 1.8615, 'learning_rate': 5.533333333333334e-06, 'epoch': 0.09} \r\n", "{'loss': 0.3423, 'learning_rate': 5.466666666666667e-06, 'epoch': 0.09} \r\n", "{'loss': 0.5502, 'learning_rate': 5.400000000000001e-06, 'epoch': 0.09} \r\n", "{'loss': 0.1772, 'learning_rate': 5.333333333333334e-06, 'epoch': 0.09} \r\n", "{'loss': 0.075, 'learning_rate': 5.2666666666666665e-06, 'epoch': 0.09} \r\n", "{'loss': 0.0059, 'learning_rate': 5.2e-06, 'epoch': 0.09} \r\n", "{'loss': 0.0042, 'learning_rate': 5.133333333333334e-06, 'epoch': 0.09} \r\n", "{'loss': 0.0213, 'learning_rate': 5.0666666666666676e-06, 'epoch': 0.09} \r\n", "{'loss': 0.0174, 'learning_rate': 5e-06, 'epoch': 0.09} \r\n", "{'loss': 1.0351, 'learning_rate': 4.933333333333334e-06, 'epoch': 0.09} \r\n", "{'loss': 0.0026, 'learning_rate': 4.866666666666667e-06, 'epoch': 0.09} \r\n", "{'loss': 0.0026, 'learning_rate': 4.800000000000001e-06, 'epoch': 0.09} \r\n", "{'loss': 0.5251, 'learning_rate': 4.7333333333333335e-06, 'epoch': 0.09} \r\n", "{'loss': 0.2601, 'learning_rate': 4.666666666666667e-06, 'epoch': 0.09} \r\n", "{'loss': 0.006, 'learning_rate': 4.600000000000001e-06, 'epoch': 0.09} \r\n", "{'loss': 0.8758, 'learning_rate': 4.533333333333334e-06, 'epoch': 0.09} \r\n", "{'loss': 0.2939, 'learning_rate': 4.4666666666666665e-06, 'epoch': 0.09} \r\n", "{'loss': 0.0141, 'learning_rate': 4.4e-06, 'epoch': 0.09} \r\n", "{'loss': 0.001, 'learning_rate': 4.333333333333334e-06, 'epoch': 0.09} \r\n", "{'loss': 0.0176, 'learning_rate': 4.266666666666668e-06, 'epoch': 0.09} \r\n", "{'loss': 0.4934, 'learning_rate': 4.2000000000000004e-06, 'epoch': 0.09} \r\n", "{'loss': 0.3413, 'learning_rate': 4.133333333333333e-06, 'epoch': 0.1} \r\n", "{'loss': 0.0059, 'learning_rate': 4.066666666666667e-06, 'epoch': 0.1} \r\n", "{'loss': 0.0379, 'learning_rate': 4.000000000000001e-06, 'epoch': 0.1} \r\n", "{'loss': 0.7871, 'learning_rate': 3.9333333333333335e-06, 'epoch': 0.1} \r\n", "{'loss': 0.0035, 'learning_rate': 3.866666666666667e-06, 'epoch': 0.1} \r\n", "{'loss': 0.3481, 'learning_rate': 3.8000000000000005e-06, 'epoch': 0.1} \r\n", "{'loss': 0.1827, 'learning_rate': 3.7333333333333337e-06, 'epoch': 0.1} \r\n", "{'loss': 0.4669, 'learning_rate': 3.6666666666666666e-06, 'epoch': 0.1} \r\n", "{'loss': 0.8078, 'learning_rate': 3.6000000000000003e-06, 'epoch': 0.1} \r\n", "{'loss': 1.1314, 'learning_rate': 3.5333333333333335e-06, 'epoch': 0.1} \r\n", "{'loss': 0.0124, 'learning_rate': 3.4666666666666672e-06, 'epoch': 0.1} \r\n", "{'loss': 0.0116, 'learning_rate': 3.4000000000000005e-06, 'epoch': 0.1} \r\n", "{'loss': 0.2296, 'learning_rate': 3.3333333333333333e-06, 'epoch': 0.1} \r\n", "{'loss': 0.0037, 'learning_rate': 3.266666666666667e-06, 'epoch': 0.1} \r\n", "{'loss': 0.0882, 'learning_rate': 3.2000000000000003e-06, 'epoch': 0.1} \r\n", "{'loss': 0.0073, 'learning_rate': 3.133333333333334e-06, 'epoch': 0.1} \r\n", "{'loss': 0.013, 'learning_rate': 3.066666666666667e-06, 'epoch': 0.1} \r\n", "{'loss': 0.0027, 'learning_rate': 3e-06, 'epoch': 0.1} \r\n", "{'loss': 0.7103, 'learning_rate': 2.9333333333333338e-06, 'epoch': 0.1} \r\n", "{'loss': 0.0022, 'learning_rate': 2.866666666666667e-06, 'epoch': 0.1} \r\n", "{'loss': 0.4343, 'learning_rate': 2.8000000000000003e-06, 'epoch': 0.1} \r\n", "{'loss': 1.1652, 'learning_rate': 2.7333333333333336e-06, 'epoch': 0.1} \r\n", "{'loss': 0.377, 'learning_rate': 2.666666666666667e-06, 'epoch': 0.1} \r\n", "{'loss': 0.0025, 'learning_rate': 2.6e-06, 'epoch': 0.1} \r\n", "{'loss': 0.072, 'learning_rate': 2.5333333333333338e-06, 'epoch': 0.1} \r\n", "{'loss': 0.0035, 'learning_rate': 2.466666666666667e-06, 'epoch': 0.11} \r\n", "{'loss': 0.0176, 'learning_rate': 2.4000000000000003e-06, 'epoch': 0.11} \r\n", "{'loss': 0.0404, 'learning_rate': 2.3333333333333336e-06, 'epoch': 0.11} \r\n", "{'loss': 0.0243, 'learning_rate': 2.266666666666667e-06, 'epoch': 0.11} \r\n", "{'loss': 0.0197, 'learning_rate': 2.2e-06, 'epoch': 0.11} \r\n", "{'loss': 0.0558, 'learning_rate': 2.133333333333334e-06, 'epoch': 0.11} \r\n", "{'loss': 0.0396, 'learning_rate': 2.0666666666666666e-06, 'epoch': 0.11} \r\n", "{'loss': 0.0042, 'learning_rate': 2.0000000000000003e-06, 'epoch': 0.11} \r\n", "{'loss': 0.2281, 'learning_rate': 1.9333333333333336e-06, 'epoch': 0.11} \r\n", "{'loss': 0.0594, 'learning_rate': 1.8666666666666669e-06, 'epoch': 0.11} \r\n", "{'loss': 0.0016, 'learning_rate': 1.8000000000000001e-06, 'epoch': 0.11} \r\n", "{'loss': 0.1489, 'learning_rate': 1.7333333333333336e-06, 'epoch': 0.11} \r\n", "{'loss': 0.0053, 'learning_rate': 1.6666666666666667e-06, 'epoch': 0.11} \r\n", "{'loss': 0.0131, 'learning_rate': 1.6000000000000001e-06, 'epoch': 0.11} \r\n", "{'loss': 0.0517, 'learning_rate': 1.5333333333333334e-06, 'epoch': 0.11} \r\n", "{'loss': 2.0054, 'learning_rate': 1.4666666666666669e-06, 'epoch': 0.11} \r\n", "{'loss': 0.863, 'learning_rate': 1.4000000000000001e-06, 'epoch': 0.11} \r\n", "{'loss': 0.0038, 'learning_rate': 1.3333333333333334e-06, 'epoch': 0.11} \r\n", "{'loss': 0.0003, 'learning_rate': 1.2666666666666669e-06, 'epoch': 0.11} \r\n", "{'loss': 0.0553, 'learning_rate': 1.2000000000000002e-06, 'epoch': 0.11} \r\n", "{'loss': 0.0076, 'learning_rate': 1.1333333333333334e-06, 'epoch': 0.11} \r\n", "{'loss': 0.0481, 'learning_rate': 1.066666666666667e-06, 'epoch': 0.11} \r\n", "{'loss': 0.5921, 'learning_rate': 1.0000000000000002e-06, 'epoch': 0.11} \r\n", "{'loss': 0.3559, 'learning_rate': 9.333333333333334e-07, 'epoch': 0.11} \r\n", "{'loss': 0.0014, 'learning_rate': 8.666666666666668e-07, 'epoch': 0.11} \r\n", "{'loss': 0.4355, 'learning_rate': 8.000000000000001e-07, 'epoch': 0.12} \r\n", "{'loss': 0.0764, 'learning_rate': 7.333333333333334e-07, 'epoch': 0.12} \r\n", "{'loss': 0.0035, 'learning_rate': 6.666666666666667e-07, 'epoch': 0.12} \r\n", "{'loss': 0.0042, 'learning_rate': 6.000000000000001e-07, 'epoch': 0.12} \r\n", "{'loss': 0.0086, 'learning_rate': 5.333333333333335e-07, 'epoch': 0.12} \r\n", "{'loss': 0.2114, 'learning_rate': 4.666666666666667e-07, 'epoch': 0.12} \r\n", "{'loss': 0.223, 'learning_rate': 4.0000000000000003e-07, 'epoch': 0.12} \r\n", "{'loss': 0.0044, 'learning_rate': 3.3333333333333335e-07, 'epoch': 0.12} \r\n", "{'loss': 0.6593, 'learning_rate': 2.666666666666667e-07, 'epoch': 0.12} \r\n", "{'loss': 0.1374, 'learning_rate': 2.0000000000000002e-07, 'epoch': 0.12} \r\n", "{'loss': 0.1037, 'learning_rate': 1.3333333333333336e-07, 'epoch': 0.12} \r\n", "{'loss': 0.0181, 'learning_rate': 6.666666666666668e-08, 'epoch': 0.12} \r\n", "{'loss': 0.0783, 'learning_rate': 0.0, 'epoch': 0.12} \r\n", "100%|█████████████████████████████████████████| 300/300 [07:31<00:00, 1.51s/it][INFO|tokenization_utils_base.py:2428] 2023-11-24 13:51:56,849 >> tokenizer config file saved in output/news-20231124-134412-2e-05/checkpoint-300/tokenizer_config.json\r\n", "[INFO|tokenization_utils_base.py:2437] 2023-11-24 13:51:56,849 >> Special tokens file saved in output/news-20231124-134412-2e-05/checkpoint-300/special_tokens_map.json\r\n", "[INFO|trainer.py:1955] 2023-11-24 13:51:56,875 >> \r\n", "\r\n", "Training completed. Do not forget to share your model on huggingface.co/models =)\r\n", "\r\n", "\r\n", "{'train_runtime': 451.0866, 'train_samples_per_second': 1.33, 'train_steps_per_second': 0.665, 'train_loss': 0.2702767427762349, 'epoch': 0.12}\r\n", "100%|█████████████████████████████████████████| 300/300 [07:31<00:00, 1.50s/it]\r\n", "[INFO|tokenization_utils_base.py:2428] 2023-11-24 13:51:56,887 >> tokenizer config file saved in output/news-20231124-134412-2e-05/tokenizer_config.json\r\n", "[INFO|tokenization_utils_base.py:2437] 2023-11-24 13:51:56,887 >> Special tokens file saved in output/news-20231124-134412-2e-05/special_tokens_map.json\r\n" ] } ], "source": [ "!which python\n", "import os\n", "from datetime import datetime\n", "import random\n", "\n", "# 定义变量\n", "lr = 2e-5\n", "num_gpus = 1\n", "lora_rank = 8\n", "lora_alpha = 32\n", "lora_dropout = 0.1\n", "max_source_len = 512\n", "max_target_len = 128\n", "dev_batch_size = 1\n", "grad_accumularion_steps = 2\n", "max_step = 300\n", "save_interval = 100\n", "max_seq_len = 512\n", "logging_steps=1\n", "\n", "run_name = \"news\"\n", "dataset_path = \"data/toutiao_cat_data_example.jsonl\"\n", "datestr = datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n", "output_dir = f\"output/{run_name}-{datestr}-{lr}\"\n", "master_port = random.randint(10000, 65535)\n", "\n", "os.makedirs(output_dir, exist_ok=True)\n", "# 构建命令\n", "command = f\"\"\"\n", "/home/zr/Code/ChatGLM3/venv/bin/torchrun --standalone --nnodes=1 --nproc_per_node={num_gpus} ../finetune_basemodel_demo/finetune.py \\\n", " --train_format input-output \\\n", " --train_file {dataset_path} \\\n", " --lora_rank {lora_rank} \\\n", " --lora_alpha {lora_alpha} \\\n", " --lora_dropout {lora_dropout} \\\n", " --max_seq_length {max_seq_len} \\\n", " --preprocessing_num_workers 1 \\\n", " --model_name_or_path {model_path} \\\n", " --output_dir {output_dir} \\\n", " --per_device_train_batch_size 1 \\\n", " --gradient_accumulation_steps 2 \\\n", " --max_steps {max_step} \\\n", " --logging_steps {logging_steps} \\\n", " --save_steps {save_interval} \\\n", " --learning_rate {lr}\n", "\"\"\"\n", "\n", "# 在 Notebook 中执行命令\n", "!{command}" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-11-24T13:44:12.468751Z", "end_time": "2023-11-24T13:51:59.534815Z" } } }, { "cell_type": "markdown", "source": [ "## 使用微调的模型进行推理预测\n", "现在，我们已经完成了模型的微调，接下来，我们将使用微调后的模型进行推理。我们使用与微调时相同的提示词，并使用一些没有出现的模型效果来复现推理结果。" ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 7, "outputs": [ { "data": { "text/plain": "Loading checkpoint shards: 0%| | 0/7 [00:00