用2卡或4卡k100测finetune.sh报错，完整记录log (#5) · Issues · ModelZoo / llama3_pytorch

用2卡或4卡k100测finetune.sh报错，完整记录log

[root@6f869190f1fa llama3_pytorch]# bash finetune.sh Export params ... starting finetune llama3 ... 06/21 18:28:47 - mmengine - WARNING - Use random port: 28966 [2024-06-21 18:28:56,677] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-06-21 18:29:02,226] torch.distributed.run: [WARNING] [2024-06-21 18:29:02,226] torch.distributed.run: [WARNING] * [2024-06-21 18:29:02,226] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. [2024-06-21 18:29:02,226] torch.distributed.run: [WARNING] * [2024-06-21 18:29:24,082] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-06-21 18:29:24,139] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-06-21 18:29:24,191] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-06-21 18:29:24,217] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) /usr/local/lib/python3.10/site-packages/mmengine/utils/dl_utils/setup_env.py:56: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. warnings.warn( [2024-06-21 18:29:43,308] [INFO] [comm.py:637:init_distributed] cdb=None [2024-06-21 18:29:43,308] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl WARNING: Logging before InitGoogleLogging() is written to STDERR I0621 18:29:43.309012 3964 ProcessGroupNCCL.cpp:686] [Rank 0] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=323813088 I0621 18:29:43.309496 3964 ProcessGroupNCCL.cpp:686] [Rank 0] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=323815872 I0621 18:29:43.309988 3964 ProcessGroupNCCL.cpp:686] [Rank 0] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=323818032 /usr/local/lib/python3.10/site-packages/mmengine/utils/dl_utils/setup_env.py:56: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. warnings.warn( [2024-06-21 18:29:43,368] [INFO] [comm.py:637:init_distributed] cdb=None WARNING: Logging before InitGoogleLogging() is written to STDERR I0621 18:29:43.369658 3967 ProcessGroupNCCL.cpp:686] [Rank 3] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=319804288 I0621 18:29:43.370335 3967 ProcessGroupNCCL.cpp:686] [Rank 0] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=319807072 I0621 18:29:43.370604 3967 ProcessGroupNCCL.cpp:686] [Rank 3] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=319809232 /usr/local/lib/python3.10/site-packages/mmengine/utils/dl_utils/setup_env.py:56: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. warnings.warn( [2024-06-21 18:29:43,431] [INFO] [comm.py:637:init_distributed] cdb=None WARNING: Logging before InitGoogleLogging() is written to STDERR I0621 18:29:43.432689 3966 ProcessGroupNCCL.cpp:686] [Rank 2] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=326017968 I0621 18:29:43.433328 3966 ProcessGroupNCCL.cpp:686] [Rank 0] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=326020752 I0621 18:29:43.433722 3966 ProcessGroupNCCL.cpp:686] [Rank 2] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=326022912 /usr/local/lib/python3.10/site-packages/mmengine/utils/dl_utils/setup_env.py:56: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. warnings.warn( [2024-06-21 18:29:43,435] [INFO] [comm.py:637:init_distributed] cdb=None WARNING: Logging before InitGoogleLogging() is written to STDERR I0621 18:29:43.436120 3965 ProcessGroupNCCL.cpp:686] [Rank 1] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=326837312 I0621 18:29:43.436677 3965 ProcessGroupNCCL.cpp:686] [Rank 0] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=326840096 I0621 18:29:43.437090 3965 ProcessGroupNCCL.cpp:686] [Rank 1] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=326842256 I0621 18:29:44.473145 3964 ProcessGroupNCCL.cpp:1340] NCCL_DEBUG: N/A /bin/sh: /opt/dtk/bin/nvcc: No such file or directory /bin/sh: /opt/dtk/bin/nvcc: No such file or directory /bin/sh: /opt/dtk/bin/nvcc: No such file or directory /bin/sh: /opt/dtk/bin/nvcc: No such file or directory 06/21 18:30:01 - mmengine - INFO -

System environment: sys.platform: linux Python: 3.10.12 (main, Mar 21 2024, 17:56:31) [GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] CUDA available: True MUSA available: False numpy_random_seed: 1998747639 GPU 0,1,2,3: Device 62b7 CUDA_HOME: /opt/dtk NVCC: Not Available GCC: gcc (GCC) 7.3.1 20180303 (Red Hat 7.3.1-5) PyTorch: 2.1.0 PyTorch compiling details: PyTorch built with:

GCC 7.3
C++ Version: 201703
Intel(R) Math Kernel Library Version 2020.0.4 Product Build 20200917 for Intel(R) 64 architecture applications
OpenMP 201511 (a.k.a. OpenMP 4.5)
LAPACK is enabled (usually provided by MKL)
NNPACK is enabled
CPU capability usage: AVX2
HIP Runtime 5.7.24164
MIOpen 2.15.4
Magma 2.7.2
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-invalid-partial-specialization -Wno-unused-private-field -Wno-aligned-allocation-unavailable -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.1.0, USE_CUDA=0, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=1, USE_GLOG=1, USE_MKL=ON, USE_MKLDNN=0, USE_MPI=1, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=1, USE_ROCM=ON,

TorchVision: 0.16.0 OpenCV: 4.9.0 MMEngine: 0.10.3

Runtime environment: launcher: pytorch randomness: {'seed': None, 'deterministic': False} cudnn_benchmark: False mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0} dist_cfg: {'backend': 'nccl'} seed: None deterministic: False Distributed launcher: pytorch Distributed training: True GPU number: 4

06/21 18:30:01 - mmengine - INFO - Config: SYSTEM = 'xtuner.utils.SYSTEM_TEMPLATE.alpaca' accumulative_counts = 16 alpaca_en = dict( dataset=dict( data_files=dict(train='datasets/alpaca_data.json'), path='json', type='datasets.load_dataset'), dataset_map_fn='xtuner.dataset.map_fns.alpaca_map_fn', max_length=2048, pack_to_max_length=True, remove_unused_columns=True, shuffle_before_pack=True, template_map_fn=dict( template='xtuner.utils.PROMPT_TEMPLATE.llama3_chat', type='xtuner.dataset.map_fns.template_map_fn_factory'), tokenizer=dict( padding_side='right', pretrained_model_name_or_path= '../../../model/Meta-Llama-3-8B-Instruct-20240421/model', trust_remote_code=True, type='transformers.AutoTokenizer.from_pretrained'), type='xtuner.dataset.process_hf_dataset', use_varlen_attn=False) batch_size = 1 betas = ( 0.9, 0.999, ) custom_hooks = [ dict( tokenizer=dict( padding_side='right', pretrained_model_name_or_path= '../../../model/Meta-Llama-3-8B-Instruct-20240421/model', trust_remote_code=True, type='transformers.AutoTokenizer.from_pretrained'), type='xtuner.engine.hooks.DatasetInfoHook'), dict( evaluation_inputs=[ '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai', ], every_n_iters=500, prompt_template='xtuner.utils.PROMPT_TEMPLATE.llama3_chat', system='xtuner.utils.SYSTEM_TEMPLATE.alpaca', tokenizer=dict( padding_side='right', pretrained_model_name_or_path= '../../../model/Meta-Llama-3-8B-Instruct-20240421/model', trust_remote_code=True, type='transformers.AutoTokenizer.from_pretrained'), type='xtuner.engine.hooks.EvaluateChatHook'), ] data_path = 'datasets/alpaca_data.json' dataloader_num_workers = 0 default_hooks = dict( checkpoint=dict( by_epoch=False, interval=500, max_keep_ckpts=2, type='mmengine.hooks.CheckpointHook'), logger=dict( interval=10, log_metric_by_epoch=False, type='mmengine.hooks.LoggerHook'), param_scheduler=dict(type='mmengine.hooks.ParamSchedulerHook'), sampler_seed=dict(type='mmengine.hooks.DistSamplerSeedHook'), timer=dict(type='mmengine.hooks.IterTimerHook')) env_cfg = dict( cudnn_benchmark=False, dist_cfg=dict(backend='nccl'), mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0)) evaluation_freq = 500 evaluation_inputs = [ '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai', ] launcher = 'pytorch' load_from = None log_level = 'INFO' log_processor = dict(by_epoch=False) lr = 0.0001 max_epochs = 3 max_length = 2048 max_norm = 1 model = dict( llm=dict( pretrained_model_name_or_path= '../../../model/Meta-Llama-3-8B-Instruct-20240421/model', torch_dtype='torch.float16', trust_remote_code=True, type='transformers.AutoModelForCausalLM.from_pretrained'), lora=dict( bias='none', lora_alpha=16, lora_dropout=0.1, r=64, task_type='CAUSAL_LM', type='peft.LoraConfig'), type='xtuner.model.SupervisedFinetune', use_varlen_attn=False) optim_type = 'torch.optim.AdamW' optim_wrapper = dict( optimizer=dict( betas=( 0.9, 0.999, ), lr=0.0001, type='torch.optim.AdamW', weight_decay=0), type='DeepSpeedOptimWrapper') pack_to_max_length = True param_scheduler = [ dict( begin=0, by_epoch=True, convert_to_iter_based=True, end=0.09, start_factor=1e-05, type='mmengine.optim.LinearLR'), dict( begin=0.09, by_epoch=True, convert_to_iter_based=True, end=3, eta_min=0.0, type='mmengine.optim.CosineAnnealingLR'), ] pretrained_model_name_or_path = '../../../model/Meta-Llama-3-8B-Instruct-20240421/model' prompt_template = 'xtuner.utils.PROMPT_TEMPLATE.llama3_chat' randomness = dict(deterministic=False, seed=None) resume = False runner_type = 'FlexibleRunner' sampler = 'mmengine.dataset.DefaultSampler' save_steps = 500 save_total_limit = 2 sequence_parallel_size = 1 strategy = dict( config=dict( bf16=dict(enabled=True), fp16=dict(enabled=False, initial_scale_power=16), gradient_accumulation_steps='auto', gradient_clipping='auto', train_micro_batch_size_per_gpu='auto', zero_allow_untested_optimizer=True, zero_force_ds_cpu_optimizer=False, zero_optimization=dict(overlap_comm=True, stage=2)), exclude_frozen_parameters=True, gradient_accumulation_steps=16, gradient_clipping=1, sequence_parallel_size=1, train_micro_batch_size_per_gpu=1, type='xtuner.engine.DeepSpeedStrategy') tokenizer = dict( padding_side='right', pretrained_model_name_or_path= '../../../model/Meta-Llama-3-8B-Instruct-20240421/model', trust_remote_code=True, type='transformers.AutoTokenizer.from_pretrained') train_cfg = dict(max_epochs=3, type='xtuner.engine.runner.TrainLoop') train_dataloader = dict( batch_size=1, collate_fn=dict( type='xtuner.dataset.collate_fns.default_collate_fn', use_varlen_attn=False), dataset=dict( dataset=dict( data_files=dict(train='datasets/alpaca_data.json'), path='json', type='datasets.load_dataset'), dataset_map_fn='xtuner.dataset.map_fns.alpaca_map_fn', max_length=2048, pack_to_max_length=True, remove_unused_columns=True, shuffle_before_pack=True, template_map_fn=dict( template='xtuner.utils.PROMPT_TEMPLATE.llama3_chat', type='xtuner.dataset.map_fns.template_map_fn_factory'), tokenizer=dict( padding_side='right', pretrained_model_name_or_path= '../../../model/Meta-Llama-3-8B-Instruct-20240421/model', trust_remote_code=True, type='transformers.AutoTokenizer.from_pretrained'), type='xtuner.dataset.process_hf_dataset', use_varlen_attn=False), num_workers=0, sampler=dict(shuffle=True, type='mmengine.dataset.DefaultSampler')) use_varlen_attn = False visualizer = None warmup_ratio = 0.03 weight_decay = 0 work_dir = './work_dirs/llama3_8b_instruct_qlora_alpaca_e3_M'

06/21 18:30:01 - mmengine - WARNING - Failed to search registry with scope "mmengine" in the "builder" registry tree. As a workaround, the current "builder" registry in "xtuner" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmengine" is a correct scope, or whether the registry is initialized. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. 06/21 18:30:02 - mmengine - INFO - Hooks will be executed in the following order: before_run: (VERY_HIGH ) RuntimeInfoHook (BELOW_NORMAL) LoggerHook

before_train: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (NORMAL ) DatasetInfoHook (LOW ) EvaluateChatHook (VERY_LOW ) CheckpointHook

before_train_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (NORMAL ) DistSamplerSeedHook

before_train_iter: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook

after_train_iter: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook (LOW ) ParamSchedulerHook (LOW ) EvaluateChatHook (VERY_LOW ) CheckpointHook

after_train_epoch: (NORMAL ) IterTimerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook

before_val: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) DatasetInfoHook

before_val_epoch: (NORMAL ) IterTimerHook

before_val_iter: (NORMAL ) IterTimerHook

after_val_iter: (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook

after_val_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook

after_val: (VERY_HIGH ) RuntimeInfoHook (LOW ) EvaluateChatHook

after_train: (VERY_HIGH ) RuntimeInfoHook (LOW ) EvaluateChatHook (VERY_LOW ) CheckpointHook

before_test: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) DatasetInfoHook

before_test_epoch: (NORMAL ) IterTimerHook

before_test_iter: (NORMAL ) IterTimerHook

after_test_iter: (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook

after_test_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook

after_test: (VERY_HIGH ) RuntimeInfoHook

after_run: (BELOW_NORMAL) LoggerHook

06/21 18:30:02 - mmengine - INFO - xtuner_dataset_timeout = 0:30:00 Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Flattening the indices (num_proc=32): 100%|████████████████████████████████████████████████████████████| 51979/51979 [00:02<00:00, 20990.04 examples/s] Map (num_proc=32): 100%|███████████████████████████████████████████████████████████████████████████████| 51979/51979 [00:02<00:00, 20246.62 examples/s] Map (num_proc=32): 100%|███████████████████████████████████████████████████████████████████████████████████| 2114/2114 [00:02<00:00, 877.28 examples/s] 06/21 18:30:38 - mmengine - WARNING - Dataset Dataset has no metainfo. dataset_meta in visualizer will be None. Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:35<00:00, 8.95s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:35<00:00, 8.95s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:35<00:00, 8.96s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:35<00:00, 8.97s/it] [2024-06-21 18:32:06,714] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.12.3, git-hash=a724046, git-branch=HEAD I0621 18:32:23.488137 3964 ProcessGroupNCCL.cpp:686] [Rank 0] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=363596784 I0621 18:32:23.860575 3965 ProcessGroupNCCL.cpp:686] [Rank 1] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=405453648 I0621 18:32:27.112511 3966 ProcessGroupNCCL.cpp:686] [Rank 2] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=404269488 I0621 18:32:27.298046 3967 ProcessGroupNCCL.cpp:686] [Rank 3] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=358885184 I0621 18:32:27.513510 3964 ProcessGroupNCCL.cpp:1340] NCCL_DEBUG: N/A [2024-06-21 18:32:27,652] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False [2024-06-21 18:32:27,662] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer [2024-06-21 18:32:27,662] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer [2024-06-21 18:32:27,801] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = AdamW [2024-06-21 18:32:27,801] [INFO] [utils.py:56:is_zero_supported_optimizer] Checking ZeRO support for optimizer=AdamW type=<class 'torch.optim.adamw.AdamW'> [2024-06-21 18:32:27,802] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 2 optimizer [2024-06-21 18:32:27,802] [INFO] [stage_1_and_2.py:147:init] Reduce bucket size 500,000,000 [2024-06-21 18:32:27,802] [INFO] [stage_1_and_2.py:148:init] Allgather bucket size 500,000,000 [2024-06-21 18:32:27,802] [INFO] [stage_1_and_2.py:149:init] CPU Offload: False [2024-06-21 18:32:27,802] [INFO] [stage_1_and_2.py:150:init] Round robin gradient partitioning: False [2024-06-21 18:32:29,414] [INFO] [utils.py:802:see_memory_usage] Before initializing optimizer states [2024-06-21 18:32:29,416] [INFO] [utils.py:803:see_memory_usage] MA 15.43 GB Max_MA 15.51 GB CA 15.7 GB Max_CA 16 GB [2024-06-21 18:32:29,416] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 46.37 GB, percent = 4.7% [2024-06-21 18:32:29,627] [INFO] [utils.py:802:see_memory_usage] After initializing optimizer states [2024-06-21 18:32:29,628] [INFO] [utils.py:803:see_memory_usage] MA 15.74 GB Max_MA 16.05 GB CA 16.32 GB Max_CA 16 GB [2024-06-21 18:32:29,628] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 46.37 GB, percent = 4.7% [2024-06-21 18:32:29,629] [INFO] [stage_1_and_2.py:514:init] optimizer state initialized [2024-06-21 18:32:29,783] [INFO] [utils.py:802:see_memory_usage] After initializing ZeRO optimizer [2024-06-21 18:32:29,785] [INFO] [utils.py:803:see_memory_usage] MA 15.74 GB Max_MA 15.74 GB CA 16.32 GB Max_CA 16 GB [2024-06-21 18:32:29,785] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 46.37 GB, percent = 4.7% [2024-06-21 18:32:29,792] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = AdamW [2024-06-21 18:32:29,793] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using client LR scheduler [2024-06-21 18:32:29,793] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = None [2024-06-21 18:32:29,793] [INFO] [logging.py:96:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0001], mom=[(0.9, 0.999)] [2024-06-21 18:32:29,801] [INFO] [config.py:974:print] DeepSpeedEngine configuration: [2024-06-21 18:32:29,802] [INFO] [config.py:978:print] activation_checkpointing_config { "partition_activations": false, "contiguous_memory_optimization": false, "cpu_checkpointing": false, "number_checkpoints": null, "synchronize_checkpoint_boundary": false, "profile": false } [2024-06-21 18:32:29,802] [INFO] [config.py:978:print] aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True} [2024-06-21 18:32:29,802] [INFO] [config.py:978:print] amp_enabled .................. False [2024-06-21 18:32:29,802] [INFO] [config.py:978:print] amp_params ................... False [2024-06-21 18:32:29,802] [INFO] [config.py:978:print] autotuning_config ............ { "enabled": false, "start_step": null, "end_step": null, "metric_path": null, "arg_mappings": null, "metric": "throughput", "model_info": null, "results_dir": "autotuning_results", "exps_dir": "autotuning_exps", "overwrite": true, "fast": true, "start_profile_step": 3, "end_profile_step": 5, "tuner_type": "gridsearch", "tuner_early_stopping": 5, "tuner_num_trials": 50, "model_info_path": null, "mp_size": 1, "max_train_batch_size": null, "min_train_batch_size": 1, "max_train_micro_batch_size_per_gpu": 1.024000e+03, "min_train_micro_batch_size_per_gpu": 1, "num_tuning_micro_batch_sizes": 3 } [2024-06-21 18:32:29,802] [INFO] [config.py:978:print] bfloat16_enabled ............. True [2024-06-21 18:32:29,802] [INFO] [config.py:978:print] checkpoint_parallel_write_pipeline False [2024-06-21 18:32:29,803] [INFO] [config.py:978:print] checkpoint_tag_validation_enabled True [2024-06-21 18:32:29,803] [INFO] [config.py:978:print] checkpoint_tag_validation_fail False [2024-06-21 18:32:29,803] [INFO] [config.py:978:print] comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7f8838398580> [2024-06-21 18:32:29,803] [INFO] [config.py:978:print] communication_data_type ...... None [2024-06-21 18:32:29,803] [INFO] [config.py:978:print] compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}} [2024-06-21 18:32:29,803] [INFO] [config.py:978:print] curriculum_enabled_legacy .... False [2024-06-21 18:32:29,803] [INFO] [config.py:978:print] curriculum_params_legacy ..... False [2024-06-21 18:32:29,803] [INFO] [config.py:978:print] data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}} [2024-06-21 18:32:29,803] [INFO] [config.py:978:print] data_efficiency_enabled ...... False [2024-06-21 18:32:29,803] [INFO] [config.py:978:print] dataloader_drop_last ......... False [2024-06-21 18:32:29,803] [INFO] [config.py:978:print] disable_allgather ............ False [2024-06-21 18:32:29,803] [INFO] [config.py:978:print] dump_state ................... False [2024-06-21 18:32:29,803] [INFO] [config.py:978:print] dynamic_loss_scale_args ...... None [2024-06-21 18:32:29,803] [INFO] [config.py:978:print] eigenvalue_enabled ........... False [2024-06-21 18:32:29,803] [INFO] [config.py:978:print] eigenvalue_gas_boundary_resolution 1 [2024-06-21 18:32:29,804] [INFO] [config.py:978:print] eigenvalue_layer_name ........ bert.encoder.layer [2024-06-21 18:32:29,804] [INFO] [config.py:978:print] eigenvalue_layer_num ......... 0 [2024-06-21 18:32:29,804] [INFO] [config.py:978:print] eigenvalue_max_iter .......... 100 [2024-06-21 18:32:29,804] [INFO] [config.py:978:print] eigenvalue_stability ......... 1e-06 [2024-06-21 18:32:29,804] [INFO] [config.py:978:print] eigenvalue_tol ............... 0.01 [2024-06-21 18:32:29,804] [INFO] [config.py:978:print] eigenvalue_verbose ........... False [2024-06-21 18:32:29,804] [INFO] [config.py:978:print] elasticity_enabled ........... False [2024-06-21 18:32:29,804] [INFO] [config.py:978:print] flops_profiler_config ........ { "enabled": false, "recompute_fwd_factor": 0.0, "profile_step": 1, "module_depth": -1, "top_modules": 1, "detailed": true, "output_file": null } [2024-06-21 18:32:29,804] [INFO] [config.py:978:print] fp16_auto_cast ............... None [2024-06-21 18:32:29,804] [INFO] [config.py:978:print] fp16_enabled ................. False [2024-06-21 18:32:29,804] [INFO] [config.py:978:print] fp16_master_weights_and_gradients False [2024-06-21 18:32:29,804] [INFO] [config.py:978:print] global_rank .................. 0 [2024-06-21 18:32:29,805] [INFO] [config.py:978:print] grad_accum_dtype ............. None [2024-06-21 18:32:29,805] [INFO] [config.py:978:print] gradient_accumulation_steps .. 16 [2024-06-21 18:32:29,805] [INFO] [config.py:978:print] gradient_clipping ............ 1 [2024-06-21 18:32:29,805] [INFO] [config.py:978:print] gradient_predivide_factor .... 1.0 [2024-06-21 18:32:29,805] [INFO] [config.py:978:print] hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8 [2024-06-21 18:32:29,805] [INFO] [config.py:978:print] initial_dynamic_scale ........ 1 [2024-06-21 18:32:29,805] [INFO] [config.py:978:print] load_universal_checkpoint .... False [2024-06-21 18:32:29,805] [INFO] [config.py:978:print] loss_scale ................... 1.0 [2024-06-21 18:32:29,805] [INFO] [config.py:978:print] memory_breakdown ............. False [2024-06-21 18:32:29,805] [INFO] [config.py:978:print] mics_hierarchial_params_gather False [2024-06-21 18:32:29,805] [INFO] [config.py:978:print] mics_shard_size .............. -1 [2024-06-21 18:32:29,805] [INFO] [config.py:978:print] monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False [2024-06-21 18:32:29,806] [INFO] [config.py:978:print] nebula_config ................ { "enabled": false, "persistent_storage_path": null, "persistent_time_interval": 100, "num_of_version_in_retention": 2, "enable_nebula_load": true, "load_path": null } [2024-06-21 18:32:29,806] [INFO] [config.py:978:print] optimizer_legacy_fusion ...... False [2024-06-21 18:32:29,806] [INFO] [config.py:978:print] optimizer_name ............... None [2024-06-21 18:32:29,806] [INFO] [config.py:978:print] optimizer_params ............. None [2024-06-21 18:32:29,806] [INFO] [config.py:978:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True} [2024-06-21 18:32:29,806] [INFO] [config.py:978:print] pld_enabled .................. False [2024-06-21 18:32:29,806] [INFO] [config.py:978:print] pld_params ................... False [2024-06-21 18:32:29,806] [INFO] [config.py:978:print] prescale_gradients ........... False [2024-06-21 18:32:29,806] [INFO] [config.py:978:print] scheduler_name ............... None [2024-06-21 18:32:29,806] [INFO] [config.py:978:print] scheduler_params ............. None [2024-06-21 18:32:29,806] [INFO] [config.py:978:print] seq_parallel_communication_data_type torch.float32 [2024-06-21 18:32:29,806] [INFO] [config.py:978:print] sparse_attention ............. None [2024-06-21 18:32:29,806] [INFO] [config.py:978:print] sparse_gradients_enabled ..... False [2024-06-21 18:32:29,806] [INFO] [config.py:978:print] steps_per_print .............. 10000000000000 [2024-06-21 18:32:29,807] [INFO] [config.py:978:print] train_batch_size ............. 64 [2024-06-21 18:32:29,807] [INFO] [config.py:978:print] train_micro_batch_size_per_gpu 1 [2024-06-21 18:32:29,807] [INFO] [config.py:978:print] use_node_local_storage ....... False [2024-06-21 18:32:29,807] [INFO] [config.py:978:print] wall_clock_breakdown ......... False [2024-06-21 18:32:29,807] [INFO] [config.py:978:print] weight_quantization_config ... None [2024-06-21 18:32:29,807] [INFO] [config.py:978:print] world_size ................... 4 [2024-06-21 18:32:29,807] [INFO] [config.py:978:print] zero_allow_untested_optimizer True [2024-06-21 18:32:29,807] [INFO] [config.py:978:print] zero_config .................. stage=2 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500,000,000 allgather_partitions=True allgather_bucket_size=500,000,000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True [2024-06-21 18:32:29,807] [INFO] [config.py:978:print] zero_enabled ................. True [2024-06-21 18:32:29,807] [INFO] [config.py:978:print] zero_force_ds_cpu_optimizer .. False [2024-06-21 18:32:29,807] [INFO] [config.py:978:print] zero_optimization_stage ...... 2 [2024-06-21 18:32:29,808] [INFO] [config.py:964:print_user_config] json = { "gradient_accumulation_steps": 16, "train_micro_batch_size_per_gpu": 1, "gradient_clipping": 1, "zero_allow_untested_optimizer": true, "zero_force_ds_cpu_optimizer": false, "zero_optimization": { "stage": 2, "overlap_comm": true }, "fp16": { "enabled": false, "initial_scale_power": 16 }, "bf16": { "enabled": true }, "steps_per_print": 1.000000e+13 } 06/21 18:32:38 - mmengine - INFO - Num train samples 2114 06/21 18:32:38 - mmengine - INFO - train example: 06/21 18:32:38 - mmengine - INFO - <|begin_of_text|><|start_header_id|>user<|end_header_id|>

Algorithm to identify the most important keywords in a sentence:

Tokenize the input sentence into separate words.
Calculate the frequency of each word in the sentence.
Compare the frequencies of each word to determine the most important keywords.
Output the most important keywords.

The Industrial Revolution was a period of major social and economic change that began in the late 18th century, marked by advances in technology, advances in manufacturing processes, and the development of new methods of producing goods. These advances led to a shift from agrarian economies to industrial ones, increased economic productivity, the increased use of machines, the factory system, improved transportation networks, increased urbanization, and population growth. The result of these changes was a massive increase in global wealth and, consequently, improved living standards.<|eot_id|><|begin_of_text|><|start_header_id|>user<|end_header_id|>

The purpose of multi-label classification is to predict multiple labels associated with a given instance of data. This is useful in applications such as text categorization, object recognition, and medical diagnosis where multiple labels can be assigned to a given data point. Multi-label classification algorithms are used to accurately and efficiently label instances of data with multiple labels that may have ambiguous boundaries such as in text categorization.<|eot_id|><|begin_of_text|><|start_header_id|>user<|end_header_id|>

Here is a list of five jobs that involve public speaking:

Politician: Politicians often need to make speeches and presentations to voters or audiences.
Salesperson: Salespeople often need to make presentations to potential customers or in conferences.
Teacher: Teachers often need to deliver lectures and presentations to their students.
Accountant: Accountants often need to present financial reports and explain financial processes.
Call Center Agent: Call center agents need to effectively deliver verbal instructions and answer questions.<|eot_id|><|begin_of_text|><|start_header_id|>user<|end_header_id|>

Companies can ensure customer satisfaction by providing excellent customer service, listening to customer feedback, and consistently striving to improve their products and services. Additionally, companies should ensure their employees are properly trained and equipped to handle customer inquiries, complaints, and support. Finally, companies should make it easy for customers to contact them if needed, and respond to inquiries quickly.<|eot_id|><|begin_of_text|><|start_header_id|>user<|end_header_id|>

The pros of AI for medical diagnosis include its potential to reduce medical errors, improve accuracy and speed in diagnosing different diseases, and reduce costs. AI technology can utilize existing medical research and data to help diagnose diseases with greater accuracy and efficiency. Additionally, there is potential for AI to make better use of limited resources by prioritizing treatments and focusing on the most serious cases first. On the other hand, there are some potential drawbacks. For one, AI systems can be prone to bias in data sets and may lead to incorrect diagnoses. Additionally, some people may have ethical concerns about overdependence on AI and its potential privacy risks. Finally, it takes time and resources to develop and implement AI systems in medical settings. All of these should be taken into consideration before implementing AI in medical diagnosis.<|eot_id|><|begin_of_text|><|start_header_id|>user<|end_header_id|>

I. Introduction A. Definition of Education B. Importance of Education II. Benefits of Education A. Financial Stability B. Advancement in Career C. Improved Mental and Emotional Well-being D. Access to Opportunity III. The Need for Accessible Education A. Cost of Education B. Barrier of Language C. Discrepancy in Quality of Education IV. Conclusion A. Summary of Points B. Conclusion<|eot_id|><|begin_of_text|><|start_header_id|>user<|end_header_id|>

Good morning everyone. Today I'm here to talk about some of the current trends in AI technology.

AI technology has made incredible advances in many areas. It has been used to help doctors detect diseases, to optimize traffic flow, and to develop autonomous vehicles. AI is becoming more and more powerful as advances are made in machine learning and natural language processing.

One of the most interesting current trends in AI technology is its use in personal assistants. AI-driven virtual assistants, such as Siri and Alexa, can be used to answer questions, do research, make purchases, and manage schedules. These virtual assistants are continually learning new skills, and they are becoming increasingly advanced.

Another trend in AI is the development of robots that have human-like capabilities. These robots are being developed in order to do tasks that are too difficult or dangerous for humans, such as exploring hazardous environments or performing medical operations.

Finally, AI-enabled computers are also being used to create artificial intelligence-based content, such as news articles and videos. This technology has great potential in areas such as education, entertainment, and journalism.

(x - 1)2 + (y + 2)2 = 25<|eot_id|><|begin_of_text|><|start_header_id|>user<|end_header_id|>

Plant-based proteins can provide essential nutrients such as vitamins, minerals, antioxidants, and fiber.
Plant-based proteins are often lower in calories and fat than proteins from animal sources.<|eot_id|><|begin_of_text|><|start_header_id|>user<|end_header_id|>

This algorithm takes a string as input.

Create two counters, "open" and "close" to store the number of open (()) and closed ()) parenthesis in the given string.
Iterate over the characters of the string and for every "(", increment the "open" counter, for every ")" increment the "close" counter.
After iterating over the characters, compare the values stored in both the counters. If the values are equal and greater than 0, then the string can be balanced, else the string cannot be balanced.
If the string can be balanced, return true, else return false.<|eot_id|><|begin_of_text|><|start_header_id|>user<|end_header_id|>

A simple circuit featuring a resistor and a capacitor could involve connecting the positive end of a battery to the resistor and then connecting the resistor to the capacitor. The capacitor should then be connected to the negative end of the battery, so that the circuit is closed. This type of circuit will allow current to flow through the resistor, first charging the capacitor and then discharging it as the current dissipates.<|eot_id|><|begin_of_text|><|start_header_id|>user<|end_header_id|>

1010<|eot_id|><|begin_of_text|><|start_header_id|>user<|end_header_id|>

One potential application of the Differential Evolution Algorithm (DEA) is to use it to control autonomous vehicles. By integrating this AI algorithm into the driving system of a car, it can be programmed to quickly evaluate the best route depending on the traffic situation and optimize it according to the given conditions. In addition, DEA can also adjust the route in real time, allowing the vehicle to make course corrections if needed. By doing this, the car can drive in a much more efficient and safe manner, allowing drivers to get to their destination faster and safer.<|eot_id|><|begin_of_text|><|start_header_id|>user<|end_header_id|>

Write a sentence describing a car 06/21 18:32:38 - mmengine - INFO - before_train in EvaluateChatHook. 06/21 18:32:54 - mmengine - INFO - Sample output: <|start_header_id|>system<|end_header_id|>

Here are 5 scenic spots in Shanghai:

The Bund (外滩): A famous waterfront area along the Huangpu River, known for its stunning views of the city skyline, especially during sunset. You can take a leisurely

06/21 18:33:07 - mmengine - INFO - Sample output: <|start_header_id|>system<|end_header_id|>

Here are five scenic spots in Shanghai:

The Bund (Waitan): A famous waterfront area along the Huangpu River, known for its stunning views of the city skyline, especially during sunset. You can take a leisurely stroll along

06/21 18:33:07 - mmengine - WARNING - "FileClient" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io 06/21 18:33:07 - mmengine - WARNING - "HardDiskBackend" is the alias of "LocalBackend" and the former will be deprecated in future. 06/21 18:33:07 - mmengine - INFO - Checkpoints will be saved to /lytest/code/algorithm/llama3_pytorch/work_dirs/llama3_8b_instruct_qlora_alpaca_e3_M. Invalid address access: 0x7f881bb90000, Error code: 1.

KERNEL VMFault !!!! <<<<<<

PID: 3967, SIGNAL: 0 !!!! <<<<<< =========> HOSTQUEUE <0x7f845c003e20>: VMFault HSA QUEUE ANALYSIS <========= =========> HOSTQUEUE <0x7f880c00d8b0>: VMFault HSA QUEUE ANALYSIS <========= =========> HOSTQUEUE <0x7f880c00b230>: VMFault HSA QUEUE ANALYSIS <========= =========> HOSTQUEUE <0x7f880c00b8a0>: VMFault HSA QUEUE ANALYSIS <========= =========> HOSTQUEUE <0x135797e0>: VMFault HSA QUEUE ANALYSIS <========= Invalid address access: 0x7fce6dd7b000, Error code: 1.

KERNEL VMFault !!!! <<<<<<

PID: 3966, SIGNAL: 0 !!!! <<<<<< =========> HOSTQUEUE <0x7fcb24004c40>: VMFault HSA QUEUE ANALYSIS <========= =========> HOSTQUEUE <0x7fcb24004540>: VMFault HSA QUEUE ANALYSIS <========= =========> HOSTQUEUE <0x7fcb24003e20>: VMFault HSA QUEUE ANALYSIS <========= =========> HOSTQUEUE <0x1371cda0>: VMFault HSA QUEUE ANALYSIS <========= Invalid address access: 0x7f35ba9c4000, Error code: 1.

KERNEL VMFault !!!! <<<<<<

PID: 3965, SIGNAL: 0 !!!! <<<<<< =========> HOSTQUEUE <0x7f3268004c40>: VMFault HSA QUEUE ANALYSIS <========= =========> HOSTQUEUE <0x7f35ac00b8a0>: VMFault HSA QUEUE ANALYSIS <========= Invalid address access: 0x7f8844fb8000, Error code: 1.

KERNEL VMFault !!!! <<<<<<

PID: 3964, SIGNAL: 0 !!!! <<<<<< =========> HOSTQUEUE <0x7f87f80096a0>: VMFault HSA QUEUE ANALYSIS <========= =========> HOSTQUEUE <0x7f87f80091d0>: VMFault HSA QUEUE ANALYSIS <========= =========> HOSTQUEUE <0x7f35ac00b230>: VMFault HSA QUEUE ANALYSIS <========= =========> HOSTQUEUE <0x137b7700>: VMFault HSA QUEUE ANALYSIS <========= =========> HOSTQUEUE <0x7f882c00d8b0>: VMFault HSA QUEUE ANALYSIS <========= =========> HOSTQUEUE <0x7f882c00b8a0>: VMFault HSA QUEUE ANALYSIS <========= =========> HOSTQUEUE <0x1393f630>: VMFault HSA QUEUE ANALYSIS <=========