debug.log

2024-12-16 09:45:24,954 INFO    MainThread:123 [wandb_setup.py:_flush():68] Current SDK version is 0.19.1
2024-12-16 09:45:24,954 INFO    MainThread:123 [wandb_setup.py:_flush():68] Configure stats pid to 123
2024-12-16 09:45:24,954 INFO    MainThread:123 [wandb_setup.py:_flush():68] Loading settings from /root/.config/wandb/settings
2024-12-16 09:45:24,954 INFO    MainThread:123 [wandb_setup.py:_flush():68] Loading settings from /mnt/xgen-mm/LAVIS/wandb/settings
2024-12-16 09:45:24,954 INFO    MainThread:123 [wandb_setup.py:_flush():68] Loading settings from environment variables
2024-12-16 09:45:24,954 INFO    MainThread:123 [wandb_init.py:_log_setup():528] Logging user logs to /mnt/xgen-mm/LAVIS/wandb/run-20241216_094524-qoi2bab2/logs/debug.log
2024-12-16 09:45:24,954 INFO    MainThread:123 [wandb_init.py:_log_setup():529] Logging internal logs to /mnt/xgen-mm/LAVIS/wandb/run-20241216_094524-qoi2bab2/logs/debug-internal.log
2024-12-16 09:45:24,954 INFO    MainThread:123 [wandb_init.py:init():644] calling init triggers
2024-12-16 09:45:24,954 INFO    MainThread:123 [wandb_init.py:init():650] wandb.init called with sweep_config: {}
config: {'model_family': 'xgenmm_v1', 'vision_encoder_path': 'google/siglip-so400m-patch14-384', 'vision_encoder_pretrained': 'google', 'lm_path': 'microsoft/Phi-3-mini-4k-instruct', 'tokenizer_path': 'microsoft/Phi-3-mini-4k-instruct', 'cross_attn_every_n_layers': 1, 'num_vision_tokens': 128, 'pretrained': '/mnt/xgen-mm/xgen-mm-phi3-mini-base-r-v1.5.pt', 'pretrained_vision_tokenizer': None, 'loss': 'supervised_finetune', 'run_name': 'finetune-xgenmmv1-phi3_4k_instruct-example_data_config', 'resume_from_checkpoint': None, 'delete_previous_checkpoint': False, 'no_save_optim_state': True, 'gradient_accumulation_steps': 1, 'seed': 42, 'learning_rate': 2e-05, 'lr_scheduler': 'cosine', 'warmup_steps': 2000, 'weight_decay': 0.0, 'precision': 'amp_bf16', 'gradient_checkpointing': True, 'num_epochs': 1, 'offline': False, 'logging_steps': 100, 'checkpoint_steps': 5000, 'data_path': '/mnt/xgen-mm/LAVIS/data_configs/example_data_config.yaml', 'batch_size': 8, 'workers': 4, 'data_sampler_group_by_length': True, 'is_multimodal': True, 'mm_use_im_start_end': False, 'conv_template_name': 'phi_3', 'image_aspect_ratio': 'anyres', 'anyres_patch_sampling': True, 'anyres_grids': [(1, 2), (2, 1), (2, 2), (3, 1), (1, 3)], 'dist_url': 'env://', 'dist_backend': 'nccl', 'horovod': False, 'no_set_device_rank': False, 'local_rank': 0, 'fsdp': True, 'fsdp_sharding_strategy': 'hybrid', 'report_to_wandb': True, 'wandb_project': 'blip3-xgenmm-finetune', 'wandb_entity': None, 'save_checkpoints_to_wandb': False, 'dryrun': False, 'use_flash_attention_2': False, 'unfreeze_vision_encoder': False, 'vision_encoder_precision': 'fp32', 'cpu_offload_gradients': False, 'rank': 0, 'world_size': 8, 'distributed': True, 'device': 'cuda:0'}
2024-12-16 09:45:24,954 INFO    MainThread:123 [wandb_init.py:init():680] starting backend
2024-12-16 09:45:24,954 INFO    MainThread:123 [wandb_init.py:init():684] sending inform_init request
2024-12-16 09:45:24,982 INFO    MainThread:123 [backend.py:_multiprocessing_setup():104] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2024-12-16 09:45:24,982 INFO    MainThread:123 [wandb_init.py:init():697] backend started and connected
2024-12-16 09:45:24,984 INFO    MainThread:123 [wandb_init.py:init():790] updated telemetry
2024-12-16 09:45:25,024 INFO    MainThread:123 [wandb_init.py:init():822] communicating run to backend with 90.0 second timeout