Skip to content

GitLab

Commit 1b9205c9 authored by yangzhong's avatar yangzhong
Browse files

v1.0

parents
Pipeline #2931 failed with stages
in 0 seconds
Hide whitespace changes
Inline Side-by-side
Showing with 344 additions and 0 deletions
wandb/run-20241216_094524-qoi2bab2/logs/debug-core.log 0 → 120000
View file @ 1b9205c9
/root/.cache/wandb/logs/core-debug-20241216_094519.log
\ No newline at end of file
wandb/run-20241216_094524-qoi2bab2/logs/debug-internal.log 0 → 100644
View file @ 1b9205c9
{"time":"2024-12-16T09:45:24.991774535+08:00","level":"INFO","msg":"using version","core version":"0.19.1"}
{"time":"2024-12-16T09:45:24.991786433+08:00","level":"INFO","msg":"created symlink","path":"/mnt/xgen-mm/LAVIS/wandb/run-20241216_094524-qoi2bab2/logs/debug-core.log"}
{"time":"2024-12-16T09:45:25.105314506+08:00","level":"INFO","msg":"created new stream","id":"qoi2bab2"}
{"time":"2024-12-16T09:45:25.105355079+08:00","level":"INFO","msg":"stream: started","id":"qoi2bab2"}
{"time":"2024-12-16T09:45:25.10538369+08:00","level":"INFO","msg":"sender: started","stream_id":"qoi2bab2"}
{"time":"2024-12-16T09:45:25.105373061+08:00","level":"INFO","msg":"writer: Do: started","stream_id":"qoi2bab2"}
{"time":"2024-12-16T09:45:25.105386768+08:00","level":"INFO","msg":"handler: started","stream_id":"qoi2bab2"}
{"time":"2024-12-16T09:45:35.115859798+08:00","level":"INFO","msg":"api: retrying error","error":"Post \"https://api.wandb.ai/graphql\": dial tcp: lookup api.wandb.ai on 114.114.114.114:53: read udp 10.16.4.6:50950->114.114.114.114:53: i/o timeout"}
wandb/run-20241216_094524-qoi2bab2/logs/debug.log 0 → 100644
View file @ 1b9205c9
2024-12-16 09:45:24,954 INFO MainThread:123 [wandb_setup.py:_flush():68] Current SDK version is 0.19.1
2024-12-16 09:45:24,954 INFO MainThread:123 [wandb_setup.py:_flush():68] Configure stats pid to 123
2024-12-16 09:45:24,954 INFO MainThread:123 [wandb_setup.py:_flush():68] Loading settings from /root/.config/wandb/settings
2024-12-16 09:45:24,954 INFO MainThread:123 [wandb_setup.py:_flush():68] Loading settings from /mnt/xgen-mm/LAVIS/wandb/settings
2024-12-16 09:45:24,954 INFO MainThread:123 [wandb_setup.py:_flush():68] Loading settings from environment variables
2024-12-16 09:45:24,954 INFO MainThread:123 [wandb_init.py:_log_setup():528] Logging user logs to /mnt/xgen-mm/LAVIS/wandb/run-20241216_094524-qoi2bab2/logs/debug.log
2024-12-16 09:45:24,954 INFO MainThread:123 [wandb_init.py:_log_setup():529] Logging internal logs to /mnt/xgen-mm/LAVIS/wandb/run-20241216_094524-qoi2bab2/logs/debug-internal.log
2024-12-16 09:45:24,954 INFO MainThread:123 [wandb_init.py:init():644] calling init triggers
2024-12-16 09:45:24,954 INFO MainThread:123 [wandb_init.py:init():650] wandb.init called with sweep_config: {}
config: {'model_family': 'xgenmm_v1', 'vision_encoder_path': 'google/siglip-so400m-patch14-384', 'vision_encoder_pretrained': 'google', 'lm_path': 'microsoft/Phi-3-mini-4k-instruct', 'tokenizer_path': 'microsoft/Phi-3-mini-4k-instruct', 'cross_attn_every_n_layers': 1, 'num_vision_tokens': 128, 'pretrained': '/mnt/xgen-mm/xgen-mm-phi3-mini-base-r-v1.5.pt', 'pretrained_vision_tokenizer': None, 'loss': 'supervised_finetune', 'run_name': 'finetune-xgenmmv1-phi3_4k_instruct-example_data_config', 'resume_from_checkpoint': None, 'delete_previous_checkpoint': False, 'no_save_optim_state': True, 'gradient_accumulation_steps': 1, 'seed': 42, 'learning_rate': 2e-05, 'lr_scheduler': 'cosine', 'warmup_steps': 2000, 'weight_decay': 0.0, 'precision': 'amp_bf16', 'gradient_checkpointing': True, 'num_epochs': 1, 'offline': False, 'logging_steps': 100, 'checkpoint_steps': 5000, 'data_path': '/mnt/xgen-mm/LAVIS/data_configs/example_data_config.yaml', 'batch_size': 8, 'workers': 4, 'data_sampler_group_by_length': True, 'is_multimodal': True, 'mm_use_im_start_end': False, 'conv_template_name': 'phi_3', 'image_aspect_ratio': 'anyres', 'anyres_patch_sampling': True, 'anyres_grids': [(1, 2), (2, 1), (2, 2), (3, 1), (1, 3)], 'dist_url': 'env://', 'dist_backend': 'nccl', 'horovod': False, 'no_set_device_rank': False, 'local_rank': 0, 'fsdp': True, 'fsdp_sharding_strategy': 'hybrid', 'report_to_wandb': True, 'wandb_project': 'blip3-xgenmm-finetune', 'wandb_entity': None, 'save_checkpoints_to_wandb': False, 'dryrun': False, 'use_flash_attention_2': False, 'unfreeze_vision_encoder': False, 'vision_encoder_precision': 'fp32', 'cpu_offload_gradients': False, 'rank': 0, 'world_size': 8, 'distributed': True, 'device': 'cuda:0'}
2024-12-16 09:45:24,954 INFO MainThread:123 [wandb_init.py:init():680] starting backend
2024-12-16 09:45:24,954 INFO MainThread:123 [wandb_init.py:init():684] sending inform_init request
2024-12-16 09:45:24,982 INFO MainThread:123 [backend.py:_multiprocessing_setup():104] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2024-12-16 09:45:24,982 INFO MainThread:123 [wandb_init.py:init():697] backend started and connected
2024-12-16 09:45:24,984 INFO MainThread:123 [wandb_init.py:init():790] updated telemetry
2024-12-16 09:45:25,024 INFO MainThread:123 [wandb_init.py:init():822] communicating run to backend with 90.0 second timeout
wandb/run-20241216_101034-bluz2d3p/logs/debug-core.log 0 → 120000
View file @ 1b9205c9
/root/.cache/wandb/logs/core-debug-20241216_101028.log
\ No newline at end of file
wandb/run-20241216_101034-bluz2d3p/logs/debug-internal.log 0 → 100644
View file @ 1b9205c9
{"time":"2024-12-16T10:10:34.060436075+08:00","level":"INFO","msg":"using version","core version":"0.19.1"}
{"time":"2024-12-16T10:10:34.060445775+08:00","level":"INFO","msg":"created symlink","path":"/mnt/xgen-mm/LAVIS/wandb/run-20241216_101034-bluz2d3p/logs/debug-core.log"}
{"time":"2024-12-16T10:10:34.170997527+08:00","level":"INFO","msg":"created new stream","id":"bluz2d3p"}
{"time":"2024-12-16T10:10:34.171047221+08:00","level":"INFO","msg":"stream: started","id":"bluz2d3p"}
{"time":"2024-12-16T10:10:34.171064676+08:00","level":"INFO","msg":"writer: Do: started","stream_id":"bluz2d3p"}
{"time":"2024-12-16T10:10:34.171071386+08:00","level":"INFO","msg":"sender: started","stream_id":"bluz2d3p"}
{"time":"2024-12-16T10:10:34.171062759+08:00","level":"INFO","msg":"handler: started","stream_id":"bluz2d3p"}
{"time":"2024-12-16T10:10:44.174006113+08:00","level":"INFO","msg":"api: retrying error","error":"Post \"https://api.wandb.ai/graphql\": dial tcp: lookup api.wandb.ai on 114.114.114.114:53: read udp 10.16.4.6:36951->114.114.114.114:53: i/o timeout"}
{"time":"2024-12-16T10:10:56.587056337+08:00","level":"INFO","msg":"api: retrying error","error":"Post \"https://api.wandb.ai/graphql\": dial tcp: lookup api.wandb.ai on 114.114.114.114:53: read udp 10.16.4.6:51531->114.114.114.114:53: i/o timeout"}
{"time":"2024-12-16T10:11:10.832560689+08:00","level":"INFO","msg":"api: retrying error","error":"Post \"https://api.wandb.ai/graphql\": dial tcp: lookup api.wandb.ai on 114.114.114.114:53: read udp 10.16.4.6:38412->114.114.114.114:53: i/o timeout"}
{"time":"2024-12-16T10:11:29.458821558+08:00","level":"INFO","msg":"api: retrying error","error":"Post \"https://api.wandb.ai/graphql\": dial tcp: lookup api.wandb.ai on 114.114.114.114:53: read udp 10.16.4.6:47026->114.114.114.114:53: i/o timeout"}
{"time":"2024-12-16T10:11:55.531697876+08:00","level":"INFO","msg":"api: retrying error","error":"Post \"https://api.wandb.ai/graphql\": dial tcp: lookup api.wandb.ai on 114.114.114.114:53: read udp 10.16.4.6:37457->114.114.114.114:53: i/o timeout"}
{"time":"2024-12-16T10:12:04.091813801+08:00","level":"INFO","msg":"stream: closing","id":"bluz2d3p"}
{"time":"2024-12-16T10:12:04.091934094+08:00","level":"INFO","msg":"handler: closed","stream_id":"bluz2d3p"}
{"time":"2024-12-16T10:12:04.091940619+08:00","level":"INFO","msg":"writer: Close: closed","stream_id":"bluz2d3p"}
{"time":"2024-12-16T10:12:04.091970789+08:00","level":"ERROR","msg":"sender: upsertRun:","error":"failed to upsert bucket: api: failed sending: context canceled"}
{"time":"2024-12-16T10:12:04.092014628+08:00","level":"WARN","msg":"runwork: ignoring record after close","work":{"Record":{"RecordType":{"Request":{"RequestType":{"Defer":{}}}},"control":{"always_send":true}}}}
{"time":"2024-12-16T10:12:04.092126412+08:00","level":"INFO","msg":"sender: closed","stream_id":"bluz2d3p"}
{"time":"2024-12-16T10:12:04.092346938+08:00","level":"INFO","msg":"stream: closed","id":"bluz2d3p"}
wandb/run-20241216_101034-bluz2d3p/logs/debug.log 0 → 100644
View file @ 1b9205c9
2024-12-16 10:10:34,054 INFO MainThread:1952 [wandb_setup.py:_flush():68] Current SDK version is 0.19.1
2024-12-16 10:10:34,054 INFO MainThread:1952 [wandb_setup.py:_flush():68] Configure stats pid to 1952
2024-12-16 10:10:34,054 INFO MainThread:1952 [wandb_setup.py:_flush():68] Loading settings from /root/.config/wandb/settings
2024-12-16 10:10:34,054 INFO MainThread:1952 [wandb_setup.py:_flush():68] Loading settings from /mnt/xgen-mm/LAVIS/wandb/settings
2024-12-16 10:10:34,054 INFO MainThread:1952 [wandb_setup.py:_flush():68] Loading settings from environment variables
2024-12-16 10:10:34,054 INFO MainThread:1952 [wandb_init.py:_log_setup():528] Logging user logs to /mnt/xgen-mm/LAVIS/wandb/run-20241216_101034-bluz2d3p/logs/debug.log
2024-12-16 10:10:34,054 INFO MainThread:1952 [wandb_init.py:_log_setup():529] Logging internal logs to /mnt/xgen-mm/LAVIS/wandb/run-20241216_101034-bluz2d3p/logs/debug-internal.log
2024-12-16 10:10:34,054 INFO MainThread:1952 [wandb_init.py:init():644] calling init triggers
2024-12-16 10:10:34,054 INFO MainThread:1952 [wandb_init.py:init():650] wandb.init called with sweep_config: {}
config: {'model_family': 'xgenmm_v1', 'vision_encoder_path': 'google/siglip-so400m-patch14-384', 'vision_encoder_pretrained': 'google', 'lm_path': 'microsoft/Phi-3-mini-4k-instruct', 'tokenizer_path': 'microsoft/Phi-3-mini-4k-instruct', 'cross_attn_every_n_layers': 1, 'num_vision_tokens': 128, 'pretrained': '/mnt/xgen-mm/xgen-mm-phi3-mini-base-r-v1.5.pt', 'pretrained_vision_tokenizer': None, 'loss': 'supervised_finetune', 'run_name': 'finetune-xgenmmv1-phi3_4k_instruct-example_data_config', 'resume_from_checkpoint': None, 'delete_previous_checkpoint': False, 'no_save_optim_state': True, 'gradient_accumulation_steps': 1, 'seed': 42, 'learning_rate': 2e-05, 'lr_scheduler': 'cosine', 'warmup_steps': 2000, 'weight_decay': 0.0, 'precision': 'amp_bf16', 'gradient_checkpointing': True, 'num_epochs': 1, 'offline': False, 'logging_steps': 100, 'checkpoint_steps': 5000, 'data_path': '/mnt/xgen-mm/LAVIS/data_configs/example_data_config.yaml', 'batch_size': 8, 'workers': 4, 'data_sampler_group_by_length': True, 'is_multimodal': True, 'mm_use_im_start_end': False, 'conv_template_name': 'phi_3', 'image_aspect_ratio': 'anyres', 'anyres_patch_sampling': True, 'anyres_grids': [(1, 2), (2, 1), (2, 2), (3, 1), (1, 3)], 'dist_url': 'env://', 'dist_backend': 'nccl', 'horovod': False, 'no_set_device_rank': False, 'local_rank': 0, 'fsdp': True, 'fsdp_sharding_strategy': 'hybrid', 'report_to_wandb': True, 'wandb_project': 'blip3-xgenmm-finetune', 'wandb_entity': None, 'save_checkpoints_to_wandb': False, 'dryrun': False, 'use_flash_attention_2': False, 'unfreeze_vision_encoder': False, 'vision_encoder_precision': 'fp32', 'cpu_offload_gradients': False, 'rank': 0, 'world_size': 8, 'distributed': True, 'device': 'cuda:0'}
2024-12-16 10:10:34,054 INFO MainThread:1952 [wandb_init.py:init():680] starting backend
2024-12-16 10:10:34,054 INFO MainThread:1952 [wandb_init.py:init():684] sending inform_init request
2024-12-16 10:10:34,058 INFO MainThread:1952 [backend.py:_multiprocessing_setup():104] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2024-12-16 10:10:34,058 INFO MainThread:1952 [wandb_init.py:init():697] backend started and connected
2024-12-16 10:10:34,061 INFO MainThread:1952 [wandb_init.py:init():790] updated telemetry
2024-12-16 10:10:34,068 INFO MainThread:1952 [wandb_init.py:init():822] communicating run to backend with 90.0 second timeout
2024-12-16 10:11:02,254 INFO Thread-2 (wrapped_target):1952 [retry.py:__call__():172] Retry attempt failed:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 199, in _new_conn
sock = connection.create_connection(
File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 789, in urlopen
response = self._make_request(
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 490, in _make_request
raise new_e
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 466, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1095, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 693, in connect
self.sock = sock = self._new_conn()
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 206, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0x7f5d48759540>: Failed to resolve 'api.wandb.ai' ([Errno -3] Temporary failure in name resolution)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 667, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 843, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 519, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.wandb.ai', port=443): Max retries exceeded with url: /graphql (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7f5d48759540>: Failed to resolve 'api.wandb.ai' ([Errno -3] Temporary failure in name resolution)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/wandb/sdk/lib/retry.py", line 131, in __call__
result = self._call_fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/wandb/sdk/internal/internal_api.py", line 393, in execute
return self.client.execute(*args, **kwargs) # type: ignore
File "/usr/local/lib/python3.10/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 52, in execute
result = self._get_result(document, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 60, in _get_result
return self.transport.execute(document, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/wandb/sdk/lib/gql_request.py", line 58, in execute
request = self.session.post(self.url, **post_args)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 700, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.wandb.ai', port=443): Max retries exceeded with url: /graphql (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7f5d48759540>: Failed to resolve 'api.wandb.ai' ([Errno -3] Temporary failure in name resolution)"))
2024-12-16 10:12:04,084 ERROR MainThread:1952 [wandb_init.py:init():849] encountered error: Run initialization has timed out after 90.0 sec. Please try increasing the timeout with the `init_timeout` setting: `wandb.init(settings=wandb.Settings(init_timeout=120))`.
2024-12-16 10:12:04,085 ERROR MainThread:1952 [wandb_init.py:init():1308] error in wandb.init()
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1298, in init
return wi.init()
File "/usr/local/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 855, in init
raise error
wandb.errors.errors.CommError: Run initialization has timed out after 90.0 sec. Please try increasing the timeout with the `init_timeout` setting: `wandb.init(settings=wandb.Settings(init_timeout=120))`.
2024-12-16 10:12:04,091 WARNING MsgRouterThr:1952 [router.py:message_loop():75] message_loop has been closed
wandb/run-20241216_101627-3las4gzs/logs/debug-core.log 0 → 120000
View file @ 1b9205c9
/root/.cache/wandb/logs/core-debug-20241216_101622.log
\ No newline at end of file
wandb/run-20241216_101627-3las4gzs/logs/debug-internal.log 0 → 100644
View file @ 1b9205c9
{"time":"2024-12-16T10:16:27.456947987+08:00","level":"INFO","msg":"using version","core version":"0.19.1"}
{"time":"2024-12-16T10:16:27.456957573+08:00","level":"INFO","msg":"created symlink","path":"/mnt/xgen-mm/LAVIS/wandb/run-20241216_101627-3las4gzs/logs/debug-core.log"}
{"time":"2024-12-16T10:16:27.567756055+08:00","level":"INFO","msg":"created new stream","id":"3las4gzs"}
{"time":"2024-12-16T10:16:27.567795709+08:00","level":"INFO","msg":"stream: started","id":"3las4gzs"}
{"time":"2024-12-16T10:16:27.567812708+08:00","level":"INFO","msg":"writer: Do: started","stream_id":"3las4gzs"}
{"time":"2024-12-16T10:16:27.567813267+08:00","level":"INFO","msg":"sender: started","stream_id":"3las4gzs"}
{"time":"2024-12-16T10:16:27.567816566+08:00","level":"INFO","msg":"handler: started","stream_id":"3las4gzs"}
{"time":"2024-12-16T10:16:37.570095655+08:00","level":"INFO","msg":"api: retrying error","error":"Post \"https://api.wandb.ai/graphql\": dial tcp: lookup api.wandb.ai on 114.114.114.114:53: read udp 10.16.4.6:33868->114.114.114.114:53: i/o timeout"}
{"time":"2024-12-16T10:16:50.053546304+08:00","level":"INFO","msg":"api: retrying error","error":"Post \"https://api.wandb.ai/graphql\": dial tcp: lookup api.wandb.ai on 114.114.114.114:53: read udp 10.16.4.6:51552->114.114.114.114:53: i/o timeout"}
{"time":"2024-12-16T10:17:04.464940038+08:00","level":"INFO","msg":"api: retrying error","error":"Post \"https://api.wandb.ai/graphql\": dial tcp: lookup api.wandb.ai on 114.114.114.114:53: read udp 10.16.4.6:37031->114.114.114.114:53: i/o timeout"}
{"time":"2024-12-16T10:17:23.179941866+08:00","level":"INFO","msg":"api: retrying error","error":"Post \"https://api.wandb.ai/graphql\": dial tcp: lookup api.wandb.ai on 114.114.114.114:53: read udp 10.16.4.6:45848->114.114.114.114:53: i/o timeout"}
{"time":"2024-12-16T10:17:49.704248141+08:00","level":"INFO","msg":"api: retrying error","error":"Post \"https://api.wandb.ai/graphql\": dial tcp: lookup api.wandb.ai on 114.114.114.114:53: read udp 10.16.4.6:58012->114.114.114.114:53: i/o timeout"}
{"time":"2024-12-16T10:17:57.499829779+08:00","level":"INFO","msg":"stream: closing","id":"3las4gzs"}
{"time":"2024-12-16T10:17:57.499987602+08:00","level":"INFO","msg":"handler: closed","stream_id":"3las4gzs"}
{"time":"2024-12-16T10:17:57.500002408+08:00","level":"INFO","msg":"writer: Close: closed","stream_id":"3las4gzs"}
{"time":"2024-12-16T10:17:57.500022511+08:00","level":"ERROR","msg":"sender: upsertRun:","error":"failed to upsert bucket: api: failed sending: context canceled"}
{"time":"2024-12-16T10:17:57.500065863+08:00","level":"WARN","msg":"runwork: ignoring record after close","work":{"Record":{"RecordType":{"Request":{"RequestType":{"Defer":{}}}},"control":{"always_send":true}}}}
{"time":"2024-12-16T10:17:57.500185254+08:00","level":"INFO","msg":"sender: closed","stream_id":"3las4gzs"}
{"time":"2024-12-16T10:17:57.500386839+08:00","level":"INFO","msg":"stream: closed","id":"3las4gzs"}
wandb/run-20241216_101627-3las4gzs/logs/debug.log 0 → 100644
View file @ 1b9205c9
2024-12-16 10:16:27,450 INFO MainThread:2689 [wandb_setup.py:_flush():68] Current SDK version is 0.19.1
2024-12-16 10:16:27,450 INFO MainThread:2689 [wandb_setup.py:_flush():68] Configure stats pid to 2689
2024-12-16 10:16:27,451 INFO MainThread:2689 [wandb_setup.py:_flush():68] Loading settings from /root/.config/wandb/settings
2024-12-16 10:16:27,451 INFO MainThread:2689 [wandb_setup.py:_flush():68] Loading settings from /mnt/xgen-mm/LAVIS/wandb/settings
2024-12-16 10:16:27,451 INFO MainThread:2689 [wandb_setup.py:_flush():68] Loading settings from environment variables
2024-12-16 10:16:27,451 INFO MainThread:2689 [wandb_init.py:_log_setup():528] Logging user logs to /mnt/xgen-mm/LAVIS/wandb/run-20241216_101627-3las4gzs/logs/debug.log
2024-12-16 10:16:27,451 INFO MainThread:2689 [wandb_init.py:_log_setup():529] Logging internal logs to /mnt/xgen-mm/LAVIS/wandb/run-20241216_101627-3las4gzs/logs/debug-internal.log
2024-12-16 10:16:27,451 INFO MainThread:2689 [wandb_init.py:init():644] calling init triggers
2024-12-16 10:16:27,451 INFO MainThread:2689 [wandb_init.py:init():650] wandb.init called with sweep_config: {}
config: {'model_family': 'xgenmm_v1', 'vision_encoder_path': 'google/siglip-so400m-patch14-384', 'vision_encoder_pretrained': 'google', 'lm_path': 'microsoft/Phi-3-mini-4k-instruct', 'tokenizer_path': 'microsoft/Phi-3-mini-4k-instruct', 'cross_attn_every_n_layers': 1, 'num_vision_tokens': 128, 'pretrained': '/mnt/xgen-mm/xgen-mm-phi3-mini-base-r-v1.5.pt', 'pretrained_vision_tokenizer': None, 'loss': 'supervised_finetune', 'run_name': 'finetune-xgenmmv1-phi3_4k_instruct-example_data_config', 'resume_from_checkpoint': None, 'delete_previous_checkpoint': False, 'no_save_optim_state': True, 'gradient_accumulation_steps': 1, 'seed': 42, 'learning_rate': 2e-05, 'lr_scheduler': 'cosine', 'warmup_steps': 2000, 'weight_decay': 0.0, 'precision': 'amp_bf16', 'gradient_checkpointing': True, 'num_epochs': 1, 'offline': False, 'logging_steps': 100, 'checkpoint_steps': 5000, 'data_path': '/mnt/xgen-mm/LAVIS/data_configs/example_data_config.yaml', 'batch_size': 8, 'workers': 4, 'data_sampler_group_by_length': True, 'is_multimodal': True, 'mm_use_im_start_end': False, 'conv_template_name': 'phi_3', 'image_aspect_ratio': 'anyres', 'anyres_patch_sampling': True, 'anyres_grids': [(1, 2), (2, 1), (2, 2), (3, 1), (1, 3)], 'dist_url': 'env://', 'dist_backend': 'nccl', 'horovod': False, 'no_set_device_rank': False, 'local_rank': 0, 'fsdp': True, 'fsdp_sharding_strategy': 'hybrid', 'report_to_wandb': True, 'wandb_project': 'blip3-xgenmm-finetune', 'wandb_entity': None, 'save_checkpoints_to_wandb': False, 'dryrun': False, 'use_flash_attention_2': False, 'unfreeze_vision_encoder': False, 'vision_encoder_precision': 'fp32', 'cpu_offload_gradients': False, 'rank': 0, 'world_size': 8, 'distributed': True, 'device': 'cuda:0'}
2024-12-16 10:16:27,451 INFO MainThread:2689 [wandb_init.py:init():680] starting backend
2024-12-16 10:16:27,451 INFO MainThread:2689 [wandb_init.py:init():684] sending inform_init request
2024-12-16 10:16:27,455 INFO MainThread:2689 [backend.py:_multiprocessing_setup():104] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2024-12-16 10:16:27,455 INFO MainThread:2689 [wandb_init.py:init():697] backend started and connected
2024-12-16 10:16:27,457 INFO MainThread:2689 [wandb_init.py:init():790] updated telemetry
2024-12-16 10:16:27,464 INFO MainThread:2689 [wandb_init.py:init():822] communicating run to backend with 90.0 second timeout
2024-12-16 10:16:55,651 INFO Thread-2 (wrapped_target):2689 [retry.py:__call__():172] Retry attempt failed:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 199, in _new_conn
sock = connection.create_connection(
File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 789, in urlopen
response = self._make_request(
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 490, in _make_request
raise new_e
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 466, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1095, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 693, in connect
self.sock = sock = self._new_conn()
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 206, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0x7f7fd65755a0>: Failed to resolve 'api.wandb.ai' ([Errno -3] Temporary failure in name resolution)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 667, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 843, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 519, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.wandb.ai', port=443): Max retries exceeded with url: /graphql (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7f7fd65755a0>: Failed to resolve 'api.wandb.ai' ([Errno -3] Temporary failure in name resolution)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/wandb/sdk/lib/retry.py", line 131, in __call__
result = self._call_fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/wandb/sdk/internal/internal_api.py", line 393, in execute
return self.client.execute(*args, **kwargs) # type: ignore
File "/usr/local/lib/python3.10/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 52, in execute
result = self._get_result(document, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 60, in _get_result
return self.transport.execute(document, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/wandb/sdk/lib/gql_request.py", line 58, in execute
request = self.session.post(self.url, **post_args)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 700, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.wandb.ai', port=443): Max retries exceeded with url: /graphql (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7f7fd65755a0>: Failed to resolve 'api.wandb.ai' ([Errno -3] Temporary failure in name resolution)"))
2024-12-16 10:17:57,494 ERROR MainThread:2689 [wandb_init.py:init():849] encountered error: Run initialization has timed out after 90.0 sec. Please try increasing the timeout with the `init_timeout` setting: `wandb.init(settings=wandb.Settings(init_timeout=120))`.
2024-12-16 10:17:57,494 ERROR MainThread:2689 [wandb_init.py:init():1308] error in wandb.init()
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1298, in init
return wi.init()
File "/usr/local/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 855, in init
raise error
wandb.errors.errors.CommError: Run initialization has timed out after 90.0 sec. Please try increasing the timeout with the `init_timeout` setting: `wandb.init(settings=wandb.Settings(init_timeout=120))`.
2024-12-16 10:17:57,500 WARNING MsgRouterThr:2689 [router.py:message_loop():75] message_loop has been closed
wandb/run-20241216_103711-tpkv930s/logs/debug-core.log 0 → 120000
View file @ 1b9205c9
/root/.cache/wandb/logs/core-debug-20241216_103706.log
\ No newline at end of file
wandb/run-20241216_103711-tpkv930s/logs/debug-internal.log 0 → 100644
View file @ 1b9205c9
{"time":"2024-12-16T10:37:11.399548112+08:00","level":"INFO","msg":"using version","core version":"0.19.1"}
{"time":"2024-12-16T10:37:11.399559227+08:00","level":"INFO","msg":"created symlink","path":"/mnt/xgen-mm/LAVIS/wandb/run-20241216_103711-tpkv930s/logs/debug-core.log"}
{"time":"2024-12-16T10:37:11.509394189+08:00","level":"INFO","msg":"created new stream","id":"tpkv930s"}
{"time":"2024-12-16T10:37:11.509442549+08:00","level":"INFO","msg":"stream: started","id":"tpkv930s"}
{"time":"2024-12-16T10:37:11.509462125+08:00","level":"INFO","msg":"writer: Do: started","stream_id":"tpkv930s"}
{"time":"2024-12-16T10:37:11.509506869+08:00","level":"INFO","msg":"handler: started","stream_id":"tpkv930s"}
{"time":"2024-12-16T10:37:11.509466637+08:00","level":"INFO","msg":"sender: started","stream_id":"tpkv930s"}
{"time":"2024-12-16T10:37:21.513277968+08:00","level":"INFO","msg":"api: retrying error","error":"Post \"https://api.wandb.ai/graphql\": dial tcp: lookup api.wandb.ai on 114.114.114.114:53: read udp 10.16.4.6:50217->114.114.114.114:53: i/o timeout"}
{"time":"2024-12-16T10:37:33.750478046+08:00","level":"INFO","msg":"api: retrying error","error":"Post \"https://api.wandb.ai/graphql\": dial tcp: lookup api.wandb.ai on 114.114.114.114:53: read udp 10.16.4.6:54015->114.114.114.114:53: i/o timeout"}
{"time":"2024-12-16T10:37:47.82458634+08:00","level":"INFO","msg":"api: retrying error","error":"Post \"https://api.wandb.ai/graphql\": dial tcp: lookup api.wandb.ai on 114.114.114.114:53: read udp 10.16.4.6:57663->114.114.114.114:53: i/o timeout"}
{"time":"2024-12-16T10:38:06.384998773+08:00","level":"INFO","msg":"api: retrying error","error":"Post \"https://api.wandb.ai/graphql\": dial tcp: lookup api.wandb.ai on 114.114.114.114:53: read udp 10.16.4.6:37102->114.114.114.114:53: i/o timeout"}
{"time":"2024-12-16T10:38:36.149856348+08:00","level":"INFO","msg":"api: retrying error","error":"Post \"https://api.wandb.ai/graphql\": dial tcp: lookup api.wandb.ai on 114.114.114.114:53: read udp 10.16.4.6:34939->114.114.114.114:53: i/o timeout"}
{"time":"2024-12-16T10:38:41.423609605+08:00","level":"INFO","msg":"stream: closing","id":"tpkv930s"}
{"time":"2024-12-16T10:38:41.423647579+08:00","level":"INFO","msg":"handler: closed","stream_id":"tpkv930s"}
{"time":"2024-12-16T10:38:41.423657977+08:00","level":"ERROR","msg":"sender: upsertRun:","error":"failed to upsert bucket: api: failed sending: context canceled"}
{"time":"2024-12-16T10:38:41.423672386+08:00","level":"INFO","msg":"writer: Close: closed","stream_id":"tpkv930s"}
{"time":"2024-12-16T10:38:41.423690712+08:00","level":"WARN","msg":"runwork: ignoring record after close","work":{"Record":{"RecordType":{"Request":{"RequestType":{"Defer":{}}}},"control":{"always_send":true}}}}
{"time":"2024-12-16T10:38:41.423816056+08:00","level":"INFO","msg":"sender: closed","stream_id":"tpkv930s"}
{"time":"2024-12-16T10:38:41.424103171+08:00","level":"INFO","msg":"stream: closed","id":"tpkv930s"}
wandb/run-20241216_103711-tpkv930s/logs/debug.log 0 → 100644
View file @ 1b9205c9
2024-12-16 10:37:11,394 INFO MainThread:5226 [wandb_setup.py:_flush():68] Current SDK version is 0.19.1
2024-12-16 10:37:11,394 INFO MainThread:5226 [wandb_setup.py:_flush():68] Configure stats pid to 5226
2024-12-16 10:37:11,394 INFO MainThread:5226 [wandb_setup.py:_flush():68] Loading settings from /root/.config/wandb/settings
2024-12-16 10:37:11,394 INFO MainThread:5226 [wandb_setup.py:_flush():68] Loading settings from /mnt/xgen-mm/LAVIS/wandb/settings
2024-12-16 10:37:11,394 INFO MainThread:5226 [wandb_setup.py:_flush():68] Loading settings from environment variables
2024-12-16 10:37:11,394 INFO MainThread:5226 [wandb_init.py:_log_setup():528] Logging user logs to /mnt/xgen-mm/LAVIS/wandb/run-20241216_103711-tpkv930s/logs/debug.log
2024-12-16 10:37:11,394 INFO MainThread:5226 [wandb_init.py:_log_setup():529] Logging internal logs to /mnt/xgen-mm/LAVIS/wandb/run-20241216_103711-tpkv930s/logs/debug-internal.log
2024-12-16 10:37:11,394 INFO MainThread:5226 [wandb_init.py:init():644] calling init triggers
2024-12-16 10:37:11,394 INFO MainThread:5226 [wandb_init.py:init():650] wandb.init called with sweep_config: {}
config: {'model_family': 'xgenmm_v1', 'vision_encoder_path': 'google/siglip-so400m-patch14-384', 'vision_encoder_pretrained': 'google', 'lm_path': 'microsoft/Phi-3-mini-4k-instruct', 'tokenizer_path': 'microsoft/Phi-3-mini-4k-instruct', 'cross_attn_every_n_layers': 1, 'num_vision_tokens': 128, 'pretrained': '/mnt/xgen-mm/xgen-mm-phi3-mini-base-r-v1.5.pt', 'pretrained_vision_tokenizer': None, 'loss': 'supervised_finetune', 'run_name': 'finetune-xgenmmv1-phi3_4k_instruct-example_data_config', 'resume_from_checkpoint': None, 'delete_previous_checkpoint': False, 'no_save_optim_state': True, 'gradient_accumulation_steps': 1, 'seed': 42, 'learning_rate': 2e-05, 'lr_scheduler': 'cosine', 'warmup_steps': 2000, 'weight_decay': 0.0, 'precision': 'amp_bf16', 'gradient_checkpointing': True, 'num_epochs': 1, 'offline': False, 'logging_steps': 100, 'checkpoint_steps': 5000, 'data_path': '/mnt/xgen-mm/LAVIS/data_configs/example_data_config.yaml', 'batch_size': 8, 'workers': 4, 'data_sampler_group_by_length': True, 'is_multimodal': True, 'mm_use_im_start_end': False, 'conv_template_name': 'phi_3', 'image_aspect_ratio': 'anyres', 'anyres_patch_sampling': True, 'anyres_grids': [(1, 2), (2, 1), (2, 2), (3, 1), (1, 3)], 'dist_url': 'env://', 'dist_backend': 'nccl', 'horovod': False, 'no_set_device_rank': False, 'local_rank': 0, 'fsdp': True, 'fsdp_sharding_strategy': 'hybrid', 'report_to_wandb': True, 'wandb_project': 'blip3-xgenmm-finetune', 'wandb_entity': None, 'save_checkpoints_to_wandb': False, 'dryrun': False, 'use_flash_attention_2': False, 'unfreeze_vision_encoder': False, 'vision_encoder_precision': 'fp32', 'cpu_offload_gradients': False, 'rank': 0, 'world_size': 8, 'distributed': True, 'device': 'cuda:0'}
2024-12-16 10:37:11,394 INFO MainThread:5226 [wandb_init.py:init():680] starting backend
2024-12-16 10:37:11,394 INFO MainThread:5226 [wandb_init.py:init():684] sending inform_init request
2024-12-16 10:37:11,397 INFO MainThread:5226 [backend.py:_multiprocessing_setup():104] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2024-12-16 10:37:11,398 INFO MainThread:5226 [wandb_init.py:init():697] backend started and connected
2024-12-16 10:37:11,400 INFO MainThread:5226 [wandb_init.py:init():790] updated telemetry
2024-12-16 10:37:11,405 INFO MainThread:5226 [wandb_init.py:init():822] communicating run to backend with 90.0 second timeout
2024-12-16 10:37:39,595 INFO Thread-2 (wrapped_target):5226 [retry.py:__call__():172] Retry attempt failed:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 199, in _new_conn
sock = connection.create_connection(
File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 789, in urlopen
response = self._make_request(
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 490, in _make_request
raise new_e
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 466, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1095, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 693, in connect
self.sock = sock = self._new_conn()
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 206, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0x7f46d5f21510>: Failed to resolve 'api.wandb.ai' ([Errno -3] Temporary failure in name resolution)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 667, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 843, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 519, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.wandb.ai', port=443): Max retries exceeded with url: /graphql (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7f46d5f21510>: Failed to resolve 'api.wandb.ai' ([Errno -3] Temporary failure in name resolution)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/wandb/sdk/lib/retry.py", line 131, in __call__
result = self._call_fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/wandb/sdk/internal/internal_api.py", line 393, in execute
return self.client.execute(*args, **kwargs) # type: ignore
File "/usr/local/lib/python3.10/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 52, in execute
result = self._get_result(document, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 60, in _get_result
return self.transport.execute(document, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/wandb/sdk/lib/gql_request.py", line 58, in execute
request = self.session.post(self.url, **post_args)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 700, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.wandb.ai', port=443): Max retries exceeded with url: /graphql (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7f46d5f21510>: Failed to resolve 'api.wandb.ai' ([Errno -3] Temporary failure in name resolution)"))
2024-12-16 10:38:41,418 ERROR MainThread:5226 [wandb_init.py:init():849] encountered error: Run initialization has timed out after 90.0 sec. Please try increasing the timeout with the `init_timeout` setting: `wandb.init(settings=wandb.Settings(init_timeout=120))`.
2024-12-16 10:38:41,418 ERROR MainThread:5226 [wandb_init.py:init():1308] error in wandb.init()
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1298, in init
return wi.init()
File "/usr/local/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 855, in init
raise error
wandb.errors.errors.CommError: Run initialization has timed out after 90.0 sec. Please try increasing the timeout with the `init_timeout` setting: `wandb.init(settings=wandb.Settings(init_timeout=120))`.
2024-12-16 10:38:41,423 WARNING MsgRouterThr:5226 [router.py:message_loop():75] message_loop has been closed
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment