Skip to content

GitLab

Commit aaebe7b6 authored by Rayyyyy's avatar Rayyyyy
Browse files

Update dtk 2404

parent 2dc606a5
Hide whitespace changes
Inline Side-by-side
Showing with 769 additions and 791 deletions
README.md
View file @ aaebe7b6
...@@ -90,8 +90,7 @@ python run.py ...@@ -90,8 +90,7 @@ python run.py
``` ```
## result ## result
日志信息可以参考**log.txt**文件 日志信息可以参考**run.log**文件，测试结果如下图所示。
<div align=center> <div align=center>
<img src="./doc/end.png"/> <img src="./doc/end.png"/>
</div> </div>
...@@ -122,3 +121,4 @@ huggingface-cli download xai-org/grok-1 --repo-type model --include ckpt-0/* --l ...@@ -122,3 +121,4 @@ huggingface-cli download xai-org/grok-1 --repo-type model --include ckpt-0/* --l
## 参考资料 ## 参考资料
- https://github.com/xai-org/grok-1 - https://github.com/xai-org/grok-1
![Alt text](image.png)
\ No newline at end of file
doc/end.png
View replaced file @ 2dc606a5 View file @ aaebe7b6
doc/end.png

11.2 KB | W: | H:

doc/end.png

63.9 KB | W: | H:

doc/end.png
doc/end.png
doc/end.png
doc/end.png
  • 2-up
  • Swipe
  • Onion skin
log.txt deleted 100644 → 0
View file @ 2dc606a5
2024-04-02 20:06:34.521652: E external/xla/xla/stream_executor/plugin_registry.cc:90] Invalid plugin kind specified: DNN
INFO:jax._src.xla_bridge:Unable to initialize backend 'cuda':
INFO:jax._src.xla_bridge:Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory
INFO:rank:Initializing mesh for self.local_mesh_config=(1, 8) self.between_hosts_config=(1, 1)...
INFO:rank:Detected 8 devices in mesh
2024-04-02 20:06:38.881608: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module jit_convert_element_type:
2024-04-02 20:06:38.881674: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:06:38.881683: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:06:38.881689: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:06:38.881695: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:06:38.881700: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:06:38.881705: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:06:38.881710: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:06:38.881716: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:06:38.881721: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:06:38.881725: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:06:38.881730: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:06:38.881735: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:06:38.881740: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:06:38.881745: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:06:38.882966: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:06:38.882973: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:06:38.882978: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:06:38.882983: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:06:38.882989: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:06:38.882994: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:06:38.882999: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:06:38.883004: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:06:38.883009: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:06:38.883014: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:06:38.883019: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:06:38.883028: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:06:38.884206: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:06:38.884214: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:06:38.884219: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:06:38.884224: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:06:38.884241: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:06:38.884246: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:06:38.884251: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:06:38.884256: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:06:38.884273: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:06:38.884280: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:06:38.885725: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:06:38.885742: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:06:38.885751: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:06:38.887291: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:06:38.887308: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:06:38.887318: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:06:38.887326: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:06:38.887335: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:06:38.888307: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:06:38.888318: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:06:38.888324: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:06:38.888329: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:06:38.888334: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:06:38.888339: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:06:38.888344: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:06:38.888349: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:06:38.888355: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:06:38.888359: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:06:38.889743: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:06:38.889751: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:06:38.889756: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:06:38.889761: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:06:38.889766: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:06:38.889771: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:06:38.889776: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:06:38.889781: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:06:38.889786: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:06:38.889797: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-04-02 20:06:43.779722: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module jit__threefry_seed:
2024-04-02 20:06:43.779803: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:06:43.779812: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:06:43.779819: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:06:43.779825: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:06:43.779830: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:06:43.779836: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:06:43.779841: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:06:43.779846: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:06:43.779851: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:06:43.779856: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:06:43.779861: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:06:43.779865: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:06:43.779870: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:06:43.779875: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:06:43.779880: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:06:43.779885: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:06:43.779890: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:06:43.779894: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:06:43.779899: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:06:43.779904: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:06:43.779909: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:06:43.779914: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:06:43.779918: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:06:43.779923: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:06:43.779928: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:06:43.779933: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:06:43.779938: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:06:43.779943: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:06:43.779949: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:06:43.782200: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:06:43.782209: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:06:43.782214: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:06:43.782225: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:06:43.782230: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:06:43.782235: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:06:43.782240: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:06:43.782245: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:06:43.782249: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:06:43.782254: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:06:43.782266: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:06:43.782273: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:06:43.783512: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:06:43.783518: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:06:43.783523: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:06:43.783528: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:06:43.783532: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:06:43.783537: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:06:43.783542: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:06:43.783547: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:06:43.783552: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:06:43.783557: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:06:43.783562: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:06:43.783568: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:06:43.784956: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:06:43.784963: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:06:43.784968: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:06:43.784973: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_detailed_logging: true
2024-04-02 20:06:43.784978: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:06:43.784983: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:06:43.784988: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:06:43.784993: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:06:43.784998: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:06:43.785003: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:06:43.785008: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:06:43.785013: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
INFO:rank:partition rules: <bound method LanguageModelConfig.partition_rules of LanguageModelConfig(model=TransformerConfig(emb_size=6144, key_size=128, num_q_heads=48, num_kv_heads=8, num_layers=64, vocab_size=131072, widening_factor=8, attn_output_multiplier=0.08838834764831845, name=None, num_experts=8, capacity_factor=1.0, num_selected_experts=2, init_scale=1.0, shard_activations=True, data_axis='data', model_axis='model'), vocab_size=131072, pad_token=0, eos_token=2, sequence_len=8192, model_size=6144, embedding_init_scale=1.0, embedding_multiplier_scale=78.38367176906169, output_multiplier_scale=0.5773502691896257, name=None, fprop_dtype=<class 'jax.numpy.bfloat16'>, model_type=None, init_scale_override=None, shard_embeddings=True)>
INFO:rank:(1, 256, 6144)
INFO:rank:(1, 256, 131072)
INFO:rank:State sharding type: <class 'model.TrainingState'>
INFO:rank:(1, 256, 6144)
INFO:rank:(1, 256, 131072)
INFO:rank:Loading checkpoint at ./checkpoints/ckpt-0
2024-04-02 20:16:02.345285: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module jit_broadcast_in_dim:
2024-04-02 20:16:02.345369: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:16:02.345378: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:16:02.345386: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:16:02.345393: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:16:02.345399: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:16:02.345404: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:16:02.345410: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:16:02.345416: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:16:02.345422: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:16:02.345428: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:16:02.345434: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:16:02.345439: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:16:02.345445: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:16:02.345451: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:16:02.345457: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:16:02.345462: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:16:02.345467: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:16:02.345473: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:16:02.345478: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:16:02.345483: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:16:02.345488: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:16:02.345494: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:16:02.345499: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:16:02.345504: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:16:02.345508: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:16:02.345513: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:16:02.345525: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:16:02.345530: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:16:02.345536: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:16:02.345542: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:16:02.345547: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:16:02.345551: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:16:02.345556: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:16:02.345561: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:16:02.345567: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:16:02.345572: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:16:02.345577: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:16:02.345582: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:16:02.345587: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:16:02.345592: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:16:02.345597: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:16:02.345603: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:16:02.345607: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:16:02.345612: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:16:02.345617: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:16:02.345622: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:16:02.345630: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:16:02.345638: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:16:02.345646: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:16:02.345654: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:16:02.345660: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:16:02.345665: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:16:02.345677: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:16:02.345683: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:16:02.345688: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:16:02.345692: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:16:02.345698: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:16:02.345702: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:16:02.345707: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:16:02.345717: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:16:02.345723: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:16:02.345729: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:16:02.345735: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:16:02.345740: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
INFO:rank:(1, 8192, 6144)
INFO:rank:(1, 8192, 131072)
2024-04-02 20:16:11.323926: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module jit__threefry_split:
2024-04-02 20:16:11.323992: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:16:11.324001: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:16:11.324007: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:16:11.324013: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:16:11.324018: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:16:11.324023: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:16:11.324028: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:16:11.324033: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:16:11.324038: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:16:11.324044: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:16:11.324049: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:16:11.324054: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:16:11.324059: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:16:11.324064: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:16:11.324069: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:16:11.324074: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:16:11.324079: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:16:11.324084: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:16:11.324089: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:16:11.324094: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:16:11.324099: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:16:11.324104: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:16:11.324109: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:16:11.324114: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:16:11.324119: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:16:11.324124: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:16:11.326912: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:16:11.326919: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:16:11.326932: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:16:11.326938: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:16:11.326943: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:16:11.326948: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:16:11.326953: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:16:11.326957: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:16:11.326962: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:16:11.326967: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:16:11.326972: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:16:11.326977: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:16:11.327978: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:16:11.327984: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:16:11.327989: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:16:11.327994: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:16:11.327999: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:16:11.328005: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:16:11.328010: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:16:11.328015: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:16:11.328020: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:16:11.328025: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:16:11.328030: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:16:11.329152: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:16:11.329158: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:16:11.329163: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:16:11.329168: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:16:11.329173: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:16:11.329179: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:16:11.329184: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:16:11.329189: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_detailed_logging: true
2024-04-02 20:16:11.329194: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:16:11.329199: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:16:11.329204: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:16:11.329210: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:16:11.330869: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:16:11.330876: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:16:11.330881: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:16:11.330886: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-04-02 20:16:11.776457: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module pjit_apply_fn:
2024-04-02 20:16:11.776524: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:16:11.776533: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:16:11.776539: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:16:11.776545: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:16:11.776551: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:16:11.776556: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:16:11.776561: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:16:11.776566: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:16:11.776571: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:16:11.776576: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:16:11.776581: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:16:11.776586: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:16:11.776591: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:16:11.776595: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:16:11.776600: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:16:11.776605: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:16:11.776610: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:16:11.776615: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:16:11.776620: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:16:11.776625: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:16:11.776629: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:16:11.776634: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:16:11.776639: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:16:11.776644: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:16:11.780464: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:16:11.780471: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:16:11.780477: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:16:11.780482: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:16:11.780487: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:16:11.780492: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:16:11.780503: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:16:11.780508: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:16:11.780513: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:16:11.780518: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:16:11.780522: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:16:11.780527: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:16:11.780532: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:16:11.782305: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:16:11.782311: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:16:11.782316: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:16:11.782321: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:16:11.782326: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:16:11.782331: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:16:11.782336: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:16:11.782341: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:16:11.782346: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:16:11.782351: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:16:11.782356: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:16:11.782361: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:16:11.788516: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:16:11.788522: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:16:11.788528: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:16:11.788532: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:16:11.788537: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:16:11.788542: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:16:11.788547: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:16:11.788553: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_detailed_logging: true
2024-04-02 20:16:11.788558: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:16:11.788563: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:16:11.788567: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:16:11.788572: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:16:11.788579: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:16:11.789651: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:16:11.789663: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:16:11.789668: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
INFO:runners:Precompile 1024
INFO:rank:(1, 1, 6144)
INFO:rank:(1, 1, 131072)
2024-04-02 20:16:37.688908: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module pjit__unnamed_wrapped_function_:
2024-04-02 20:16:37.688982: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:16:37.688992: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:16:37.688998: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:16:37.689004: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:16:37.689010: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:16:37.689015: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:16:37.689020: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:16:37.689025: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:16:37.689030: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:16:37.689034: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:16:37.689039: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:16:37.689044: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:16:37.689049: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:16:37.689054: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:16:37.689059: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:16:37.689064: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:16:37.689069: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:16:37.689074: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:16:37.690290: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:16:37.690297: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:16:37.690302: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:16:37.690307: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:16:37.690312: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:16:37.690317: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:16:37.690322: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:16:37.690327: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:16:37.690332: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:16:37.690337: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:16:37.690342: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:16:37.690347: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:16:37.690364: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:16:37.690371: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:16:37.690376: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:16:37.690381: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:16:37.690386: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:16:37.690390: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:16:37.690395: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:16:37.690400: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:16:37.690405: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:16:37.690409: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:16:37.690414: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:16:37.690419: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:16:37.690424: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:16:37.690429: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:16:37.690434: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:16:37.690438: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:16:37.690443: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:16:37.690448: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:16:37.690453: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:16:37.690457: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:16:37.690462: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:16:37.690467: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:16:37.690472: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:16:37.690477: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:16:37.690482: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:16:37.690486: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:16:37.690491: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_detailed_logging: true
2024-04-02 20:16:37.690496: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:16:37.690501: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:16:37.690505: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:16:37.690510: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:16:37.690515: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:16:37.690519: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:16:37.690524: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:16:37.690539: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-04-02 20:18:16.404824: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:18:18.721123: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:18:20.185634: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:18:20.185783: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
INFO:runners:Compiling...
INFO:rank:(1, 1, 6144)
INFO:rank:(1, 1, 131072)
2024-04-02 20:19:38.707084: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module pjit_apply_fn:
2024-04-02 20:19:38.707204: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:19:38.707213: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:19:38.707219: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:19:38.707225: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:19:38.707230: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:19:38.707235: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:19:38.707240: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:19:38.707245: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:19:38.707250: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:19:38.707255: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:19:38.707266: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:19:38.707272: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:19:38.707277: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:19:38.707281: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:19:38.707286: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:19:38.707291: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:19:38.707296: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:19:38.707301: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:19:38.707311: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:19:38.707316: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:19:38.707321: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:19:38.707326: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:19:38.708908: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:19:38.708917: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:19:38.708922: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:19:38.708927: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:19:38.708942: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:19:38.708948: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:19:38.708952: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:19:38.708957: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:19:38.708963: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:19:38.708968: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:19:38.708974: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:19:38.710184: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:19:38.710191: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:19:38.710197: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:19:38.710202: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:19:38.710207: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:19:38.710212: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:19:38.710217: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:19:38.710222: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:19:38.710227: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:19:38.710232: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:19:38.710239: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:19:38.711294: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:19:38.711301: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:19:38.711306: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:19:38.711311: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:19:38.711315: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:19:38.711320: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:19:38.711325: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:19:38.711330: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:19:38.711335: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:19:38.711339: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:19:38.711345: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:19:38.711350: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:19:38.711356: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_detailed_logging: true
2024-04-02 20:19:38.712443: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:19:38.712450: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:19:38.712455: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:19:38.712465: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:19:38.712469: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:19:38.712474: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:19:38.712479: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:19:38.712484: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
INFO:runners:Done compiling.
2024-04-02 20:24:05.210361: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:24:05.606087: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:24:05.609344: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:24:05.634597: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:24:05.868559: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:24:05.871576: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:24:15.149612: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:24:15.149871: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:24:15.151427: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:24:15.153198: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:24:15.154609: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:24:15.155907: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:25:18.110561: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:25:18.124629: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:25:18.131321: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:25:18.256010: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:25:18.275243: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:25:18.437802: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:25:18.466272: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:25:23.385495: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:25:23.385636: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:25:23.385793: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:25:23.387155: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:25:23.388295: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:25:23.388400: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:25:23.389957: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:30:06.824387: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module jit_convert_element_type:
2024-04-02 20:30:06.824465: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:30:06.824477: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:30:06.824486: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:30:06.824495: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:30:06.824501: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:30:06.824508: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:30:06.824513: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:30:06.824521: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:30:06.824530: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:30:06.824549: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:30:06.824557: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:30:06.824565: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:30:06.824573: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:30:06.824581: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:30:06.824590: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:30:06.824596: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:30:06.824602: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:30:06.824608: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:30:06.824613: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:30:06.824620: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:30:06.824626: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:30:06.824633: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:30:06.824638: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:30:06.824643: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:30:06.824647: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:30:06.824655: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:30:06.824664: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:30:06.824670: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:30:06.824677: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:30:06.824685: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:30:06.824700: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:30:06.824706: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:30:06.824711: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:30:06.824718: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:30:06.824725: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:30:06.824732: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:30:06.824738: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:30:06.824742: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:30:06.824748: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:30:06.824756: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:30:06.824762: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:30:06.824767: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:30:06.824771: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:30:06.824776: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:30:06.824781: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:30:06.824786: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:30:06.824795: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:30:06.824801: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:30:06.824808: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:30:06.824814: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:30:06.824819: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:30:06.824826: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:30:06.824831: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:30:06.824838: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:30:06.824844: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:30:06.824849: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:30:06.824856: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:30:06.824861: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:30:06.824866: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:30:06.824872: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:30:06.824880: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:30:06.824887: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:30:06.824894: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:30:06.824904: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-04-02 20:30:06.870851: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module jit_broadcast_in_dim:
2024-04-02 20:30:06.870932: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:30:06.870941: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:30:06.870950: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:30:06.870955: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:30:06.870960: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:30:06.870966: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:30:06.870970: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:30:06.870976: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:30:06.870981: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:30:06.870986: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:30:06.870991: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:30:06.870996: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:30:06.871001: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:30:06.871006: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:30:06.873138: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:30:06.873146: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:30:06.873153: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:30:06.873158: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:30:06.873164: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:30:06.873169: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:30:06.873174: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:30:06.873179: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:30:06.873184: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:30:06.873190: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:30:06.873196: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:30:06.875107: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:30:06.875114: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:30:06.875119: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:30:06.875124: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:30:06.875130: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:30:06.875135: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:30:06.875140: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:30:06.875157: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:30:06.875162: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:30:06.875170: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:30:06.878198: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:30:06.878207: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:30:06.878213: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:30:06.878219: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:30:06.878224: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:30:06.878229: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:30:06.878234: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:30:06.878240: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:30:06.878244: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:30:06.878249: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:30:06.878255: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:30:06.878266: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:30:06.878271: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:30:06.878276: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:30:06.878281: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:30:06.880486: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:30:06.880492: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:30:06.880497: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:30:06.880503: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:30:06.880509: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:30:06.880513: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:30:06.880519: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:30:06.880523: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:30:06.880529: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:30:06.880535: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:30:06.880540: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:30:06.880545: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:30:06.880551: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:30:06.881900: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-04-02 20:30:07.020453: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module jit__squeeze:
2024-04-02 20:30:07.020526: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:30:07.020542: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:30:07.020549: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:30:07.020554: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:30:07.020559: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:30:07.020564: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:30:07.020569: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:30:07.020574: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:30:07.020579: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:30:07.020584: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:30:07.020589: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:30:07.020594: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:30:07.020599: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:30:07.020603: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:30:07.020608: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:30:07.020613: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:30:07.020621: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:30:07.023268: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:30:07.023276: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:30:07.023281: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:30:07.023286: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:30:07.023291: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:30:07.023296: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:30:07.023300: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:30:07.023305: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:30:07.023310: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:30:07.023316: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:30:07.023321: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:30:07.023325: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:30:07.023330: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:30:07.023334: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:30:07.023339: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:30:07.023344: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:30:07.023351: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:30:07.024915: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:30:07.024927: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:30:07.024932: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:30:07.024937: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:30:07.024942: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:30:07.024947: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:30:07.024951: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:30:07.024956: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:30:07.024961: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:30:07.024965: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:30:07.024970: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:30:07.024975: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:30:07.024980: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:30:07.024984: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:30:07.024990: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:30:07.024994: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:30:07.024999: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:30:07.025004: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:30:07.025009: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:30:07.025013: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:30:07.025020: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:30:07.026867: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:30:07.026875: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:30:07.026880: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:30:07.026885: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:30:07.026890: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:30:07.026895: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:30:07.026900: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:30:07.026905: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:30:07.026910: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-04-02 20:30:07.065030: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module jit_scatter:
2024-04-02 20:30:07.065103: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:30:07.065111: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:30:07.065118: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:30:07.065133: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:30:07.065138: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:30:07.065143: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:30:07.065148: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:30:07.065153: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:30:07.065158: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:30:07.065162: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:30:07.065167: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:30:07.065172: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:30:07.065177: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:30:07.065187: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:30:07.067717: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:30:07.067726: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:30:07.067731: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:30:07.067736: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:30:07.067741: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:30:07.067746: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:30:07.067750: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:30:07.067755: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:30:07.067760: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:30:07.067764: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:30:07.067772: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:30:07.069614: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:30:07.069621: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:30:07.069627: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:30:07.069632: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:30:07.069637: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:30:07.069642: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:30:07.069648: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:30:07.069653: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:30:07.069658: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:30:07.069663: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:30:07.069668: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:30:07.069673: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:30:07.069678: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:30:07.072042: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:30:07.072048: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:30:07.072053: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:30:07.072058: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:30:07.072063: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:30:07.072068: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:30:07.072072: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:30:07.072077: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:30:07.072081: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:30:07.072086: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:30:07.072091: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:30:07.072095: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:30:07.072104: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:30:07.073627: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:30:07.073634: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:30:07.073639: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:30:07.073644: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:30:07.073649: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:30:07.073654: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:30:07.073659: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:30:07.073664: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:30:07.073669: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:30:07.073674: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:30:07.073679: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:30:07.073684: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:30:07.073690: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
infer time： 4.5299530029296875e-06 秒
Output for prompt: The answer to life the universe and everything is of course 42.
But what is the answer to the question of how to get a job in the games industry?
Well, it’s not 42.
It’s not even 42000.
It’s actually 420000.
That’s the number of people who applied for jobs at EA last year.
And that’s just EA.
So how do you get a job in the games industry?
run.log 0 → 100644
View file @ aaebe7b6
2024-05-18 11:04:39.673769: E external/xla/xla/stream_executor/plugin_registry.cc:90] Invalid plugin kind specified: DNN
INFO:jax._src.xla_bridge:Unable to initialize backend 'cuda':
INFO:jax._src.xla_bridge:Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory
INFO:rank:Initializing mesh for self.local_mesh_config=(1, 8) self.between_hosts_config=(1, 1)...
INFO:rank:Detected 8 devices in mesh
2024-05-18 11:04:40.894257: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module jit_convert_element_type:
2024-05-18 11:04:40.894296: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:04:40.894302: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:04:40.894307: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:04:40.894312: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:04:40.894316: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:04:40.894321: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:04:40.894325: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:04:40.894329: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:04:40.894334: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:04:40.894338: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:04:40.894342: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:04:40.894346: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:04:40.894350: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:04:40.894354: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:04:40.894358: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:04:40.894363: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:04:40.894367: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:04:40.894372: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:04:40.894376: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:04:40.894380: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:04:40.894384: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:04:40.894388: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:04:40.894392: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:04:40.894396: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:04:40.894400: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:04:40.894404: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:04:40.894409: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:04:40.894413: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:04:40.894417: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:04:40.894421: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:04:40.894433: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:04:40.894437: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:04:40.894441: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:04:40.894445: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:04:40.894449: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:04:40.894454: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:04:40.894458: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:04:40.894462: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:04:40.894466: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:04:40.894470: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:04:40.894474: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:04:40.894479: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:04:40.894484: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:04:40.894488: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:04:40.894492: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:04:40.894496: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:04:40.894500: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:04:40.894504: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:04:40.894508: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:04:40.894513: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:04:40.894517: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:04:40.894521: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:04:40.894525: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:04:40.894529: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:04:40.894534: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:04:40.894538: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:04:40.894542: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:04:40.894546: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:04:40.894551: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:04:40.894555: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:04:40.894559: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:04:40.894563: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:04:40.894567: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:04:40.894575: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-05-18 11:04:46.281731: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module jit__threefry_seed:
2024-05-18 11:04:46.281778: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:04:46.281784: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:04:46.281789: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:04:46.281794: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:04:46.281798: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:04:46.281803: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:04:46.281808: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:04:46.281813: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:04:46.281817: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:04:46.281821: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:04:46.281825: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:04:46.281830: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:04:46.281834: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:04:46.281838: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:04:46.281842: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:04:46.281846: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:04:46.281851: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:04:46.281855: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:04:46.281860: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:04:46.281864: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:04:46.281868: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:04:46.281872: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:04:46.281876: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:04:46.281880: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:04:46.281884: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:04:46.281889: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:04:46.281893: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:04:46.281897: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:04:46.281902: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:04:46.281907: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:04:46.281911: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:04:46.281915: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:04:46.281925: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:04:46.281929: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:04:46.281934: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:04:46.281938: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:04:46.281942: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:04:46.281946: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:04:46.281951: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:04:46.281955: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:04:46.281959: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:04:46.281964: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:04:46.281968: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:04:46.281972: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:04:46.281976: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:04:46.281980: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:04:46.281984: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:04:46.281989: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:04:46.281993: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:04:46.281997: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:04:46.282002: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:04:46.282006: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:04:46.282010: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:04:46.282014: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:04:46.282018: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:04:46.282023: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:04:46.282027: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_detailed_logging: true
2024-05-18 11:04:46.282031: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:04:46.282035: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:04:46.282039: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:04:46.282043: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:04:46.282048: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:04:46.282052: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:04:46.282057: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:04:46.282061: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
INFO:rank:partition rules: <bound method LanguageModelConfig.partition_rules of LanguageModelConfig(model=TransformerConfig(emb_size=6144, key_size=128, num_q_heads=48, num_kv_heads=8, num_layers=64, vocab_size=131072, widening_factor=8, attn_output_multiplier=0.08838834764831845, name=None, num_experts=8, capacity_factor=1.0, num_selected_experts=2, init_scale=1.0, shard_activations=True, data_axis='data', model_axis='model'), vocab_size=131072, pad_token=0, eos_token=2, sequence_len=8192, model_size=6144, embedding_init_scale=1.0, embedding_multiplier_scale=78.38367176906169, output_multiplier_scale=0.5773502691896257, name=None, fprop_dtype=<class 'jax.numpy.bfloat16'>, model_type=None, init_scale_override=None, shard_embeddings=True)>
INFO:rank:(1, 256, 6144)
INFO:rank:(1, 256, 131072)
INFO:rank:State sharding type: <class 'model.TrainingState'>
INFO:rank:(1, 256, 6144)
INFO:rank:(1, 256, 131072)
INFO:rank:Loading checkpoint at ./checkpoints/ckpt-0
2024-05-18 11:13:29.395845: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module jit_broadcast_in_dim:
2024-05-18 11:13:29.395987: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:13:29.395995: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:13:29.396003: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:13:29.396010: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:13:29.396015: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:13:29.396020: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:13:29.396025: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:13:29.396031: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:13:29.396044: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:13:29.396049: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:13:29.396055: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:13:29.396060: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:13:29.396066: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:13:29.396075: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:13:29.396081: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:13:29.396091: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:13:29.396097: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:13:29.396101: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:13:29.396105: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:13:29.396111: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:13:29.396116: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:13:29.396121: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:13:29.396126: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:13:29.396130: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:13:29.396136: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:13:29.396141: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:13:29.396151: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:13:29.396156: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:13:29.396161: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:13:29.396167: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:13:29.396171: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:13:29.396176: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:13:29.396182: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:13:29.396187: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:13:29.396193: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:13:29.396198: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:13:29.396202: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:13:29.396206: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:13:29.396214: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:13:29.396221: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:13:29.396229: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:13:29.396244: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:13:29.396250: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:13:29.396256: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:13:29.396267: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:13:29.396276: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:13:29.396286: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:13:29.396305: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:13:29.396319: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:13:29.396327: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:13:29.396337: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:13:29.396348: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:13:29.396366: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:13:29.396373: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:13:29.396380: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:13:29.396384: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:13:29.396390: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:13:29.396396: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:13:29.396402: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:13:29.396412: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:13:29.396420: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:13:29.396425: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:13:29.396430: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:13:29.396434: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
INFO:rank:(1, 8192, 6144)
INFO:rank:(1, 8192, 131072)
2024-05-18 11:13:49.857423: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module jit__threefry_split:
2024-05-18 11:13:49.857468: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:13:49.857474: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:13:49.857480: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:13:49.857485: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:13:49.857490: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:13:49.857494: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:13:49.857499: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:13:49.857503: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:13:49.857508: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:13:49.857512: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:13:49.857516: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:13:49.857521: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:13:49.857526: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:13:49.857531: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:13:49.857535: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:13:49.857539: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:13:49.857544: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:13:49.857548: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:13:49.857552: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:13:49.857556: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:13:49.857561: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:13:49.857565: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:13:49.857570: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:13:49.857574: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:13:49.857578: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:13:49.857583: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:13:49.857587: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:13:49.857591: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:13:49.857603: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:13:49.857608: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:13:49.857613: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:13:49.857618: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:13:49.857622: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:13:49.857626: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:13:49.857631: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:13:49.857635: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:13:49.857639: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:13:49.857643: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:13:49.857648: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:13:49.857652: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:13:49.857656: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:13:49.857661: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:13:49.857666: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:13:49.857681: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:13:49.857685: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:13:49.857689: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:13:49.857694: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:13:49.857698: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:13:49.857702: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:13:49.857707: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:13:49.857712: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:13:49.857716: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:13:49.857720: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:13:49.857724: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:13:49.857729: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:13:49.857733: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:13:49.857737: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_detailed_logging: true
2024-05-18 11:13:49.857741: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:13:49.857745: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:13:49.857750: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:13:49.857755: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:13:49.857762: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:13:49.857767: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:13:49.857771: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:13:49.857775: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-05-18 11:13:50.351060: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module pjit_apply_fn:
2024-05-18 11:13:50.351114: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:13:50.351120: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:13:50.351125: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:13:50.351130: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:13:50.351134: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:13:50.351138: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:13:50.351143: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:13:50.351147: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:13:50.351151: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:13:50.351156: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:13:50.351161: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:13:50.351165: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:13:50.351169: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:13:50.351173: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:13:50.351177: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:13:50.351182: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:13:50.351186: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:13:50.351190: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:13:50.351194: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:13:50.351199: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:13:50.351203: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:13:50.351208: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:13:50.351213: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:13:50.351217: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:13:50.351221: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:13:50.351225: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:13:50.351229: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:13:50.351233: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:13:50.351237: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:13:50.351242: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:13:50.351251: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:13:50.351256: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:13:50.351261: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:13:50.351265: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:13:50.351269: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:13:50.351274: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:13:50.351278: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:13:50.351282: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:13:50.351286: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:13:50.351290: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:13:50.351294: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:13:50.351299: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:13:50.351303: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:13:50.351308: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:13:50.351312: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:13:50.351316: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:13:50.351320: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:13:50.351324: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:13:50.351328: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:13:50.351333: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:13:50.351337: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:13:50.351341: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:13:50.351345: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:13:50.351350: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:13:50.351355: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:13:50.351359: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:13:50.351363: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_detailed_logging: true
2024-05-18 11:13:50.351367: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:13:50.351372: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:13:50.351376: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:13:50.351380: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:13:50.351384: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:13:50.351388: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:13:50.351397: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:13:50.351403: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
INFO:runners:Precompile 1024
INFO:rank:(1, 1, 6144)
INFO:rank:(1, 1, 131072)
2024-05-18 11:14:16.143780: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module pjit__unnamed_wrapped_function_:
2024-05-18 11:14:16.143831: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:14:16.143837: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:14:16.143842: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:14:16.143847: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:14:16.143852: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:14:16.143856: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:14:16.143860: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:14:16.143864: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:14:16.143868: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:14:16.143873: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:14:16.143877: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:14:16.143881: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:14:16.143886: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:14:16.143890: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:14:16.143894: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:14:16.143898: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:14:16.143902: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:14:16.143906: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:14:16.143911: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:14:16.143915: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:14:16.143919: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:14:16.143923: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:14:16.143928: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:14:16.143932: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:14:16.143936: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:14:16.143940: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:14:16.143944: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:14:16.143948: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:14:16.143952: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:14:16.143956: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:14:16.143970: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:14:16.143974: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:14:16.143979: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:14:16.143983: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:14:16.143987: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:14:16.143991: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:14:16.143995: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:14:16.143999: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:14:16.144003: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:14:16.144008: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:14:16.144012: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:14:16.144016: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:14:16.144020: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:14:16.144025: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:14:16.144029: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:14:16.144033: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:14:16.144037: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:14:16.144041: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:14:16.144045: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:14:16.144050: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:14:16.144054: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:14:16.144058: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:14:16.144062: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:14:16.144066: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:14:16.144070: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:14:16.144074: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:14:16.144079: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_detailed_logging: true
2024-05-18 11:14:16.144083: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:14:16.144087: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:14:16.144092: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:14:16.144096: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:14:16.144100: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:14:16.144104: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:14:16.144108: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:14:16.144116: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-05-18 11:15:55.217264: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-05-18 11:15:57.751730: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-05-18 11:16:00.376632: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-05-18 11:16:01.263488: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-05-18 11:16:01.263654: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-05-18 11:16:01.263783: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
INFO:runners:Compiling...
INFO:rank:(1, 1, 6144)
INFO:rank:(1, 1, 131072)
2024-05-18 11:17:15.752139: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module pjit_apply_fn:
2024-05-18 11:17:15.752192: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:17:15.752198: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:17:15.752203: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:17:15.752208: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:17:15.752212: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:17:15.752216: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:17:15.752221: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:17:15.752225: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:17:15.752229: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:17:15.752234: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:17:15.752238: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:17:15.752242: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:17:15.752246: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:17:15.752250: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:17:15.752254: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:17:15.752258: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:17:15.752263: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:17:15.752267: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:17:15.752271: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:17:15.752275: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:17:15.752280: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:17:15.752284: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:17:15.752288: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:17:15.752292: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:17:15.752304: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:17:15.752308: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:17:15.752312: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:17:15.752317: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:17:15.752321: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:17:15.752325: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:17:15.752330: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:17:15.752334: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:17:15.752338: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:17:15.752342: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:17:15.752346: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:17:15.752350: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:17:15.752355: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:17:15.752359: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:17:15.752363: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:17:15.752367: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:17:15.752371: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:17:15.752375: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:17:15.752379: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:17:15.752384: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:17:15.752388: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:17:15.752392: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:17:15.752396: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:17:15.752400: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:17:15.752404: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:17:15.752408: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:17:15.752412: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:17:15.752417: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:17:15.752421: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:17:15.752425: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:17:15.752429: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:17:15.752433: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:17:15.752437: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_detailed_logging: true
2024-05-18 11:17:15.752444: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:17:15.752449: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:17:15.752453: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:17:15.752457: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:17:15.752461: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:17:15.752465: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:17:15.752469: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:17:15.752473: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
INFO:runners:Done compiling.
2024-05-18 11:24:31.379439: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module jit_convert_element_type:
2024-05-18 11:24:31.379494: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:24:31.379502: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:24:31.379510: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:24:31.379518: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:24:31.379523: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:24:31.379530: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:24:31.379536: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:24:31.379543: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:24:31.379551: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:24:31.379559: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:24:31.379565: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:24:31.379573: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:24:31.379580: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:24:31.379587: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:24:31.379595: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:24:31.379600: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:24:31.379606: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:24:31.379612: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:24:31.379616: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.379623: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:24:31.379629: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:24:31.379635: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:24:31.379640: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:24:31.379644: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:24:31.379649: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:24:31.379659: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:24:31.379677: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:24:31.379683: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:24:31.379690: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:24:31.379700: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:24:31.379704: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:24:31.379709: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:24:31.379714: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:24:31.379720: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:24:31.379727: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:24:31.379732: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:24:31.379738: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:24:31.379742: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:24:31.379748: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:24:31.379756: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:24:31.379762: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:24:31.379766: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.379770: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.379775: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:24:31.379779: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:24:31.379785: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:24:31.379793: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:24:31.379798: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:24:31.379805: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:24:31.379810: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:24:31.379814: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:24:31.379820: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:24:31.379827: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:24:31.379833: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:24:31.379839: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:24:31.379843: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:24:31.379850: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:24:31.379856: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:24:31.379860: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:24:31.379868: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:24:31.379874: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:24:31.379881: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:24:31.379890: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:24:31.379895: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-05-18 11:24:31.444282: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module jit_broadcast_in_dim:
2024-05-18 11:24:31.444327: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:24:31.444333: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:24:31.444338: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:24:31.444343: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:24:31.444348: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:24:31.444352: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:24:31.444357: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:24:31.444361: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:24:31.444365: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:24:31.444370: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:24:31.444375: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:24:31.444379: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:24:31.444383: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:24:31.444387: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:24:31.444391: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:24:31.444395: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:24:31.444400: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:24:31.444404: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:24:31.444409: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.444414: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:24:31.444418: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:24:31.444423: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:24:31.444427: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:24:31.444431: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:24:31.444436: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:24:31.444440: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:24:31.444445: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:24:31.444449: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:24:31.444462: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:24:31.444466: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:24:31.444470: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:24:31.444475: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:24:31.444479: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:24:31.444484: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:24:31.444488: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:24:31.444492: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:24:31.444496: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:24:31.444502: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:24:31.444506: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:24:31.444511: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:24:31.444515: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:24:31.444519: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.444524: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.444528: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:24:31.444533: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:24:31.444537: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:24:31.444542: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:24:31.444547: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:24:31.444552: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:24:31.444556: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:24:31.444560: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:24:31.444565: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:24:31.444569: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:24:31.444573: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:24:31.444578: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:24:31.444582: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:24:31.444586: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:24:31.444591: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:24:31.444596: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:24:31.444601: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:24:31.444605: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:24:31.444611: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:24:31.444616: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:24:31.444620: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-05-18 11:24:31.577404: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module jit__squeeze:
2024-05-18 11:24:31.577458: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:24:31.577464: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:24:31.577469: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:24:31.577474: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:24:31.577479: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:24:31.577483: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:24:31.577488: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:24:31.577492: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:24:31.577496: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:24:31.577500: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:24:31.577504: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:24:31.577508: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:24:31.577513: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:24:31.577517: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:24:31.577521: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:24:31.577526: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:24:31.577530: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:24:31.577534: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:24:31.577538: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.577542: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:24:31.577546: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:24:31.577550: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:24:31.577554: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:24:31.577559: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:24:31.577563: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:24:31.577567: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:24:31.577572: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:24:31.577576: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:24:31.577580: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:24:31.577584: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:24:31.577588: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:24:31.577601: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:24:31.577605: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:24:31.577610: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:24:31.577614: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:24:31.577619: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:24:31.577623: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:24:31.577627: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:24:31.577631: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:24:31.577635: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:24:31.577640: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:24:31.577644: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.577648: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.577652: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:24:31.577656: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:24:31.577661: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:24:31.577665: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:24:31.577679: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:24:31.577684: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:24:31.577688: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:24:31.577692: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:24:31.577697: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:24:31.577701: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:24:31.577705: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:24:31.577709: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:24:31.577714: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:24:31.577718: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:24:31.577722: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:24:31.577726: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:24:31.577730: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:24:31.577734: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:24:31.577738: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:24:31.577742: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:24:31.577746: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-05-18 11:24:31.629288: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module jit_scatter:
2024-05-18 11:24:31.629340: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:24:31.629346: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:24:31.629351: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:24:31.629356: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:24:31.629360: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:24:31.629364: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:24:31.629369: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:24:31.629373: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:24:31.629377: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:24:31.629381: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:24:31.629386: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:24:31.629390: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:24:31.629394: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:24:31.629398: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:24:31.629403: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:24:31.629407: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:24:31.629411: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:24:31.629416: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:24:31.629420: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.629424: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:24:31.629428: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:24:31.629432: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:24:31.629436: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:24:31.629440: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:24:31.629444: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:24:31.629448: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:24:31.629457: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:24:31.629461: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:24:31.629465: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:24:31.629469: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:24:31.629473: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:24:31.629478: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:24:31.629482: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:24:31.629499: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:24:31.629504: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:24:31.629508: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:24:31.629512: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:24:31.629516: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:24:31.629520: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:24:31.629525: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:24:31.629529: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:24:31.629533: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.629537: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.629541: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:24:31.629545: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:24:31.629550: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:24:31.629554: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:24:31.629558: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:24:31.629562: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:24:31.629566: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:24:31.629570: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:24:31.629574: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:24:31.629578: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:24:31.629582: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:24:31.629586: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:24:31.629591: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:24:31.629595: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:24:31.629599: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:24:31.629603: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:24:31.629607: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:24:31.629611: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:24:31.629616: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:24:31.629620: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:24:31.629624: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
forward ...
forward ...
infer time： 5.0067901611328125e-06 秒
Output for prompt: The answer to life the universe and everything is of course 42.
But what is the answer to the question of how to get a job in the games industry?
Well, it’s not 42.
It’s not even a number.
It’s a question.
The question is:
“What do you want to do?”
The answer to that question is the answer to the question of how to get a job in the games industry.
You see, the games industry is a very competitive place
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment