2024-04-02 20:06:34.521652: E external/xla/xla/stream_executor/plugin_registry.cc:90] Invalid plugin kind specified: DNN
INFO:jax._src.xla_bridge:Unable to initialize backend 'cuda':
INFO:jax._src.xla_bridge:Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory
INFO:rank:Initializing mesh for self.local_mesh_config=(1, 8) self.between_hosts_config=(1, 1)...
INFO:rank:Detected 8 devices in mesh
2024-04-02 20:06:38.881608: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module jit_convert_element_type:
2024-04-02 20:06:38.881674: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:06:38.881683: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:06:38.881689: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:06:38.881695: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:06:38.881700: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:06:38.881705: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:06:38.881710: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:06:38.881716: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:06:38.881721: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:06:38.881725: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:06:38.881730: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:06:38.881735: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:06:38.881740: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:06:38.881745: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:06:38.882966: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:06:38.882973: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:06:38.882978: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:06:38.882983: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:06:38.882989: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:06:38.882994: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:06:38.882999: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:06:38.883004: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:06:38.883009: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:06:38.883014: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:06:38.883019: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:06:38.883028: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:06:38.884206: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:06:38.884214: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:06:38.884219: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:06:38.884224: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:06:38.884241: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:06:38.884246: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:06:38.884251: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:06:38.884256: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:06:38.884273: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:06:38.884280: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:06:38.885725: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:06:38.885742: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:06:38.885751: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:06:38.887291: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:06:38.887308: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:06:38.887318: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:06:38.887326: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:06:38.887335: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:06:38.888307: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:06:38.888318: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:06:38.888324: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:06:38.888329: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:06:38.888334: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:06:38.888339: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:06:38.888344: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:06:38.888349: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:06:38.888355: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:06:38.888359: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:06:38.889743: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:06:38.889751: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:06:38.889756: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:06:38.889761: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:06:38.889766: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:06:38.889771: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:06:38.889776: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:06:38.889781: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:06:38.889786: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:06:38.889797: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-04-02 20:06:43.779722: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module jit__threefry_seed:
2024-04-02 20:06:43.779803: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:06:43.779812: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:06:43.779819: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:06:43.779825: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:06:43.779830: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:06:43.779836: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:06:43.779841: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:06:43.779846: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:06:43.779851: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:06:43.779856: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:06:43.779861: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:06:43.779865: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:06:43.779870: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:06:43.779875: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:06:43.779880: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:06:43.779885: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:06:43.779890: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:06:43.779894: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:06:43.779899: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:06:43.779904: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:06:43.779909: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:06:43.779914: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:06:43.779918: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:06:43.779923: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:06:43.779928: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:06:43.779933: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:06:43.779938: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:06:43.779943: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:06:43.779949: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:06:43.782200: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:06:43.782209: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:06:43.782214: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:06:43.782225: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:06:43.782230: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:06:43.782235: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:06:43.782240: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:06:43.782245: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:06:43.782249: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:06:43.782254: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:06:43.782266: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:06:43.782273: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:06:43.783512: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:06:43.783518: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:06:43.783523: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:06:43.783528: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:06:43.783532: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:06:43.783537: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:06:43.783542: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:06:43.783547: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:06:43.783552: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:06:43.783557: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:06:43.783562: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:06:43.783568: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:06:43.784956: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:06:43.784963: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:06:43.784968: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:06:43.784973: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_detailed_logging: true
2024-04-02 20:06:43.784978: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:06:43.784983: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:06:43.784988: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:06:43.784993: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:06:43.784998: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:06:43.785003: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:06:43.785008: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:06:43.785013: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
INFO:rank:partition rules: <bound method LanguageModelConfig.partition_rules of LanguageModelConfig(model=TransformerConfig(emb_size=6144, key_size=128, num_q_heads=48, num_kv_heads=8, num_layers=64, vocab_size=131072, widening_factor=8, attn_output_multiplier=0.08838834764831845, name=None, num_experts=8, capacity_factor=1.0, num_selected_experts=2, init_scale=1.0, shard_activations=True, data_axis='data', model_axis='model'), vocab_size=131072, pad_token=0, eos_token=2, sequence_len=8192, model_size=6144, embedding_init_scale=1.0, embedding_multiplier_scale=78.38367176906169, output_multiplier_scale=0.5773502691896257, name=None, fprop_dtype=<class 'jax.numpy.bfloat16'>, model_type=None, init_scale_override=None, shard_embeddings=True)>
INFO:rank:(1, 256, 6144)
INFO:rank:(1, 256, 131072)
INFO:rank:State sharding type: <class 'model.TrainingState'>
INFO:rank:(1, 256, 6144)
INFO:rank:(1, 256, 131072)
INFO:rank:Loading checkpoint at ./checkpoints/ckpt-0
2024-04-02 20:16:02.345285: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module jit_broadcast_in_dim:
2024-04-02 20:16:02.345369: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:16:02.345378: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:16:02.345386: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:16:02.345393: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:16:02.345399: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:16:02.345404: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:16:02.345410: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:16:02.345416: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:16:02.345422: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:16:02.345428: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:16:02.345434: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:16:02.345439: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:16:02.345445: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:16:02.345451: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:16:02.345457: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:16:02.345462: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:16:02.345467: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:16:02.345473: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:16:02.345478: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:16:02.345483: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:16:02.345488: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:16:02.345494: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:16:02.345499: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:16:02.345504: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:16:02.345508: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:16:02.345513: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:16:02.345525: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:16:02.345530: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:16:02.345536: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:16:02.345542: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:16:02.345547: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:16:02.345551: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:16:02.345556: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:16:02.345561: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:16:02.345567: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:16:02.345572: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:16:02.345577: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:16:02.345582: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:16:02.345587: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:16:02.345592: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:16:02.345597: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:16:02.345603: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:16:02.345607: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:16:02.345612: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:16:02.345617: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:16:02.345622: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:16:02.345630: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:16:02.345638: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:16:02.345646: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:16:02.345654: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:16:02.345660: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:16:02.345665: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:16:02.345677: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:16:02.345683: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:16:02.345688: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:16:02.345692: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:16:02.345698: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:16:02.345702: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:16:02.345707: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:16:02.345717: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:16:02.345723: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:16:02.345729: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:16:02.345735: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:16:02.345740: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
INFO:rank:(1, 8192, 6144)
INFO:rank:(1, 8192, 131072)
2024-04-02 20:16:11.323926: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module jit__threefry_split:
2024-04-02 20:16:11.323992: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:16:11.324001: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:16:11.324007: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:16:11.324013: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:16:11.324018: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:16:11.324023: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:16:11.324028: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:16:11.324033: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:16:11.324038: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:16:11.324044: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:16:11.324049: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:16:11.324054: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:16:11.324059: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:16:11.324064: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:16:11.324069: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:16:11.324074: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:16:11.324079: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:16:11.324084: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:16:11.324089: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:16:11.324094: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:16:11.324099: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:16:11.324104: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:16:11.324109: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:16:11.324114: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:16:11.324119: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:16:11.324124: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:16:11.326912: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:16:11.326919: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:16:11.326932: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:16:11.326938: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:16:11.326943: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:16:11.326948: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:16:11.326953: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:16:11.326957: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:16:11.326962: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:16:11.326967: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:16:11.326972: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:16:11.326977: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:16:11.327978: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:16:11.327984: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:16:11.327989: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:16:11.327994: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:16:11.327999: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:16:11.328005: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:16:11.328010: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:16:11.328015: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:16:11.328020: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:16:11.328025: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:16:11.328030: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:16:11.329152: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:16:11.329158: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:16:11.329163: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:16:11.329168: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:16:11.329173: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:16:11.329179: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:16:11.329184: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:16:11.329189: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_detailed_logging: true
2024-04-02 20:16:11.329194: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:16:11.329199: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:16:11.329204: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:16:11.329210: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:16:11.330869: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:16:11.330876: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:16:11.330881: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:16:11.330886: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-04-02 20:16:11.776457: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module pjit_apply_fn:
2024-04-02 20:16:11.776524: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:16:11.776533: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:16:11.776539: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:16:11.776545: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:16:11.776551: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:16:11.776556: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:16:11.776561: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:16:11.776566: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:16:11.776571: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:16:11.776576: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:16:11.776581: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:16:11.776586: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:16:11.776591: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:16:11.776595: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:16:11.776600: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:16:11.776605: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:16:11.776610: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:16:11.776615: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:16:11.776620: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:16:11.776625: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:16:11.776629: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:16:11.776634: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:16:11.776639: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:16:11.776644: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:16:11.780464: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:16:11.780471: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:16:11.780477: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:16:11.780482: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:16:11.780487: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:16:11.780492: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:16:11.780503: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:16:11.780508: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:16:11.780513: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:16:11.780518: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:16:11.780522: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:16:11.780527: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:16:11.780532: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:16:11.782305: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:16:11.782311: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:16:11.782316: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:16:11.782321: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:16:11.782326: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:16:11.782331: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:16:11.782336: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:16:11.782341: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:16:11.782346: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:16:11.782351: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:16:11.782356: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:16:11.782361: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:16:11.788516: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:16:11.788522: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:16:11.788528: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:16:11.788532: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:16:11.788537: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:16:11.788542: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:16:11.788547: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:16:11.788553: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_detailed_logging: true
2024-04-02 20:16:11.788558: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:16:11.788563: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:16:11.788567: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:16:11.788572: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:16:11.788579: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:16:11.789651: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:16:11.789663: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:16:11.789668: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
INFO:runners:Precompile 1024
INFO:rank:(1, 1, 6144)
INFO:rank:(1, 1, 131072)
2024-04-02 20:16:37.688908: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module pjit__unnamed_wrapped_function_:
2024-04-02 20:16:37.688982: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:16:37.688992: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:16:37.688998: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:16:37.689004: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:16:37.689010: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:16:37.689015: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:16:37.689020: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:16:37.689025: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:16:37.689030: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:16:37.689034: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:16:37.689039: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:16:37.689044: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:16:37.689049: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:16:37.689054: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:16:37.689059: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:16:37.689064: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:16:37.689069: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:16:37.689074: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:16:37.690290: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:16:37.690297: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:16:37.690302: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:16:37.690307: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:16:37.690312: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:16:37.690317: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:16:37.690322: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:16:37.690327: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:16:37.690332: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:16:37.690337: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:16:37.690342: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:16:37.690347: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:16:37.690364: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:16:37.690371: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:16:37.690376: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:16:37.690381: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:16:37.690386: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:16:37.690390: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:16:37.690395: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:16:37.690400: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:16:37.690405: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:16:37.690409: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:16:37.690414: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:16:37.690419: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:16:37.690424: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:16:37.690429: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:16:37.690434: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:16:37.690438: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:16:37.690443: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:16:37.690448: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:16:37.690453: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:16:37.690457: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:16:37.690462: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:16:37.690467: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:16:37.690472: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:16:37.690477: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:16:37.690482: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:16:37.690486: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:16:37.690491: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_detailed_logging: true
2024-04-02 20:16:37.690496: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:16:37.690501: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:16:37.690505: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:16:37.690510: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:16:37.690515: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:16:37.690519: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:16:37.690524: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:16:37.690539: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-04-02 20:18:16.404824: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:18:18.721123: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:18:20.185634: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:18:20.185783: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
INFO:runners:Compiling...
INFO:rank:(1, 1, 6144)
INFO:rank:(1, 1, 131072)
2024-04-02 20:19:38.707084: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module pjit_apply_fn:
2024-04-02 20:19:38.707204: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:19:38.707213: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:19:38.707219: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:19:38.707225: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:19:38.707230: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:19:38.707235: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:19:38.707240: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:19:38.707245: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:19:38.707250: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:19:38.707255: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:19:38.707266: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:19:38.707272: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:19:38.707277: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:19:38.707281: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:19:38.707286: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:19:38.707291: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:19:38.707296: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:19:38.707301: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:19:38.707311: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:19:38.707316: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:19:38.707321: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:19:38.707326: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:19:38.708908: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:19:38.708917: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:19:38.708922: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:19:38.708927: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:19:38.708942: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:19:38.708948: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:19:38.708952: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:19:38.708957: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:19:38.708963: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:19:38.708968: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:19:38.708974: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:19:38.710184: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:19:38.710191: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:19:38.710197: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:19:38.710202: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:19:38.710207: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:19:38.710212: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:19:38.710217: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:19:38.710222: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:19:38.710227: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:19:38.710232: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:19:38.710239: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:19:38.711294: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:19:38.711301: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:19:38.711306: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:19:38.711311: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:19:38.711315: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:19:38.711320: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:19:38.711325: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:19:38.711330: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:19:38.711335: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:19:38.711339: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:19:38.711345: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:19:38.711350: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:19:38.711356: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_detailed_logging: true
2024-04-02 20:19:38.712443: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:19:38.712450: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:19:38.712455: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:19:38.712465: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:19:38.712469: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:19:38.712474: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:19:38.712479: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:19:38.712484: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
INFO:runners:Done compiling.
2024-04-02 20:24:05.210361: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:24:05.606087: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:24:05.609344: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:24:05.634597: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:24:05.868559: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:24:05.871576: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:24:15.149612: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:24:15.149871: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:24:15.151427: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:24:15.153198: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:24:15.154609: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:24:15.155907: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:25:18.110561: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:25:18.124629: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:25:18.131321: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:25:18.256010: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:25:18.275243: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:25:18.437802: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:25:18.466272: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-04-02 20:25:23.385495: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:25:23.385636: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:25:23.385793: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:25:23.387155: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:25:23.388295: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:25:23.388400: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:25:23.389957: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-04-02 20:30:06.824387: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module jit_convert_element_type:
2024-04-02 20:30:06.824465: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:30:06.824477: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:30:06.824486: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:30:06.824495: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:30:06.824501: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:30:06.824508: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:30:06.824513: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:30:06.824521: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:30:06.824530: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:30:06.824549: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:30:06.824557: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:30:06.824565: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:30:06.824573: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:30:06.824581: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:30:06.824590: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:30:06.824596: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:30:06.824602: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:30:06.824608: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:30:06.824613: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:30:06.824620: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:30:06.824626: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:30:06.824633: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:30:06.824638: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:30:06.824643: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:30:06.824647: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:30:06.824655: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:30:06.824664: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:30:06.824670: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:30:06.824677: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:30:06.824685: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:30:06.824700: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:30:06.824706: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:30:06.824711: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:30:06.824718: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:30:06.824725: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:30:06.824732: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:30:06.824738: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:30:06.824742: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:30:06.824748: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:30:06.824756: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:30:06.824762: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:30:06.824767: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:30:06.824771: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:30:06.824776: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:30:06.824781: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:30:06.824786: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:30:06.824795: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:30:06.824801: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:30:06.824808: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:30:06.824814: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:30:06.824819: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:30:06.824826: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:30:06.824831: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:30:06.824838: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:30:06.824844: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:30:06.824849: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:30:06.824856: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:30:06.824861: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:30:06.824866: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:30:06.824872: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:30:06.824880: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:30:06.824887: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:30:06.824894: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:30:06.824904: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-04-02 20:30:06.870851: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module jit_broadcast_in_dim:
2024-04-02 20:30:06.870932: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:30:06.870941: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:30:06.870950: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:30:06.870955: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:30:06.870960: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:30:06.870966: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:30:06.870970: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:30:06.870976: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:30:06.870981: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:30:06.870986: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:30:06.870991: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:30:06.870996: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:30:06.871001: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:30:06.871006: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:30:06.873138: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:30:06.873146: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:30:06.873153: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:30:06.873158: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:30:06.873164: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:30:06.873169: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:30:06.873174: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:30:06.873179: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:30:06.873184: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:30:06.873190: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:30:06.873196: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:30:06.875107: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:30:06.875114: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:30:06.875119: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:30:06.875124: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:30:06.875130: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:30:06.875135: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:30:06.875140: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:30:06.875157: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:30:06.875162: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:30:06.875170: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:30:06.878198: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:30:06.878207: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:30:06.878213: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:30:06.878219: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:30:06.878224: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:30:06.878229: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:30:06.878234: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:30:06.878240: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:30:06.878244: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:30:06.878249: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:30:06.878255: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:30:06.878266: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:30:06.878271: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:30:06.878276: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:30:06.878281: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:30:06.880486: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:30:06.880492: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:30:06.880497: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:30:06.880503: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:30:06.880509: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:30:06.880513: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:30:06.880519: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:30:06.880523: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:30:06.880529: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:30:06.880535: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:30:06.880540: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:30:06.880545: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:30:06.880551: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:30:06.881900: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-04-02 20:30:07.020453: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module jit__squeeze:
2024-04-02 20:30:07.020526: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:30:07.020542: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:30:07.020549: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:30:07.020554: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:30:07.020559: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:30:07.020564: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:30:07.020569: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:30:07.020574: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:30:07.020579: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:30:07.020584: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:30:07.020589: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:30:07.020594: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:30:07.020599: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:30:07.020603: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:30:07.020608: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:30:07.020613: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:30:07.020621: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:30:07.023268: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:30:07.023276: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:30:07.023281: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:30:07.023286: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:30:07.023291: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:30:07.023296: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:30:07.023300: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:30:07.023305: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:30:07.023310: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:30:07.023316: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:30:07.023321: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:30:07.023325: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:30:07.023330: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:30:07.023334: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:30:07.023339: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:30:07.023344: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:30:07.023351: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:30:07.024915: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:30:07.024927: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:30:07.024932: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:30:07.024937: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:30:07.024942: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:30:07.024947: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:30:07.024951: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:30:07.024956: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:30:07.024961: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:30:07.024965: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:30:07.024970: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:30:07.024975: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:30:07.024980: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:30:07.024984: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:30:07.024990: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:30:07.024994: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:30:07.024999: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:30:07.025004: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:30:07.025009: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:30:07.025013: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:30:07.025020: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:30:07.026867: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:30:07.026875: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:30:07.026880: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:30:07.026885: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:30:07.026890: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:30:07.026895: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:30:07.026900: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:30:07.026905: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:30:07.026910: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-04-02 20:30:07.065030: W external/xla/xla/service/gpu/gpu_compiler.cc:549] GpuCompilationEnvironment of hlo_module jit_scatter:
2024-04-02 20:30:07.065103: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_backend_optimization_level: 3
2024-04-02 20:30:07.065111: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_eliminate_hlo_implicit_broadcast: true
2024-04-02 20:30:07.065118: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_multi_thread_eigen: true
2024-04-02 20:30:07.065133: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-04-02 20:30:07.065138: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_alias_scope_metadata: true
2024-04-02 20:30:07.065143: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_noalias_metadata: true
2024-04-02 20:30:07.065148: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_llvm_enable_invariant_load_metadata: true
2024-04-02 20:30:07.065153: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_force_host_platform_device_count: 1
2024-04-02 20:30:07.065158: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_nans: true
2024-04-02 20:30:07.065162: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_infs: true
2024-04-02 20:30:07.065167: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_allow_excess_precision: true
2024-04-02 20:30:07.065172: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_autotune_level: 4
2024-04-02 20:30:07.065177: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_division: true
2024-04-02 20:30:07.065187: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_fast_math_honor_functions: true
2024-04-02 20:30:07.067717: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_max_hlo_modules: -1
2024-04-02 20:30:07.067726: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_multiheap_size_constraint_per_heap: -1
2024-04-02 20:30:07.067731: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_async_all_reduce: true
2024-04-02 20:30:07.067736: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_strict_conv_algorithm_picker: true
2024-04-02 20:30:07.067741: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-04-02 20:30:07.067746: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_frontend: true
2024-04-02 20:30:07.067750: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_nccl_termination_timeout_seconds: -1
2024-04-02 20:30:07.067755: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_shared_constants: true
2024-04-02 20:30:07.067760: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-04-02 20:30:07.067764: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_simplify_all_fp_conversions: true
2024-04-02 20:30:07.067772: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_xla_runtime_executable: true
2024-04-02 20:30:07.069614: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_shape_checks: RUNTIME
2024-04-02 20:30:07.069621: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_normalize_layouts: true
2024-04-02 20:30:07.069627: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-04-02 20:30:07.069632: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_dump_enable_mlir_pretty_form: true
2024-04-02 20:30:07.069637: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_triton_gemm: true
2024-04-02 20:30:07.069642: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-04-02 20:30:07.069648: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_experimental_deallocation: true
2024-04-02 20:30:07.069653: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_enable_mlir_fusion_outlining: true
2024-04-02 20:30:07.069658: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_m_dim: 8
2024-04-02 20:30:07.069663: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_n_dim: 8
2024-04-02 20:30:07.069668: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_cpu_matmul_tiling_k_dim: 8
2024-04-02 20:30:07.069673: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_num_runs_to_instantiate: -1
2024-04-02 20:30:07.069678: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-04-02 20:30:07.072042: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_inflation_factor: 1
2024-04-02 20:30:07.072048: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_min_graph_size: 5
2024-04-02 20:30:07.072053: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reassociation_for_converted_ar: true
2024-04-02 20:30:07.072058: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-04-02 20:30:07.072063: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-04-02 20:30:07.072068: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_highest_priority_async_stream: true
2024-04-02 20:30:07.072072: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-04-02 20:30:07.072077: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_redzone_padding_bytes: 8388608
2024-04-02 20:30:07.072081: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_triton_fusion_level: 2
2024-04-02 20:30:07.072086: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_graph_eviction_timeout_seconds: 60
2024-04-02 20:30:07.072091: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_gpu2_hal: true
2024-04-02 20:30:07.072095: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_copy_insertion_use_region_analysis: true
2024-04-02 20:30:07.072104: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-04-02 20:30:07.073627: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_split_k_autotuning: true
2024-04-02 20:30:07.073634: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduction_epilogue_fusion: true
2024-04-02 20:30:07.073639: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_cublas_fallback: true
2024-04-02 20:30:07.073644: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-04-02 20:30:07.073649: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_debug_buffer_assignment_show_max: 15
2024-04-02 20:30:07.073654: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_enable_dumping: true
2024-04-02 20:30:07.073659: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_all_gather_combine_by_dim: true
2024-04-02 20:30:07.073664: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-04-02 20:30:07.073669: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: FUSION
2024-04-02 20:30:07.073674: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_command_buffer: CUBLAS
2024-04-02 20:30:07.073679: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_enable_cub_radix_sort: true
2024-04-02 20:30:07.073684: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_memory_limit_slop_factor: 95
2024-04-02 20:30:07.073690: W external/xla/xla/service/gpu/gpu_compiler.cc:549] xla_gpu_threshold_for_windowed_einsum_mib: 100000
infer time： 4.5299530029296875e-06 秒
Output for prompt: The answer to life the universe and everything is of course 42.

But what is the answer to the question of how to get a job in the games industry?

Well, it’s not 42.

It’s not even 42000.

It’s actually 420000.

That’s the number of people who applied for jobs at EA last year.

And that’s just EA.

So how do you get a job in the games industry?