2024-05-18 11:04:39.673769: E external/xla/xla/stream_executor/plugin_registry.cc:90] Invalid plugin kind specified: DNN
INFO:jax._src.xla_bridge:Unable to initialize backend 'cuda': 
INFO:jax._src.xla_bridge:Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory
INFO:rank:Initializing mesh for self.local_mesh_config=(1, 8) self.between_hosts_config=(1, 1)...
INFO:rank:Detected 8 devices in mesh
2024-05-18 11:04:40.894257: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module jit_convert_element_type:
2024-05-18 11:04:40.894296: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:04:40.894302: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:04:40.894307: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:04:40.894312: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:04:40.894316: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:04:40.894321: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:04:40.894325: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:04:40.894329: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:04:40.894334: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:04:40.894338: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:04:40.894342: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:04:40.894346: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:04:40.894350: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:04:40.894354: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:04:40.894358: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:04:40.894363: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:04:40.894367: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:04:40.894372: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:04:40.894376: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:04:40.894380: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:04:40.894384: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:04:40.894388: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:04:40.894392: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:04:40.894396: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:04:40.894400: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:04:40.894404: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:04:40.894409: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:04:40.894413: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:04:40.894417: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:04:40.894421: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:04:40.894433: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:04:40.894437: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:04:40.894441: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:04:40.894445: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:04:40.894449: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:04:40.894454: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:04:40.894458: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:04:40.894462: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:04:40.894466: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:04:40.894470: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:04:40.894474: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:04:40.894479: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:04:40.894484: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:04:40.894488: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:04:40.894492: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:04:40.894496: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:04:40.894500: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:04:40.894504: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:04:40.894508: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:04:40.894513: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:04:40.894517: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:04:40.894521: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:04:40.894525: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:04:40.894529: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:04:40.894534: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:04:40.894538: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:04:40.894542: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:04:40.894546: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:04:40.894551: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:04:40.894555: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:04:40.894559: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:04:40.894563: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:04:40.894567: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:04:40.894575: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-05-18 11:04:46.281731: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module jit__threefry_seed:
2024-05-18 11:04:46.281778: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:04:46.281784: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:04:46.281789: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:04:46.281794: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:04:46.281798: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:04:46.281803: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:04:46.281808: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:04:46.281813: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:04:46.281817: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:04:46.281821: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:04:46.281825: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:04:46.281830: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:04:46.281834: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:04:46.281838: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:04:46.281842: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:04:46.281846: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:04:46.281851: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:04:46.281855: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:04:46.281860: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:04:46.281864: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:04:46.281868: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:04:46.281872: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:04:46.281876: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:04:46.281880: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:04:46.281884: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:04:46.281889: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:04:46.281893: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:04:46.281897: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:04:46.281902: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:04:46.281907: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:04:46.281911: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:04:46.281915: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:04:46.281925: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:04:46.281929: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:04:46.281934: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:04:46.281938: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:04:46.281942: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:04:46.281946: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:04:46.281951: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:04:46.281955: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:04:46.281959: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:04:46.281964: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:04:46.281968: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:04:46.281972: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:04:46.281976: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:04:46.281980: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:04:46.281984: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:04:46.281989: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:04:46.281993: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:04:46.281997: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:04:46.282002: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:04:46.282006: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:04:46.282010: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:04:46.282014: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:04:46.282018: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:04:46.282023: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:04:46.282027: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_detailed_logging: true
2024-05-18 11:04:46.282031: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:04:46.282035: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:04:46.282039: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:04:46.282043: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:04:46.282048: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:04:46.282052: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:04:46.282057: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:04:46.282061: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
INFO:rank:partition rules: <bound method LanguageModelConfig.partition_rules of LanguageModelConfig(model=TransformerConfig(emb_size=6144, key_size=128, num_q_heads=48, num_kv_heads=8, num_layers=64, vocab_size=131072, widening_factor=8, attn_output_multiplier=0.08838834764831845, name=None, num_experts=8, capacity_factor=1.0, num_selected_experts=2, init_scale=1.0, shard_activations=True, data_axis='data', model_axis='model'), vocab_size=131072, pad_token=0, eos_token=2, sequence_len=8192, model_size=6144, embedding_init_scale=1.0, embedding_multiplier_scale=78.38367176906169, output_multiplier_scale=0.5773502691896257, name=None, fprop_dtype=<class 'jax.numpy.bfloat16'>, model_type=None, init_scale_override=None, shard_embeddings=True)>
INFO:rank:(1, 256, 6144)
INFO:rank:(1, 256, 131072)
INFO:rank:State sharding type: <class 'model.TrainingState'>
INFO:rank:(1, 256, 6144)
INFO:rank:(1, 256, 131072)
INFO:rank:Loading checkpoint at ./checkpoints/ckpt-0
2024-05-18 11:13:29.395845: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module jit_broadcast_in_dim:
2024-05-18 11:13:29.395987: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:13:29.395995: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:13:29.396003: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:13:29.396010: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:13:29.396015: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:13:29.396020: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:13:29.396025: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:13:29.396031: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:13:29.396044: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:13:29.396049: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:13:29.396055: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:13:29.396060: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:13:29.396066: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:13:29.396075: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:13:29.396081: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:13:29.396091: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:13:29.396097: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:13:29.396101: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:13:29.396105: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:13:29.396111: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:13:29.396116: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:13:29.396121: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:13:29.396126: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:13:29.396130: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:13:29.396136: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:13:29.396141: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:13:29.396151: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:13:29.396156: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:13:29.396161: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:13:29.396167: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:13:29.396171: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:13:29.396176: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:13:29.396182: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:13:29.396187: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:13:29.396193: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:13:29.396198: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:13:29.396202: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:13:29.396206: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:13:29.396214: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:13:29.396221: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:13:29.396229: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:13:29.396244: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:13:29.396250: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:13:29.396256: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:13:29.396267: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:13:29.396276: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:13:29.396286: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:13:29.396305: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:13:29.396319: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:13:29.396327: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:13:29.396337: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:13:29.396348: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:13:29.396366: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:13:29.396373: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:13:29.396380: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:13:29.396384: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:13:29.396390: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:13:29.396396: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:13:29.396402: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:13:29.396412: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:13:29.396420: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:13:29.396425: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:13:29.396430: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:13:29.396434: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
INFO:rank:(1, 8192, 6144)
INFO:rank:(1, 8192, 131072)
2024-05-18 11:13:49.857423: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module jit__threefry_split:
2024-05-18 11:13:49.857468: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:13:49.857474: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:13:49.857480: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:13:49.857485: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:13:49.857490: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:13:49.857494: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:13:49.857499: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:13:49.857503: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:13:49.857508: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:13:49.857512: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:13:49.857516: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:13:49.857521: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:13:49.857526: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:13:49.857531: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:13:49.857535: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:13:49.857539: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:13:49.857544: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:13:49.857548: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:13:49.857552: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:13:49.857556: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:13:49.857561: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:13:49.857565: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:13:49.857570: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:13:49.857574: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:13:49.857578: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:13:49.857583: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:13:49.857587: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:13:49.857591: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:13:49.857603: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:13:49.857608: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:13:49.857613: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:13:49.857618: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:13:49.857622: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:13:49.857626: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:13:49.857631: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:13:49.857635: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:13:49.857639: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:13:49.857643: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:13:49.857648: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:13:49.857652: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:13:49.857656: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:13:49.857661: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:13:49.857666: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:13:49.857681: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:13:49.857685: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:13:49.857689: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:13:49.857694: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:13:49.857698: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:13:49.857702: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:13:49.857707: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:13:49.857712: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:13:49.857716: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:13:49.857720: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:13:49.857724: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:13:49.857729: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:13:49.857733: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:13:49.857737: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_detailed_logging: true
2024-05-18 11:13:49.857741: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:13:49.857745: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:13:49.857750: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:13:49.857755: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:13:49.857762: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:13:49.857767: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:13:49.857771: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:13:49.857775: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-05-18 11:13:50.351060: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module pjit_apply_fn:
2024-05-18 11:13:50.351114: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:13:50.351120: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:13:50.351125: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:13:50.351130: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:13:50.351134: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:13:50.351138: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:13:50.351143: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:13:50.351147: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:13:50.351151: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:13:50.351156: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:13:50.351161: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:13:50.351165: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:13:50.351169: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:13:50.351173: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:13:50.351177: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:13:50.351182: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:13:50.351186: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:13:50.351190: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:13:50.351194: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:13:50.351199: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:13:50.351203: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:13:50.351208: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:13:50.351213: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:13:50.351217: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:13:50.351221: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:13:50.351225: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:13:50.351229: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:13:50.351233: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:13:50.351237: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:13:50.351242: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:13:50.351251: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:13:50.351256: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:13:50.351261: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:13:50.351265: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:13:50.351269: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:13:50.351274: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:13:50.351278: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:13:50.351282: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:13:50.351286: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:13:50.351290: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:13:50.351294: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:13:50.351299: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:13:50.351303: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:13:50.351308: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:13:50.351312: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:13:50.351316: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:13:50.351320: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:13:50.351324: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:13:50.351328: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:13:50.351333: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:13:50.351337: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:13:50.351341: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:13:50.351345: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:13:50.351350: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:13:50.351355: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:13:50.351359: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:13:50.351363: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_detailed_logging: true
2024-05-18 11:13:50.351367: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:13:50.351372: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:13:50.351376: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:13:50.351380: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:13:50.351384: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:13:50.351388: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:13:50.351397: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:13:50.351403: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
INFO:runners:Precompile 1024
INFO:rank:(1, 1, 6144)
INFO:rank:(1, 1, 131072)
2024-05-18 11:14:16.143780: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module pjit__unnamed_wrapped_function_:
2024-05-18 11:14:16.143831: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:14:16.143837: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:14:16.143842: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:14:16.143847: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:14:16.143852: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:14:16.143856: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:14:16.143860: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:14:16.143864: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:14:16.143868: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:14:16.143873: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:14:16.143877: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:14:16.143881: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:14:16.143886: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:14:16.143890: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:14:16.143894: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:14:16.143898: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:14:16.143902: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:14:16.143906: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:14:16.143911: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:14:16.143915: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:14:16.143919: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:14:16.143923: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:14:16.143928: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:14:16.143932: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:14:16.143936: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:14:16.143940: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:14:16.143944: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:14:16.143948: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:14:16.143952: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:14:16.143956: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:14:16.143970: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:14:16.143974: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:14:16.143979: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:14:16.143983: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:14:16.143987: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:14:16.143991: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:14:16.143995: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:14:16.143999: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:14:16.144003: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:14:16.144008: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:14:16.144012: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:14:16.144016: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:14:16.144020: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:14:16.144025: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:14:16.144029: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:14:16.144033: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:14:16.144037: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:14:16.144041: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:14:16.144045: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:14:16.144050: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:14:16.144054: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:14:16.144058: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:14:16.144062: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:14:16.144066: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:14:16.144070: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:14:16.144074: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:14:16.144079: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_detailed_logging: true
2024-05-18 11:14:16.144083: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:14:16.144087: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:14:16.144092: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:14:16.144096: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:14:16.144100: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:14:16.144104: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:14:16.144108: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:14:16.144116: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-05-18 11:15:55.217264: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-05-18 11:15:57.751730: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-05-18 11:16:00.376632: E external/xla/xla/service/rendezvous.cc:31] This thread has been waiting for 10 seconds and may be stuck:
2024-05-18 11:16:01.263488: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-05-18 11:16:01.263654: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
2024-05-18 11:16:01.263783: E external/xla/xla/service/rendezvous.cc:36] Thread is unstuck! Warning above was a false-positive. Perhaps the timeout is too short.
INFO:runners:Compiling...
INFO:rank:(1, 1, 6144)
INFO:rank:(1, 1, 131072)
2024-05-18 11:17:15.752139: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module pjit_apply_fn:
2024-05-18 11:17:15.752192: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:17:15.752198: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:17:15.752203: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:17:15.752208: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:17:15.752212: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:17:15.752216: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:17:15.752221: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:17:15.752225: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:17:15.752229: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:17:15.752234: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:17:15.752238: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:17:15.752242: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:17:15.752246: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:17:15.752250: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:17:15.752254: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:17:15.752258: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:17:15.752263: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:17:15.752267: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:17:15.752271: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:17:15.752275: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:17:15.752280: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:17:15.752284: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:17:15.752288: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:17:15.752292: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:17:15.752304: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:17:15.752308: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:17:15.752312: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:17:15.752317: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:17:15.752321: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:17:15.752325: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:17:15.752330: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:17:15.752334: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:17:15.752338: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:17:15.752342: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:17:15.752346: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:17:15.752350: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:17:15.752355: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:17:15.752359: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:17:15.752363: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:17:15.752367: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:17:15.752371: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:17:15.752375: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:17:15.752379: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:17:15.752384: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:17:15.752388: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:17:15.752392: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:17:15.752396: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:17:15.752400: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:17:15.752404: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:17:15.752408: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:17:15.752412: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:17:15.752417: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:17:15.752421: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:17:15.752425: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:17:15.752429: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:17:15.752433: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:17:15.752437: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_detailed_logging: true
2024-05-18 11:17:15.752444: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:17:15.752449: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:17:15.752453: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:17:15.752457: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:17:15.752461: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:17:15.752465: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:17:15.752469: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:17:15.752473: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
INFO:runners:Done compiling.
2024-05-18 11:24:31.379439: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module jit_convert_element_type:
2024-05-18 11:24:31.379494: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:24:31.379502: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:24:31.379510: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:24:31.379518: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:24:31.379523: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:24:31.379530: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:24:31.379536: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:24:31.379543: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:24:31.379551: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:24:31.379559: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:24:31.379565: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:24:31.379573: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:24:31.379580: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:24:31.379587: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:24:31.379595: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:24:31.379600: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:24:31.379606: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:24:31.379612: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:24:31.379616: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.379623: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:24:31.379629: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:24:31.379635: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:24:31.379640: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:24:31.379644: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:24:31.379649: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:24:31.379659: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:24:31.379677: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:24:31.379683: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:24:31.379690: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:24:31.379700: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:24:31.379704: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:24:31.379709: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:24:31.379714: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:24:31.379720: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:24:31.379727: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:24:31.379732: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:24:31.379738: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:24:31.379742: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:24:31.379748: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:24:31.379756: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:24:31.379762: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:24:31.379766: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.379770: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.379775: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:24:31.379779: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:24:31.379785: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:24:31.379793: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:24:31.379798: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:24:31.379805: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:24:31.379810: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:24:31.379814: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:24:31.379820: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:24:31.379827: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:24:31.379833: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:24:31.379839: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:24:31.379843: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:24:31.379850: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:24:31.379856: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:24:31.379860: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:24:31.379868: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:24:31.379874: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:24:31.379881: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:24:31.379890: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:24:31.379895: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-05-18 11:24:31.444282: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module jit_broadcast_in_dim:
2024-05-18 11:24:31.444327: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:24:31.444333: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:24:31.444338: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:24:31.444343: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:24:31.444348: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:24:31.444352: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:24:31.444357: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:24:31.444361: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:24:31.444365: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:24:31.444370: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:24:31.444375: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:24:31.444379: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:24:31.444383: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:24:31.444387: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:24:31.444391: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:24:31.444395: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:24:31.444400: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:24:31.444404: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:24:31.444409: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.444414: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:24:31.444418: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:24:31.444423: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:24:31.444427: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:24:31.444431: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:24:31.444436: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:24:31.444440: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:24:31.444445: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:24:31.444449: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:24:31.444462: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:24:31.444466: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:24:31.444470: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:24:31.444475: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:24:31.444479: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:24:31.444484: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:24:31.444488: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:24:31.444492: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:24:31.444496: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:24:31.444502: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:24:31.444506: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:24:31.444511: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:24:31.444515: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:24:31.444519: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.444524: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.444528: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:24:31.444533: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:24:31.444537: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:24:31.444542: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:24:31.444547: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:24:31.444552: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:24:31.444556: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:24:31.444560: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:24:31.444565: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:24:31.444569: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:24:31.444573: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:24:31.444578: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:24:31.444582: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:24:31.444586: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:24:31.444591: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:24:31.444596: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:24:31.444601: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:24:31.444605: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:24:31.444611: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:24:31.444616: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:24:31.444620: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-05-18 11:24:31.577404: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module jit__squeeze:
2024-05-18 11:24:31.577458: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:24:31.577464: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:24:31.577469: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:24:31.577474: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:24:31.577479: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:24:31.577483: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:24:31.577488: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:24:31.577492: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:24:31.577496: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:24:31.577500: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:24:31.577504: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:24:31.577508: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:24:31.577513: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:24:31.577517: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:24:31.577521: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:24:31.577526: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:24:31.577530: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:24:31.577534: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:24:31.577538: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.577542: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:24:31.577546: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:24:31.577550: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:24:31.577554: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:24:31.577559: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:24:31.577563: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:24:31.577567: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:24:31.577572: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:24:31.577576: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:24:31.577580: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:24:31.577584: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:24:31.577588: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:24:31.577601: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:24:31.577605: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:24:31.577610: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:24:31.577614: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:24:31.577619: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:24:31.577623: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:24:31.577627: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:24:31.577631: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:24:31.577635: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:24:31.577640: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:24:31.577644: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.577648: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.577652: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:24:31.577656: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:24:31.577661: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:24:31.577665: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:24:31.577679: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:24:31.577684: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:24:31.577688: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:24:31.577692: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:24:31.577697: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:24:31.577701: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:24:31.577705: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:24:31.577709: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:24:31.577714: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:24:31.577718: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:24:31.577722: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:24:31.577726: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:24:31.577730: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:24:31.577734: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:24:31.577738: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:24:31.577742: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:24:31.577746: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
2024-05-18 11:24:31.629288: W external/xla/xla/service/gpu/gpu_compiler.cc:555] GpuCompilationEnvironment of hlo_module jit_scatter:
2024-05-18 11:24:31.629340: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_backend_optimization_level: 3
2024-05-18 11:24:31.629346: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_eliminate_hlo_implicit_broadcast: true
2024-05-18 11:24:31.629351: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_multi_thread_eigen: true
2024-05-18 11:24:31.629356: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cuda_data_dir: "./cuda_sdk_lib"
2024-05-18 11:24:31.629360: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_alias_scope_metadata: true
2024-05-18 11:24:31.629364: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_noalias_metadata: true
2024-05-18 11:24:31.629369: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_llvm_enable_invariant_load_metadata: true
2024-05-18 11:24:31.629373: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_force_host_platform_device_count: 1
2024-05-18 11:24:31.629377: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_nans: true
2024-05-18 11:24:31.629381: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_infs: true
2024-05-18 11:24:31.629386: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_allow_excess_precision: true
2024-05-18 11:24:31.629390: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_autotune_level: 4
2024-05-18 11:24:31.629394: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_division: true
2024-05-18 11:24:31.629398: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_fast_math_honor_functions: true
2024-05-18 11:24:31.629403: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_max_hlo_modules: -1
2024-05-18 11:24:31.629407: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_multiheap_size_constraint_per_heap: -1
2024-05-18 11:24:31.629411: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_async_all_reduce: true
2024-05-18 11:24:31.629416: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_strict_conv_algorithm_picker: true
2024-05-18 11:24:31.629420: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_reduce_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.629424: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_frontend: true
2024-05-18 11:24:31.629428: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_nccl_termination_timeout_seconds: -1
2024-05-18 11:24:31.629432: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_shared_constants: true
2024-05-18 11:24:31.629436: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_scratch_max_megabytes: 4096
2024-05-18 11:24:31.629440: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_simplify_all_fp_conversions: true
2024-05-18 11:24:31.629444: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_xla_runtime_executable: true
2024-05-18 11:24:31.629448: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_shape_checks: RUNTIME
2024-05-18 11:24:31.629457: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_normalize_layouts: true
2024-05-18 11:24:31.629461: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_tiling_and_fusion: true
2024-05-18 11:24:31.629465: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_dump_enable_mlir_pretty_form: true
2024-05-18 11:24:31.629469: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_triton_gemm: true
2024-05-18 11:24:31.629473: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cudnn_int8x32_convolution_reordering: true
2024-05-18 11:24:31.629478: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_experimental_deallocation: true
2024-05-18 11:24:31.629482: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_enable_mlir_fusion_outlining: true
2024-05-18 11:24:31.629499: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_m_dim: 8
2024-05-18 11:24:31.629504: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_n_dim: 8
2024-05-18 11:24:31.629508: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_cpu_matmul_tiling_k_dim: 8
2024-05-18 11:24:31.629512: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_num_runs_to_instantiate: -1
2024-05-18 11:24:31.629516: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_lhs_enable_gpu_async_tracker: true
2024-05-18 11:24:31.629520: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_inflation_factor: 1
2024-05-18 11:24:31.629525: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_min_graph_size: 5
2024-05-18 11:24:31.629529: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reassociation_for_converted_ar: true
2024-05-18 11:24:31.629533: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_all_gather_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.629537: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_reduce_scatter_combine_threshold_bytes: 31457280
2024-05-18 11:24:31.629541: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_highest_priority_async_stream: true
2024-05-18 11:24:31.629545: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_auto_spmd_partitioning_memory_budget_ratio: 1.1
2024-05-18 11:24:31.629550: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_redzone_padding_bytes: 8388608
2024-05-18 11:24:31.629554: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_triton_fusion_level: 2
2024-05-18 11:24:31.629558: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_graph_eviction_timeout_seconds: 60
2024-05-18 11:24:31.629562: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_gpu2_hal: true
2024-05-18 11:24:31.629566: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_copy_insertion_use_region_analysis: true
2024-05-18 11:24:31.629570: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_collective_permute_decomposer_threshold: 9223372036854775807
2024-05-18 11:24:31.629574: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_split_k_autotuning: true
2024-05-18 11:24:31.629578: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduction_epilogue_fusion: true
2024-05-18 11:24:31.629582: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_cublas_fallback: true
2024-05-18 11:24:31.629586: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_filter_kernels_spilling_registers_on_autotuning: true
2024-05-18 11:24:31.629591: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_debug_buffer_assignment_show_max: 15
2024-05-18 11:24:31.629595: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_enable_dumping: true
2024-05-18 11:24:31.629599: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_all_gather_combine_by_dim: true
2024-05-18 11:24:31.629603: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_reduce_scatter_combine_by_dim: true
2024-05-18 11:24:31.629607: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: FUSION
2024-05-18 11:24:31.629611: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_command_buffer: CUBLAS
2024-05-18 11:24:31.629616: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_enable_cub_radix_sort: true
2024-05-18 11:24:31.629620: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_memory_limit_slop_factor: 95
2024-05-18 11:24:31.629624: W external/xla/xla/service/gpu/gpu_compiler.cc:555] xla_gpu_threshold_for_windowed_einsum_mib: 100000
forward ...
forward ...
infer time： 5.0067901611328125e-06 秒
Output for prompt: The answer to life the universe and everything is of course 42.

But what is the answer to the question of how to get a job in the games industry?

Well, it’s not 42.

It’s not even a number.

It’s a question.

The question is:

“What do you want to do?”

The answer to that question is the answer to the question of how to get a job in the games industry.

You see, the games industry is a very competitive place