Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add API to assemble CPU shards to a sharded tensor #5630

Merged
merged 4 commits into from
Oct 5, 2023

Improve tests and error handling

c03a567
Select commit
Loading
Failed to load commit list.
Merged

Add API to assemble CPU shards to a sharded tensor #5630

Improve tests and error handling
c03a567
Select commit
Loading
Failed to load commit list.
Google Cloud Build / ci-tpu-test-trigger (tpu-pytorch) succeeded Oct 5, 2023 in 54m 53s

Summary

Build Information

Trigger ci-tpu-test-trigger
Build 4aec2951-776b-453a-9cfe-d163eda793f7
Start 2023-10-05T12:48:06-07:00
Duration 54m2.436s
Status SUCCESS

Steps

Step Status Duration
git_fetch SUCCESS 8.201s
git_checkout SUCCESS 698ms
build_pytorch-xla-test_docker_image SUCCESS 43m4.828s
push_pytorch-xla-test_docker_image SUCCESS 1m14.108s
run_e2e_tests SUCCESS 9m29.97s

Details

starting build "4aec2951-776b-453a-9cfe-d163eda793f7"

FETCHSOURCE
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint: 
hint: 	git config --global init.defaultBranch <name>
hint: 
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint: 
hint: 	git branch -m <name>
Initialized empty Git repository in /workspace/.git/
From https://github.com/pytorch/xla
 * branch            c03a5672354c0fc088812dbd4031e6cbafeb12d5 -> FETCH_HEAD
HEAD is now at c03a567 Improve tests and error handling
BUILD
Starting Step #0 - "git_fetch"
Step #0 - "git_fetch": Already have image (with digest): gcr.io/cloud-builders/git
Step #0 - "git_fetch": From https://github.com/pytorch/xla
Step #0 - "git_fetch":  * [new branch]          2-0-python-3-10         -> origin/2-0-python-3-10
Step #0 - "git_fetch":  * [new branch]          4483-improve-resnet50-data-loading-performance -> origin/4483-improve-resnet50-data-loading-performance
Step #0 - "git_fetch":  * [new branch]          5d-unit-test            -> origin/5d-unit-test
Step #0 - "git_fetch":  * [new branch]          FixTransferToServer     -> origin/FixTransferToServer
Step #0 - "git_fetch":  * [new branch]          JackCaaoG/spmd_device_python_test -> origin/JackCaaoG/spmd_device_python_test
Step #0 - "git_fetch":  * [new branch]          JackCao/Disable_hlo_dump_test_on_tpu -> origin/JackCao/Disable_hlo_dump_test_on_tpu
Step #0 - "git_fetch":  * [new branch]          JackCao/cherry_pick_dynamo_1 -> origin/JackCao/cherry_pick_dynamo_1
Step #0 - "git_fetch":  * [new branch]          JackCao/cherry_pick_dynamo_2 -> origin/JackCao/cherry_pick_dynamo_2
Step #0 - "git_fetch":  * [new branch]          JackCao/cherry_pick_dynamo_3 -> origin/JackCao/cherry_pick_dynamo_3
Step #0 - "git_fetch":  * [new branch]          JackCao/spmd_python_device -> origin/JackCao/spmd_python_device
Step #0 - "git_fetch":  * [new branch]          JackCao/unordered_sharding_spec -> origin/JackCao/unordered_sharding_spec
Step #0 - "git_fetch":  * [new branch]          JackCao/update_to_2.0   -> origin/JackCao/update_to_2.0
Step #0 - "git_fetch":  * [new branch]          JackCao/workaround_einsum_faketensor -> origin/JackCao/workaround_einsum_faketensor
Step #0 - "git_fetch":  * [new branch]          JackCaoG-patch-1        -> origin/JackCaoG-patch-1
Step #0 - "git_fetch":  * [new branch]          JackCaoG-patch-2        -> origin/JackCaoG-patch-2
Step #0 - "git_fetch":  * [new branch]          JackCaoG-patch-3        -> origin/JackCaoG-patch-3
Step #0 - "git_fetch":  * [new branch]          JackCaoG-patch-4        -> origin/JackCaoG-patch-4
Step #0 - "git_fetch":  * [new branch]          JackCaoG-remove-pin     -> origin/JackCaoG-remove-pin
Step #0 - "git_fetch":  * [new branch]          JackCaoG/1_13_revert_addcmul -> origin/JackCaoG/1_13_revert_addcmul
Step #0 - "git_fetch":  * [new branch]          JackCaoG/2.0_patch      -> origin/JackCaoG/2.0_patch
Step #0 - "git_fetch":  * [new branch]          JackCaoG/AddDynamoSpmdMetrics -> origin/JackCaoG/AddDynamoSpmdMetrics
Step #0 - "git_fetch":  * [new branch]          JackCaoG/CI_build_cpp_test -> origin/JackCaoG/CI_build_cpp_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/Dyanmo_execute_async -> origin/JackCaoG/Dyanmo_execute_async
Step #0 - "git_fetch":  * [new branch]          JackCaoG/Dynamo_training_test -> origin/JackCaoG/Dynamo_training_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/PJRT_wait_buffer_sync -> origin/JackCaoG/PJRT_wait_buffer_sync
Step #0 - "git_fetch":  * [new branch]          JackCaoG/SPMD_execute_metrics -> origin/JackCaoG/SPMD_execute_metrics
Step #0 - "git_fetch":  * [new branch]          JackCaoG/SPMD_int_to_cpu -> origin/JackCaoG/SPMD_int_to_cpu
Step #0 - "git_fetch":  * [new branch]          JackCaoG/Update_short_metrics -> origin/JackCaoG/Update_short_metrics
Step #0 - "git_fetch":  * [new branch]          JackCaoG/Update_wheel_docker_to_table -> origin/JackCaoG/Update_wheel_docker_to_table
Step #0 - "git_fetch":  * [new branch]          JackCaoG/add_dynamo_test_counter_check -> origin/JackCaoG/add_dynamo_test_counter_check
Step #0 - "git_fetch":  * [new branch]          JackCaoG/add_dynamo_test_to_tpu_ci -> origin/JackCaoG/add_dynamo_test_to_tpu_ci
Step #0 - "git_fetch":  * [new branch]          JackCaoG/add_gpu_doc    -> origin/JackCaoG/add_gpu_doc
Step #0 - "git_fetch":  * [new branch]          JackCaoG/add_gpu_doc_readme -> origin/JackCaoG/add_gpu_doc_readme
Step #0 - "git_fetch":  * [new branch]          JackCaoG/add_test_dynamo_graph_dump -> origin/JackCaoG/add_test_dynamo_graph_dump
Step #0 - "git_fetch":  * [new branch]          JackCaoG/api_init_runtime -> origin/JackCaoG/api_init_runtime
Step #0 - "git_fetch":  * [new branch]          JackCaoG/barrier_comment -> origin/JackCaoG/barrier_comment
Step #0 - "git_fetch":  * [new branch]          JackCaoG/better_dynamo_error_message -> origin/JackCaoG/better_dynamo_error_message
Step #0 - "git_fetch":  * [new branch]          JackCaoG/build_audio_whl -> origin/JackCaoG/build_audio_whl
Step #0 - "git_fetch":  * [new branch]          JackCaoG/bump+gpu_coverage_timeout -> origin/JackCaoG/bump+gpu_coverage_timeout
Step #0 - "git_fetch":  * [new branch]          JackCaoG/bump_build_timeout -> origin/JackCaoG/bump_build_timeout
Step #0 - "git_fetch":  * [new branch]          JackCaoG/cache_sharded_placeholder_dynamo -> origin/JackCaoG/cache_sharded_placeholder_dynamo
Step #0 - "git_fetch":  * [new branch]          JackCaoG/cherry_pick_dynamo_test -> origin/JackCaoG/cherry_pick_dynamo_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/cherry_picl_xrt_crash_2.1 -> origin/JackCaoG/cherry_picl_xrt_crash_2.1
Step #0 - "git_fetch":  * [new branch]          JackCaoG/clean_up_use_virtual_device -> origin/JackCaoG/clean_up_use_virtual_device
Step #0 - "git_fetch":  * [new branch]          JackCaoG/cleanup_docs   -> origin/JackCaoG/cleanup_docs
Step #0 - "git_fetch":  * [new branch]          JackCaoG/colab_whl_abi  -> origin/JackCaoG/colab_whl_abi
Step #0 - "git_fetch":  * [new branch]          JackCaoG/colab_whl_abi_2 -> origin/JackCaoG/colab_whl_abi_2
Step #0 - "git_fetch":  * [new branch]          JackCaoG/data_to_string -> origin/JackCaoG/data_to_string
Step #0 - "git_fetch":  * [new branch]          JackCaoG/disable_coverage -> origin/JackCaoG/disable_coverage
Step #0 - "git_fetch":  * [new branch]          JackCaoG/disable_dynamo_different_shape_test -> origin/JackCaoG/disable_dynamo_different_shape_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/disable_flaky_test -> origin/JackCaoG/disable_flaky_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/disable_more_dynamic -> origin/JackCaoG/disable_more_dynamic
Step #0 - "git_fetch":  * [new branch]          JackCaoG/disable_more_dynamic_test -> origin/JackCaoG/disable_more_dynamic_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/disable_test_xla_sharding -> origin/JackCaoG/disable_test_xla_sharding
Step #0 - "git_fetch":  * [new branch]          JackCaoG/disbale_docker_sandbox -> origin/JackCaoG/disbale_docker_sandbox
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dump_output_sharding -> origin/JackCaoG/dump_output_sharding
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_2.0_cherry_pick -> origin/JackCaoG/dynamo_2.0_cherry_pick
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_debug_message -> origin/JackCaoG/dynamo_debug_message
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_different_graph -> origin/JackCaoG/dynamo_different_graph
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_doc     -> origin/JackCaoG/dynamo_doc
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_dump_tensor_file -> origin/JackCaoG/dynamo_dump_tensor_file
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_note_backport_2.1 -> origin/JackCaoG/dynamo_note_backport_2.1
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_only_warm_up_cache -> origin/JackCaoG/dynamo_only_warm_up_cache
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_optimizer -> origin/JackCaoG/dynamo_optimizer
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_reduce_execute -> origin/JackCaoG/dynamo_reduce_execute
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_rng_seed_without_mark_step -> origin/JackCaoG/dynamo_rng_seed_without_mark_step
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_spmd    -> origin/JackCaoG/dynamo_spmd
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_spmd_input_sharding_change -> origin/JackCaoG/dynamo_spmd_input_sharding_change
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_spmd_optimization -> origin/JackCaoG/dynamo_spmd_optimization
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_warning -> origin/JackCaoG/dynamo_warning
Step #0 - "git_fetch":  * [new branch]          JackCaoG/enable_dynamo_upstream_test -> origin/JackCaoG/enable_dynamo_upstream_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/enable_fsdp_gpu_test -> origin/JackCaoG/enable_fsdp_gpu_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_cc_linter_once -> origin/JackCaoG/fix_cc_linter_once
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_deprecated_test_oepration_api -> origin/JackCaoG/fix_deprecated_test_oepration_api
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_dyanmo_additional_tracing -> origin/JackCaoG/fix_dyanmo_additional_tracing
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_dynamo     -> origin/JackCaoG/fix_dynamo
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_dynamo_crash -> origin/JackCaoG/fix_dynamo_crash
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_dynamo_inplace -> origin/JackCaoG/fix_dynamo_inplace
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_dynamo_text_allclose -> origin/JackCaoG/fix_dynamo_text_allclose
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_hlo_dump   -> origin/JackCaoG/fix_hlo_dump
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_hlo_dump_spmd -> origin/JackCaoG/fix_hlo_dump_spmd
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_mark_step_on_spmd_device -> origin/JackCaoG/fix_mark_step_on_spmd_device
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_mp_save_test -> origin/JackCaoG/fix_mp_save_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_send_cpu_data -> origin/JackCaoG/fix_send_cpu_data
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_spmd_buff_is_null -> origin/JackCaoG/fix_spmd_buff_is_null
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_tpu_ci_accuracy -> origin/JackCaoG/fix_tpu_ci_accuracy
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_vision_build -> origin/JackCaoG/fix_vision_build
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fsdp_autowrap_test -> origin/JackCaoG/fsdp_autowrap_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/gpu_build_pytorch_cuda -> origin/JackCaoG/gpu_build_pytorch_cuda
Step #0 - "git_fetch":  * [new branch]          JackCaoG/handle_dynamo_fn_without_arg -> origin/JackCaoG/handle_dynamo_fn_without_arg
Step #0 - "git_fetch":  * [new branch]          JackCaoG/improve_data_tostring -> origin/JackCaoG/improve_data_tostring
Step #0 - "git_fetch":  * [new branch]          JackCaoG/increase_timeout_docker -> origin/JackCaoG/increase_timeout_docker
Step #0 - "git_fetch":  * [new branch]          JackCaoG/ir_value_sharding -> origin/JackCaoG/ir_value_sharding
Step #0 - "git_fetch":  * [new branch]          JackCaoG/lower_dropout_backward -> origin/JackCaoG/lower_dropout_backward
Step #0 - "git_fetch":  * [new branch]          JackCaoG/move_op_lowering_guide -> origin/JackCaoG/move_op_lowering_guide
Step #0 - "git_fetch":  * [new branch]          JackCaoG/move_pin_2_0   -> origin/JackCaoG/move_pin_2_0
Step #0 - "git_fetch":  * [new branch]          JackCaoG/move_pjrt_lock_location -> origin/JackCaoG/move_pjrt_lock_location
Step #0 - "git_fetch":  * [new branch]          JackCaoG/move_return_future -> origin/JackCaoG/move_return_future
Step #0 - "git_fetch":  * [new branch]          JackCaoG/new_alias_api  -> origin/JackCaoG/new_alias_api
Step #0 - "git_fetch":  * [new branch]          JackCaoG/parallel_spmd  -> origin/JackCaoG/parallel_spmd
Step #0 - "git_fetch":  * [new branch]          JackCaoG/pin_update_20230808 -> origin/JackCaoG/pin_update_20230808
Step #0 - "git_fetch":  * [new branch]          JackCaoG/r20_backportdoc -> origin/JackCaoG/r20_backportdoc
Step #0 - "git_fetch":  * [new branch]          JackCaoG/r2_0_ABI_0     -> origin/JackCaoG/r2_0_ABI_0
Step #0 - "git_fetch":  * [new branch]          JackCaoG/r2_0_back_port_profile_flag -> origin/JackCaoG/r2_0_back_port_profile_flag
Step #0 - "git_fetch":  * [new branch]          JackCaoG/r2_0_backport_loader_thread -> origin/JackCaoG/r2_0_backport_loader_thread
Step #0 - "git_fetch":  * [new branch]          JackCaoG/r2_0_backport_loader_thread_2 -> origin/JackCaoG/r2_0_backport_loader_thread_2
Step #0 - "git_fetch":  * [new branch]          JackCaoG/r2_0_cherry_pick_defer_init -> origin/JackCaoG/r2_0_cherry_pick_defer_init
Step #0 - "git_fetch":  * [new branch]          JackCaoG/remove_pin     -> origin/JackCaoG/remove_pin
Step #0 - "git_fetch":  * [new branch]          JackCaoG/restrict_mark_sharding -> origin/JackCaoG/restrict_mark_sharding
Step #0 - "git_fetch":  * [new branch]          JackCaoG/revert_addcmul -> origin/JackCaoG/revert_addcmul
Step #0 - "git_fetch":  * [new branch]          JackCaoG/revert_protobuf_tf_patch -> origin/JackCaoG/revert_protobuf_tf_patch
Step #0 - "git_fetch":  * [new branch]          JackCaoG/shard_device_data_ir -> origin/JackCaoG/shard_device_data_ir
Step #0 - "git_fetch":  * [new branch]          JackCaoG/skip_tpu_test_precision -> origin/JackCaoG/skip_tpu_test_precision
Step #0 - "git_fetch":  * [new branch]          JackCaoG/spmd_aliasing  -> origin/JackCaoG/spmd_aliasing
Step #0 - "git_fetch":  * [new branch]          JackCaoG/spmd_dynamo_sharding_changed -> origin/JackCaoG/spmd_dynamo_sharding_changed
Step #0 - "git_fetch":  * [new branch]          JackCaoG/spmd_incorrect_result -> origin/JackCaoG/spmd_incorrect_result
Step #0 - "git_fetch":  * [new branch]          JackCaoG/tensor_debugging_api -> origin/JackCaoG/tensor_debugging_api
Step #0 - "git_fetch":  * [new branch]          JackCaoG/tensor_with_xla_data_sharding -> origin/JackCaoG/tensor_with_xla_data_sharding
Step #0 - "git_fetch":  * [new branch]          JackCaoG/test_torch_compile_usage -> origin/JackCaoG/test_torch_compile_usage
Step #0 - "git_fetch":  * [new branch]          JackCaoG/tf_update_20230201 -> origin/JackCaoG/tf_update_20230201
Step #0 - "git_fetch":  * [new branch]          JackCaoG/unordered_partial_sharding_spec -> origin/JackCaoG/unordered_partial_sharding_spec
Step #0 - "git_fetch":  * [new branch]          JackCaoG/updateDocker   -> origin/JackCaoG/updateDocker
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_2_0_ABI -> origin/JackCaoG/update_2_0_ABI
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_API_GUIDE -> origin/JackCaoG/update_API_GUIDE
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_XLA_IR_SHAPE_CACHE_SIZE -> origin/JackCaoG/update_XLA_IR_SHAPE_CACHE_SIZE
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_build_guide -> origin/JackCaoG/update_build_guide
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_colab_2_10_whl -> origin/JackCaoG/update_colab_2_10_whl
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_doc1    -> origin/JackCaoG/update_doc1
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_doc_remove_xrt -> origin/JackCaoG/update_doc_remove_xrt
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_docker_instruction -> origin/JackCaoG/update_docker_instruction
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_dynamo_backend_name -> origin/JackCaoG/update_dynamo_backend_name
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_dynamo_backend_nograd -> origin/JackCaoG/update_dynamo_backend_nograd
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_dynamo_doc -> origin/JackCaoG/update_dynamo_doc
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_dynamo_number_2.1 -> origin/JackCaoG/update_dynamo_number_2.1
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_dynamo_readme -> origin/JackCaoG/update_dynamo_readme
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_dynamo_save_graph_path -> origin/JackCaoG/update_dynamo_save_graph_path
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_dynamo_torch_equal -> origin/JackCaoG/update_dynamo_torch_equal
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_gpu_doc -> origin/JackCaoG/update_gpu_doc
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_gpu_wheel_link -> origin/JackCaoG/update_gpu_wheel_link
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_gpu_wheellink -> origin/JackCaoG/update_gpu_wheellink
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_pjrt_future -> origin/JackCaoG/update_pjrt_future
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_readme_2 -> origin/JackCaoG/update_readme_2
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_tf_2023018 -> origin/JackCaoG/update_tf_2023018
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_to_2.2  -> origin/JackCaoG/update_to_2.2
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_torch_compile_usage -> origin/JackCaoG/update_torch_compile_usage
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_wheel_docker_format -> origin/JackCaoG/update_wheel_docker_format
Step #0 - "git_fetch":  * [new branch]          JackCaoG/use_new_alias_api -> origin/JackCaoG/use_new_alias_api
Step #0 - "git_fetch":  * [new branch]          JackCaoG/xrt_cherry_pick_07_10 -> origin/JackCaoG/xrt_cherry_pick_07_10
Step #0 - "git_fetch":  * [new branch]          JackCaoG/xrt_cherrypick_0721 -> origin/JackCaoG/xrt_cherrypick_0721
Step #0 - "git_fetch":  * [new branch]          JackCaog/dynamo_input_sharding -> origin/JackCaog/dynamo_input_sharding
Step #0 - "git_fetch":  * [new branch]          JackCaog/quick_spmd_fix -> origin/JackCaog/quick_spmd_fix
Step #0 - "git_fetch":  * [new branch]          JackCaog/spmd_device_default -> origin/JackCaog/spmd_device_default
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-1       -> origin/ManfeiBai-patch-1
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-10      -> origin/ManfeiBai-patch-10
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-11      -> origin/ManfeiBai-patch-11
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-12      -> origin/ManfeiBai-patch-12
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-13      -> origin/ManfeiBai-patch-13
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-14      -> origin/ManfeiBai-patch-14
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-15      -> origin/ManfeiBai-patch-15
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-16      -> origin/ManfeiBai-patch-16
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-17      -> origin/ManfeiBai-patch-17
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-18      -> origin/ManfeiBai-patch-18
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-19      -> origin/ManfeiBai-patch-19
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-2       -> origin/ManfeiBai-patch-2
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-20      -> origin/ManfeiBai-patch-20
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-21      -> origin/ManfeiBai-patch-21
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-22      -> origin/ManfeiBai-patch-22
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-23      -> origin/ManfeiBai-patch-23
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-24      -> origin/ManfeiBai-patch-24
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-25      -> origin/ManfeiBai-patch-25
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-26      -> origin/ManfeiBai-patch-26
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-27      -> origin/ManfeiBai-patch-27
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-28      -> origin/ManfeiBai-patch-28
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-29      -> origin/ManfeiBai-patch-29
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-3       -> origin/ManfeiBai-patch-3
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-30      -> origin/ManfeiBai-patch-30
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-31      -> origin/ManfeiBai-patch-31
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-32      -> origin/ManfeiBai-patch-32
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-4       -> origin/ManfeiBai-patch-4
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-5       -> origin/ManfeiBai-patch-5
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-6       -> origin/ManfeiBai-patch-6
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-7       -> origin/ManfeiBai-patch-7
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-8       -> origin/ManfeiBai-patch-8
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-9       -> origin/ManfeiBai-patch-9
Step #0 - "git_fetch":  * [new branch]          XLASymNodeImplSub       -> origin/XLASymNodeImplSub
Step #0 - "git_fetch":  * [new branch]          XLA_HLO_DEBUG_METADATA  -> origin/XLA_HLO_DEBUG_METADATA
Step #0 - "git_fetch":  * [new branch]          a_tensor_cleanup        -> origin/a_tensor_cleanup
Step #0 - "git_fetch":  * [new branch]          accu_rate               -> origin/accu_rate
Step #0 - "git_fetch":  * [new branch]          adapt_lazy_fun_1        -> origin/adapt_lazy_fun_1
Step #0 - "git_fetch":  * [new branch]          add-imagenet-learning-rate-optimizer -> origin/add-imagenet-learning-rate-optimizer
Step #0 - "git_fetch":  * [new branch]          add-imagenet-learning-rate-scheduler -> origin/add-imagenet-learning-rate-scheduler
Step #0 - "git_fetch":  * [new branch]          add-imagenet-learning-schedule -> origin/add-imagenet-learning-schedule
Step #0 - "git_fetch":  * [new branch]          add-summary-writer      -> origin/add-summary-writer
Step #0 - "git_fetch":  * [new branch]          addAddDivToXLASymIntNodeImpl -> origin/addAddDivToXLASymIntNodeImpl
Step #0 - "git_fetch":  * [new branch]          addAnotherExpandSymintPyTest -> origin/addAnotherExpandSymintPyTest
Step #0 - "git_fetch":  * [new branch]          addAnsibleToR112        -> origin/addAnsibleToR112
Step #0 - "git_fetch":  * [new branch]          addAnsibleToR113        -> origin/addAnsibleToR113
Step #0 - "git_fetch":  * [new branch]          addDSModelTest          -> origin/addDSModelTest
Step #0 - "git_fetch":  * [new branch]          addExpandSymintTest     -> origin/addExpandSymintTest
Step #0 - "git_fetch":  * [new branch]          addMoreTestsForSizeAdd  -> origin/addMoreTestsForSizeAdd
Step #0 - "git_fetch":  * [new branch]          addPatch                -> origin/addPatch
Step #0 - "git_fetch":  * [new branch]          addPin                  -> origin/addPin
Step #0 - "git_fetch":  * [new branch]          addPy310Cuda118NightlyWheel -> origin/addPy310Cuda118NightlyWheel
Step #0 - "git_fetch":  * [new branch]          addTestForNonzero       -> origin/addTestForNonzero
Step #0 - "git_fetch":  * [new branch]          addVersionSelectionToR111 -> origin/addVersionSelectionToR111
Step #0 - "git_fetch":  * [new branch]          addVersionSelectionToR112 -> origin/addVersionSelectionToR112
Step #0 - "git_fetch":  * [new branch]          addVersionSelectionToR113 -> origin/addVersionSelectionToR113
Step #0 - "git_fetch":  * [new branch]          add_gitignore           -> origin/add_gitignore
Step #0 - "git_fetch":  * [new branch]          add_gitrev              -> origin/add_gitrev
Step #0 - "git_fetch":  * [new branch]          add_ne_size_node        -> origin/add_ne_size_node
Step #0 - "git_fetch":  * [new branch]          add_os_import_cifar     -> origin/add_os_import_cifar
Step #0 - "git_fetch":  * [new branch]          add_py_bench            -> origin/add_py_bench
Step #0 - "git_fetch":  * [new branch]          add_rendezvous_test     -> origin/add_rendezvous_test
Step #0 - "git_fetch":  * [new branch]          add_roots_tags          -> origin/add_roots_tags
Step #0 - "git_fetch":  * [new branch]          add_set_dimension_sizes -> origin/add_set_dimension_sizes
Step #0 - "git_fetch":  * [new branch]          add_symint_to_ci_docker -> origin/add_symint_to_ci_docker
Step #0 - "git_fetch":  * [new branch]          add_symint_util         -> origin/add_symint_util
Step #0 - "git_fetch":  * [new branch]          add_telemetry_env_var   -> origin/add_telemetry_env_var
Step #0 - "git_fetch":  * [new branch]          add_test_filter         -> origin/add_test_filter
Step #0 - "git_fetch":  * [new branch]          add_toxla_test          -> origin/add_toxla_test
Step #0 - "git_fetch":  * [new branch]          add_zach_comment        -> origin/add_zach_comment
Step #0 - "git_fetch":  * [new branch]          addcmul                 -> origin/addcmul
Step #0 - "git_fetch":  * [new branch]          addneg                  -> origin/addneg
Step #0 - "git_fetch":  * [new branch]          addr                    -> origin/addr
Step #0 - "git_fetch":  * [new branch]          aifix_warnings          -> origin/aifix_warnings
Step #0 - "git_fetch":  * [new branch]          ailing_fix_sums         -> origin/ailing_fix_sums
Step #0 - "git_fetch":  * [new branch]          ailzhang-patch-1        -> origin/ailzhang-patch-1
Step #0 - "git_fetch":  * [new branch]          alanwaketan/addcdiv     -> origin/alanwaketan/addcdiv
Step #0 - "git_fetch":  * [new branch]          alanwaketan/alias       -> origin/alanwaketan/alias
Step #0 - "git_fetch":  * [new branch]          alanwaketan/all_gather  -> origin/alanwaketan/all_gather
Step #0 - "git_fetch":  * [new branch]          alanwaketan/all_gather_token -> origin/alanwaketan/all_gather_token
Step #0 - "git_fetch":  * [new branch]          alanwaketan/all_reduce  -> origin/alanwaketan/all_reduce
Step #0 - "git_fetch":  * [new branch]          alanwaketan/all_reduce_d -> origin/alanwaketan/all_reduce_d
Step #0 - "git_fetch":  * [new branch]          alanwaketan/all_reduce_t -> origin/alanwaketan/all_reduce_t
Step #0 - "git_fetch":  * [new branch]          alanwaketan/as_strided  -> origin/alanwaketan/as_strided
Step #0 - "git_fetch":  * [new branch]          alanwaketan/as_strided_2 -> origin/alanwaketan/as_strided_2
Step #0 - "git_fetch":  * [new branch]          alanwaketan/build_input_output -> origin/alanwaketan/build_input_output
Step #0 - "git_fetch":  * [new branch]          alanwaketan/cc          -> origin/alanwaketan/cc
Step #0 - "git_fetch":  * [new branch]          alanwaketan/ci          -> origin/alanwaketan/ci
Step #0 - "git_fetch":  * [new branch]          alanwaketan/conv        -> origin/alanwaketan/conv
Step #0 - "git_fetch":  * [new branch]          alanwaketan/copy        -> origin/alanwaketan/copy
Step #0 - "git_fetch":  * [new branch]          alanwaketan/cpp_test    -> origin/alanwaketan/cpp_test
Step #0 - "git_fetch":  * [new branch]          alanwaketan/custom_sharding -> origin/alanwaketan/custom_sharding
Step #0 - "git_fetch":  * [new branch]          alanwaketan/debug       -> origin/alanwaketan/debug
Step #0 - "git_fetch":  * [new branch]          alanwaketan/detach      -> origin/alanwaketan/detach
Step #0 - "git_fetch":  * [new branch]          alanwaketan/detach_2    -> origin/alanwaketan/detach_2
Step #0 - "git_fetch":  * [new branch]          alanwaketan/diagonal_scatter -> origin/alanwaketan/diagonal_scatter
Step #0 - "git_fetch":  * [new branch]          alanwaketan/dist_2      -> origin/alanwaketan/dist_2
Step #0 - "git_fetch":  * [new branch]          alanwaketan/ds          -> origin/alanwaketan/ds
Step #0 - "git_fetch":  * [new branch]          alanwaketan/dump_hlo    -> origin/alanwaketan/dump_hlo
Step #0 - "git_fetch":  * [new branch]          alanwaketan/dynamic_shape -> origin/alanwaketan/dynamic_shape
Step #0 - "git_fetch":  * [new branch]          alanwaketan/experiments -> origin/alanwaketan/experiments
Step #0 - "git_fetch":  * [new branch]          alanwaketan/exponential -> origin/alanwaketan/exponential
Step #0 - "git_fetch":  * [new branch]          alanwaketan/fix         -> origin/alanwaketan/fix
Step #0 - "git_fetch":  * [new branch]          alanwaketan/flag        -> origin/alanwaketan/flag
Step #0 - "git_fetch":  * [new branch]          alanwaketan/fsdp        -> origin/alanwaketan/fsdp
Step #0 - "git_fetch":  * [new branch]          alanwaketan/fsdp_data   -> origin/alanwaketan/fsdp_data
Step #0 - "git_fetch":  * [new branch]          alanwaketan/fsdp_detach -> origin/alanwaketan/fsdp_detach
Step #0 - "git_fetch":  * [new branch]          alanwaketan/func_test   -> origin/alanwaketan/func_test
Step #0 - "git_fetch":  * [new branch]          alanwaketan/graph_exe   -> origin/alanwaketan/graph_exe
Step #0 - "git_fetch":  * [new branch]          alanwaketan/gru         -> origin/alanwaketan/gru
Step #0 - "git_fetch":  * [new branch]          alanwaketan/hide_token  -> origin/alanwaketan/hide_token
Step #0 - "git_fetch":  * [new branch]          alanwaketan/hlo_spmd    -> origin/alanwaketan/hlo_spmd
Step #0 - "git_fetch":  * [new branch]          alanwaketan/inputoutput -> origin/alanwaketan/inputoutput
Step #0 - "git_fetch":  * [new branch]          alanwaketan/lazy_graph  -> origin/alanwaketan/lazy_graph
Step #0 - "git_fetch":  * [new branch]          alanwaketan/lazy_graph_2 -> origin/alanwaketan/lazy_graph_2
Step #0 - "git_fetch":  * [new branch]          alanwaketan/lazy_graph_3 -> origin/alanwaketan/lazy_graph_3
Step #0 - "git_fetch":  * [new branch]          alanwaketan/lazy_graph_4 -> origin/alanwaketan/lazy_graph_4
Step #0 - "git_fetch":  * [new branch]          alanwaketan/lazy_graph_5 -> origin/alanwaketan/lazy_graph_5
Step #0 - "git_fetch":  * [new branch]          alanwak
...
[Logs truncated due to log size limitations. For full logs, see https://console.cloud.google.com/cloud-build/builds;region=us-central1/4aec2951-776b-453a-9cfe-d163eda793f7?project=164006649440.]
...
e2e_tests": The following NEW packages will be installed:
Step #4 - "run_e2e_tests":   gettext gettext-base
Step #4 - "run_e2e_tests": 0 upgraded, 2 newly installed, 0 to remove and 5 not upgraded.
Step #4 - "run_e2e_tests": Need to get 1486 kB of archives.
Step #4 - "run_e2e_tests": After this operation, 6658 kB of additional disk space will be used.
Step #4 - "run_e2e_tests": Get:1 http://deb.debian.org/debian bullseye/main amd64 gettext-base amd64 0.21-4 [175 kB]
Step #4 - "run_e2e_tests": Get:2 http://deb.debian.org/debian bullseye/main amd64 gettext amd64 0.21-4 [1311 kB]
Step #4 - "run_e2e_tests": debconf: delaying package configuration, since apt-utils is not installed
Step #4 - "run_e2e_tests": Fetched 1486 kB in 0s (6775 kB/s)
Step #4 - "run_e2e_tests": Selecting previously unselected package gettext-base.
Step #4 - "run_e2e_tests": (Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 103311 files and directories currently installed.)
Step #4 - "run_e2e_tests": Preparing to unpack .../gettext-base_0.21-4_amd64.deb ...
Step #4 - "run_e2e_tests": Unpacking gettext-base (0.21-4) ...
Step #4 - "run_e2e_tests": Selecting previously unselected package gettext.
Step #4 - "run_e2e_tests": Preparing to unpack .../gettext_0.21-4_amd64.deb ...
Step #4 - "run_e2e_tests": Unpacking gettext (0.21-4) ...
Step #4 - "run_e2e_tests": Setting up gettext-base (0.21-4) ...
Step #4 - "run_e2e_tests": Setting up gettext (0.21-4) ...
Step #4 - "run_e2e_tests": Processing triggers for libc-bin (2.31-13+deb11u7) ...
Step #4 - "run_e2e_tests": + apt-get install google-cloud-sdk-gke-gcloud-auth-plugin -o DPkg::options::=--force-overwrite
Step #4 - "run_e2e_tests": Reading package lists...
Step #4 - "run_e2e_tests": Building dependency tree...
Step #4 - "run_e2e_tests": Reading state information...
Step #4 - "run_e2e_tests": The following NEW packages will be installed:
Step #4 - "run_e2e_tests":   google-cloud-sdk-gke-gcloud-auth-plugin
Step #4 - "run_e2e_tests": 0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
Step #4 - "run_e2e_tests": Need to get 3213 kB of archives.
Step #4 - "run_e2e_tests": After this operation, 11.2 MB of additional disk space will be used.
Step #4 - "run_e2e_tests": Get:1 https://packages.cloud.google.com/apt cloud-sdk-bullseye/main amd64 google-cloud-sdk-gke-gcloud-auth-plugin amd64 449.0.0-0 [3213 kB]
Step #4 - "run_e2e_tests": debconf: delaying package configuration, since apt-utils is not installed
Step #4 - "run_e2e_tests": Fetched 3213 kB in 0s (24.2 MB/s)
Step #4 - "run_e2e_tests": Selecting previously unselected package google-cloud-sdk-gke-gcloud-auth-plugin.
Step #4 - "run_e2e_tests": (Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 103532 files and directories currently installed.)
Step #4 - "run_e2e_tests": Preparing to unpack .../google-cloud-sdk-gke-gcloud-auth-plugin_449.0.0-0_amd64.deb ...
Step #4 - "run_e2e_tests": Unpacking google-cloud-sdk-gke-gcloud-auth-plugin (449.0.0-0) ...
Step #4 - "run_e2e_tests": dpkg: warning: overriding problem because --force enabled:
Step #4 - "run_e2e_tests": dpkg: warning: trying to overwrite '/usr/lib/google-cloud-sdk/.install/gke-gcloud-auth-plugin-linux-x86_64.snapshot.json', which is also in package google-cloud-cli-gke-gcloud-auth-plugin 449.0.0-0
Step #4 - "run_e2e_tests": dpkg: warning: overriding problem because --force enabled:
Step #4 - "run_e2e_tests": dpkg: warning: trying to overwrite '/usr/lib/google-cloud-sdk/.install/gke-gcloud-auth-plugin.snapshot.json', which is also in package google-cloud-cli-gke-gcloud-auth-plugin 449.0.0-0
Step #4 - "run_e2e_tests": dpkg: warning: overriding problem because --force enabled:
Step #4 - "run_e2e_tests": dpkg: warning: trying to overwrite '/usr/lib/google-cloud-sdk/.install/gke-gcloud-auth-plugin.manifest', which is also in package google-cloud-cli-gke-gcloud-auth-plugin 449.0.0-0
Step #4 - "run_e2e_tests": dpkg: warning: overriding problem because --force enabled:
Step #4 - "run_e2e_tests": dpkg: warning: trying to overwrite '/usr/lib/google-cloud-sdk/.install/gke-gcloud-auth-plugin-linux-x86_64.manifest', which is also in package google-cloud-cli-gke-gcloud-auth-plugin 449.0.0-0
Step #4 - "run_e2e_tests": dpkg: warning: overriding problem because --force enabled:
Step #4 - "run_e2e_tests": dpkg: warning: trying to overwrite '/usr/lib/google-cloud-sdk/bin/gke-gcloud-auth-plugin', which is also in package google-cloud-cli-gke-gcloud-auth-plugin 449.0.0-0
Step #4 - "run_e2e_tests": dpkg: warning: overriding problem because --force enabled:
Step #4 - "run_e2e_tests": dpkg: warning: trying to overwrite '/usr/bin/gke-gcloud-auth-plugin', which is also in package google-cloud-cli-gke-gcloud-auth-plugin 449.0.0-0
Step #4 - "run_e2e_tests": Setting up google-cloud-sdk-gke-gcloud-auth-plugin (449.0.0-0) ...
Step #4 - "run_e2e_tests": + export USE_GKE_GCLOUD_AUTH_PLUGIN=True
Step #4 - "run_e2e_tests": + USE_GKE_GCLOUD_AUTH_PLUGIN=True
Step #4 - "run_e2e_tests": + apt-get install kubectl
Step #4 - "run_e2e_tests": Reading package lists...
Step #4 - "run_e2e_tests": Building dependency tree...
Step #4 - "run_e2e_tests": Reading state information...
Step #4 - "run_e2e_tests": kubectl is already the newest version (1.28.2-00).
Step #4 - "run_e2e_tests": 0 upgraded, 0 newly installed, 0 to remove and 5 not upgraded.
Step #4 - "run_e2e_tests": + gcloud container clusters get-credentials tpu-cluster --zone europe-west4-a
Step #4 - "run_e2e_tests": Fetching cluster endpoint and auth data.
Step #4 - "run_e2e_tests": kubeconfig entry generated for tpu-cluster.
Step #4 - "run_e2e_tests": + export IMAGE=us-central1-docker.pkg.dev/tpu-pytorch/docker/pytorch-xla-test:4aec2951-776b-453a-9cfe-d163eda793f7
Step #4 - "run_e2e_tests": + IMAGE=us-central1-docker.pkg.dev/tpu-pytorch/docker/pytorch-xla-test:4aec2951-776b-453a-9cfe-d163eda793f7
Step #4 - "run_e2e_tests": + export PROJECT_ID=tpu-pytorch
Step #4 - "run_e2e_tests": + PROJECT_ID=tpu-pytorch
Step #4 - "run_e2e_tests": ++ envsubst
Step #4 - "run_e2e_tests": ++ kubectl create -f - -o name
Step #4 - "run_e2e_tests": + pod_name=pod/xla-test-job-jtjzg
Step #4 - "run_e2e_tests": ++ kubectl wait --for condition=ready --timeout=60m pod/xla-test-job-jtjzg -o name
Step #4 - "run_e2e_tests": + pod_name=pod/xla-test-job-jtjzg
Step #4 - "run_e2e_tests": + kubectl logs -f pod/xla-test-job-jtjzg --container=xla-test
Step #4 - "run_e2e_tests": + pip install expecttest
Step #4 - "run_e2e_tests": Collecting expecttest
Step #4 - "run_e2e_tests":   Downloading expecttest-0.1.6-py3-none-any.whl (6.5 kB)
Step #4 - "run_e2e_tests": Installing collected packages: expecttest
Step #4 - "run_e2e_tests": Successfully installed expecttest-0.1.6
Step #4 - "run_e2e_tests": WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Step #4 - "run_e2e_tests": 
Step #4 - "run_e2e_tests": [notice] A new release of pip is available: 23.0.1 -> 23.2.1
Step #4 - "run_e2e_tests": [notice] To update, run: pip install --upgrade pip
Step #4 - "run_e2e_tests": + python3 /src/pytorch/xla/test/test_operations.py -v
Step #4 - "run_e2e_tests": /usr/local/lib/python3.10/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
Step #4 - "run_e2e_tests":   warn(
Step #4 - "run_e2e_tests": test_mp_decorator (__main__.MpDecoratorTest) ... ok
Step #4 - "run_e2e_tests": test_multi_init_xla_backend (__main__.RegisterXLAKeyTest) ... ok
Step #4 - "run_e2e_tests": test_rng_seed_transfer (__main__.TestAsyncScalar) ... ok
Step #4 - "run_e2e_tests": test_scalar_transfer (__main__.TestAsyncScalar) ... ok
Step #4 - "run_e2e_tests": test (__main__.TestAtenTensorTo) ... ok
Step #4 - "run_e2e_tests": test_add_mixed_device (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_addmm_integer_types (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_ailing_slice (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_amp_foreach_non_finite_check_and_unscale_ (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_arange_nan (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_as_strided_r1 (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_as_strided_r1_dim1 (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_as_strided_r1_slice (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_as_strided_r1_t (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_as_strided_r1_t_off (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_as_strided_r1_t_slice (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_as_strided_r2_t_update (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_baddmm_integer_types (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_basic_bfloat16 (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_bfloat16_float32_cast (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_binaryop_order (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_bitwise_and_not (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_byte_dtype (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_cached_addcdiv (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_cat_empty_tensor (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_clamp (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_conv2d_backward (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_deepcopy (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_diagonal_write (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_diagonal_write_transposed (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_diagonal_write_transposed_r3 (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_div_mixed_device (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_emb_bf16 (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_empty_advanced_indexing (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_empty_strided (__main__.TestAtenXlaTensor) ... skipped 'grad_input produces wrong results after functionalization. pytorch/pytorch#91199'
Step #4 - "run_e2e_tests": test_expand_default (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_flip (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_flip_check_throws (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_flip_empty_tensor (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_flip_expand (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_flip_rectangular (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_flip_shape (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_float32_bfloat16_cast (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_frac_negative (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_get_real_xla_devices (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_index_bool (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_index_put (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_index_select_0dim (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_index_select_out (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_index_types (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_copy_different_sizes (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_view_backprop_base (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_view_backprop_view (__main__.TestAtenXlaTensor) ... /src/pytorch/xla/test/test_operations.py:906: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /src/pytorch/build/aten/src/ATen/core/TensorBody.h:489.)
Step #4 - "run_e2e_tests":   self.assertIsNone(a.grad)
Step #4 - "run_e2e_tests": ok
Step #4 - "run_e2e_tests": test_inplace_view_backprop_view_of_view (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_view_gradcheck (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_view_makes_base_require_grad (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_view_modify_base (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_view_multiple_outputs (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_view_non_contig (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_view_of_view (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_view_python (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_masked_fill_in_out_place (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_masked_fill_with_tensor (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_matmul_integer_types (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_max_broadcast (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_mul_mixed_device (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_nan_to_num_in_place (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_nan_to_num_in_place_with_inf (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_negative_cat (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_negative_slice (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_no_storage (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_norm_p0 (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_patched_linear_1D_bias (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_patched_linear_2D_bias (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_patched_linear_3D (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_patched_linear_3D_bias (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_pow_constant (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_pow_integer_types (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_pred_and_u8 (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_pred_one_hot (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_pred_type (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_print (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_rand (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_rand_like (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_randint_like (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_randn_like (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_randperm (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_reduction_0dim (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_reduction_unordered_dim (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_replace_xla_tensor (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_rrelu_module (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_s_copy_dtype (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_save (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_save_api (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_save_bf16 (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_save_tuple (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_save_view_alias_check (__main__.TestAtenXlaTensor) ... skipped 'When functionalization is enabled, views do not exist.'
Step #4 - "run_e2e_tests": test_scatter_add_bool (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_scatter_add_small_target (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_serialization_api (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_set (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_sgn (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_sigmoid_bounds (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_slice_assign (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_slice_copy (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_slice_rnd_stepped_assign (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_slice_start_end (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_slice_stepped_assign (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_slice_stepped_other_assign (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_slice_zero_sized_dim (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_split_empty_dim (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_spooky_ailing (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_squeeze_nonzero (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_stack_pred (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_sub_mixed_device (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_too_many_parameter (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_transpose_1d (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_transpose_1d_inplace (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_view_1718 (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_view_and_copy_ (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_view_and_multi_mark_step (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_view_empty (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_view_out_computation (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_writeable_tensors_updates (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_cross_entropy_loss (__main__.TestBinaryCrossEntropyLimitValue) ... ok
Step #4 - "run_e2e_tests": test_mixed_dtype_tuple (__main__.TestDataType) ... ok
Step #4 - "run_e2e_tests": test_get_xla_tensor_debug_info (__main__.TestDebuggingUtil) ... ok
Step #4 - "run_e2e_tests": test_masked_select_shape (__main__.TestDynamicShape) ... ok
Step #4 - "run_e2e_tests": test_nonzero_cast (__main__.TestDynamicShape) ... ok
Step #4 - "run_e2e_tests": test_nonzero_shape (__main__.TestDynamicShape) ... ok
Step #4 - "run_e2e_tests": test_data_wrapper (__main__.TestGeneric) ... ok
Step #4 - "run_e2e_tests": test_git_revisons (__main__.TestGeneric) ... ok
Step #4 - "run_e2e_tests": test_nms (__main__.TestGeneric) ... ok
Step #4 - "run_e2e_tests": test_send_to_device_grad (__main__.TestGeneric) ... ok
Step #4 - "run_e2e_tests": test_send_to_device_single (__main__.TestGeneric) ... ok
Step #4 - "run_e2e_tests": test_util_foreach_api (__main__.TestGeneric) ... ok
Step #4 - "run_e2e_tests": test_util_foreach_api_cycle (__main__.TestGeneric) ... ok
Step #4 - "run_e2e_tests": test_zeros_like_patch (__main__.TestGeneric) ... ok
Step #4 - "run_e2e_tests": test_inter_op_sync (__main__.TestInterOpSyncTensors) ... ok
Step #4 - "run_e2e_tests": test (__main__.TestLongGraphChain) ... ok
Step #4 - "run_e2e_tests": test (__main__.TestModelComparator) ... ok
Step #4 - "run_e2e_tests": test_nll_loss_inf (__main__.TestNllLossLimitValue) ... ok
Step #4 - "run_e2e_tests": test_nll_loss_nan (__main__.TestNllLossLimitValue) ... ok
Step #4 - "run_e2e_tests": test_add (__main__.TestOpBuilder) ... ok
Step #4 - "run_e2e_tests": test_conditional (__main__.TestOpBuilder) ... ok
Step #4 - "run_e2e_tests": test_mul (__main__.TestOpBuilder) ... ok
Step #4 - "run_e2e_tests": test_triangular_solve (__main__.TestOpBuilder) ... /src/pytorch/xla/test/test_operations.py:2025: UserWarning: torch.triangular_solve is deprecated in favor of torch.linalg.solve_triangularand will be removed in a future PyTorch release.
Step #4 - "run_e2e_tests": torch.linalg.solve_triangular has its arguments reversed and does not return a copy of one of the inputs.
Step #4 - "run_e2e_tests": X = torch.triangular_solve(B, A).solution
Step #4 - "run_e2e_tests": should be replaced with
Step #4 - "run_e2e_tests": X = torch.linalg.solve_triangular(A, B). (Triggered internally at /src/pytorch/aten/src/ATen/native/BatchLinearAlgebra.cpp:2192.)
Step #4 - "run_e2e_tests":   return torch.triangular_solve(
Step #4 - "run_e2e_tests": ok
Step #4 - "run_e2e_tests": test_while (__main__.TestOpBuilder) ... ok
Step #4 - "run_e2e_tests": test_optimization_barrier_correctness (__main__.TestOptimizationBarrier) ... ok
Step #4 - "run_e2e_tests": test (__main__.TestParallelLoader) ... ok
Step #4 - "run_e2e_tests": test (__main__.TestParallelTensorMNIST) ... ok
Step #4 - "run_e2e_tests": test (__main__.TestParallelTensorResnet18) ... ok
Step #4 - "run_e2e_tests": test_random_from_to_bool (__main__.TestRandom) ... ok
Step #4 - "run_e2e_tests": test_get_xla_tensor (__main__.TestSelect) ... ok
Step #4 - "run_e2e_tests": test (__main__.TestToXlaTensorArena) ... ok
Step #4 - "run_e2e_tests": test_wait_device_ops (__main__.TestWaitDeviceOps) ... ok
Step #4 - "run_e2e_tests": test_non_empty_scope (__main__.XpTraceTest) ... ok
Step #4 - "run_e2e_tests": test_non_empty_scope_decorator (__main__.XpTraceTest) ... ok
Step #4 - "run_e2e_tests": 
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Ran 158 tests in 163.277s
Step #4 - "run_e2e_tests": 
Step #4 - "run_e2e_tests": OK (skipped=12)
Step #4 - "run_e2e_tests": + python3 /src/pytorch/xla/test/pjrt/test_runtime_tpu.py
Step #4 - "run_e2e_tests": Running tests under Python 3.10.13: /usr/local/bin/python3
Step #4 - "run_e2e_tests": [ RUN      ] TestExperimentalPjrtTpu.test_default_xla_devices
Step #4 - "run_e2e_tests": [       OK ] TestExperimentalPjrtTpu.test_default_xla_devices
Step #4 - "run_e2e_tests": [ RUN      ] TestExperimentalPjrtTpu.test_execute_time_metric
Step #4 - "run_e2e_tests": [       OK ] TestExperimentalPjrtTpu.test_execute_time_metric
Step #4 - "run_e2e_tests": [ RUN      ] TestExperimentalPjrtTpu.test_global_ordinal_pjrt
Step #4 - "run_e2e_tests": [       OK ] TestExperimentalPjrtTpu.test_global_ordinal_pjrt
Step #4 - "run_e2e_tests": [ RUN      ] TestExperimentalPjrtTpu.test_global_ordinal_xla_model
Step #4 - "run_e2e_tests": [       OK ] TestExperimentalPjrtTpu.test_global_ordinal_xla_model
Step #4 - "run_e2e_tests": [ RUN      ] TestExperimentalPjrtTpu.test_global_runtime_device_attributes
Step #4 - "run_e2e_tests": [       OK ] TestExperimentalPjrtTpu.test_global_runtime_device_attributes
Step #4 - "run_e2e_tests": [ RUN      ] TestExperimentalPjrtTpu.test_local_ordinal_pjrt
Step #4 - "run_e2e_tests": [       OK ] TestExperimentalPjrtTpu.test_local_ordinal_pjrt
Step #4 - "run_e2e_tests": [ RUN      ] TestExperimentalPjrtTpu.test_local_ordinal_with_discontiguous_global_ordinal_v4
Step #4 - "run_e2e_tests": [  SKIPPED ] TestExperimentalPjrtTpu.test_local_ordinal_with_discontiguous_global_ordinal_v4
Step #4 - "run_e2e_tests": [ RUN      ] TestExperimentalPjrtTpu.test_local_ordinal_with_discontiguous_global_ordinal_v4_threaded
Step #4 - "run_e2e_tests": [  SKIPPED ] TestExperimentalPjrtTpu.test_local_ordinal_with_discontiguous_global_ordinal_v4_threaded
Step #4 - "run_e2e_tests": [ RUN      ] TestExperimentalPjrtTpu.test_local_ordinal_xla_model
Step #4 - "run_e2e_tests": [       OK ] TestExperimentalPjrtTpu.test_local_ordinal_xla_model
Step #4 - "run_e2e_tests": [ RUN      ] TestExperimentalPjrtTpu.test_runtime_device_attributes
Step #4 - "run_e2e_tests": [       OK ] TestExperimentalPjrtTpu.test_runtime_device_attributes
Step #4 - "run_e2e_tests": [ RUN      ] TestExperimentalPjrtTpu.test_spawn_threads
Step #4 - "run_e2e_tests": [       OK ] TestExperimentalPjrtTpu.test_spawn_threads
Step #4 - "run_e2e_tests": [ RUN      ] TestExperimentalPjrtTpu.test_xla_devices_multiprocess
Step #4 - "run_e2e_tests": [       OK ] TestExperimentalPjrtTpu.test_xla_devices_multiprocess
Step #4 - "run_e2e_tests": [ RUN      ] TestExperimentalPjrtTpu.test_xla_devices_single_process_all_chips
Step #4 - "run_e2e_tests": [       OK ] TestExperimentalPjrtTpu.test_xla_devices_single_process_all_chips
Step #4 - "run_e2e_tests": [ RUN      ] TestExperimentalPjrtTpu.test_xla_devices_single_process_one_chip
Step #4 - "run_e2e_tests": [       OK ] TestExperimentalPjrtTpu.test_xla_devices_single_process_one_chip
Step #4 - "run_e2e_tests": [ RUN      ] TestExperimentalPjrtTpu.test_xla_devices_single_process_one_chip_one_device_spawn
Step #4 - "run_e2e_tests": [       OK ] TestExperimentalPjrtTpu.test_xla_devices_single_process_one_chip_one_device_spawn
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Ran 15 tests in 119.963s
Step #4 - "run_e2e_tests": 
Step #4 - "run_e2e_tests": OK (skipped=2)
Step #4 - "run_e2e_tests": + python3 /src/pytorch/xla/test/pjrt/test_collective_ops_tpu.py
Step #4 - "run_e2e_tests": Running tests under Python 3.10.13: /usr/local/bin/python3
Step #4 - "run_e2e_tests": [ RUN      ] TestCollectiveOpsTpu.test_all_gather_pinned
Step #4 - "run_e2e_tests": [       OK ] TestCollectiveOpsTpu.test_all_gather_pinned
Step #4 - "run_e2e_tests": [ RUN      ] TestCollectiveOpsTpu.test_all_gather_unpinned
Step #4 - "run_e2e_tests": [       OK ] TestCollectiveOpsTpu.test_all_gather_unpinned
Step #4 - "run_e2e_tests": [ RUN      ] TestCollectiveOpsTpu.test_all_reduce_pinned
Step #4 - "run_e2e_tests": [       OK ] TestCollectiveOpsTpu.test_all_reduce_pinned
Step #4 - "run_e2e_tests": [ RUN      ] TestCollectiveOpsTpu.test_all_reduce_unpinned
Step #4 - "run_e2e_tests": [       OK ] TestCollectiveOpsTpu.test_all_reduce_unpinned
Step #4 - "run_e2e_tests": [ RUN      ] TestCollectiveOpsTpu.test_all_to_all_pinned
Step #4 - "run_e2e_tests": [       OK ] TestCollectiveOpsTpu.test_all_to_all_pinned
Step #4 - "run_e2e_tests": [ RUN      ] TestCollectiveOpsTpu.test_all_to_all_unpinned
Step #4 - "run_e2e_tests": [       OK ] TestCollectiveOpsTpu.test_all_to_all_unpinned
Step #4 - "run_e2e_tests": [ RUN      ] TestCollectiveOpsTpu.test_broadcast_master_param_synchronized_parameters
Step #4 - "run_e2e_tests": [       OK ] TestCollectiveOpsTpu.test_broadcast_master_param_synchronized_parameters
Step #4 - "run_e2e_tests": [ RUN      ] TestCollectiveOpsTpu.test_broadcast_master_param_unsynchronized_parameters
Step #4 - "run_e2e_tests": [       OK ] TestCollectiveOpsTpu.test_broadcast_master_param_unsynchronized_parameters
Step #4 - "run_e2e_tests": [ RUN      ] TestCollectiveOpsTpu.test_reduce_scatter_pinned
Step #4 - "run_e2e_tests": [       OK ] TestCollectiveOpsTpu.test_reduce_scatter_pinned
Step #4 - "run_e2e_tests": [ RUN      ] TestCollectiveOpsTpu.test_reduce_scatter_unpinned
Step #4 - "run_e2e_tests": [       OK ] TestCollectiveOpsTpu.test_reduce_scatter_unpinned
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Ran 10 tests in 124.602s
Step #4 - "run_e2e_tests": 
Step #4 - "run_e2e_tests": OK
Step #4 - "run_e2e_tests": + python3 /src/pytorch/xla/test/spmd/test_xla_sharding.py
Step #4 - "run_e2e_tests": .......E...ss......................................
Step #4 - "run_e2e_tests": ======================================================================
Step #4 - "run_e2e_tests": ERROR: test_from_cpu_shards_2d (__main__.BasicShardingTest)
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Traceback (most recent call last):
Step #4 - "run_e2e_tests":   File "/src/pytorch/xla/test/spmd/test_xla_sharding.py", line 948, in test_from_cpu_shards_2d
Step #4 - "run_e2e_tests":     self.assertTrue(torch.allclose(global_tensor.cpu(), expected))
Step #4 - "run_e2e_tests": RuntimeError: The size of tensor a (2) must match the size of tensor b (4) at non-singleton dimension 1
Step #4 - "run_e2e_tests": 
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Ran 51 tests in 2.859s
Step #4 - "run_e2e_tests": 
Step #4 - "run_e2e_tests": FAILED (errors=1, skipped=2)
Step #4 - "run_e2e_tests": ++ kubectl get pod/xla-test-job-jtjzg -o 'jsonpath={.status.containerStatuses[?(@.name=="xla-test")].state.terminated.exitCode}'
Step #4 - "run_e2e_tests": + exit
Finished Step #4 - "run_e2e_tests"
PUSH
DONE

Build Log: https://console.cloud.google.com/cloud-build/builds;region=us-central1/4aec2951-776b-453a-9cfe-d163eda793f7?project=164006649440