Add API to assemble CPU shards to a sharded tensor #5630
Merged
Google Cloud Build / ci-tpu-test-trigger (tpu-pytorch)
succeeded
Oct 5, 2023 in 54m 53s
Summary
Build Information
Trigger | ci-tpu-test-trigger |
Build | 4aec2951-776b-453a-9cfe-d163eda793f7 |
Start | 2023-10-05T12:48:06-07:00 |
Duration | 54m2.436s |
Status | SUCCESS |
Steps
Step | Status | Duration |
---|---|---|
git_fetch | SUCCESS | 8.201s |
git_checkout | SUCCESS | 698ms |
build_pytorch-xla-test_docker_image | SUCCESS | 43m4.828s |
push_pytorch-xla-test_docker_image | SUCCESS | 1m14.108s |
run_e2e_tests | SUCCESS | 9m29.97s |
Details
starting build "4aec2951-776b-453a-9cfe-d163eda793f7"
FETCHSOURCE
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m <name>
Initialized empty Git repository in /workspace/.git/
From https://github.com/pytorch/xla
* branch c03a5672354c0fc088812dbd4031e6cbafeb12d5 -> FETCH_HEAD
HEAD is now at c03a567 Improve tests and error handling
BUILD
Starting Step #0 - "git_fetch"
Step #0 - "git_fetch": Already have image (with digest): gcr.io/cloud-builders/git
Step #0 - "git_fetch": From https://github.com/pytorch/xla
Step #0 - "git_fetch": * [new branch] 2-0-python-3-10 -> origin/2-0-python-3-10
Step #0 - "git_fetch": * [new branch] 4483-improve-resnet50-data-loading-performance -> origin/4483-improve-resnet50-data-loading-performance
Step #0 - "git_fetch": * [new branch] 5d-unit-test -> origin/5d-unit-test
Step #0 - "git_fetch": * [new branch] FixTransferToServer -> origin/FixTransferToServer
Step #0 - "git_fetch": * [new branch] JackCaaoG/spmd_device_python_test -> origin/JackCaaoG/spmd_device_python_test
Step #0 - "git_fetch": * [new branch] JackCao/Disable_hlo_dump_test_on_tpu -> origin/JackCao/Disable_hlo_dump_test_on_tpu
Step #0 - "git_fetch": * [new branch] JackCao/cherry_pick_dynamo_1 -> origin/JackCao/cherry_pick_dynamo_1
Step #0 - "git_fetch": * [new branch] JackCao/cherry_pick_dynamo_2 -> origin/JackCao/cherry_pick_dynamo_2
Step #0 - "git_fetch": * [new branch] JackCao/cherry_pick_dynamo_3 -> origin/JackCao/cherry_pick_dynamo_3
Step #0 - "git_fetch": * [new branch] JackCao/spmd_python_device -> origin/JackCao/spmd_python_device
Step #0 - "git_fetch": * [new branch] JackCao/unordered_sharding_spec -> origin/JackCao/unordered_sharding_spec
Step #0 - "git_fetch": * [new branch] JackCao/update_to_2.0 -> origin/JackCao/update_to_2.0
Step #0 - "git_fetch": * [new branch] JackCao/workaround_einsum_faketensor -> origin/JackCao/workaround_einsum_faketensor
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-1 -> origin/JackCaoG-patch-1
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-2 -> origin/JackCaoG-patch-2
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-3 -> origin/JackCaoG-patch-3
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-4 -> origin/JackCaoG-patch-4
Step #0 - "git_fetch": * [new branch] JackCaoG-remove-pin -> origin/JackCaoG-remove-pin
Step #0 - "git_fetch": * [new branch] JackCaoG/1_13_revert_addcmul -> origin/JackCaoG/1_13_revert_addcmul
Step #0 - "git_fetch": * [new branch] JackCaoG/2.0_patch -> origin/JackCaoG/2.0_patch
Step #0 - "git_fetch": * [new branch] JackCaoG/AddDynamoSpmdMetrics -> origin/JackCaoG/AddDynamoSpmdMetrics
Step #0 - "git_fetch": * [new branch] JackCaoG/CI_build_cpp_test -> origin/JackCaoG/CI_build_cpp_test
Step #0 - "git_fetch": * [new branch] JackCaoG/Dyanmo_execute_async -> origin/JackCaoG/Dyanmo_execute_async
Step #0 - "git_fetch": * [new branch] JackCaoG/Dynamo_training_test -> origin/JackCaoG/Dynamo_training_test
Step #0 - "git_fetch": * [new branch] JackCaoG/PJRT_wait_buffer_sync -> origin/JackCaoG/PJRT_wait_buffer_sync
Step #0 - "git_fetch": * [new branch] JackCaoG/SPMD_execute_metrics -> origin/JackCaoG/SPMD_execute_metrics
Step #0 - "git_fetch": * [new branch] JackCaoG/SPMD_int_to_cpu -> origin/JackCaoG/SPMD_int_to_cpu
Step #0 - "git_fetch": * [new branch] JackCaoG/Update_short_metrics -> origin/JackCaoG/Update_short_metrics
Step #0 - "git_fetch": * [new branch] JackCaoG/Update_wheel_docker_to_table -> origin/JackCaoG/Update_wheel_docker_to_table
Step #0 - "git_fetch": * [new branch] JackCaoG/add_dynamo_test_counter_check -> origin/JackCaoG/add_dynamo_test_counter_check
Step #0 - "git_fetch": * [new branch] JackCaoG/add_dynamo_test_to_tpu_ci -> origin/JackCaoG/add_dynamo_test_to_tpu_ci
Step #0 - "git_fetch": * [new branch] JackCaoG/add_gpu_doc -> origin/JackCaoG/add_gpu_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/add_gpu_doc_readme -> origin/JackCaoG/add_gpu_doc_readme
Step #0 - "git_fetch": * [new branch] JackCaoG/add_test_dynamo_graph_dump -> origin/JackCaoG/add_test_dynamo_graph_dump
Step #0 - "git_fetch": * [new branch] JackCaoG/api_init_runtime -> origin/JackCaoG/api_init_runtime
Step #0 - "git_fetch": * [new branch] JackCaoG/barrier_comment -> origin/JackCaoG/barrier_comment
Step #0 - "git_fetch": * [new branch] JackCaoG/better_dynamo_error_message -> origin/JackCaoG/better_dynamo_error_message
Step #0 - "git_fetch": * [new branch] JackCaoG/build_audio_whl -> origin/JackCaoG/build_audio_whl
Step #0 - "git_fetch": * [new branch] JackCaoG/bump+gpu_coverage_timeout -> origin/JackCaoG/bump+gpu_coverage_timeout
Step #0 - "git_fetch": * [new branch] JackCaoG/bump_build_timeout -> origin/JackCaoG/bump_build_timeout
Step #0 - "git_fetch": * [new branch] JackCaoG/cache_sharded_placeholder_dynamo -> origin/JackCaoG/cache_sharded_placeholder_dynamo
Step #0 - "git_fetch": * [new branch] JackCaoG/cherry_pick_dynamo_test -> origin/JackCaoG/cherry_pick_dynamo_test
Step #0 - "git_fetch": * [new branch] JackCaoG/cherry_picl_xrt_crash_2.1 -> origin/JackCaoG/cherry_picl_xrt_crash_2.1
Step #0 - "git_fetch": * [new branch] JackCaoG/clean_up_use_virtual_device -> origin/JackCaoG/clean_up_use_virtual_device
Step #0 - "git_fetch": * [new branch] JackCaoG/cleanup_docs -> origin/JackCaoG/cleanup_docs
Step #0 - "git_fetch": * [new branch] JackCaoG/colab_whl_abi -> origin/JackCaoG/colab_whl_abi
Step #0 - "git_fetch": * [new branch] JackCaoG/colab_whl_abi_2 -> origin/JackCaoG/colab_whl_abi_2
Step #0 - "git_fetch": * [new branch] JackCaoG/data_to_string -> origin/JackCaoG/data_to_string
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_coverage -> origin/JackCaoG/disable_coverage
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_dynamo_different_shape_test -> origin/JackCaoG/disable_dynamo_different_shape_test
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_flaky_test -> origin/JackCaoG/disable_flaky_test
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_more_dynamic -> origin/JackCaoG/disable_more_dynamic
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_more_dynamic_test -> origin/JackCaoG/disable_more_dynamic_test
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_test_xla_sharding -> origin/JackCaoG/disable_test_xla_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/disbale_docker_sandbox -> origin/JackCaoG/disbale_docker_sandbox
Step #0 - "git_fetch": * [new branch] JackCaoG/dump_output_sharding -> origin/JackCaoG/dump_output_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_2.0_cherry_pick -> origin/JackCaoG/dynamo_2.0_cherry_pick
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_debug_message -> origin/JackCaoG/dynamo_debug_message
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_different_graph -> origin/JackCaoG/dynamo_different_graph
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_doc -> origin/JackCaoG/dynamo_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_dump_tensor_file -> origin/JackCaoG/dynamo_dump_tensor_file
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_note_backport_2.1 -> origin/JackCaoG/dynamo_note_backport_2.1
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_only_warm_up_cache -> origin/JackCaoG/dynamo_only_warm_up_cache
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_optimizer -> origin/JackCaoG/dynamo_optimizer
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_reduce_execute -> origin/JackCaoG/dynamo_reduce_execute
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_rng_seed_without_mark_step -> origin/JackCaoG/dynamo_rng_seed_without_mark_step
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_spmd -> origin/JackCaoG/dynamo_spmd
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_spmd_input_sharding_change -> origin/JackCaoG/dynamo_spmd_input_sharding_change
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_spmd_optimization -> origin/JackCaoG/dynamo_spmd_optimization
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_warning -> origin/JackCaoG/dynamo_warning
Step #0 - "git_fetch": * [new branch] JackCaoG/enable_dynamo_upstream_test -> origin/JackCaoG/enable_dynamo_upstream_test
Step #0 - "git_fetch": * [new branch] JackCaoG/enable_fsdp_gpu_test -> origin/JackCaoG/enable_fsdp_gpu_test
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_cc_linter_once -> origin/JackCaoG/fix_cc_linter_once
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_deprecated_test_oepration_api -> origin/JackCaoG/fix_deprecated_test_oepration_api
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dyanmo_additional_tracing -> origin/JackCaoG/fix_dyanmo_additional_tracing
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo -> origin/JackCaoG/fix_dynamo
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo_crash -> origin/JackCaoG/fix_dynamo_crash
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo_inplace -> origin/JackCaoG/fix_dynamo_inplace
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo_text_allclose -> origin/JackCaoG/fix_dynamo_text_allclose
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_hlo_dump -> origin/JackCaoG/fix_hlo_dump
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_hlo_dump_spmd -> origin/JackCaoG/fix_hlo_dump_spmd
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_mark_step_on_spmd_device -> origin/JackCaoG/fix_mark_step_on_spmd_device
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_mp_save_test -> origin/JackCaoG/fix_mp_save_test
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_send_cpu_data -> origin/JackCaoG/fix_send_cpu_data
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_spmd_buff_is_null -> origin/JackCaoG/fix_spmd_buff_is_null
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_tpu_ci_accuracy -> origin/JackCaoG/fix_tpu_ci_accuracy
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_vision_build -> origin/JackCaoG/fix_vision_build
Step #0 - "git_fetch": * [new branch] JackCaoG/fsdp_autowrap_test -> origin/JackCaoG/fsdp_autowrap_test
Step #0 - "git_fetch": * [new branch] JackCaoG/gpu_build_pytorch_cuda -> origin/JackCaoG/gpu_build_pytorch_cuda
Step #0 - "git_fetch": * [new branch] JackCaoG/handle_dynamo_fn_without_arg -> origin/JackCaoG/handle_dynamo_fn_without_arg
Step #0 - "git_fetch": * [new branch] JackCaoG/improve_data_tostring -> origin/JackCaoG/improve_data_tostring
Step #0 - "git_fetch": * [new branch] JackCaoG/increase_timeout_docker -> origin/JackCaoG/increase_timeout_docker
Step #0 - "git_fetch": * [new branch] JackCaoG/ir_value_sharding -> origin/JackCaoG/ir_value_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/lower_dropout_backward -> origin/JackCaoG/lower_dropout_backward
Step #0 - "git_fetch": * [new branch] JackCaoG/move_op_lowering_guide -> origin/JackCaoG/move_op_lowering_guide
Step #0 - "git_fetch": * [new branch] JackCaoG/move_pin_2_0 -> origin/JackCaoG/move_pin_2_0
Step #0 - "git_fetch": * [new branch] JackCaoG/move_pjrt_lock_location -> origin/JackCaoG/move_pjrt_lock_location
Step #0 - "git_fetch": * [new branch] JackCaoG/move_return_future -> origin/JackCaoG/move_return_future
Step #0 - "git_fetch": * [new branch] JackCaoG/new_alias_api -> origin/JackCaoG/new_alias_api
Step #0 - "git_fetch": * [new branch] JackCaoG/parallel_spmd -> origin/JackCaoG/parallel_spmd
Step #0 - "git_fetch": * [new branch] JackCaoG/pin_update_20230808 -> origin/JackCaoG/pin_update_20230808
Step #0 - "git_fetch": * [new branch] JackCaoG/r20_backportdoc -> origin/JackCaoG/r20_backportdoc
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_ABI_0 -> origin/JackCaoG/r2_0_ABI_0
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_back_port_profile_flag -> origin/JackCaoG/r2_0_back_port_profile_flag
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_backport_loader_thread -> origin/JackCaoG/r2_0_backport_loader_thread
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_backport_loader_thread_2 -> origin/JackCaoG/r2_0_backport_loader_thread_2
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_cherry_pick_defer_init -> origin/JackCaoG/r2_0_cherry_pick_defer_init
Step #0 - "git_fetch": * [new branch] JackCaoG/remove_pin -> origin/JackCaoG/remove_pin
Step #0 - "git_fetch": * [new branch] JackCaoG/restrict_mark_sharding -> origin/JackCaoG/restrict_mark_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/revert_addcmul -> origin/JackCaoG/revert_addcmul
Step #0 - "git_fetch": * [new branch] JackCaoG/revert_protobuf_tf_patch -> origin/JackCaoG/revert_protobuf_tf_patch
Step #0 - "git_fetch": * [new branch] JackCaoG/shard_device_data_ir -> origin/JackCaoG/shard_device_data_ir
Step #0 - "git_fetch": * [new branch] JackCaoG/skip_tpu_test_precision -> origin/JackCaoG/skip_tpu_test_precision
Step #0 - "git_fetch": * [new branch] JackCaoG/spmd_aliasing -> origin/JackCaoG/spmd_aliasing
Step #0 - "git_fetch": * [new branch] JackCaoG/spmd_dynamo_sharding_changed -> origin/JackCaoG/spmd_dynamo_sharding_changed
Step #0 - "git_fetch": * [new branch] JackCaoG/spmd_incorrect_result -> origin/JackCaoG/spmd_incorrect_result
Step #0 - "git_fetch": * [new branch] JackCaoG/tensor_debugging_api -> origin/JackCaoG/tensor_debugging_api
Step #0 - "git_fetch": * [new branch] JackCaoG/tensor_with_xla_data_sharding -> origin/JackCaoG/tensor_with_xla_data_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/test_torch_compile_usage -> origin/JackCaoG/test_torch_compile_usage
Step #0 - "git_fetch": * [new branch] JackCaoG/tf_update_20230201 -> origin/JackCaoG/tf_update_20230201
Step #0 - "git_fetch": * [new branch] JackCaoG/unordered_partial_sharding_spec -> origin/JackCaoG/unordered_partial_sharding_spec
Step #0 - "git_fetch": * [new branch] JackCaoG/updateDocker -> origin/JackCaoG/updateDocker
Step #0 - "git_fetch": * [new branch] JackCaoG/update_2_0_ABI -> origin/JackCaoG/update_2_0_ABI
Step #0 - "git_fetch": * [new branch] JackCaoG/update_API_GUIDE -> origin/JackCaoG/update_API_GUIDE
Step #0 - "git_fetch": * [new branch] JackCaoG/update_XLA_IR_SHAPE_CACHE_SIZE -> origin/JackCaoG/update_XLA_IR_SHAPE_CACHE_SIZE
Step #0 - "git_fetch": * [new branch] JackCaoG/update_build_guide -> origin/JackCaoG/update_build_guide
Step #0 - "git_fetch": * [new branch] JackCaoG/update_colab_2_10_whl -> origin/JackCaoG/update_colab_2_10_whl
Step #0 - "git_fetch": * [new branch] JackCaoG/update_doc1 -> origin/JackCaoG/update_doc1
Step #0 - "git_fetch": * [new branch] JackCaoG/update_doc_remove_xrt -> origin/JackCaoG/update_doc_remove_xrt
Step #0 - "git_fetch": * [new branch] JackCaoG/update_docker_instruction -> origin/JackCaoG/update_docker_instruction
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_backend_name -> origin/JackCaoG/update_dynamo_backend_name
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_backend_nograd -> origin/JackCaoG/update_dynamo_backend_nograd
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_doc -> origin/JackCaoG/update_dynamo_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_number_2.1 -> origin/JackCaoG/update_dynamo_number_2.1
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_readme -> origin/JackCaoG/update_dynamo_readme
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_save_graph_path -> origin/JackCaoG/update_dynamo_save_graph_path
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_torch_equal -> origin/JackCaoG/update_dynamo_torch_equal
Step #0 - "git_fetch": * [new branch] JackCaoG/update_gpu_doc -> origin/JackCaoG/update_gpu_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/update_gpu_wheel_link -> origin/JackCaoG/update_gpu_wheel_link
Step #0 - "git_fetch": * [new branch] JackCaoG/update_gpu_wheellink -> origin/JackCaoG/update_gpu_wheellink
Step #0 - "git_fetch": * [new branch] JackCaoG/update_pjrt_future -> origin/JackCaoG/update_pjrt_future
Step #0 - "git_fetch": * [new branch] JackCaoG/update_readme_2 -> origin/JackCaoG/update_readme_2
Step #0 - "git_fetch": * [new branch] JackCaoG/update_tf_2023018 -> origin/JackCaoG/update_tf_2023018
Step #0 - "git_fetch": * [new branch] JackCaoG/update_to_2.2 -> origin/JackCaoG/update_to_2.2
Step #0 - "git_fetch": * [new branch] JackCaoG/update_torch_compile_usage -> origin/JackCaoG/update_torch_compile_usage
Step #0 - "git_fetch": * [new branch] JackCaoG/update_wheel_docker_format -> origin/JackCaoG/update_wheel_docker_format
Step #0 - "git_fetch": * [new branch] JackCaoG/use_new_alias_api -> origin/JackCaoG/use_new_alias_api
Step #0 - "git_fetch": * [new branch] JackCaoG/xrt_cherry_pick_07_10 -> origin/JackCaoG/xrt_cherry_pick_07_10
Step #0 - "git_fetch": * [new branch] JackCaoG/xrt_cherrypick_0721 -> origin/JackCaoG/xrt_cherrypick_0721
Step #0 - "git_fetch": * [new branch] JackCaog/dynamo_input_sharding -> origin/JackCaog/dynamo_input_sharding
Step #0 - "git_fetch": * [new branch] JackCaog/quick_spmd_fix -> origin/JackCaog/quick_spmd_fix
Step #0 - "git_fetch": * [new branch] JackCaog/spmd_device_default -> origin/JackCaog/spmd_device_default
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-1 -> origin/ManfeiBai-patch-1
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-10 -> origin/ManfeiBai-patch-10
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-11 -> origin/ManfeiBai-patch-11
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-12 -> origin/ManfeiBai-patch-12
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-13 -> origin/ManfeiBai-patch-13
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-14 -> origin/ManfeiBai-patch-14
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-15 -> origin/ManfeiBai-patch-15
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-16 -> origin/ManfeiBai-patch-16
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-17 -> origin/ManfeiBai-patch-17
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-18 -> origin/ManfeiBai-patch-18
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-19 -> origin/ManfeiBai-patch-19
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-2 -> origin/ManfeiBai-patch-2
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-20 -> origin/ManfeiBai-patch-20
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-21 -> origin/ManfeiBai-patch-21
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-22 -> origin/ManfeiBai-patch-22
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-23 -> origin/ManfeiBai-patch-23
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-24 -> origin/ManfeiBai-patch-24
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-25 -> origin/ManfeiBai-patch-25
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-26 -> origin/ManfeiBai-patch-26
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-27 -> origin/ManfeiBai-patch-27
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-28 -> origin/ManfeiBai-patch-28
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-29 -> origin/ManfeiBai-patch-29
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-3 -> origin/ManfeiBai-patch-3
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-30 -> origin/ManfeiBai-patch-30
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-31 -> origin/ManfeiBai-patch-31
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-32 -> origin/ManfeiBai-patch-32
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-4 -> origin/ManfeiBai-patch-4
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-5 -> origin/ManfeiBai-patch-5
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-6 -> origin/ManfeiBai-patch-6
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-7 -> origin/ManfeiBai-patch-7
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-8 -> origin/ManfeiBai-patch-8
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-9 -> origin/ManfeiBai-patch-9
Step #0 - "git_fetch": * [new branch] XLASymNodeImplSub -> origin/XLASymNodeImplSub
Step #0 - "git_fetch": * [new branch] XLA_HLO_DEBUG_METADATA -> origin/XLA_HLO_DEBUG_METADATA
Step #0 - "git_fetch": * [new branch] a_tensor_cleanup -> origin/a_tensor_cleanup
Step #0 - "git_fetch": * [new branch] accu_rate -> origin/accu_rate
Step #0 - "git_fetch": * [new branch] adapt_lazy_fun_1 -> origin/adapt_lazy_fun_1
Step #0 - "git_fetch": * [new branch] add-imagenet-learning-rate-optimizer -> origin/add-imagenet-learning-rate-optimizer
Step #0 - "git_fetch": * [new branch] add-imagenet-learning-rate-scheduler -> origin/add-imagenet-learning-rate-scheduler
Step #0 - "git_fetch": * [new branch] add-imagenet-learning-schedule -> origin/add-imagenet-learning-schedule
Step #0 - "git_fetch": * [new branch] add-summary-writer -> origin/add-summary-writer
Step #0 - "git_fetch": * [new branch] addAddDivToXLASymIntNodeImpl -> origin/addAddDivToXLASymIntNodeImpl
Step #0 - "git_fetch": * [new branch] addAnotherExpandSymintPyTest -> origin/addAnotherExpandSymintPyTest
Step #0 - "git_fetch": * [new branch] addAnsibleToR112 -> origin/addAnsibleToR112
Step #0 - "git_fetch": * [new branch] addAnsibleToR113 -> origin/addAnsibleToR113
Step #0 - "git_fetch": * [new branch] addDSModelTest -> origin/addDSModelTest
Step #0 - "git_fetch": * [new branch] addExpandSymintTest -> origin/addExpandSymintTest
Step #0 - "git_fetch": * [new branch] addMoreTestsForSizeAdd -> origin/addMoreTestsForSizeAdd
Step #0 - "git_fetch": * [new branch] addPatch -> origin/addPatch
Step #0 - "git_fetch": * [new branch] addPin -> origin/addPin
Step #0 - "git_fetch": * [new branch] addPy310Cuda118NightlyWheel -> origin/addPy310Cuda118NightlyWheel
Step #0 - "git_fetch": * [new branch] addTestForNonzero -> origin/addTestForNonzero
Step #0 - "git_fetch": * [new branch] addVersionSelectionToR111 -> origin/addVersionSelectionToR111
Step #0 - "git_fetch": * [new branch] addVersionSelectionToR112 -> origin/addVersionSelectionToR112
Step #0 - "git_fetch": * [new branch] addVersionSelectionToR113 -> origin/addVersionSelectionToR113
Step #0 - "git_fetch": * [new branch] add_gitignore -> origin/add_gitignore
Step #0 - "git_fetch": * [new branch] add_gitrev -> origin/add_gitrev
Step #0 - "git_fetch": * [new branch] add_ne_size_node -> origin/add_ne_size_node
Step #0 - "git_fetch": * [new branch] add_os_import_cifar -> origin/add_os_import_cifar
Step #0 - "git_fetch": * [new branch] add_py_bench -> origin/add_py_bench
Step #0 - "git_fetch": * [new branch] add_rendezvous_test -> origin/add_rendezvous_test
Step #0 - "git_fetch": * [new branch] add_roots_tags -> origin/add_roots_tags
Step #0 - "git_fetch": * [new branch] add_set_dimension_sizes -> origin/add_set_dimension_sizes
Step #0 - "git_fetch": * [new branch] add_symint_to_ci_docker -> origin/add_symint_to_ci_docker
Step #0 - "git_fetch": * [new branch] add_symint_util -> origin/add_symint_util
Step #0 - "git_fetch": * [new branch] add_telemetry_env_var -> origin/add_telemetry_env_var
Step #0 - "git_fetch": * [new branch] add_test_filter -> origin/add_test_filter
Step #0 - "git_fetch": * [new branch] add_toxla_test -> origin/add_toxla_test
Step #0 - "git_fetch": * [new branch] add_zach_comment -> origin/add_zach_comment
Step #0 - "git_fetch": * [new branch] addcmul -> origin/addcmul
Step #0 - "git_fetch": * [new branch] addneg -> origin/addneg
Step #0 - "git_fetch": * [new branch] addr -> origin/addr
Step #0 - "git_fetch": * [new branch] aifix_warnings -> origin/aifix_warnings
Step #0 - "git_fetch": * [new branch] ailing_fix_sums -> origin/ailing_fix_sums
Step #0 - "git_fetch": * [new branch] ailzhang-patch-1 -> origin/ailzhang-patch-1
Step #0 - "git_fetch": * [new branch] alanwaketan/addcdiv -> origin/alanwaketan/addcdiv
Step #0 - "git_fetch": * [new branch] alanwaketan/alias -> origin/alanwaketan/alias
Step #0 - "git_fetch": * [new branch] alanwaketan/all_gather -> origin/alanwaketan/all_gather
Step #0 - "git_fetch": * [new branch] alanwaketan/all_gather_token -> origin/alanwaketan/all_gather_token
Step #0 - "git_fetch": * [new branch] alanwaketan/all_reduce -> origin/alanwaketan/all_reduce
Step #0 - "git_fetch": * [new branch] alanwaketan/all_reduce_d -> origin/alanwaketan/all_reduce_d
Step #0 - "git_fetch": * [new branch] alanwaketan/all_reduce_t -> origin/alanwaketan/all_reduce_t
Step #0 - "git_fetch": * [new branch] alanwaketan/as_strided -> origin/alanwaketan/as_strided
Step #0 - "git_fetch": * [new branch] alanwaketan/as_strided_2 -> origin/alanwaketan/as_strided_2
Step #0 - "git_fetch": * [new branch] alanwaketan/build_input_output -> origin/alanwaketan/build_input_output
Step #0 - "git_fetch": * [new branch] alanwaketan/cc -> origin/alanwaketan/cc
Step #0 - "git_fetch": * [new branch] alanwaketan/ci -> origin/alanwaketan/ci
Step #0 - "git_fetch": * [new branch] alanwaketan/conv -> origin/alanwaketan/conv
Step #0 - "git_fetch": * [new branch] alanwaketan/copy -> origin/alanwaketan/copy
Step #0 - "git_fetch": * [new branch] alanwaketan/cpp_test -> origin/alanwaketan/cpp_test
Step #0 - "git_fetch": * [new branch] alanwaketan/custom_sharding -> origin/alanwaketan/custom_sharding
Step #0 - "git_fetch": * [new branch] alanwaketan/debug -> origin/alanwaketan/debug
Step #0 - "git_fetch": * [new branch] alanwaketan/detach -> origin/alanwaketan/detach
Step #0 - "git_fetch": * [new branch] alanwaketan/detach_2 -> origin/alanwaketan/detach_2
Step #0 - "git_fetch": * [new branch] alanwaketan/diagonal_scatter -> origin/alanwaketan/diagonal_scatter
Step #0 - "git_fetch": * [new branch] alanwaketan/dist_2 -> origin/alanwaketan/dist_2
Step #0 - "git_fetch": * [new branch] alanwaketan/ds -> origin/alanwaketan/ds
Step #0 - "git_fetch": * [new branch] alanwaketan/dump_hlo -> origin/alanwaketan/dump_hlo
Step #0 - "git_fetch": * [new branch] alanwaketan/dynamic_shape -> origin/alanwaketan/dynamic_shape
Step #0 - "git_fetch": * [new branch] alanwaketan/experiments -> origin/alanwaketan/experiments
Step #0 - "git_fetch": * [new branch] alanwaketan/exponential -> origin/alanwaketan/exponential
Step #0 - "git_fetch": * [new branch] alanwaketan/fix -> origin/alanwaketan/fix
Step #0 - "git_fetch": * [new branch] alanwaketan/flag -> origin/alanwaketan/flag
Step #0 - "git_fetch": * [new branch] alanwaketan/fsdp -> origin/alanwaketan/fsdp
Step #0 - "git_fetch": * [new branch] alanwaketan/fsdp_data -> origin/alanwaketan/fsdp_data
Step #0 - "git_fetch": * [new branch] alanwaketan/fsdp_detach -> origin/alanwaketan/fsdp_detach
Step #0 - "git_fetch": * [new branch] alanwaketan/func_test -> origin/alanwaketan/func_test
Step #0 - "git_fetch": * [new branch] alanwaketan/graph_exe -> origin/alanwaketan/graph_exe
Step #0 - "git_fetch": * [new branch] alanwaketan/gru -> origin/alanwaketan/gru
Step #0 - "git_fetch": * [new branch] alanwaketan/hide_token -> origin/alanwaketan/hide_token
Step #0 - "git_fetch": * [new branch] alanwaketan/hlo_spmd -> origin/alanwaketan/hlo_spmd
Step #0 - "git_fetch": * [new branch] alanwaketan/inputoutput -> origin/alanwaketan/inputoutput
Step #0 - "git_fetch": * [new branch] alanwaketan/lazy_graph -> origin/alanwaketan/lazy_graph
Step #0 - "git_fetch": * [new branch] alanwaketan/lazy_graph_2 -> origin/alanwaketan/lazy_graph_2
Step #0 - "git_fetch": * [new branch] alanwaketan/lazy_graph_3 -> origin/alanwaketan/lazy_graph_3
Step #0 - "git_fetch": * [new branch] alanwaketan/lazy_graph_4 -> origin/alanwaketan/lazy_graph_4
Step #0 - "git_fetch": * [new branch] alanwaketan/lazy_graph_5 -> origin/alanwaketan/lazy_graph_5
Step #0 - "git_fetch": * [new branch] alanwak
...
[Logs truncated due to log size limitations. For full logs, see https://console.cloud.google.com/cloud-build/builds;region=us-central1/4aec2951-776b-453a-9cfe-d163eda793f7?project=164006649440.]
...
e2e_tests": The following NEW packages will be installed:
Step #4 - "run_e2e_tests": gettext gettext-base
Step #4 - "run_e2e_tests": 0 upgraded, 2 newly installed, 0 to remove and 5 not upgraded.
Step #4 - "run_e2e_tests": Need to get 1486 kB of archives.
Step #4 - "run_e2e_tests": After this operation, 6658 kB of additional disk space will be used.
Step #4 - "run_e2e_tests": Get:1 http://deb.debian.org/debian bullseye/main amd64 gettext-base amd64 0.21-4 [175 kB]
Step #4 - "run_e2e_tests": Get:2 http://deb.debian.org/debian bullseye/main amd64 gettext amd64 0.21-4 [1311 kB]
Step #4 - "run_e2e_tests": debconf: delaying package configuration, since apt-utils is not installed
Step #4 - "run_e2e_tests": Fetched 1486 kB in 0s (6775 kB/s)
Step #4 - "run_e2e_tests": Selecting previously unselected package gettext-base.
Step #4 - "run_e2e_tests": (Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 103311 files and directories currently installed.)
Step #4 - "run_e2e_tests": Preparing to unpack .../gettext-base_0.21-4_amd64.deb ...
Step #4 - "run_e2e_tests": Unpacking gettext-base (0.21-4) ...
Step #4 - "run_e2e_tests": Selecting previously unselected package gettext.
Step #4 - "run_e2e_tests": Preparing to unpack .../gettext_0.21-4_amd64.deb ...
Step #4 - "run_e2e_tests": Unpacking gettext (0.21-4) ...
Step #4 - "run_e2e_tests": Setting up gettext-base (0.21-4) ...
Step #4 - "run_e2e_tests": Setting up gettext (0.21-4) ...
Step #4 - "run_e2e_tests": Processing triggers for libc-bin (2.31-13+deb11u7) ...
Step #4 - "run_e2e_tests": + apt-get install google-cloud-sdk-gke-gcloud-auth-plugin -o DPkg::options::=--force-overwrite
Step #4 - "run_e2e_tests": Reading package lists...
Step #4 - "run_e2e_tests": Building dependency tree...
Step #4 - "run_e2e_tests": Reading state information...
Step #4 - "run_e2e_tests": The following NEW packages will be installed:
Step #4 - "run_e2e_tests": google-cloud-sdk-gke-gcloud-auth-plugin
Step #4 - "run_e2e_tests": 0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
Step #4 - "run_e2e_tests": Need to get 3213 kB of archives.
Step #4 - "run_e2e_tests": After this operation, 11.2 MB of additional disk space will be used.
Step #4 - "run_e2e_tests": Get:1 https://packages.cloud.google.com/apt cloud-sdk-bullseye/main amd64 google-cloud-sdk-gke-gcloud-auth-plugin amd64 449.0.0-0 [3213 kB]
Step #4 - "run_e2e_tests": debconf: delaying package configuration, since apt-utils is not installed
Step #4 - "run_e2e_tests": Fetched 3213 kB in 0s (24.2 MB/s)
Step #4 - "run_e2e_tests": Selecting previously unselected package google-cloud-sdk-gke-gcloud-auth-plugin.
Step #4 - "run_e2e_tests": (Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 103532 files and directories currently installed.)
Step #4 - "run_e2e_tests": Preparing to unpack .../google-cloud-sdk-gke-gcloud-auth-plugin_449.0.0-0_amd64.deb ...
Step #4 - "run_e2e_tests": Unpacking google-cloud-sdk-gke-gcloud-auth-plugin (449.0.0-0) ...
Step #4 - "run_e2e_tests": dpkg: warning: overriding problem because --force enabled:
Step #4 - "run_e2e_tests": dpkg: warning: trying to overwrite '/usr/lib/google-cloud-sdk/.install/gke-gcloud-auth-plugin-linux-x86_64.snapshot.json', which is also in package google-cloud-cli-gke-gcloud-auth-plugin 449.0.0-0
Step #4 - "run_e2e_tests": dpkg: warning: overriding problem because --force enabled:
Step #4 - "run_e2e_tests": dpkg: warning: trying to overwrite '/usr/lib/google-cloud-sdk/.install/gke-gcloud-auth-plugin.snapshot.json', which is also in package google-cloud-cli-gke-gcloud-auth-plugin 449.0.0-0
Step #4 - "run_e2e_tests": dpkg: warning: overriding problem because --force enabled:
Step #4 - "run_e2e_tests": dpkg: warning: trying to overwrite '/usr/lib/google-cloud-sdk/.install/gke-gcloud-auth-plugin.manifest', which is also in package google-cloud-cli-gke-gcloud-auth-plugin 449.0.0-0
Step #4 - "run_e2e_tests": dpkg: warning: overriding problem because --force enabled:
Step #4 - "run_e2e_tests": dpkg: warning: trying to overwrite '/usr/lib/google-cloud-sdk/.install/gke-gcloud-auth-plugin-linux-x86_64.manifest', which is also in package google-cloud-cli-gke-gcloud-auth-plugin 449.0.0-0
Step #4 - "run_e2e_tests": dpkg: warning: overriding problem because --force enabled:
Step #4 - "run_e2e_tests": dpkg: warning: trying to overwrite '/usr/lib/google-cloud-sdk/bin/gke-gcloud-auth-plugin', which is also in package google-cloud-cli-gke-gcloud-auth-plugin 449.0.0-0
Step #4 - "run_e2e_tests": dpkg: warning: overriding problem because --force enabled:
Step #4 - "run_e2e_tests": dpkg: warning: trying to overwrite '/usr/bin/gke-gcloud-auth-plugin', which is also in package google-cloud-cli-gke-gcloud-auth-plugin 449.0.0-0
Step #4 - "run_e2e_tests": Setting up google-cloud-sdk-gke-gcloud-auth-plugin (449.0.0-0) ...
Step #4 - "run_e2e_tests": + export USE_GKE_GCLOUD_AUTH_PLUGIN=True
Step #4 - "run_e2e_tests": + USE_GKE_GCLOUD_AUTH_PLUGIN=True
Step #4 - "run_e2e_tests": + apt-get install kubectl
Step #4 - "run_e2e_tests": Reading package lists...
Step #4 - "run_e2e_tests": Building dependency tree...
Step #4 - "run_e2e_tests": Reading state information...
Step #4 - "run_e2e_tests": kubectl is already the newest version (1.28.2-00).
Step #4 - "run_e2e_tests": 0 upgraded, 0 newly installed, 0 to remove and 5 not upgraded.
Step #4 - "run_e2e_tests": + gcloud container clusters get-credentials tpu-cluster --zone europe-west4-a
Step #4 - "run_e2e_tests": Fetching cluster endpoint and auth data.
Step #4 - "run_e2e_tests": kubeconfig entry generated for tpu-cluster.
Step #4 - "run_e2e_tests": + export IMAGE=us-central1-docker.pkg.dev/tpu-pytorch/docker/pytorch-xla-test:4aec2951-776b-453a-9cfe-d163eda793f7
Step #4 - "run_e2e_tests": + IMAGE=us-central1-docker.pkg.dev/tpu-pytorch/docker/pytorch-xla-test:4aec2951-776b-453a-9cfe-d163eda793f7
Step #4 - "run_e2e_tests": + export PROJECT_ID=tpu-pytorch
Step #4 - "run_e2e_tests": + PROJECT_ID=tpu-pytorch
Step #4 - "run_e2e_tests": ++ envsubst
Step #4 - "run_e2e_tests": ++ kubectl create -f - -o name
Step #4 - "run_e2e_tests": + pod_name=pod/xla-test-job-jtjzg
Step #4 - "run_e2e_tests": ++ kubectl wait --for condition=ready --timeout=60m pod/xla-test-job-jtjzg -o name
Step #4 - "run_e2e_tests": + pod_name=pod/xla-test-job-jtjzg
Step #4 - "run_e2e_tests": + kubectl logs -f pod/xla-test-job-jtjzg --container=xla-test
Step #4 - "run_e2e_tests": + pip install expecttest
Step #4 - "run_e2e_tests": Collecting expecttest
Step #4 - "run_e2e_tests": Downloading expecttest-0.1.6-py3-none-any.whl (6.5 kB)
Step #4 - "run_e2e_tests": Installing collected packages: expecttest
Step #4 - "run_e2e_tests": Successfully installed expecttest-0.1.6
Step #4 - "run_e2e_tests": WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Step #4 - "run_e2e_tests":
Step #4 - "run_e2e_tests": [notice] A new release of pip is available: 23.0.1 -> 23.2.1
Step #4 - "run_e2e_tests": [notice] To update, run: pip install --upgrade pip
Step #4 - "run_e2e_tests": + python3 /src/pytorch/xla/test/test_operations.py -v
Step #4 - "run_e2e_tests": /usr/local/lib/python3.10/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
Step #4 - "run_e2e_tests": warn(
Step #4 - "run_e2e_tests": test_mp_decorator (__main__.MpDecoratorTest) ... ok
Step #4 - "run_e2e_tests": test_multi_init_xla_backend (__main__.RegisterXLAKeyTest) ... ok
Step #4 - "run_e2e_tests": test_rng_seed_transfer (__main__.TestAsyncScalar) ... ok
Step #4 - "run_e2e_tests": test_scalar_transfer (__main__.TestAsyncScalar) ... ok
Step #4 - "run_e2e_tests": test (__main__.TestAtenTensorTo) ... ok
Step #4 - "run_e2e_tests": test_add_mixed_device (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_addmm_integer_types (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_ailing_slice (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_amp_foreach_non_finite_check_and_unscale_ (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_arange_nan (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_as_strided_r1 (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_as_strided_r1_dim1 (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_as_strided_r1_slice (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_as_strided_r1_t (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_as_strided_r1_t_off (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_as_strided_r1_t_slice (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_as_strided_r2_t_update (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_baddmm_integer_types (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_basic_bfloat16 (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_bfloat16_float32_cast (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_binaryop_order (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_bitwise_and_not (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_byte_dtype (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_cached_addcdiv (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_cat_empty_tensor (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_clamp (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_conv2d_backward (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_deepcopy (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_diagonal_write (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_diagonal_write_transposed (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_diagonal_write_transposed_r3 (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_div_mixed_device (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_emb_bf16 (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_empty_advanced_indexing (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_empty_strided (__main__.TestAtenXlaTensor) ... skipped 'grad_input produces wrong results after functionalization. pytorch/pytorch#91199'
Step #4 - "run_e2e_tests": test_expand_default (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_flip (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_flip_check_throws (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_flip_empty_tensor (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_flip_expand (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_flip_rectangular (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_flip_shape (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_float32_bfloat16_cast (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_frac_negative (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_get_real_xla_devices (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_index_bool (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_index_put (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_index_select_0dim (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_index_select_out (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_index_types (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_copy_different_sizes (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_view_backprop_base (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_view_backprop_view (__main__.TestAtenXlaTensor) ... /src/pytorch/xla/test/test_operations.py:906: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /src/pytorch/build/aten/src/ATen/core/TensorBody.h:489.)
Step #4 - "run_e2e_tests": self.assertIsNone(a.grad)
Step #4 - "run_e2e_tests": ok
Step #4 - "run_e2e_tests": test_inplace_view_backprop_view_of_view (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_view_gradcheck (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_view_makes_base_require_grad (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_view_modify_base (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_view_multiple_outputs (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_view_non_contig (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_view_of_view (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_inplace_view_python (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_masked_fill_in_out_place (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_masked_fill_with_tensor (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_matmul_integer_types (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_max_broadcast (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_mul_mixed_device (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_nan_to_num_in_place (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_nan_to_num_in_place_with_inf (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_negative_cat (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_negative_slice (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_no_storage (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_norm_p0 (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_patched_linear_1D_bias (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_patched_linear_2D_bias (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_patched_linear_3D (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_patched_linear_3D_bias (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_pow_constant (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_pow_integer_types (__main__.TestAtenXlaTensor) ... skipped 'Not supported on TPU'
Step #4 - "run_e2e_tests": test_pred_and_u8 (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_pred_one_hot (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_pred_type (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_print (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_rand (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_rand_like (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_randint_like (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_randn_like (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_randperm (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_reduction_0dim (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_reduction_unordered_dim (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_replace_xla_tensor (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_rrelu_module (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_s_copy_dtype (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_save (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_save_api (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_save_bf16 (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_save_tuple (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_save_view_alias_check (__main__.TestAtenXlaTensor) ... skipped 'When functionalization is enabled, views do not exist.'
Step #4 - "run_e2e_tests": test_scatter_add_bool (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_scatter_add_small_target (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_serialization_api (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_set (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_sgn (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_sigmoid_bounds (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_slice_assign (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_slice_copy (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_slice_rnd_stepped_assign (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_slice_start_end (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_slice_stepped_assign (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_slice_stepped_other_assign (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_slice_zero_sized_dim (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_split_empty_dim (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_spooky_ailing (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_squeeze_nonzero (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_stack_pred (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_sub_mixed_device (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_too_many_parameter (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_transpose_1d (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_transpose_1d_inplace (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_view_1718 (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_view_and_copy_ (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_view_and_multi_mark_step (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_view_empty (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_view_out_computation (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_writeable_tensors_updates (__main__.TestAtenXlaTensor) ... ok
Step #4 - "run_e2e_tests": test_cross_entropy_loss (__main__.TestBinaryCrossEntropyLimitValue) ... ok
Step #4 - "run_e2e_tests": test_mixed_dtype_tuple (__main__.TestDataType) ... ok
Step #4 - "run_e2e_tests": test_get_xla_tensor_debug_info (__main__.TestDebuggingUtil) ... ok
Step #4 - "run_e2e_tests": test_masked_select_shape (__main__.TestDynamicShape) ... ok
Step #4 - "run_e2e_tests": test_nonzero_cast (__main__.TestDynamicShape) ... ok
Step #4 - "run_e2e_tests": test_nonzero_shape (__main__.TestDynamicShape) ... ok
Step #4 - "run_e2e_tests": test_data_wrapper (__main__.TestGeneric) ... ok
Step #4 - "run_e2e_tests": test_git_revisons (__main__.TestGeneric) ... ok
Step #4 - "run_e2e_tests": test_nms (__main__.TestGeneric) ... ok
Step #4 - "run_e2e_tests": test_send_to_device_grad (__main__.TestGeneric) ... ok
Step #4 - "run_e2e_tests": test_send_to_device_single (__main__.TestGeneric) ... ok
Step #4 - "run_e2e_tests": test_util_foreach_api (__main__.TestGeneric) ... ok
Step #4 - "run_e2e_tests": test_util_foreach_api_cycle (__main__.TestGeneric) ... ok
Step #4 - "run_e2e_tests": test_zeros_like_patch (__main__.TestGeneric) ... ok
Step #4 - "run_e2e_tests": test_inter_op_sync (__main__.TestInterOpSyncTensors) ... ok
Step #4 - "run_e2e_tests": test (__main__.TestLongGraphChain) ... ok
Step #4 - "run_e2e_tests": test (__main__.TestModelComparator) ... ok
Step #4 - "run_e2e_tests": test_nll_loss_inf (__main__.TestNllLossLimitValue) ... ok
Step #4 - "run_e2e_tests": test_nll_loss_nan (__main__.TestNllLossLimitValue) ... ok
Step #4 - "run_e2e_tests": test_add (__main__.TestOpBuilder) ... ok
Step #4 - "run_e2e_tests": test_conditional (__main__.TestOpBuilder) ... ok
Step #4 - "run_e2e_tests": test_mul (__main__.TestOpBuilder) ... ok
Step #4 - "run_e2e_tests": test_triangular_solve (__main__.TestOpBuilder) ... /src/pytorch/xla/test/test_operations.py:2025: UserWarning: torch.triangular_solve is deprecated in favor of torch.linalg.solve_triangularand will be removed in a future PyTorch release.
Step #4 - "run_e2e_tests": torch.linalg.solve_triangular has its arguments reversed and does not return a copy of one of the inputs.
Step #4 - "run_e2e_tests": X = torch.triangular_solve(B, A).solution
Step #4 - "run_e2e_tests": should be replaced with
Step #4 - "run_e2e_tests": X = torch.linalg.solve_triangular(A, B). (Triggered internally at /src/pytorch/aten/src/ATen/native/BatchLinearAlgebra.cpp:2192.)
Step #4 - "run_e2e_tests": return torch.triangular_solve(
Step #4 - "run_e2e_tests": ok
Step #4 - "run_e2e_tests": test_while (__main__.TestOpBuilder) ... ok
Step #4 - "run_e2e_tests": test_optimization_barrier_correctness (__main__.TestOptimizationBarrier) ... ok
Step #4 - "run_e2e_tests": test (__main__.TestParallelLoader) ... ok
Step #4 - "run_e2e_tests": test (__main__.TestParallelTensorMNIST) ... ok
Step #4 - "run_e2e_tests": test (__main__.TestParallelTensorResnet18) ... ok
Step #4 - "run_e2e_tests": test_random_from_to_bool (__main__.TestRandom) ... ok
Step #4 - "run_e2e_tests": test_get_xla_tensor (__main__.TestSelect) ... ok
Step #4 - "run_e2e_tests": test (__main__.TestToXlaTensorArena) ... ok
Step #4 - "run_e2e_tests": test_wait_device_ops (__main__.TestWaitDeviceOps) ... ok
Step #4 - "run_e2e_tests": test_non_empty_scope (__main__.XpTraceTest) ... ok
Step #4 - "run_e2e_tests": test_non_empty_scope_decorator (__main__.XpTraceTest) ... ok
Step #4 - "run_e2e_tests":
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Ran 158 tests in 163.277s
Step #4 - "run_e2e_tests":
Step #4 - "run_e2e_tests": OK (skipped=12)
Step #4 - "run_e2e_tests": + python3 /src/pytorch/xla/test/pjrt/test_runtime_tpu.py
Step #4 - "run_e2e_tests": Running tests under Python 3.10.13: /usr/local/bin/python3
Step #4 - "run_e2e_tests": [ RUN ] TestExperimentalPjrtTpu.test_default_xla_devices
Step #4 - "run_e2e_tests": [ OK ] TestExperimentalPjrtTpu.test_default_xla_devices
Step #4 - "run_e2e_tests": [ RUN ] TestExperimentalPjrtTpu.test_execute_time_metric
Step #4 - "run_e2e_tests": [ OK ] TestExperimentalPjrtTpu.test_execute_time_metric
Step #4 - "run_e2e_tests": [ RUN ] TestExperimentalPjrtTpu.test_global_ordinal_pjrt
Step #4 - "run_e2e_tests": [ OK ] TestExperimentalPjrtTpu.test_global_ordinal_pjrt
Step #4 - "run_e2e_tests": [ RUN ] TestExperimentalPjrtTpu.test_global_ordinal_xla_model
Step #4 - "run_e2e_tests": [ OK ] TestExperimentalPjrtTpu.test_global_ordinal_xla_model
Step #4 - "run_e2e_tests": [ RUN ] TestExperimentalPjrtTpu.test_global_runtime_device_attributes
Step #4 - "run_e2e_tests": [ OK ] TestExperimentalPjrtTpu.test_global_runtime_device_attributes
Step #4 - "run_e2e_tests": [ RUN ] TestExperimentalPjrtTpu.test_local_ordinal_pjrt
Step #4 - "run_e2e_tests": [ OK ] TestExperimentalPjrtTpu.test_local_ordinal_pjrt
Step #4 - "run_e2e_tests": [ RUN ] TestExperimentalPjrtTpu.test_local_ordinal_with_discontiguous_global_ordinal_v4
Step #4 - "run_e2e_tests": [ SKIPPED ] TestExperimentalPjrtTpu.test_local_ordinal_with_discontiguous_global_ordinal_v4
Step #4 - "run_e2e_tests": [ RUN ] TestExperimentalPjrtTpu.test_local_ordinal_with_discontiguous_global_ordinal_v4_threaded
Step #4 - "run_e2e_tests": [ SKIPPED ] TestExperimentalPjrtTpu.test_local_ordinal_with_discontiguous_global_ordinal_v4_threaded
Step #4 - "run_e2e_tests": [ RUN ] TestExperimentalPjrtTpu.test_local_ordinal_xla_model
Step #4 - "run_e2e_tests": [ OK ] TestExperimentalPjrtTpu.test_local_ordinal_xla_model
Step #4 - "run_e2e_tests": [ RUN ] TestExperimentalPjrtTpu.test_runtime_device_attributes
Step #4 - "run_e2e_tests": [ OK ] TestExperimentalPjrtTpu.test_runtime_device_attributes
Step #4 - "run_e2e_tests": [ RUN ] TestExperimentalPjrtTpu.test_spawn_threads
Step #4 - "run_e2e_tests": [ OK ] TestExperimentalPjrtTpu.test_spawn_threads
Step #4 - "run_e2e_tests": [ RUN ] TestExperimentalPjrtTpu.test_xla_devices_multiprocess
Step #4 - "run_e2e_tests": [ OK ] TestExperimentalPjrtTpu.test_xla_devices_multiprocess
Step #4 - "run_e2e_tests": [ RUN ] TestExperimentalPjrtTpu.test_xla_devices_single_process_all_chips
Step #4 - "run_e2e_tests": [ OK ] TestExperimentalPjrtTpu.test_xla_devices_single_process_all_chips
Step #4 - "run_e2e_tests": [ RUN ] TestExperimentalPjrtTpu.test_xla_devices_single_process_one_chip
Step #4 - "run_e2e_tests": [ OK ] TestExperimentalPjrtTpu.test_xla_devices_single_process_one_chip
Step #4 - "run_e2e_tests": [ RUN ] TestExperimentalPjrtTpu.test_xla_devices_single_process_one_chip_one_device_spawn
Step #4 - "run_e2e_tests": [ OK ] TestExperimentalPjrtTpu.test_xla_devices_single_process_one_chip_one_device_spawn
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Ran 15 tests in 119.963s
Step #4 - "run_e2e_tests":
Step #4 - "run_e2e_tests": OK (skipped=2)
Step #4 - "run_e2e_tests": + python3 /src/pytorch/xla/test/pjrt/test_collective_ops_tpu.py
Step #4 - "run_e2e_tests": Running tests under Python 3.10.13: /usr/local/bin/python3
Step #4 - "run_e2e_tests": [ RUN ] TestCollectiveOpsTpu.test_all_gather_pinned
Step #4 - "run_e2e_tests": [ OK ] TestCollectiveOpsTpu.test_all_gather_pinned
Step #4 - "run_e2e_tests": [ RUN ] TestCollectiveOpsTpu.test_all_gather_unpinned
Step #4 - "run_e2e_tests": [ OK ] TestCollectiveOpsTpu.test_all_gather_unpinned
Step #4 - "run_e2e_tests": [ RUN ] TestCollectiveOpsTpu.test_all_reduce_pinned
Step #4 - "run_e2e_tests": [ OK ] TestCollectiveOpsTpu.test_all_reduce_pinned
Step #4 - "run_e2e_tests": [ RUN ] TestCollectiveOpsTpu.test_all_reduce_unpinned
Step #4 - "run_e2e_tests": [ OK ] TestCollectiveOpsTpu.test_all_reduce_unpinned
Step #4 - "run_e2e_tests": [ RUN ] TestCollectiveOpsTpu.test_all_to_all_pinned
Step #4 - "run_e2e_tests": [ OK ] TestCollectiveOpsTpu.test_all_to_all_pinned
Step #4 - "run_e2e_tests": [ RUN ] TestCollectiveOpsTpu.test_all_to_all_unpinned
Step #4 - "run_e2e_tests": [ OK ] TestCollectiveOpsTpu.test_all_to_all_unpinned
Step #4 - "run_e2e_tests": [ RUN ] TestCollectiveOpsTpu.test_broadcast_master_param_synchronized_parameters
Step #4 - "run_e2e_tests": [ OK ] TestCollectiveOpsTpu.test_broadcast_master_param_synchronized_parameters
Step #4 - "run_e2e_tests": [ RUN ] TestCollectiveOpsTpu.test_broadcast_master_param_unsynchronized_parameters
Step #4 - "run_e2e_tests": [ OK ] TestCollectiveOpsTpu.test_broadcast_master_param_unsynchronized_parameters
Step #4 - "run_e2e_tests": [ RUN ] TestCollectiveOpsTpu.test_reduce_scatter_pinned
Step #4 - "run_e2e_tests": [ OK ] TestCollectiveOpsTpu.test_reduce_scatter_pinned
Step #4 - "run_e2e_tests": [ RUN ] TestCollectiveOpsTpu.test_reduce_scatter_unpinned
Step #4 - "run_e2e_tests": [ OK ] TestCollectiveOpsTpu.test_reduce_scatter_unpinned
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Ran 10 tests in 124.602s
Step #4 - "run_e2e_tests":
Step #4 - "run_e2e_tests": OK
Step #4 - "run_e2e_tests": + python3 /src/pytorch/xla/test/spmd/test_xla_sharding.py
Step #4 - "run_e2e_tests": .......E...ss......................................
Step #4 - "run_e2e_tests": ======================================================================
Step #4 - "run_e2e_tests": ERROR: test_from_cpu_shards_2d (__main__.BasicShardingTest)
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Traceback (most recent call last):
Step #4 - "run_e2e_tests": File "/src/pytorch/xla/test/spmd/test_xla_sharding.py", line 948, in test_from_cpu_shards_2d
Step #4 - "run_e2e_tests": self.assertTrue(torch.allclose(global_tensor.cpu(), expected))
Step #4 - "run_e2e_tests": RuntimeError: The size of tensor a (2) must match the size of tensor b (4) at non-singleton dimension 1
Step #4 - "run_e2e_tests":
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Ran 51 tests in 2.859s
Step #4 - "run_e2e_tests":
Step #4 - "run_e2e_tests": FAILED (errors=1, skipped=2)
Step #4 - "run_e2e_tests": ++ kubectl get pod/xla-test-job-jtjzg -o 'jsonpath={.status.containerStatuses[?(@.name=="xla-test")].state.terminated.exitCode}'
Step #4 - "run_e2e_tests": + exit
Finished Step #4 - "run_e2e_tests"
PUSH
DONE
Loading