Add all-gather coalescing for FSDP/ZeRO1 #5624

jeffhataws · 2023-09-20T00:20:53Z

(Replaced by #5950)

This PR adds all-gather coalescence support and use that in FSDP/ZeRO1. This PR is to be used in conjunction with openxla/xla#5740 .

A separate and related PR for reduce-scatter coalescence that also enables using reduce-scatter's scale param in FSDP is #5938.

This is a revival of #4145 . Will need to address the comments.

JackCaoG · 2023-09-20T01:29:07Z

@alanwaketan can you take a look?

alanwaketan · 2023-10-03T21:37:19Z

@jeffhataws Can you double check the CI failures?

jeffhataws · 2023-10-04T21:46:46Z

@jeffhataws Can you double check the CI failures?

Thanks. Since it depends on openxla/xla#5740 I will need to take care of merging that first. So let's leave this open for now.

Imported from GitHub PR #5740 This PR adds tuple input support to all-gather and reduce-scatter. This is a revival of part of tensorflow/tensorflow#58377 and to be used in conjunction with pytorch/xla#5624 . In FSDP, different layers' weights need to be all-gathered/reduced-scatter during training. If some layers are small, multiple layers' weights can be aggregated for more efficient data transfer (same concept as bucket_cap_mb in DDP). With existing all-gather and reduce-scatter in PyTorch-XLA, you would have to do the bucketing and decomposing outside of the operation. This PR enables multiple different tensors to be all-gathered/reduce-scatter, keeping the original tensor shapes to enable bucketing and decomposing optimizations inside the operation. Original PR has token support like the token used for allreduce to ensure order between CCops. That will be separate PR if needed. Copybara import of the project: -- 7ea1159 by Junmin Hao <junminh@amazon.com>: Add Tuple input and token support to all-gather and reduce-scatter. Committer: Junmin Hao <junminh@amazon.com> -- cdb873e by Junmin Hao <junminh@amazon.com>: lint fix -- aad3521 by Jeffrey Huynh <jthuynh@amazon.com>: Fix hlo_verifier_test failure due to changed expectation -- 32e8145 by Jeffrey Huynh <jthuynh@amazon.com>: Separate the token change out into a separate PR with RFC. -- b301c2a by Jeffrey Huynh <jthuynh@amazon.com>: Change *WithToken tests to *WithTuple -- 5890278 by Jeffrey Huynh <jthuynh@amazon.com>: Fix missing parenthesis Merging this change closes #5740 COPYBARA_INTEGRATE_REVIEW=#5740 from jeffhataws:ag_rs_coalesce_revived 14e09f0 PiperOrigin-RevId: 573976449

Imported from GitHub PR openxla/xla#5740 This PR adds tuple input support to all-gather and reduce-scatter. This is a revival of part of #58377 and to be used in conjunction with pytorch/xla#5624 . In FSDP, different layers' weights need to be all-gathered/reduced-scatter during training. If some layers are small, multiple layers' weights can be aggregated for more efficient data transfer (same concept as bucket_cap_mb in DDP). With existing all-gather and reduce-scatter in PyTorch-XLA, you would have to do the bucketing and decomposing outside of the operation. This PR enables multiple different tensors to be all-gathered/reduce-scatter, keeping the original tensor shapes to enable bucketing and decomposing optimizations inside the operation. Original PR has token support like the token used for allreduce to ensure order between CCops. That will be separate PR if needed. Copybara import of the project: -- 7ea1159a1464efddebe9384e87ed6df504d89b2e by Junmin Hao <junminh@amazon.com>: Add Tuple input and token support to all-gather and reduce-scatter. Committer: Junmin Hao <junminh@amazon.com> -- cdb873e6d97f5f12b3d3c587bb5782d58e3554c5 by Junmin Hao <junminh@amazon.com>: lint fix -- aad352117ba950ac5ae62330e3980f4b5898a701 by Jeffrey Huynh <jthuynh@amazon.com>: Fix hlo_verifier_test failure due to changed expectation -- 32e814524b88a474af5e4e904c0dd19841430b86 by Jeffrey Huynh <jthuynh@amazon.com>: Separate the token change out into a separate PR with RFC. -- b301c2a2a5b52180f9e9626173e6b67a78782960 by Jeffrey Huynh <jthuynh@amazon.com>: Change *WithToken tests to *WithTuple -- 5890278fc16c9f900782d32a92d40ecf548aea85 by Jeffrey Huynh <jthuynh@amazon.com>: Fix missing parenthesis Merging this change closes #5740 PiperOrigin-RevId: 573976449

jeffhataws · 2023-10-20T02:44:38Z

@alanwaketan what's the best way to check this against the openxla with merged openxla/xla#5740?

alanwaketan · 2023-10-20T23:49:56Z

@alanwaketan what's the best way to check this against the openxla with merged openxla/xla#5740?

I'm working on a pin update. Will loop you in once that PR is up.

jeffhataws · 2023-10-31T19:51:22Z

@alanwaketan what's the best way to check this against the openxla with merged openxla/xla#5740?

I'm working on a pin update. Will loop you in once that PR is up.

@alanwaketan just want to check how things are going with the pin update. Also, how do I make the automatic checks to run on this PR?

jeffhataws · 2023-11-09T03:40:15Z

@alanwaketan @JackCaoG will you help ensure this is in 2.2?

JackCaoG · 2023-11-09T17:49:36Z

@jeffhataws can you rebase this pr? Then we can start reviewing it. Thanks!

alanwaketan · 2023-11-14T19:43:19Z

@alanwaketan what's the best way to check this against the openxla with merged openxla/xla#5740?

I'm working on a pin update. Will loop you in once that PR is up.

The pin update is completed. So, please rebase and then I will take a look.

jeffhataws · 2023-11-16T22:06:52Z

@alanwaketan what's the best way to check this against the openxla with merged openxla/xla#5740?

I'm working on a pin update. Will loop you in once that PR is up.

The pin update is completed. So, please rebase and then I will take a look.

Thanks alanwaketan. I have rebased. Please take a look.

jeffhataws · 2023-11-17T23:43:16Z

@alanwaketan @JackCaoG I am not sure how to reproduce/debug the above errors.

test/run_tests.sh passes for me on CPU machine with export PJRT_DEVICE=CPU.

When I run test/cpp/run_tests.sh with export PJRT_DEVICE=CPU I get a hang:

Running all cpp test...
+ '[' /tmp/pytorch_cpp_test.log '!=' '' ']'
+ bazel test --config=tpu //torch_xla/csrc/runtime:all //test/cpp:all --test_timeout 1000
(hangs)

jeffhataws · 2023-11-18T00:44:55Z

@alanwaketan @JackCaoG I am not sure how to reproduce/debug the above errors.

test/run_tests.sh passes for me on CPU machine with export PJRT_DEVICE=CPU.

When I run test/cpp/run_tests.sh with export PJRT_DEVICE=CPU I get a hang:
Running all cpp test...
+ '[' /tmp/pytorch_cpp_test.log '!=' '' ']'
+ bazel test --config=tpu //torch_xla/csrc/runtime:all //test/cpp:all --test_timeout 1000
(hangs)

It wasn't hanging. After a while, it shows:

+ bazel test --config=tpu //torch_xla/csrc/runtime:all //test/cpp:all --test_timeout 1000
^[[A^[[A^[[A^[[A//test/cpp:test_aten_xla_tensor_1                                        PASSED in 23.4s
//test/cpp:test_aten_xla_tensor_2                                        PASSED in 18.5s
//test/cpp:test_aten_xla_tensor_3                                        PASSED in 44.9s
//test/cpp:test_aten_xla_tensor_4                                        PASSED in 15.4s
//test/cpp:test_aten_xla_tensor_5                                        PASSED in 85.5s
//test/cpp:test_aten_xla_tensor_6                                        PASSED in 44.5s
//test/cpp:test_ir                                                       PASSED in 0.5s
//test/cpp:test_lazy                                                     PASSED in 0.5s
//test/cpp:test_replication                                              PASSED in 0.6s
//test/cpp:test_tensor                                                   PASSED in 101.9s
//test/cpp:test_xla_sharding                                             PASSED in 0.5s
//torch_xla/csrc/runtime:cache_test                                      PASSED in 0.0s
//torch_xla/csrc/runtime:pjrt_computation_client_test                    PASSED in 0.7s
//torch_xla/csrc/runtime:sys_util_test                                   PASSED in 0.0s
//torch_xla/csrc/runtime:util_test                                       PASSED in 0.1s
//torch_xla/csrc/runtime:xla_util_test                                   PASSED in 0.4s

Executed 16 out of 16 tests: 16 tests pass.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.

JackCaoG · 2023-11-18T01:28:04Z

You didn't add any cpp test and our cpp test doesn't test distributed so that should be fine? I do see a bunch of python test failing in the CI with

ImportError: /opt/conda/lib/python3.8/site-packages/torch_xla-2.2.0+git08075a8-py3.8-linux-x86_64.egg/_XLAC.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN9torch_xla14tensor_methods10all_gatherERKN3c1013intrusive_ptrINS_9XLATensorENS1_6detail34intrusive_target_default_null_typeIS3_EEEEllSt6vectorISA_IlSaIlEESaISC_EEb

in https://github.com/pytorch/xla/actions/runs/6896526398/job/18766614593?pr=5624

If I have to guess it is that the build pass but when we try to import the XLAC, we find that there are some functions that only has the header but missing in the cpp file? this one seems to be tensor_methods::all_gather. If you try to build this locally and just do a import torch_xla you should see the same failure.

JackCaoG · 2023-11-18T01:28:29Z

@alanwaketan is out next week, I will work with you to try to land this change before branch cut(or we will cherry-pick).

Also allow using reduce-scatter's scale param in FSDP. (revived pytorch#4145)

jeffhataws · 2023-11-18T23:30:19Z

You didn't add any cpp test and our cpp test doesn't test distributed so that should be fine? I do see a bunch of python test failing in the CI with
ImportError: /opt/conda/lib/python3.8/site-packages/torch_xla-2.2.0+git08075a8-py3.8-linux-x86_64.egg/_XLAC.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN9torch_xla14tensor_methods10all_gatherERKN3c1013intrusive_ptrINS_9XLATensorENS1_6detail34intrusive_target_default_null_typeIS3_EEEEllSt6vectorISA_IlSaIlEESaISC_EEb
in https://github.com/pytorch/xla/actions/runs/6896526398/job/18766614593?pr=5624

If I have to guess it is that the build pass but when we try to import the XLAC, we find that there are some functions that only has the header but missing in the cpp file? this one seems to be tensor_methods::all_gather. If you try to build this locally and just do a import torch_xla you should see the same failure.

Fixed. Removed the previous all_gather method by mistake.

…tter Imported from GitHub PR openxla#5740 This PR adds tuple input support to all-gather and reduce-scatter. This is a revival of part of tensorflow/tensorflow#58377 and to be used in conjunction with pytorch/xla#5624 . In FSDP, different layers' weights need to be all-gathered/reduced-scatter during training. If some layers are small, multiple layers' weights can be aggregated for more efficient data transfer (same concept as bucket_cap_mb in DDP). With existing all-gather and reduce-scatter in PyTorch-XLA, you would have to do the bucketing and decomposing outside of the operation. This PR enables multiple different tensors to be all-gathered/reduce-scatter, keeping the original tensor shapes to enable bucketing and decomposing optimizations inside the operation. Original PR has token support like the token used for allreduce to ensure order between CCops. That will be separate PR if needed. Copybara import of the project: -- 7ea1159 by Junmin Hao <junminh@amazon.com>: Add Tuple input and token support to all-gather and reduce-scatter. Committer: Junmin Hao <junminh@amazon.com> -- cdb873e by Junmin Hao <junminh@amazon.com>: lint fix -- aad3521 by Jeffrey Huynh <jthuynh@amazon.com>: Fix hlo_verifier_test failure due to changed expectation -- 32e8145 by Jeffrey Huynh <jthuynh@amazon.com>: Separate the token change out into a separate PR with RFC. -- b301c2a by Jeffrey Huynh <jthuynh@amazon.com>: Change *WithToken tests to *WithTuple -- 5890278 by Jeffrey Huynh <jthuynh@amazon.com>: Fix missing parenthesis Merging this change closes openxla#5740 COPYBARA_INTEGRATE_REVIEW=openxla#5740 from jeffhataws:ag_rs_coalesce_revived 14e09f0 PiperOrigin-RevId: 573976449

jeffhataws · 2023-11-20T18:52:27Z

Fixed the lint error.

jeffhataws · 2023-11-22T21:43:20Z

We can keep trying to fix test errors, but I am not sure if it will be eaiser to split all-gather and reduce-scatter into separate prs. @jeffhataws I take that this pr works in your local forks, not sure what's the difference. Test failures seem real.

Yeah the code was ported from an old version of torch/xla so there were some merge errors. Plus the final version of openxla change openxla/xla#5740 doesn't have token support, so I need to make the corresponding change here.

…thout token

JackCaoG · 2023-11-27T20:08:07Z

hmm, this error seems real

2023-11-27 19:09:57.910045: E external/xla/xla/status_macros.cc:54] INTERNAL: RET_CHECK failure (external/xla/xla/service/shape_inference.cc:2155) scatter_dimension < operand_shape->rank() 
*** Begin stack trace ***
	tsl::CurrentStackTrace[abi:cxx11]()
	
	xla::status_macros::MakeErrorStream::Impl::GetStatus()
	xla::ShapeInference::InferReduceScatterShape(absl::lts_20230125::Span<xla::Shape const* const>, long, long)
	
	
	xla::XlaBuilder::ReportErrorOrReturn(absl::lts_20230125::FunctionRef<absl::lts_20230125::StatusOr<xla::XlaOp> ()>)
	xla::XlaBuilder::ReduceScatter(xla::XlaOp, xla::XlaComputation const&, long, long,

…ter tuple change without token

jeffhataws · 2023-11-28T21:12:23Z

One of the CPU workflows failed with this:

[25,304 / 29,344] Compiling xla/mlir_hlo/mhlo/transforms/legalize_to_linalg/legalize_to_linalg.cc; 33s local ... (48 actions, 47 running)
[28,230 / 30,817] Compiling xla/service/gpu/cub_sort_kernel.cu.cc; 31s local ... (48 actions, 47 running)
[29,163 / 30,817] Compiling xla/service/algebraic_simplifier.cc; 29s local ... (48 actions, 47 running)
/home/ec2-user/actions-runner/_work/_temp/c18e5d69-957a-4e96-8ea6-714d1254694a.sh: line 1: 27242 Killed                  docker exec --privileged -u jenkins "${pid}" bash -c '.circleci/test.sh'
Error: Process completed with exit code 137.

JackCaoG · 2023-11-28T21:27:00Z

hmm, seems like vm oom when building pytorch/xla...

JackCaoG · 2023-11-28T21:28:52Z

Let's ignore the CPU failure and focus on GPU for now. GPU failures seems real.

jeffhataws · 2023-11-29T19:41:36Z

All tests passing with Reduce-Scatter change separated out in #5938 .

torch_xla/csrc/ops/all_gather.cpp

torch_xla/csrc/cross_replica_reduces.cpp

test/test_torch_distributed_xla_backend.py

torch_xla/core/xla_model.py

JackCaoG · 2023-11-30T02:12:38Z

torch_xla/csrc/cross_replica_reduces.cpp

+    xla::XlaOp all_gather_result;
+    if (pin_layout) {
+      all_gather_result = xla::AllGather(
+          xla::Tuple(inputs[0].builder(), type_ctx.second.ops), dim,


I think this means even if there is single element in the all-gather, we will wrap it inside the tuple.. I need to check with xla teams whether this has any speed implications.

JackCaoG · 2023-11-30T02:15:40Z

torch_xla/csrc/cross_replica_reduces.cpp

+      }
+    } else {
+      result[0] = all_gather_result;
+    }


I don't see how token is being used here, in all_reduce, we manually append it at the end of ops

for (auto& type_ctx : redux.contexts) { xla::XlaOp token_op = MaybeConvertTo(chained_token, type_ctx.first); type_ctx.second.ops.push_back(token_op); type_ctx.second.operand_shapes.push_back( ShapeHelper::ShapeOfXlaOp(token_op));

I saw the GetOperandList below, but I think this does not gurante when you have multiple types, each types has a token.

Fix the usage of token.

For multiple types case, I think for now we should ensure same type in the list.

torch_xla/csrc/init_python_bindings.cpp

JackCaoG · 2023-11-30T02:29:38Z

torch_xla/csrc/tensor_methods.cpp

+  }
+  return {result, torch::lazy::Value(node, inputs.size())};
+}
+
 XLATensorPtr all_gather(const XLATensorPtr& input, int64_t dim,


I am not why does it compile when below code still pass a single IR variable to torch::lazy::MakeNode<AllGather>, while you change the constructor to take arrayRef, maybe arrayRef have a default constructor.

JackCaoG

I left a few comments, overall I think it is better to implement a new BuildAllGatherCoalesced instead of modifying the existing BuildAllGather. The way you do it today will change the HLO generated for the single tensor all-gather. Given that branch cut is this Friday, I don't think we have time to do the performance benchmarking to make sure this is regression free, it is safer to add new features while no touch the existing logic.

JackCaoG · 2023-11-30T02:38:44Z

oh I know what's going on. Can you create a branch from pytorch/xla directly instead of creating from a fork. Fork can not used our cache so compilation will take much longer and easier to fail. I already give you write access to the project so you should be able to create a new branch from our repo directly.

jeffhataws · 2023-11-30T04:03:59Z

oh I know what's going on. Can you create a branch from pytorch/xla directly instead of creating from a fork. Fork can not used our cache so compilation will take much longer and easier to fail. I already give you write access to the project so you should be able to create a new branch from our repo directly.

Thanks. Changed to PR from a branch on pytorch/xla #5950 .

…lass

…tter Imported from GitHub PR openxla#5740 This PR adds tuple input support to all-gather and reduce-scatter. This is a revival of part of tensorflow/tensorflow#58377 and to be used in conjunction with pytorch/xla#5624 . In FSDP, different layers' weights need to be all-gathered/reduced-scatter during training. If some layers are small, multiple layers' weights can be aggregated for more efficient data transfer (same concept as bucket_cap_mb in DDP). With existing all-gather and reduce-scatter in PyTorch-XLA, you would have to do the bucketing and decomposing outside of the operation. This PR enables multiple different tensors to be all-gathered/reduce-scatter, keeping the original tensor shapes to enable bucketing and decomposing optimizations inside the operation. Original PR has token support like the token used for allreduce to ensure order between CCops. That will be separate PR if needed. Copybara import of the project: -- 7ea1159 by Junmin Hao <junminh@amazon.com>: Add Tuple input and token support to all-gather and reduce-scatter. Committer: Junmin Hao <junminh@amazon.com> -- cdb873e by Junmin Hao <junminh@amazon.com>: lint fix -- aad3521 by Jeffrey Huynh <jthuynh@amazon.com>: Fix hlo_verifier_test failure due to changed expectation -- 32e8145 by Jeffrey Huynh <jthuynh@amazon.com>: Separate the token change out into a separate PR with RFC. -- b301c2a by Jeffrey Huynh <jthuynh@amazon.com>: Change *WithToken tests to *WithTuple -- 5890278 by Jeffrey Huynh <jthuynh@amazon.com>: Fix missing parenthesis Merging this change closes openxla#5740 COPYBARA_INTEGRATE_REVIEW=openxla#5740 from jeffhataws:ag_rs_coalesce_revived 14e09f0 PiperOrigin-RevId: 573976449

JackCaoG requested a review from alanwaketan September 20, 2023 01:28

jeffhataws mentioned this pull request Sep 22, 2023

Add tuple input support to all-gather and reduce-scatter openxla/xla#5740

Closed

jeffhataws force-pushed the cc_coalesce_revival branch from 79676e7 to 78fc6d3 Compare October 12, 2023 16:07

jeffhataws closed this Oct 18, 2023

jeffhataws force-pushed the cc_coalesce_revival branch from 78fc6d3 to 8f45cae Compare October 18, 2023 03:19

jeffhataws reopened this Oct 19, 2023

jeffhataws force-pushed the cc_coalesce_revival branch 2 times, most recently from 2e861ff to 76a2f0f Compare October 20, 2023 02:42

jeffhataws force-pushed the cc_coalesce_revival branch from 76a2f0f to 1d8671f Compare November 16, 2023 21:09

Add all-gather and reduce-scatter coalescence support for FSDP.

57e6b36

Also allow using reduce-scatter's scale param in FSDP. (revived pytorch#4145)

jeffhataws force-pushed the cc_coalesce_revival branch from 1d8671f to 57e6b36 Compare November 18, 2023 23:29

Fix allgather to be compatible with openxla allgather tuple change wi…

ec27f90

…thout token

Fix reduce-scatter-coalesce to be compatible with openxla reduce-scat…

f9f3a71

…ter tuple change without token

jeffhataws requested a review from JackCaoG November 28, 2023 20:36

Separate out the reduce-scatter-coalesce changes into a separate PR

5843b43

jeffhataws changed the title ~~Add all-gather/reduce-scatter coalescee for FSDP/ZeRO1~~ Add all-gather coalescing for FSDP/ZeRO1 Nov 29, 2023

jeffhataws mentioned this pull request Nov 29, 2023

Add reduce-scatter coalescing for FSDP/ZeRO1 #5938

Closed

jeffhataws commented Nov 29, 2023

View reviewed changes

torch_xla/csrc/ops/all_gather.cpp Outdated Show resolved Hide resolved

jeffhataws commented Nov 29, 2023

View reviewed changes

torch_xla/csrc/cross_replica_reduces.cpp Outdated Show resolved Hide resolved

Some cleanups

8e6715e

JackCaoG reviewed Nov 30, 2023

View reviewed changes

test/test_torch_distributed_xla_backend.py Show resolved Hide resolved

JackCaoG reviewed Nov 30, 2023

View reviewed changes

torch_xla/core/xla_model.py Show resolved Hide resolved

JackCaoG reviewed Nov 30, 2023

View reviewed changes

torch_xla/csrc/init_python_bindings.cpp Show resolved Hide resolved

JackCaoG reviewed Nov 30, 2023

View reviewed changes

jeffhataws mentioned this pull request Nov 30, 2023

Add all-gather coalescing for FSDP/ZeRO1 #5950

Merged

Add separate BuildAllGatherCoalesced builder and AllGatherCoalesced c…

3cd39ea

…lass

jeffhataws mentioned this pull request Nov 30, 2023

Add reduce-scatter coalescing for FSDP/ZeRO1 #5956

Merged

jeffhataws closed this Nov 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add all-gather coalescing for FSDP/ZeRO1 #5624

Add all-gather coalescing for FSDP/ZeRO1 #5624

jeffhataws commented Sep 20, 2023 •

edited

Loading

JackCaoG commented Sep 20, 2023

alanwaketan commented Oct 3, 2023

jeffhataws commented Oct 4, 2023

jeffhataws commented Oct 20, 2023

alanwaketan commented Oct 20, 2023

jeffhataws commented Oct 31, 2023

jeffhataws commented Nov 9, 2023

JackCaoG commented Nov 9, 2023

alanwaketan commented Nov 14, 2023

jeffhataws commented Nov 16, 2023

jeffhataws commented Nov 17, 2023 •

edited

Loading

jeffhataws commented Nov 18, 2023

JackCaoG commented Nov 18, 2023

JackCaoG commented Nov 18, 2023

jeffhataws commented Nov 18, 2023

jeffhataws commented Nov 20, 2023

jeffhataws commented Nov 22, 2023

JackCaoG commented Nov 27, 2023

jeffhataws commented Nov 28, 2023

JackCaoG commented Nov 28, 2023

JackCaoG commented Nov 28, 2023

jeffhataws commented Nov 29, 2023

JackCaoG Nov 30, 2023

JackCaoG Nov 30, 2023

JackCaoG Nov 30, 2023

jeffhataws Nov 30, 2023

JackCaoG Nov 30, 2023

JackCaoG left a comment

JackCaoG commented Nov 30, 2023

jeffhataws commented Nov 30, 2023

Add all-gather coalescing for FSDP/ZeRO1 #5624

Add all-gather coalescing for FSDP/ZeRO1 #5624

Conversation

jeffhataws commented Sep 20, 2023 • edited Loading

JackCaoG commented Sep 20, 2023

alanwaketan commented Oct 3, 2023

jeffhataws commented Oct 4, 2023

jeffhataws commented Oct 20, 2023

alanwaketan commented Oct 20, 2023

jeffhataws commented Oct 31, 2023

jeffhataws commented Nov 9, 2023

JackCaoG commented Nov 9, 2023

alanwaketan commented Nov 14, 2023

jeffhataws commented Nov 16, 2023

jeffhataws commented Nov 17, 2023 • edited Loading

jeffhataws commented Nov 18, 2023

JackCaoG commented Nov 18, 2023

JackCaoG commented Nov 18, 2023

jeffhataws commented Nov 18, 2023

jeffhataws commented Nov 20, 2023

jeffhataws commented Nov 22, 2023

JackCaoG commented Nov 27, 2023

jeffhataws commented Nov 28, 2023

JackCaoG commented Nov 28, 2023

JackCaoG commented Nov 28, 2023

jeffhataws commented Nov 29, 2023

JackCaoG Nov 30, 2023

Choose a reason for hiding this comment

JackCaoG Nov 30, 2023

Choose a reason for hiding this comment

JackCaoG Nov 30, 2023

Choose a reason for hiding this comment

jeffhataws Nov 30, 2023

Choose a reason for hiding this comment

JackCaoG Nov 30, 2023

Choose a reason for hiding this comment

JackCaoG left a comment

Choose a reason for hiding this comment

JackCaoG commented Nov 30, 2023

jeffhataws commented Nov 30, 2023

jeffhataws commented Sep 20, 2023 •

edited

Loading

jeffhataws commented Nov 17, 2023 •

edited

Loading