Implement ordered list state for FnApi. #30317

shunping · 2024-02-14T20:49:36Z

First attempt to implement ordered list state for FnApi. Notice that caching has not been implemented yet.

Reference:
https://docs.google.com/document/d/1U77sAvE6Iy9XsVruRYHxPdFji7nqS6HPi1XU8fhyrxs/

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

If there are changes on a state after we obtain iterators from calling read() and readRange(), the behavior of these pre-existing iterators were incorrect in the previous implementation. The change introduced here will make sure that these iterators will still work as if no local change is made.

shunping · 2024-02-15T16:53:53Z

Run Python PreCommit 3.9

scwhittle

Didn't get yet to implementation/test

scwhittle · 2024-02-15T19:40:11Z

model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto

+// A response to the get state request for an ordered list.
+message OrderedListStateGetResponse {
+  bytes continuation_token = 1;
+  bytes data = 2;


how do we return multiple elements if the request was a range?

should we return the sort-key for elements, that seems like part of the user-data for example if it's some id/timestamp the user might want it back instead of having to duplicate it in the payload as well.

should this be repeated OrderedListEntry instead of data?

For efficiency reasons we have generally not split individual elements up into individual field protos, and provided them as contiguous bytes of data. Makes sense to do that here. But the coding should be specified (e.g. is this a concatenation of several encoded KV<sort-key, value>s?

We should also consider if there are considerations in the encoding of sort-key. If they're (typically) timestamps, bigendian might be preferable to varint.

how do we return multiple elements if the request was a range?

We concatenate the encoded entries in a byte array and send them back in chunks with corresponding continuation token.

should this be repeated OrderedListEntry instead of data?

I have considered this option. Besides the efficiency reason @robertwb mentioned, I also find that representing the response as bytes has an advantage of reusing the existing code in https://github.com/apache/beam/blob/3693174c0421d0ff049042ca283db633431892ef/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java. This iterator is used to parsed the returned data block (in not only OrderedListState but also Multimap, Bag, etc) and it supports blocks even if the boundary is not aligned with entries/elements. I think this is not achievable with OrderedListEntry.

But the coding should be specified (e.g. is this a concatenation of several encoded KV<sort-key, value>s

That's right. It is basically the concatenation of encoded TimestampedValue, since TimestampedValue is already in use in the sdk interface of OrderedListState

beam/sdks/java/core/src/main/java/org/apache/beam/sdk/state/OrderedListState.java

Line 29 in 3693174

extends GroupingState<TimestampedValue<T>, Iterable<TimestampedValue<T>>> {

scwhittle · 2024-02-15T19:41:18Z

model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto

+// timestamp range is not specified, the runner should use [MIN_INT64,
+// MAX_INT64) by default.
+message OrderedListStateGetRequest {
+  bytes continuation_token = 1;


comment on continuation token, ie should be returned by previous response and the key/range should match the previous request generating the token

Do we want to require returning the range if there's a continuation token involved? Or should they be mutually exclusive?

comment on continuation token ...

Good idea! I will add that when I revise the code.

Do we want to require returning the range if there's a continuation token involved? Or should they be mutually exclusive?

I don't think the range is a hard requirement.
In fact in my simple implementation of the fake client, I put the current sort key and the current block index into the continuation token:

beam/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/FakeBeamFnStateClient.java

Line 228 in 5d7cd5e

// The continuation format here is the sort key (long) followed by an index (int)

.
In other implementation, you may need the range, but I think it is implementation-dependent. That's why I hesitate to put a range/sort key as a separate field when continuation token is present.

I thought the continuation token should allow the runner to uniquely identify where to find the next block of data. Is that what you mean by "mutually exclusive"?

Yes, that's what I meant by mutually exclusive. If we allow providing both, we have to figure out what to do when they disagree, and there's no good usecase for that.

sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/FnApiStateAccessor.java

model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto

robertwb · 2024-02-15T22:17:08Z

model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto

+// timestamp range is not specified, the runner should use [MIN_INT64,
+// MAX_INT64) by default.
+message OrderedListStateGetRequest {
+  bytes continuation_token = 1;


Do we want to require returning the range if there's a continuation token involved? Or should they be mutually exclusive?

robertwb · 2024-02-15T22:18:56Z

model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto

+// A response to the get state request for an ordered list.
+message OrderedListStateGetResponse {
+  bytes continuation_token = 1;
+  bytes data = 2;


For efficiency reasons we have generally not split individual elements up into individual field protos, and provided them as contiguous bytes of data. Makes sense to do that here. But the coding should be specified (e.g. is this a concatenation of several encoded KV<sort-key, value>s?

robertwb · 2024-02-15T22:27:17Z

model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto

+
+// A request to update an ordered list
+message OrderedListStateUpdateRequest {
+  // when the request is processed, deletes should always happen before inserts.


Should we instead split this up into two separate requests, e.g. OrderedListStateInsertRequest and OrderedListStateDeleteRequest?

Discussion 2 in the design doc talked about this, but there is no conclusion on that https://docs.google.com/document/d/1U77sAvE6Iy9XsVruRYHxPdFji7nqS6HPi1XU8fhyrxs/

I am fine with splitting them though.

It would be helpful to outline the pro/con in the design doc of little decisions like, and note which one was chosen and why.

For example one benefit to splitting the requests is to avoid ordering issues. We would have to spec that either the inserts or deletes happen first, even though they are in one request together. It is a bit confusing. And then if you want them in the other order, you still have to make two requests but each one has an empty field.

And note whether there is an efficiency consideration.

Yep. I am planning to add an addendum to the original design doc to summarize the decisions we make here. We should have that after this round of review completes.

model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto

robertwb · 2024-02-15T22:37:05Z

model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto

+// A response to the get state request for an ordered list.
+message OrderedListStateGetResponse {
+  bytes continuation_token = 1;
+  bytes data = 2;


We should also consider if there are considerations in the encoding of sort-key. If they're (typically) timestamps, bigendian might be preferable to varint.

sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/FakeBeamFnStateClient.java

* The range information is placed in the state key of ordered list * For consistency, we reuse the existing get request and response mesasages of other states like Bag, MultiMap, etc.

* Reuse existing messages of clear and append.

* Replace String::size() > 0 with String::isEmpty() * Return this in readLater and readRangeLater instead of throwing an exception * Remove the added SupressWarnings("unchecked")

Previously, we used a repeated OrderedListEntry field in the AppendRequest particularly for ordered list state. For consistency, we now get rid of that and use the same data field as other states.

github-actions · 2024-03-06T17:36:06Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @robertwb for label java.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

shunping · 2024-03-06T20:51:16Z

Run Java PreCommit

sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/FakeBeamFnStateClient.java

sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/OrderedListUserState.java

sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/FakeBeamFnStateClient.java

acrites · 2024-06-11T16:27:51Z

sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/FakeBeamFnStateClient.java

@@ -47,6 +60,7 @@ public class FakeBeamFnStateClient implements BeamFnStateClient {
  private static final int DEFAULT_CHUNK_SIZE = 6;
  private final Map<StateKey, List<ByteString>> data;
  private int currentId;
+  private final Map<StateKey, NavigableSet<Long>> orderedListSortKeysFromStateKey;


This seems like a pretty inefficient data structure to use. You might look at SortedSet (which requires adding a deduplicating id) unless we can use SortedMultiset from Guava. This would allow us to do things like remove all elements in a range without first copying over all keys in the range and then removing them one by one (which requires a hash each time).

Oh, I see this is part of the "fake" client, so maybe we don't care as much about performance?

The structure is used to support multiple ordered list states within one DoFn.

For example, we can have order list state A and B, which have different StateKeys. The outer map is only used once when looking up the right ordered list.

Then under each state key, the NavigableSet, which is internally initialized with a TreeSet, can support removing a range of keys efficiently.

acrites · 2024-06-11T16:30:24Z

sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/FakeBeamFnStateClient.java

+
+      StateKey.Builder keyBuilder = key.toBuilder();
+
+      // clear the range in the state key before using it as a key to store, because ordered list


I don't understand this comment. Are we trying to handle the case where the same state key shows up in initialData multiple times?

The initialData is a list of entries to be stored in the ordered list, which is an idea borrowed from the MultimapUserStateTest.

An entry is specified by the value and a state key (including the name of the state and the range this entry will fall in). Multiple entries may belong to the same state because only the name of the state is the identifier.

When we need to put these entries in the internal structure, we will remove the range field in the state key so that entries of the same state identifier will be put into one bucket.

acrites · 2024-06-11T16:34:58Z

sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/OrderedListUserState.java

+    @Override
+    public void verifyDeterministic() throws NonDeterministicException {
+      verifyDeterministic(
+          this, "TimestampedValueCoder requires a deterministic valueCoder", valueCoder);


Just curious, why is this? It seems like we have no way of overwriting an individual element in the ordered list, so if it gets coded differently it wouldn't matter... It seems weird to be using non-deterministic coders anyway, but how would it affect correctness here?

The coder here is based on the one defined at

beam/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TimestampedValue.java

Line 122 in 462c833

public void verifyDeterministic() throws NonDeterministicException {

.

This is just part of the contract that coders must define, though I agree it's not something that would be likely to get leaked to somewhere that it matters...

In this first version of implementation. I prefer keeping it there so it is consistent with the existing TimeStampedValueCoder defined in TimestampedValue.java. We can surely improve that later.

sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/OrderedListUserState.java

sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/OrderedListUserStateTest.java

…viwer comments.

codecov · 2024-06-11T20:20:42Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.13%. Comparing base (8a64641) to head (b481880).
Report is 1 commits behind head on master.

❗ Current head b481880 differs from pull request most recent head f4c3440

Please upload reports for the commit f4c3440 to get more accurate results.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #30317      +/-   ##
============================================
- Coverage     71.18%   71.13%   -0.06%     
  Complexity     3014     3014              
============================================
  Files          1058     1055       -3     
  Lines        134076   133466     -610     
  Branches       3254     3254              
============================================
- Hits          95437    94935     -502     
+ Misses        35498    35400      -98     
+ Partials       3141     3131      -10

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/FnApiStateAccessor.java

robertwb · 2024-06-11T21:21:47Z

sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/OrderedListUserStateTest.java

+import org.junit.runners.JUnit4;
+
+@RunWith(JUnit4.class)
+public class OrderedListUserStateTest {


Are there also pipeline-level tests that we can now enable for portable runners (and, once it's implemented, Dataflow)?

We have pipeline level tests for states including OrderedListState at java/org/apache/beam/sdk/transforms/ParDoTest.java.

sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/OrderedListUserState.java

robertwb · 2024-06-11T21:26:14Z

sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/OrderedListUserState.java

+    @Override
+    public void verifyDeterministic() throws NonDeterministicException {
+      verifyDeterministic(
+          this, "TimestampedValueCoder requires a deterministic valueCoder", valueCoder);


This is just part of the contract that coders must define, though I agree it's not something that would be likely to get leaked to somewhere that it matters...

robertwb · 2024-06-11T21:28:04Z

sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/OrderedListUserState.java

+
+  public void add(TimestampedValue<T> value) {
+    checkState(
+        !isClosed,


When would the user see a closed state? Is it if they held onto it after the bundle closed or something like that?

The way I implement the closed state in OrderedListUserState is consistent with the other states. The flag isClosed is set when the class method asyncClose() is called.

beam/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/OrderedListUserState.java

Line 276 in 65aef1a

isClosed = true;

This class method is registered in stateFinalizers at FnApiStateAccessor.java

beam/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/FnApiStateAccessor.java

Line 929 in 65aef1a

stateFinalizers.add(rval::asyncClose);

It is then called after each processElement in FnApiDoFnRunner.java

beam/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java

Lines 836 to 842 in 65aef1a

doFnInvoker.invokeGetInitialWatermarkEstimatorState(processContext)))));

} finally {

currentElement = null;

currentRestriction = null;

}

this.stateAccessor.finalizeState();

.

robertwb · 2024-06-11T21:35:15Z

sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/OrderedListUserStateTest.java

+                    OrderedListRange.newBuilder().setStart(sortKey).setEnd(sortKey + 1).build()))
+        .build();
+  }
+}


You've covered a lot of cases here, but this seems like the kind of thing where it'd be good to have fuzz tests too. Say you do dozens of random interspersed reads, writes, and clears. You could do the same thing on a naive List (as long as you kept your range small enough) and verify you got the same result. (Make sure you record a seed so you can reproduce failures.)

That's a good idea. Maybe we can do that in a follow-up PR?

OK. Please file an issue and assign it to yourself.

Done. Opened a issue: #31592.

Thanks @robertwb!

sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/OrderedListUserState.java

robertwb · 2024-06-11T21:44:11Z

sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/OrderedListUserState.java

+  private NavigableMap<Instant, Collection<T>> pendingAdds = Maps.newTreeMap();
+  private TreeRangeSet<Instant> pendingRemoves = TreeRangeSet.create();
+
+  private boolean isCleared = false;


What does this represent, as one can clear various sub-ranges.

This is a flag when a user calls the class method clear() to wipe out all elements in the order list. In this case, we have some special code path, as any previously persistent storage for this state will be cleared once async_close() is invoked.

Might be good to add that to a comment for this variable.

sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/OrderedListUserState.java

… and pendingRemoves during async_close()

shunping · 2024-06-12T13:51:20Z

Run Java PreCommit

shunping · 2024-06-12T13:51:49Z

Run Java_IOs_Direct PreCommit

shunping · 2024-06-13T03:18:36Z

Run Go PreCommit

sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/FnApiStateAccessor.java

robertwb · 2024-06-13T17:29:43Z

sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/OrderedListUserStateTest.java

+                    OrderedListRange.newBuilder().setStart(sortKey).setEnd(sortKey + 1).build()))
+        .build();
+  }
+}


OK. Please file an issue and assign it to yourself.

damccorm

Thanks!

shunping added 8 commits February 13, 2024 15:16

Add request and response proto messages for ordered list state.

b978ca7

Initial implementation of OrderedListState for fnApi.

68abe65

Discard the use of the value coder in FakeBeamFnStateClient.

d06e0a5

Support continuation token for ordered list get request in fake client

f74fc60

Add copyright notices to the new files

f9e48a3

Add binding for ordered list state in fnapi state accessor

a76b523

Clean up comments

8a3b719

shunping marked this pull request as draft February 14, 2024 20:49

github-actions bot added java model labels Feb 14, 2024

Apply spotless and checkStyle to reformat

5d7cd5e

shunping force-pushed the ordered-list-state-2 branch from ac40716 to 5d7cd5e Compare February 15, 2024 15:16

scwhittle requested changes Feb 15, 2024

View reviewed changes

robertwb reviewed Feb 15, 2024

View reviewed changes

shunping added 9 commits February 21, 2024 16:20

Add an encode-only coder for the use in the fake client.

33e748d

Remove request and response messages for ordered list state get.

91b770e

* The range information is placed in the state key of ordered list * For consistency, we reuse the existing get request and response mesasages of other states like Bag, MultiMap, etc.

Remove request and response messages for ordered list state update

4b512ea

* Reuse existing messages of clear and append.

Minor fixes based on feedbacks from reviewers

09db01c

* Replace String::size() > 0 with String::isEmpty() * Return this in readLater and readRangeLater instead of throwing an exception * Remove the added SupressWarnings("unchecked")

Apply spotless

44b6679

Use data field in AppendRequest for ordered list state

868e493

Previously, we used a repeated OrderedListEntry field in the AppendRequest particularly for ordered list state. For consistency, we now get rid of that and use the same data field as other states.

Merge branch 'master' into ordered-list-state-2

ea7ca70

Apply spotless

83bac80

Minor renaming of a variable

90b7f7d

shunping force-pushed the ordered-list-state-2 branch from 5a438a3 to 90b7f7d Compare March 6, 2024 16:40

shunping marked this pull request as ready for review March 6, 2024 16:42

github-actions bot added the Next Action: Reviewers label Mar 6, 2024

Add urn for ordered list state.

a76034f

github-actions bot added the model label Jun 8, 2024

Add ordered list spec to ParDoTranslation.

b481880

shunping force-pushed the ordered-list-state-2 branch from bd5838b to b481880 Compare June 8, 2024 15:49

acrites reviewed Jun 11, 2024

View reviewed changes

sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/FakeBeamFnStateClient.java Outdated Show resolved Hide resolved

acrites reviewed Jun 11, 2024

View reviewed changes

sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/OrderedListUserState.java Show resolved Hide resolved

acrites reviewed Jun 11, 2024

View reviewed changes

sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/FakeBeamFnStateClient.java Outdated Show resolved Hide resolved

acrites reviewed Jun 11, 2024

View reviewed changes

sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/OrderedListUserState.java Outdated Show resolved Hide resolved

acrites reviewed Jun 11, 2024

View reviewed changes

sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/OrderedListUserState.java Show resolved Hide resolved

acrites reviewed Jun 11, 2024

View reviewed changes

sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/OrderedListUserState.java Show resolved Hide resolved

acrites reviewed Jun 11, 2024

View reviewed changes

sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/OrderedListUserStateTest.java Show resolved Hide resolved

Fix an edge case when async called after clear. Minor fix based on re…

65aef1a

…viwer comments.

robertwb reviewed Jun 11, 2024

View reviewed changes

Refactor some variable names. Add a notes on the order of pendingAdds…

c158b32

… and pendingRemoves during async_close()

Merge branch 'master' into ordered-list-state-2

f4c3440

acrites reviewed Jun 13, 2024

View reviewed changes

sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/FnApiStateAccessor.java Show resolved Hide resolved

acrites approved these changes Jun 13, 2024

View reviewed changes

robertwb approved these changes Jun 13, 2024

View reviewed changes

This was referenced Jun 13, 2024

[Feature Request]: Add fuzz tests to ordered list states. #31592

Open

[Bug]: Correctly handle Int64_MAX as a key in ordered list state #31593

Open

damccorm approved these changes Jun 13, 2024

View reviewed changes

damccorm merged commit af31d35 into apache:master Jun 13, 2024
112 of 113 checks passed

shunping mentioned this pull request Aug 29, 2024

Support ordered list states in python sdk and fnapi runner #32326

Merged


		StateKey.Builder keyBuilder = key.toBuilder();

		// clear the range in the state key before using it as a key to store, because ordered list

	doFnInvoker.invokeGetInitialWatermarkEstimatorState(processContext)))));
	} finally {
	currentElement = null;
	currentRestriction = null;
	}

	this.stateAccessor.finalizeState();

Implement ordered list state for FnApi. #30317

Implement ordered list state for FnApi. #30317

Conversation

shunping commented Feb 14, 2024 • edited Loading

GitHub Actions Tests Status (on master branch)

shunping commented Feb 15, 2024

scwhittle left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shunping Feb 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shunping Feb 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shunping Feb 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Mar 6, 2024

shunping commented Mar 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shunping Jun 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jun 11, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shunping Jun 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shunping Jun 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shunping commented Jun 12, 2024

shunping commented Jun 12, 2024

shunping commented Jun 13, 2024

Choose a reason for hiding this comment

damccorm left a comment

Choose a reason for hiding this comment

shunping commented Feb 14, 2024 •

edited

Loading

shunping Feb 16, 2024 •

edited

Loading

shunping Feb 16, 2024 •

edited

Loading

shunping Feb 23, 2024 •

edited

Loading

shunping Jun 11, 2024 •

edited

Loading

codecov bot commented Jun 11, 2024 •

edited

Loading

shunping Jun 12, 2024 •

edited

Loading

shunping Jun 12, 2024 •

edited

Loading