Cluster Local Model CR #3839

greenmoon55 · 2024-08-06T17:27:32Z

What this PR does / why we need it:
This PR introduces a cluster model cache CR and a node group CR. With these two PRs, you can define a model that would be downloaded to multiple nodes which are defined by a node group CR (node selectors).

Example Model Cache CR

apiVersion: serving.kserve.io/v1alpha1
kind: ClusterLocalModel
metadata:
  name: iris
spec:
  modelSize: 1Gi
  nodeGroups:
    - gpu1
  sourceModelUri: gs://kfserving-examples/models/sklearn/1.0/model

NodeGroup CR

apiVersion: serving.kserve.io/v1alpha1
kind: LocalModelNodeGroup 
metadata:
  name: gpu1
spec:
  persistentVolumeSpec:
    accessModes:
    - ReadWriteOnce
    volumeMode: Filesystem
    capacity:
      storage: 2Gi
    hostPath:
      path: /models
      type: ""
    persistentVolumeReclaimPolicy: Delete
    storageClassName: standard
    nodeAffinity:
      required:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node.kubernetes.io/instance-type
            operator: In
            values:
              - gpu1
  persistentVolumeClaimSpec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 2Gi
    storageClassName: standard
    volumeMode: Filesystem
    volumeName: blah

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Type of changes
Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Feature/Issue validation/testing:

Please describe the tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

Test A
Test B
Logs

Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Checklist:

Have you added unit/e2e tests that prove your fix is effective or that this feature works?
Has code been commented, particularly in hard-to-understand areas?
Have you made corresponding changes to the documentation?

Release note:

Re-running failed tests

/rerun-all - rerun all failed workflows.
/rerun-workflow <workflow name> - rerun a specific failed workflow. Only one workflow name can be specified. Multiple /rerun-workflow commands are allowed per comment.

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

greenmoon55 · 2024-08-07T14:51:25Z

pkg/apis/serving/v1alpha1/cached_model_status.go

+
+type CachedModelStatus struct {
+	OverallStatus OverallStatus         `json:"overallStatus,omitempty"`
+	NodeStatus    map[string]NodeStatus `json:"nodeStatus,omitempty"`


example: nodeX: Downloading

suggest NodeModelStatus

greenmoon55 · 2024-08-07T14:51:42Z

pkg/apis/serving/v1alpha1/cached_model_status.go

+
+	// How many nodes have the model available
+	// +optional
+	ModelCopies       *ModelCopies     `json:"copies,omitempty"`


for example: Suceessful: 10 total: 20

greenmoon55 · 2024-08-07T14:52:05Z

pkg/apis/serving/v1alpha1/cached_model_status.go

+	// How many nodes have the model available
+	// +optional
+	ModelCopies       *ModelCopies     `json:"copies,omitempty"`
+	InferenceServices []NamespacedName `json:"infereneceServices,omitempty"`


ns1: isvc1, ns2: isvc2 uses this cached model

greenmoon55 · 2024-08-07T14:52:55Z

pkg/apis/serving/v1alpha1/cached_model_types.go

+	// Container spec for the storage initializer init container
+
+	// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="StorageUri is immutable"
+	StorageUri string            `json:"storageUri" validate:"required"`


Full match vs match prefix like "storageUriPrefix"

greenmoon55 · 2024-08-07T14:55:38Z

pkg/apis/serving/v1alpha1/cached_model_types.go

+	// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="StorageUri is immutable"
+	StorageUri string            `json:"storageUri" validate:"required"`
+	ModelSize  resource.Quantity `json:"modelSize" validate:"required"`
+	NodeGroups []string          `json:"nodeGroups" validate:"required"`


one model can be cached on multiple nodegroups (gpu-1, gpu-2, gpu-3)

greenmoon55 · 2024-08-07T14:57:57Z

pkg/apis/serving/v1alpha1/cached_model_types.go

+	NodeGroups []string          `json:"nodeGroups" validate:"required"`
+	// only local is supported for now
+	StorageType   StorageType   `json:"storageType" validate:"required"`
+	CleanupPolicy CleanupPolicy `json:"cleanupPolicy" validate:"required"`


Maybe we also need a "download policy" that expects the model is on the node without creating jobs. Just for special maintenances.

Should we remove this field?

greenmoon55 · 2024-08-07T14:58:51Z

pkg/apis/serving/v1alpha1/cached_model_types.go

+
+// StorageContainerSpec defines the container spec for the storage initializer init container, and the protocols it supports.
+// +k8s:openapi-gen=true
+type ClusterCachedModelSpec struct {


better name?

Currently this is cluster scope only.

I still prefer the name ClusterLocalModel as the implementation of this CR is based on caching models locally.

greenmoon55 · 2024-08-07T14:59:22Z

pkg/apis/serving/v1alpha1/node_group_types.go

+// ModelCacheNodeGroupSpec defines a group of nodes for to download the model to.
+// +k8s:openapi-gen=true
+type ModelCacheNodeGroupSpec struct {
+	StorageLimit resource.Quantity `json:"storageLimit"`


We can use the storageLimit on the PersistentVolume instead of this field. However that field is immutable which is less flexible.

greenmoon55 · 2024-08-07T15:01:22Z

pkg/apis/serving/v1alpha1/node_group_types.go

+// +k8s:openapi-gen=true
+type ModelCacheNodeGroupSpec struct {
+	StorageLimit resource.Quantity `json:"storageLimit"`
+	NodeSelector map[string]string `json:"nodeSelector"`


Discussed with @sukumargaonkar. Two approaches here.
Approach 1 - It's up to the user to make sure they use the correct label selector so two node groups never have overlaps.
Approach 2 - Require a specific label set on nodes. e.g. kserve-nodegroup-name: xyz
Otherwise if we allow overlapping in nodegroups. storage size calculation would be tricky.

greenmoon55 · 2024-08-07T15:06:07Z

pkg/apis/serving/v1alpha1/node_group_types.go

+	NodeSelector map[string]string `json:"nodeSelector"`
+	// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="persistentVolume is immutable"
+	// PV spec template
+	PersistentVolume corev1.PersistentVolume `json:"persistentVolume"`


We will always create PVs for now. In the future, users might be able to leverage PersistentVolumeClaim with a custom CSI driver which makes sure different PVs map to the same disk location.

In the future we may have a namespace-scoped CR for NFS backed PV which does not share across namespaces. For non-local disks, sharing across namespaces is tricky as you would need different PVs and the volume plugin need to make sure they map to the same storage.

How can we make it namespace scoped? Now the PV/PVC is defined on the node group CR but this probably never should be made namespace scoped.

another namespace-scoped model cache CR that doesn't use nodegroup cr.

greenmoon55 · 2024-08-07T18:23:25Z

pkg/apis/serving/v1alpha1/cached_model_types.go

+	// only local is supported for now
+	StorageType   StorageType   `json:"storageType" validate:"required"`
+	CleanupPolicy CleanupPolicy `json:"cleanupPolicy" validate:"required"`
+}


Envvars for auth can be put in a configmap and added to the pod spec.

envFrom: - configMapRef: name: <config-file>

- name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: key: access-key name: name

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

yuzisun · 2024-08-15T02:12:37Z

pkg/apis/serving/v1alpha1/cached_model_types.go

+
+	// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="StorageUri is immutable"
+	// Original StorageUri
+	StorageUri string `json:"storageUri" validate:"required"`


suggest renaming to SourceModelUri

yuzisun · 2024-08-15T02:19:41Z

pkg/apis/serving/v1alpha1/cached_model_types.go

+	// group of nodes to cache the model on.
+	NodeGroups []string `json:"nodeGroups" validate:"required"`
+	// only local is supported for now
+	StorageType StorageType `json:"storageType" validate:"required"`


I think this CR is very specifically designed for local model cache, for other type of storage like AWS EBS fields like NodeGroup, cleanup policy would not apply I think.

I was thinking about temporarily disabling deletion jobs for some reason, say we want to delete and then recreate the CR but keeping the files on the disk.

yuzisun · 2024-08-15T02:23:20Z

pkg/apis/serving/v1alpha1/cached_model_types.go

+)
+
+// CleanupPolicy enum
+// +kubebuilder:validation:Enum="";DeleteModel;Ignore


what are the other possible values? this sounds like a flag

yeah, I can't think of other values for now.

yuzisun · 2024-08-15T02:26:31Z

pkg/apis/serving/v1alpha1/storage_container_types.go

@@ -32,6 +32,7 @@ type StorageContainerSpec struct {

 	// List of URI formats that this container supports
 	SupportedUriFormats []SupportedUriFormat `json:"supportedUriFormats" validate:"required"`
+	UseCase             UseCase              `json:"useCase" validate:"required"`


suggest workloadType: InitContainer or Job

yuzisun · 2024-08-15T02:27:11Z

pkg/apis/serving/v1alpha1/cached_model_status.go

+package v1alpha1
+
+type CachedModelStatus struct {
+	OverallStatus OverallStatus         `json:"overallStatus,omitempty"`


can we just call status?

yuzisun · 2024-08-25T21:21:14Z

pkg/apis/serving/v1alpha1/cluster_local_model_types.go

+}
+
+// +k8s:openapi-gen=true
+// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object


deepcopy-gen is deprecated
https://kubebuilder.io/migration/legacy/migration_guide_v1tov2#migrate-the-apis

replaced with +kubebuilder:object:root=true

Somehow we still call deepcopy-gen and removing it would cause an error while generating swagger files.

yuzisun · 2024-08-25T21:22:43Z

pkg/apis/serving/v1alpha1/cluster_local_model_types.go

+}
+
+// +k8s:openapi-gen=true
+// +kubebuilder:object:root=true


We only need this line

same as above

yuzisun · 2024-08-25T21:23:38Z

pkg/apis/serving/v1alpha1/cluster_local_model_types.go

+type ClusterLocalModelSpec struct {
+	// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="StorageUri is immutable"
+	// Original StorageUri
+	SourceModelUri string `json:"sourceModelUri" validate:"required"`


maybe we need to consider supporting multiple source modelUris

We can make it a list of models here or support multiple local model crs for one inference service.

yuzisun · 2024-08-25T21:24:25Z

pkg/apis/serving/v1alpha1/local_model_node_group_status.go

@@ -0,0 +1,24 @@
+/*
+Copyright 2021 The KServe Authors.


Suggested change

Copyright 2021 The KServe Authors.

Copyright 2024 The KServe Authors.

yuzisun · 2024-08-25T21:24:38Z

pkg/apis/serving/v1alpha1/cluster_local_model_types.go

@@ -0,0 +1,60 @@
+/*
+Copyright 2023 The KServe Authors.


Suggested change

Copyright 2023 The KServe Authors.

Copyright 2024 The KServe Authors.

pkg/apis/serving/v1alpha1/local_model_node_group_status.go

yuzisun · 2024-08-25T21:27:14Z

pkg/apis/serving/v1alpha1/local_model_node_group_types.go

+// +k8s:openapi-gen=true
+// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
+// +genclient


Suggested change

// +k8s:openapi-gen=true

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// +genclient

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

greenmoon55 · 2024-08-26T17:42:14Z

pkg/apis/serving/v1alpha1/cluster_local_model_types.go

+}
+
+// +k8s:openapi-gen=true
+// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object


Somehow we still call deepcopy-gen and removing it would cause an error while generating swagger files.

pkg/apis/serving/v1alpha1/local_model_node_group_status.go

greenmoon55 · 2024-08-26T17:45:27Z

pkg/apis/serving/v1alpha1/cluster_local_model_types.go

+type ClusterLocalModelSpec struct {
+	// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="StorageUri is immutable"
+	// Original StorageUri
+	SourceModelUri string `json:"sourceModelUri" validate:"required"`


We can make it a list of models here or support multiple local model crs for one inference service.

greenmoon55 · 2024-08-26T17:47:10Z

pkg/apis/serving/v1alpha1/cluster_local_model_status.go

+	InferenceServices []NamespacedName `json:"inferenceServices,omitempty"`
+}
+
+type NamespacedName struct {


Defining a new struct here because I got encountered struct field "Namespace" without JSON tag in type "NamespacedName" if I use types.NamespacedName.

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

greenmoon55 · 2024-08-29T20:48:04Z

pkg/apis/serving/v1alpha1/cluster_local_model_types.go

+}
+
+// +k8s:openapi-gen=true
+// +kubebuilder:object:root=true


same as above

greenmoon55 · 2024-08-29T20:51:26Z

pkg/apis/serving/v1alpha1/cluster_local_model_status.go

+type NodeStatus string
+
+// NodeStatus Enum values
+const (


~~With the daemonset in the future, Node status should be ModelExists or ModelDoesNotExist reported by the daemonset. For other status, we can look at individual jobs?~~
It still makes sense if we do not go down the daemonset approach.

…ntainers.yaml Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

greenmoon55 · 2024-09-10T20:21:42Z

/rerun

greenmoon55 · 2024-09-10T20:22:47Z

/rerun-all

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

yuzisun · 2024-09-14T18:20:24Z

pkg/apis/serving/v1alpha1/cluster_local_model_status.go

+
+type ClusterLocalModelStatus struct {
+	// Status of the model on a node, like NodeDownloaded or NodeNotReady
+	NodeStatus map[string]NodeStatus `json:"nodeStatus,omitempty"`


I think we can create a LocalModelNodeGroup controller and create the LocalModelNode resource per node, in this way we can spawn the model reconciler daemonset and update the status on each LocalModelNode.

yuzisun · 2024-09-17T15:01:02Z

/lgtm
/approve

yuzisun · 2024-09-17T15:03:43Z

Discussed with @greenmoon55 to create a follow up PR to add a LocalModelNode custom resource to track the model status for each node in the group.

* new model cache cr Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * update crd Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Fix genereted python tests Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Fix test failure Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Make nodegroup a list field in model cache cr Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * fix lint Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * minor updates to model cache cr Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Add usecase field to cluster storage container Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Fix test failures Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Change variable name Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Fix lint Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Fix default storage container cr Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * fix defualt.yaml Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Remove storagelimit field from node group Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Fix python code Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Change some fields Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Rename crd Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Fix lint error in python test files Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Rename CR Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Add status to local model node group Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Add missing node status Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Remove files related to ClusterLocalNodeGroup Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Add default value for workload type Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Fix StorageContainerSpec WorkloadType default value Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * nodegroups -> nodegroup Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Add comments Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Add back storageLimit Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Update charts/kserve-crd/templates/serving.kserve.io_clusterstoragecontainers.yaml Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> --------- Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

new model cache cr

072109d

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

greenmoon55 force-pushed the cluster-model-cache-cr-new branch from ca5d830 to 072109d Compare August 6, 2024 17:32

update crd

6ad65f0

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

greenmoon55 force-pushed the cluster-model-cache-cr-new branch from 69eafde to 6ad65f0 Compare August 6, 2024 19:35

greenmoon55 added 6 commits August 6, 2024 15:57

Fix lint

40d7e2f

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

Fix genereted python tests

4478bf5

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

Fix test failure

2d88907

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

Fix crd

210fcf1

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

Make nodegroup a list field in model cache cr

873ccb3

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

fix lint

76954f2

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

greenmoon55 commented Aug 7, 2024

View reviewed changes

greenmoon55 added 11 commits August 7, 2024 17:09

minor updates to model cache cr

6b19cd8

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

Add usecase field to cluster storage container

7e12a75

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

Fix test failures

3fa471e

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

Change variable name

6331af5

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

Fix lint

2d25e89

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

Fix default storage container cr

587c26b

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

Merge branch 'master' into cluster-model-cache-cr-new

b5d2c79

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

fix defualt.yaml

0677742

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

Remove storagelimit field from node group

2edef3a

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

Fix python code

596ee82

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

Change some fields

35733a3

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

greenmoon55 mentioned this pull request Aug 14, 2024

Cluster local model controller #3860

Merged

9 tasks

yuzisun reviewed Aug 15, 2024

View reviewed changes

yuzisun reviewed Aug 25, 2024

View reviewed changes

pkg/apis/serving/v1alpha1/local_model_node_group_status.go Show resolved Hide resolved

yuzisun reviewed Aug 25, 2024

View reviewed changes

Add comments

0975cca

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

greenmoon55 commented Aug 26, 2024

View reviewed changes

greenmoon55 added 2 commits August 28, 2024 18:05

Add back storageLimit

7b8c3ca

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

Merge branch 'master' into cluster-model-cache-cr-new

47e4be1

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

greenmoon55 commented Aug 29, 2024

View reviewed changes

greenmoon55 added 4 commits September 5, 2024 14:49

Update charts/kserve-crd/templates/serving.kserve.io_clusterstorageco…

9ad7835

…ntainers.yaml Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

Merge branch 'master' into cluster-model-cache-cr-new

8af2b50

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

Merge branch 'master' into cluster-model-cache-cr-new

8ec2b3a

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

Merge branch 'master' into cluster-model-cache-cr-new

5e0ad5e

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

greenmoon55 force-pushed the cluster-model-cache-cr-new branch from 4a78624 to 531ae53 Compare September 12, 2024 18:20

Merge branch 'master' into cluster-model-cache-cr-new

c6fef73

Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>

greenmoon55 force-pushed the cluster-model-cache-cr-new branch from 531ae53 to c6fef73 Compare September 12, 2024 18:29

yuzisun reviewed Sep 14, 2024

View reviewed changes

yuzisun added lgtm approved labels Sep 17, 2024

yuzisun approved these changes Sep 17, 2024

View reviewed changes

yuzisun merged commit acfc887 into kserve:master Sep 17, 2024
57 checks passed

greenmoon55 deleted the cluster-model-cache-cr-new branch December 23, 2024 20:46

	Copyright 2021 The KServe Authors.
	Copyright 2024 The KServe Authors.

	Copyright 2023 The KServe Authors.
	Copyright 2024 The KServe Authors.

	// +k8s:openapi-gen=true
	// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
	// +genclient

Cluster Local Model CR #3839

Cluster Local Model CR #3839

Conversation

greenmoon55 commented Aug 6, 2024 • edited by yuzisun Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

greenmoon55 Aug 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuzisun Aug 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

greenmoon55 Aug 29, 2024 • edited Loading

Choose a reason for hiding this comment

greenmoon55 commented Sep 10, 2024

greenmoon55 commented Sep 10, 2024

yuzisun Sep 14, 2024 • edited Loading

Choose a reason for hiding this comment

yuzisun commented Sep 17, 2024

yuzisun commented Sep 17, 2024

greenmoon55 commented Aug 6, 2024 •

edited by yuzisun

Loading

greenmoon55 Aug 7, 2024 •

edited

Loading

yuzisun Aug 25, 2024 •

edited

Loading

greenmoon55 Aug 29, 2024 •

edited

Loading

yuzisun Sep 14, 2024 •

edited

Loading