Ensure PartitionedOutputOperator is run with fixed local distribution #13834

arhimondr · 2022-08-24T19:30:59Z

Description

PartitionedOutputOperator maintains buffers for each output partition.
When the operator is run in the same pipeline as the TableScanOperator the
buffers are flushed after each split resulting in small pages being created.

Is this change a fix, improvement, new feature, refactoring, or other?

Improvement

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

Core engine

How would you describe this change to a non-technical end user or system administrator?

Performance improvement

Related issues, pull requests, and links

Documentation

(X) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

(X) No release notes entries required.
( ) Release notes entries required with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

losipiuk

Up to my limited expertise with this code it looks good. @sopel39 please take a look.

Also we should run benchmarks on this PR before merging IMO.

sopel39

This change will introduce performance regression (wall time) due to extra buffer and thread switching. I experimented with something similar for PA aggregation. Due to the issue, we also disabled spill in source stage joins (#315)

Usually source stage loop is super efficient from TS to output because it lacks any context switching.

If anything, IMO we should move merging of pages out of partition output operator to buffering layer (maybe with some opportunistic locking)

przemekak · 2022-09-01T06:16:53Z

Attaching benchmarks for this change.
As you can see it is a little bit slower, especially visible on TPCDS q67.
partitioned-output-fixed-distribution.pdf

arhimondr · 2022-11-02T21:19:57Z

@losipiuk @sopel39 @przemekak

Updated

The biggest regressions (q67, q22) were due to the fact that the plan shaped has changed and the AddExchangesBelowPartialAggregationOverGroupIdRuleSet could no longer perform necessary optimization.

To address the small discrepancy that came out during the last benchmark I optimized local exchange a bit. By removing PageReference and changing the data structure within a buffer from a linked list based Queue to an array based queue I reduced number of allocations. Also I decreased the number of lock acquisitions.

Currently the benchmark results look nearly identical:
Benchmarks comparison.pdf

I also tried to simulate an alleged worst case locally (when splits are fully consumed and the exchange is not necessary):

set session hive.compression_codec = 'NONE';
create table lineitem as select returnflag, quantity, extendedprice, discount from tpch.sf100.lineitem;

SELECT returnflag, sum(quantity), avg(extendedprice), sum(discount) FROM lineitem GROUP BY 1
UNION ALL
SELECT returnflag, sum(quantity), avg(extendedprice), sum(discount) FROM lineitem GROUP BY 1
UNION ALL
SELECT returnflag, sum(quantity), avg(extendedprice), sum(discount) FROM lineitem GROUP BY 1
UNION ALL
SELECT returnflag, sum(quantity), avg(extendedprice), sum(discount) FROM lineitem GROUP BY 1;

And I didn't find any difference in latency and cpu utilization with and without an additional local exchange.

The profiler also doesn't show any significant differences neither in CPU utilization not in lock contention:
flamegraphs.zip. Honestly I couldn't even find any frames associated with a local exchange.

Since there's no observable difference in a common case i think we should consider merging the change as it helps with providing better efficiency for corner cases (high cardinallity partitioning + selective predicates)

losipiuk · 2022-11-03T10:24:04Z

.../trino/sql/planner/iterative/rule/AddExchangesBelowPartialAggregationOverGroupIdRuleSet.java

@@ -104,25 +104,33 @@
            typeOf(ExchangeNode.class)
                    .with(scope().equalTo(REMOTE))
                    .with(source().matching(
-                            // PushPartialAggregationThroughExchange adds a projection. However, it can be removed if RemoveRedundantIdentityProjections is run in the mean-time.
-                            typeOf(ProjectNode.class).capturedAs(PROJECTION)
+                            typeOf(ProjectNode.class)


any ProjectNode is fine?

My understanding is that one of the rules adds identity projections that are later removed. This rule is trying to capture these projections. @findepi could you please take a look and validate whether my understanding is correct?

losipiuk · 2022-11-03T10:24:40Z

.../trino/sql/planner/iterative/rule/AddExchangesBelowPartialAggregationOverGroupIdRuleSet.java

-                                                    .with(step().equalTo(AggregationNode.Step.PARTIAL))
-                                                    .with(nonEmpty(groupingColumns()))
+                                            typeOf(ExchangeNode.class)
+                                                    .with(scope().equalTo(LOCAL))


any Local exchange is fine?

In my understanding this rule is trying to capture a specific shape of a plan that got change with introduction of an additional local exchange. Thus in this plane shape other types of local exchange shouldn't be expected. But it would be better to wait for @findepi to take a look to make sure my understanding is correct.

@arhimondr sorry, i don't remember. it feels like i wrote this code like 5 years ago

losipiuk · 2022-11-03T10:29:04Z

core/trino-main/src/main/java/io/trino/sql/planner/OptimizerConfig.java

+        return forceFixedDistributionForPartitionedOutputOperatorEnabled;
+    }
+
+    @Config("experimental.force-fixed-distribution-for-partitioned-output-operator-enabled")


nit: this is super long. Alsy why experimental
maybe optimizer.force-partitioned-output-fixed-distribution

Also add some documentation here and maybe in some .rst if we have any.

I intend to remove this property after there's enough confidence that no regression is introduced. End users shouldn't be changing this property, as the plan shape alterations may result in some planner rules (such as AddExchangesBelowPartialAggregationOverGroupIdRuleSet to no longer kick in)

losipiuk

LGTM

sopel39 · 2022-11-04T16:45:39Z

I wonder if this approach will cause limit queries to run for longer than needed, e.g. output rows will be now buffered until full page is collected.

losipiuk · 2022-11-04T19:16:11Z

I wonder if this approach will cause limit queries to run for longer than needed, e.g. output rows will be now buffered until full page is collected.

Do we have limit queries in benchmark? Should we extend benchmark with those?

przemekak · 2022-11-04T19:52:12Z

I wonder if this approach will cause limit queries to run for longer than needed, e.g. output rows will be now buffered until full page is collected.

Do we have limit queries in benchmark? Should we extend benchmark with those?

Actually most of TPCDS queruies have LIMIT 100 in them, e,g,:
https://github.com/trinodb/trino/blob/master/testing/trino-benchmark-queries/src/main/resources/sql/presto/tpcds/q57.sql#L58

sopel39 · 2022-11-08T12:09:38Z

Actually most of TPCDS queruies have LIMIT 100 in them, e,g,:

These are on aggregation output, so entire subquery must execute

losipiuk · 2022-11-08T14:27:16Z

Actually most of TPCDS queruies have LIMIT 100 in them, e,g,:

These are on aggregation output, so entire subquery must execute

We probably should benchmark with a query which just does super selective table scan with limit.

arhimondr · 2022-11-08T17:11:36Z

I wonder if this approach will cause limit queries to run for longer than needed, e.g. output rows will be now buffered until full page is collected.

@sopel39 It is not necessarily true. The TableScan pipeline would be allowed to "run" as long as there is buffer space (32MB). But it doesn't prevent a consumer pipeline to run if there's a free thread available.

We probably should benchmark with a query which just does super selective table scan with limit.

@losipiuk Yeah, given the complex interactions involved in short circuiting limit and the fact that we had problems with it in the past I don't think it's a bad idea to validate. Let me check.

sopel39 · 2022-11-09T12:56:50Z

@arhimondr could you add a test that partial limit is above the local RR exchange that you are adding. That should mitigate potential limit issues

But it doesn't prevent a consumer pipeline to run if there's a free thread available.

Consumer won't be able to fetch page until page builder is considered full.

arhimondr · 2022-11-09T16:18:00Z

@sopel39 I think I now understand the problem. Great catch.

Discussed with @losipiuk offline and it looks like could you add a test that partial limit is above the local RR exchange that you are adding might not even be sufficient. As @losipiuk suggested there could be a situation when a single task produces less rows than the limit, so the partial limit won't be triggered and the result won't get delivered to a downstream task until the upstream task finishes. Let me think about it more.

arhimondr · 2022-11-09T23:29:37Z

@sopel39 @losipiuk It feels like it's a classic throughput vs latency problem that is intrinsic to buffering. It is hard to say what is the right flush strategy.

I tried to approach this problem by flushing after a certain delay (see the last commit: a43b3c5) However I'm not sure what is the best strategy.

CC: @martint

sopel39 · 2022-11-10T09:56:42Z

@sopel39 @losipiuk It feels like it's a classic throughput vs latency problem that is intrinsic to buffering. It is hard to say what is the right flush strategy.

@arhimondr if you have a plan like:

RemoteExchange
     |
 PartialLimit
     |
LocalExchange
     |
TableScan

then RemoteExchange wouldn't have to wait until full page is buffered. I don't think it would introduce regressions then, would it?

losipiuk · 2022-11-10T14:32:33Z

@sopel39 @losipiuk It feels like it's a classic throughput vs latency problem that is intrinsic to buffering. It is hard to say what is the right flush strategy.

@arhimondr if you have a plan like:
RemoteExchange
     |
 PartialLimit
     |
LocalExchange
     |
TableScan
then RemoteExchange wouldn't have to wait until full page is buffered. I don't think it would introduce regressions then, would it?

Yeah. But it still will wait until PartialLimit triggers. Which may never happen if single task did not produce enought matching rows.

sopel39 · 2022-11-10T15:05:27Z

Which may never happen if single task did not produce enought matching rows.

That's FTE issue, right? In pipeline mode it would work

losipiuk · 2022-11-10T17:25:42Z

Which may never happen if single task did not produce enought matching rows.

That's FTE issue, right? In pipeline mode it would work

I think it would not. Unless I am missing sth. We can chat offline.

arhimondr · 2022-11-15T17:33:55Z

Actually it looks like today it's not possible for a partial limit to endup between a table scan and a PartitionedOutputOperator. Any LimitNode (event a partial created by the PushLimitThroughOuterJoin) is unconditionally split into two by the AddExchanges rule: https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/sql/planner/optimizations/AddExchanges.java#L539

So the plan of a LEFT OUTER JOIN (seemingly the only query shape that can be impacted by this change) looks like following:

Fragment 0 [SINGLE]                                                                                                                           
     Output layout: [comment, comment_15]                                                                                                      
     Output partitioning: SINGLE []                                                                                                            
     Output[columnNames = [comment, comment]]                                                                                                  
     │   Layout: [comment:varchar(44), comment_15:varchar(44)]                                                                                 
     │   Estimates: {rows: 10 (631B), cpu: 0, memory: 0B, network: 0B}                                                                         
     │   comment := comment_15                                                                                                                 
     └─ Limit[count = 10]                                                                                                                      
        │   Layout: [comment:varchar(44), comment_15:varchar(44)]                                                                              
        │   Estimates: {rows: 10 (631B), cpu: 631, memory: 0B, network: 0B}                                                                    
        └─ LocalExchange[partitioning = SINGLE]                                                                                                
           │   Layout: [comment:varchar(44), comment_15:varchar(44)]                                                                           
           │   Estimates: {rows: 10 (631B), cpu: 0, memory: 0B, network: 0B}                                                                   
           └─ RemoteSource[sourceFragmentIds = [1]]                                                                                            
                  Layout: [comment:varchar(44), comment_15:varchar(44)]                                                                        
                                                                                                                                               
 Fragment 1 [HASH]                                                                                                                             
     Output layout: [comment, comment_15]                                                                                                      
     Output partitioning: SINGLE []                                                                                                            
     LimitPartial[count = 10]                                                                                                                  
     │   Layout: [comment:varchar(44), comment_15:varchar(44)]                                                                                 
     │   Estimates: {rows: 10 (631B), cpu: 631, memory: 0B, network: 0B}                                                                       
     └─ LeftJoin[criteria = ("linenumber" = "linenumber_3"), hash = [$hashvalue, $hashvalue_23], distribution = PARTITIONED]                   
        │   Layout: [comment:varchar(44), comment_15:varchar(44)]                                                                              
        │   Estimates: {rows: 85969 (5.18MB), cpu: 7.79M, memory: 2.61MB, network: 0B}                                                         
        │   Distribution: PARTITIONED                                                                                                          
        ├─ RemoteSource[sourceFragmentIds = [2]]                                                                                               
        │      Layout: [linenumber:integer, comment:varchar(44), $hashvalue:bigint]                                                            
        └─ LocalExchange[partitioning = SINGLE]                                                                                                
           │   Layout: [linenumber_3:integer, comment_15:varchar(44), $hashvalue_23:bigint]                                                    
           │   Estimates: {rows: 60175 (2.61MB), cpu: 0, memory: 0B, network: 0B}                                                              
           └─ RemoteSource[sourceFragmentIds = [4]]                                                                                            
                  Layout: [linenumber_3:integer, comment_15:varchar(44), $hashvalue_24:bigint]                                                 
                                                                                                                                               
 Fragment 2 [SINGLE]                                                                                                                           
     Output layout: [linenumber, comment, $hashvalue_19]                                                                                       
     Output partitioning: HASH [linenumber][$hashvalue_19]                                                                                     
     LocalExchange[partitioning = ROUND_ROBIN]                                                                                                 
     │   Layout: [linenumber:integer, comment:varchar(44), $hashvalue_19:bigint]                                                               
     │   Estimates: {rows: 10 (456B), cpu: 456, memory: 0B, network: 0B}                                                                       
     └─ Limit[count = 10]                                                                                                                      
        │   Layout: [linenumber:integer, comment:varchar(44), $hashvalue_20:bigint]                                                            
        │   Estimates: {rows: 10 (456B), cpu: 456, memory: 0B, network: 0B}                                                                    
        └─ LocalExchange[partitioning = SINGLE]                                                                                                
           │   Layout: [linenumber:integer, comment:varchar(44), $hashvalue_20:bigint]                                                         
           │   Estimates: {rows: 10 (456B), cpu: 0, memory: 0B, network: 0B}                                                                   
           └─ RemoteSource[sourceFragmentIds = [3]]                                                                                            
                  Layout: [linenumber:integer, comment:varchar(44), $hashvalue_21:bigint]                                                      
                                                                                                                                               
 Fragment 3 [SOURCE]                                                                                                                           
     Output layout: [linenumber, comment, $hashvalue_22]                                                                                       
     Output partitioning: SINGLE []                                                                                                            
     Project[]                                                                                                                                 
     │   Layout: [linenumber:integer, comment:varchar(44), $hashvalue_22:bigint]                                                               
     │   Estimates: {rows: 10 (456B), cpu: 456, memory: 0B, network: 0B}                                                                       
     │   $hashvalue_22 := combine_hash(bigint '0', COALESCE("$operator$hash_code"("linenumber"), 0))                                           
     └─ LimitPartial[count = 10]                                                                                                               
        │   Layout: [linenumber:integer, comment:varchar(44)]                                                                                  
        │   Estimates: {rows: 10 (366B), cpu: 366, memory: 0B, network: 0B}                                                                    
        └─ TableScan[table = hive:tpch:lineitem]                                                                                               
               Layout: [linenumber:integer, comment:varchar(44)]                                                                               
               Estimates: {rows: 60175 (2.10MB), cpu: 2.10M, memory: 0B, network: 0B}                                                          
               linenumber := linenumber:int:REGULAR                                                                                            
               comment := comment:varchar(44):REGULAR                                                                                          
                                                                                                                                               
 Fragment 4 [SOURCE]                                                                                                                           
     Output layout: [linenumber_3, comment_15, $hashvalue_25]                                                                                  
     Output partitioning: HASH [linenumber_3][$hashvalue_25]                                                                                   
     LocalExchange[partitioning = ROUND_ROBIN]                                                                                                 
     │   Layout: [linenumber_3:integer, comment_15:varchar(44), $hashvalue_25:bigint]                                                          
     │   Estimates: {rows: 60175 (2.61MB), cpu: 2.61M, memory: 0B, network: 0B}                                                                
     └─ ScanProject[table = hive:tpch:lineitem]                                                                                                
            Layout: [linenumber_3:integer, comment_15:varchar(44), $hashvalue_26:bigint]                                                       
            Estimates: {rows: 60175 (2.61MB), cpu: 2.10M, memory: 0B, network: 0B}/{rows: 60175 (2.61MB), cpu: 2.61M, memory: 0B, network: 0B} 
            $hashvalue_26 := combine_hash(bigint '0', COALESCE("$operator$hash_code"("linenumber_3"), 0))                                      
            comment_15 := comment:varchar(44):REGULAR                                                                                          
            linenumber_3 := linenumber:int:REGULAR

Basically on the left side the plan looks like following: SCAN -> Partial Limit -> Remote Gather -> Local Gather -> PartitionedOutputOperator.

This plan shape may be problematic for very large limits (e.g.: LIMIT 1_000_000_000) as it requires all the rows to be processed by a single node before passing them to a join and we may need to address that at some point. However at this point It doesn't seem like this PR should introduce any regression.

I'm dropping the Flush partitioned output on timeout to allow LIMIT queries finish early commit for now. @losipiuk @sopel39 I wonder if there's anything else here to be done before we can merge it?

sopel39 · 2022-11-16T15:08:52Z

core/trino-main/src/main/java/io/trino/operator/exchange/LocalExchange.java

-            exchangerSupplier = () -> new BroadcastExchanger(buffers, memoryManager);
-        }
-        else if (partitioning.equals(FIXED_ARBITRARY_DISTRIBUTION)) {
+        if (partitioning.equals(SINGLE_DISTRIBUTION) || partitioning.equals(FIXED_ARBITRARY_DISTRIBUTION)) {
            exchangerSupplier = () -> new RandomExchanger(buffers, memoryManager);


Is this more expensive potentially?

You mean for single distribution? I don't think so. The broadcast implementation seemed to be more complex.

sopel39 · 2022-11-16T15:09:49Z

core/trino-main/src/main/java/io/trino/operator/exchange/LocalExchangeSource.java

@@ -41,8 +42,11 @@

    private final Consumer<LocalExchangeSource> onFinish;

-    private final BlockingQueue<PageReference> buffer = new LinkedBlockingDeque<>();


so we never relied on buffer to be blocking?

No. new LinkedBlockingDeque<>() is unbounded

sopel39 · 2022-11-16T15:11:57Z

core/trino-main/src/main/java/io/trino/operator/exchange/LocalExchange.java

@@ -13,6 +13,7 @@
 */


add a comment that PageReference is no longer needed after removing broadcast exchange

Should I add it to the commit message? Not sure if it makes sense to keep mentions of the PageReference in the codebase, since the PageReference itself is going away

don't block PR

PartitionedOutputOperator maintains buffers for each output partition. When the operator is run in the same pipeline as the TableScanOperator the buffers are flushed after each split resulting in small pages being created.

Broadcast local exchanges are never performed

Use ArrayDeque to avoid unnecessary allocations associated with a LinkedList based queue

Avoid creating PageReference objects for every page

A new instance is created for each ExchangeSink

pettyjamesm · 2022-12-09T19:34:17Z

core/trino-main/src/main/java/io/trino/sql/planner/optimizations/AddLocalExchanges.java

@@ -743,6 +744,9 @@ public PlanWithProperties visitExchange(ExchangeNode node, StreamPreferredProper
                        any().withOrderSensitivity(),
                        any().withOrderSensitivity());
            }
+            if (isForceFixedDistributionForPartitionedOutputOperatorEnabled(session) && node.isHashPartitionedExchange()) {


Shouldn't this behavior only be induced for partitioned output operators on top of table scans? Intermediate stages shouldn't have the same flushing behaviors, so condition seems potentially overly-broad, but maybe that's handled by planAndEnforceChildren somehow?

cla-bot bot added the cla-signed label Aug 24, 2022

github-actions bot added the tests:hive label Aug 24, 2022

arhimondr force-pushed the partitioned-output-fixed-distribution branch from dce8c8d to 2c1b725 Compare August 25, 2022 01:38

arhimondr requested review from sopel39 and losipiuk August 25, 2022 01:38

losipiuk reviewed Aug 25, 2022

View reviewed changes

sopel39 previously requested changes Aug 25, 2022

View reviewed changes

arhimondr force-pushed the partitioned-output-fixed-distribution branch 4 times, most recently from b88f4c0 to 3400d4e Compare November 2, 2022 21:09

arhimondr requested a review from sopel39 November 2, 2022 21:09

losipiuk reviewed Nov 3, 2022

View reviewed changes

arhimondr force-pushed the partitioned-output-fixed-distribution branch from 3400d4e to a43b3c5 Compare November 9, 2022 23:26

arhimondr force-pushed the partitioned-output-fixed-distribution branch from a43b3c5 to 06c056b Compare November 10, 2022 21:35

arhimondr force-pushed the partitioned-output-fixed-distribution branch from 06c056b to 5355abe Compare November 15, 2022 17:57

losipiuk approved these changes Nov 16, 2022

View reviewed changes

sopel39 reviewed Nov 16, 2022

View reviewed changes

arhimondr force-pushed the partitioned-output-fixed-distribution branch from 5355abe to b05a554 Compare November 18, 2022 16:25

arhimondr added 5 commits November 19, 2022 05:53

Drop BroadcastExchanger

9c3a69a

Broadcast local exchanges are never performed

Optimize LocalExchangeSource

7d7b95e

Use ArrayDeque to avoid unnecessary allocations associated with a LinkedList based queue

Optimize local exchange

ab6cc6b

Avoid creating PageReference objects for every page

Drop unnecessary synchronization

af5cc04

A new instance is created for each ExchangeSink

arhimondr force-pushed the partitioned-output-fixed-distribution branch from b05a554 to af5cc04 Compare November 19, 2022 10:56

arhimondr merged commit 45e111c into trinodb:master Nov 20, 2022

arhimondr deleted the partitioned-output-fixed-distribution branch November 20, 2022 03:00

github-actions bot added this to the 404 milestone Nov 20, 2022

colebow mentioned this pull request Nov 21, 2022

Add Trino 405 release notes #15139

Merged

pettyjamesm mentioned this pull request Dec 9, 2022

Cleanup PartitioningExchanger #15357

Merged

pettyjamesm reviewed Dec 9, 2022

View reviewed changes

This was referenced Dec 9, 2022

Revert "Ensure PartitionedOutputOperator is run with fixed local distribution" #15358

Merged

Avoid flushing PagePartitioner buffer on each split #15361

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensure PartitionedOutputOperator is run with fixed local distribution #13834

Ensure PartitionedOutputOperator is run with fixed local distribution #13834

arhimondr commented Aug 24, 2022

losipiuk left a comment

sopel39 left a comment •

edited

Loading

przemekak commented Sep 1, 2022

arhimondr commented Nov 2, 2022

losipiuk Nov 3, 2022

arhimondr Nov 8, 2022

losipiuk Nov 3, 2022

arhimondr Nov 8, 2022

findepi Nov 9, 2022

losipiuk Nov 3, 2022

losipiuk Nov 3, 2022

arhimondr Nov 8, 2022

losipiuk left a comment

sopel39 commented Nov 4, 2022

losipiuk commented Nov 4, 2022

przemekak commented Nov 4, 2022

sopel39 commented Nov 8, 2022

losipiuk commented Nov 8, 2022

arhimondr commented Nov 8, 2022

sopel39 commented Nov 9, 2022 •

edited

Loading

arhimondr commented Nov 9, 2022

arhimondr commented Nov 9, 2022

sopel39 commented Nov 10, 2022

losipiuk commented Nov 10, 2022

sopel39 commented Nov 10, 2022

losipiuk commented Nov 10, 2022

arhimondr commented Nov 15, 2022

sopel39 Nov 16, 2022

arhimondr Nov 16, 2022

sopel39 Nov 16, 2022

arhimondr Nov 16, 2022

sopel39 Nov 16, 2022

arhimondr Nov 16, 2022

pettyjamesm Dec 9, 2022

		@@ -41,8 +42,11 @@

		private final Consumer<LocalExchangeSource> onFinish;

		private final BlockingQueue<PageReference> buffer = new LinkedBlockingDeque<>();

Ensure PartitionedOutputOperator is run with fixed local distribution #13834

Ensure PartitionedOutputOperator is run with fixed local distribution #13834

Conversation

arhimondr commented Aug 24, 2022

Description

Related issues, pull requests, and links

Documentation

Release notes

losipiuk left a comment

Choose a reason for hiding this comment

sopel39 left a comment • edited Loading

Choose a reason for hiding this comment

przemekak commented Sep 1, 2022

arhimondr commented Nov 2, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

losipiuk left a comment

Choose a reason for hiding this comment

sopel39 commented Nov 4, 2022

losipiuk commented Nov 4, 2022

przemekak commented Nov 4, 2022

sopel39 commented Nov 8, 2022

losipiuk commented Nov 8, 2022

arhimondr commented Nov 8, 2022

sopel39 commented Nov 9, 2022 • edited Loading

arhimondr commented Nov 9, 2022

arhimondr commented Nov 9, 2022

sopel39 commented Nov 10, 2022

losipiuk commented Nov 10, 2022

sopel39 commented Nov 10, 2022

losipiuk commented Nov 10, 2022

arhimondr commented Nov 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sopel39 left a comment •

edited

Loading

sopel39 commented Nov 9, 2022 •

edited

Loading