[SPARK-37326][SQL] Support TimestampNTZ in CSV data source #34596

sadikovi · 2021-11-15T03:53:58Z

What changes were proposed in this pull request?

This PR adds support for TimestampNTZ type in the CSV data source.

Most of the functionality has already been added, this patch verifies that writes + reads work for TimestampNTZ type and adds schema inference depending on the timestamp value format written. The following applies:

If there is a mixture of TIMESTAMP_NTZ and TIMESTAMP_LTZ values, use TIMESTAMP_LTZ.
If there are only TIMESTAMP_NTZ values, resolve using the the default timestamp type configured with spark.sql.timestampType.

In addition, I introduced a new CSV option timestampNTZFormat which is similar to timestampFormat but it allows to configure read/write pattern for TIMESTAMP_NTZ types. It is basically a copy of timestamp pattern but without timezone.

The schema inference works in the following way:

We test if the field can be parsed a timestamp without timezone using timestampNTZFormat.
If the field has the timezone component, parseWithoutTimeZone method throws QueryExecutionErrors.cannotParseStringAsDataTypeError which is a RuntimeException.
Move on to parsing the field as timestamp with timezone (the existing logic).

Why are the changes needed?

The current CSV source could write values as TimestampNTZ into a file but could not preserve this type when reading the file back, this PR fixes the issue.

Does this PR introduce any user-facing change?

Previously, CSV data source would infer timestamp values as TimestampType when reading a CSV file. Now, the data source would infer the timestamp value type based on the format (with or without timezone) and default timestamp type based on spark.sql.timestampType.

A new CSV option timestampNTZFormat is added to control the way values are formatted during writes or parsed during reads.

Now if the timestamp cannot be parsed as a timestamp without timezone, e.g. contains the zone-offset or zone-id component, parseWithTimeZone throws RuntimeException signalling the inference code to try the next type.

How was this patch tested?

I extended CSVSuite with a few unit tests to verify that write-read roundtrip works for TIMESTAMP_NTZ and TIMESTAMP_LTZ values.

sadikovi · 2021-11-15T04:53:58Z

cc @gengliangwang @MaxGekk

SparkQA · 2021-11-15T05:13:19Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49690/

SparkQA · 2021-11-15T05:57:40Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49690/

SparkQA · 2021-11-15T06:28:12Z

Test build #145220 has finished for PR 34596 at commit 6bfc1b1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sadikovi · 2021-11-15T22:11:57Z

Let me fix the tests first.

sadikovi · 2021-11-16T02:24:40Z

@gengliangwang @MaxGekk @cloud-fan could you review this PR? Thanks.

SparkQA · 2021-11-16T03:23:05Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49732/

SparkQA · 2021-11-16T04:11:00Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49732/

SparkQA · 2021-11-16T07:16:22Z

Test build #145262 has finished for PR 34596 at commit ea47b94.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sadikovi · 2021-11-17T00:10:11Z

I would like to discuss how to handle the following case:

User writes TIMESTAMP_NTZ value in CSV.
User set spark.sql.legacy.timeParserPolicy to "legacy"

This case is ambiguous: default timestamp pattern for legacy requires timezone but the value does not have any. However, the code works just fine when the pattern does not have timezone. In that case we would write the value successfully.

To resolve ambiguity, I identified 2 options:

throw an exception saying that TIMESTAMP_NTZ values are not supported in "legacy" time parsing, regardless of whether timestamp format has the timezone component or not.
Write timestamp value with the default timezone, basically converting LocalDateTime into java.sql.Timestamp (currently implemented in the PR).

@gengliangwang Do you have any preferences here? Thanks.

cloud-fan · 2021-11-17T05:39:28Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVInferSchema.scala

@@ -38,6 +39,13 @@ class CSVInferSchema(val options: CSVOptions) extends Serializable {
    legacyFormat = FAST_DATE_FORMAT,
    isParsing = true)

+  private val timestampNTZFormatter = TimestampFormatter(
+    options.timestampFormatInRead,


I think we should fail earlier if the pattern string contains timezone? It doesn't make sense to have timezone in NTZ string.

moreover, maybe we should add a new CSV option to define pattern for timestamp ntz.

sadikovi · 2021-11-17T23:32:07Z

cc @MaxGekk @cloud-fan @gengliangwang for review.

SparkQA · 2021-11-18T00:17:24Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49823/

SparkQA · 2021-11-18T01:16:20Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49823/

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVInferSchema.scala

gengliangwang · 2021-11-18T02:35:47Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVInferSchema.scala

+  }
+
+  private def tryParseTimestampNTZ(field: String): DataType = {
+    if ((allCatch opt !timestampNTZFormatter.isTimeZoneSet(field)).getOrElse(false) &&


nit: let's add one-line comment here.

Added, thanks.

gengliangwang · 2021-11-18T02:52:43Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala

@@ -164,6 +164,20 @@ class CSVOptions(
      s"${DateFormatter.defaultPattern}'T'HH:mm:ss[.SSS][XXX]"
    })

+  val timestampNTZFormatInRead: Option[String] = parameters.get("timestampNTZFormat").orElse {
+    if (SQLConf.get.legacyTimeParserPolicy == LegacyBehaviorPolicy.LEGACY) {


Let's ignore the legacy behavior in TimestampNTZ.
For the casting between string and TimestampNTZ, we didn't respect it either.

gengliangwang · 2021-11-18T02:52:49Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala

+    }
+  }
+  val timestampNTZFormatInWrite: String = parameters.getOrElse("timestampNTZFormat",
+    if (SQLConf.get.legacyTimeParserPolicy == LegacyBehaviorPolicy.LEGACY) {


SparkQA · 2021-11-18T03:26:38Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49834/

gengliangwang · 2021-11-18T03:36:13Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala

+      val parsed = formatter.parse(s)
+      val parsedZoneId = parsed.query(TemporalQueries.zone())
+      parsedZoneId != null
+    } catch checkParsedDiff(s, legacyFormatter.isTimeZoneSet)


gengliangwang · 2021-11-18T03:37:11Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala

+    try {
+      val (_, zoneIdOpt, _) = parseTimestampString(UTF8String.fromString(s))
+      zoneIdOpt.isDefined
+    } catch checkParsedDiff(s, legacyFormatter.isTimeZoneSet)


sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala

gengliangwang

Thanks for the work

sadikovi · 2021-11-28T21:23:24Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala

+   * refer to `parseTimestampString` for the allowed formats.
+   */
+  def stringToTimestampWithoutTimeZone(s: UTF8String): Option[Long] = {
+    stringToTimestampWithoutTimeZone(s, false)


I used false to preserve the original behaviour in this method and the one below.

sadikovi · 2021-11-28T22:13:42Z

@gengliangwang @cloud-fan I updated the code, could you do the penultimate review? Thanks.

SparkQA · 2021-11-28T23:01:59Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50156/

SparkQA · 2021-11-28T23:41:49Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50156/

SparkQA · 2021-11-29T03:11:19Z

Test build #145686 has finished for PR 34596 at commit 043edb6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-11-29T22:07:39Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50201/

SparkQA · 2021-11-29T23:08:47Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50201/

SparkQA · 2021-11-30T02:06:44Z

Test build #145731 has finished for PR 34596 at commit 1edef2d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2021-11-30T06:54:02Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVInferSchema.scala

+    // We can only parse the value as TimestampNTZType if it does not have zone-offset or
+    // time-zone component and can be parsed with the timestamp formatter.
+    // Otherwise, it is likely to be a timestamp with timezone.
+    if ((allCatch opt timestampNTZFormatter.parseWithoutTimeZone(field, true)).isDefined) {


Should maybe we skip the parsing if SQLConf.get.timestampType is set to TIMESTAMP_LTZ since parsing is non-trivial op?

Could you elaborate a bit more? Thanks.

My understanding was that the config indicated whether the output of parsing should be treated as TimestampNTZ or TimestampLTZ.

HyukjinKwon · 2021-11-30T06:56:16Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVInferSchema.scala

+    options.zoneId,
+    legacyFormat = FAST_DATE_FORMAT,
+    isParsing = true,
+    forTimestampNTZ = true)


this part I'd defer to @MaxGekk to review.

HyukjinKwon

Looks good otherwise

MaxGekk · 2021-11-30T08:12:26Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala

@@ -164,6 +164,10 @@ class CSVOptions(
      s"${DateFormatter.defaultPattern}'T'HH:mm:ss[.SSS][XXX]"
    })

+  val timestampNTZFormatInRead: Option[String] = parameters.get("timestampNTZFormat")
+  val timestampNTZFormatInWrite: String = parameters.getOrElse("timestampNTZFormat",
+    s"${DateFormatter.defaultPattern}'T'HH:mm:ss[.SSS]")


I wonder what's the reason to have the optional field [.SSS] in write. How should CSV writer decide whether to write milliseconds or not?

Another question, why the precision is in milliseconds but not in microseconds?

It is the same reason as for timestampFormat above, I just copy-pasted it for timestampNTZFormat and removed the zone component.

It is the same reason as for timestampFormat above ...

The option was added when Spark's timestamp type had milliseconds precision. Now the precision is microsecond, and don't see any reasons to loose info in write, by default.

I think we can change it for both LTZ and NTZ, if the precision lose here is a problem.

I would rather change it separately, sounds more like a general problem with timestamps.

MaxGekk · 2021-11-30T08:28:00Z

sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala

+          StructType(StructField("a", TimestampNTZType) :: Nil),
+          Map.empty[String, String]) as "value")
+      .selectExpr("value.a")
+    checkAnswer(fromCsvDF, Row(null))


How about positive test for the functions from_csv/to_csv and schema_of_csv?

There is a test right above that verifies the happy path for those functions. I only added a new test because it was missing from the original patch.

MaxGekk · 2021-11-30T08:32:29Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala

+      val path = s"${dir.getCanonicalPath}/csv"
+
+      val exp = spark.sql("select timestamp_ntz'2020-12-12 12:12:12' as col0")
+      exp.write.format("csv").option("timestampNTZFormat", "yyyy-MM-dd HH:mm:ss").save(path)


Could you test max precision with pattern .SSSSSS, please.

MaxGekk · 2021-11-30T08:34:23Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala

@@ -1012,6 +1012,196 @@ abstract class CSVSuite
    }
  }

+  test("SPARK-37326: Use different pattern to write and infer TIMESTAMP_NTZ values") {
+    withTempDir { dir =>
+      val path = s"${dir.getCanonicalPath}/csv"


Just write directly to dir

Oh, I could not because the directory already exists by the time the method is called. withTempDir creates a directory, but it needs to be non-existent for Spark to write into.

There is withTempPath which doesn't create any dirs:

spark/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/SQLHelper.scala

Lines 62 to 66 in 0ba1d38

/**

* Generates a temporary path without creating the actual file/directory, then pass it to `f`. If

* a file/directory is created there by `f`, it will be delete after `f` returns.

*/

protected def withTempPath(f: File => Unit): Unit = {

+1 to use withTempPath

It seems there is a mixture of withTempDir and withTempPath in this file arguably being used with the same purpose. I will open a PR to fix the tests I added but I am not going to update other occurrences.

MaxGekk · 2021-11-30T08:36:02Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala

+          .option("timestampNTZFormat", "yyyy-MM-dd HH:mm:ss")
+          .load(path)
+
+        checkAnswer(res, exp)


checkAnswer doesn't check the type. Could you check that the inferred type is correct.

Wouldn't that fail the answer since the values would be different? Yes, I can check the type explicitly, thanks.

MaxGekk · 2021-11-30T08:37:54Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala

@@ -1012,6 +1012,196 @@ abstract class CSVSuite
    }
  }

+  test("SPARK-37326: Use different pattern to write and infer TIMESTAMP_NTZ values") {


The test title says about different patterns but the patterns are the same in write and in inferring, in fact.

Could you elaborate please? The code does use different timestamp format compared to the default one.

The test title confuses because it can be read as the patterns in write and infer are different in the test.

Let me rewrite this.

MaxGekk · 2021-11-30T08:43:14Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala

+        .option("header", "true")
+        .load(path)
+
+      for (policy <- Seq("exception", "corrected", "legacy")) {


Where is policy used?

Oh, good point. I forgot to change. Now it is updated.

sadikovi · 2021-11-30T22:43:59Z

I addressed the comments, can you review again?

SparkQA · 2021-12-01T00:22:42Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50251/

SparkQA · 2021-12-01T01:08:59Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50251/

SparkQA · 2021-12-01T04:11:27Z

Test build #145778 has finished for PR 34596 at commit feb3715.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2021-12-01T07:14:05Z

+1, LGTM. Merging to master.
Thank you, @sadikovi and @gengliangwang @HyukjinKwon @cloud-fan for review.

cloud-fan · 2021-12-01T14:12:54Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala

   * The return type is [[Option]] in order to distinguish between 0L and null. Please
   * refer to `parseTimestampString` for the allowed formats.
   */
-  def stringToTimestampWithoutTimeZone(s: UTF8String): Option[Long] = {
+  def stringToTimestampWithoutTimeZone(s: UTF8String, failOnError: Boolean): Option[Long] = {


The name failOnError is misleading, as this method never fail. how about allowTimeZone?

Respectfully, I have changed this exact code and variable 3 times by now. Yes, I will update 😞.

cloud-fan · 2021-12-01T14:14:42Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala

@@ -66,10 +68,23 @@ sealed trait TimestampFormatter extends Serializable {
  @throws(classOf[DateTimeParseException])
  @throws(classOf[DateTimeException])
  @throws(classOf[IllegalStateException])
-  def parseWithoutTimeZone(s: String): Long =
+  def parseWithoutTimeZone(s: String, failOnError: Boolean): Long =


ditto, allowTimeZone?

Another question is, instead of adding a new parameter, can the caller side pick one of parse and parseWithoutTimeZone?

I am not sure we can do it, because parseWithoutTimeZone method can be used with different options depending on the context, and it is, in fact. Unfortunately, this is the artifact of the problem that we discussed earlier - some functions are not supposed to fail while parsing the value and others are.

cloud-fan · 2021-12-01T14:21:02Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala

@@ -2489,10 +2693,6 @@ abstract class CSVSuite
  }

  test("SPARK-36536: use casting when datetime pattern is not set") {
-    def isLegacy: Boolean = {


I think we should still test the legacy behavior for LTZ?

Oh, this is for TimestampNTZ change - isLegacy flag was only applied to TimestampNTZ, TimestampLTZ code is unchanged.

### What changes were proposed in this pull request? This PR fixes an issue that the test added in SPARK-37326 (#34596) fails with Java 11. https://github.com/apache/spark/runs/4381645820?check_suite_focus=true#step:9:11681 The reason is that the error message `DateTimeFormatter` was changed as of Java 9. https://bugs.openjdk.java.net/browse/JDK-8085887 http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/28df1af8e872 ### Why are the changes needed? To keep the build stable. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Tests with the following command success. ``` JAVA_HOME=/path/to/java11/ build/sbt -Phive -Phive-thriftserver "testOnly org.apache.spark.sql.execution.datasources.csv.CSVv*Suite" ``` Closes #34771 from sarutak/followup-SPARK-37326. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…support in CSV data source ### What changes were proposed in this pull request? This is a follow-up PR to #34596. There were a few comments and suggestions raised after the PR was merged, so I addressed them in this follow-up: - Instead of using `failOnError`, which was confusing as no error was thrown in the method, we use `allowTimeZone` which has an opposite meaning of `failOnError` and far more descriptive. - I updated a few test names to resolve ambiguity. - I changed the tests to use `withTempPath` as was suggested in the original PR. ### Why are the changes needed? Code cleanup and clarifications. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing unit and integration tests. Closes #34777 from sadikovi/timestamp-ntz-csv-follow-up. Authored-by: Ivan Sadikov <ivan.sadikov@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

sadikovi added 2 commits November 15, 2021 16:38

add timestamp ntz read in csv

7e73e59

update tests

6bfc1b1

github-actions bot added the SQL label Nov 15, 2021

update code to handle legacy time parsing

ea47b94

cloud-fan reviewed Nov 17, 2021

View reviewed changes

address comments, update code

e439ae2

github-actions bot added the DOCS label Nov 17, 2021

update tests

7eca451

gengliangwang reviewed Nov 18, 2021

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVInferSchema.scala Show resolved Hide resolved

gengliangwang reviewed Nov 18, 2021

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala Outdated Show resolved Hide resolved

gengliangwang mentioned this pull request Nov 18, 2021

[SPARK-37360][SQL] Support TimestampNTZ in JSON data source #34638

Closed

gengliangwang approved these changes Nov 26, 2021

View reviewed changes

sadikovi commented Nov 28, 2021

View reviewed changes

minor updates, add test

043edb6

minor updates

1edef2d

HyukjinKwon reviewed Nov 30, 2021

View reviewed changes

MaxGekk reviewed Nov 30, 2021

View reviewed changes

update tests, address comments

feb3715

MaxGekk approved these changes Dec 1, 2021

View reviewed changes

MaxGekk closed this in ce1f97f Dec 1, 2021

cloud-fan reviewed Dec 1, 2021

View reviewed changes

sarutak mentioned this pull request Dec 1, 2021

[SPARK-37326][SQL][FOLLOWUP] Fix the test for Java 11 #34771

Closed

sadikovi mentioned this pull request Dec 2, 2021

[SPARK-37326][SQL][FOLLOW-UP] Update code and tests for TimestampNTZ support in CSV data source #34777

Closed

	/**
	* Generates a temporary path without creating the actual file/directory, then pass it to `f`. If
	* a file/directory is created there by `f`, it will be delete after `f` returns.
	*/
	protected def withTempPath(f: File => Unit): Unit = {

[SPARK-37326][SQL] Support TimestampNTZ in CSV data source #34596

[SPARK-37326][SQL] Support TimestampNTZ in CSV data source #34596

Conversation

sadikovi commented Nov 15, 2021 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

sadikovi commented Nov 15, 2021

SparkQA commented Nov 15, 2021

SparkQA commented Nov 15, 2021

SparkQA commented Nov 15, 2021

sadikovi commented Nov 15, 2021

sadikovi commented Nov 16, 2021

SparkQA commented Nov 16, 2021

SparkQA commented Nov 16, 2021

SparkQA commented Nov 16, 2021

sadikovi commented Nov 17, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sadikovi commented Nov 17, 2021

SparkQA commented Nov 18, 2021

SparkQA commented Nov 18, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Nov 18, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gengliangwang left a comment

Choose a reason for hiding this comment

sadikovi Nov 28, 2021 • edited Loading

Choose a reason for hiding this comment

sadikovi commented Nov 28, 2021

SparkQA commented Nov 28, 2021

SparkQA commented Nov 28, 2021

SparkQA commented Nov 29, 2021

SparkQA commented Nov 29, 2021

SparkQA commented Nov 29, 2021

SparkQA commented Nov 30, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HyukjinKwon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sadikovi commented Nov 30, 2021

SparkQA commented Dec 1, 2021

SparkQA commented Dec 1, 2021

SparkQA commented Dec 1, 2021

MaxGekk commented Dec 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sadikovi Dec 2, 2021 • edited Loading

Choose a reason for hiding this comment

sadikovi commented Nov 15, 2021 •

edited

Loading

sadikovi Nov 28, 2021 •

edited

Loading

sadikovi Dec 2, 2021 •

edited

Loading