DTS-3385: Switch getMetadata to use delta-kernel #505
Conversation
pranavsuku-db
requested review from
linzhou-db
and removed request for
linzhou-db
server/src/main/scala/io/delta/kernelsharedtable/DeltaSharedTableKernel.scala
Outdated
Show resolved
Hide resolved
| case e: io.delta.kernel.exceptions.TableNotFoundException => | ||
| throw new FileNotFoundException(e.getMessage) | ||
| case e: Throwable => | ||
| throw new FileNotFoundException(e.getMessage) |
There was a problem hiding this comment.
@linzhou-db I needed to catch all throwabales here to fix the issue in this test case: table_changes_exception: cdf_table_missing_log from DeltaSharingSuite.scala. I get a runtime error for this test case on the table.getLatestSnapshot(engine).asInstanceOf[SnapshotImpl] line and it needs to be handled as a 400 error and not a 500 error so I need to throw a FileNotFound, but I could not figure out what I need to catch, and so I caught all throwables.
There was a problem hiding this comment.
what's the error and stacktrace?
There was a problem hiding this comment.
doing string matching on error message now since fix is not in delta-kernel yet
|
|
||
| case class PropertyAllowedValues(property: String, allowedValues: Seq[String]) | ||
|
|
||
| /** |
There was a problem hiding this comment.
we can remove these for now and add them until they are needed.
until L77
| } | ||
| } | ||
|
|
||
| /** |
There was a problem hiding this comment.
L84 to L125 can be removed too?
There was a problem hiding this comment.
I need DeletionVectorsTableFeature and ColumnMappingTableFeature but removed the rest
There was a problem hiding this comment.
let's see if these two can be removed once the usage of clientReaderFeatures can be removed.
| case e: io.delta.kernel.exceptions.TableNotFoundException => | ||
| throw new FileNotFoundException(e.getMessage) | ||
| case e: Throwable => | ||
| throw new FileNotFoundException(e.getMessage) |
There was a problem hiding this comment.
what's the error and stacktrace?
|
|
||
| // Validate snapshot version, throw error if it's before the valid start version. | ||
| val snapshotVersion = snapshot.getVersion(engine) | ||
| if (snapshotVersion < validStartVersion) { |
There was a problem hiding this comment.
I don't think we need the logic around validStartVersion in the oss code?
| ) | ||
| } | ||
|
|
||
| val refreshTokenOpt = refreshToken.map(decodeAndValidateRefreshToken) |
There was a problem hiding this comment.
this can be removed and added back later as well.
There was a problem hiding this comment.
its needed to get snapshot, dont think I can delete this yet:
val snapshot = getSharedTableSnapshot(
table,
engine,
version.orElse(refreshTokenOpt.map(_.getVersion)),
timestamp,
validStartVersion = tableConfig.startVersion,
clientReaderFeatures =
allowedClientReaderFeatures(clientReaderFeatures = clientReaderFeaturesSet)
)
There was a problem hiding this comment.
let me know if I missed anything, I remember we don't have any test cases setting client features in the request, in that case, we can always just go with the condition on L175, always assuming clientReaderFeature is empty.
There was a problem hiding this comment.
And add this back when we actually add those tests, to make this code well tested.
There was a problem hiding this comment.
For refreshTokenOpt, it's only used when queryTable(includeFiles = true).
So you can just use version in getSharedTableSnapshot without the orElse.
There was a problem hiding this comment.
makes sense, fixed
| ) | ||
| } | ||
|
|
||
| val refreshTokenOpt = refreshToken.map(decodeAndValidateRefreshToken) |
There was a problem hiding this comment.
let me know if I missed anything, I remember we don't have any test cases setting client features in the request, in that case, we can always just go with the condition on L175, always assuming clientReaderFeature is empty.
| ) | ||
| } | ||
|
|
||
| val refreshTokenOpt = refreshToken.map(decodeAndValidateRefreshToken) |
There was a problem hiding this comment.
And add this back when we actually add those tests, to make this code well tested.
| ) | ||
| } | ||
|
|
||
| val refreshTokenOpt = refreshToken.map(decodeAndValidateRefreshToken) |
There was a problem hiding this comment.
For refreshTokenOpt, it's only used when queryTable(includeFiles = true).
So you can just use version in getSharedTableSnapshot without the orElse.
| } | ||
| } | ||
|
|
||
| /** |
There was a problem hiding this comment.
let's see if these two can be removed once the usage of clientReaderFeatures can be removed.
server/src/main/scala/io/delta/kernelsharedtable/DeltaSharedTableKernel.scala
Show resolved
Hide resolved
server/src/main/scala/io/delta/kernelsharedtable/DeltaSharedTableKernel.scala
Outdated
Show resolved
Hide resolved
server/src/main/scala/io/delta/kernelsharedtable/DeltaSharedTableKernel.scala
Outdated
Show resolved
Hide resolved
server/src/main/scala/io/delta/kernelsharedtable/DeltaSharedTableKernel.scala
Outdated
Show resolved
Hide resolved
server/src/main/scala/io/delta/sharing/server/DeltaAction.scala
Outdated
Show resolved
Hide resolved
server/src/main/scala/io/delta/kernelsharedtable/DeltaSharedTableKernel.scala
Show resolved
Hide resolved
getMetadata now uses delta-kernel