-
Notifications
You must be signed in to change notification settings - Fork 813
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fractional seconds are lost #103
Comments
Would you be willing to release a 0.3.2 with this fix? If not, when do you think you might release the library with this fix? |
Also, if these methods were not |
Hi @rhauch
Of course, I'll do it right after work.
All deserialize* are Anyway, thank you for the bug report (and all the details)! I'll let you know when 0.3.2 is in Maven Central. |
@shyiko, sounds great. Thank you very much!
I actually looked at overriding, but because of the private method it would have required copying a lot of code. Making all the methods protected would be great for situations exactly like this, and would reduce the need for a patch release. BTW, please double-check my logic, too. |
I understand. The reason I'm reluctant to marking all of them
You can count on it :) |
0.3.2 deployed to Maven Central. Cheers 🍻 |
Added an integration test case to diagnose the loss of the fractional seconds from MySQL temporal values. The problem appears to be a bug in the MySQL Binary Log Connector library that we used, and this bug was reported as shyiko/mysql-binlog-connector-java#103. That was fixed in version 0.3.2 of the library, which Stanley was kind enough to release for us. During testing, though, several issues were discovered in how temporal values are handled and converted from the MySQL events, through the MySQL Binary Log client library, and through the Debezium MySQL connector to conform with Kafka Connect's various temporal logical schema types. Most of the issues involved converting most of the temporal values from local time zone (which is how they are created by the MySQL Binary Log client) into UTC (which is how Kafka Connect expects them). Really, java.util.Date doesn't have time zone information and instead tracks the number of milliseconds past epoch, but the conversion of normal timestamp information to the milliseconds past epoch in UTC depends on the time zone in which that conversion happens.
Added an integration test case to diagnose the loss of the fractional seconds from MySQL temporal values. The problem appears to be a bug in the MySQL Binary Log Connector library that we used, and this bug was reported as shyiko/mysql-binlog-connector-java#103. That was fixed in version 0.3.2 of the library, which Stanley was kind enough to release for us. During testing, though, several issues were discovered in how temporal values are handled and converted from the MySQL events, through the MySQL Binary Log client library, and through the Debezium MySQL connector to conform with Kafka Connect's various temporal logical schema types. Most of the issues involved converting most of the temporal values from local time zone (which is how they are created by the MySQL Binary Log client) into UTC (which is how Kafka Connect expects them). Really, java.util.Date doesn't have time zone information and instead tracks the number of milliseconds past epoch, but the conversion of normal timestamp information to the milliseconds past epoch in UTC depends on the time zone in which that conversion happens.
Fix circleci build, in the integration test suite multiple events could be in the binary log
I'm using 0.3.1, and think there is an issue with how the library is deserializing fractional seconds. In particular, the
AbstractRowsEventDataDeserializer.getFractionalSeconds(...)
method is defined as follows:Here,
meta
is the metadata for the column, which for temporal types is the fractional seconds part, or fsp, as described in the MySQL documentation.Consider an example table with this definition (taken from the MySQL documentation):
and the following insert statement:
Here, the fsp -- and thus the value of
meta
-- for all three columns is2
. The value of thefractionalSeconds
variable read from the stream is then78
(since rounding occurs within the database). However, the following lines in the method:seem to not make any sense, because the result will always be 0 unless fsp is 4, 5 or 6, and the result will always be incorrect.
If the function is to return the number of milliseconds, then it seems like the code should simply multiply by the correct factor of 10 (based on the number of digits in the precision, or
meta
):So for
fractionalSeconds=78
andmeta=2
, thenfactor=10
and the method return 780, which is the correct number of milliseconds.For an fsp of 1,
meta=1' and
fractionalSeconds=8` (assuming MySQL rounded), then the method would return 800, which is the correct number of milliseconds.Now, MySQL fsp values can be integers between 1 and 6, so the above code needs to amended to handle
meta
values that are greater than 3. A simple way to do that would be to change the factor to use exponentials, but a more efficient way is to do the following:I'm sure there are other ways this could be written, but the above code uses simple integer math and is at least correct.
Note that this same problem appears in the master branch, too.
The text was updated successfully, but these errors were encountered: