Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parse binlog timestamp in UTC #1322

Merged
merged 2 commits into from
Oct 10, 2023
Merged

Parse binlog timestamp in UTC #1322

merged 2 commits into from
Oct 10, 2023

Conversation

meiji163
Copy link
Contributor

@meiji163 meiji163 commented Oct 8, 2023

Description

This PR addresses a situation where TIMESTAMP fields can be corrupted. Quoting @dm-2:

gh-ost is incorrectly modifying the timezone of timestamp fields of rows when the following criteria are met:

  • database is running in non-UTC timezone, and
  • row is changed whilst a migration is running, and
  • field is timestamp, and
  • field contains a non-UTC timestamp

This is caused by how go-mysql/replication package parses binlog timestamps based on timezone. In the latest release this behaviour was disabled by editing the vendored package 7e0bdbe , but in the master branch it is enabled.

  • contributed code is using same conventions as original code
  • script/cibuild returns with no formatting errors, build errors or unit test errors.

@meiji163 meiji163 requested a review from rashiq October 9, 2023 03:34
@meiji163
Copy link
Contributor Author

meiji163 commented Oct 10, 2023

This has run successfully on testing replicas on 175 production tables total, so I am fairly confident to merge.

@meiji163 meiji163 merged commit 03f7fc8 into master Oct 10, 2023
9 checks passed
@timvaillancourt
Copy link
Collaborator

@meiji163 I'm so glad someone finally figured this out and it's great it didn't require modifying vendored code. Thank you!

@meiji163 meiji163 mentioned this pull request Oct 31, 2023
RainbowDashy pushed a commit to bytebase/gh-ost that referenced this pull request Jan 2, 2024
d-bytebase pushed a commit to bytebase/gh-ost that referenced this pull request Jan 2, 2024
* Add `focal` build

* upgrade codeQL action to v2

* Pin docker image to stretch (github#1324)

* pin stretch image

* update stretch repo

* Update Dockerfile.test

* parse binlog timestamp in UTC (github#1322)

* upgrade standard library deps (github#1332)

* fix: typo (github#1331)

Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Remove stale RELEASE_VERSION file (github#1349)

Signed-off-by: Tim Vaillancourt <tvaillancourt@slack-corp.com>

* Force index in unique key range queries (github#1237)

* WIP

* Pass entire sql.UniqueKey

* newline for limit

* Rename var

---------

Co-authored-by: meiji163 <meiji163@github.com>

* Include git commit in version (github#1347)

* Include git commit in version

Signed-off-by: Tim Vaillancourt <tvaillancourt@slack-corp.com>

* Make --version output change less-breaking

---------

Signed-off-by: Tim Vaillancourt <tvaillancourt@slack-corp.com>

* Fix tokens in inspector logging (github#1226)

* Cleanup whitespace and comments in SQL query text (github#1246)

* Cleanup whitespace in SQL query text

* cleanup

* Add indent

* Update unit tests

* Update unit tests, pt 2

* Fix tweaks

* Fix merge conflict resolution

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

---------

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Fix `--version` output (github#1352)

* Fix `--version` output

Signed-off-by: Tim Vaillancourt <tvaillancourt@slack-corp.com>

* Always fallback version/commit if undef

Signed-off-by: Tim Vaillancourt <tvaillancourt@slack-corp.com>

---------

Signed-off-by: Tim Vaillancourt <tvaillancourt@slack-corp.com>

* Update to `go1.21` (github#1351)

* go1.21 + bullseye

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* go1.21 + bullseye pt 2

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* checkout before setup-go

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* go fmt

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Use golangci-lint 1.54.2 to support go1.21

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* stop using io/ioutil to make linter happy

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Fix typo

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Lint

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* revert replica-tests CI to ubuntu 20 due to linker errors

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Update ensure-go-installed

* use `ubuntu-latest` for `ci` job

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* stretch -> bullseye

Signed-off-by: Tim Vaillancourt <tvaillancourt@slack-corp.com>

---------

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Signed-off-by: Tim Vaillancourt <tvaillancourt@slack-corp.com>

* Replace deprecated `github.com/satori/go.uuid` lib (github#1354)

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Add `CODEOWNERS` file to auto-request reviews (github#1350)

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* go mod tidy

* update test

* fix lint

---------

Signed-off-by: Tim Vaillancourt <tvaillancourt@slack-corp.com>
Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Co-authored-by: Rashiq <rashiq@users.noreply.github.com>
Co-authored-by: meiji163 <meiji163@github.com>
Co-authored-by: guangwu <guoguangwu@magic-shield.com>
Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants