Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON Encoding: Use Type_RAW for marshalling json #16637

Merged
merged 9 commits into from
Aug 29, 2024

Conversation

rohit-nayak-ps
Copy link
Contributor

@rohit-nayak-ps rohit-nayak-ps commented Aug 22, 2024

Description

This PR cherry-picks commits from #16615, but reverts the changes made in insert_select.go to treat Type_JSON differently, but uses all the added tests. #16615 is an attempt to fix the issue #16614 with sending json type as bind variable to vttablet.

This PR attempts to fix that issue by not sending the marshalled json objects in vreplication workflows as a newly introduced internal type Type_RAW which is encoded as raw bytes. Previously we used to encode Type_JSON as raw, which is incorrect since we do want to quote json strings. In vreplication workflows, we use json functions like json_object() to specify the json document to preserve precision of large integers.

This bug is present since several versions, so we backport to all supported versions.

Marked Do Not Merge while waiting for more 👀 on this change.

Related Issue(s)

Checklist

  • "Backport to:" labels have been added if this change should be back-ported to release branches
  • If this change is to be back-ported to previous releases, a justification is included in the PR description
  • Tests were added or are not required
  • Did the new or modified tests pass consistently locally and on CI?
  • Documentation was added or is not required

Copy link
Contributor

vitess-bot bot commented Aug 22, 2024

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

  • Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

  • Apply the release notes (needs details) label if users need to know about this change.
  • New features should be documented.
  • There should be some code comments as to why things are implemented the way they are.
  • There should be a comment at the top of each new or modified test to explain what the test does.

New flags

  • Is this flag really necessary?
  • Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

  • Each item in Jobs should be named in order to mark it as required.
  • If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • RPC changes should be compatible with vitess-operator
  • If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
  • vtctl command output order should be stable and awk-able.

@vitess-bot vitess-bot bot added NeedsBackportReason If backport labels have been applied to a PR, a justification is required NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsIssue A linked issue is missing for this Pull Request NeedsWebsiteDocsUpdate What it says labels Aug 22, 2024
@github-actions github-actions bot added this to the v21.0.0 milestone Aug 22, 2024
@rohit-nayak-ps rohit-nayak-ps added Type: Bug Component: VReplication and removed NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsWebsiteDocsUpdate What it says NeedsIssue A linked issue is missing for this Pull Request labels Aug 22, 2024
Copy link

codecov bot commented Aug 22, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.92%. Comparing base (feb845a) to head (223efa6).
Report is 32 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #16637      +/-   ##
==========================================
- Coverage   68.98%   68.92%   -0.06%     
==========================================
  Files        1562     1564       +2     
  Lines      200697   201371     +674     
==========================================
+ Hits       138446   138798     +352     
- Misses      62251    62573     +322     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@harshit-gangal harshit-gangal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes looks good to me

@rohit-nayak-ps rohit-nayak-ps changed the title WIP: VReplication JSON Encoding: Use Type_RAW for marshalling json VReplication JSON Encoding: Use Type_RAW for marshalling json Aug 23, 2024
@rohit-nayak-ps rohit-nayak-ps marked this pull request as ready for review August 23, 2024 13:37
@rohit-nayak-ps rohit-nayak-ps added Backport to: release-18.0 Needs to be back ported to release-18.0 Backport to: release-19.0 Needs to be back ported to release-19.0 Backport to: release-20.0 Needs to be backport to release-20.0 and removed NeedsBackportReason If backport labels have been applied to a PR, a justification is required labels Aug 23, 2024
@rohit-nayak-ps rohit-nayak-ps changed the title VReplication JSON Encoding: Use Type_RAW for marshalling json JSON Encoding: Use Type_RAW for marshalling json Aug 23, 2024
harshit-gangal and others added 8 commits August 28, 2024 23:13
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
…n binlog events instead of Type_JSON, which is now treated as a regular quoted type

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
@rohit-nayak-ps rohit-nayak-ps merged commit e89f684 into vitessio:main Aug 29, 2024
132 checks passed
@rohit-nayak-ps rohit-nayak-ps deleted the rohit/vrep-json-as-raw-type branch August 29, 2024 09:59
shlomi-noach pushed a commit that referenced this pull request Sep 4, 2024
…) (#16681)

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com>
Co-authored-by: Rohit Nayak <rohit@planetscale.com>
shlomi-noach pushed a commit that referenced this pull request Sep 4, 2024
…) (#16682)

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com>
Co-authored-by: Rohit Nayak <rohit@planetscale.com>
shlomi-noach pushed a commit that referenced this pull request Sep 4, 2024
…) (#16680)

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com>
Co-authored-by: Rohit Nayak <rohit@planetscale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Backport to: release-18.0 Needs to be back ported to release-18.0 Backport to: release-19.0 Needs to be back ported to release-19.0 Backport to: release-20.0 Needs to be backport to release-20.0 Component: Query Serving Component: VReplication Type: Bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bug Report: syntax error while trying to use INSERT INTO ... SELECT * FROM ... to copy data between keyspaces
4 participants