Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

util: Add JSONWriter #10244

Merged
merged 5 commits into from
Dec 8, 2023
Merged

util: Add JSONWriter #10244

merged 5 commits into from
Dec 8, 2023

Conversation

breezewish
Copy link
Member

@breezewish breezewish commented Dec 5, 2023

What problem does this PR solve?

Issue Number: ref #1799

This PR is extracted from #10197

What is changed and how it works?

This PR introduces JSONWriter to build JSON in an efficient and structural way.

Usage:

// out is a io.Writer, for example, 
// out := &bytes.Buffer{}

w := BorrowJSONWriter(out)

w.WriteObject(func() {
  w.WriteObjectField("payload", func() {
    w.WriteObjectField("dml", func() {
      w.WriteStringField("statement", "INSERT")
      w.WriteUint64Field("ts", 100)
    })
  })
  w.WriteObjectField("source", func() {
    w.WriteStringField("source", "TiCDC")
    w.WriteInt64Field("version", 1)
  })
})

ReturnJSONWriter(w)

Output (formatted):

{
  "payload": {
    "dml": {
      "statement": "INSERT",
      "ts": 100
    }
  },
  "source": {
    "source": "TiCDC",
    "version": 1
  }
}

Performance:

Debezium JSON using JSON Writer:

BenchmarkEncodeOneTinyColumn-10    	 2771671	       430.0 ns/op	       0 B/op	       0 allocs/op
BenchmarkEncodeLargeText-10        	  974575	      1250 ns/op	       0 B/op	       0 allocs/op
BenchmarkEncodeLargeBinary-10      	  978442	      1143 ns/op	    1154 B/op	       3 allocs/op

Debezium JSON using golang's JSON Marshaler:

BenchmarkEncodeOneTinyColumn-10    	 1000000	      1007 ns/op	    1072 B/op	      10 allocs/op
BenchmarkEncodeLargeText-10        	  526785	      2144 ns/op	    3169 B/op	      12 allocs/op
BenchmarkEncodeLargeBinary-10      	  363991	      3232 ns/op	    5348 B/op	      13 allocs/op

Canal JSON using easyjson:

BenchmarkEncodeOneTinyColumn-10    	 2046062	       593.4 ns/op	     819 B/op	       6 allocs/op
BenchmarkEncodeLargeText-10        	  703260	      1544 ns/op	    2928 B/op	       7 allocs/op
BenchmarkEncodeLargeBinary-10      	  432828	      2508 ns/op	    3968 B/op	       9 allocs/op

Check List

Tests

  • Unit test

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note

None

Signed-off-by: Wish <breezewish@outlook.com>
@ti-chi-bot ti-chi-bot bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Dec 5, 2023
@sre-bot
Copy link

sre-bot commented Dec 5, 2023

CLA assistant check
All committers have signed the CLA.

@ti-chi-bot ti-chi-bot bot added contribution This PR is from a community contributor. first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. labels Dec 5, 2023
Copy link
Contributor

ti-chi-bot bot commented Dec 5, 2023

Hi @breezewish. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ti-chi-bot ti-chi-bot bot added needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. release-note-none Denotes a PR that doesn't merit a release note. and removed release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Dec 5, 2023
@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Dec 6, 2023
Signed-off-by: Wish <breezewish@outlook.com>
Signed-off-by: Wish <breezewish@outlook.com>
@ti-chi-bot ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Dec 6, 2023
Signed-off-by: Wish <breezewish@outlook.com>
@breezewish
Copy link
Member Author

/ok-to-test

Copy link
Contributor

ti-chi-bot bot commented Dec 6, 2023

@breezewish: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/ok-to-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Dec 6, 2023
Copy link
Contributor

ti-chi-bot bot commented Dec 6, 2023

[LGTM Timeline notifier]

Timeline:

  • 2023-12-06 02:51:52.589430902 +0000 UTC m=+1582341.254657098: ☑️ agreed by asddongmen.
  • 2023-12-06 06:12:11.133902703 +0000 UTC m=+1594359.799128899: ☑️ agreed by 3AceShowHand.

Signed-off-by: Wish <breezewish@outlook.com>
@breezewish
Copy link
Member Author

/retest

@breezewish
Copy link
Member Author

/retest

@asddongmen
Copy link
Contributor

/merge

Copy link
Contributor

ti-chi-bot bot commented Dec 8, 2023

@asddongmen: We have migrated to builtin LGTM and approve plugins for reviewing.

Please use /approve when you want approve this pull request.

The changes announcement: LGTM plugin changes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@asddongmen
Copy link
Contributor

/retest

@asddongmen
Copy link
Contributor

/lgtm

Copy link
Contributor

ti-chi-bot bot commented Dec 8, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3AceShowHand, asddongmen, sdojjy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [3AceShowHand,asddongmen,sdojjy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot merged commit bfc1e00 into pingcap:master Dec 8, 2023
14 checks passed
@breezewish breezewish deleted the json-writer branch December 8, 2023 06:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved contribution This PR is from a community contributor. first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. lgtm needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants