Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: ujson #1038

Closed
wants to merge 4 commits into from
Closed

feat: ujson #1038

wants to merge 4 commits into from

Conversation

n0izn0iz
Copy link
Contributor

@n0izn0iz n0izn0iz commented Aug 9, 2023

This is a basic json (un)marshaller

It ports strings (un)quoting directly from golang's encoding/json

It can work on interfaces to ease writing marshallers/unmarshallers but can't fully deduct types since no reflection, for example, for objects you need to implement FromJSONAble and/or JSONAble interface

It also adds unicode/utf16 to stdlibs, copied from golang 1.17.5, this is needed for JSON strings unquoting

Caveats:

  • big numbers are not supported
  • all other *int* types (int, int32, uint, uint32, etc...) use strconv.Atoi for unmarshal so may loose precision
  • floats are not supported
  • objects require custom marshaller/unmarshaller (which are easy to write with this library API)
  • avl.Trees are automatically supported in marshaller but require a custom parser in unmarshaller since the values' types can't de deducted
  • slices require custom marshaller/unmarshaller since I did not find a way to match on []interface{}

A version of this package is already used in Teritori's dao and feed contracts to return complex data to ui and also to encode actions that can be executed in proposals

See the tests for example usages

Although this might be improved, I think it's already useful as-is and could be merged if it passes review

Contributors' checklist...
  • Added new tests, or not needed, or not feasible
  • Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory
  • Updated the official documentation or not needed
  • No breaking changes were made, or a BREAKING CHANGE: xxx message was included in the description
  • Added references to related issues and PRs
  • Provided any useful hints for running manual tests
  • Added new benchmarks to generated graphs, if any. More info here.

@github-actions github-actions bot added the 🧾 package/realm Tag used for new Realms or Packages. label Aug 9, 2023
@moul
Copy link
Member

moul commented Aug 10, 2023

As we discussed earlier, we're working on a more natural way to handle this situation. However, since that might take a while, I'm really impressed with how you've tackled this complex contract. It's great to see you diving into this, and I believe your work will continue to be a good resource for learning—even when it's eventually replaced by an official update.

So please, keep up the good work! Many thanks.

@n0izn0iz n0izn0iz force-pushed the ujson branch 3 times, most recently from db6dc02 to 72b8699 Compare August 24, 2023 17:48
@github-actions github-actions bot added the 📦 🤖 gnovm Issues or PRs gnovm related label Aug 24, 2023
@n0izn0iz n0izn0iz force-pushed the ujson branch 2 times, most recently from c7dd55f to 42ed2d7 Compare August 24, 2023 18:02
@n0izn0iz n0izn0iz marked this pull request as ready for review August 24, 2023 18:04
@n0izn0iz n0izn0iz requested review from jaekwon, moul and a team as code owners August 24, 2023 18:04
@n0izn0iz n0izn0iz force-pushed the ujson branch 5 times, most recently from 972ebfd to dfb9f7b Compare August 24, 2023 18:36
Signed-off-by: Norman Meier <norman@berty.tech>
Signed-off-by: Norman Meier <norman@berty.tech>
Copy link
Member

@thehowl thehowl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your efforts, Norman.

One thing I wonder about is whether doing proper tokenization and then AST is the proper way to go around making a JSON parser. This is a topic where I've personally played around with1 before, so I have a bit of an idea of what libraries are out there.

I tend to like jsonparser for reflection-less JSON parsing. Though as the name suggests, the library doesn't do marshaling, so someting different would still be needed for Gno.

Still, the design is something that I think we can borrow: instead of working with ASTs (which package users would have to support and learn about when starting to use ujson) maybe we can let the end users just work directly with []byte? (Also further strengthening the point I was making of using Unmarshal/MarshalJSON for custom types, rather than a new interface)

Footnotes

  1. https://gitlab.com/tyge/nanojson

}

// does not work for slices, use ast exploration instead
func (ast *JSONASTNode) ParseAny(ptr *interface{}) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

*interface{} is not what you think it is, you probably want to use interface{} here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you elaborate? I was thinking this was enforcing that I get a pointer as argument so I can assign to what it's pointing at

Comment on lines +14 to +16
type FromJSONAble interface {
FromJSON(ast *JSONASTNode)
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like this personally. JSON is a simple format with well-established conventions in Go; the most known of these is that types which should do custom JSON En/decoding should implement MarshalJSON and UnmarshalJSON, with well-defined signatures. I'd prefer if we kept it that way, even while we don't have encoding/json, so that when we do have reflection-based json encoding/decoding, it is relatively painless to port the old code to the new one.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, as a bonus, I also like MarshalText if you want to implement support for that :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying to avoid using the same naming as encoding/json to prevent confusion

@n0izn0iz
Copy link
Contributor Author

n0izn0iz commented Sep 8, 2023

I chose to pass the AST because it makes writing the unmarshalers very easy, the goal here was ease of use since it will be superseded by a proper generic object encoding which will not be in gno code if I understand correctly

Signed-off-by: Norman Meier <norman@berty.tech>
Signed-off-by: Norman Meier <norman@berty.tech>
@netlify
Copy link

netlify bot commented Sep 12, 2023

Deploy Preview for gno-docs failed.

Name Link
🔨 Latest commit 5fe3bc0
🔍 Latest deploy log https://app.netlify.com/sites/gno-docs/deploys/65005e78d66f010008f5792d

@n0izn0iz
Copy link
Contributor Author

now in #1154

@n0izn0iz n0izn0iz closed this Feb 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
📦 🤖 gnovm Issues or PRs gnovm related 🧾 package/realm Tag used for new Realms or Packages.
Projects
Status: 🌟 Wanted for Launch
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants