Locally load schemas #269

decentralgabe · 2022-12-13T04:57:06Z

Fixes #134

Don't think this will work for schemas that are not pre-defined. Meaning, if we're in local mode and try to load a http/s resource that's not defined it'll break. This should be fine...

It is 'turned on' by default with an init() block which isn't ideal. Could use some ideas to make this better.

codecov-commenter · 2022-12-13T05:06:33Z

Codecov Report

Merging #269 (28bc011) into main (d46fd8a) will increase coverage by 0.18%.
The diff coverage is 71.26%.

❗ Current head 28bc011 differs from pull request most recent head a674489. Consider uploading reports for the commit a674489 to get more accurate results

@@            Coverage Diff             @@
##             main     #269      +/-   ##
==========================================
+ Coverage   60.18%   60.35%   +0.18%     
==========================================
  Files          38       39       +1     
  Lines        4329     4399      +70     
==========================================
+ Hits         2605     2655      +50     
- Misses       1298     1315      +17     
- Partials      426      429       +3

Impacted Files	Coverage Δ
credential/rendering/schema.go	`0.00% <0.00%> (ø)`
schema/jsonschema.go	`52.00% <ø> (-2.72%)`	⬇️
credential/exchange/schema.go	`36.36% <66.67%> (ø)`
schema/loader.go	`72.60% <72.60%> (ø)`
credential/manifest/schema.go	`52.63% <100.00%> (ø)`
credential/schema/vcjsonschema.go	`46.00% <100.00%> (ø)`

decentralgabe · 2022-12-13T05:08:03Z

Tested by turning off wifi and running the tests 😉

nitro-neal · 2022-12-13T14:49:08Z

schema/known_schemas/pe-submission-claim-format-designations.json

@@ -2,17 +2,10 @@
  "$schema": "http://json-schema.org/draft-07/schema#",


did everything else get deleted but not this file?

schema/jsonschema.go

nitro-neal

Good job! We the way we had it before was so janky

dwn/model.go

andresuribe87 · 2022-12-13T10:58:37Z

schema/known_schemas/pe-submission-claim-format-designations.json

@@ -2,17 +2,10 @@
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Presentation Submission Claim Format Designations",
  "type": "object",
-  "additionalProperties": false,
-  "properties": {
+  "definitions": {


What's different about this schema that you didn't delete it? Is it because it's not published in the web?

I didn't delete most schemas. I deleted the -deref.json files. This schema actually had an update which I pulled in. cc: @nitro-neal

andresuribe87 · 2022-12-13T15:23:39Z

schema/jsonschema.go

 func init() {
 	httploader.Client = &http.Client{
 		Timeout: time.Second * 10,
 	}
 }

+const (
+	// defaultSchemaURL is a placeholder that's needed to load any schema
+	defaultSchemaURL = "schema.json"


Not sure I follow why this is needed, can you elaborate?

the library requires you to provide a url for the schema and we don't really care what this is but it needs to be a placeholder https://github.com/santhosh-tekuri/jsonschema/blob/master/compiler.go#L65

schema/loader.go

andresuribe87 · 2022-12-13T15:32:29Z

schema/loader.go

+}
+
+// DisableLocalLoad disables the ability to load schemas locally, clearing current locally loaded schemas
+func DisableLocalLoad() {


Is this being used?

no, but I believe it should remain as an option for users of the lib

andresuribe87 · 2022-12-13T15:56:40Z

schema/loader.go

+	jsonschema.Loaders["https"] = func(url string) (io.ReadCloser, error) {
+		schema, ok := localSchemas[strings.TrimPrefix(url, "https://")]
+		if !ok {
+			return nil, fmt.Errorf("%q not found", url)
+		}
+		return io.NopCloser(strings.NewReader(schema)), nil
+	}
+	jsonschema.Loaders["http"] = func(url string) (io.ReadCloser, error) {
+		schema, ok := localSchemas[strings.TrimPrefix(url, "http://")]
+		if !ok {
+			return nil, fmt.Errorf("%q not found", url)
+		}
+		return io.NopCloser(strings.NewReader(schema)), nil
+	}


I have a couple of suggestions for your consideration:

Rework LocalLoader into LocalLoaderFactory.

Remove the jsonschema.Loader directives from this file, and the init functions (let users of the SDK do it themselves).

Provide a method for creating loaders, something like what's below.

type Loader func(url string) (io.ReadCloser, error) func (l *LocalLoaderFactory) ProtocolLoader(protocol string) Loader { return func(url string) (io.ReadCloser, error) { schema, ok := localSchemas[strings.TrimPrefix(url, protocol + "://")] if !ok { return nil, fmt.Errorf("%q not found", url) } return io.NopCloser(strings.NewReader(schema)), nil } }

Once that is done, then user's of the SDK can decide where to override the jsonschema loader.

// Somewhere in the ssi-service loaderFactory := schema.NewLocalLoaderFactory() jsonschema.Loader["http"] = loaderFactory.ProtocolLoader("http") jsonschema.Loader["https"] = loaderFactory.ProtocolLoader("https")

Thoughts?

I'm open to this but I think it's important we're able to enable local loading for CI. how do you think we should go about that?

How about defining the TestMain (as described in this https://pkg.go.dev/testing#hdr-Main)? Within that function we can patch the jsonschema lib.

decentralgabe · 2022-12-13T21:30:25Z

schema/loader.go

+}
+
+// AddCachedSchema adds a schema to be cached
+func (cl *CachingLoader) AddCachedSchema(schemaURI, schema string) error {


adjusted this to be a caching loader with a test. no more init.

andresuribe87 · 2022-12-13T21:33:27Z

schema/loader.go

+// NewCachingLoader returns a new CachingLoader that enables the ability to cache http and https schemas
+func NewCachingLoader() CachingLoader {
+	localSchemas := make(map[string]string)
+	jsonschema.Loaders["https"] = func(url string) (io.ReadCloser, error) {


I find it odd that a constructor has side effects. What's the rationale of patching jsonschema within the constructor?

most, if not all, schemas will be loaded by http/s. having users configure additional loaders leaks details of the library that they shouldn't know or need to care about. this loader will work fine for the 99%

schema/loader.go

andresuribe87 · 2022-12-13T21:36:53Z

schema/loader_test.go

+	"github.com/stretchr/testify/assert"
+)
+
+func TestCachingLoader(t *testing.T) {


Consider splitting this large test into multiple smaller unit tests that exercise a single behavior.

split into 2

andresuribe87 · 2022-12-13T21:50:04Z

schema/loader.go

+func NewCachingLoader() CachingLoader {
+	localSchemas := make(map[string]string)
+	jsonschema.Loaders["https"] = func(url string) (io.ReadCloser, error) {
+		schema, ok := localSchemas[strings.TrimPrefix(url, "https://")]


This might not be safe for concurrent use (and cause a panic), depending on how users of the SDK exercise it.

A goroutine might be reading from the map, while another one writes to the map by calling AddCachedSchema.

Consider writing once to the map only, in the constructor. Concurrent reads on a map are safe.

this constructor should only be called once - and I don't imagine callers will be further configuring the loaders...if they do then yes they may have concurrency issues. is this worth caring about?

decentralgabe · 2022-12-14T19:05:12Z

schema/loader.go

+	jsonschema.Loaders["https"] = cl.cachingLoaderForProtocol("https")
+}
+
+func (cl *CachingLoader) cachingLoaderForProtocol(protocol string) func(url string) (io.ReadCloser, error) {


updated to lock on map writes.

decentralgabe · 2022-12-14T19:05:20Z

schema/loader.go

+		}
+
+		// read the contents and cache and prevent future lookups
+		contents, err := io.ReadAll(loaded)


added cache on load from http

andresuribe87 · 2022-12-14T19:52:26Z

schema/loader.go

+}
+
+func (cl *CachingLoader) cachingLoaderForProtocol(protocol string) func(url string) (io.ReadCloser, error) {
+	var m sync.Mutex


I think this should be a shared lock across all loaders (i.e. move this variable to be a member of CachingLoader. As it stands, you can have a goroutine writing via a schema fetched from the http protocol, while another goroutine reads a schema fetched from the https protocol. Does this make sense?

yes but isn't that fine, since they're different protocols?

schema/loader.go

andresuribe87 · 2022-12-14T20:05:28Z

schema/loader.go

+}
+
+func (cl *CachingLoader) cachingLoaderForProtocol(protocol string) func(url string) (io.ReadCloser, error) {
+	var m sync.Mutex


Did you consider using a sync.RWMutex in order to lock more granularly? It could increase read throughput (though that's a small optimization).

Co-authored-by: Andres Uribe <auribe@tbd.email>

andresuribe87

This is awesome, I'm so excited about this change. Thanks for tackling!

schema/loader.go

andresuribe87 · 2022-12-14T21:55:41Z

schema/loader.go

+
+// NewCachingLoader returns a new CachingLoader that enables the ability to cache http and https schemas
+func NewCachingLoader(schemas map[string]string) (*CachingLoader, error) {
+	// make sure only one process can write to the map at a time


nit: move this to be a comment for the schemas function.

Co-authored-by: Andres Uribe <auribe@tbd.email>

decentralgabe added 5 commits December 12, 2022 18:00

move schemas to their respective folders

e7bdc7e

playing with loaders

4730f6f

temp

690291c

local loading partially working

790c136

caching working

08654fb

lints

619bd82

decentralgabe marked this pull request as ready for review December 13, 2022 05:08

decentralgabe requested review from nitro-neal and andresuribe87 as code owners December 13, 2022 05:08

empty block

df79526

nitro-neal reviewed Dec 13, 2022

View reviewed changes

schema/jsonschema.go Outdated Show resolved Hide resolved

nitro-neal approved these changes Dec 13, 2022

View reviewed changes

andresuribe87 reviewed Dec 13, 2022

View reviewed changes

decentralgabe added 4 commits December 13, 2022 11:53

caching loader

3819a41

loader test

8ec1c5b

local loading

8bced79

add testmain to all schema tests

c945078

decentralgabe commented Dec 13, 2022

View reviewed changes

decentralgabe mentioned this pull request Dec 13, 2022

Speedup tests by caching jsonschemas TBD54566975/ssi-service#214

Closed

andresuribe87 reviewed Dec 13, 2022

View reviewed changes

decentralgabe added 4 commits December 13, 2022 14:28

cr comments

5718a53

break up test

5be3d9c

remove dwn code-not used

1809e72

pr feedback

5e3bc95

decentralgabe commented Dec 14, 2022

View reviewed changes

lint

403dc7e

andresuribe87 reviewed Dec 14, 2022

View reviewed changes

decentralgabe and others added 3 commits December 14, 2022 13:26

Apply suggestions from code review

b511cca

Co-authored-by: Andres Uribe <auribe@tbd.email>

use sync map

5364fe9

Update schema/loader.go

4360444

Co-authored-by: Andres Uribe <auribe@tbd.email>

andresuribe87 approved these changes Dec 14, 2022

View reviewed changes

decentralgabe and others added 2 commits December 14, 2022 14:01

Update schema/loader.go

28bc011

Co-authored-by: Andres Uribe <auribe@tbd.email>

merge

a674489

decentralgabe merged commit e2badfb into main Dec 14, 2022

decentralgabe deleted the cache-schemas branch December 14, 2022 22:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Locally load schemas #269

Locally load schemas #269

decentralgabe commented Dec 13, 2022 •

edited

Loading

codecov-commenter commented Dec 13, 2022 •

edited

Loading

decentralgabe commented Dec 13, 2022

nitro-neal Dec 13, 2022

nitro-neal left a comment

andresuribe87 Dec 13, 2022

decentralgabe Dec 13, 2022

andresuribe87 Dec 13, 2022

decentralgabe Dec 13, 2022

andresuribe87 Dec 13, 2022

decentralgabe Dec 13, 2022

andresuribe87 Dec 13, 2022

decentralgabe Dec 13, 2022

andresuribe87 Dec 13, 2022

decentralgabe Dec 13, 2022

andresuribe87 Dec 13, 2022

decentralgabe Dec 13, 2022

andresuribe87 Dec 13, 2022

decentralgabe Dec 13, 2022

andresuribe87 Dec 13, 2022

decentralgabe Dec 13, 2022

decentralgabe Dec 14, 2022

decentralgabe Dec 14, 2022

andresuribe87 Dec 14, 2022

decentralgabe Dec 14, 2022

andresuribe87 Dec 14, 2022

andresuribe87 left a comment

andresuribe87 Dec 14, 2022

		@@ -2,17 +2,10 @@
		"$schema": "http://json-schema.org/draft-07/schema#",

Locally load schemas #269

Locally load schemas #269

Conversation

decentralgabe commented Dec 13, 2022 • edited Loading

codecov-commenter commented Dec 13, 2022 • edited Loading

Codecov Report

decentralgabe commented Dec 13, 2022

Choose a reason for hiding this comment

nitro-neal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andresuribe87 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

decentralgabe commented Dec 13, 2022 •

edited

Loading

codecov-commenter commented Dec 13, 2022 •

edited

Loading