[POC] Adds a "dynamic" module to dynamically load services from smithy models

[Baccata]

This relies on a one-pass + suspension to build a map of schemas that work against dynamic data structures, backing product types with Array[Any], and union-types with (Int, Any)

The goal is notably to get to a state where services could be proxied dynamically, whether for actual proxyfication purposes, or to create CLIs that do not need recompilation, or create UIs/playgrounds dynamically.

[Baccata]

Don't trust the CI, the new module is not currently wired (nor do I intend to do it right away)

[kubukoz]

I can now confirm this works on JS :)

[kubukoz]

@Baccata, eeef6b8 (#17) adds a failing test for something I've experienced when trying to compile dynamic schemas in the other CLI module. The stack trace in this test is as follows:

[E]   NoSuchElementException: key not found: smithy.api#String
[E] 
[E]   Map.scala:274                           scala.collection.MapOps#default
[E]   Map.scala:273                           scala.collection.MapOps#default$
[E]   Map.scala:405                           scala.collection.AbstractMap#default
[E]   HashMap.scala:425                       scala.collection.mutable.HashMap#apply
[E]   DynamicModelCompiler.scala:82           smithy4s.dynamic.Compiler$CompileVisitor#schema
[E]   DynamicModelCompiler.scala:180          smithy4s.dynamic.Compiler$CompileVisitor#$anonfun$structureShape$3
[E]   suspended.scala:32                      schematic.suspended$Schema#deferredSchema$lzycompute
[E]   suspended.scala:32                      schematic.suspended$Schema#deferredSchema
[E]   suspended.scala:35                      schematic.suspended$Schema#$anonfun$compile$1
[E]   DynamicModelSpec.scala:131              smithy4s.dynamic.model.DynamicModelSpec$Interpreter$GetFieldNames$#$anonfun$suspend$1
[E]   DynamicModelSpec.scala:127              smithy4s.dynamic.model.DynamicModelSpec$Interpreter$GetFieldNames$#$anonfun$genericStruct$2
[E]   StrictOptimizedIterableOps.scala:118    scala.collection.StrictOptimizedIterableOps#flatMap
[E]   StrictOptimizedIterableOps.scala:105    scala.collection.StrictOptimizedIterableOps#flatMap$
[E]   Vector.scala:113                        scala.collection.immutable.Vector#flatMap
[E]   DynamicModelSpec.scala:126              smithy4s.dynamic.model.DynamicModelSpec$Interpreter$GetFieldNames$#$anonfun$genericStruct$1
[E]   DynamicModelSpec.scala:131              smithy4s.dynamic.model.DynamicModelSpec$Interpreter$GetFieldNames$#$anonfun$suspend$1
[E]   DynamicModelSpec.scala:163              smithy4s.dynamic.model.DynamicModelSpec$Interpreter$#$anonfun$toFieldNames$1
[E]   List.scala:293                          scala.collection.immutable.List#flatMap
[E]   DynamicModelSpec.scala:162              smithy4s.dynamic.model.DynamicModelSpec$Interpreter$#toFieldNames
[E]   DynamicModelSpec.scala:174              smithy4s.dynamic.model.DynamicModelSpec$#$anonfun$new$8
[E]   suites.scala:82                         delay @ weaver.MutableFSuite#$anonfun$pureTest$1
// only weaver stuff below

[kubukoz]

Using expected as a base as it's already been verified as the correct output of model dump in the test above.

[Baccata]

The test fails because the smithy prelude is not dumped in the json representation. We would prepopulate the visitor by running it on a handcrafted model containing just the primitives

[kubukoz]

makes sense. I'll let you know if I get to working on it first :)

[kubukoz]

Tried that, turns out the floatShape visitor was not doing anything in particular - still the case for other cases. E.g. when I tried to compile PizzaSpec's model, it's doesn't recognize Food (a union). I can look at this again after the weekend.

[Baccata]

Tried that, turns out the floatShape visitor was not doing anything in particular - still the case for other cases. E.g. when I tried to compile PizzaSpec's model, it's doesn't recognize Food (a union). I can look at this again after the weekend.

Yeah, I had left a fair amount there as TODOs

[Baccata]

Most of it should be implemented in eb05278

[kubukoz]

also, can we include this in the root project? It makes local publishing much easier.

[Baccata]

also, can we include this in the root project? It makes local publishing much easier.

@kubukoz sure yeah, go for it :)

[kubukoz]

Added new spec, trying to dump the model of PizzaSpec to find more issues - currently it's failing because it doesn't recognize the @simpleRestJson trait, so we probably need to change how the model assembler is used in that spec. Then there's the union issue: #17 (comment) - some other cases are also probably not being visited

[Baccata]

@kubukoz, I've created this to facilitate testing. Given a service given by the dynamic model, we can create a "proxy" function, that round trips a Document => IO[Document] kleisli through the fromJsonIO and toJsonIO functions.

Then, if we take that proxy and create a corresponding statically generated stub from it, in theory the data should go through the proxy back and forth, in an untampered fashion.

We can add some middleware to intercept and store the incoming and outgoing json in a CE-Ref, and either perform assertions on the json itself, or use it for logging purposes when we see that the data has not gone through the dynamic proxy as expected.

[kubukoz]

This seems useful for working with schemas, maybe it could be exposed to the users in some module closer to core?

[Baccata]

I thought about, but decided against it because of a lack for a better motivation, at this time. When the time arise, we'll move it

[kubukoz]

Sooo this is currently working in all capacity that I'd need it to, is there anything major that needs to happen before we merge?

[Baccata]

@kubukoz sorry I had missed that. I need to do another pass, clean some things up (make most of them private) and move stuff to an internalspackage. I'll try to do it next week.

[Baccata]

@kubukoz I've cleaned things up, renamed some things, etc. Time for some bikeshedding :)