Provide Rust serde-like flatten behavior for flattening/sharing/capturing fields (500USD Bounty)

[DamianReeves]

---

From the maintainer Li Haoyi: I'm putting a 500USD bounty on this issue, payable by bank transfer on a merged PR implementing this.

---

Sometimes when processing JSON you want to use a shared structure across various models. The serde crate in Rust wanted to provide support for this feature and provided it through the flatten attribute.
It would be very useful if upickle also supported this.

Lets say we have an API which supports pagination as a general concept and both our users and products endpoints support pagination.

Usecase: Sharing via Flattening

{
  "limit": 100,
  "offset": 200,
  "total": 1053,
  "users": [
    {"id": "49824073-979f-4814-be10-5ea416ee1c2f", "username": "john_doe"},
    ...
  ]
}

{
  "limit": 50,
  "offset": 100,
  "total": 2103,
  "users": [
    {"id": "abf24073-979f-4814-be10-5ea416ee1c2f", "shortname": "blender", "description":"It slices it dices."},
    ...
  ]
}

It would be useful if we could use the following models to capture this.

case class Pagination(limit:Int, offset:Int, total:Int)
case class User(id:String, username:String)
case class Users(users:List[User], @upickle.implicits.flatten pagination:Pagination)

case class ProductInfo(id:String, shortname:String, description:String)
case class Products(products:List[Product], @upickle.implicits.flatten pagination:Pagination)

Usecase: Extra Fields

This can also be used as a mechanism for handling extra fields like the Rust crate also does.

{
   "name":"myLibrary",
   "devDependencies": {
        "vite": "^5.4.2",
        "typescript":"^5.0.0"   
    },
   "nonStandardField1":"blah",
   "customDependencies": {
       "@myEnterprise/enterpriseLib": "1.0.0"
   }
}

case class PackageManifest(
    name:String, 
    devDependencies:Map[String,String], 
    @upickle.implicits.flatten otherStuff:Map[String, ujson.Value]
}

[mrdziuban]

I'm taking a look at this

[pawelsadlo]

I have been working on something similar ,and I think it would be good to have the following questions answered sooner or later:

1. What to do with collisions:

case class Foo(a:Int, @flatten b:Bar)
case class Bar(a:Int)

Do we allow overriding? What are the overriding rules (do we allow overriding with subclasses?)

If not, how we report, do we emit compiletime errors?

2. Are nested flattens allowed?

case class Foo(a:Int, @flatten b:Bar)
case class Bar(@flatten c: Baz,d:String)
case class Baz(e:Int)

3. Is @flatten allowed for generics?

case class Foo[T](@flatten a:T)

It might be impossible to emit compilation errors in case of generics.

[mrdziuban]

Adding another question:

4. Should we allow @flatten on fields referring to members of sealed hierarchies? If so:
   1. How can singletons be flattened? They're just encoded as a String
   2. For products, should the $type (or other configured discriminator key) field also be flattened?

sealed trait Foo
case object Bar extends Foo
case class Baz() extends Foo

case class Test(str: String, @flatten foo: Foo)

write(Bar) // "Bar"
write(Baz()) // {"$type":"Baz"}
write(Test("test", Bar)) // how would this work?
write(Test("test", Baz())) // would this be encoded like this? {"str":"test","$type":"Baz"}

[lihaoyi]

1. Honestly I have no idea. Maybe report an error?
2. I think nested flattens could be allowed, but you'd need it to be @flatten c: Baz in your example
3. If possible, it would be nice. Not sure what technical limitations there may be
4. I think limiting @flatten only to case classes is fine for now, no need to worry about sealed traits.

Overall I think I'd be happy with any answer to these questions, as long as the answer is documented, tested and properly justified. Some limitations are inevitable and we just need to be clear about where we draw the line and why.

[pawelsadlo]

2. I think nested flattens could be allowed, but you'd need it to be @flatten c: Baz in your example

Yeah , I forgot @flatten c:Baz, will edit