Could Swift macros enhance the GRDB experience?

[groue]

Hello GRDB users,

Do you think the library could benefit from Swift macros? Could they solve a pain point, or help shipping a new feature?

Macros are a big deal. Please share your ideas!

[freak4pc]

Are they a big deal really? In ObjC they were always so painful to use, read & maintain. Seems to me like most things expressed by macros are much better expressed with functions, but I'll be happy to be made wrong 🤷‍♂️

[groue]

I'm not sure either :-) But the first pitched macro-based feature, Predicates (direct link to the pitch) is impressive.

The GRDB way of writing predicates has always been Column-based, with variations that are all more or less verbose:

Player.filter(Column("score") > 1000)
Player.filter(Player.Columns.score > 1000)

This works, but this is not quite pretty.

I've always been reluctant at providing a keypath-based solution, because 1. there is no "official" way to turn a keypath into a string (hack required), and 2. keypaths can only be used on public/interface properties, and this is in direct conflict with my attempts at letting record types hide database details:

// Pretty, but requires a hack, and
// does not scale with records that
// hide database details.
Player.filter(\.score > 1000)

As your app grows more complex, and leaves the simple world of toy examples, it is important that it can evolve as gracefully as possible. I don't like techniques that only work at simple scales and require a hard and long refactoring when a new requirement enters the scene.

The Predicate pitch still has the problem that it forces database details to be exposed to public/internal consumption. But maybe we could derive something from CodingKeys (which are the database interface, unlike the public/internal properties), that could feed those predicates.

[gmedori]

For what it's worth @freak4pc Swift macros are a far cry from the C macros available in Objective-C. Namely they:

1. Are far more difficult to write since they require some understanding of the SwiftSyntax library
2. Cannot live alongside application code; they must be written in a special compiler plugin
3. Must fully type check and produce compile-able code
4. Can emit custom error and warning messages

These all lead to Swift macros being safer and less pervasive, at the cost of speed of development and flexibility. Lack of flexibility is not to say they aren't powerful; they are very powerful. They just aren't arbitrary in the same way since they don't do a blind string replace.

[aaronpearce]

Here is my first pass at a quick macro to parse stored properties of a model into a Column.

@Columns
struct Player: Identifiable, Equatable {
    var id: Int64?
    var name: String
    var score: Int
 }

Which then generates:

struct Player: Identifiable, Equatable {
   var id: Int64?
   var name: String
   var score: Int
   
   enum Columns {
       static let id = Column(CodingKeys.id)
       static let name = Column(CodingKeys.name)
       static let score = Column(CodingKeys.score)
   }
}

https://gist.github.com/aaronpearce/8421f743c28c5349c7d5abfbb7fa1b11

After doing this, I think there could be a nicer solution of annotating each column manually so that we can still have transient properties if need be. Something more like the below, which could also be applied at the top model as well if we wished.

struct Player: Identifiable, Equatable {
   @Column var id: Int64?
   @Column var name: String
   @Column var score: Int

   @Transient var temporaryProperty: String?
}

Another option around that would be make it on the whole model, then let a user opt-out instead for transient properties with a @Transient macro which the parent @Columns macro could read and then skip over.

[groue]

Oh, slight difficulty 😬: CodingKeys is not visible by the macro. We don't know its inner cases. This means that our macro will need to build the list of keys from properties, not from the case of CodingKeys. But users have the right to customize CodingKeys 😖

// User input
@DatabaseRecord
struct Player {
    var id: Int64
    var name: String
    var transientFlag = false
}

extension Player {
    enum CodingKeys: String, CodingKey {
        // No key for transientFlag
        case id, name
    }
}

// Generated
struct Player: Decodable, FetchableRecord, ... {
    var id: Int64
    var name: String
    var transientFlag = false

    enum Columns {
        static let id = Column(CodingKeys.id)
        static let name = Column(CodingKeys.name)
        // ❌ compiler error
        static let transientFlag = Column(CodingKeys.transientFlag)
    }
}

We're not yet out of the woods.

This can be solved if we require that the customized CodingKeys enum is declared inside the record definition.

// User input
@DatabaseRecord
struct Player {
    var id: Int64
    var name: String
    var transientFlag = false
    
    enum CodingKeys: String, CodingKey {
        // No key for transientFlag
        case id, name
    }
}

// Generated
// ✅ Cases of CodingKeys are seen by the macro
struct Player: Decodable, FetchableRecord, ... {
    var id: Int64
    var name: String
    var transientFlag = false

    enum Columns {
        static let id = Column(CodingKeys.id)
        static let name = Column(CodingKeys.name)
    }
}

Poor users who don't make it right on a single stroke. They'll see weird errors 😕

[aaronpearce]

Oh, that's a case I didn't consider.

[gmedori]

@groue couldn't you scan the syntax tree for the type the macro is on, look for the CodingKeys implementation and emit a custom diagnostic saying "hey, for this to work correctly, put your coding keys in here instead of an extension"?

[groue]

You're right. This is possible. But is it desirable? Maybe we can find other designs that avoid this kind of error message.

I'm not thrilled by macros that are unable to interact well with the large variety of normal ways of coding. Errors that read "yeah, no, don't do it this way, do it MY way instead" would quite upset me if I had to meet them, especially if the "way of the macro" goes against my superior needs, for no good reason. Imagine yourself, discovering the library, unaware of this conversation, and reading "hey, put your coding keys in here instead of an extension": what would you feel? Maybe you'd feel a lack of respect for your work, the coding conventions of your team, and maybe even your technical requirements.

Sure, we may have no other option at some point. And the documentation will be very clear about the kind of Swift dialect that users will have to use. But it's very important that we avoid forcing a dialect, a subset of Swift, if we can.

[gmedori]

That's super valid. Thanks for pointing it out, and you're right, it'd make me feel crummy (as SwiftData is currently doing as I'm experimenting with it). What if there are two possible macros someone could use? One that attaches to the CodingKeys definition and generates the columns as a peer (sibling) to the CodingKeys, and another that attaches to the type and generates it based on the properties. Then all that needs doing is document which one is used where. If you have custom coding keys, use this one, if not, use the other one. I know it complicates things; I'm mostly trying to brainstorm solutions to the problem.

[freak4pc]

Now that we know what Macros really mean in the Swift context, this feels like a great possible move especially for GRDB which can try and mirror some of the niceties of SwiftData API without being crappy in the inside (uhm Core Data uhm)

[groue]

Exactly 💪 😉

[groue]

I'm compiling below a few user feedbacks:

• https://twitter.com/rhysmorgan/status/1666047540968472576 and https://twitter.com/freak4pc/status/1666346436521349120 (please keep the ability to add retroactive conformance to GRDB protocols)
• https://mastodon.social/@aaron_pearce/110499620318160040 (have macros generate columns)
• https://mastodon.social/@helge/110499702696464299 (have macros generate SQL)

If you want to reply to one of them, please don't reply in this thread, but create a new one in this discussion, so that we can keep focused on each topic 🙏

[groue]

Columns are discussed in this thread.

[groue]

GRDB has never required Codable, but only Codable could help declare record types in a convenient way, due to the compiler-generated Codable boilerplate. Macros could remove the need for Codable. And this would greatly enhance performance as well.

This is currently not possible, because macros that generate initializers prevent the compiler generation of the memberwise initializer (https://forums.swift.org/t/allow-attached-macros-to-not-suppress-the-memberwise-initializer/65392). But, assuming this issue with macros is solved eventually:

// What user writes
@DatabaseRecord
struct Player {
    var id: String
    var name: String
    var score: Int
}

// What compiler sees
struct Player: FetchableRecord, PersistableRecord {
    var id: String
    var name: String
    var score: Int
    
    init(row: Row) {
        self.id = row["id"]
        self.name = row["name"]
        self.score = row["score"]
    }
    
    func encode(to container: inout PersistenceContainer) throws {
        container["id"] = id
        container["name"] = name
        container["score"] = score
    }
}

We could even generate optimized records out of the box:

// What compiler could also see
struct Player: FetchableRecord, PersistableRecord {
    var id: String
    var name: String
    var score: Int
    
    static var databaseTableName: String {
        // Improvement: this one is now computed at compile time
        "player"
    }
    
    static var databaseSelection: [any SQLSelectable] {
        // Improvement: if struct has less properties than there are table columns,
        // unused columns are not selected.
        //
        // Before macros: SELECT * FROM player
        // After macros: SELECT id, name, score FROM player
        [Column("id"), Column("name"), Column("score")]
    }
    
    init(row: Row) {
        // Improvement: we can access columns by index (fast)
        // instead of names (slow), because we have generated
        // the selection.
        self.id = row[0]
        self.name = row[1]
        self.score = row[2]
    }
    
    func encode(to container: inout PersistenceContainer) throws {
        container["id"] = id
        container["name"] = name
        container["score"] = score
    }
}

We'd probably need to link this generated code to the generated columns

[pocketpixels]

In general I really like the idea of using macros to make GRDB easier to use.
I much prefer the way GRDB works compared to Core Data (and also now SwiftData). But there is a fair amount of boilerplate and protocols to remember and conform to, which makes the learning curve fairly steep. It would be great if macros could make it as easy to adopt as SwiftData.

[groue]

Fully agreed. Macros are a tremendous opportunity. There are a lot of details to iron out, though. Concrete ideas and support are welcome.

[groue]

This thread exemplifies how it is not reasonable to attempt at dealing with the memberwise init from macros. They just CAN NOT deal with all use cases users need.

Shipping known and unfixable bugs has never been the GRDB method: either issues have a fix, or they are not issues. Otherwise I could not handle the support requests. And I don't talk about the moral issue of shipping known-to-fail code. That's why the lib has never aimed at dealing with the simple cases, and has always aimed at dealing with the worst 😉

So some very strong argument has to be made until this lib ships known-crappy code and the documentation that prevents it from hurting too much (i.e. a hand-made memberwise init).

[AnthonyEgerton]

Just to confirm I understand the problem.

• We want to be able to add an init(row: Row) throws method to types with a macro.
• But we don't want to stop the compiler generating a memberwise init for structs.
• With the current macros we can't add an init in an extension.

Assuming this is the case I think I have a workaround.

If we define a protocol with a static factory method like this:

public protocol DBRecord: FetchableRecord, PersistableRecord {
    static func make(row: Row) -> Self
}

Then we add an extension like this:

extension DBRecord {
    init(row: Row) throws {
        self = .make(row: row)
    }
}

We can write a macro that will take this code:

@DBRecord
struct Player {
    var id: Int64?
    var name: String
    var score: Int
}

and expand it to:

@DBRecord
struct Player {
    var id: Int64?
    var name: String
    var score: Int
    static let databaseTableName = "player"

    enum Columns: String, ColumnExpression {
        case id, name, score
    }

    static var databaseSelection: [any SQLSelectable] {
        [
            Column("id"),
            Column("name"),
            Column("score")
        ]
    }

    static func make(row: Row) -> Self {
        .init(
            id: row[0],
            name: row[1],
            score: row[2]
        )
    }

    func encode(to container: inout PersistenceContainer) throws {
        container["id"] = id
        container["name"] = name
        container["score"] = score
    }
}
extension Player : DBRecord  {}

Which allows us to write this:

let record1: FetchableRecord = Player(id: 1, name: "Sam", score: 24)
let row = Row(["id": 1, "name": "Sam", "score": 24])
let record2: FetchableRecord = try Player(row: row)

I've written a macro which does this using Xcode 15 Beta 2 and it seems to work.

What do you think?

[groue]

Hey, that'a a pretty nice idea! Thank you @AnthonyEgerton :-)

[groue]

Macros could improve associations.

Simple associations can be defined as:

extension Author {
    static let books = hasMany(Book.self)
}

But there is some boilerplate for users who want to access books from an Author instance:

extension Author {
    static let books = hasMany(Book.self)
    var books: QueryInterfaceRequest<Book> {
        request(for: Self.books)
    }
}

And users who refine associations have to take care of using association keys. In the example below, the custom keys are necessary, or requests that involve both novels and essays would not give the correct results:

extension Author {
    static let books = hasMany(Book.self)
    static let novels = books.filter(kind: .novel).forKey("novels")
    static let essays = books.filter(kind: .essay).forKey("essays")
}

And still some boilerplate:

extension Author {
    static let books = hasMany(Book.self)
    static let novels = books.filter(kind: .novel).forKey("novels")
    static let essays = books.filter(kind: .essay).forKey("essays")

    var books: QueryInterfaceRequest<Book> {
        request(for: Self.books)
    }

    var novels: QueryInterfaceRequest<Book> {
        request(for: Self.novels)
    }

    var essays: QueryInterfaceRequest<Book> {
        request(for: Self.essays)
    }
}

Custom association keys are also useful for associations that do not have the same name as their target table. In the example below, the custom key is required if we want to refer to an array of ShelfItem with the name items:

struct ShelfItem { ... }
struct Shelf {
    static let items = hasMany(ShelfItem.self).forKey("items")
}

struct ShelfWithItems: Decodable, FetchableRecord {
    var shelf: Shelf
    var items: [ShelfItem] // named after the association key
}
let shelvesWithItems = try dbQueue.read { db in
    try Shelf
        .including(all: Shelf.items)
        .asRequest(of: ShelfWithItems.self)
        .fetchAll(db)
}

---

The pattern is usually the same:

• Each association generally should use, as its association key, the name of the static property that stores it:

  extension Author {
      static let novels = books.filter(kind: .novel).forKey("novels")
      static let essays = books.filter(kind: .essay).forKey("essays")
      //         ~~~~~~                                      ~~~~~~
  }
  
  struct Shelf {
      static let items = hasMany(ShelfItem.self).forKey("items")
      //         ~~~~~                                   ~~~~~
  }

  Macros should be able to help there.

• Static properties have a company instance property:

  extension Author {
      static let books = hasMany(Book.self)
      //         ~~~~~           ~~~~
      var books: QueryInterfaceRequest<Book> {
      //  ~~~~~                        ~~~~
          request(for: Self.books)
      //                    ~~~~~
      }
  }

  Macros should be able to help there as well.

[pocketpixels]

Yes please, that would be amazing.

[hiimtmac]

Ive been playing with a couple macros that sorta remind me of Vapor's Fluent models & property wrappers.

The pattern I have tended to skew towards with GRDB for a long time is:

• write a memberwise init
• conform to TableRecord with explicit static let databaseTableName
• create a enum Columns: String, ColumnExpression {} in the model with all the applicable properties
• conform to FetchableRecord with convenience init(row: Row) that calls memberwise init using Columns for subscripting
• conform to PersistableRecord with encode(to container: inout PersistenceContainer) again using Columns for subscripting

I know that you can get a lot for free with some of existing codable helpers in the library and reduce some of that boilerplate but ¯\_(ツ)_/¯ .

Most of this boilerplate can be eliminated with macros. Along with some helpers for relationships as well (happy to share source code if anyone is interested):

Usage

@Table(name: "clubs")
public final class GKClub {
    @Column(name: "id") public var id: UUID
    @Column(name: "name") public var name: String
    @Column(name: "abbreviation") public var abbreviation: String
    @Column(name: "bag_id") public var bagId: GKBag.ID?
    @BelongsTo public var bag: QueryInterfaceRequest<GKBag>
    @HasMany public var shots: QueryInterfaceRequest<GKShot>
}

Expansion

@Table(name: "clubs")
public final class GKClub {
    @Column(name: "id") public var id: UUID
    @Column(name: "name") public var name: String
    @Column(name: "abbreviation") public var abbreviation: String
    @Column(name: "bag_id") public var bagId: GKBag.ID?
    @BelongsTo public var bag: QueryInterfaceRequest<GKBag>
    {
        get {
            request(for: Self.bag)
        }
    }
    @HasMany public var shots: QueryInterfaceRequest<GKShot>
    {
        get {
            request(for: Self.shots)
        }
    }
    @ColumnAccessor(name: "id") public static var id
    
    @ColumnAccessor(name: "name") public static var name
    
    @ColumnAccessor(name: "abbreviation") public static var abbreviation
    
    @ColumnAccessor(name: "bag_id") public static var bagId
    
    public init(id: UUID, name: String, abbreviation: String, bagId: GKBag.ID?) {
        self.id = id
        self.name = name
        self.abbreviation = abbreviation
        self.bagId = bagId
    }
    
    public convenience init(row: Row) throws {
        self.init(id: row[GKClub.$id], name: row[GKClub.$name], abbreviation: row[GKClub.$abbreviation], bagId: row[GKClub.$bagId])
    }
    
    public func encode(to container: inout PersistenceContainer) throws {
        container[GKClub.$id] = id
        container[GKClub.$name] = name
        container[GKClub.$abbreviation] = abbreviation
        container[GKClub.$bagId] = bagId
    }
    
    public static let databaseTableName = "clubs"
    
    public static let bag = belongsTo(GKBag.self, key: GKBag.databaseTableName)
    
    public static let shots = hasMany(GKShot.self)
}
extension GKClub : TableRecord  {}

extension GKClub : FetchableRecord  {}

extension GKClub : PersistableRecord  {}

Im using a property wrapper ColumnAccessor to implement my Columns for subscript usage like here

ColumnAccessor implementation

@propertyWrapper
public struct ColumnAccessor {
    public init(name: String) {
        self.wrappedValue = name
    }
    
    public let wrappedValue: String
    public var projectedValue: some ColumnExpression {
        Column(wrappedValue)
    }
}

Current Limitations

• Currently models must be classes and marked final (no technical limitation for this - just that's what I tend to do so that's how I made it for now)
• I can't seem to figure out how to make @HasMany(though: Something.hasMany, using: Another.belongsTo) compile (the macro generation code works in tests, but I think since Something.hasMany and Another.belongsTo are macro generated, they are not available to be consumed by this macro as well?)
• I don't love the fact that you can access the wrappedValue of the ColumnAccessor - perhaps this is overkill and the static member that the macro generates should just be the Column itself, but I sorta like the $ syntax to visually show "hey this is a column in a table" as opposed to some actual value to use/consume (ie GKClub.$name vs GKClub.name)

Pre macro work

Prior to macros I had made a property wrapper with a similar(stolen) implementation to Field property wrapper in Fluent, and then a static subscript on my own Table protocol that could derive a Column from that. But with that implementation I still had to write all the protocol conformances by hand and there was a requirement to have an empty init() {} in all my models (like fluent models).

Example of property wrapper way with static subscript

public final class GKClub: Table, ... {
    @Field(key: "id") public var id: UUID
    @OptionalField(key: "name") public var name: String?

    ...
}

_ = row[GKClub.$name]

[pnewell]

Would you still be willing to share this?

[ErrorErrorError]

Hey y'all!

I am currently working on a GRDBMacros library that allows users to generate Record with macros. It is still a work-in-progress, but so far you are able to use a @DatabaseRecord to build a FetchableRecord & PersistableRecord struct, while also generating columns.

@DatabaseRecord
struct Author {
  var id: Int64?
  var name: String
  var countryCode: String?

  @Transient var somePropertyToExclude = true

  @HasMany var books: QueryInterfaceRequest<Book>
}

Will expand to:

struct Author {
  var id: Int64?
  @Column("title") var name: String
  var countryCode: String?

  @Transient var somePropertyToExclude = true

  var books: QueryInterfaceRequest<Book> {
    get {
      request(for: Self.books)
    }
  }

  static let books = hasMany(Book.self)
 }

extension Author: GRDB.FetchableRecord {
  init(row: GRDB.Row) {
    self.id = row[Columns.id]
    self.name = row[Columns.name]
    self.countryCode = row[Columns.countryCode]
  }
}

extension Author: GRDB.PersistableRecord {
  func encode(to container: inout GRDB.PersistenceContainer) throws {
    container[Columns.id] = id
    container[Columns.name] = name
    container[Columns.countryCode] = countryCode
  }
}

extension Author {
  enum Columns {
    static let id = GRDB.Column("id")
    static let name = GRDB.Column("title")
    static let countryCode = GRDB.Column("countryCode")
  }
}

extension Author {
  static let databaseSelection: [any GRDB.SQLSelectable] = [
    Columns.id,
    Columns.name,
    Columns.countryCode
  ]
}

Most of the suggestions come from #1323 (comment) but I am planning on adding more features, like generating SQL tables! If there's any features/additions you would like for me to add let me know!

As of now the source code will be up on https://github.com/ErrorErrorError/GRDBMacros, but if the dev would want for it to be on GRDB.swift, feel free to let me know :D

[groue]

That's a fantastic exploration, @ErrorErrorError 👍