[CT-2035] [Spike] Parsing: model ‘version’ attribute

[MichelleArk]

Introduce an optional ‘versions’ attribute on model yaml entry. versions is a list of 'version' structures. Pseudocode:

ModelVersion:
    version: str
    implemented_by: Optional[str] = None # defaults to <model_name>.v<version>
    # all model properties (e.g. config, description, etc) 
    columns: Optional[List[Union[ColumnInfo, IncludeExclude]]]

versions will be specified within a single model yaml entry, but each version should maps to an individual node in the manifest with a unique_id: model.<project_name>.<model_name>.v<version>.

Example:

models:
  - name: dim_customers
    latest_version: 2
    config:
      materialized: table
    columns:
      - name: customer_id
        description: This is the primary key
        data_type: int

    versions: # new!
      - version: 2
        # implemented_by: dim_customers.v2 - optional, this is the default naming convention
      - version: 1
        # implemented_by: dim_customers.v1 - optional, this is the default naming convention
        config:
          alias: dim_customers
        columns:
          - include: * 
            exclude: customer_id
          - name: customer_id
            data_type: float  # this was previously a float in version 1 for some reason

Notes:

• The model name is defined in the yaml file, and is globally unique but shared among versions of that model
• Each version of this model is "implemented by" a model SQL/python file. The implemented_by property points to the name of a .sql (or .py) file that implements that model version. By convention, this file is <model_name>.v<version>.sql, e.g. dim_customers.v2.sql. However, users could set implemented_by to point to a different SQL file that doesn't meet the default naming convention. (These file names must be globally unique, even though they are not the actual model name.)

[jtcohen6]

version should be a valid integer (for now)

#6736 (comment)

We should leave space for adding other versioning schemes in the future (namely SemVer). For now, that can be as simple as creating lightweight abstractions over the logic to:

• validate a version input (this issue)
• sort versions & determine which is "latest" ([CT-2036] Parsing: model ‘latest’ attribute #6867)

[jtcohen6]

@MichelleArk assigning us for refinement :)

[emmyoop]

Are we going to have multiple models with the same name separated only by the version? What would the unique id look like in this case? It's a departure from how we're handling unique id now. Would it be handled similar to how we handle disabled models?

Do all versions of the same model get linked into the DAG?

[jtcohen6]

Conversation during Language team estimation yesterday:

• Scariest thing here is updating a model's unique_id between "parsing" (SQL file name, model.<project>.<implemented_by>) and "patching" (model.<project>.<name>.<vX>). That said, we agree that it's long-term better, and more correct for the interface, if the unique_id includes the canonical model name and the version number, versus being tied to the file name of the SQL implementation.
• We should spike this to start, to understand where the rough edges might be.
• We might want to add an abstraction over looking up nodes by unique_id, rather than just indexing a dictionary it should be a method with an understanding of versions.

[jtcohen6]

Idea from @heysweet: Now that we're requiring users to tell us which model version is latest, should we hoist it up to the top-level? So instead of latest: true nested within versions (and validate that there's exactly one with that value), the user specifies latest: 2 up top.

[MichelleArk]

I like it! Less validation necessary and it's easier on the eyes in terms of sifting through versions to find the latest one. I've updated the description.

[jtcohen6]

I've proposed v: instead of name: as the uniquely identifying field for each version

    versions:
      - v: 2
      - v: 1
        config:
          alias: dim_customers
          ...
      - v: 3

Relevant comment in discussion:

• Model versions #6736 (reply in thread)

[MichelleArk]

closing in favor of implementation issue: #7263