Enhancement: Allow frontmatter.json to be split in multiple files

[landure]

When describing complex content types, frontmatter.json quickly become a long and unreadable file.
I just finished creating a content type describing Hugo various front-matter fields, and my frontmatter.json has 211 lines of content, and this is just with one complex content type and some frontmatter settings.

Hugo allows to split is configuration file in multiple files stored in config folder, and named according to the config section they store.
An example of Hugo splitted config can be found at https://github.com/razonyang/hugo-theme-bootstrap-skeleton/tree/main/config/_default

Frontmatter could allow a similar setup in .frontmatter/config to allow an atomic configuration.

Another option is to allow the use of an import directive in frontmatter.json where the user explicitly import another file in lieu of a parameter value.

[estruyf]

This would be an interesting approach. It also needs to be carefully thought through with the #407 feature.

[michaeltlombardi]

I like this idea, this seems very similar to extending a local configuration proposal in #407 - maybe that would solve both?

I do think the DevX for being able to automatically import and build the configs in .frontmatter/config would be very useful even without a way to publish/consume configs not kept (and checked in with) the current project.

[estruyf]

@michaeltlombardi there is room for both, as I don't think this will work for your scenario.

@landure let us start creating a list of configuration/settings we want to split up in separate files. That will help to get the development for this started.

[landure]

@estruyf From the top of my head, I see at least three configuration settings that would gain to be stored in separate files:

• frontMatter.taxonomy.contentTypes array contents, that could be stored for example in .frontmatter/config/taxonomy.contentTypes.json or .frontmatter/config/taxonomy.contentTypes/*.json
• frontMatter.content.snippets array contents, that could be stored for example in .frontmatter/config/content.snippets.json or .frontmatter/config/content.snippets/*.json
• frontMatter.content.placeholders array contents, that could be stored for example in .frontmatter/config/content.placeholders.json or .frontmatter/config/content.placeholders/*.json

The idea being to be able to easily import reproducible configuration settings from elsewhere, and to make the configuration more readable and maintainable.

[michaeltlombardi]

My only suggestion to the proposal outlined is that the convention always replace . with folder level, so instead of (not reordered for alphabetic sorting, sorry):

.
├── .frontmatter/
│   └── config/
│       ├── taxonomy.contentTypes/
│       │   └── *.json
│       ├── taxonomy.contentTypes.json
│       ├── content.placeholders/
│       │   └── *.json
│       ├── content.placeholders.json
│       ├── content.snippets/
│       │   └── *.json
│       └── content.snippets.json
└── .frontmatter.json

It would be:

.
├── .frontmatter/
│   └── config/
│       └── taxonomy/
│       │   ├── contentTypes/
│       │   │   └── *.json
│       │   ├── contentTypes.json
│       └── content/
│           ├── placeholders/
│           │   └── *.json
│           ├── placeholders.json
│           ├── snippets/
│           │   └── *.json
│           └── snippets.json
└── .frontmatter.json

I think those three sections - taxonomy.contentTypes, content.snippets, and content.placeholders make enormous sense for a first pass. I think in the future it would absolutely be useful to be able to do data.*, taxonomy.fieldGroups, and taxonomy.customTaxonomy.

For me, the ideal end state is to be able to have most or all fields definable via this split definition, with non-arrays defined in their nearest ancestor .json. For example:

// .frontmatter/config/taxonomy.json
{
  "frontMatter.taxonomy.seoContentLength": 1760,
  "frontMatter.taxonomy.seoDescriptionField": "description",
  // etc for the non-arrays
}
// .frontmatter/config/taxonomy/contentTypes/foo.json
{
  "name": "foo",
  "pageBundle": false,
  "fields": [
    {
      "title": "Title",
      "name": "title",
      "type": "string",
      "single": true
    },
    // snipped for brevity
  ]
}
// .frontmatter/config/taxonomy/contentTypes/bar.json
{
  "name": "bar",
  "pageBundle": false,
  "fields": [
    {
      "title": "Title",
      "name": "title",
      "type": "string",
      "single": true
    },
    // snipped for brevity
  ]
}

With the deterministic file/folder name, we could know what schema to apply where.

On possible future DevX

In the example above, I used the full key names in the split config. To be very clear, I believe that's how this should be implemented initially because I expect it's the least amount of work. However, I think a possible enhanced UX would be this instead:

// .frontmatter/config/taxonomy.json
{
  "seoContentLength": 1760,
  "seoDescriptionField": "description",
  // etc for the non-arrays
}

But that would require some schema rework (or copying the schema partially and maintaining it separately, which seems rough - def more thought needs to go into the maintainability of something like this). I just wanted to note it here because if I didn't do it now, I would entirely forget.

I really, really like this proposal. I also think sorting this out is something of a prerequisite for #407, especially since this model would make maintaining "packaged" configurations easier and will already require some work on merging. This is also much more relevant to the majority of the user base.

My own config file is 767 lines long without the hundreds of lines I would need for my data type definitions.

[estruyf]

I started to work on how @michaeltlombardi defined it.

In the Front Matter: Diagnostic logging you'll be able to see the merged config, this will be useful to test things out, or see what the actual config is that gets applied.

[estruyf]

The example config folders and files:

The outcome:

{
  "$schema": "https://beta.frontmatter.codes/frontmatter.schema.json",
  "frontMatter.framework.id": "docusaurus",
  "frontMatter.content.publicFolder": "static",
  "frontMatter.taxonomy.tags": [
    "docusaurus",
    "facebook",
    "hello",
    "hola"
  ],
  "frontMatter.taxonomy.categories": [],
  "frontMatter.content.pageFolders": [
    {
      "title": "blog",
      "path": "[[workspace]]/blog"
    },
    {
      "title": "docs",
      "path": "[[workspace]]/docs"
    }
  ],
  "frontMatter.content.placeholders": [
    {
      "id": "ogImage",
      "script": "./scripts/og-image.js",
      "command": "~/.nvm/versions/node/v16.11.1/bin/node"
    }
  ],
  "frontMatter.content.snippets": {
    "blockquote": {
      "body": "{{< blockquote type=\"[[type]]\" text=\"[[&selection]]\" >}}",
      "description": "Creates a blockquote",
      "fields": [
        {
          "name": "type",
          "title": "Type",
          "type": "choice",
          "choices": [
            "info",
            "important"
          ],
          "default": "info"
        },
        {
          "name": "selection",
          "title": "Selection",
          "type": "string",
          "default": "FM_SELECTED_TEXT"
        }
      ]
    },
    "highlight": {
      "description": "Creates a code highlighting box",
      "body": [
        "{{< highlight \"[[type]]\" \"linenos=table,noclasses=false\" >}}",
        "  [[selection]]",
        "{{< / highlight >}}"
      ],
      "fields": [
        {
          "name": "type",
          "title": "Language",
          "type": "choice",
          "choices": [
            "html",
            "css",
            "typescript"
          ],
          "default": "typescript"
        },
        {
          "name": "selection",
          "title": "Selection",
          "type": "string",
          "default": "FM_SELECTED_TEXT"
        }
      ]
    },
    "image-snippet": {
      "body": "{{< caption-new \"[[&mediaUrl]]\" \"[[caption]]\" \"[[customCaption]]\" >}}",
      "isMediaSnippet": true,
      "description": "",
      "fields": [
        {
          "name": "customCaption",
          "title": "Custom caption",
          "type": "string",
          "default": "FM_SELECTED_TEXT"
        }
      ]
    },
    "video-snippet": {
      "body": [
        "{{< video \"[[&mediaUrl]]\" \"[[caption]]\" >}}"
      ],
      "isMediaSnippet": true
    }
  },
  "frontMatter.taxonomy.contentTypes": [
    {
      "name": "blog",
      "pageBundle": false,
      "previewPath": null,
      "fields": [
        {
          "title": "Title",
          "name": "title",
          "type": "string"
        },
        {
          "title": "Description",
          "name": "description",
          "type": "string"
        }
      ]
    },
    {
      "name": "default",
      "pageBundle": false,
      "previewPath": null,
      "fields": [
        {
          "title": "Title",
          "name": "title",
          "type": "string"
        },
        {
          "title": "Description",
          "name": "description",
          "type": "string"
        },
        {
          "title": "Publishing date",
          "name": "date",
          "type": "datetime",
          "default": "{{now}}",
          "isPublishDate": true
        },
        {
          "title": "Content preview",
          "name": "preview",
          "type": "image"
        },
        {
          "title": "Is in draft",
          "name": "draft",
          "type": "draft"
        },
        {
          "title": "Tags",
          "name": "tags",
          "type": "tags"
        },
        {
          "title": "Categories",
          "name": "categories",
          "type": "categories"
        }
      ]
    },
    {
      "name": "post",
      "pageBundle": false,
      "previewPath": null,
      "fields": [
        {
          "title": "Title",
          "name": "title",
          "type": "string"
        },
        {
          "title": "Description",
          "name": "description",
          "type": "string"
        },
        {
          "title": "Tags",
          "name": "tags",
          "type": "tags"
        }
      ]
    }
  ]
}

[estruyf]

For the snippets, it would be useful to have a new title property, as the filename is currently used as the key. An optional title property will allow overriding of the title shown on the dashboard.

This is how it gets rendered at the moment:

[estruyf]

Pushed the current change for the new beta. The feature is not yet complete, but already available to give it a try to see if we're on the right path.

Current settings which can be split into folders/files are:

• Content-types: .frontmatter/config/taxonomy/contentTypes
• Page folders: .frontmatter/config/taxonomy/pageFolders
• Placeholders: .frontmatter/config/taxonomy/placeholders
• Snippets: .frontmatter/config/taxonomy/snippets

I've also created a sample project: https://github.com/FrontMatter/project-samples/tree/main/docusaurus

Let me know what you think.

[estruyf]

Testing out if we can split the JSON schema into multiple files. That will make it easier to reuse these in the config files. For instance, this is what the content-type schema looks like: https://beta.frontmatter.codes/config/taxonomy.contenttype.schema.json

{
  "$schema": "https://beta.frontmatter.codes/config/taxonomy.contenttype.schema.json",
  "name": "blog",
  "pageBundle": false,
  "previewPath": null,
  "fields": [
    {
      "title": "Title",
      "name": "title",
      "type": "string"
    },
    {
      "title": "Description",
      "name": "description",
      "type": "string"
    }
  ]
}

That should make managing these individual config files easier, as you will have IntelliSense on them.

[estruyf]

The sample was updated to support custom scripts and data files, folders, and types.

The following settings are now supported to be split in multiple files:

Setting name	JSON Schema	Folder path
frontMatter.content.pageFolders	https://beta.frontmatter.codes/config/content.pagefolders.schema.json	./frontmatter/config/content/pagefolders/
frontMatter.content.placeholders	https://beta.frontmatter.codes/config/content.placeholders.schema.json	./frontmatter/config/content/placeholders/
frontMatter.content.snippets	https://beta.frontmatter.codes/config/content.snippets.schema.json	./frontmatter/config/content/snippets/
frontMatter.custom.scripts	https://beta.frontmatter.codes/config/custom.scripts.schema.json	./frontmatter/config/custom/scripts/
frontMatter.data.files	https://beta.frontmatter.codes/config/data.files.schema.json	./frontmatter/config/data/files/
frontMatter.data.folders	https://beta.frontmatter.codes/config/data.folders.schema.json	./frontmatter/config/data/folders/
frontMatter.data.types	https://beta.frontmatter.codes/config/data.types.schema.json	./frontmatter/config/data/types/
frontMatter.taxonomy.contentTypes	https://beta.frontmatter.codes/config/taxonomy.contenttypes.schema.json	./frontmatter/config/taxonomy/contenttypes/

The sample (https://github.com/FrontMatter/project-samples/tree/main/docusaurus) got updated to test and show how this functionality works.

[estruyf]

Documentation has been updated as well with more information: https://beta.frontmatter.codes/docs/settings#splitting-your-settings-in-multiple-files

[michaeltlombardi]

Just decomposed more than 700 lines of configuration - this is much more maintainable for me and if/when #407 becomes more feasible, it will make for vastly improved maintainability for "packaged" configurations.

[michaeltlombardi]

Just ran across something possibly worth addressing - as I started to do some work today on my theme/helpers to make it easier to configure sites for my users, I noticed that my data types definitions in particular were getting a little out of hand and thought it would be good to break them into subfolders inside of `.frontmatter/config/data/types.

However, this doesn't seem to work - anything in a subfolder does get processed, but the file doesn't show up in the data view.

I suspect it's to do with how the file names/settings are being parsed:

vscode-front-matter/src/helpers/SettingsHelper.ts

Lines 470 to 476 in b9a0c65

	// Get the path without the filename
	const configFolder = parseWinPath(dirname(configFilePath));
	let relSettingName = configFolder.split('/').join('.');
	if (relSettingName.startsWith('.')) {
	relSettingName = relSettingName.substring(1);
	}
	relSettingName = relSettingName.toLowerCase();

vscode-front-matter/src/helpers/SettingsHelper.ts

Lines 499 to 509 in b9a0c65

	else if (relSettingName === SETTING_DATA_FILES.toLowerCase()) {
	Settings.updateGlobalConfigArraySetting(SETTING_DATA_FILES, "id", configJson);
	}
	// Data folders
	else if (relSettingName === SETTING_DATA_FOLDERS.toLowerCase()) {
	Settings.updateGlobalConfigArraySetting(SETTING_DATA_FOLDERS, "id", configJson);
	}
	// Data types
	else if (relSettingName === SETTING_DATA_TYPES.toLowerCase()) {
	Settings.updateGlobalConfigArraySetting(SETTING_DATA_TYPES, "id", configJson);
	}

It looks like the relSettingName for .frontmatter/config/data/types/bar/baz.json becomes data.types.bar while .frontmatter/config/data/types/foo.json becomes data.types and therefore works.

Would it be possible to see instead if we could use something like

if (relSettingName.includes(SETTING_DATA_TYPES.toLowerCase())) {
  ...
}

So that if the path includes the correct key, we can assume it belongs to that key?

Alternatively, if the file includes a $schema key, maybe it could use that key to determine where to insert the settings?

For example, this definition (in .frontmatter/config/data/types/bar/baz.json)

{
  "$schema": "https://beta.frontmatter.codes/config/data.types.schema.json",
  "id": "hugo.params.baz",
  "schema": {
    "title": "Baz Site Parameters for hugo-toroidal",
    "type": "object",
    "properties": {
      "First": {
        "title": "First Property",
        "description": "First Baz",
        "type": "string",
        "default": ""
      }
    }
  }
}

could be reasonably inferred to map to data.types given the value of $schema, I think - though I don't know the performance cost here vs using the folder structure.

[estruyf]

@michaeltlombardi indeed, sub-folders are, as you mentioned, not working, as I currently linked it to that particular folder structure.

The logic can be changed to support sub-folders. The performance should still be fine.

Not sure if the schema would be the best, as this is not required, and not sure if everyone would use it.

[estruyf]

• Make sure the frontmatter.json file wins over the folder - as this makes local changes easier, instead of searching in the folders
• Allow sub-folder usage

[michaeltlombardi]

Not sure if the schema would be the best, as this is not required, and not sure if everyone would use it.

I realize I didn't explicitly say this, but I was thinking of that check as a fallback. With subfolder support though I don't think it's necessary.

[michaeltlombardi]

I also just realized, I'll make a PR to the docs for clarifying that you can check the composed settings with the Diagnostic logging command.

[estruyf]

@michaeltlombardi sub-folder should now be supported, feel free to give it a try

[michaeltlombardi]

Works a treat, extremely happy with being able to use subfolders to organize the data bits! I'm now looking over my snippets and having a think about those, because that list is also growing long, but I know that the naming for the snippets maps directly to their file name.

I think any changes here might have to be addressed in a different way, if at all. 🤔 I don't exactly want "folders" of snippets in the UI I think, but there's also no way to contend with having a bunch of snippets in the existing UI (as addressed in #440).

[estruyf]

Would it help to have a title field in the snippet? That way, if not set or used, it will use the file as key, otherwise it will override it will the title from within the snippet.

[michaeltlombardi]

That would help enormously and mean I don't have to have spaces in file names 😅

It would also bring snippets more in line with the UX surface of other configurable items.

[estruyf]

I will also need to change how to edit the snippets. For now, when they are external, they get a new action to open the file instead of editing these.

[estruyf]

🚀 you can now add titles to your snippets, which makes most sense when adding snippets by individual files, not when you add these within your settings directly.

[michaeltlombardi]

Rad!!