-
-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add experimental content layer flag #11652
Conversation
|
packages/astro/src/@types/astro.ts
Outdated
* @version 4.16.0 | ||
* @description | ||
* | ||
* The Content Layer API is a new way to handle content and data in Astro. It builds upon [content collections](https://docs.astro.build/en/guides/content-collections/), taking them beyond local files in `src/content` and allowing you to fetch content from anywhere, including remote APIs, or files anywhere in your project. As well as being more powerful, the Content Layer is designed to be more performant, helping sites scale to thousands of pages. Data is cached between builds and updated incrementally. Markdown parsing is also 5-10x faster, with similar scale reductions in memory. While the feature is experimental and subject to breaking changes, we invite you to try it today and let us know how it works for you. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Docs comment re: terminology (with full understanding this can change in the experimental stage easily enough!) Sorry for the length!
I see in the RFC we still use "collection" and "entry". As I'm going through the text, it feels the most natural to me (outsider, thinking of explaining this feature) to refer to "content layer" in e.g. the following ways
-
"use the Content Layer API"
-
"collections powered by the Content Layer API"
-
"migrating your existing content collections to use the Content Layer API" (vs migrating from content collections to content layer" <- this doesn't sound quite right, because implies that you don't still have collections")
-
"defining new (content?) collections (with the Content Layer API)" <- this last part will be unnecessary once this is the only way. And, with all our marketing having gone into "Content collections" and being a "content-first" project, we may even want to keep the phrase content collection to still refer to the noun, irrespective of the new implementation.
"content layer" doesn't feel like it can replace "content collections" one-to-one. You don't have "different layers." You do have "a content layer" but that's not instead of having collections, and that's not the main part of this that the user interacts with: they make collections that have entries. I don't think "layer collections" feels very ergonomic, but that's really the closest analogy. 😅
So again, we don't have to have this all nailed down, but as I'm reviewing this I'll be suggesting what I think feels both clear and helpful to someone using this new API to power their collections!
"The layer" may be the underlying interesting tech, but I really think "collection" is going to still be the main word that resonates with users the most. Once the layer is powering everything, no one is going to be thinking of a "content layer loader" when they write their collection schema, it's a loader they define in their collection, using defineCollection()
. The word "layer" doesn't appear anywhere, and if we can start off closer to that kind of documentation, we'll be ahead of the game!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I think the right summary is that these new content collections are powered by the content layer API
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sarah11918 I've made some updates to the terminology. See if that's better.
!preview contentlayer |
Snapshots have been released for the following packages:
Publish Log
Build Log
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code wise, it looks good
@@ -2184,6 +2185,258 @@ export interface AstroUserConfig { | |||
* For a complete overview, and to give feedback on this experimental API, see the [Server Islands RFC](https://github.com/withastro/roadmap/pull/963). | |||
*/ | |||
serverIslands?: boolean; | |||
|
|||
/** | |||
* @docs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, to make things simpler, I have mocked up something in a branch of docs itself with an idea of what I'm thinking: https://content-layer-draft--astro-docs-2.netlify.app/en/reference/configuration-reference/#experimentalcontentlayer
I did take take a little from the RFC and mix it in here. I mostly tried to laser-focus things a bit because it's already quite long, and for experimental docs, we can be a bit more minimal and let the RFC do some of the work.
Putting on my "seeing this for the first time, want to play with it" hat, I feel like it's the loader
stuff that people need to get is fundamentally different, and most important, in all this. So I spent a bit more time there, and didn't spend as much time on the querying and rendering (even combined them into one section!) because they work so much like content collections do already.
So see what you think about something that kind of takes this form! When we have a good shape, then we can squish it in here to this file!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's great. Is it on a branch somewhere? Also do you have any suggestions for a workflow for working on this? It's really hard editing inside the JSDoc!
@sarah11918 I'm going to merge this as-is so we have the flag in the content-layer branch. I can then do another PR just for the docs updates. |
* Empty commit * Changeset * feat: add Content Layer loader (#11334) * wip * wip * wip * Update demo * Add meta * wip * Add file loader * Add schema validation * Remove log * Changeset * Format * Lockfile * Fix type * Handle loading for data store JSON * Use rollup util to import JSON * Fix types * Format * Add tests * Changes from review * fix: sync content layer in dev (#11365) * wip * wip * wip * Update demo * Add meta * wip * Add file loader * Add schema validation * Remove log * Changeset * Format * Lockfile * Fix type * Handle loading for data store JSON * Use rollup util to import JSON * Fix types * Format * Add tests * Changes from review * Sync content layer in dev * feat: add typegen for loaders (#11358) * fix: watch for content layer changes (#11371) * fix: watch for content layer changes * Add test * feat: adds simple loader (#11386) * wip * Add simple loader * Fix type guard * Tighten loader schema * Add loader function to type * Reinstall vitest * feat: add glob loader (#11398) * feat: add glob loader * Enable watching and fix paths * Store the full entry object, not just data * Add generateId support * Fix test * Rename loaders to sync * Refacctor imports * Use getEntry * Format * Fix import * Remove type from output * Windows path * Add test for absolute path * Update lockfile * Debugging windows * Allow file URL for base dir * Reset time limit * feat: add markdown rendering to content layer (#11440) * feat: add glob loader * Enable watching and fix paths * Store the full entry object, not just data * Add generateId support * Fix test * Rename loaders to sync * Refacctor imports * Use getEntry * Format * Fix import * Remove type from output * Windows path * Add test for absolute path * Update lockfile * Debugging windows * Allow file URL for base dir * Reset time limit * wip: add markdown rendering to content layer * use cached entries * CLean up types * Instrument more of the build * Add digest helper * Add comments * Make image extraction work * feat: image support for content layer (#11469) * wip * wip * Add image to benchmark * Stub assets if missing * Resolve assets in data * Ignore virtual module * Format * rm log * Handle images when using cached data * Fix CCC * Add a comment * Changes from review * Format * Use relative paths for asset files * Pass all md props to getImage * Ensure dotastro dir exists * Fix tests * Changes from review * Don't use temp array in getcollection * Add error handling * Format * Handle paths that are already relative * Dedupe sync runs * Fix syncing in dev * Changes from review * Windows paths ftw * feat(content-layer): support references in content layer (#11494) * Support references in content layer * Fix utf8 rendering * Warn for invalid entries * Fix test * lol windows paths * Remove assertion * chore: fix content layer types (#11527) * Add experimental_content type * Fix import * Make data store methods generic * fix loader types * Lockfile * Clean content layer with `--force` (#11541) * Clearn content layer with `--force` * Add tests * Document --force flag * Fixes to content layer render types (#11558) * Lockfile * feat: use devalue to serialize content layer data (#11562) * feat: use devalue to serialize content layer data * Fix import * Use devalue stringify * Unused import * Propagate error messages correctly * Support --force flag in sync and dev (#11581) * Support --force flag in sync and dev * Fix test * Separate render function and merge content layer types (#11579) * Separate render function and merge content layer types * Changes from review * fix: clear content layer cache if config has changed (#11591) * fix: clear content layer cache if config has changed * Add test * Watch config * Change from review * fix: skip glob files in content dir (#11622) * fix: skip glob files in content dir * Changes from review * Log pattern * Refactor content layer into shared instance (#11625) * Refactor content layer into shared instance * Clean up when testing * Handle cleanup * fix: support filters in content layer getCollection (#11631) * Throw when using deprecated getEntryByX functions with content layer (#11637) * Updates to content layer types and jsdocs (#11643) * Add hot key to reload content layer (#11626) * Add hot key to reload content layer * Fix filename * Remove cli message * Update example * Change key to "s" * feat: handle simple mdx rendering (#11633) * feat: handle simple mdx rendering * cleanup * feedback * fix regression * remove log * flip condition * update tests * log collections to understand the error * let's try this alternative * try parallel test to understand the issue * chore: use a new fixture to fix tests * rebase and docs * fix regressions * remove old code * address feedback * rename param * log error * rebase * chore: try a different cache dir to solve the error test * fix invalidation of the module when there's no store available * address suggestion * run formatter * update lock file * Lint * Add experimental content layer flag (#11652) * Add experimental content layer flag * Syntax and format * Aside * Format * Reset content config between runs * Update fixture * Update terminology * Lint * wut * Normalize render function return value (#11663) * Add markdoc support to content layer (#11664) * Add markdoc support to content layer * Switch test to cheerio * Update benchmarks * update lock file * Update content layer flag docs (#11682) * Update content layer flag docs * Apply suggestions from code review Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * More markdoc * Typo * Apply suggestions from code review Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * Update --------- Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * Add changeset for content layer experimental release (#11644) * Add changeset for content layer experimental release * Update changeset * Update .changeset/smooth-chicken-wash.md Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> --------- Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * feat: injectTypes (#11551) * feat: make inline config 1st arg * fix: run config done in sync * feat: start working on injectTypes * feat: write files * feat: adapt core features * feat: migrate db to injectTypes * feat: special db handling * feat: update settings instead of workarounds * fix: create dotAstroDir * feat: refactor sync tests * fix: path * fix: paths * chore: add comments * feat: overwrite content file if exists * chore: remove unused db env related code * feat: use dotAstroDir for settings * chore: simplify astro env sync * feat: use dotAstroDir for preferences * feat: handle db in integration api * chore: reorganize * feat: format * feat: add test * Discard changes to examples/basics/astro.config.mjs * Discard changes to examples/basics/package.json * Discard changes to pnpm-lock.yaml * chore: remove test files * feat: update examples dts * fix: dts * chore: changesets * fix: indentation * Apply suggestions from code review Co-authored-by: Chris Swithinbank <swithinbank@gmail.com> * Apply suggestions from code review Co-authored-by: Chris Swithinbank <swithinbank@gmail.com> * chore: format * Update packages/astro/src/integrations/hooks.ts * Update .changeset/mean-horses-kiss.md * feat: remove formatting * feat: handle fs errors * feat: remove astro:db special path handling * feat: add fs error * Update packages/astro/src/content/types-generator.ts * Update .changeset/mean-horses-kiss.md * Update errors-data.ts * Update .changeset/mean-horses-kiss.md Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * Update .changeset/mean-horses-kiss.md Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> --------- Co-authored-by: Chris Swithinbank <swithinbank@gmail.com> Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> Co-authored-by: Erika <3019731+Princesseuh@users.noreply.github.com> * Add file generation and flag for content intellisense (#11639) * feat: add type to infer input type of collection * refactor: * feat: generate json schema for content too * feat: generate a manifest of all the collections * refactor: unnecessary type * fix: only add content collections to manifest * chore: changeset * fix: generate file URLs * fix: flag it properly * fix: save in lower case * docs: add jsdoc to experimental option * nit: move function out * fix: match vscode flag name * Update packages/astro/src/@types/astro.ts Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * Update packages/astro/src/@types/astro.ts Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * Update serious-pumas-run.md * test: add tests * Add content layer support * Apply suggestions from code review * fix: test * Update .changeset/serious-pumas-run.md Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * Apply suggestions from code review * Remove check for json --------- Co-authored-by: Matt Kane <m@mk.gg> Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * nit: use same filesystem error as injectTypes * fix: code component was missing support for meta string (#11605) * fix: code component was missing support for meta string Fixed #11604 * Create odd-buttons-pay.md * <Code>: add reference link for meta prop * Apply suggestions from code review Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * Update .changeset/odd-buttons-pay.md * Update .changeset/odd-buttons-pay.md Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> --------- Co-authored-by: Erika <3019731+Princesseuh@users.noreply.github.com> Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> Co-authored-by: Emanuele Stoppa <my.burning@gmail.com> * Deprecates exporting prerender with dynamic values (#11657) * wip * done i think * Add changeset * Use hook instead * Reorder hooks [skip ci] * Update .changeset/eleven-pens-glow.md Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * Fix run * Fix link * Add link Co-authored-by: Sarah Rainsberger <sarah11918@users.noreply.github.com> * More accurate migration [skip ci] --------- Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> Co-authored-by: Sarah Rainsberger <sarah11918@users.noreply.github.com> * Use node parseArgs instead of yargs-parser and arg (#11645) * wip * done * Add changeset * Format * Update * Fix houston * Fix test * Fix test * [ci] format * resolve conflict --------- Co-authored-by: Emanuele Stoppa <my.burning@gmail.com> Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> Co-authored-by: Florian Lefebvre <contact@florian-lefebvre.dev> Co-authored-by: Chris Swithinbank <swithinbank@gmail.com> Co-authored-by: Erika <3019731+Princesseuh@users.noreply.github.com> Co-authored-by: Julien Cayzac <jcayzac@users.noreply.github.com> Co-authored-by: Bjorn Lu <bjornlu.dev@gmail.com> Co-authored-by: Sarah Rainsberger <sarah11918@users.noreply.github.com> Co-authored-by: Bjorn Lu <ematipico@users.noreply.github.com>
Changes
Adds an experimental flag for content layer. This includes the docs for the flag. It removes the
type: "experimental_content"
, and instead automatically sets the type if the loader is defined.The flag is enforced when the content layer tries to sync: if there are any collections that have a loader but the flag is not set then it throws an error. The flag is also used to enable the hotkey for syncing the content layer.
Testing
Tests updated with the flag, and the type removed.
Docs
This has experimental flag docs