Add next export for static builds

[matthewmueller]

This PR adds support for a next export -o <out> <dir> command and closes #604. This PR is a non-breaking change and intentionally makes very few changes to the existing codebase.

The way it works is it first performs a next build and then takes that output and turns it into a self-executing static bundle. The lifecycle hooks in a static build are a bit different than a dynamic build:

• During the export, next will call each page's getStaticInitialProps() static method if they have one. This will enable things like pulling in and compiling markdown files at build time. Any errors that happen here will halt the build. The reason to go with getStaticInitialProps() as opposed to getInitialProps({ build: true }) is that getStaticInitialProps() won't break any existing next.js plugins (ex. https://github.com/matthewmueller/next-cookies/blob/master/index.js#L20-L28) and can be treated as a progressive enhancement.
• The client will always call getInitialProps() on page load. This is a bit different because in the dynamic build, the server will call getInitialProps(), but since we don't have control over the server in static builds, we'll call it here. This is where you'll load user data.

I made 4 changes to the existing codebase:

• __NEXT_DATA__ will now receive { exported: true } so it can adjust the way the router fetches new pages.
• Document now knows about { exported: true } to adjust the location of the app.js common bundle.
• Using the { exported: true } from __NEXT_DATA__, we also trigger getInitialProps on page load before rehydrating (as mentioned above).
• Add an export command in bin/next

There was a lot of discussion in #604 about custom routing. I decided not to include custom routing in this PR because most static file servers support redirects and by using the as parameter in <Link as={as} /> and Router.push(url, as), you can support dynamic routes like /blog/:slug.

Wasn't sure how to setup the test environment for this project, but happy to add tests once the implementation is settled on.

Note that there's a couple TODOs in server/export.js. Please have a look as I wasn't sure if those pieces are relevant for production or if they're just used in development.

Let me know what you think and if there's anything I can do to improve this PR!

[sedubois]

I think you should remove this, add it in your ~/.gitconfig

[matthewmueller]

i sort of feel like it's the project's job to prevent stuff like this from getting committed, but i don't care much either way 😄. let's see what others think

[impronunciable]

Agreed with Matthew

[albinekb]

Agree it should always be in your global gitignore

[jstcki]

Nice! 👏 You beat me to it 😊

I have a few cursory observations:

• <dir> in next build <dir> refers to the root of the app, so next export <dir> should work the same. Therefore a -o, --out option would be nice for changing <dir>/build to <dir>/whatever
• I think the bundle paths (app.js, pages) should still include the hash/build ID for long-term caching.
• Should the build parameter in getInitialProps() rather be called staticBuild to make it more clear that it doesn't apply during regular next build?
• Support for CDN URLs would be nice (probably through an option in next.config.js)

[matthewmueller]

@herrstucki

in next build refers to the root of the app, so next export should work the same. Therefore a -o, --out option would be nice for changing /build to /whatever

+1, I'll fix that up

I think the bundle paths (app.js, pages) should still include the hash/build ID for long-term caching.

+1, but I wasn't sure how to add this. Did you get this working in your branch? It'd probably look something like this: <script src="/app.js?{hash}"></script>

Should the build parameter in getInitialProps() rather be called staticBuild to make it more clear that it doesn't apply during regular next build?

I ended up moving this trigger into getBuildProps() to avoid conflicts with existing plugins. I'm open to a better name if anyone has one. I'm also thinking that separate lifecycle functions make more sense in the long run anyway (perhaps even a getServerInitialProps()).

Support for CDN URLs would be nice (probably through an option in next.config.js)

+1, though I think we could proceed with this PR without including that for now. I think you could set that the URLs up as aliases in the next.config.js similar to how webpack does it.

[jstcki]

FWIW, I think not adding another static method to pages would be preferable. In my opinion switching to a static export should ideally require no changes to the code.

Also, by adding another static method, static pages will get broken when the user forgets to add that method to the page, because getInitialProps() only gets called in the client when you navigate to a page but not when you load the server-rendered page.

In most cases (e.g. when you just do a fetch() in getInitialProps) you don't even need to branch on server/client environment anyway, no?

[matthewmueller]

FWIW, I think not adding another static method to pages would be preferable. In my opinion switching to a static export should ideally require no changes to the code.

So this was actually put in place so you don't need to make any changes to existing code. I've managed to export a large web app without any code changes. You can think of this lifecycle hook as something extra that you can now attach onto.

Also, by adding another static method, static pages will get broken when the user forgets to add that method to the page, because getInitialProps() only gets called in the client when you navigate to a page but not when you load the server-rendered page.

Actually it gets called on load now and then when you navigate around, so assuming your getInitialProp() functions are isomorphic, it just works™

[jstcki]

I think the bundle paths (app.js, pages) should still include the hash/build ID for long-term caching.

+1, but I wasn't sure how to add this. Did you get this working in your branch? It'd probably look something like this: <script src="/app.js?id=hash"></script>

I managed to do it by moving the built files to the locations where they're expected, i.e. _next/<hash>/app.js and _next/<buildId>/pages. I re-purposed server/replace for this (note that my version is not fully functional).

[jstcki]

FWIW, I think not adding another static method to pages would be preferable. In my opinion switching to a static export should ideally require no changes to the code.

So this was actually put in place so you don't need to make any changes to existing code. I've exported a large web app without any code changes.

Hmm, but don't you lose the "server"-provided initial props in a static build then? Wouldn't it be nice if the server and static render would be the same out of the box? 🤔

So ideally a page with

// pages/foo.js
static async getInitialProps() {
  return fetch('stuff.json').then(r => r.json());
}

Would have the data from stuff.json also directly baked into /foo/index.html, no?

But you're probably right that existing implementations which rely only on req/res being present to detect a server environment would break …

[matthewmueller]

Hmm, but don't you lose the "server"-provided initial props in a static build then? Wouldn't it be nice if the server and static render would be the same out of the box? 🤔

Yah, I definitely agree that would be nice and it's originally what I was proposing, but I don't think it really works that well in practice. The server-side getInitialProps has access to headers, dynamic paths and query parameters that don't really apply in a static build. I also found that there's a bit more cognitive load when you start considering your function running in 3 different contexts.

I'll definitely think about this some more and happy to discuss this some more, but I feel like progressive enhancement may be a better approach where you can get your site working the normal way and then call into this additional hook to bring static data into your views.

I think I'd prefer less potential breakage and more verbosity. Something like this:

// pages/foo.js
static async getInitialProps() {
  return await fetchData()
}

static async getBuildProps() {
  return await fetchData()
}

function fetchData() {
  return fetch('stuff.json').then(r => r.json());
}

[jstcki]

Yeah, you're probably right about this. Just dropping more args into a function doesn't make it easier to use it correctly 😉 (and also impossible to typecheck).

In the long term separating the different environments into separate methods (e.g. getServerInitialProps, getClientInitialProps, getStaticInitialProps) would probably make sense. Or a single context argument with different explicit types akin to a redux action (e.g. {env: "SERVER", ...}, {env: "STATIC", ...}, {env: "CLIENT", ...}). But that's another issue …

[matthewmueller]

Sweet, I also changed getBuildProps to getStaticInitialProps because that makes way more sense :-)

[impronunciable]

Maybe I'm missing something but if route is falsy then we are returning incorrect routes like undefined/index.js

[jstcki]

Probably better rewritten as

return route
  ? route.replace(/\/$/, '') + '/index.json'
  : '/index.json'

(also doesn't reassign the function argument)

[matthewmueller]

yep, good catch!

[impronunciable]

Why not passing a static prop to getInitialProps I can see how most "initial props" will be shared between a static snapshot of the site and the "dynamic" version with maybe a small difference

[foxyblocks]

Why not passing a static prop to getInitialProps I can see how most "initial props" will be shared between a static snapshot of the site and the "dynamic" version with maybe a small difference

I'm not sure I see the use case of having a single app support both being hosted statically and dynamically.

[jstcki]

@impronunciable see discussion above? 😇

[impronunciable]

@herrstucki I don't agree adding new methods is the way to go, I saw the convo but it's something we should discuss further before a change as big as this one

[foxyblocks]

As far as the cognitive overload of having the getInitialProps running in 3 different contexts I don't think it will actually be a common issue. I think in most use cases it will only ever be at most 2 contexts.
One context is either export or req and the other is the client side context. Can you envisage a scenario that would use all three?

[matthewmueller]

TL;DR I'd like next export to work out of the box for everyone without any code changes. I originally had getInitialProps({ build: true }) but that broke a large web app because the plugins I was using didn't consider a build type that wasn't server or client.

---

It's definitely a deviation, but I think it's in right direction since it allows you to introduce new hooks in a non-breaking way and each function only runs in a single context, so you don't need to consider each environment the function could be run in.

The example I keep thinking about is React itself. React could be done as one big update method:

function render ({ componentDidMount, componentWillUpdate }) {
  if (componentDidMount) {
     // ...
  } else {
     // render
  }
}

But that would get really confusing. Now I can't really envision next.js introducing many more of these lifecycle methods, so maybe it's fine to just keep it as getInitialProps() and then we'll update our plugins.

Another consideration is { static: true } and { export: true } are reserved words, so it would have to be something like { build: true }.

[pklada]

What's the status on this? We'd love to use this internally -- I'm curious if this is something that will be merged any time soon. Is it under review?

[MoOx]

@matthewmueller there is now some conflicts on this PR. Hope a solution will be found (I can't really help since I am not a next.js expert). I agree that avoiding a good method seems a good idea. Will a breaking change be a real problem @impronunciable @rauchg (=> should this feature need to wait for 3.0)?

[matthewmueller]

@MoOx i'm still waiting on feedback from the questions above. i'll plan on updating the PR when there's more direction and priority. what i can say is that i've been using this branch for about a month in production and it works great, though there are still a few rough edges that need to get ironed out with the development server.

the development server would need to be updated to work better with export. next export relies on getInitialProps getting called on page load on the client-side. The development server calls it on the server-side, but not on page load on the client-side. So right now, using the development server can lead you astray because the lifecycles are not identical yet. what this means for you right now is that you'll probably want to create your own static file server and run next export each time you make a change.

[craigmulligan]

@matthewmueller I've been using it for a little while too, working pretty well, I actually haven't hit the lifecycle issue yet.

One thing I would add is the build hash for cache busting, currently it just exports a app.js bundle.

I'm now starting to look at how to add the dynamic routing to mimic the server-side api. Don't want to get too ahead but have you looked at this at all?

[tscanlin]

@matthewmueller Thanks for creating this PR and getting export functionality working! This is awesome and very useful for things like github pages where it needs to be static.

One issue I ran into though is trying to get this to play nice with non-root paths. For example, I have a project (http://tscanlin.github.io/tocbot/) and the url is not the root of the domain so right now I am fixing in a very hacky way... (https://github.com/tscanlin/tocbot/blob/master/script/fix-export.sh)
Some how it has this weird flash of mis styling that I can't figure out though. Any ideas?

But otherwise it works great locally, maybe there could be a way to export pure js files instead of json files so that eval isn't needed. Let me know if you want any help with this but it sounds like you are still waiting for some author feedback. Nice work!

[tscanlin]

Well.. I decided I couldn't wait for this to land so I created this: https://github.com/tscanlin/next-export

And I successfully deployed what I'm assuming is the first static next.js site on github pages, check it out!
http://tscanlin.github.io/tocbot/

My script draws heavily on the work of @matthewmueller so thank you man! Cheerio.js is also amazing :)

The main thing it does away with is the getStaticInitialProps bit for the _document.js which threw me off initially. But I kept it for components to use for added flexibility. Feedback welcome.

[jstcki]

next export now is available in v3@beta.

So 🎉 for the feature but 😕 for the work that went into this PR. Thanks @matthewmueller!

[arunoda]

@herrstucki That's not a waste.
I could built next-export comes with 3.0 so quickly due to all the experiments and research going over this PR and in the related issue.

Thanks @matthewmueller for your great work.

---

I'm closing this due to this: https://zeit.co/blog/next3-preview

[jstcki]

Good to hear. Sorry if I came across grumpy @arunoda. I'm very happy that this became a core feature!

[matthewmueller]

Happy to see this getting some attention, regardless of if this work was used or not. One thing:

Basically, you will not be able to render content dynamic in the server as we prebuilt HTML files. If you need that, you need run your app with next start.

Correct me if I'm wrong here, but it sounds like there is no way to dynamically fetch content before your page is rendered. In other words, getInitialProps() does not get called on the client-side before the page is rendered.

If that's the case, it greatly reduces the usefulness of static exports. Static deploys are not only useful when your content is static, but also when you have dynamic content but desire operationally simple deployments and fewer moving parts.

Will this be addressed in a future release?

[tscanlin]

This is awesome, so glad to hear this has landed in core!

I tried it out and it worked well. I had a few ideas to offer.

1. One thing that would be a nice improvement is during the export, delete all of the files out of the export directory, but not the export directory itself. This makes testing easier because you can keep a web server running without getting permission errors from deleting the folder underneath it. del('${exportDir}/*')

2. 2 lines could be added to normalizeRoute in page-loader.js to make multiple static apps easily work from subfolders on the same domain using the already available assetPrefix. The lines are:

route = route.replace(this.assetPrefix, '');
if (!route) { route = '/' }

Without this the file that's requested is like:
/${assetPrefix}/_next/c0bae9f4-37d0-4f02-8a91-08fe02d6fbfb/page/${assetPrefix}/index.js

instead of:
/${assetPrefix}/_next/c0bae9f4-37d0-4f02-8a91-08fe02d6fbfb/page/index.js

like you'd want.

What do you say @herrstucki / @arunoda any way one or both of these changes could make it in??

[jstcki]

@matthewmueller shouldn't you just fetch dynamic data in componentDidMount then? With static pages you can't wait for that data before displaying anything like you can with server-rendered ones.

[matthewmueller]

@herrstucki

shouldn't you just fetch dynamic data in componentDidMount then? With static pages you can't wait for that data before displaying anything like you can with server-rendered ones.

You definitely can, but then you run into a situation where the router does different things depending on how you arrive at the page. If you arrive at it via page load you'd put your fetch logic inside a componentDidMount but if you come to the page via the client-side router it'd call getInitialProps, so navigating around and hitting the back button would do different things. Maybe the solution is to put all fetching inside componentDidMount, but then I think your router fetched view wouldn't be ready to go after the AJAX request.

Also, it's not quite a blank slate, because you have statically-rendered HTML that it serves up first. From what I can tell, the version of next export that's in 3.0-alpha will call getInitialProps during the build phase to give you HTML, and not during the client-side page loading phase. The PR here calls getStaticInitialProps during the build phase, and getInitialProps on page load.

[jstcki]

@matthewmueller but calling getInitialProps on page load would essentially call it twice (once during export and once in the browser), no? And you can't block the HTML from rendering anyway. So unless I'm missing simething completely, I see two options:

1. export all dynamic pages ahead of time
2. defer dynamic data loading completely to the client using React's normal lifecycle

Either way, getInitialProps is called only either during export or when you route to the page.

[matthewmueller]

@herrstucki yep, I'm proposing it would be called twice but in different contexts. Looks like this:

1. At next export: getInitialProps({ query, path }) + some way to differentiate that this is a build step. This will generate raw HTML and will serve as the shell of your application.
2. On page load: getInitialProps({ query, path }). At this point your application shell is visible, this step fills in your application with dynamic data

The 1st one isn't useful for loading anything dynamic (just building your application shell or static content), the 2nd one is useful for loading dynamic data (like user data).

What I'd like to see is some way to do 2. in a way that's consistent with client-side routing. I'm not partial to the way in which this is achieved, just that it's possible to do.

/cc @arunoda

[arunoda]

@herrstucki @matthewmueller In our next-export implementation, it works exactly as how it's works normally with Next.js.

getInitialProps runs when running next-export as it's similar to server rendering of the page.

When the page loads, getInitialProps won't run. But when you are doing a client side routing it will.

[jstcki]

@matthewmueller but isn't your 2nd point already solved with my 2nd point (using React's lifecycle)? That's what you'd do in a static app without next.js as well, right?

[matthewmueller]

It's worth a try, but imagine you have two pages where you need to load user data and you have a <Link /> that goes between them. Where do you put your logic to fetch user data? getInitialProps or componentDidMount?

If you do all the loading in componentDidMount which is what I think you're suggesting (correct me if I'm wrong), the <Link /> will transition to the upcoming page before you load the data. Maybe this is desirable, but I'd rather see my view + data when I click and show loading (if need be) on the current page rather than the upcoming page.

It also makes the router pretty useless for all static builds, which makes next.js less of a win.

[vysakh0]

Since it doesn't use hash router it is not possible to host in sites like s3 or firebase or other static sites? The url comes as /about, not /#/about so will result in document not found error in s3 Or is there a config that i missed? Thank you so much for the feature, it's one thing i was looking forward for in nextjs

// my next.config.js
exports.exportPathMap = () => ({
  "/": { page: "/" },
 "/about": { page: "/about" },
})

[jstcki]

@vysakh0 next export will build individual HTML files in the necessary directories, so it will work with regular URLs on static sites. E.g. the path /about will create a file /about/index.html

[tscanlin]

@herrstucki is there any way we can change the behavior to delete all the files under the export directory but not the export directory itself? del('${exportDir}/*')

[jstcki]

@tscanlin I think that's a great idea but I'm not actually a contributor to next.js – I've just been particularly interested in this feature 😅

So probably the best way to get this in is to make a PR on the v3-beta branch.

[tscanlin]

Haha, no worries, thanks!