Abilitty to skipping static pages generating at build time

[hlege]

Describe the feature you'd like to request

In next 13 with app directory there is no way to opt out from build time static page generation without loosing ISR. (Only dynamic render). If we want to use the same code base with different backend server (e.g configured in .env files) we force to use SSR and loose ISR.

These feature allows use case like dynamic portals with different backends servers and different data but the same pages and business logic.

Other issues is that the backend server often cannot be reach by developer/build machine.

In a big portal site this feature will also allow faster build and deploy time too, but preserve the ability to use ISR if needed.

Describe the solution you'd like

Configuration based settings:

const nextConfig = {
   ssg: false // means build time the pages wont generate static html, only runtime with ISR
};

Describe alternatives you've considered

Per paged configuration:

export const ssg = false;

or

export const preventSSG = true;

[Xevion]

Why don't you simply modify generateStaticParams to not generate any of the parameters at build time? Is there are particular use-case that makes a site-wide SSG disabling configuration necessary or abundantly better?

[neslob]

export async function generateStaticParams() {
  if (env.SKIP_GENERATE_STATIC_PARAMS === 'true') return [];

  // your existing code follows here
}

I am using @t3-oss/env-nextjs and added:

SKIP_GENERATE_STATIC_PARAMS: z.optional(z.string())

[hlege]

generateStaticParams only work pages with dynamic parameters (slug), if you has a page e.g "/about" with dynamic data coming from backend, then you cannot prevenet SSG, only export const revalidate = 0 works but then its become SSR only.

[dbk91]

If your app directory component is marked as a client component, using the useSearchParams hook will omit the component from the static builds and force it to fetch client-side only.

During static rendering, any component using the useSearchParams hook will skip the prerendering step and instead be rendered on the client at runtime.

But I think I am confused about your use case. Can you clarify your first statement?

In next 13 with app directory there is no way to opt out from build time static page generation without loosing ISR. (Only dynamic render). If we want to use the same code base with different backend server (e.g configured in .env files) we force to use SSR and loose ISR.

Opting out of static page generation causes you to lose ISR because there is nothing to statically generate, hence Incremental Static Regeneration. The only option when opting out of a build runtime is to fetch it dynamically server side (e.g. getServerSideProps) because you'd be choosing to not run anything at build time. I'm not sure how you would be able to skip statically generating a page while simultaneously using ISR without that being the definition of fetching data dynamically server-side.

Please let me know if I misunderstood your problem.

[hlege]

I will try to explain it via example.

Given a page named /about . This page uses data fetched from backend. The backend data will change rarely or never change at all.

My problem is that the backend server cannot be reached at build time. Also I want to use the same application with different backend server.

So right now in order to build the application I am forced to use dynamic render (SSR). The problem with SSR is that every request to the /about page will dynamically fetch the data (which "never" changes ) and render the html. Of course I can make a custom cache that store the fetched data, but still the html will render every request. Also there is no cache control headers because it's SSR, so every request will hit backend even if it is still the same result every time.

So if we generate a static html on demand (not in build time) the nextjs features like cache headers control and ISR still works without the need to create a custom server.

[Sharpiro]

If we're explicitly saying "use client" at the top of the file, can't this disable the server-side stuff. Annoying to have errors about "localStorage" not available because a client component is trying to render on the server initially for some reason.

[Sharpiro]

There should be a “use client only”

[dbk91]

@hlege You simply want to avoid data fetching during the build process due to backend resources potentially being unavailable, but you want it to fetch and render statically with ISR once deployed, correct?

If this is the case, I think you could simply avoid fetching the data within your server component by referencing a build time constant.

async function getPosts() {
  const res = await fetch(`https://.../posts`, { next: { revalidate: 60 } });
  const data = await res.json();

  return data.posts;
}

export default async function PostList() {
  let posts = [];
  if (process.env.BUILD_ENVIRONMENT !== 'local') {
    posts = await getPosts();
  }

  return posts.map((post) => <div>{post.name}</div>);
};

In the example, process.env.BUILD_ENVIRONMENT would be a user-defined build constant used to differentiate between your local environment and the deployed environment. This is of course assuming you are using a single static route and not any dynamic segments. For dynamic segments, I would recommend the solution @Xevion mentioned as it is more elegant.

[benderillo]

@dbk91

@hlege You simply want to avoid data fetching during the build process due to backend resources potentially being unavailable, but you want it to fetch and render statically with ISR once deployed, correct?

If this is the case, I think you could simply avoid fetching the data within your server component by referencing a build time constant.

async function getPosts() {
  const res = await fetch(`https://.../posts`, { next: { revalidate: 60 } });
  const data = await res.json();

  return data.posts;
}

export default async function PostList() {
  let posts = [];
  if (process.env.BUILD_ENVIRONMENT !== 'local') {
    posts = await getPosts();
  }

  return posts.map((post) => <div>{post.name}</div>);
};

In the example, process.env.BUILD_ENVIRONMENT would be a user-defined build constant used to differentiate between your local environment and the deployed environment. This is of course assuming you are using a single static route and not any dynamic segments. For dynamic segments, I would recommend the solution @Xevion mentioned as it is more elegant.

Your solution is like a hack around a limitation of the framework. And what @Xevion suggested applies only to the dynamic segments and even then it does not work (maybe a bug in nextjs13): if you have a dynamic segment page say you have a folders like this: "/posts/[slug]" and inside there is a page.tsx just like in their documentation

type Props = {
  params: {
    slug: string;
  };
};

export default async function Page({ params: { slug } }: Props) {
  const post = await getPost(slug);
  return (
    <article>
      {post.title}
    </article>
  );
}

When I try to build this with yarn build, I get the build error because getPost is called with "[slug]" as the value of the slug parameter passed into Page(). That is because I do not provide the generateStaticParams function.
Silly me assumed that this will just skip the SSG and I can have SSR.
I can make it work by exporting the export const dynamic = 'force-dynamic'; then it will force "no-cache" and I get SSR.
Like explained here: https://beta.nextjs.org/docs/api-reference/segment-config#dynamic

But what the topic starter and I actually want is the ability to have a cache control with "generate on first request" (if I understood the request correctly).

Imagine you have a page that is not generated at build time but set to SSR with a cache control that says "revalidate = false"; basically once rendered it stays cached ( as if it was SSG).

This is what ideally would be the most suitable solution for the author of this ticket.
Imagine a blogs website with a couple thousands of posts. I don't want to SSG them. Ideally I want them to be SSR on the first request with cache TTL set to like forever. So the first time someone tries to open a post, it gets rendered and cached.
Next time it is the cache all the way.
And If I want to invalidate, I can have an API route that would do manual revalidation of any page like it is supported in Nextjs 12.

[dbk91]

Your solution is like a hack around a limitation of the framework. And what @Xevion suggested applies only to the dynamic segments ...

Yes, I am aware. I called both of those points out specifically in my original comment. My intent was to provide some solution to an uncommon use-case.

Imagine a blogs website with a couple thousands of posts. I don't want to SSG them.

I don't think a blog is the best example for the author's original scenario. The author has a limited access to the resources required to generate the pages at build time. I don't think this issue was about avoiding the cost of generating content at build time. It's typically ideal to generate these at build time so that the first user to visit the page doesn't incur the cost of waiting for the page to generate using SSR. This is also what enables the effectiveness ISR — the worst-case performance scenario is the user views stale content as opposed to waiting for a page generated using SSR.

[hlege]

i am well aware that i can use build time env for that, but than everything need to be checked and its a little "ugly" and i thinks nextjs can support it out of box. Also i think a common cms/portal like webpage is use a render strategy like: render when user request a page -> then cache it until its expired or content is changed.

the worst-case performance scenario is the user views stale content as opposed to waiting for a page generated using SSR.

In nextjs 13 the streaming rendering already solve this problem with a proper page layout structure too.

[dbk91]

@hlege Totally agreed. I realize in retrospect that my eagerness to help you find a near-term solution, albeit a hack, was seemingly dismissive of your original feature request. My apologies for that. To be clear, I do agree something like this would be useful for the framework to expose — in particular the per page configuration at a minimum.

Tangentially, I found an issue specific to client components when trying to leverage the loading and error special file structure with SWR/suspense that a solution like this could fix more elegantly. SWR requires fallbackData when using suspense but that causes the application to flash content with the content of fallback data. This doesn't work well since client-side data might be specific to a user or frequently updated and will flash intermittently between the loading state and the pre-generated content. But if you don't provide content, it throws an error at build-time. The only workaround I found was to use useSearchParams to opt out at build time and then omit the fallbackData, but you have to place it before the useSWR hook, which is not obvious. It feels inelegant if you're not referencing search parameters in your component.

Of course, that example is not applicable to your issue since that's specific to client components and client-side fetching. I simply wanted to provide that to present another use-case to support this feature request.

[benderillo]

@dbk91

I don't think a blog is the best example for the author's original scenario.
No, it is not :) I added it as a generalisation of the issue. Because what the author asks will handle both.

the worst-case performance scenario is the user views stale content as opposed to waiting for a page generated using SSR.
I don't think we should be worrying about SSR performance too much these days. Look at the Remix framework that is all SSR deliberately but I doubt it has performance drawbacks compared to NextJS (they actually show in their blog that they are faster).
Rendering is moving to the Edge and that makes SSR blazingly fast :)

[dbk91]

@dbk91

I don't think a blog is the best example for the author's original scenario.
No, it is not :) I added it as a generalisation of the issue. Because what the author asks will handle both.

the worst-case performance scenario is the user views stale content as opposed to waiting for a page generated using SSR.
I don't think we should be worrying about SSR performance too much these days. Look at the Remix framework that is all SSR deliberately but I doubt it has performance drawbacks compared to NextJS (they actually show in their blog that they are faster).
Rendering is moving to the Edge and that makes SSR blazingly fast :)

There are some good points here. I think it's important to keep in mind that SSR can be negatively impacted if the API you're integrating with is slow. For example, I have some personal projects where I serve content from my Notion via their API. In my experience, it's been slower than most so I try to do everything at build time. 😅 Having said that, you're correct in that it's up to the developer to determine those scenarios and edge runtimes can help this tremendously. And as @hlege pointed out, the appDir does provide improved performance if you leverage the layout structures in a fashion where you can build the majority of the static content and then display instant loading states otherwise.

I think this feature could be an overall win for the framework.

[agusterodin]

We have a similar requirement at my organization.

Our application has whitelabeling requirements (eg: custom brand name in browser title bar). We need to deploy run multiple versions of the same exact application build multiple times but with slightly different environment variables. The application needs to be deployed via Docker to custom infrastructure (customer requirements, not ours). We want to avoid performing multiple docker builds and maintaining many redundant docker images just to change a few build-time environment variables. Our proposed solution is to use runtime environment variables.

We are able to accomplish runtime environment variables with publicRuntimeConfig. I want to use getStaticProps so we can take advantage of incremental static regeneration and not take the performance hit that SSR would cause. Unfortunately there is no means to perform something similar to fallback: 'blocking' for non-dynamic routes.

I don't even need to revalidate as we don't have any blocking data requirements. For all I care I would set revalidate to an arbitrarily high value so that revalidation never happens.

Ideally there would be a way to tell the server to generate and cache ALL pages when the built server process starts (similar to what happens with static optimization during build time).

Basically what i'm looking for is "run-time" static page optimization instead of "build-time" static page optimization. We have a very small amount of pages, so run-time static optimization slowness isn't a concern.

If this isn't currently possible, does the app directory / Suspense make this possible?

[ngwie]

I have the same problem with @agusterodin, which is variables aren’t available at build time.

At the moment I fixed it with deleting generated static file after build.

RUN yarn build

# Remove SSG file at build time
# Also removes files in .next/standalone/.next folder if next config output is 'standalone'
RUN rm ./.next/**/*.html ./.next/standalone/.next/**/*.html

I hope this helps the others. But options to skip SSG at build time is nice to have.

[CaptainN]

Is it required to run next experimental-generate before next start after next experimental-compile? Ideally, if you don't run generate, then it'll generate pages in to the ISR cache at runtime.

[hlege]

Totally agreed!

[arekrgw]

What should be the commands order while using output mode as standalone?

Assume a multi stage docker image build. Is it possible to run next experimental-generate after copying files from standalone folder to runner docker image?

Overall, how should this process look like, or maybe it is possible to skip next experimental-generate so the static will be generated at runtime at first request?

[ReesMorris]

This isn't a direct solution, but does work for our use-case. We have a single Next.js codebase which is deployed to different customers, all with different environment variables (such as API URLs, etc).

We have a docker-entrypoint.sh file:

#!/bin/sh
ENV_PATH="/app/.env"

# Overwrite the `.env` file with variables in the current environment
printenv > $ENV_PATH

# Delete all .html and .rsc files within the .next directory
find /app/.next -type f \( -name "*.html" -o -name "*.rsc" \) -delete

# Start the Next.js application using the exec command
exec "$@"

And in our Dockerfile we simply reference it as an ENTRYPOINT:

# Copy the entry point script into the image and make it executable
COPY docker-entrypoint.sh ./docker-entrypoint.sh
RUN chmod +x ./docker-entrypoint.sh
...
ENTRYPOINT ["./docker-entrypoint.sh"]
CMD ["node", "server.js"]

Essentially, we build our application using the environment variables for one of our customers (typically a test instance with the smallest amounts of data/latency) then, just before the container runs, the docker-entrypoint.sh script will overwrite the .env file in the image (which gets generated along with the build) and delete all pre-rendered static pages that used the dummy data. This forces Next to re-compile each page on first visit, but then (as far as we can see) fall back to ISR behaviour. Since all our customer APIs return the exact same data structure, just with their own CMS' content, there's no issues.

A few notes:

• printenv will copy every environment variable over from the container
• this is designed for the standalone output, which has a different data structure
• if Next changes how pages are built/stored, this solution very likely will break.

[DarkPromise]

next experimental-compile seems to be broken in >14.2.0, prior to this, it was working with 14.1.4.

NOTE This is running in docker

The error I get in 14.2.1 now is:

[builder 4/4] RUN npm run build-experimental:
0.748
0.748 > scst@0.1.0 build-experimental
0.748 > next experimental-compile
0.748
0.878  ⚠ You are using a non-standard "NODE_ENV" value in your environment. This creates inconsistencies in the project and is strongly advised against. Read more: https://nextjs.org/docs/messages/non-standard-node-env
1.019  ⨯ Invalid project directory provided, no such directory: /app/experimental-compile

Prior to the command running, the following environment variables are injected:
ENV NEXT_TELEMETRY_DISABLED 1
ENV NODE_ENV production

Some things has changed between the version upgrades, does anyone else have this issue?

For context, this is the full build step I used:

# Rebuild the source code only when needed
FROM base AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .

ENV NEXT_TELEMETRY_DISABLED 1
ENV NODE_ENV production

# Build
# See https://github.com/vercel/next.js/discussions/46544#discussioncomment-7115457
RUN npm run build-experimental

Thanks

[arekrgw]

Confirming the issue.

For our project it's critical to work, would like to know if this will be fixed or should we migrate to something else

[arekrgw]

Okay, the scripts were refactored and now this command should be invoked differently

Now we should use the next build --experimental-build-mode compile/generate

Ref:

next.js/packages/next/src/bin/next.ts

Line 116 in c0ae6f6

'--experimental-build-mode [mode]',

next.js/docs/02-app/02-api-reference/08-next-cli.mdx

Line 189 in c0ae6f6

--experimental-build-mode [mode] Uses an experimental build mode. (choices: "compile"

[DarkPromise]

Cool, thanks for the heads up

[jarosik10]

So next build --experimental-build-mode generate has to be run with application source code and it is necessary to copy all app source files to docker image?

[m-sanders]

I have just tried moving to this for our setup and ultimately reverted the change.

I was running the generate step as an npm pre-step on our start command. Our containers have memory limits of 2GB and normally remain stable around the 700-900MB mark. The generate step, eats memory and very quickly causes an OOM error and kills the pod. I understand that Vercel uses 8GB for their builders. I bumped memory limits to 8GB to match, and it worked but seems excessive. While I could rework this to a separate job, having the added generate step also slows down releases as we have to pre-render quite a lot of static routes for all of our locales.

This is a step in the right direction but it is not the ideal solution. What I would like is to make all routes behave like Page Router fallback mode on dynamic routes. Build the code and manifest but do not generate any prerenders, then rely on fallback to lazily build things at runtime. I can see that others have hacked their way to this solution by doing a build and then deleting the *.html and *.rsc output in the next directory.

EDIT: I got to the bottom of this. We do not set CPU limits on our pods but we do set memory limits. We end up with too many workers for the available memory. Setting experimental.cpus to a sensible number based on the memory available fixed this. I found that with 4 cpus and 3GB resulted in consistent crash-free generates using around 2GB. I also tried experimental.memoryBasedWorkersCount: true but that option still resulted in OOM. I felt that it was safest to explicitly set the count than rely on the implementation of memoryBasedWorkersCount.

[CaptainN]

Couple questions:

• Is it required to use the generate step? That has to be done at runtime (because it access external resources which aren't even accessible to the build environment), which requires a bunch of support files including the full source of the application and node_modules be available in that environment. It'd be stellar if we could simply skip that, and let the ISR cache be filled at runtime for all routes. It seems this is not yet possible.
• Given the above, this split between compile/generate doesn't seem applicable to standalone mode. Is that right?

[tusmenko]

NEXT_PUBLIC_ is another kind of obstacle to the "build once - deploy many" place. It's sad how many next-libraries require them instead of explicit configuration - even the Next itself. Even if that is relevant primarily to React itself and the Web in general, there are good examples of avoiding this limitation in other React frameworks.

[CaptainN]

I'm aware. We have simply duplicated those values - it's not ideal, but since it doesn't hit any of the URLs defined in those variables during build time, it's fine for us. For variable output (for us there are only 2 - staging and prod), we have created multiple configs, with the differing values in there.

The only other kind of odd thing is that we are using Sentry.io - and its integration does reach out to upload sourcemaps during compile time - that's not ideal, as it's technically a "side effect" which we are supposed to avoid in Docker, but whatevs, it's fine, I guess.

[achton]

With NextJS 14 and App Router, I've been through the whole shebang:

• normal next build - build env did not have the required env vars
• next-runtime-env and rewriting env var usage - hydration errors everywhere, and it also forces dynamic rendering
• now trying to avoid NEXT_PUBLIC_* entirely, but without luck - client requires some dynamic env var values unavailable at the compile step

How have you solved the scenario where the variable values are environment-specific and thus totally unknown until runtime (or the generate step)? @CaptainN @tusmenko

[tusmenko]

@achton, unfortunately, there is no straight way. In my case, I am avoiding NEXT_PUBLIC_ variables where possible, with just a few of them remaining required (for Sentry and NextAuth). I'm keeping them in sync with runtime in my build pipeline. And it's not a huge problem, as they are changing not often. Just smell...
Luckily, most of the libs are not relying on NEXT_PUBLIC_ stuff, and hopefully, those who do - will stop soon, as it's a dumb practice.

[CaptainN]

I just embraced duplicating the NEXT_PUBLIC_* vars - like I said above, we have separate build pipelines for the varying deployments. If you have dozens, that might get a bit overwhelming, but we have only 2, so it's fine for us. It is not optimal to have to maintain those settings in multiple places (both sets of action files, the Dockerfile which needs to fold those in, and our kustomization files for runtime. It's not DRY at all.)

[CaptainN]

I may have encountered a bug related to this? (Maybe not?) - basically, the chunk filenames are not deterministic based on BUILD_ID - is that unexpected behavior?

[hannanstd]

Add:

export const dynamic = 'force-dynamic';

in your root layout.tsx.

[tinhochu]

this works!

[sleroq]

With 'force-dynamic' you are losing ISR and everything becomes SSR.

[hannanstd]

you can put it in the pages which use prisma rather than putting on root layout

[achton]

@ijjk Is there any chance that the next build --experimental-build-mode generate step could inline client side vars (NEXT_PUBLIC_) instead of compile?

In my view, this would complete the featureset of the experimental build mode with the compile step being available for building a static, build-once-deploy-everywhere type bundle, which can then be pre-rendered after the fact with dynamic values for the client-side.

[ijjk]

This is being investigated

[RareSecond]

next build --experimental-build-mode generate doesn't seem to run the Linting and checking validity of types step either. Is this by design?

[ijjk]

Yes by design, this step is only meant to generate the prerendered assets

[hlege]

#46544 (comment)

Is it required to run next experimental-generate before next start after next experimental-compile? Ideally, if you don't run generate, then it'll generate pages in to the ISR cache at runtime.

@ijjk Is there any upcoming change regarding this? Currently, if we run only next build experimental-mode compile, all request end up being rendered dynamically on the server, without opting in to ISR cache.

The original request is still not achievable.

• next build experimental-mode compile makes all routes dynamically rendered without opting in to ISR cache.
• next build experimental-mode generate cannot be used in standalone deploy mode, as it requires all the source code, node_modules, etc.

[CaptainN]

The generate step also adds minutes to my pod's startup time, which is not ideal.

[CaptainN]

Is this feature documented anywhere? It's worth noting, that this is REQUIRED if you want to scale a self hosted set of pods for next.js, when you don't want to (or can't) do the generate step in your build environment. Without it, you hit this bug, which prevents deterministic chunk hashes. But I don't think these experimental build flags are documented anywhere outside of this issue.

(In my case, I can't do the generate step in my build environment, because that env doesn't have access to the backend services which supply the data for the that step, and I can't do the compile step in my runtime environment, because it either takes 15 minutes per pod, or produces variable build artifacts, even with a deterministic buildId, or it crashes my entire cluster (especially if I have Sentry enabled). Also, Sentry doesn't seem to support compile/generate at all, which is kind of a bummer.)

[hdodov]

I've found a workaround that you can use if you have middleware.

1. Change your route from /app/foo/page.tsx to /app/foo/[...dynamic]/page.tsx

2. Add export const revalidate = 1800 in it

3. Add the following rewrite in your middleware.ts:

   if (path === '/foo') return NextResponse.rewrite(new URL('/foo/dynamic', request.url))

This will opt the route out of generation during build, but keep the ISR cache. You can't do this for sitemap.ts, though, which was my use case...

P.S. next build experimental-mode compile is useless for me too - it just switches all routes to SSR (Next.js v15.0.1).
P.S. It's crazy that you can't do something so simple. Really starting to regret picking Next.....

[CaptainN]

You can force static rendering - which IMHO, should be a requirement.

export const static = "force-static";

The automagic config thing is an absolute nightmare, reminiscent of Meteor.js. It was a mistake. Better - force declaring a default (static, dynamic, etc.) then use the detection methods to WARN when something won't work in the console.

[hdodov]

But with force-static it'll attempt to generate it during build as well, which is the main issue. By putting [...dynamic] in the middle, Next doesn't attempt to generate on build.

[hdodov]

There's another solution.

It seems like Next sets a NEXT_PHASE env variable during build. You can use it to skip the database call like this:

export const revalidate = 1800;

export default async function sitemap() {
  if (process.env.NEXT_PHASE === "phase-production-build") return [];

  return db.find(/* ... */).map(/* ... */);
}

With only this, however, the first render of the page/route would render the empty data generated during the build. So you have to remove the accompanying *.body and *.meta files that are generated. In my case, I use Docker, so I do it like this:

# ...

COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static

# This right here:
RUN rm -rf /app/.next/server/app/sitemap.xml.*

USER nextjs

# ...

My Dockerfile is based on the official one here. With this setup, I get ISR in production mode and with no issues during build.

[Quozzo]

Interestingly I get this error and needed to pass in an empty array/object to the unstable_cache function instead of directly returning the empty.
More interestingly, the error doesn't occur on the (formerly) static pages but on the dyncamic ones.