Bloated zip on npm

[stuft2]

Prerequisites

• I have written a descriptive issue title
• I have searched existing issues to ensure the issue has not already been raised

Issue

NIT: The package on NPM includes tests, examples, and .github in the download which unnecessarily increases the size of the package. Not a huge deal but an easy fix.

[Uzlopak]

Moving this issue to help as it is a topic regarding all repos/packages

[Eomm]

Thank you for your interest in this topic. Please see the discussion in fastify/skeleton#42 (comment) for the reasoning against such a change.

[stuft2]

Thanks for the insight. I generally understand the issue but may lack in the open source development experience to fully understand.

The reasoning for not using the files property in the package.json seems incorrect to me:

The result is that once the files field is added then a new entry must be added for every file added to the project.

Using typescript results in a known input and output location. And using a glob pattern on one (or maybe both) of those locations is all that's needed. The license and readme are automatically included. Are there other artifacts that would need to be in npm that I'm omitting?

[Eomm]

I got the point, and I must say that I'm pro reducing the package size tbh

but:

Are there other artifacts that would need to be in npm that I'm omitting?

Are you suggesting migrating ~60 repositories to typescript?
In this case, the fastify team has few contributors that are TS devs, and others are not interested in learning it - it would be unfeasible

The big deal to re-open the discussion with the whole team is:
What is the PRO to cut some KB on the backend side vs creating a package artifact that in 2300 will still be self-complete?

[stuft2]

Are you suggesting migrating ~60 repositories to typescript?

No, sorry. The way I phrased my rebuttal has more to do with how I code and was probably confusing. However, supporting type definitions in the repository requires a known file structure to allow type completion to work (e.g. using the (package.json).types property or just using index.js and index.d.ts at the root).

The point being that the file structure, though inconsistent between fastify repos, is still well organized, so glob patterns could be leveraged with confidence. Some fastify repos could use (package.json).files = ['lib/*.js', 'types/*.d.ts'] while others could use (package.json).files = [ '*.d.ts', '*.js' ]. If glob patterns won't work, then I guess I would have to question the organization of the repo further.

What is the PRO to cut some KB on the backend side vs creating a package artifact that in 2300 will still be self-complete?

A self-complete package artifact on NPM in 2300 seems duplicative to me if you already have the self-complete source in GitHub. The previous discussion covers the list of pros very well, I think.

I resurfaced this discussion purely because I thought it would be an easy win and because this isn't just a problem for fastify. Too many packages in NPM include vestigial artifacts and if everyone does it, it can make installing dependencies much more burdensome. Where fastify is renown in the javascript/nodejs community and developers may look to it for an example, more care should be given to code leanly.

[jsumners]

They could also learn how to use the source code already on their systems. At least a few of us use the included assets when debugging things.

[stuft2]

They could also learn how to use the source code already on their systems. At least a few of us use the included assets when debugging things.

I'm not sure what you mean. Why would I need .github, tests, examples, docs, etc. if I'm debugging my project? If I'm going to debug something in fastify itself, shouldn't I clone the repo from github instead? At least that way I can create a PR, look through the commit history, etc. which is more helpful than downloading and debugging fastify from npm.

[Uzlopak]

To be honest I also prefer only having active code in the npm packages because nothing is more annoying than installing node modules with burger king internet. Also happened that i could reconstruct the codebase of some abandoned npm packages because it was containing more than the active code (mainly tests and docs).

[mcollina]

The latter is why we include all the relevant artifacts in our releases.

[stuft2]

I'm okay with backups, but NPM probably is not the best place for them IMO. And if @Uzlopak's scenario is based on repos that he doesn't maintain, then it makes sense but fastify doesn't fit that scenario either.

[Eomm]

A self-complete package artifact on NPM in 2300 seems duplicative to me if you already have the self-complete source in GitHub

An npm package is forever.
A GitHub repo is not.

I have plenty examples, here is one:

https://www.npmjs.com/package/outpipe
Click on the repo link and you will get a 404

The only though I got is:

• compress the additional stuff (it requires a prepublish/postpublish script)
• maintain 2 registry: npm full payload, gh-registry light payload

In any case, it is more work

[climba03003]

I still see no gain in saving that files space or bandwidth especially we are not talking about >= 5 MB (Even the main repo only takes 2.04 MB).
Today is 2023, I believe most of the internet can support the bandwidth of such size.

If anyone do consider the size, they have plenty of way to remove it.
For example,

• docker using multi-stage to reduce the size
• yarn provide a way to reduce the unnecessary files
• pnpm used to reduce the network bandwidth
• setup a cache proxy if you are about download speed (I personally setup a proxy npm registry to cache and provide faster installation speed)

[stuft2]

If you're really interested in backups, GitHub recommends using backup utilities on GitHub marketplace.

Just for reference, fastify weekly downloads currently add up to 2TB vs 73GB if the package wasn't being used as a repo backup. That's a 96.35% decrease. Over a year, that adds up to 100TB decrease which might not seem like a lot until we imagine everyone using NPM in the same way year after year.

As much as I hope you'll reconsider, keeping fastify stable, maintainable, and long-living is a great goal regardless of how its achieved. I'm happy to continue to respond as needed, but I think I'll just leave this in your hands now. Thanks all.

[mikhail-g]

To sum up the pros and cons of having non-production files in the npm package

Pros:

1. As a backup (mainly tests and docs) when GitHub repo gets deleted the dev code remains in npm package
2. To test how the lib works in the client environment (node version, OS)

Cons:

1. Test files are potential source of failures in client systems as quality standards for them are normally lower than for prod files, e.g. whitespaces in folder names may lead to failures in Linux systems, cli, etc.
2. Increased size of the package that leads to increased bandwidth and time on install

Solutions:

1. maintain 2 registry: npm full payload, gh-registry light payload
2. compress the additional stuff (it requires a prepublish/postpublish script)
3. docker using multi-stage to reduce the size
4. yarn provide a way to reduce the unnecessary files
5. pnpm used to reduce the network bandwidth
6. setup a cache proxy if you are about download speed
7. backup utilities on GitHub marketplace.

[Uzlopak]

Solutions:
8. Keep it as is.