Reducing package size

[frankdejonge]

Describe the feature

It would be great if this library could publish sub-splits to publish smaller sub-packages. The current artefact is 29MB, a substantial amount of data. Many users only use a fraction of the total package size, yet the code is shipped over and over. As an example, the Symfony codebase uses sub-splits to allow independent distribution of components. Flysystem, the project I maintain, does the same for all of the adapters. I've personally set this up and would be willing to help out setting it up or answering questions about it. I believe this would be hugely beneficial for our users and also reduces the environmental impact.

Please consider this 🙏

Use Case

When I download this package, I download 29MB while I use but a fraction of the source code.

Proposed Solution

Setup splish/lite (https://github.com/splitsh/lite)

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

SDK version used

Irrelevant

Environment details (Version of PHP (php -v)? OS name and version, etc.)

Irrelevant

[stobrien89]

Hi @frankdejonge,

Thanks for the feature request. We've given this serious consideration, but we're not in a position to manage sub-trees/repositories at this time. We plan to make the SDK modular in the next major version and will likely use Satis or similar.

That said, we realize package size is an issue for a number of customers and would like to know how many people would be interested in a feature that cleans up unused services. We're open to other suggestions as well— I'm going to convert this to a discussion for now so we can get an idea of what path would be best for everyone.

[frankdejonge]

Hi @stobrien89 I think any effort to reduce package size is great. When it comes to the proposed solution, there are might be some unfortunate downsides. With containers being a common artefact form for distributing application, each command results in a layer, so people need to be very aware that these commands need to not suffer from having an intermediate layer where does contain the code. That said, it does make it possible to reduce package size, which is great.

[tomasfejfar]

The feature to clean up unused stuff afterwards is really not ideal. One example being using Docker for Windows. Disk IO is rather slow in docker mounted volume and usual composer install call takes 10s to finish everything except aws sdk, which takes additional 100s or so. Also it's one more different layer of knowledge for the developers to maintain - when you need another AWS service, you need to know that there is this proprietary way of dealing with AWS SDK dependency. I sense many "I have aws-sdk-php required in composer, but S3 client cannot be found" bug reports.

I think the split into multiple packages could be very crude with very basic dependencies you already have (to build the "cleanup tool") and still solve the problem completely.
There could be some "composer dependency magic" to resolve the complexities into versioning the different packages. For example every package could require some aws-sdk-php/package-lock dummy package in a same version the package is released under.

Example:

• aws-sdk-php/s3:123.1.8 requires aws-sdk-php/package-lock:123.1.8
• aws-sdk-php/sqs:123.1.8 requires aws-sdk-php/package-lock:123.1.8
• aws-sdk-php/sns:130.0.0 requires aws-sdk-php/package-lock:130.0.0

This way you don't have to deal with compatibility between aws-sdk-php/sns:130.0.0 and aws-sdk-php/sqs:123.1.8, because those have conflicting dependency on aws-sdk-php/package-lock:130.0.0 and cannot be installed together.

[stobrien89]

Hi @tomasfejfar,

Thanks for the feedback. We realize the Composer script isn't the ideal solution, but we wanted to offer a compromise since we've made the decision not to modularize/split this major version of the SDK. Modularization is something we've committed to for the next major version. That being said, we're still willing to entertain other ideas that may make this easier for folks in containerized environments.

[stobrien89]

Just to clarify, anyone who is in favor of the feature I mentioned above can upvote my first comment. Anyone who is not in favor and/or has other ideas or suggestions is welcome to provide feedback.

[tomasfejfar]

Another case for reducing the package size is reducing the time to index the library in IDE (looking at you, PhpStorm). AWS SDK is by far the most released library and it takes by far the most time to index.

[stobrien89]

Hi all,

Just wanted to mention we've just merged a feature which allows you to remove unused services. It should be available in tomorrow's release. Any feedback is welcomed and appreciated.

[tomasfejfar]

I don't think it works like you describe @longshadowdev. I don't think the hooks are executed for each dependency.

Instead you as end user are responsible to put all the required packages in your root composer.json's extra section. If you don't put there all the required packages, it will fail. If you don't have the hook set up for your root repo, nothing will happen IMHO.

[lechatthecat]

Does this mean it's user's responsibility to check which service is depending on which service and put all of the required services in composer.json? users need to manually create such dependency mapping as long as users use the reducing package size feature?

[stobrien89]

@lechatthecat There are only a few services that the SDK's core functionality depends on. As far as I can remember, there are not services that are dependent on other services. All services that are unsafe to delete are marked as such and will not be deleted under any circumstance.

In terms of services that depend on one another in a customer's application architecture, it's entirely up to the customer to determine those dependencies and account for them in the services they specify to keep.

[lechatthecat]

@stobrien89
Thank you for your answer!

[Tofandel]

Note: this is only a 1/3rd solution, it will still require the full download and then removal of the lib via composer, this is not optimal, uses a lot of SSD write cycles and a lot of bandwidth for no valid reason

A switch to a private packagist or https://packagist.org/packages/symplify/monorepo-builder that creates readonly github repos under a new org, would be a much better and future proof solution (of course this will require all current users to require one package per actual package under different names instead of the whole thing) and this can be done by deprecating the old package with an indication about it

[fredericgboutin-yapla]

Another case for reducing the package size is when using AWS Lambda over a custom PHP provider - for example with https://bref.sh/.

There is a limit to the size of the produced package which is something like 220MB. This vendor component alone accounts for almost 16% of that quota, which is huge.

The dumped documentation in src/data subfolders are entirely to blame here.

I suppose I'll give a shot to #2420 (comment) 🤞

[sc0ttdav3y]

I'm running on bref.sh and have found the tool #2420 (comment) has been useful to take a 40 MB dependency down to 5 MB. Great job and thanks @stobrien89 and team.

[yenfryherrerafeliz]

Hello everyone, currently we are exploring making potential improvements in the AWS SDK for PHP. We have created this GitHub issue to better gather feedback from our community. Please look into the options there and vote for the things you think will add value to you, if present in the AWS SDK for PHP.

Thank you!