Long Porter Publish Times

[samjohnsonCF]

Hey all!
I'm curious as to what the expected time it takes to do porter publish? (I know this might vary depending how much you have in your porter.yaml)

I'm seeing somewhere between a 4 to 5 minute time while the porter publish sits at the "Building bundle ===>" line, then it finally starts running through the docker template. Another 4 or so minutes and it finishes building and pushing the image up to our Repository. Is there anyway to speed this process up? I even have the docker template pulling an already built image so it really don't have to do much as far as any installs of software.
Average run times seem to be anywhere from about 8 to 10 minutes with a mostly pre-built docker image and if its building the entire docker image it can take 15 to 20 minutes to do the entire publish command.

Thanks!
Sam - CloudFit

[carolynvs]

Hey Sam, those are pretty long build times and not what I would expect for an image unless it was really quite large or had slow RUN commands in the Dockerfile. For reference, when I build and publish porter-hello (the output of porter create which is ~350MB) it takes about 1m depending on my network.

Can you provide a copy of the output and which version of Porter you are on? With recent releases of v1 you should see the colorized progress-enabled buildkit output which isn't as great for troubleshooting. I recommend running PORTER_VERBOSITY=debug porter build &> porter-build-log.txt so that we can look at the full docker output.

[samjohnsonCF]

Hey Carolyn! So we do have a fairly big bundle. We are pulling in about ~44 images (need to remove some :-p), and it sits around 5.5 GB. Though, the odd part is the actual copy/upload of the bundle is not that long. We are running all this in the Azure pipelines and uploading to an Azure ACR. So speeds as far as copying are not that bad. The odd part is when it initially starts it sits on that buildkit output for a solid 4 or 5 minutes without any movement. Kinda hard to show that as there isn't times in the log. I'll go grab the output for you though as I can just get one from our runs. We are running the latest version of porter as well.

[samjohnsonCF]

porter_publish_output.txt

Attached is our porter publish. I scrubbed all the image names and such, but the times are the important part.

You'll notice between the building bundle and the generating dockerfile, there is a very long pause. Starts at 1:15 and ends at 1:22.

2023-02-14T01:15:31.8289760Z Building bundle ===>  

2023-02-14T01:15:31.8290303Z Using buildkit build driver  

2023-02-14T01:22:48.2275098Z Running linters for each mixin used in the manifest...  

2023-02-14T01:22:48.2839845Z Copying porter runtime ===>

2023-02-14T01:22:48.5158132Z Copying mixins ===> 

2023-02-14T01:22:48.5158623Z Copying mixin exec ===> 

2023-02-14T01:22:48.5679940Z Generating Dockerfile

The actual porter publish part and copying all the images up only takes maybe 2 minutes to complete. I appreciate all your help on this!

[carolynvs]

When you run build, are the images referenced by your porter.yaml file already in the Docker cache? I see that when we generate the .cnab/bundle.json file that we first pull each image to get its repository digest. If you don't have them cached, I think that's probably the culprit and I can work on a fix (we shouldn't pull the entire image).

[samjohnsonCF]

Haha right!? You would think it would be in the massive amount of images being uploaded at the end.

I appreciate your time looking into this. Please let me know if you need anything else from me. Also, just to make sure, that run pulled v1.0.7 of porter.

[samjohnsonCF]

Sorry, I missed your last message there. No, the images are not cached at all. Every time our scripts run its on a net new image via the Azure Pipelines.

[carolynvs]

Thanks for confirming that how we are getting referenced image digests is the cause of that particular performance problem! I've made an issue and will get a fix out soon but it won't be in this week's release.

#2576

[samjohnsonCF]

Oh fantastic! No worries on not getting it out this week. I appreciate the quick response!

[samjohnsonCF]

@carolynvs I wanted to post back here, cause this fix was in the latest version (1.0.13). Just ran a porter publish with it and what was taking 10-12 mins to run, took 1 minute 30 seconds to run! :-)
So amazing, thank you for getting this fixed!

[carolynvs]

HOT DAMN! 🎉 That's great news, thanks for letting me know.