Deployment is *very* slow (nodeJS with publish profile)

[0gust1]

I'm using a workflow file adapted from "build and deploy a Node.js Web app to Azure using publish profile".

(side remark) : I had issues with symbolic links too (see #54 (comment))

Upon analysis, it seems that the whole copy of the node_modules folders to the webroot of the webapp takes ages (10-12 minutes for the Azure Webapp deploy step !) It's very inefficient to process and send many little files over network.

=> Shouldn't the final npm install (and npm run build – I'm using typescript) commands be executed on the target machine instead of a github action container ?

IMHO, the workflow should be like this for the "nodeJS with publish profile" scenario :

• the github action should only send source files to the targeted runtime machine
• npm install, npm run build and npm start should be done on the targeted runtime machine
• It's up to the developer to ensure that is app is building correctly before calling azure/webapps-deploy

Related questions :

• Also how this Github action is related to Kudu ?
• Does it use Kudu behind the scenes ? How ? (bare copy ? zip deploy ?)
• If yes, are Kudu capabilities useable ? I.E. .deployment file, deployment hooks, etc.
  references : https://github.com/projectkudu/kudu/wiki/Customizing-deployments and https://github.com/projectkudu/kudu/wiki/Deployment-hooks

[sfabriece]

My deployments are taking 10 minutes and it is slowing us down. How can we improve the situation?

[rachael-ross]

Our deployments are taking over 30 mins!

[AmrutaKawade]

@0gust1 Sorry for delayed response.
action uses kudu zipdeploy api for deployment with zip file input
https://github.com/projectkudu/kudu/wiki/REST-API#zip-deployment

[sfabriece]

Any way to speed it up @AmrutaKawade ?

[AmrutaKawade]

@sfabriece custom deloyment scripts should work.
Do not include node_modules in zip. Instead add .deployment and deploy.cmd / deploy.sh files at root of your repo with npm install command. it will execute npm install on target machine

some examples
https://docs.microsoft.com/en-us/azure/app-service/configure-language-nodejs?pivots=platform-windows#run-gruntbowergulp
https://github.com/projectkudu/kudu/wiki/Deployment-hooks#easiest-way-to-get-started-with-a-custom-deployment-script
https://github.com/projectkudu/kudu/wiki/Custom-Deployment-Script

[0gust1]

Thanks @AmrutaKawade !

Ideally, hints could be given in documentations and workflow files (for example: https://github.com/Azure/actions-workflow-samples/blob/master/AppService/node.js-webapp-on-azure.yml), as every nodeJS app have the same behavior on this, and people using the actions-workflow-samples will encounter the same issue.

[sfabriece]

@AmrutaKawade as far as I can tell, the deploy scripts control how the app is deployment once on the target machine. How do I control what ends up in the zip file?

[AmrutaKawade]

@sfabriece you can provide directly zip as a input to action or folder location which will be zipped in action

- name: 'Run Azure webapp deploy action using publish profile credentials'
      uses: azure/webapps-deploy@v2
      with:
        app-name: <app-name>
        publish-profile: ${{ secrets.azureWebAppPublishProfile }}
        package: <provide path of your folder or zip file>

[kaverma]

@sfabriece you can provide directly zip as a input to action or folder location which will be zipped in action

- name: 'Run Azure webapp deploy action using publish profile credentials'
      uses: azure/webapps-deploy@v2
      with:
        app-name: <app-name>
        publish-profile: ${{ secrets.azureWebAppPublishProfile }}
        package: <provide path of your folder or zip file>

Closing the issue as the question has been addressed and for performance we already have npm install in the sample workflow which optimises the time

[ywaserblat-excelforce]

Had the same issue, tried with the suggested solution of creating a custom deploy (btw, custom deploy for a standard task?)
but got the following error:
/opt/Kudu/Scripts/starter.sh: line 2: exec: deploy.cmd: not found
/opt/Kudu/Scripts/starter.sh: line 2: exec: deploy.cmd: not found\n/opt/Kudu/Scripts/starter.sh deploy.cmd
so far I wasn't able to solve this issue, so I am falling back to the default inefficient procedure.
I do believe that @0gust1's suggestion is the right approach:
the github action should only send source files to the targeted runtime machine
npm install, npm run build and npm start should be done on the targeted runtime machine
It's up to the developer to ensure that is app is building correctly before calling azure/webapps-deploy
this should be the default w/o any customization needed as it is the standard behavior for node.js apps

[JonCognioDigital]

Could somebody please explain how this can be solved?

I am using the latest version of the script automatically installed by the Azure portal when using the deployment center. It seems to be running npm install on Github then uploading every file individually. Did anyone find a way to (A) zip them up and unzip them on the target webapp or (B) just do the npm install on the target webapp (although that would stop you from doing any other steps using github actions like testing)?

[ghost]

Seconding this one I am having the same problem and the solutions provided above do not seem to be working. I am using the standard file automatically generated by the Azure portal as well.

[kaverma]

@BALAGA-GAYATRI , can you please help here with the above queries? If the solution isn't working, can you please reopen the issue?

[abacaj]

Takes 30+ min for my deploy step on Azure, just < 5min on DigitalOcean app service.
Sucks that microsoft hasn't provided any guidance here.

[MoatezNG]

@stefanoz-cloud sure

name: Build and deploy Node.js app to Azure Web App

on:
  push:
    branches:
      - main
  workflow_dispatch:

env:
  AZURE_WEBAPP_PACKAGE_PATH: '.' # set this to the path to your web app project, defaults to the repository root
  NODE_VERSION: '16.x' # set this to the node version to use

permissions:
  contents: read

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: npm install, build
        run: |
          npm install
          npm run build --if-present
      - name: Zip artifact for deployment
        run: zip release.zip ./* -r
      - name: Upload artifact for deployment job
        uses: actions/upload-artifact@v3
        with:
          name: node-app
          path: release.zip

  deploy:
    permissions:
      contents: none
    runs-on: ubuntu-latest
    needs: build
    environment:
      name: 'Development'
      url: ${{ steps.deploy-to-webapp.outputs.webapp-url }}
    steps:
      - name: Download artifact from build job
        uses: actions/download-artifact@v3
        with:
          name: node-app
      - name: unzip artifact for deployment
        run: unzip release.zip
      - name: 'Deploy to Azure WebApp'
        id: 'id'
        uses: azure/webapps-deploy@v2
        with:
          app-name: 'NAME'
          slot-name: 'SLOT'
          publish-profile: 'SECRET'
          package: ${{ env.AZURE_WEBAPP_PACKAGE_PATH }}

[sven5]

The deployment is slow when there are many files to deploy.
When uploading a Zip file it's been extracted on the target App service (Linux). This takes a while. I think this is just a serial operation.
We're deploying about ~10.000 static web files (html, js, css) to our App service and it takes ~20min. It's an output of a Gatsby build.

The only way to speed up deployment time is to decrease the artifact size.

[abacaj]

The deployment is slow when there are many files to deploy. When uploading a Zip file it's been extracted on the target App service (Linux). This takes a while. I think this is just a serial operation. We're deploying about ~10.000 static web files (html, js, css) to our App service and it takes ~20min. It's an output of a Gatsby build.

The only way to speed up deployment time is to decrease the artifact size.

Not sure how other platforms handle it, but I’ve only seen this delay deploying to Azure.

Vercel, digitalocean and a few others do not have this issue, same codebase.

[sven5]

I could imagine that the deployment process, especially the Zip unpacking on the App Service, would be faster when running in parallel.

Another option would be using Docker images. You create your custom image and upload it to a container registry. After that the image is applied on the App Service. Perhaps this is faster than deploying a Zip file.

[psierak]

@MoatezNG your answer worked for me. Brought my build and deploy from 2.5 hrs to 20 min. Thank you

[tefkah]

Something to add:
While the deployment process takes about 30 mins, after which it crashes with the mysterious "Refer to logs" error, I do already see the changes reflected after about 5ish minutes. No idea why it doesn't just complete, kind of a waste of energy, but at the very least iteration time isn't twice per hour.

[kumaraksh1]

Similar issue (#229) is going on, we will be fixing this soon.
Closing this issue for now and the progress can be tracked from the mentioned issue.

[tefkah]

great to hear, thank you!

[hharb3]

In upload artifacts section include only build/ directory

• name: Upload artifact for deployment job

    uses: actions/upload-artifact@v2
    with:
      name: node-app
      path: build/
  

[sabeeshvk]

Making the zip file from GitHub build process and doing unzip on the Azure side seems to work much better.
I used the steps as mentioned by MoatezNG above.

[mavyfaby]

I significantly reduce my deployment time from 11 minutes to 1 minute by zipping after npm build. Thanks to @MoatezNG.

Before:

After:

[lhammarstrom]

I was using Azure Devops to build and package my NextJS app with fully acceptable build times (~1 minute with caching).
Before zipping I removed all files except the ones needed (including not deleting node_modules). I agree that npm install should be run on the target machine but I never managed to get it working using .deployment and deploy.sh files, so I opted for the easy route and adding node_modules to the bundle. This had no effect on deploy times which were still horrendous. It could take 20 minutes or more using the Zip deploy.

However when I changed the WEBSITE_RUN_FROM_PACKAGE application setting to 1 my deploys took 20 seconds. So for everyone banging their heads: go to configuration and set:

WEBSITE_RUN_FROM_PACKAGE=1

[0gust1]

A small update (as I originally created the issue...)

• At work, we're no longer subject to this problem, as we switched to a Docker-managed deployment model for our project. In this model, the update is much quicker and simpler (we needed containerization for native/OS-dependencies management).
• In some cases, the WEBSITE_RUN_FROM_PACKAGE=1 solution is not ideal / won't work: if you rely on mounting in you app folder some folders from Azure Storages. Try it, but your mileage may vary.

[Zhaph]

Website_Run_From_Package is fine if your app can run from inside the zip file. If you're using Incremental Static Regeneration to build new pages on the fly from data stores, etc. then this won't work for you, and you'll need to unpack the Zip file at one end or the other.

[lhammarstrom]

Website_Run_From_Package is fine if your app can run from inside the zip file. If you're using Incremental Static Regeneration to build new pages on the fly from data stores, etc. then this won't work for you, and you'll need to unpack the Zip file at one end or the other.

Well it continues to be an issue to deploy anything other than .NET on Azure

[Zhaph]

Indeed. Just switched to the AzureWebApp task, increased the memory with NODE_OPTIONS: --max_old_space_size=4096 and now it's failing due to path lengths when trying to unpack and sort the files out:

Cleaning up temp folders from previous zip deployments and extracting pushed zip file C:\local\Temp\zipdeploy\xeqz5oqd.zip (205.23 MB) to C:\local\Temp\zipdeploy\extracted
The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.

Not sure who's really at fault here, Windows with its legacy path requirements, or node and npm with it's 90k+ JS files required to run a website...

[IliaZolas]

Hey guys, I wanted to validate @MoatezNG answer as successful.

After you connect your Azure resource to your Github account, you will have a secret and yaml file appear in your respective repo.

You need to access that .yml file and modify it according to @MoatezNG instructions though it might differ for some people. Also remember that indentation is important and will result in errors if done incorrectly.

Here is my new yml file which resulted in deployment in under 4 minutes (remember most of this is automatically generated so you just need to add specific parts):

name: Build and deploy Node.js app to Azure Web App - <your-app-name-is automatically generated here>

on:
  push:
    branches:
      - main
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2

      - name: Set up Node.js version
        uses: actions/setup-node@v1
        with:
          node-version: '16.x'

      - name: npm install, build, and test
        run: |
          npm install
          CI=false npm run build --if-present // I did this to skip warnings as errors, remove if you want
          CI=false npm run test --if-present // I did this to skip warnings as errors, remove if you want
          
      - name: Zip artifact for deployment
        run: zip release.zip ./*

      - name: Upload artifact for deployment job
        uses: actions/upload-artifact@v2
        with:
          name: release.zip
          path: release.zip

  deploy:
    runs-on: ubuntu-latest
    needs: build
    environment:
      name: 'Production'
      url: ${{ steps.deploy-to-webapp.outputs.webapp-url }}

    steps:
      - name: Download artifact from build job
        uses: actions/download-artifact@v2
        with:
          name: release.zip
      - name: unzip artifact for deployment
        run: unzip release.zip

      - name: 'Deploy to Azure Web App'
        id: deploy-to-webapp
        uses: azure/webapps-deploy@v2
        with:
          app-name: '<your-app-name-is automatically generated here>'
          slot-name: 'Production'
          publish-profile: ${{ secrets.AZUREAPPSERVICE... is automatically generated here> }}
          package: .

P.S. if you're publishing a react app you need to include your build file in your repo (npm run build will generate this file on your machine) ✌️

[starthecode]

I was using Azure Devops to build and package my NextJS app with fully acceptable build times (~1 minute with caching). Before zipping I removed all files except the ones needed (including not deleting node_modules). I agree that npm install should be run on the target machine but I never managed to get it working using .deployment and deploy.sh files, so I opted for the easy route and adding node_modules to the bundle. This had no effect on deploy times which were still horrendous. It could take 20 minutes or more using the Zip deploy.

However when I changed the WEBSITE_RUN_FROM_PACKAGE application setting to 1 my deploys took 20 seconds. So for everyone banging their heads: go to configuration and set:

WEBSITE_RUN_FROM_PACKAGE=1

this solve the delay to deployment but adding this, breaking application and getting this error - code: 'MODULE_NOT_FOUND',

[sgollapudi77]

Hi Everyone, we've made changes to the workflow file created by default from the portal. Also, for others who are already using the workflow file you can add 2 steps as mentioned in this comment. Please reply on the other issue #229 if you're still facing the issue.

[braedonwatkins]

commenting to keep reference. having a similar situation of ~15min build time even with a zipped directory.

i tried excluding the node_modules folder from the zip and then doing npm install in the startup command. i verified it was targeting the correct directory and that it was correctly installing modules but the apiapp just says there was an error.

may try using a bash script in the /home directory, as some others in the thread suggest, but i cant see why that would work as opposed to a startup command.
maybe the order of operations is such that the CI/CD does its build and runs the api.js file then runs the startup command? but still when the api fails wouldnt it retry and then work? why does it continue to fail?

[MrSharpp]

I was using Azure Devops to build and package my NextJS app with fully acceptable build times (~1 minute with caching). Before zipping I removed all files except the ones needed (including not deleting node_modules). I agree that npm install should be run on the target machine but I never managed to get it working using .deployment and deploy.sh files, so I opted for the easy route and adding node_modules to the bundle. This had no effect on deploy times which were still horrendous. It could take 20 minutes or more using the Zip deploy.
However when I changed the WEBSITE_RUN_FROM_PACKAGE application setting to 1 my deploys took 20 seconds. So for everyone banging their heads: go to configuration and set:
WEBSITE_RUN_FROM_PACKAGE=1

this solve the delay to deployment but adding this, breaking application and getting this error - code: 'MODULE_NOT_FOUND',

fixed this issue by changing the zip command to

 - name: Zip artifact for deployment
        run: zip release.zip ./* .next -qr

after it, the app runs perfect and deploying is fast too

[jasperfirecai2]

going by the deployment logs in azure, a lot of time taken is azure not being very smart about handling zip deploys, and the workflow staying connected during that. WHen you deploy a zip file it puts it in a /tmp directory, then extracts it to /tmp/zipdeploy/extracted, THEN copies the files over to your destination. it would be a whole lot faster if it just extracted into the target directly

[ogysha]

Here is the workflow which runs fast.

name: FE on Azure App Service CI/CD

on:
  workflow_dispatch:
  push:
    branches:
      - main
      - develop
  pull_request:
    types: [opened, synchronize, reopened, closed]
    branches:
      - main
      - develop

env:
  AZURE_WEBAPP_NAME: fe-app-dev
  AZURE_WEBAPP_PACKAGE_PATH: build
  NODE_VERSION: 20.x

permissions:
  contents: read

jobs:
  build-and-deploy:
    name: Build and deploy
    runs-on: ubuntu-latest
    env:
      NODE_ENV: production
      VITE_MY_VAR: ${{ secrets.MY_VAR }}
    environment:
      name: dev
      url: ${{ steps.deploy-to-webapp.outputs.webapp-url }}
    steps:
      - uses: actions/checkout@v4

      - name: Set up Node.js version
        uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}

      - name: yarn install and build
        run: |
          yarn install --frozen-lockfile --production=false
          yarn build

      - name: Deploy to Azure Web App
        id: deploy-to-webapp
        uses: azure/webapps-deploy@v3
        with:
          app-name: ${{ env.AZURE_WEBAPP_NAME }}
          publish-profile: ${{ secrets.AZURE_WEB_APP_PUBLISH_PROFILE }}
          package: ${{ env.AZURE_WEBAPP_PACKAGE_PATH }}

[copy-ninja1]

only zip and deploy what you need.
Only zip the node_modules if there is a change in the package.json

      - name: Build application
        run: |
          npm run build --if-present

      - name: Zip artifact for deployment
        run: |
          if [ "$PACKAGE_CHANGED" = "true" ]; then
            zip -r release.zip dist nest-cli.json package.json tsconfig.json tsconfig.build.json node_modules
          else
            zip -r release.zip dist nest-cli.json package.json tsconfig.json tsconfig.build.json
          fi

on:
  push:
    branches:
      - dev
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Set up Node.js version
        uses: actions/setup-node@v3
        with:
          node-version: '20.x'

      - name: Check if package.json has changed
        id: package-check
        run: |
          if git diff --quiet HEAD~1 HEAD -- package.json; then
            echo "package_changed=false" >> $GITHUB_ENV
          else
            echo "package_changed=true" >> $GITHUB_ENV
          fi

      - name: Install dependencies if package.json has changed
        if: env.package_changed == 'true'
        run: |
          npm install --legacy-peer-deps

      - name: Build application
        run: |
          npm run build --if-present

      - name: Zip artifact for deployment
        run: |
          if [ "$PACKAGE_CHANGED" = "true" ]; then
            zip -r release.zip dist nest-cli.json package.json tsconfig.json tsconfig.build.json node_modules
          else
            zip -r release.zip dist nest-cli.json package.json tsconfig.json tsconfig.build.json
          fi

      - name: Upload artifact for deployment job
        uses: actions/upload-artifact@v4
        with:
          name: node-app
          path: release.zip

  deploy:
    runs-on: ubuntu-latest
    needs: build
    environment:
      name: 'dev'
      url: ${{ steps.deploy-to-webapp.outputs.webapp-url }}

    steps:
      - name: Download artifact from build job
        uses: actions/download-artifact@v4
        with:
          name: node-app

      - name: Unzip artifact for deployment
        run: unzip release.zip

      - name: 'Deploy to Azure Web App'
        id: deploy-to-webapp
        uses: azure/webapps-deploy@v3
        with:
          app-name: 'saltechfreshapp'
          slot-name: 'v2-prd'
          package: .
          publish-profile: ${{ secrets.AZUREAPPSERVICE_PUBLISHPROFILE }}

[copy-ninja1]

@sfabriece custom deloyment scripts should work. Do not include node_modules in zip. Instead add .deployment and deploy.cmd / deploy.sh files at root of your repo with npm install command. it will execute npm install on target machine

some examples https://docs.microsoft.com/en-us/azure/app-service/configure-language-nodejs?pivots=platform-windows#run-gruntbowergulp https://github.com/projectkudu/kudu/wiki/Deployment-hooks#easiest-way-to-get-started-with-a-custom-deployment-script https://github.com/projectkudu/kudu/wiki/Custom-Deployment-Script

Or you can deploy include the node_modules only when there is a change to the package.json