Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

True support for multiple function apps in a monorepo #2521

Open
aaronadamsCA opened this issue Nov 9, 2020 · 29 comments
Open

True support for multiple function apps in a monorepo #2521

aaronadamsCA opened this issue Nov 9, 2020 · 29 comments
Assignees
Milestone

Comments

@aaronadamsCA
Copy link

aaronadamsCA commented Nov 9, 2020

I have a "real" monorepo with multiple workspaces under a single root repository (not a multi-root workspace). This extension doesn't support multiple function apps in a true, single-root monorepo. I think it should.

Current impact is that my team can't really develop and deploy multiple function apps in parallel in a single development container. Now I need to set up CI for our function apps, which is fine, but it's also work I was hoping I wouldn't have to do right now. Deploying from VS Code using this extension is a nice fit with rapid prototyping, and it was a surprise to discover you don't support multiple workspaces in a repository.

@ejizba
Copy link
Contributor

ejizba commented Nov 11, 2020

What is your desired behavior in this situation? Just prompt to select the function app each time?

@ejizba ejizba added the feature label Nov 11, 2020
@aaronadamsCA
Copy link
Author

Maybe another level of the tree? Honestly I'm not sure. I suppose my expectation as I added a couple more function apps was that I'd be able to, say, right-click deploy to each one in the tree.

FYI my use case was webhooks and queue workers; it makes sense for these to be separate function apps in a shared repository, since they share most dependencies (e.g. TypeScript configs and types) but have different runtime requirements (e.g. webhooks need to stay warm whereas queue workers are fine to start cold).

Now that I've set up GitHub Actions to deploy each of my Azure function apps, I'm no longer in need of this extension; it was a great stopgap solution, I just wanted to provide the feedback on what would have kept it useful for us. Thanks!

@ejizba ejizba added this to the backlog candidates milestone Dec 7, 2020
@AzCode-Bot
Copy link
Collaborator

This issue has become stale and is at risk of being closed. The community has 60 days to upvote the issue. If it receives 5 upvotes we will keep it open and take another look. If not, we will close it. To learn more about how we handle issues, please see our documentation.

Happy Coding!

@AzCode-Bot
Copy link
Collaborator

🙁 In the last 60 days, this issue has received less than 5 community upvotes and we closed it. Still a big Thank You to you for taking the time to create it! To learn more about how we handle issues, please see our documentation.

Happy Coding!

@AzCode-Bot AzCode-Bot added the out of scope See https://aka.ms/azcodeissuetriaging label Jul 11, 2021
@aaronadamsCA
Copy link
Author

That's alright, we're switching to Google Cloud Run which offers far better means for running function apps. Still a big Thank You for taking the time to consider making Azure Functions worth using, better luck next time 👋

@lensbart
Copy link

@AzCode-Bot the issue has 5 upvotes now, please reopen; thanks

@ejizba ejizba reopened this Jul 13, 2021
@ejizba ejizba removed the out of scope See https://aka.ms/azcodeissuetriaging label Jul 13, 2021
@ashishapy
Copy link

I came across same problem. I have been trying several hacks to keep mono git repo for multiple Azure function project. It would be good if it supports out of box.

@AzCode-Bot
Copy link
Collaborator

🙂 This feature request received a sufficient number of community upvotes and we moved it to our backlog. To learn more about how we handle feature requests, please see our documentation.

Happy Coding!

@brandon-burciaga
Copy link

Love that this got to the backlog! I am wanting a monorepo as well with durable functions.

@joelybahh
Copy link

Hey, just want to politely bump this thread. I too have recently moved our team's codebase under a mono-repo, while it has its obvious simplifications and joys, it comes with its speedbumps and learning experiences.

The first obvious problem is the deployment of an azure function via the vs code extension relies on a traditional folder structure, therefore, has no greater awareness of any broader local packages outside of the scope of the function app folder. Given most mono-repo environments utilize workspaces, which is a feature in most if not all modern package managers, I do imagine a solution could exist for this that is abstract enough to not require a really niche solution.

I could be missing something here so please let me know if I'm off the ball, but just some controls for multiple function apps, and the ability to define install & build commands would go a long way in helping this broader problem. 😄

@sanderkooger
Copy link

Hey Guys is there a way to make a function app actually work in a monorepo? if so i could use some hints. We have turbo-repo with PNPM to run our monorepo.

@sittingbool
Copy link

sittingbool commented Jan 8, 2023

I am not sure if the desired behavior is still clear. No matter what monorepo tools you are using, the basis for all of them nowadays seems to be npm workspaces and then there are some features added to cache, bump versions etc. But all based on npm workspaces.

So lets get the requirements straight. I can start explaining the setup that I am for, you guys can chip in yours and we see where we can align.

My primary issue is that I want to have a set of function apps that together make my software. In that I want to have shared code and shared libraries.

like this:

repo-root

  • function-apps
    • app1
    • app2
  • libraries
    • domain-models
    • shared-code
  • node_modules // all of them

where package.json would contain

  "workspaces": [
    "function-apps/*",
    "libraries/*"
  ]

My proposals:

  • in npm workspaces I can actually install dependencies on the root level of my repo and npm knows that the packages in the root node_modules can be used in the workspaces. If I want to deploy that I would expect that the azure function will not deploy those dependencies. Because each function app has its own node_modules. I guess thats one thing that should be working. Solution: the functions app knows that its within a monorepo (property in local-settings.json or such) and what the repo root is. It will copy the node_modules from there for deployment. Downside: maybe too many modules will be uploaded that the function does not actually need.
  • also I want to habe some self written shared code like project specific libraries, domain models, etc. So there should be a way to have workspaces that are not an azure function app. but it should be possible to use them inside the function app. That would work in npm workspaces. Again here the issue so far ist only the deployment of the function app.

Note: I usually deploy using the azure cli.

Workarounds (I will try this now and let you know if it works out):

  • put your shared code in a packages folder, keep the apps separate.
  • publish your shared code to a tar (uning npm pack) instead of to npm
  • install that tar to your functions dependencies
  • use lerna or turborepo or similar to create a dependeny on build that rebuilds the tar files first

@sittingbool
Copy link

sittingbool commented Jan 13, 2023

As promised a WORKAROUND to fulfill shared code and multiple azure function apps within one repo, for javascript and typescript:

I am on a Mac, maybe someone could help with a Windows solution of the build script. I can also create one with node. But I think on Windows it should work with Git-Bash where you can run bash-scripts as well.

following folder structure:

my-project/
  apps/
    function-app-a/
      function-a-1/
      function-a-2/
      ...
      package.json
      ...
    function-app-b/
      function-b-1/
      function-b-2/
      ...
      package.json
      ...
  lib/
    out/
    core/
    domain-model/
    whatever-you-need/
    package.json
  build.sh

Please put your eye on:

  • the lib/out
  • the package.json files in each function app
  • the build.sh

The build.sh gets following content:

#!/usr/bin/env bash
# the output of the tgz file
OUT_DIR=out
# the name of your lib (package,json > name)
LIB_NAME=my-project-lib
# the path of your shared code (relative to the root path)
LIB_DIR=lib
# the directory of this file
BASE_DIR=$(dirname "$0")
# the origin path of the script execution (your function app)
ORIGIN_DIR=$(realpath "$PWD")
# the root directory of your project
ROOT_DIR=$(realpath "$PWD/$BASE_DIR")

# go to root and build dependencies as tgz
cd "$ROOT_DIR" || exit
cd "$LIB_DIR" || exit

if [ -d "$OUT_DIR" ]; then rm -Rf $OUT_DIR/*.tgz; fi

mkdir $OUT_DIR;

npm run build || exit
npm pack --pack-destination $OUT_DIR "$@" || exit

cd "$ORIGIN_DIR" || exit
npm i "$LIB_NAME"

Now configure:

  • LIB_NAME = whatever you put in the name property of lib/package.json
    optional:
  • OUT_DIR = lib/out in this example. you can change the folder name if you want to and the set the same dir-name here
  • LIB_DIR = lib or whatever you want to call that folder

make sure build.sh is executable. on unix this would be:

chmod +x build.sh

run the build file. on unix:

./build.sh

ignore the 404 error.

check that it created a tgz-file in lib/out

go to each of your function apps and install the tgz like so:

npm i ../../lib/out/my-project-lib.tgz

why tgz? because you can install npm packages in your lib and they will also be within you function app nodemodules whenever you build.

Last step:

in your function apps (each one) modify your build script in the package.json.
for typescript:

"scripts": {
    "build": "../../build.sh && tsc",

for jsvascript:

"scripts": {
    "build": "../../build.sh",
    "prestart": "npm run build",

Make sure you start your apps with:
npm run start so that the prestart is being executed
or call func start to not rebuild the lib.

I believe the approach to create a local library as a dependency in each function app could also be adjusted to other languages like Java and C# but it will be a while until I would get around to that. Maybe we can create a repo to store templates for this workaround.

@AlexPshul
Copy link

After browsing the web for quite a while in hopes to get a good solution for that, I've ended up creating my own NX workspace plugin for that. You're welcome to use it. :)
https://www.npmjs.com/package/@nxazure/func

@joelybahh
Copy link

Hey Guys is there a way to make a function app actually work in a monorepo? if so I could use some hints. We have turbo-repo with PNPM to run our monorepo.

Idk about PNPM specifically, but for yarn, I had success getting deployments to work via nohoist in the package.json of the function app:

"workspaces": {
    "nohoist": [
        "**"
    ]
}

While nohoist is useful, it does come with drawbacks. The most obvious one is modules will now be duplicated in multiple locations, denying the benefit of hoisting. This solution becomes more problematic when you want to have multiple function apps, so that double up of packages could quickly cascade out, if you only have the one, this might work for your needs, its not optimal, but it is functional (with yarn atleast).

This thread might help around hoisting behavior and pnpm

@joelybahh
Copy link

joelybahh commented May 26, 2023

I have found a workaround to use as a temporary solution (Note the below is quite bespoke to my environment so a pure copy and paste may not work)

path: scipts/azure-deploy.js

// eslint-disable-next-line @typescript-eslint/no-var-requires
const fs = require('fs');

// eslint-disable-next-line @typescript-eslint/no-var-requires
const path = require('path');

const packageName = process.argv[2];  // Get the package name from command-line arguments

// Define the paths to the JSON files
const settingsPath = path.join(__dirname, '..', '.vscode', 'settings.json');
const launchPath = path.join(__dirname, '..', '.vscode', 'launch.json');
const tasksPath = path.join(__dirname, '..', '.vscode', 'tasks.json');

// Read the JSON files
const settings = JSON.parse(fs.readFileSync(settingsPath, 'utf8'));
const launch = JSON.parse(fs.readFileSync(launchPath, 'utf8'));
const tasks = JSON.parse(fs.readFileSync(tasksPath, 'utf8'));

// Modify the JSON data
settings["azureFunctions.projectSubpath"] = path.join('functions', packageName);
settings["azureFunctions.deploySubpath"] = path.join('functions', packageName);

// Modify the tasks.json to reflect the correct package
tasks.tasks.forEach((task) => {
    if (task.options && task.options.cwd) {
        task.options.cwd = "${workspaceFolder}/functions/" + packageName;
    }
});

// Save the JSON data back to the files
fs.writeFileSync(settingsPath, JSON.stringify(settings, null, 4), 'utf8');
fs.writeFileSync(launchPath, JSON.stringify(launch, null, 4), 'utf8');
fs.writeFileSync(tasksPath, JSON.stringify(tasks, null, 4), 'utf8');

// Ready to deploy with the updated paths

Basically just rewriting the paths do where its being deployed from, and making sure I call it properly prior to deploying. Its not ideal, but it allows multiple apps to be deployable from vscode for now.

example in package.json

{
    "scripts": {
        "predeploy": "node ./scripts/function-deploy.js",  
    }
}

yarn predeploy subscriptions updates all the hardcoded paths to be /functions/subscriptions
yarn predeploy api updates all the hardcoded paths to be /functions/api

⚠️ This incurs added risk due to developer error as you could 'accidentally' deploy the wrong function app the wrong place. We just weighed this risk as not to concerning for us right now.

@ravitejaavv
Copy link

Can any one help me on How do we build and deploy multiple go-lang azure functions under single repo code base?

@ingmaramzan
Copy link

Can we have an update or a timeline for when this plan to be available, at least as a preview maybe?

@sudosean
Copy link

sudosean commented May 1, 2024

I am also looking to set up a monorepo with a React UI and Azure Functions. @ejizba would you recommend any one of the posted workarounds while this issue is open? thank you

@GustavoOS
Copy link

Any updates?

@DrPye
Copy link

DrPye commented Sep 4, 2024

4 years later still waiting, do we really have to abandon azure functions just because we use a mono repo?

@sanderkooger
Copy link

4 years later still waiting, do we really have to abandon azure functions just because we use a mono repo?

Pretty much i guess. I migrated all our functions to a new nestjs app. Can run that one anywhere i want. Cheaper too

@DrPye
Copy link

DrPye commented Sep 4, 2024

Ok after some research we have learned this isnt just multiple azure functions, even if you have 1 azure function in a monorepo (turborepo in our case) it will just break deployment. The files are uploaded but none of the endpoints show/ are registered)

We can probably work around this with a deployment pipeline but it is more work for us and this was supposed to be a quick and easy implementation!

@AlexPshul
Copy link

@DrPye if you're open to using NX for managing your monorepo, I've created a plugin that allows you to add azure function apps to your monorepo.
It supports using libs, it supports running functions in parallel and it supports publishing each function independently.
Have a look, maybe it will suit your needs.

https://www.npmjs.com/package/@nxazure/func

@DrPye
Copy link

DrPye commented Sep 5, 2024

@AlexPshul I would love to but we've just finished migrating to turbo and I think id lose my job if I do another refactor ha!

@AndrewDucDanhDo
Copy link

AndrewDucDanhDo commented Sep 16, 2024

hello, after much suffering, wanted to post something because my team, and I got a workaround working today and maybe it might help others if you're looking for a hacky fix.

CONTEXT
We recently shifted to a monorepo using npm workspaces and a Azure Functions (typescript) implementation. Our "monorepo" had our backend directory nested with our azure function implementation. At the top level, we have github workflows with deploy .yml files that followed this microsoft tutorial: https://learn.microsoft.com/en-us/azure/azure-functions/functions-how-to-github-actions?tabs=windows%2Cdotnet&pivots=method-template

We realised that az functions only cared about having the dist build package being at the top level and doesn't like the directory with the az funcs in a nested directory... lol.

OUR NESTED AZ FUNC DIRECTORY

PoC-Template
├── apps
│   ├── frontend
│   ├── backend
│   │   ├── dist (from npm run build)
│   │   ├── src
│   │   ├── package.json
│   │   ├── tsconfig.json 
│   │   └── src
│   │       └── functions
├── package.json (updated this file)
├── tsconfig.json  (added this file)
└── ...

OUR FIX
We pretty much copied our /dist out build package directory that contained our javascript azure functions to the root level. This meant we had to add a tsconfig.json to the top level as well (and added an "exclude" for any directories that you don't want built during the "npm run build") and add to the package.json the ""main": "dist/functions/*.js" assuming your root is in a "../src".

Some mocked code snippets:
tsconfig.json at root

{
  "compilerOptions": {
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "target": "ES2022",
    "outDir": "dist",
    "rootDir": "apps/backend/src",
    "sourceMap": true,
    "strict": true,
    "skipLibCheck": true
  },
  "exclude": [
    "apps/frontend",
    ...
  ]
}

package.json

{
  "name": "",
  "version": "1.0.0",
  "description": "",
  "scripts": {
   ...
  },
  ...
  "main": "dist/functions/*.js"
}

.github/deploy-backend.yml

# Step to build the package in /apps/backend
- name: Build and publish package (Backend)
  shell: bash
  run: |
    echo "Building the package in /apps/backend"
    pushd 'apps/backend'
    npm install --no-audit --no-fund # Install backend-specific packages only
    npm run build # This creates the /dist folder
    popd

# Step to copy /apps/backend/dist to the root directory
- name: Copy dist folder to root
  run: |
    echo "Copying dist folder from /apps/backend to root"
    cp -R apps/backend/dist . # Copy the /dist folder from /apps/backend to the root
    ls -l dist # Verify that dist is now in the root directory

# Step to deploy Azure Functions pointing to the root-level dist folder
- name: "Deploy Azure Functions"
  uses: Azure/functions-action@v1
  id: fa
  with:
    app-name: ${{ env.FUNCTIONS_APP_NAME }}
    package: ${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}
    publish-profile: ${{ secrets.AZURE_FUNCTIONAPP_PUBLISH_PROFILE }}

hope this helps someone (or maybe myself in the future) xx

@DrPye
Copy link

DrPye commented Sep 17, 2024

Hi all, if your using turbo repo (possibly pnpm deploy will do the same trick) I wrote a blog post for sorting this out, no extra tsconfig needed, no moving of files, just a bit of a build script for deployments. Please let me know if you feel steps are missing or need clarification!

https://pyedpiper.pyenet.co.uk/blog/serverless-function-deploy-mono

@tyleralbee
Copy link

tyleralbee commented Nov 5, 2024

UPDATE: I figured this out, leaving here in case others run into same issues

Hi all, if your using turbo repo (possibly pnpm deploy will do the same trick) I wrote a blog post for sorting this out, no extra tsconfig needed, no moving of files, just a bit of a build script for deployments. Please let me know if you feel steps are missing or need clarification!

https://pyedpiper.pyenet.co.uk/blog/serverless-function-deploy-mono

This is incredibly helpful @DrPye - would you mind sharing your tsup.config.ts? Right now I am bundling internal packages like so:

const tsupConfig = defineConfig((options) => {
    return {
        // ...
        noExternal: [/^@my-monorepo\/*/], // Bundle internal packages - results in dynamic require is not supported error
    }
})

But I am running into the esbuild issue outlined here (dynamic require not supported). Are you implementing one of the fixes in that thread or are you bundling internal packages in a different way?

Edit: after adding the dynamic require banner and shims for __dirname it worked, thank you so much again @DrPye

I needed to include the following in my tsup.config.ts:

const tsupConfig = defineConfig((options) => {
    return {
        noExternal: [/^@my-monorepo\/*/],
        banner: {
            js: "import { createRequire } from 'module'; const require = createRequire(import.meta.url);",
        },
        shims: true,
    }
})

@DrPye
Copy link

DrPye commented Nov 5, 2024

@tyleralbee Im really happy it helped, that was alot of smashing my head into a wall to get it working.

Im glad your sorted your problem but to answer your question, we simply use noExternal and cjs as our format. We havent come across the issue you had.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests