finish event never fires

[jousi592]

I am trying to upload some images to my cloud storage.
I am using form data to send the data down to the cloud function. The data arrives correctly, but the Busboy process never finishes processing it and the function timeouts after 60s.

Here is the implementation:

const firebase = require("firebase");
const Busboy = require("busboy");
require("firebase/storage");
const firebase_db = firebase_admin.database();
const firebase_storage = firebase.storage();


module.exports = (req, res) => {
  const busboy = new Busboy({headers: req.headers});
  const fields = {};
  const files = {};

  busboy.on('field', (fieldname, val) => {
    fields[fieldname] = val;
  });

  busboy.on('file', (fieldname, file, filename) => {
    files[fieldname] = file;
  });

  busboy.on('finish', async () => { // ----> Function never fires
    try {
      let project_id = await firebase_db.ref("/project_config/project_id").once("value")
      project_id = project_id.val()
   
      if (!project_id) return res.send({ message: "Project ID not found" })
      const new_image_storage_ref = firebase_storage.ref().child(`${project_id}/${fields.route}`)
      const snapshot = await new_image_storage_ref.put(files.image)
      const download_url = await snapshot.ref.getDownloadURL()
 
      res.send(download_url)
    } catch (error) {
      console.log("----- Error during saving content data ", error)
      res.send({ message: "Error during saving your image, please re-load page and try again or contact support." })
    }
  });
}

Do you have any idea what might be causing it?

[charmander]

You don’t seem to be writing anything into the busboy instance.

req.pipe(busboy);

[jousi592]

@charmander
I dont completly grasp the logic behind the busboy library yet, I am solely looking for something to help me get files from FE to BE. It seems like every middleware library for consuming form data is a little more difficult to setup than I expected.

Are you saying I need to add req.pipe(busboy); after each file call back?
Like:

busboy.on('file', (fieldname, file, filename) => {
    files[fieldname] = file;
    req.pipe(busboy);
});

[mscdex]

You would add that line in the parent scope, after you've set up all of your event handlers.

Another thing to be aware of is that if you're running this as a "cloud function" (e.g. Amazon Lambda) it could be that the form data is already buffered and stored somewhere, so piping to your busboy instance won't do anything in that case. For that you'd just have to find where it's being stored (e.g. on req) and write that manually to the busboy instance (e.g. busboy.end(req.body)).

[jousi592]

Thanks for the comment. Its starting to make a little more sence.

I have added:

  busboy.end(req.body);
  req.pipe(busboy);

Since the in example here https://cloud.google.com/functions/docs/writing/http#multipart_data they also add it.

However the function keep timing out, because it still isnt getting into the finish state.

The code now looks like:

const firebase_admin = require("firebase-admin");
const firebase = require("firebase");
const Busboy = require("busboy");
require("firebase/storage");
const firebase_db = firebase_admin.database();
const firebase_storage = firebase.storage();


module.exports = (req, res) => {
  const busboy = new Busboy({headers: req.headers});
  const fields = {};
  const files = {};

  // This code will process each non-file field in the form.
  busboy.on('field', (fieldname, val) => {
    fields[fieldname] = val;
  });

  // This code will process each file uploaded.
  busboy.on('file', (fieldname, file, filename) => {
    files[fieldname] = file;
  });

  busboy.on('finish', async () => {
    console.log(files)
    console.log(fields)
    try {
      let project_id = await firebase_db.ref("/project_config/project_id").once("value")
      project_id = project_id.val()
   
      if (!project_id) return res.send({ message: "Project ID not found" })
      const new_image_storage_ref = firebase_storage.ref().child(`${project_id}/${fields.route}`)
      const snapshot = await new_image_storage_ref.put(files.image)
      const download_url = await snapshot.ref.getDownloadURL()
  
      console.log(download_url)
      res.send(download_url)
    } catch (error) {
      console.log("----- Error during saving content data ", error)
      res.send({ message: "Error during saving your image, please re-load page and try again or contact support." })
    }
  });

  busboy.end(req.body);
  req.pipe(busboy);
}

[mscdex]

If you're using the Google Cloud, then if you look closely at the example you linked to they are just using:

busboy.end(req.rawBody);

so you'd use that instead of:

busboy.end(req.body);
req.pipe(busboy);

[jousi592]

I have tried that also, but the function still times out :/

Function execution took 60006 ms, finished with status: 'timeout'

Not sure what else might be causing it, since I have just slightly modified the example mentioned before...

[jousi592]

Anyway I will keep trying and let you know once I figure it out 👍

[kersten]

Got the same problem. But this only happens if I have an file field in the request. If I only use other fields with enctype="multipart/form-data" it works like a charm.

[peterhuffer]

The finish event never fires because:

Note: if you listen for this event, you should always handle the stream no matter if you care about the file contents or not (e.g. you can simply just do stream.resume(); if you want to discard the contents), otherwise the 'finish' event will never fire on the Busboy instance

I have this same issue where I want to accumulate my form data and then perform an operation with all of it, but I don't want to handle the file at the time of the file event. I've moved my fields to be headers in the mean time, and handle the file when the event is received, but I'd be curious if there is a way to do this all within the form data.

[pizzarob]

I am having a similar issue. I am trying to write to a path and it does in fact write some of the data, but not all of it and the finish event is never emitted. I've tried to add other events to the streams to figure out was is going on but can't seem to figure it out. This happens when running in a docker container on ECS. It seems to work fine locally.

export default (req) =>
  new Promise((resolve, reject) => {
    const busboy = new Busboy({ headers: req.headers });
    const fieldNameCache = {};
    const output = {
      fields: [],
      files: [],
    };

    busboy.on(
      "file",
      async (fieldName, fileStream, fileName, encoding, mimeType) => {
        if (INVALID_MIME_TYPE.includes(mimeType)) {
          return reject({ message: "mime type not allowed", data: mimeType });
        }
        if (!VALID_FIELD_NAMES.includes(fieldName)) {
          return reject(new Error(`Erroneous value ${fieldName}`));
        }

        const folder = `${uuid()}`;

        await fs.promises.mkdir(folder, { recursive: true });

        const filePath = `${folder}/${fileName}`;

        output.files.push({
          fieldName,
          fileName,
          path: filePath,
          folder,
          mimeType,
        });

        function cb(e) {
          console.log("ENDING", e);
        }

        const ws = fs
          .createWriteStream(filePath)
          .on("error", cb)
          .on("finish", cb)
          .on("end", cb);

        fileStream
          .on("error", cb)
          .on("finish", cb)
          .on("end", cb);

        pipeline(fileStream, ws, (err) => {
          if (err) {
            console.log("Pipeline failed", err);
          } else {
            console.log("Pipleline complete");
          }
        });
      }
    );

    busboy.on(
      "field",
      (
        fieldName,
        value,
        fieldnameTruncated,
        valTruncated,
        encoding,
        mimeType
      ) => {
        if (!VALID_FIELD_NAMES.includes(fieldName)) {
          return reject(new Error(`Erroneous value ${fieldName}`));
        }
        if (fieldNameCache[fieldName]) {
          return reject(new Error(`Duplicate field name ${fieldName}`));
        }
        fieldNameCache[fieldName] = true;
        output.fields.push({ fieldName, value, mimeType });
      }
    );

    busboy.on("finish", async () => {
      // Never happens
      resolve(output);
    });

    // Added these to try to find errors, but there are none
    req.pipe(busboy).on("error", (e) => {
      console.log("error", e);
    });
    req.on("aborted", (e) => {
      console.log(`Request aborted`, e);
    });
    busboy.on("error", (e) => {
      console.log("busboy error");
      console.log(e);
    });
  });

[mscdex]

If you can provide a minimal reproduction against the current master branch, let me know.

[thecuvii]

Node version matters. Works fine with node v14, but in some case with v16, the problem occurs.

[gongshoudao]

If you're using the Google Cloud, then if you look closely at the example you linked to they are just using:

busboy.end(req.rawBody);

so you'd use that instead of:

busboy.end(req.body);
req.pipe(busboy);

you solved my problem!!!
Thanks