Survey questions for end users to decide what to benchmark

[mhdawson]

Generate list of questions, then work with community committee to see if we can send out survey from Node.js Foundation/community.

@bmeurer will come up with some initial ideas.

[bmeurer]

You got the wrong handle.

[mhdawson]

@bmeurer sorry about that.

[bmeurer]

Here's a preliminary list of bullet points:

These should be text entry to leave the answers open:

• What is your primary use case for Node? (Web Developer Tooling / Standalone Servers / Cloud Services / Others)

• Do you have infrastructure in place for tracking performance on your applications.

• What kind of language dialect do you use? (ES5 / ES6 / ESNext whatever supported by most recent Node version / TypeScript / Other)

• Do you run optimization passes on your JavaScript before you deploy it to Node? (Minification / Bundling / Obfuscation / Other)

• Where do you report performance regressions if you see them when you upgrade Node.js versions

• What's your preferred way to write asynchronous code? (Callbacks / Promises / async/await / Other)

• Which Node.js feature do you believe affects your performance the most.

• What tools do you use to investigate performance issues ? What command line options do you use ?

• Do you have use cases were you cannot use Node.js because it is single threaded.

• Would a worker model were additional threads are available but data is exchanged as messages or shared array buffers be useful for your applications.

• Is startup time important to your use case ?

• Do garbage collector or optimizer pauses cause problems for your application ?

• If you could choose which JavaScript language/Node.js feature should be optimized next, what would that be? Using the latest Node release as the basis.

• If you had to choose one, would you want the Node.js community to improve CPU or Memory efficiency ?

• What are the top 5 Node modules that you use most often?

Mostly WIP at this point, and targeted towards figuring out what JavaScript focused benchmarks are needed. That doesn't really say a lot about the other performance aspects of Node.

[bmeurer]

There was already a survey run earlier this year, which might be related, results were published here.

As mentioned in the meeting, these points above are highly focused on the underlying JavaScript engine. It would be interesting if folks with experience/interest in other areas of Node could add questions related to that to the list.

[mcollina]

We are missing a good framework in the ecosystem to microbenchmark I/O operations with statistic significance. Something similar to what we have in core, but something we can use to measure promises, callbacks, database drivers. I had several shot at this, none of them was great.

A typical question I get is: "how do async/await stack up against callbacks".

Some folks (including myself) have been putting some time in https://github.com/fastify/benchmarks. This is not comprehensive, but the various frameworks do more or less the same things.

[mhdawson]

What is your primary use case for Node? (Web Developer Tooling / Standalone Servers / Cloud Services / Others)
What kind of language dialect do you use? (ES5 / ES6 / ESNext whatever supported by most recent Node version / TypeScript / Other)
Do you run optimization passes on your JavaScript before you deploy it to Node? (Minification / Bundling / Obfuscation / Other)
What's your preferred way to write asynchronous code? (Callbacks / Promises / async/await / Other)
What are the top 5 Node modules that you use most often?
If you could choose which JavaScript language/Node.js feature should be optimized next, what would that be? Using the latest Node release as the basis.

[mhdawson]

Opened request for help sending out survey nodejs/community-committee#153

[mhdawson]

Survey is live here: https://www.surveymonkey.com/r/NodeBenchmarking

[mhdawson]

We have a dump of the responses so far. I'm not going to post them here as I believe we should wait for the poll to close before we make the results public in order to avoid influencing the results. If members of the @nodejs/benchmarking team would like a copy just let me know and I'll send them on through email.

[bmeurer]

@mhdawson Can you send me a copy of the responses so far?

[mhdawson]

@bmeurer I realized some data might have to be stripped out (for example any identifying info) and have just asked the question I should have asked before which is if any of the data is sensitive. Of course we'll need to share with benchmarking team but I want to agree with the Foundation contact doing the survey how/what we should share. It may take me a bit more time to do that.

[mhdawson]

Closing since the survey is complete and we are just waiting for the final data from the user-feedback team.