2017 Benchmarking WG Survey

[dshaw]

The Benchmarking Working Group (WG) reached out to the CommComm for help gathering feedback from Node.js Users. nodejs/community-committee#153

The Node.js End User Feedback team has taken on this project with the support of the Node.js Foundation team.

[dshaw]

I edited down the questions, adding options wherever possible. I also added an intro.
https://docs.google.com/document/d/1qiepQ-FkP6uiFjwLanZo5Bqu6DaiPNyARPLo97Ap3Xs/edit?ts=5a2e9441

[dshaw]

@gtewallace The survey is ready to convert to Survey Monkey.

[dshaw]

Update @gtewallace has the survey in Survey Monkey now. There was a minor issue. It should be ready tomorrow.

[dshaw]

The Benchmarking WG Survey is live and ready for input and distribution. Thank you @gtewallace of the Node.js Foundation for setting this up.

https://www.surveymonkey.com/r/NodeBenchmarking

/cc @nodejs/community-committee @nodejs/evangelism

[gtewallace]

☺ @ZibbyKeaton after the holidays, do you mind putting this out on Twitter? Tnx!

[boneskull]

Thanks @dshaw for pointing me here.

I tried taking this survey but immediately found it difficult due to ambiguity in the way the questions were posed.

1. How many Node.js applications does your organization have in production?
2. What version of Node.js do you currently use for production? [select all that apply]
3. If you are not using the 8.x LTS release when are you planning to upgrade to that version?

On 1, I answered "more than 100", but in an organization the size of IBM, I'm unsure if anyone knows what that number actually is or how to get it. 2 and 3 suffer from the same problem, but these are asking about "you" instead of "your organization", which is makes me unsure about what the question really wants to know.

5. What are the top 5 Node modules that you use most often?

What does "use" mean? Does it mean "what are your top 5 favorite modules"? Or "what do I npm install the most?" "What gets deployed most often?" The "correct" answers could be deep dependencies, as well. It's tough to make a guess at any of these without infrastructure in place for tracking it. I don't even know how to answer for my own hobby projects...

Anyway, that's where I dropped out of the survey. 😉

[dshaw]

@boneskull Thank you for taking the time to start the survey and share what blocked you.

If you're coming to this thread and have had similar issues to @boneskull, please +1 the comment above and leave a comment.

/cc @mhdawson (also IBM 😉 ) and @nodejs/benchmarking

[mhdawson]

@boneskull thanks for the feedback, we will try to reflect on how to refine the questions in future surveys. I agree its more complicated to respond on behalf of a large org like IBM. I'd try to narrow the focus to the group you work with as opposed to trying to get it right for IBM as a whole.

In terms of the question on top 5 modules we are trying to get at what are the most important modules to Node.js users as background to frame some of the other questions. For example if you say express versus grunt it gives us a possible hint as to your use case. In addition if X is in everybody's top 5 then we should probably think about what we can do to avoid regressing performance for that module.

In any case we (the benchmarking team) are not data analysts/experts so the questions are a bit rougher than they might be otherwise but I think we are going to get some interesting info and as well as learning how we might better ask in future surveys.

[boneskull]

@boneskull thanks for the feedback, we will try to reflect on how to refine the questions in future surveys. I agree its more complicated to respond on behalf of a large org like IBM. I'd try to narrow the focus to the group you work with as opposed to trying to get it right for IBM as a whole.

Yes, I tried that, but realized I didn't even know those answers. 😉

I'm not a quantitative researcher by trade, but I have worked in the domain for much of the past 10 years. @dshaw @mhdawson I'm happy to help with any future surveys.

[mhdawson]

@boneskull thanks for the offer, we'll get you to help out for the next one.

[bnb]

Links to my shares of the Benchmarking Survey:

Twitter: https://twitter.com/bitandbang/status/956939288024879112

LinkedIn: https://www.linkedin.com/feed/update/urn:li:activity:6362705334787801088

[ZibbyKeaton]

How is this doing on survey responses? Happy to continue to promote it on Node.js handle, just let me know when you are thinking of closing it.

[Knighton910]

/cc @nodejs/community-committee

If you guys could use an additional hand - Im available
Just contact me & we will link up from there.

[bnb]

@ZibbyKeaton We're planning on closing on the 31st 👍

[ZibbyKeaton]

@bnb I will stage a few more tweets and let folks know they need to submit by then.

[dshaw]

The Benchmarking WG survey has now officially ended. https://www.surveymonkey.com/r/NodeBenchmarking

A big thank you to all the teams and individuals that helped support this:
@nodejs/benchmarking @nodejs/user-feedback @nodejs/community-committee @nodejs/foundation @gtewallace