Added initial event loop copy #373

TheHollidayInn · 2019-04-09T00:23:45Z

Here is the initial rough draft for #294

goldbergyoni · 2019-04-10T10:01:14Z

@TheHollidayInn I will review this in the next few days. Looks like a great start!

goldbergyoni · 2019-04-10T10:01:28Z

/remind me in 4 days

TheHollidayInn · 2019-05-15T17:53:23Z

@i0natan and updates on this?

goldbergyoni · 2019-05-24T10:43:37Z

@TheHollidayInn Sorry for the delay, I'm just struggling to jungle so many balls :)

If works for you, I can prioritize this bullet and run now a simulntanous discussion to finalize it next week.

In overall, I believe we should separate this into 3 bullets, sorted by priority:

Avoiding CPU heavy tasks - "Other CPU intensive tasks, such as DNS Lookup or Crypto will be offload to lubuv, take up other threads and starve new CPU intensive tasks"
Monitor the event loop
Avoid sync operations - "TLDR; Avoid using syncronous and other Event Loop blocking functions"

TheHollidayInn · 2019-05-27T16:53:11Z

No problem! I'm happy to split/work on the three

goldbergyoni · 2019-05-28T17:57:31Z

@TheHollidayInn Cool, so here's my proposed TOC for 1':

[TLDR Basics] The Node.js userland is single-threaded, keeping the CPU busy might put ~2000 concurrent users in idle.
[TLDR spice it up a little] Yet, it's important to understand that most reasonable CPU operations, including simple cryptography and basic loops, barely affects the end user latency. One has iterate over arrays of million items to generate a noticeable block. Mitigating this demands no more than a simple monitor and developer awareness
[Otherwise] If only one request is doing some crazy computations (e.g. 100 crypto methods per request), all the other 1000 will wait patiently for that one to complete

[One paragraph]

Two lines about the event loop model - our app code is executed using a single thread (as opposed to the IO calls that go outside), this is great for minimizing context switching but assume that all behave nicely as they share the same...

[Examples & code]

Diagram of the event loop that illustrates the block
7 lines code example that shows loop of crypto and a setTimeout that is called-back after 5 seconds instead of 2 (settime is set to 2000 ms but it will never called before the crypto part ends)
Some nice quote
Benchmark! I have this ready, Express route with 1...1000 loop, if I approach it using 1000 concurrent users, the loop add 3ms latency to the end-user (negelectible)

What do you think?

TheHollidayInn · 2019-05-28T19:10:41Z

Hmm. Rethinking this and I might be missing how to separate all three. The issue with using CPU intensive tasks is that they are sync and block the event loop. I think all bullet points might be the same.

Taking a look at your copy now. Also, have you had a chance to read through my copy in this PR?

goldbergyoni · 2019-06-03T11:43:40Z

@TheHollidayInn

Yes, I read your copy which was good but felt like we mix concerns, hence we need to separate,
I don't see how the 3 are the same.

Avoid CPU intensive tasks - this issues is related to code that is CPU intensive like crypto, see the proposed TOC
Is about monitoring it...
sync operations - When you use fs.readfilesync then there's no CPU work here at all, it just blocks your loop until the IO work is over. That's a very specific way of blocking the loop and hurting performance. Very few people do that

Thoughts?

TheHollidayInn · 2019-06-05T02:46:36Z

Ah okay, after reading everything, I am seeing what you mean.

I think where I am getting confused is that both IO and CPU operations cause the same problem which is blocking the loop. And both are run on worker threads: https://nodejs.org/en/docs/guides/dont-block-the-event-loop/#what-code-runs-on-the-worker-pool

So in general the idea is to watch for any loop blockers - dns, file io, crypto, encryption, etc.

But there are quite a few to watch for. I'd agree to separate them up for sure. I'll go ahead and write those up :D

goldbergyoni · 2019-06-06T14:13:18Z

@TheHollidayInn

Sounds good, I suggest starting with 1 bullet, see my proposed TOC for the 1st one. Does this TOC resonate with you?

Minor technical remark:

"And both are run on worker threads"

Most CPU intensive tasks won't run on worker thread, any user code except some Crypto/DNS libraries will get executed within the main thread. This article refers to the Node core code, not to the user-land code

TheHollidayInn · 2019-06-06T16:31:58Z

@i0natan Yep! Works for me :D

I'm not sure I follow the difference between your distinction of node core and user-land. Crypto is considered CPI intensive tasks. DNS is IO. Both go to the worker pool. The section I linked refers to apis you can use in node modules.

However, your first TOC is "Avoid CPU intensive tasks - this issues is related to code that is CPU intensive like crypto, see the proposed TOC". Are you referring to the cypto module: https://nodejs.org/api/crypto.html - because the cpu intensive tasks from that module are run on the Worker Pool: crypto.pbkdf2(), crypto.scrypt(), crypto.randomBytes(), crypto.randomFill(), crypto.generateKeyPair()

[editing multiple times - sorry :P]

So I guess my new thought is there are operations to avoid that are cpu expensive - not on worker threads:

algorithms with large number of operations (like your while loop)
Large json parsing
Certain regex queries
long promises/awaits

Then there are synchronous apis (includes some cpu intensive, io intensive and worker pool)

Encryption:
        crypto.randomBytes (synchronous version)
        crypto.randomFillSync
        crypto.pbkdf2Sync
        You should also be careful about providing large input to the encryption and decryption routines.
    Compression:
        zlib.inflateSync
        zlib.deflateSync
    File system:
        Do not use the synchronous file system APIs. For example, if the file you access is in a distributed file system like NFS, access times can vary widely.
    Child process:
        child_process.spawnSync
        child_process.execSync
        child_process.execFileSync

goldbergyoni · 2019-06-10T16:53:26Z

So I guess my new thought is there are operations to avoid that are cpu expensive - not on worker threads:

algorithms with large number of operations (like your while loop)

Large json parsing

Certain regex queries

long promises/awaits

Yes! Those examples make this bullet shine. Crypto is just another example.

Now let's discuss Crypto which is just another bullet:

Crypto work (except some built-in crypto utils that are executed using the libuv threads pool)

Only few crypto libraries, a subset of those that are part of Node, are executed using the thread pool, this is why crypto is still a good example

Synchronous APIs are a different bullet, keep it focused on a single challenge - CPU. Because soon you'll have to remind solutions and mitigating CPU intensive tasks take a different form than mitigating sync-IO tasks (which are very eay to remedy - just avoid them :))

Makes sense?

TheHollidayInn · 2019-06-22T04:44:06Z

Alright, got a new draft up. Can you let me know what you think?

goldbergyoni

Great start, very challenging topic, let's perform 2 iterations until we perfect it?

@BrunoScheufler Need your help here, seems like there are few formatting issues?

goldbergyoni · 2019-07-01T16:20:18Z

README.md

+
+## ![✔] 7.1. Don't block the event loop
+
+**TL;DR:** TLDR; Avoid CPU intensive tasks as they will block the Event Loop.


Duplicate TLDR?

Maybe add "block the mostly single-threaded event loop"?

goldbergyoni · 2019-07-01T16:21:41Z

sections/performance/block-loop.md

@@ -0,0 +1,57 @@
+Node handles the Event Loop on a single thread rotating through multiple queues. Operations with high complexity, large json parsing, unsafe regex querieies, and long awaited promises are some of the operations that can cause the event loop to stall. Preventing this is as easy as setting up monitoring for the the Event Loop and resolving the blocker.


Did you forget the page title? see other 'more info' files, they all contain a title

goldbergyoni · 2019-07-01T16:27:02Z

sections/performance/block-loop.md

@@ -0,0 +1,57 @@
+Node handles the Event Loop on a single thread rotating through multiple queues. Operations with high complexity, large json parsing, unsafe regex querieies, and long awaited promises are some of the operations that can cause the event loop to stall. Preventing this is as easy as setting up monitoring for the the Event Loop and resolving the blocker.


"handles the Event Loop on a single thread " -> "mostly on a single thread" (just to be precise)

"Operations with high CPU cost" (it's not about complexity, IO can be too complex)

"querieies" - typo?

"long awaited promises" - what do you mean here? not sure this is in-general true, usually promises are used to wait for IO

"Preventing this is as easy as setting up monitoring for the the Event Loop and resolving the blocker." - monitoring is detection, not prevention. Let's keep the monitoring to a dedicated bullet. My thoughts "Avoid this off-loading CPU intensive tasks to dedicated workload that doesn't serer users (e.g. job server), or by breaking long tasks into small steps that stand on the event queue lines, by using crypto libraries that utilize the event loop thread pool. To name a few techniques that can keep the event loop alive"

A lot of my info comes from this article: https://nodejs.org/en/docs/guides/dont-block-the-event-loop/

From this section: https://nodejs.org/en/docs/guides/dont-block-the-event-loop/#don-t-block-the-event-loop

For promises: "You should make sure you never block the Event Loop. In other words, each of your JavaScript callbacks should complete quickly. This of course also applies to your await's, your Promise.then's, and so on."

For computational complexity
"A good way to ensure this is to reason about the "computational complexity" of your callbacks."

For non-sync IO, should I state that you should also watch for large IO tasks?

For promises: "You should make sure you never block the Event Loop. In other words, each of your JavaScript callbacks should complete quickly. This of course also applies to your await's, your Promise.then's, and so on."

Note there is a signficant difference between what you wrote and the article - you alluded to 'long awaited promises', they refer to the .then part. Practically, most promises wrapping some IO work, if you wait for long time then it's not related to CPU work. Their advice of not slowing down the .then handler makes sense, but it's too specific, we can generalize this advice to any other code...

"A good way to ensure this is to reason about the "computational complexity" of your callbacks."

Well, it's probably right technically, is it a simple and well understood term? I guess the answer is subjective. I suggest that our guide should curate and simplify those technical guides

For non-sync IO, should I state that you should also watch for large IO tasks?

I think it doesn't belong here

Note that I also included in my comment multiple mitigation techniques

I'll definitely include many of your suggestions. I'm just trying to hone in on "high CPU usage" because it is super generic. I want to tell people what to look for.

I believe that these are three examples: Operations with high complexity, large json parsing, unsafe regex queries. So "high complexity" is just one example.

Also, promises are also used a lot for db access. If you have long queries that you are are awaiting you could block the loop like our example, I think.

Anyway, I'm just trying to give devs examples of "cpu work" to look for :D. I'll get your edits in this weekend.

OKay, after reading this I'd retract my comment about awaiting. I was missing the idea that promises are added to the job queue: https://github.com/getify/You-Dont-Know-JS/blob/master/async%20%26%20performance/ch1.md. Soooo sorry about that :P

Yes, if you await on IO operation (E.g. DB call) - this will not the event loop

goldbergyoni · 2019-07-01T16:28:06Z

sections/performance/block-loop.md

+
+server.listen(3000)
+
+process.on('SIGINT', function () {


Can you shorten this example by 50%? the SIGINT for example is not relevant to this topic

goldbergyoni · 2019-07-01T16:32:15Z

sections/performance/block-loop.md

+```
+
+## Image of the Event Loop
+![1*aU5dr98pxTsZ4AMfnA6lNA.png](https://cdn-images-1.medium.com/max/1600/1*aU5dr98pxTsZ4AMfnA6lNA.png)


I'm not sure that this image is descriptive enough to explain why the loop is sensitive to CPU-bounded tasks. Can we find a better one?

TheHollidayInn · 2019-07-18T13:49:12Z

@i0natan Can you review my updates :D

goldbergyoni

Hey, thanks for improving, absolutely getting closer to 'production grade'. I estimate that we would need ~2 more iterations to make it 100% concise and fun to read. If preferable we can work on it simultaneously (e.g. git chat or hangouts).

goldbergyoni · 2019-07-22T09:08:03Z

README.md

+
+## ![✔] 7.1. Don't block the event loop
+
+**TL;DR:** Avoid CPU intensive tasks as they will block the mostly single-threaded Event Loop.


You stated what NOT to do, let's include your recommendation? for example "Avoid CPU intensive tasks as they will block the mostly single-threaded Event Loop and offload those to a dedicated thread, process or even a different technology based on the context"

goldbergyoni · 2019-07-22T09:08:43Z

README.md

+
+**TL;DR:** Avoid CPU intensive tasks as they will block the mostly single-threaded Event Loop.
+
+**Otherwise:** As the Event Loop is blocked, Node.js will be unable to handle other request thus causing delays for concurrent users.


Consider emphasizing the severity of this in a bolder language - "3000 users are waiting for a response, the content is ready to be served, but one single request blocks the server from dispatching the results back"

goldbergyoni · 2019-07-22T09:09:52Z

sections/performance/block-loop.md

+
+<br/><br/>
+
+Node handles the Event Loop mostly on a single thread rotating through multiple queues. Operations with high complexity, large json parsing, unsafe regex querieies, and large IO operations are some of the operations that can cause the Event Loop to stall. Avoid this off-loading CPU intensive tasks to a dedicated service (e.g. job server), or breaking long tasks into small steps then using the Worker Pool are some examples of how to avoid blocking the Event Loop.


One more example "large JSON parsing, applying logic over huge arrays, unsafe regex..."

goldbergyoni · 2019-07-22T09:13:22Z

sections/performance/block-loop.md

+
+<br/><br/>
+
+Node handles the Event Loop mostly on a single thread rotating through multiple queues. Operations with high complexity, large json parsing, unsafe regex querieies, and large IO operations are some of the operations that can cause the Event Loop to stall. Avoid this off-loading CPU intensive tasks to a dedicated service (e.g. job server), or breaking long tasks into small steps then using the Worker Pool are some examples of how to avoid blocking the Event Loop.


Let's now take it one step up and make our advice even more accurate, I'll be glad to explain why is this true:

"When handling CPU-intensive tasks, different techniques should be applied based on of two different scenarios: if these CPU-intensive tasks are needed rarely (e.g. triggered by a few requests only) - then they can be offloaded to a dedicate process, thread or get handled in chunks (See code example below). However, the more challenging scenario is where most or all requests need to trigger CPU-intensive code - in that case, this task must be offloaded to a different workload, one that doesn't dispatch content to the end-users, like a job-server, Serverless, and similar

Sounds good to me and makes sense. Should I add this as a separate paragraph?

Yes, or maybe to the main one?

goldbergyoni · 2019-07-22T09:13:57Z

sections/performance/block-loop.md

+
+Node handles the Event Loop mostly on a single thread rotating through multiple queues. Operations with high complexity, large json parsing, unsafe regex querieies, and large IO operations are some of the operations that can cause the Event Loop to stall. Avoid this off-loading CPU intensive tasks to a dedicated service (e.g. job server), or breaking long tasks into small steps then using the Worker Pool are some examples of how to avoid blocking the Event Loop.
+
+## Example


Formatting issue - in all other pages, the title contains the main message, e.g. "Example: blocking the event loop"

goldbergyoni · 2019-07-22T09:14:39Z

sections/performance/block-loop.md

+## Example
+Let's take a look at an example from [Node Clinic](https://clinicjs.org/documentation/doctor/05-fixing-event-loop-problem).
+```
+const restify = require('restify')


Code examples should be really short and concise, the first two lines and the last one are not needed (they make it look long)

sections/performance/block-loop.md

goldbergyoni · 2019-07-22T09:46:25Z

sections/performance/block-loop.md

+```
+
+## Image of the Event Loop
+   ┌───────────────────────────┐


This image is better, it gives a sense of the queues that might stay idle. Thinking out loud whether it's the best we can get? does it illustrate how some request blocks others? I tried to sketch something, see below, what do you think?

TheHollidayInn · 2019-07-25T22:24:16Z

I missed your comment but I'm happy to do git chat or hangouts :)

js-kyle · 2019-07-28T01:19:17Z

Awesome work going on here! Nice

goldbergyoni · 2019-07-29T10:24:04Z

@TheHollidayInn Let me know when do you want me to review the whole, I noted that you answered some of the comments and other don't

BrunoScheufler

Content looks great to me, awesome work 🙌

goldbergyoni · 2019-08-01T09:34:34Z

@TheHollidayInn Any chance that you're available for a 1-hour finalization session afterwhich, we will publish this? when might be a good time next week?

Best if you can merge all my remarks into the PR by then so we just focus on final touches

goldbergyoni · 2019-08-01T09:36:20Z

@BrunoScheufler Can you review the lexical part + formatting?

TheHollidayInn · 2019-08-05T14:32:07Z

Sweeeet!

@i0natan Do you have any time Tuesday or Thursday after 6pm UTC?

goldbergyoni · 2019-08-06T13:23:17Z

@TheHollidayInn What timezone are you in? my calendar seems busy at those timeslots, do you happen to be available on Friday 6-8am UTC or Sunday 12-16 UTC or Monday 12-16?

goldbergyoni · 2019-08-11T13:33:04Z

@TheHollidayInn Do you have some availability this week and can proposes 2-3 slots?

TheHollidayInn · 2019-08-11T15:43:53Z

Sorry! I actually had to do a last minute travel for work.

I am back on Wednesday. I can do Wed-Fri after 6pm UTC. I'll need to check Sat, but I'm more flexible on Sun.

I'm in MST, btw. In Colorado. What about you?

TheHollidayInn · 2019-08-19T13:05:55Z

@goldbergyoni Just following up. Are you free this week?

goldbergyoni · 2019-08-20T16:50:30Z

@TheHollidayInn Sorry for the delay, on a very tight trip now in US and struggle to find time.

My suggestion - let's merge this item as-is now, then there is a couple of enrichment I'd like us to perform as additional PRs. Works for you?

@BrunoScheufler @js-kyle Could you please help with reviewing the formatting/grammar/wording?

p.s. @TheHollidayInn Did you see the credit I gave you in my testing practices guide?

TheHollidayInn · 2019-08-20T18:45:00Z

Works for me! I hope your trip is going well :D

I have your testing practices on my list to read. I just saw it in Node Weekly, congrats!!
I now see the link to me! Thank you :D :D

goldbergyoni · 2019-09-03T17:56:58Z

@BrunoScheufler Could you please help with format and language review?

goldbergyoni · 2019-09-11T18:02:19Z

@TheHollidayInn I've merged this, big congrats to us and many thanks for the high effort here! 🎇 🚀 🙏

I believe we can and should take 1/2 level up, I'll pr with some suggestions

Added initial event loop copy

Added initial event loop copy

2c63fee

TheHollidayInn mentioned this pull request Apr 9, 2019

Performance: Don't block the event loop #294

Closed

Change to discuss cpu usage blocking

b34549e

goldbergyoni requested changes Jul 1, 2019

View reviewed changes

Copy updates

221f176

goldbergyoni requested changes Jul 22, 2019

View reviewed changes

Copy updates.

7d65352

BrunoScheufler reviewed Jul 30, 2019

View reviewed changes

Merge branch 'master' into event-loop

40f0dc3

goldbergyoni merged commit d237500 into goldbergyoni:master Sep 11, 2019

elite0226 pushed a commit to elite0226/nodebestpractices that referenced this pull request Oct 31, 2022

Merge pull request goldbergyoni#373 from TheHollidayInn/event-loop

3552ed0

Added initial event loop copy


		## ![✔] 7.1. Don't block the event loop

		TL;DR: TLDR; Avoid CPU intensive tasks as they will block the Event Loop.

		@@ -0,0 +1,57 @@
		Node handles the Event Loop on a single thread rotating through multiple queues. Operations with high complexity, large json parsing, unsafe regex querieies, and long awaited promises are some of the operations that can cause the event loop to stall. Preventing this is as easy as setting up monitoring for the the Event Loop and resolving the blocker.


		## ![✔] 7.1. Don't block the event loop

		TL;DR: Avoid CPU intensive tasks as they will block the mostly single-threaded Event Loop.


		TL;DR: Avoid CPU intensive tasks as they will block the mostly single-threaded Event Loop.

		Otherwise: As the Event Loop is blocked, Node.js will be unable to handle other request thus causing delays for concurrent users.


		<br/><br/>

		Node handles the Event Loop mostly on a single thread rotating through multiple queues. Operations with high complexity, large json parsing, unsafe regex querieies, and large IO operations are some of the operations that can cause the Event Loop to stall. Avoid this off-loading CPU intensive tasks to a dedicated service (e.g. job server), or breaking long tasks into small steps then using the Worker Pool are some examples of how to avoid blocking the Event Loop.


		Node handles the Event Loop mostly on a single thread rotating through multiple queues. Operations with high complexity, large json parsing, unsafe regex querieies, and large IO operations are some of the operations that can cause the Event Loop to stall. Avoid this off-loading CPU intensive tasks to a dedicated service (e.g. job server), or breaking long tasks into small steps then using the Worker Pool are some examples of how to avoid blocking the Event Loop.

		## Example


		server.listen(3000)

		process.on('SIGINT', function () {

Added initial event loop copy #373

Added initial event loop copy #373

Conversation

TheHollidayInn commented Apr 9, 2019

goldbergyoni commented Apr 10, 2019

goldbergyoni commented Apr 10, 2019

TheHollidayInn commented May 15, 2019

goldbergyoni commented May 24, 2019

TheHollidayInn commented May 27, 2019

goldbergyoni commented May 28, 2019 • edited Loading

TheHollidayInn commented May 28, 2019

goldbergyoni commented Jun 3, 2019

TheHollidayInn commented Jun 5, 2019

goldbergyoni commented Jun 6, 2019

TheHollidayInn commented Jun 6, 2019 • edited Loading

goldbergyoni commented Jun 10, 2019

TheHollidayInn commented Jun 22, 2019

goldbergyoni left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheHollidayInn Jul 1, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheHollidayInn commented Jul 18, 2019

goldbergyoni left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

goldbergyoni Jul 22, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheHollidayInn commented Jul 25, 2019 • edited Loading

js-kyle commented Jul 28, 2019

goldbergyoni commented Jul 29, 2019

BrunoScheufler left a comment

Choose a reason for hiding this comment

goldbergyoni commented Aug 1, 2019

goldbergyoni commented Aug 1, 2019

TheHollidayInn commented Aug 5, 2019

goldbergyoni commented Aug 6, 2019

goldbergyoni commented Aug 11, 2019

TheHollidayInn commented Aug 11, 2019

TheHollidayInn commented Aug 19, 2019

goldbergyoni commented Aug 20, 2019

TheHollidayInn commented Aug 20, 2019

goldbergyoni commented Sep 3, 2019

goldbergyoni commented Sep 11, 2019

goldbergyoni commented May 28, 2019 •

edited

Loading

TheHollidayInn commented Jun 6, 2019 •

edited

Loading

TheHollidayInn Jul 1, 2019 •

edited

Loading

goldbergyoni Jul 22, 2019 •

edited

Loading

TheHollidayInn commented Jul 25, 2019 •

edited

Loading