Parallel Requests

Issuing Parallel Requests

em-http asynchronous API allows you to issue many requests in parallel – no need to serialize and wait for each one, or spool up any threads. In fact, you can achieve parallel execution with em-http through several approaches.

Issuing independent parallel requests

EventMachine.run do
  http1 = EventMachine::HttpRequest.new('http://google.com/').get
  http2 = EventMachine::HttpRequest.new('http://yahoo.com/').get

  http1.callback { }
  http2.callback { } 
end

In the case above, two requests are issued in parallel, and two callbacks are setup – each will be executed by EventMachine when the server responds.

Synchronizing with Multi interface

EventMachine.run do
  multi = EventMachine::MultiRequest.new

  multi.add :google, EventMachine::HttpRequest.new('http://www.google.com/').get
  multi.add :yahoo, EventMachine::HttpRequest.new('http://www.yahoo.com/').get

  multi.callback do
    puts multi.responses[:callback]
    puts multi.responses[:errback]
    EventMachine.stop
  end
end

Sometimes you want to issue multiple requests in parallel, but you need to wait until all of the complete before you proceed any further in your code. That’s where the Multi interface comes in: create an instance, add as many requests to it as you wish, and setup a callback on the Multi instance. The callback will be invoked only when all of the requests are done. Note that Multi will always invoke the callback function, regardless of whether the request succeed or failed.

Controlling concurrency with Iterators

EventMachine ships with asynchronous iterators which allow you to control the concurrency of the executing requests. For example, let’s say you have 100 requests you have to make, but you only want to have at most 10 in progress at any given point. That’s where asynchronous iterators come in. Take a look at the EventMachine repo & documentation on how to set this up.