Parse server with Heroku App totally crash after long running action

[AnChiChang]

Hi

I had run Parse Server on Heroku, recently, sometimes server totally crash, and can't get any response. Cause client App can't connect with Server. I had discussed with Heroku technical support and monitor the situation.

We found the first crash always:
Connection closed w/o response: Heroku/xxxxx

2017-01-24 04:18:09.986
371 1 2017-01-24T04:18:09.813106+00:00 heroku router - - at=error code=H13 desc="Connection closed without response" method=PUT path="/parse/classes/_Installation/KfuCX2uZa3" host=xxxxx.herokuapp.com request_id=c089ed29-3bc4-4a65-91b5-c616ba177e66 fwd="210.63.206.33" dyno=web.1 connect=1ms service=101ms status=503 bytes=0

Go to the log: https://api.heroku.com/myapps/app46892815@heroku.com/addons/logentries

You are receiving this email because your Logentries alarm "Connection closed w/o response"
has been triggered.

In context:
2017-01-24 04:17:53.241 309 1 2017-01-24T04:17:53.030785+00:00 heroku router - - at=info method=POST path="/parse/classes/OrderHeader" host=xxxxx.herokuapp.com request_id=04a3231f-dcc9-4e45-8b75-5817b39a1a27 fwd="54.167.80.205" dyno=web.1 connect=0ms service=112ms status=200 bytes=532
2017-01-24 04:17:53.253 309 1 2017-01-24T04:17:53.028848+00:00 heroku router - - at=info method=POST path="/parse/functions/getOrderQty" host=xxxxx.herokuapp.com request_id=cb8dadf4-4491-4136-8aa8-26cedcf8ddeb fwd="77.167.9.48" dyno=web.1 connect=0ms service=524ms status=200 bytes=525
2017-01-24 04:17:53.336 309 1 2017-01-24T04:17:53.027281+00:00 heroku router - - at=info method=POST path="/parse/classes/OrderHeader" host=xxxxx.herokuapp.com request_id=5a986cb6-f604-49f5-a345-b60e770b52ec fwd="54.167.80.205" dyno=web.1 connect=0ms service=106ms status=200 bytes=532
2017-01-24 04:17:54.620 314 1 2017-01-24T04:17:54.476007+00:00 heroku router - - at=info method=GET path="/parse/classes/Products/sKecipBbID" host=xxxxx.herokuapp.com request_id=614a52c0-de6a-4a85-b814-b45cda655e0f fwd="77.167.9.48" dyno=web.1 connect=0ms service=19ms status=200 bytes=1453
2017-01-24 04:17:54.728 308 1 2017-01-24T04:17:54.493833+00:00 heroku router - - at=info method=POST path="/parse/classes/ProductImage" host=xxxxx.herokuapp.com request_id=8ed5e98d-ec50-4a28-9b18-3d84a72d26c6 fwd="77.167.9.48" dyno=web.1 connect=0ms service=23ms status=200 bytes=1358
2017-01-24 04:17:55.195 306 1 2017-01-24T04:17:54.888491+00:00 heroku router - - at=info method=POST path="/parse/classes/ProductSpec" host=xxxxx.herokuapp.com request_id=f9c154b8-2a6d-4830-959e-30a5504bf630 fwd="77.167.9.48" dyno=web.1 connect=0ms service=43ms status=200 bytes=775
2017-01-24 04:17:55.554 308 1 2017-01-24T04:17:55.273767+00:00 heroku router - - at=info method=POST path="/parse/functions/getOrderQty" host=xxxxx.herokuapp.com request_id=ff93ac54-53d5-4fbb-93b7-5e0cd6d06a95 fwd="77.167.9.48" dyno=web.1 connect=0ms service=82ms status=200 bytes=525
2017-01-24 04:17:55.593 300 1 2017-01-24T04:17:55.320901+00:00 heroku router - - at=info method=POST path="/parse/classes/Chat" host=xxxxx.herokuapp.com request_id=00d569fe-3461-477e-980a-614c2fd96a42 fwd="77.167.9.48" dyno=web.1 connect=0ms service=79ms status=200 bytes=6248
2017-01-24 04:18:08.459 310 1 2017-01-24T04:18:08.350748+00:00 heroku router - - at=info method=POST path="/parse/functions/pushToChannel" host=xxxxx.herokuapp.com request_id=50453d4e-a9fc-475b-b039-992bbc3819b0 fwd="77.167.9.48" dyno=web.1 connect=1ms service=84ms status=200 bytes=502
2017-01-24 04:18:08.785 293 1 2017-01-24T04:18:08.509356+00:00 heroku router - - at=info method=POST path="/parse/push" host=xxxxx.herokuapp.com request_id=421a20da-611c-4022-8293-50daf6d75858 fwd="54.167.80.205" dyno=web.1 connect=0ms service=90ms status=200 bytes=533
2017-01-24 04:18:09.986 371 1 2017-01-24T04:18:09.813106+00:00 heroku router - - at=error code=H13 desc="Connection closed without response" method=PUT path="/parse/classes/_Installation/KfuCX2uZa3" host=xxxxx.herokuapp.com request_id=c089ed29-3bc4-4a65-91b5-c616ba177e66 fwd="210.63.206.33" dyno=web.1 connect=1ms service=101ms status=503 bytes=0

after that, Parse Server totally crash. can't support anymore. And after analysis by Heroku Technical support. He give me some advise like below:

To explain a little more H13 errors often occur alongside the H12 errors that Annie mentioned. We generally see this pattern where one long-running action starts hogging the queue which in turn affects any subsequent requests.

Our router will drop a long-running request after 30 seconds, but the dyno behind it will continue processing the request until completion. Our router is unaware of it, though, so it'll dispatch new requests to that busy dyno. This effect tends to compound, and you'll eventually see H12 errors even for unrelated URLs, such as static assets. H13 errors are similar in what causes them, but are primarily related to concurrent web servers.

As your app is using ExpressJS, you will also want to install something like timeout, which will ensure that a long running request is dropped at the dyno-level as well. Specifically, timeout raise a Response timeout exception when that happens.

With that in place, the compound effect is less likely to occur, but long-running actions still need to be addressed.

Please let me know if I can be of further help.

Have any timeout handle tool that he mentioned I need to install also? Or have any good idea for solving this problem? Thanks for helping.

Kevin

[iatek]

I've had this problem with any long-running requests on Heroku Node too, but it isn't really specific to Parse Server.

The simplest way to handle is to set a timeout on the Parse Server endpoint request so that it can be "gracefully" handled before Heroku drops it (at 30 seconds). For example, I usually use Express and "Request" (https://github.com/request/request#requestoptions-callback) together which accepts a "timeout" parameter. This way you can abort the request before Heroku does. This article on Heroku is also helpful:
https://devcenter.heroku.com/articles/request-timeout

request.get({url:'http://myparserver/someclass/'+id,
        json:true,
        timeout: 10000, // some time in MS less than 30000 when Heroku will drop the request
        headers:conf.parse.headers},
        function (e,r,b){
        if (e || r.statusCode != 200 || (b && b.error)) {
            console.log("error------"+JSON.stringify((e)||b));
            // handle the error
        }
        else {
            // success
        }
});

[AnChiChang]

hello Iatek

first, thanks for your patience to explain.
I understand Heroku router behavior, but I don't quite understand, I have to adjust Parse Server side to setup time out or control in my client App(iOS and Android App)?
I don't understand that you mention "Parse Server endpoint"
Please advise. Thanks

[AnChiChang]

And I also found all H13(connection close without response) always from my cloud code.
one is from Parse.Cloud.afterSave, after save without any response parameter, I don't know why cause long run.
The other one is from Parse.Cloud.define, but that function only do some Parse Query and save.

[joeyslack]

Heroku will not allow your process to run for longer than 30 seconds. They do it by design. Either talk to them, change providers, or offload complicated computations into offline jobs.