GAE prohibits spawning new threads directly

[mattrjacobs]

Issue found after #734 change

[mattrjacobs]

(copied from #734 (comment))

Originally written by @chrisbianca:

@mattrjacobs Sadly, whilst this has removed Unsafe as a stumbling block on GAE, it fails at the next hurdle instead which is the spawning of new Threads:

[INFO] SEVERE: Unhandled exception
[INFO] java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "modifyThreadGroup")
[INFO]  at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
[INFO]  at java.security.AccessController.checkPermission(AccessController.java:559)
[INFO]  at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
[INFO]  at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkPermission(DevAppServerFactory.java:429)
[INFO]  at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkAccess(DevAppServerFactory.java:454)
[INFO]  at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:315)
[INFO]  at java.lang.Thread.init(Thread.java:391)
[INFO]  at java.lang.Thread.init(Thread.java:349)
[INFO]  at java.lang.Thread.<init>(Thread.java:548)
[INFO]  at com.netflix.hystrix.util.HystrixTimer$ScheduledExecutor$1.newThread(HystrixTimer.java:155)

GAE restricts the spawning of new threads other than through it's ThreadManager (https://cloud.google.com/appengine/docs/java/javadoc/com/google/appengine/api/ThreadManager).

To work around this, the creation of new threads in Hystrix would have to make use of a ThreadFactory instead of creating new Threads directly. Do you think this might be possible?

There's a good example in Guava of how they've worked around the limitation on GAE: http://docs.guava-libraries.googlecode.com/git/javadoc/src-html/com/google/common/util/concurrent/MoreExecutors.html#line.766

[mattrjacobs]

(copied from #734 (comment))

(originally written by @chrisbianca)

@mattrjacobs : I had a bit of a play round with this myself. Unfortunately whilst using the ThreadFactory does alleviate the error, it means that you're not able to name the threads as they are currently: Thread.setName() is also a restricted GAE class.

Is there a reason the threads have to be named hystrix-XXX?

[chrisbianca]

@mattrjacobs : Removing setName() and setDaemon() means that GAE will actually execute the HystrixCommand without any errors. I've not studied the Hystrix code enough to establish what knock on effects this might have on functionality though.

[mattrjacobs]

AFAIK, changing names is a change that humans would notice but the Hystrix runtime wouldn't care about. I'll need to review the code to confirm this.

setDaemon was a fairly recent change, so not performing that in GAE would be OK. This only affects behavior at shutdown.

If those changes would be reasonable, then I would likely gate them to change the behavior for GAE, and leave all other users as-is. The implementation from Guava looks right (and I'm sure Google knows better than I do about how to detect AppEngine)

[chrisbianca]

Yes, that seems like a reasonable approach, and certainly hacking in those changes on a local build has allowed me to at least make use of the core Hystrix functionality on GAE. I haven't had a chance to check any of the other functionality, e.g. servlet, as yet. I'll try and do that next week, but this will certainly be a good step in the right direction.

[mattrjacobs]

For reference, the places in code which call newThread currently are:

• HystrixTimer
• HystrixConcurrencyStrategy
• HystrixMetricsPoller

[mattrjacobs]

@chrisbianca Could you take a look at #1066?

[wendelicious]

It seems that we are seeing the same issue. Do these changes also apply in Google app engine flex/compat runtime?

[mattrjacobs]

You'll have to forgive me - I know nothing about GAE. The changed outlined above were:

1. not use sun.misc.Unsafe
2. detect if running in GAE. if so, use the GAE-provided method for creating threads instead of calling new Thread(...).

2 was never tested

If you could report what version of Hystrix you're using and what error you're encountering, that would help understand what the next step is