Upgrading to gunicorn >19.4.1 causes server to crash

[kevgliss]

There is currently an issue with how argparse is handling an upstream change in gunicorn.

Here https://github.com/Netflix/lemur/blob/master/lemur/manage.py#L387 we take the settings generated by gunicorn and combine them with our own options.

If we take a look at the difference between the two versions:

19.4.1

['--proxy-protocol'] store_true
['--worker-connections'] None
['--statsd-host'] None
['--max-requests-jitter'] None
['--pythonpath'] None
['-R', '--enable-stdio-inheritance'] store_true
['-k', '--worker-class'] None
['--suppress-ragged-eofs'] store_true
['--ssl-version'] None
['--log-syslog-facility'] None
['--log-syslog'] store_true
['--cert-reqs'] None
['--preload'] store_true
['--keep-alive'] None
['--access-logfile'] None
['-g', '--group'] None
['--graceful-timeout'] None
['--do-handshake-on-connect'] store_true
['--spew'] store_true
['-w', '--workers'] None
['-n', '--name'] None
['-m', '--umask'] None
['--worker-tmp-dir'] None
['--limit-request-fields'] None
['-c', '--config'] None
['--log-config'] None
['-p', '--pid'] None
['--no-sendfile'] store_false
['--check-config'] store_true
['--statsd-prefix'] None
['--proxy-allow-from'] None
['--forwarded-allow-ips'] None
['--max-requests'] None
['--threads'] None
['-u', '--user'] None
['--limit-request-line'] None
['--access-logformat'] None
['--certfile'] None
['--paste', '--paster'] None
['--log-syslog-prefix'] None
['-D', '--daemon'] store_true
['--chdir'] None
['--log-syslog-to'] None
['--ciphers'] None
['--log-level'] None
['-b', '--bind'] append
['--error-logfile', '--log-file'] None
['-e', '--env'] append
['--reload'] store_true
['--limit-request-field_size'] None
['-t', '--timeout'] None
['--ca-certs'] None
['--settings'] None
['--keyfile'] None
['--backlog'] None
['--logger-class'] None

19.4.4

['--proxy-protocol'] store_true
['--worker-connections'] None
['--statsd-host'] None
['--max-requests-jitter'] None
['--pythonpath'] None
['-R', '--enable-stdio-inheritance'] store_true
['-k', '--worker-class'] None
['--suppress-ragged-eofs'] store_true
['--ssl-version'] None
['--log-syslog-facility'] None
['--log-syslog'] store_true
['--cert-reqs'] None
['--preload'] store_true
['--keep-alive'] None
['--access-logfile'] None
['-g', '--group'] None
['--graceful-timeout'] None
['--do-handshake-on-connect'] store_true
['--spew'] store_true
['-w', '--workers'] None
['-n', '--name'] None
['-m', '--umask'] None
['--worker-tmp-dir'] None
['--limit-request-fields'] None
['-c', '--config'] None
['--log-config'] None
['-p', '--pid'] None
['--no-sendfile'] store_const
['--check-config'] store_true
['--statsd-prefix'] None
['--proxy-allow-from'] None
['--forwarded-allow-ips'] None
['--max-requests'] None
['--threads'] None
['-u', '--user'] None
['--limit-request-line'] None
['--access-logformat'] None
['--certfile'] None
['--paste', '--paster'] None
['--log-syslog-prefix'] None
['-D', '--daemon'] store_true
['--chdir'] None
['--log-syslog-to'] None
['--ciphers'] None
['--log-level'] None
['-b', '--bind'] append
['--error-logfile', '--log-file'] None
['-e', '--env'] append
['--reload'] store_true
['--limit-request-field_size'] None
['-t', '--timeout'] None
['--ca-certs'] None
['--settings'] None
['--keyfile'] None
['--backlog'] None
['--logger-class'] None

The only change I make out is the change from ['--no-sendfile'] store_false to ['--no-sendfile'] store_const. This change corresponds with this change:
benoitc/gunicorn@d8b6f0a#commitcomment-15256412

[tilgovi]

@kevgliss I think you just need to pass along const=cls.const to the Option constructor there.

[tilgovi]

Sorry, that's const=klass.const as you call it here.

[kevgliss]

@tilgovi That didn't seem to help.

Stacktrace without klass.const:

Traceback (most recent call last):
  File "manage.py", line 778, in <module>
    main()
  File "manage.py", line 775, in main
    manager.run()
  File "/Users/kglisson/.virtualenvs/lemur/lib/python2.7/site-packages/Flask_Script-2.0.5-py2.7.egg/flask_script/__init__.py", line 412, in run
    result = self.handle(sys.argv[0], sys.argv[1:])
  File "/Users/kglisson/.virtualenvs/lemur/lib/python2.7/site-packages/Flask_Script-2.0.5-py2.7.egg/flask_script/__init__.py", line 350, in handle
    app_parser = self.create_parser(prog)
  File "/Users/kglisson/.virtualenvs/lemur/lib/python2.7/site-packages/Flask_Script-2.0.5-py2.7.egg/flask_script/__init__.py", line 193, in create_parser
    command_parser = command.create_parser(name, func_stack=func_stack, parent=self)
  File "/Users/kglisson/.virtualenvs/lemur/lib/python2.7/site-packages/Flask_Script-2.0.5-py2.7.egg/flask_script/commands.py", line 202, in create_parser
    parser.add_argument(*option.args, **option.kwargs)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1294, in add_argument
    action = action_class(**kwargs)
TypeError: __init__() takes at least 4 arguments (3 given)

With your suggestion:

Traceback (most recent call last):
  File "manage.py", line 778, in <module>
    main()
  File "manage.py", line 775, in main
    manager.run()
  File "/Users/kglisson/.virtualenvs/lemur/lib/python2.7/site-packages/Flask_Script-2.0.5-py2.7.egg/flask_script/__init__.py", line 412, in run
    result = self.handle(sys.argv[0], sys.argv[1:])
  File "/Users/kglisson/.virtualenvs/lemur/lib/python2.7/site-packages/Flask_Script-2.0.5-py2.7.egg/flask_script/__init__.py", line 350, in handle
    app_parser = self.create_parser(prog)
  File "/Users/kglisson/.virtualenvs/lemur/lib/python2.7/site-packages/Flask_Script-2.0.5-py2.7.egg/flask_script/__init__.py", line 193, in create_parser
    command_parser = command.create_parser(name, func_stack=func_stack, parent=self)
  File "/Users/kglisson/.virtualenvs/lemur/lib/python2.7/site-packages/Flask_Script-2.0.5-py2.7.egg/flask_script/commands.py", line 202, in create_parser
    parser.add_argument(*option.args, **option.kwargs)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1294, in add_argument
    action = action_class(**kwargs)
TypeError: __init__() got an unexpected keyword argument 'const'

I am thinking that this has more to do with Flask-Script implementation more than anything else. It's weird because store_const is a valid argparse action.

[john-bodley]

@kevgliss it seems like the the const argument is only valid for options where klass.const is not None, i.e.

options = (
    Option(
        *klass.cli,
        action=klass.action,
        **({'const': klass.const} if klass.const is not None else {})
    ) for setting, klass in settings.iteritems() if klass.cli
)