Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

uWSGI with libyaml: static-map parsing #2097

Closed
almahmoud opened this issue Nov 14, 2019 · 1 comment · Fixed by #2122
Closed

uWSGI with libyaml: static-map parsing #2097

almahmoud opened this issue Nov 14, 2019 · 1 comment · Fixed by #2122

Comments

@almahmoud
Copy link

We are running uWSGI with libyaml, accomplished by specifying a profile as mentioned in: #863 (comment), and after much debugging found that the YAML configuration has some oddly specific requirements when it comes to listing static-map entries.
Specifically, the following configuration does not work:

uwsgi:
  static-map:
  - /static/style=/server/static/style/blue
  - /static=/server/static
  - /favicon.ico=/server/static/favicon.ico

appearing as if there is no static mapping, although it is technically valid YAML, but the following does work:

uwsgi:
  static-map:
    - /static/style=/server/static/style/blue
    - /static=/server/static
    - /favicon.ico=/server/static/favicon.ico

Additionally, we noticed that all the keys after static-map are inexplicably ignored.
Specifically, the following will properly parse all the configs:

uwsgi:
  buffer-size: 16384
  die-on-term: true
  enable-threads: true
  hook-master-start: 'unix_signal:2 gracefully_kill_them_all'
  http: 0.0.0.0:8080
  manage-script-name: true
  master: true
  offload-threads: 2
  processes: 1
  py-call-osafterfork: true
  threads: 4
  thunder-lock: true
  static-map:
    - /static/style=/server/static/style/blue
    - /static=/server/static
    - /favicon.ico=/server/static/favicon.ico

but, if the static-map is specified earlier in the config, everything after it seems to be ignored. For example, this will result in uWSGI signaling that thunder lock is disabled:

uwsgi:
  buffer-size: 16384
  die-on-term: true
  enable-threads: true
  hook-master-start: 'unix_signal:2 gracefully_kill_them_all'
  http: 0.0.0.0:8080
  manage-script-name: true
  master: true
  offload-threads: 2
  processes: 1
  py-call-osafterfork: true
  threads: 4
  static-map:
    - /static/style=/server/static/style/blue
    - /static=/server/static
    - /favicon.ico=/server/static/favicon.ico
  thunder-lock: true

This is specifically affecting us because we want to use valid YAML (so duplicate keys are problematic, hence the need for the libyaml version) and are generating the YAML config file through the Helm templating engine (ultimately running in a Kubernetes environment), and thus do not have direct control over the order of the keys or the indentation unless we special case the key which we are trying to avoid doing. Our presumption was that since it is libyaml, it should be able to parse any valid YAML config file, but that seems to not be the case. Are we missing something?
@almahmoud almahmoud mentioned this issue Nov 14, 2019
Merged
@nuwang nuwang mentioned this issue Dec 16, 2019
Closed
awelzel added a commit to awelzel/uwsgi that referenced this issue Jan 25, 2020
@awelzel
yaml/libyaml: Fix keys ignored after sequence, support non-indented s…
f13843b 
…equence entries

This patch re-purposes the in_uwsgi_section variable to track the depth
of the configuration. Previously, leaving a nested sequence would
stop the parsing process, ignoring any keys following the sequence.

Further, support non-indented sequences for which no
YAML_BLOCK_SEQUENCE_START_TOKEN is emitted. The entries are preceded by
the YAML_BLOCK_ENTRY_TOKEN, however, which can be used instead to
recognize these entries.

The following configuration is parsed sensibly after this patch when
compiled with libyaml:

    uwsgi:
      http-socket: ':4040'
      wsgi: tests.hello
      static-map:
        - /static/js=/server/static/js
        - /static/css=/server/static/css
      cache2: [
          "name=thumbnails,items=200",
          "name=videos,items=100"
      ]
      env:  # this is missing the SEQUENCE_START_TOKEN
      - ENVIRONMENT=prod
      - "DATABASE_URL=mysql://localhost"
      workers: 4
      thunder-lock: true

    # Verified with --show-config
    ./uwsgi -y ./config.yaml --show-config
    ...
    ;uWSGI instance configuration
    [uwsgi]
    yaml = ./config.yaml
    http-socket = :4040
    wsgi = tests.hello
    static-map = /static/js=/server/static/js
    static-map = /static/css=/server/static/css
    cache2 = name=thumbnails,items=200
    cache2 = name=videos,items=100
    env = ENVIRONMENT=prod
    env = DATABASE_URL=mysql://localhost
    workers = 4
    thunder-lock = true
    show-config = true
    ;end of configuration

Fixes unbit#2097 (but *only* if libyaml is used).
awelzel added a commit to awelzel/uwsgi that referenced this issue Jan 25, 2020
@awelzel
yaml/libyaml: Fix keys ignored after sequence, support non-indented s…
8db8622 
…equence entries

This patch re-purposes the in_uwsgi_section variable to track the depth
of the configuration. Previously, leaving a nested sequence would
stop the parsing process, ignoring any keys following the sequence.

Further, support non-indented sequences for which no
YAML_BLOCK_SEQUENCE_START_TOKEN is emitted. The entries are preceded by
the YAML_BLOCK_ENTRY_TOKEN, however, which can be used instead to
recognize these entries.

The following configuration is parsed sensibly after this patch when
compiled with libyaml:

    uwsgi:
      http-socket: ':4040'
      wsgi: tests.hello
      static-map:
        - /static/js=/server/static/js
        - /static/css=/server/static/css
      cache2: [
          "name=thumbnails,items=200",
          "name=videos,items=100"
      ]
      env:  # this is missing the SEQUENCE_START_TOKEN
      - ENVIRONMENT=prod
      - "DATABASE_URL=mysql://localhost"
      workers: 4
      thunder-lock: true

    # Verified with --show-config
    ./uwsgi -y ./config.yaml --show-config
    ...
    ;uWSGI instance configuration
    [uwsgi]
    yaml = ./config.yaml
    http-socket = :4040
    wsgi = tests.hello
    static-map = /static/js=/server/static/js
    static-map = /static/css=/server/static/css
    cache2 = name=thumbnails,items=200
    cache2 = name=videos,items=100
    env = ENVIRONMENT=prod
    env = DATABASE_URL=mysql://localhost
    workers = 4
    thunder-lock = true
    show-config = true
    ;end of configuration

Fixes unbit#2097 (but *only* if libyaml is used).
awelzel added a commit to awelzel/uwsgi that referenced this issue Jan 25, 2020
@awelzel
yaml/libyaml: Fix keys ignored after sequence, support non-indented s…
37f1a31 
…equence entries

This patch re-purposes the in_uwsgi_section variable to track the depth
of the configuration. Previously, leaving a nested sequence would
stop the parsing process, ignoring any keys following the sequence.

Further, support non-indented sequences for which no
YAML_BLOCK_SEQUENCE_START_TOKEN is emitted. The entries are preceded by
the YAML_BLOCK_ENTRY_TOKEN, however, which can be used instead to
recognize these entries.

The following configuration is parsed sensibly after this patch when
compiled with libyaml:

    uwsgi:
      http-socket: ':4040'
      wsgi: tests.hello
      static-map:
        - /static/js=/server/static/js
        - /static/css=/server/static/css
      cache2: [
          "name=thumbnails,items=200",
          "name=videos,items=100"
      ]
      env:  # this is missing the SEQUENCE_START_TOKEN
      - ENVIRONMENT=prod
      - "DATABASE_URL=mysql://localhost"
      workers: 4
      thunder-lock: true

    # Verified with --show-config
    ./uwsgi -y ./config.yaml --show-config
    ...
    ;uWSGI instance configuration
    [uwsgi]
    yaml = ./config.yaml
    http-socket = :4040
    wsgi = tests.hello
    static-map = /static/js=/server/static/js
    static-map = /static/css=/server/static/css
    cache2 = name=thumbnails,items=200
    cache2 = name=videos,items=100
    env = ENVIRONMENT=prod
    env = DATABASE_URL=mysql://localhost
    workers = 4
    thunder-lock = true
    show-config = true
    ;end of configuration

Fixes unbit#2097
@awelzel awelzel mentioned this issue Jan 25, 2020
Merged
@awelzel
Copy link
Contributor

awelzel commented Jan 25, 2020

@almahmoud , @nuwang

nice find! I've opened #2122 which should fix both behaviors, but it would be good to have it tested. I have pushed a uwsgi-2.0 branch plus the fix at:

https://github.com/awelzel/uwsgi/tree/2097-libyaml-fixes-2.0

Mind giving this a try and report back?

$ UWSGI_PROFILE_OVERRIDE=yaml=libyaml pip install git+https://github.com/awelzel/uwsgi.git@2097-libyaml-fixes-2.0

Our presumption was that since it is libyaml, it should be able to parse any valid YAML config file, but that seems to not be the case. Are we missing something?

You didn't miss anything, the yaml config parsing is "interesting" ;-)

@awelzel awelzel mentioned this issue Jan 25, 2020
Open
xrmx pushed a commit to xrmx/uwsgi that referenced this issue Mar 11, 2020
@awelzel @xrmx
yaml/libyaml: Fix keys ignored after sequence, support non-indented s…
ff48d4a 
…equence entries

This patch re-purposes the in_uwsgi_section variable to track the depth
of the configuration. Previously, leaving a nested sequence would
stop the parsing process, ignoring any keys following the sequence.

Further, support non-indented sequences for which no
YAML_BLOCK_SEQUENCE_START_TOKEN is emitted. The entries are preceded by
the YAML_BLOCK_ENTRY_TOKEN, however, which can be used instead to
recognize these entries.

The following configuration is parsed sensibly after this patch when
compiled with libyaml:

    uwsgi:
      http-socket: ':4040'
      wsgi: tests.hello
      static-map:
        - /static/js=/server/static/js
        - /static/css=/server/static/css
      cache2: [
          "name=thumbnails,items=200",
          "name=videos,items=100"
      ]
      env:  # this is missing the SEQUENCE_START_TOKEN
      - ENVIRONMENT=prod
      - "DATABASE_URL=mysql://localhost"
      workers: 4
      thunder-lock: true

    # Verified with --show-config
    ./uwsgi -y ./config.yaml --show-config
    ...
    ;uWSGI instance configuration
    [uwsgi]
    yaml = ./config.yaml
    http-socket = :4040
    wsgi = tests.hello
    static-map = /static/js=/server/static/js
    static-map = /static/css=/server/static/css
    cache2 = name=thumbnails,items=200
    cache2 = name=videos,items=100
    env = ENVIRONMENT=prod
    env = DATABASE_URL=mysql://localhost
    workers = 4
    thunder-lock = true
    show-config = true
    ;end of configuration

Fixes unbit#2097 (but *only* if libyaml is used).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

yaml/libyaml: Fix keys ignored after sequence, support non-indented sequence entries awelzel/uwsgi
2 participants
@awelzel @almahmoud