Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dispatcher utility #37

Merged
merged 18 commits into from
Sep 25, 2019
Merged

Conversation

mraspaud
Copy link
Member

Add the dispatcher utility.

client1:
  host: ftp://ftp.client1.com
  connection_parameters:
    credential_file: .netrc
  filepattern: '{platform_name}_{start_time}.{format}'
  directory: /input_data/{sensor}
  dispatch:
    - topics:
        - /level2/viirs
        - /level2/avhrr
      conditions:
        # key matches metadata items or provides default
        - product: [green_snow, true_color]
          sensor: viirs
        - product: [green_snow, overview]
          sensor: avhrr
          # special section "except" for negating
          except:
            platform_name: NOAA-15
    - topics:
        - /level3/cloudtype
      directory: /input/cloud_products
      conditions:
        - area: omerc_bb
          daylight: '<30'
          coverage: '>50'

@mraspaud mraspaud requested a review from pnuu September 11, 2019 17:38
@mraspaud mraspaud self-assigned this Sep 11, 2019
@codecov
Copy link

codecov bot commented Sep 12, 2019

Codecov Report

❗ No coverage uploaded for pull request base (master@358a759). Click here to learn what that means.
The diff coverage is 65.19%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master      #37   +/-   ##
=========================================
  Coverage          ?   29.82%           
=========================================
  Files             ?       12           
  Lines             ?     1750           
  Branches          ?        0           
=========================================
  Hits              ?      522           
  Misses            ?     1228           
  Partials          ?        0
Impacted Files Coverage Δ
trollmoves/server.py 0% <0%> (ø)
setup.py 0% <0%> (ø)
trollmoves/utils.py 64.47% <100%> (ø)
trollmoves/movers.py 28.04% <28.04%> (ø)
trollmoves/hooks.py 75% <75%> (ø)
trollmoves/dispatcher.py 89.67% <89.67%> (ø)
trollmoves/tests/test_dispatcher.py 98.06% <98.06%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 358a759...168697a. Read the comment docs.

Copy link
Member

@pnuu pnuu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Couple of typos in the docstrings

trollmoves/dispatcher.py Outdated Show resolved Hide resolved
trollmoves/dispatcher.py Outdated Show resolved Hide resolved
@pnuu
Copy link
Member

pnuu commented Sep 12, 2019

Oh, and inotify needs to be added to install_requires.

setup.py Show resolved Hide resolved
@mraspaud mraspaud marked this pull request as ready for review September 12, 2019 09:24
@pnuu
Copy link
Member

pnuu commented Sep 23, 2019

I'll give some more comments tomorrow morning.

trollmoves/dispatcher.py Outdated Show resolved Hide resolved
trollmoves/dispatcher.py Outdated Show resolved Hide resolved
@pnuu
Copy link
Member

pnuu commented Sep 24, 2019

Could there be a condition for filename patterns? I'm thinking about messages coming from trollmoves transfers, which might need additional transfers to servers without trollmoves client capabilities, like external customers. Although now that I look at those messages I think there might be enought information even without this feature 🤔

@mraspaud
Copy link
Member Author

I was hoping to be free from the filenames and just work with metadata...

@pnuu
Copy link
Member

pnuu commented Sep 25, 2019

Isn't filename (or uid/uri) also metadata? 😁 Anyway, this can also be handled by having specific topics.

@pnuu
Copy link
Member

pnuu commented Sep 25, 2019

LGTM, feel free to merge if you don't find anything to change.

@mraspaud
Copy link
Member Author

Thanks. I'll test my current dispatch chains with the modifications to make sure it still works

@mraspaud mraspaud merged commit 4d22fe7 into pytroll:master Sep 25, 2019
@mraspaud mraspaud deleted the feature-dispatcher branch September 25, 2019 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants