Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for query requesting information about partition step distribution across workers #2025

Merged
merged 2 commits into from
Feb 2, 2018

Conversation

jtfmumm
Copy link
Contributor

@jtfmumm jtfmumm commented Feb 2, 2018

You can now send an external message to a running
Wallaroo cluster and receive a response with a json
payload containing the location of all steps (by id)
in each partition (state and stateless) across all
workers.

This commit also includes an update to external_sender
adding the ability to send partition queries and print
the response to stdout.

Closes #2024

Our initial query protocol for checking the layout
of state and stateless partitions across workers will
use the json format.
You can now send an external message to a running
Wallaroo cluster and receive a response with a json
payload containing the location of all steps (by id)
in each partition (state and stateless) across all
workers.

This commit also includes an update to external_sender
adding the ability to send partition queries and print
the response to stdout.
@jtfmumm
Copy link
Contributor Author

jtfmumm commented Feb 2, 2018

To test this, start up any application with a partition (word count is best because it has both state and stateless partitions). Run it with multiple workers and/or grow the cluster. Use the external_sender tool with the following command (provided you're listening on an external channel of 127.0.0.1:5050):

./external_sender -e 127.0.0.1:5050 -t partition-query

You should see a json map printed out to stdout with the following structure:

{partition_type: {partition_id: {worker: step_ids}}}

You can query after grow events to see that it's updating (and doing so, you'll quickly discover #2026 as I did).

@SeanTAllen
Copy link
Contributor

@john this seems like something that should have some unit tests.

@jtfmumm
Copy link
Contributor Author

jtfmumm commented Feb 2, 2018

@SeanTAllen There are unit tests around the actual json encoding/decoding. Which part do you mean?

@SeanTAllen
Copy link
Contributor

Sorry I completely missed those.

@jtfmumm jtfmumm merged commit bb87c61 into master Feb 2, 2018
@jtfmumm jtfmumm deleted the query-protocol branch February 2, 2018 15:05
wallaroolabs-wally added a commit that referenced this pull request Feb 2, 2018
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants