Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-160] Import NexMark integration tests. #99

Closed
wants to merge 14 commits into from
Closed

[BEAM-160] Import NexMark integration tests. #99

wants to merge 14 commits into from

Conversation

mshields822
Copy link
Contributor

BUILDS ON #85

This is an initial import of the 'NexMark' integration tests.
All Google Dataflow specific code has been isolated.
It could do with more cleanup, but wanted to get it out in the sunlight.

@davorbonaci
Copy link
Member

R: @davorbonaci

@davorbonaci
Copy link
Member

Build fix, please?

Caused by: hudson.plugins.git.GitException: Command "git merge 30585ad3529b0799043d3cef9d2824337b631baa" returned status code 1:
stdout: Auto-merging sdks/java/core/pom.xml
Auto-merging pom.xml
CONFLICT (content): Merge conflict in pom.xml
Automatic merge failed; fix conflicts and then commit the result.

@davorbonaci
Copy link
Member

I'm not sure between integration/java and sdks/java/integration. I think I prefer the second one a bit more. Even more, we might want to do sdks/java/integration/nexmark.

@mshields822
Copy link
Contributor Author

I don't think we should take the build seriously until the pubsub pull
request is in since this builds on top. I've put it here so I can point to
it this fri and get the early stuff done.

Not really fussed about the directory name, whatever seems best is fine.
But it will depend on at least one runner so putting under sdks/... seems
misleading. Thought about examples/... but the code quality there is
significantly higher :-)

On Wed, Mar 30, 2016 at 4:04 PM, Davor Bonaci notifications@github.com
wrote:

I'm not sure between integration/java and sdks/java/integration. I think
I prefer the second one a bit more. Even more, we might want to do
sdks/java/integration/nexmark.


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#99 (comment)

@kennknowles
Copy link
Member

+1 to Mark's last point. When modules are all split, we are going to want this dependency direction:

integration tests → runner(s) → SDK

Otherwise we have a cycle between runners and SDK. Is there another option?

@kennknowles
Copy link
Member

Pinging because this has been sitting for a while. Maybe rebase to kick the tests against the new code layout?

@mshields822
Copy link
Contributor Author

I'm still chipping away on this in the background. I've left it as a
dormant PR since I've been pointing customers to it (esp for Q10). Will
ping back when actually ready, and we need the 3 pubsub ones to go in first.

On Fri, Apr 8, 2016 at 8:46 AM, Kenn Knowles notifications@github.com
wrote:

Pinging because this has been sitting for a while. Maybe rebase to kick
the tests against the new code layout?


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#99 (comment)

iemejia referenced this pull request in iemejia/beam Jan 12, 2018
mareksimunek pushed a commit to mareksimunek/beam that referenced this pull request May 9, 2018
* #! - eclipse compilation

* apache#40 - Sort operator with in-memory RSBK sorting

* apache#40 - Spark SortTranslator

* apache#40 - Flink SortTranslator

* apache#40 - RangePartitioning

* apache#40 - PR#99
dmvk pushed a commit to dmvk/beam that referenced this pull request May 15, 2018
* #! - eclipse compilation

* apache#40 - Sort operator with in-memory RSBK sorting

* apache#40 - Spark SortTranslator

* apache#40 - Flink SortTranslator

* apache#40 - RangePartitioning

* apache#40 - PR#99
mareksimunek pushed a commit to seznam/beam that referenced this pull request Jul 9, 2018
* #! - eclipse compilation

* #40 - Sort operator with in-memory RSBK sorting

* #40 - Spark SortTranslator

* #40 - Flink SortTranslator

* #40 - RangePartitioning

* #40 - PR#99
dmvk pushed a commit to seznam/beam that referenced this pull request Aug 17, 2018
* #! - eclipse compilation

* #40 - Sort operator with in-memory RSBK sorting

* #40 - Spark SortTranslator

* #40 - Flink SortTranslator

* #40 - RangePartitioning

* #40 - PR#99
dmvk pushed a commit to seznam/beam that referenced this pull request Oct 5, 2018
* #! - eclipse compilation

* #40 - Sort operator with in-memory RSBK sorting

* #40 - Spark SortTranslator

* #40 - Flink SortTranslator

* #40 - RangePartitioning

* #40 - PR#99
hengfengli pushed a commit to hengfengli/beam that referenced this pull request Mar 21, 2022
Fixes test cleanup, the change stream needs to be dropped before the
table, otherwise the ddl update fails.

Improves reliability of the integration test by inserting the records
before the pipeline starts and querying from 1 second before / after.
alnzng added a commit to alnzng/beam that referenced this pull request Jun 12, 2023
(cherry-pick)Refactor key coder for Samza portable runner (apache#27095)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants