Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implements ideas described in #8 and #9 #10

Closed
wants to merge 8 commits into from

Conversation

dadrus
Copy link
Contributor

@dadrus dadrus commented Sep 16, 2017

WIP: This pull request is just to show how the implementation of this library can be enhanced to solve limitations/issues described in #8 and #9.

The main changes are:

  • EntityBuilder is not a singleton any more
  • EntityBuilder uses a new DSL object, which implements the create method, each time a new script file needs to be parsed and returns an EntityStore object, which holds all the references to the loaded objects
  • DataTestLoader is adopted to the new EntityBuilder API
  • tests adopt the API changes where applicable

Open topics, respectively topics to be discussed (from my point of view) are:

  • EntityBuilder became immutable, thus it is a real builder now. But the object it returns, the EntityStore (a new introduced interface) is not really immutable. It defines a clear method, which is actually not required (the entire object can just be dropped instead of using this method. Adding of new objects is anyway not possible to it). The background of this method was to retain the previous functionality of EntityBuilder.
  • unit tests for new classes need to be implemented. Since I don't know, whether the approach represented by this PR is acceptable for you, I have not implemented any tests which directly address the new classes. The corresponding functionality is however tested through already available tests.
  • Your code threw an Exception if different groovy dsl files had entity definitions for entities with same names. This is now done only for definitions from the same file (responsibility of EntityBuilder). Since DataTestLoader is now responsible for handling multiple files, the corresponding functionality has to be implemented here.
  • some code simplifications and alignments.
    • EntityBuildingException and NoSuchElementException are used for same purposes but in different methods. I would suggest to get rid of EntityBuildingException and use NoSuchElementException only.
    • Instead of checking for object existence after getting it from a map/list, one could use .withDefault { ... } on map/list definition. Things like this would made the code more clear.
    • made more use of Groovy features, which will made the code more clear as well.

What do you think?

… (handling of the defined DSL) and one responsible for the managing of the built objects in an EntityManager. New tests for the builder part implemented.
@dadrus
Copy link
Contributor Author

dadrus commented Nov 15, 2017

According to our discussion in #8 the commits above represent my view on how I would like to see the implementation of the library (not a WIP anymore :) ).

Following has been done:

  • the project has been split (package level) into three parts:
    • the builder package contains everything related to the DSL and the building of the entities.
    • the loader package contains everything which is related to script handling and persisting and deleting of the built entities
    • the util package contains the FileReader
  • Since the implementation in the builder package was not covered by the old tests, corresponding tests have been implemented
  • All old tests where adopted to the new structure
  • The pom.xml has been modified in the following way:
    • the failsafe plugin is now used to run integration tests. The one and only IT test which was there, was the Demo test. It has been renamed to TestDataLoaderIT to let failsafe plugin see it.
    • gmavenplus-plugin is now used to compile the groovy code
    • license plugin config has been changed to not to expect license header in resource files, like Jenkinsfile, pom.xml, etc.

What could be done in addition:

  • migrate tests for classes in loader package to groovy Spock. This will simplify the test code a lot. In addition it will become easier to use/generate DSL scripts without having predefined one in the resource folder. But this is maybe something for the future

@schnatterer
Copy link
Member

Hi @dadrus,

first of all, thanks for your efforts & your patience. As mentioned in #8 we were quite busy in the last months. Still, we hope to do better in the future.

Phew! That's quite a lot of changes in the diffs.
Also, the information is quite fragmented in #8, #9 and your two comments in this PR, where some parts of the first are made obsolete by the commits from yesterday (e.g. no more EntityStore). Here's what I think

Breaking changes.

The new package structure results in breaking API changes. This would force a major version change. While this is possible, we should take a moment to consider if this is necessary.

TestDataLoader

From: de.triology.blog.testdataloader.TestDataLoader;
To: de.triology.blog.testdata.loader.TestDataLoader;

We probably could avoid this one by just keeping the old de.triology.blog.testdataloader.TestDataLoader. That's a bit opposed to your restructuring of the package structure. But is this a good reason for breaking compatibility?

EntityBuilder, Singleton (#8)

In the data groovy scripts

import static de.triology.blog.testdataloader.EntityBuilder.create

offered code completion. With the PR this is not possible anymore, because a static import is no longer possible.
For me, this is a major drawback. Do you have code completion in these files in Eclipse at all? Did you read our article on the functionality and design of test-data-loader? It elaborates on the benefits of code completion. Maybe you could try it with IntelliJ to learn about the charms of this feature.

I understand that the singleton that realizes the code completion is directly opposed to your requirements from #8. However, getting rid of it also gets rid of code completion.
Do you have any ideas on how to find a compromise?

I'm also not quite sure about the use case of #8. Where would you need multiple EntityBuilders within one test and different scripts with entities that have the same name? Could you elaborate on this, or maybe write a test that illustrates this?

Minor issues

  • You reformatted the whole pom.xml to an indent of 2 spaces. Why? Is this is an eclipse-specific formatter? I created an .editorconfig in master. Does that work with eclipse?
  • Why did you remove the licenses from resources and pom.xml? Do you know about mvn license:format?
  • Please don't bloat pom.xml with eclipse stuff. It's been a while since I used eclipse, but I remember that eclipse offers another quickfix, that stores changes in the workspace and leaves the pom.xml as is.
  • You renamed Demo, which is referenced in README.md (and in the article mentioned above). However, I like your idea of running it as IT using failsafe.

#9

There a so much changes, I fail to see which ones are necessary for #9.

Where to go from here

So, after spending several hours reviewing this, I must admit that the PR is just a bit too huge for me.

Also, in general, you wrote a lot about what you changed but not always why. The why which is more important for me, because I cannot see it in the code. It's difficult to see from the commits to which issue they relate. I'm also convinced that you did a lot of changes that are not directly related to the two issues.

I think it would be more transparent and easier to review if we split this PR in multiple smaller PRs that clearly show what was changed and why.
Maybe one for your general optimizations, and two separate ones for issue #8 issue #9.

I hope that that this does not scare you off. We're definitely interested in your work and want to enable other tools such as your dadrus/jpa-unit to use test-data-loader.

@dadrus
Copy link
Contributor Author

dadrus commented Nov 19, 2017

Hi @schnatterer.

Thank you very much also from my side for this extensive review. The enormous work you did is exactly what I wanted to avoid by initially defining this PR as WIP and by asking you to verify whether the direction I took in my initial commit is inline with your strategy for this project.

I'll start my reply with more simple cases from your comments.

Minor Issues:

  • Code formatting in general, including xml files, like the pom.xml. To be honest, I have not taken .editorconfig into consideration while looking for formatter settings. I'm using the default community formatter settings enforced by Sonar in all my projects. In my case reformatting the modified code according to this settings is also configured as part of the eclipse save action. Since code formatting can also happen on a repository level during e.g a merge, I have not really taken care of it. I can easily apply your format settings as well.
  • license. The original license settings in the pom.xml were incomplete. Just running mvn clean install led to errors originating from the license plugin and telling that some files, among others src/test/resources/tests/FileReaderTest, Jenkinsfile and README.md do not include the mandatory license header. To avoid breaking any tests or builds, I've extended the license configuration not to cover resource files at all. This also includes the pom.xml.
  • pom.xml and eclipse stuff. I think you are talking about org.eclipse.m2e:lifecycle-mapping. On one hand it dies not have any effects on neither maven build, nor on other IDEs. On other hand I can easily remove this setting from the pom. I've added it because of gmavenplus-plugin which I've configured instead of the configuration you did to compile groovy files. I don't know why, but the Spock tests were not executed while groovy-eclipse-compiler configuration was in place.
  • Demo renaming. I should have changed the description in the README.md. Just forgot it. This change and the introduction of maven-failsafe-plugin go definitely beyond the scope of what was required according to Usage of TestDataLoader for different groovy scripts, which use same names for objects of same, or different types is not possible #8. I can reverse this particular change to have this PR more scope aware.

Breaking changes

  • package structure. Since EntityBuilder has now nothing in common with the initially developed EntityBuilder (it now defines its own public API) and the binary interface of TestDataLoader has changed as well, the version of the library (if you are going to accept the changes) should reflect this on a major version level. At least my understanding. By renaming the packages, thus restructuring the code and herewith the API of the library I wanted highlight the abovesaid changes.
  • EntityBuilder is not a singleton anymore. I had the code completion by statically importing EntityBuilder.create in eclipse as well. Having IDE support for code completion is however not limited to static imports, thus to singleton implementations if stateawareness is required. Otherwise all the DSLs around there would lack this definitely desired functionality. Both, IntelliJ as eclipse (and this is not limited to these two IDEs) implement support for DSL descriptors, which allow the DSL author to give the IDE a hint on how code within a (in our case groovy) script can be resolved so that code completion is available. The drawback of dsld is that these descriptors are IDE specific. Btw, this is also one of those topics we could have discussed after my initial PR. I am not in dsld, so I am now looking on how the descriptor needs to be defined to have code completion available again (at least in eclipse).
  • multiple EntityBuilders in one test. Here an example where one would use different scripts with same names. In case of jpa-unit one can seed the db by using the @InitialDataSets annotation for each test (implemented by a test methid). On one hand this annotation does accept multiple paths to script files to be used to seed the db. On the other hand the annotation can be applied on different test methods. That means the EntityBuilder will be used multiple times in one and/or in different tests. Having this class implementd as a singleton following issues arise:
    • usage of same script from different test methods will not be possible since the corresponding builder object will retain the state from the previous test run.
    • usage of different scripts by same test method which use same names for the built entities will not be possible as the name is already used. Why should different scripts use same names? They are independent. Why should their scopes interfere? One should be able to evolve indelendent scripts also independently. In case of the integration with jpa-unit, there is no need for names at all. As with EntityPersister, used by TestDataLoader the names are not used. Thus it should be possible to use the following distinct scripts in all above cases without any limitations:
create Customer, "customer", {
 name = "Foo"
 account = create Account, "account", {
    // whatever comes here
  }
}
create Customer, "customer", {
 name = "Moo"
 account = create Account, "account", {
    // whatever comes here
  }
}

To be honest, I would even like to see an additional create method without a name argument. The name could be then generated internally, e.g as a UUID. For that change, TestDataLoader would however need a further method the enable a retrieval of such "unnamed" entities.

One additional note regarding EntityBuilder as singleton. One could have it as a singleton and thus have to register and unregister the listeners for each test method execution. But this just doesn't sound right and will lead to some artificial complexity, which I would like to avoid. In general singletons should be just avoided whenever possible.

I hope, I could give you enough insights about the why. Regarding the PR split: the general PR for #8 will remain anyway a massive one. It will be same as today, but using your formatter rules (if I can use them in eclipse), without the package structure change and Demo renaming. The main issue however remains with the code completion requirement. Are you going to accept the loss of singleton and willing to go for dsld? If this approach is a no go for you, there is no need to invest furthet time and effort.

@schnatterer
Copy link
Member

Hi @dadrus,

thanks for your endurance and your detailed answer.

Let's try to get us to a pragmatic merge.

Most importantly, we're willing to get rid of the singleton. So let's make the breaking change from test-data-loader 0.x to 1.
We would need however

  • a brief migration guide from 0.x to 1.x (e.g. change package from x to y, etc.) in the README and
  • the dsld in the repo with a brief description on how to use them in the README.

As for the formatting, we've had a lot of discussions about formatting in different IDEs in the past.
There is a eclipse formatter plugin for IntelliJ, but it does not format 100% in the same way as eclipse.
Also, some developers just don't use auto format at all, because they say the code is hard to read after formatting (fluent APIs and such).

So our compromise is: Format the code you write code yourself (manually or with formatter). Don't use save actions, don't auto format complete files.
This has been working out for us very well.
BTW, auto formatting the pom.xml is the reason why the license check failed :-P
We should create a CONTRIBUTING.md that contains this. Since you're the first external committer, we unfortunately don't have that, yet.

So, how are we going to get this thing merged?
Are you willing to apply your changes again in a "as small as possible to fix #8"-PR or do you think reverting/changing/extending the things we talked about is easier?

Some final notes:
I just recognized a typo in the test packages: they are named "trilogy" instead of "triology".
Also, note that before releasing a 1.x version we will probably remove the "blog" from the package after your PR has been merged.
We're planning on releasing to maven central in December 2017 or Jan 2018.

@dadrus
Copy link
Contributor Author

dadrus commented Nov 23, 2017

Hi @schnatterer,

Cool. Let's try to approach it that way.

In the meantime I implemented the dsld for eclipse, you can find in the above referenced commit. According to the documentation of both eclipse and IntelliJ, the dsld file just needs to be on the classpath to have code completion working. Thus it can reside in the resulting jar. The end user doesn't need to do anything. In case of eclipse the dsld file needs to reside in a special folder dsld. IntelliJ does not impose that type of restriction. I don't use IntelliJ; because of this it will be problematic for me to develop a dsld for it. Will you be able to implement it on your own?

What about the following approach to make the review of the changes as easy as possible:

  • I'll submit a new PR in which every commit will have a logical history (I'll squash multiple commits into one which belong to a specific "feature")
  • The first set of commits will implement the changes in the EntityBuilder
  • New commits will follow, which will include the changes for Demo becoming an IT.
  • If the above will be ok for you, package structure change will follow.
  • Followed by the required descriptions in the README.

That way every commit will build on top of the other and you will have a meaningful history.

Or do you rather like each of these topics become a separate PR?

Regarding the CONTRIBUTING.md - take a look at one I have in the jpa-unit project. In my case nobody has contributed so far, so I don't have any experience, whether it covers all the relevant aspects. But based on our discussion, I think it might be a pretty good starting point:)

@dadrus
Copy link
Contributor Author

dadrus commented Nov 23, 2017

Please take a look at following commits. If this type of history is ok for you, I'll create a new PR which will include them.

@schnatterer
Copy link
Member

Hi @dadrus,

I had a short look at the new_master commits. This looks good. It seems much more structured to me now.
Thanks for your thorough work!

I'll have to inspect da324f7 in detail, but I can do this as part of the new PR.

So, yes please create a new PR (one is enough). I 'm glad to review (and hopefully merge) it by the end of next week.

@dadrus
Copy link
Contributor Author

dadrus commented Nov 25, 2017

replaced by #12

@dadrus dadrus closed this Nov 25, 2017
schnatterer added a commit that referenced this pull request Dec 18, 2017
Implements #8 according to our discussions in PR #10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants