Mutually exclude mainSourceFile of different configurations

[omerfirmak]

Unless explicitly added to sourceFiles, a configuration will exclude
other configurations' mainSourceFile's.

[dlang-bot]

Thanks for your pull request and interest in making D better, @omerfirmak! We are looking forward to reviewing it, and you should be hearing from a maintainer soon.
Please verify that your PR follows this checklist:

• My PR is fully covered with tests (you can see the coverage diff by visiting the details link of the codecov check)
• My PR is as minimal as possible (smaller, focused PRs are easier to review than big ones)
• I have provided a detailed rationale explaining my changes
• New or modified functions have Ddoc comments (with Params: and Returns:)

Please see CONTRIBUTING.md for more information.

---

If you have addressed all reviews or aren't sure how to proceed, don't hesitate to ping us with a simple comment.

[Geod24]

Nice! This needs a release note entry. Suggestion:

Dub will now automatically exclude mainSourceFile from other configurations

By default, Dub uses all files it can find under its sourcePaths.
However, a common pattern when dealing with multiple targets is to use
configurations to represent said targets. In the case those targets are executables,
users would be forced to add main files from other configurations to the
excludedSourceFiles list, or store the main in a different directory outside of
the sourcePaths.

To simplify this workflow, Dub will now exclude files listed in mainSourceFile
for other configuration. In case this is not desirable, the files need to be manually
added to the sourceFiles list.

Side note: When writing release notes, we always try to include the following 3 things:

• Previous situation;
• Reasoning for the change;
• Mitigation process (in case the user actually wants the old behavior);

[Geod24]

Just in case: https://github.com/dlang/dub/tree/master/changelog

[omerfirmak]

Just in case: https://github.com/dlang/dub/tree/master/changelog

Thanks! Added your suggestion to changelog.

[schveiguy]

A configuration's settings should not be affected by other configurations.

This has broken mysql-native. I'm not sure why (will ask the original maintainer), but it uses the package file in one configuration as the mainSourceFile.

With this change, how does one write a dub recipe file that works across multiple compilers? I'm assuming that "mainSourceFile" was there for a reason.

I get the idea behind the change, and it's a good idea. However, I think we can do this without breaking existing projects. Maybe introduce an "exclusiveMainSourceFile" or something?

[schveiguy]

Another artifact here is that dub is spitting out these exclusions into the local package cache, so even using a previous version of dub will not work once the cache has been set up by the 2.095 version.

[Geod24]

This has broken mysql-native. I'm not sure why (will ask the original maintainer), but it uses the package file in one configuration as the mainSourceFile.

First, sorry for breaking your library. We weren't aware of that. I'd recommend adding mysql-native to the project tester to prevent this from happening again. Looking at the git history and the file itself, I assume the usage was because of this warning, but it would be good to get the maintainer's word on it.

A configuration's settings should not be affected by other configurations.

While I usually agree, I think this is a (useful) concession to a common use-case, and the work-around for this is easy.
I also consider configurations to be independent units, although specialization of the "main" configuration, but unless dub starts to detect what source files are used based on mainSourceFile & co, this seemed like the best course of action.

With this change, how does one write a dub recipe file that works across multiple compilers? I'm assuming that "mainSourceFile" was there for a reason.

I'm not sure I follow why this change would make writing cross-compiler recipes harder ? Looking at your recipe, you just have to remove the offending line to make things work again. From what I can see of https://github.com/mysql-d/mysql-native/blob/27ea51f8b0c6aa41aab00fc288a2a1dec734905f/dub.sdl, half of the file is just working around dub limitations (such as wanting optional dependencies to actually work). And given that mainSourceFile does not accept platform customization, I really don't see the issue here.

I get the idea behind the change, and it's a good idea. However, I think we can do this without breaking existing projects. Maybe introduce an "exclusiveMainSourceFile" or something?

We had another change that was halted because it broke projects. Unfortunately it's pretty hard to know unless we either 1) do nothing or 2) have a comprehensive test-suite, hence my suggestion to add mysql-native to the project tester.
I don't think adding more fields with subtly different behaviors makes sense, it just adds complexity, pushes problem to the users, and create more risk of bugs when two features are used together.

IMO we should really come up with a list of needed changes for the near future and implement them in DUB 2.0. Because I really don't want to break anyone's workflow out there, but at the same time, no one is happy with their DUB workflow.

[schveiguy]

I'm not sure I follow why this change would make writing cross-compiler recipes harder ?

Not what I meant. What I meant was, if, for instance, I want to write a dub file that works with both dmd 2.095 and 2.094, dub has subtly changed the meaning of this directive between those 2 compilers. Changing the meaning (significantly) in one compiler release without deprecations or warnings is not good practice. Granted, dub can be installed separately from the compiler, but it's usually packaged in all of the installers.

Looking at the git history and the file itself, I assume the usage was because of this warning, but it would be good to get the maintainer's word on it.

I'd also recommend getting rid of that warning. The original maintainer said he can't really remember why he added it, but he thinks he added it to be consistent with rdmd, which requires you to name a "root" file from which all other things are imported. But that warning probably had something to do with it.

We had another change that was halted because it broke projects. Unfortunately it's pretty hard to know unless we either 1) do nothing or 2) have a comprehensive test-suite, hence my suggestion to add mysql-native to the project tester.
I don't think adding more fields with subtly different behaviors makes sense, it just adds complexity, pushes problem to the users, and create more risk of bugs when two features are used together.

I think this change has to be reverted. Changing the meaning breaks the intent of previous recipe files (even if that intent was misplaced). I shouldn't have to add a project to some CI tester to ensure the language doesn't change from under my feet, that's not a way to develop a widely used tool. At least a deprecation cycle is in order.

I think the original directive was poorly defined to begin with, and should be deprecated in favor of something more spelled-out. Thinking about it, it really should be "excludeInOtherConfigs", which conveys the actual effect, rather than some fuzzy "this is where the main file lives" to be acted on (poorly) by dub test (the goal of which, by the way, is moot now that unittests do not run main by default).

Oh, and this should NOT be done the way it was in any case -- injecting excludedSourceFiles directives into the parsed dub config means when you regurgitate that file (when dub downloads the config from the repo and spits out the dub.json file), you are changing the original directives unconditionally. For instance, this meant that if you use 2.095 dub to build your project, realize that it doesn't link, and then revert to 2.094, the 2.094 dub now sees those excludedSourceFiles directives and has the same problem. It shouldn't depend on which dub I use to download a version of a package.

[andre2007]

I just tried the new dub version and this change breaks all of my repositories I wrote at work.

I am used to the pattern to have in app.d a version statement version(unittest) which calls DUnit testrunner, else call the productive main. In dub.json app.d is set as mainSourceFile for configuration unittest.

Due to this pr changes, it fails now with linker errors.

The solution seems to be to layout the test main to a separate testapp.d . But this not really fortunate ):

It is even more worse. I am somehow experienced with dub and therefore know why there are linker failures. But for less dub experienced user they have no chance to find out why linker errors occurs.

[PetarKirov]

@schveiguy @andre2007 I discussed with @Geod24 and we think the best course of action is to revert this change, but more specifically to revert to the old behavior and make the new one opt-in. What do you guys think?

[andre2007]

Yes, I agree

[s-ludwig]

Thinking about it, it really should be "excludeInOtherConfigs", which conveys the actual effect, rather than some fuzzy "this is where the main file lives" to be acted on (poorly) by dub test (the goal of which, by the way, is moot now that unittests do not run main by default).

Actually, the main reason for the existence of this directive is to support rdmd based builds, dub test came later and just conveniently used this piece of information for that workaround.

[s-ludwig]

BTW, @Geod24, I don't agree with the push for 2.0:

1. It doesn't really matter in practice whether a minor or a major version silently breaks recepies, most likely it simply fragments people/packages into those staying with 1.x.x and those that upgrade to 2.x.x - definitely not something we want
2. There are lots of opportunities for improvement that require absolutely no breaking changes, I really think those should be tackled first before considering a breaking alternative with questionable benefits

but at the same time, no one is happy with their DUB workflow.

I hope we are on the same page that you will really only ever see the complaints, but not the cases of people simply being happy. I have no idea how the actual percentages are for the various levels of problem severity, but I certainly know that this kind of black/white painting doesn't help overall. Personally, of course I also have some places with workarounds or solutions that could be more nicely integrated, but overall I'm pretty happy and I know that non-breaking additions, such as extension support or parallel builds, will get rid of at least my remaining issues.

[Geod24]

Not what I meant. What I meant was, if, for instance, I want to write a dub file that works with both dmd 2.095 and 2.094, dub has subtly changed the meaning of this directive between those 2 compilers. Changing the meaning (significantly) in one compiler release without deprecations or warnings is not good practice. Granted, dub can be installed separately from the compiler, but it's usually packaged in all of the installers.

Well, you're just describing breaking changes here. This has nothing to do with compiler version.

I'd also recommend getting rid of that warning.

Yep, I wanted to look into this.

I think this change has to be reverted.

I agree.

I shouldn't have to add a project to some CI tester to ensure the language doesn't change from under my feet, that's not a way to develop a widely used tool.

That is not at all what I was suggesting. I suggested that, as a mature and widely used project, mysql-native would be a good fit for Buildkite, as it would allow to widen the coverage for D core development. It doesn't mean we can freely make breaking change, it just means we don't end up in a situation where a breaking change is merged in the first place.

At least a deprecation cycle is in order.

In case that wasn't clear, the breaking change was not anticipated (and yes, in hindsight, it should have been, but hindsight is 20/20.

I think the original directive was poorly defined to begin with, and should be deprecated in favor of something more spelled-out. Thinking about it, it really should be "excludeInOtherConfigs", which conveys the actual effect, rather than some fuzzy "this is where the main file lives" to be acted on (poorly) by dub test (the goal of which, by the way, is moot now that unittests do not run main by default).

I'm going to strongly disagree here. We have already too many ways to do things, and adding an extra field for a slightly-different behavior is just bad design. The core issue that this PR was trying to solve was that it's still convoluted to have multiple projects in the same source tree.

Oh, and this should NOT be done the way it was in any case [...]

That's a good point, thanks for the feedback.

[Geod24]

1. It doesn't really matter in practice whether a minor or a major version silently breaks recepies, most likely it simply fragments people/packages into those staying with 1.x.x and those that upgrade to 2.x.x - definitely not something we want

You're making an argument against any change in general here. We're stuck in a situation where we have an application, which is also a library, that is extremely resistant to change as it must not break compatibility. We end up with things being piled on top of one another with no coherence. We have two package formats, configurations vs sub-packages (and to some extent, build modes), a few ways to override package providers (such as file system providers vs add-path vs add-local vs add-override), etc...

What I'm suggesting is defining a supported set of features, for a supported period, and every once in a while (e.g. 2 years), simply revise things and improve the status quo. npm puts out a major version every 6 months, with LTS every other release, and their user base is order of magnitude bigger.

There are lots of opportunities for improvement that require absolutely no breaking changes, I really think those should be tackled first before considering a breaking alternative with questionable benefits

I'd be happy if you provide a list. Because the improvement that mattered to us are:

• This one, which is going to be reverted;
• Make sourcePaths work as documented #2041 : Not going to be merged because breaking change;
• Allow custom dflags to be defined for dependencies #2040 : Which is really a hack to avoid making a breaking change;
• Use DC env variable as the default compiler #2047 : Which could be argued to be a break change;
• Dont export DFLAGS & LFLAGS by default #2048 : Ditto

We also had #2035 which is also a change of behavior, but we got this bug reported so many times that I don't think anyone would dare to argue against it.

I hope we are on the same page that you will really only ever see the complaints, but not the cases of people simply being happy. I have no idea how the actual percentages are for the various levels of problem severity, but I certainly know that this kind of black/white painting doesn't help overall. Personally, of course I also have some places with workarounds or solutions that could be more nicely integrated, but overall I'm pretty happy and I know that non-breaking additions, such as extension support or parallel builds, will get rid of at least my remaining issues.

We are on the same page. I'm sorry if you felt I was being too negative towards dub. It's no easy feat to maintain a language's package manager for as long as you have been, and I don't think you get enough credit for. However, I do think you have a workflow that suits the tool you built. Not everyone does (I know my workflow for instance is pretty unique), and I think by polling the combination of people's workflow, we could end up with a tool that is much more satisfying to use than it is today.

But to integrate the feedback we got, we'll need to make some changes which will break users' recipe in subtle ways. In the case of this specific PR, there is a backward compatible change one can do to their recipe to make it work both with the old and new dub. And as mentioned, both use cases are here to work around previous limitations.

[schveiguy]

revert to the old behavior and make the new one opt-in

I agree. But what does this mean? A command line switch? A new directive? I prefer a new directive.

[schveiguy]

Actually, the main reason for the existence of this directive is to support rdmd based builds, dub test came later and just conveniently used this piece of information for that workaround.

Thanks for this bit of history, it makes things clearer to me. The mysql-native maintainer before me said this was probably due to using rdmd in some cases.

But still, the idea that dub test excludes the whole file is somewhat problematic. However, that's existing behavior, and people should already have dealt with that.

[schveiguy]

Well, you're just describing breaking changes here. This has nothing to do with compiler version.

dub ships with the compiler. In fact, on my mac, I use dvm to manage compilers, and it uses the dub that installs with that compiler.

True, it's possible to use dub as a separate installation. Not many do that though.

[s-ludwig]

It doesn't really matter in practice whether a minor or a major version silently breaks recepies, most likely it simply fragments people/packages into those staying with 1.x.x and those that upgrade to 2.x.x - definitely not something we want

You're making an argument against any change in general here.

What I meant is that making any sorts of backwards compatibility "cuts" in the form of a major version has the risk of people locking in to the old version, because it is more convenient or the only possibility (external dependencies). Shipping with the compiler helps of course, but also makes the problem worse when having to support multiple compiler versions.

Also, "breaking change" is something that in the end needs to be defined in terms of practical risk (probability and severity) with some threshold, otherwise we would be at version 25.0.0 by now. At some point there will be a 2.0.0, no doubt, but even then I'd strongly prioritize backwards compatibility w.r.t. the package recipe. That may mean that some compatibility code needs to be kept around across major versions, but packages that declare a minimum dub version for example would run with only the new semantics.

We're stuck in a situation where we have an application, which is also a library, that is extremely resistant to change as it must not break compatibility. We end up with things being piled on top of one another with no coherence. We have two package formats, configurations vs sub-packages (and to some extent, build modes), a few ways to override package providers (such as file system providers vs add-path vs add-local vs add-override), etc...

• Two packages formats: Unfortunate and a topic I'd rather not talk about, but not really an issue from an implementation POV and not much we can realistically do about it anymore, except for making it worse by adding a third format
• Sub packages are a distinct feature - they have a different purpose and work in a different way, even thought you may be able to formulate a sub set of problems with either of them. But you could throw in other things in then, too. The distinction is one of the things that needs to be clarified in a proper introductory documentation.
• add-path/add-local are not and are not meant to specify package providers. Instead they are meant to support multi-repository projects, where working copies of each project are checked out. File system package suppliers have been implemented by someone at some point for reasons unknown to me, and I would be fine with deprecating them for 2.0.0 unless they are actually in use. But they are really not adding any significant complexity either and are certainly not suited to justify a 2.x release.

Apart from what's there, what possible features on the radar do you think would be incoherent with the current feature set?

What I'm suggesting is defining a supported set of features, for a supported period, and every once in a while (e.g. 2 years), simply revise things and improve the status quo. npm puts out a major version every 6 months, with LTS every other release, and their user base is order of magnitude bigger.

If that means breaking existing features every two years then I simply don't think this is a good idea. With appropriate deprecation periods and a compelling justification for removing a feature, this is fine, but the trade-off needs to result in a net positive for the ecosystem as a whole. Just breaking or removing little things is hardly solving the real issues. And certainly not removing core features such as configurations or sub packages.

There are lots of opportunities for improvement that require absolutely no breaking changes, I really think those should be tackled first before considering a breaking alternative with questionable benefits

I'd be happy if you provide a list. Because the improvement that mattered to us are:

This one, which is going to be reverted;
#2041 : Not going to be merged because breaking change;

Commented on the issue. This is obviously not critical, but makes sense to be fixed long term.

#2040 : Which is really a hack to avoid making a breaking change;

The way this is supposed to work in the usual cases is that DUB knows which flags need to or may be propagated and acts accordingly. This is currently works for build options (inheritedBuildOptions), but should be extended as necessary. I wouldn't mind a manual way to do this, although the recursion aspect of #2040 seems to be difficult to define right. What would be the alternative breaking change?

#2047 : Which could be argued to be a break change;

I think overall this still lies in the non-breaking spectrum.

#2048 : Ditto

This is definitely a breaking change, especially since this is officially documented and probably should be reverted. The overlap of input and output variables is indeed an issue, though, so a transition starting with the documentation and a deprecation message when used directly in a build commands directive is definitely desirable.

We also had #2035 which is also a change of behavior, but we got this bug reported so many times that I don't think anyone would dare to argue against it.

Agreed, the risk of this breaking anything seems to be low and it definitely was a stepping stone.

I hope we are on the same page that you will really only ever see the complaints, but not the cases of people simply being happy. I have no idea how the actual percentages are for the various levels of problem severity, but I certainly know that this kind of black/white painting doesn't help overall. Personally, of course I also have some places with workarounds or solutions that could be more nicely integrated, but overall I'm pretty happy and I know that non-breaking additions, such as extension support or parallel builds, will get rid of at least my remaining issues.

We are on the same page. I'm sorry if you felt I was being too negative towards dub. It's no easy feat to maintain a language's package manager for as long as you have been, and I don't think you get enough credit for. However, I do think you have a workflow that suits the tool you built. Not everyone does (I know my workflow for instance is pretty unique), and I think by polling the combination of people's workflow, we could end up with a tool that is much more satisfying to use than it is today.

But to integrate the feedback we got, we'll need to make some changes which will break users' recipe in subtle ways. In the case of this specific PR, there is a backward compatible change one can do to their recipe to make it work both with the old and new dub. And as mentioned, both use cases are here to work around previous limitations.

I agree with all of this, except for "But to integrate the feedback we got, we'll need to make some changes which will break users' recipe in subtle ways" as a generalization. Of course any change that is visible from the outside has the potential to result in subtle or obvious breakage, so every change will always be a judgment call in that regard. But even if the direct change is deemed breaking, usually there is an acceptable transition to the desired behavior and I think most open changes that have a high impact are well outside of the breaking territory anyway.

Just for this particular issue it is obvious that there is a real chance that this will mess with existing code, it's just a convenience feature, and I agree with Steve that configuration directives should not interfere with each other, as it adds an unnecessary element of surprise.

[s-ludwig]

But still, the idea that dub test excludes the whole file is somewhat problematic. However, that's existing behavior, and people should already have dealt with that.

I think with Druntime now supporting to skip the execution of main in unit test mode, we can actually just remove that logic, as well as the main() defined in the generated test main file.

[schveiguy]

So is there a plan to revert this and reintroduce via something else? Just asking, because mysql-native is broken with the latest dub until this is resolved.

[schveiguy]

Just starting to get into cleaning out the mysql-d cobwebs. Has this change been reverted or fixed?

[andre2007]

Users still having issues due to this change https://forum.dlang.org/post/menbaobefkvwrychtgar@forum.dlang.org