Make the solver aware of pkg-config constraints #3023

garetxe · 2016-01-03T11:13:10Z

This pull request fixes #3016 (it does not entirely fix #335, but it goes in that direction). As explained in more detail in #3016, having the solver aware of available pkg-config dependencies is important for bindings for recent versions of external APIs.

The implementation is fairly similar to that in #2873, the only noteworthy point is that on startup we scan the list of all installed pkg-config packages in order to obtain their versions. Obtaining the list of installed packages is straightforward (pkg-config --list-all), based on this list we then call pkg-config --modversion pkg1 pkg2....

If there are a lot of packages the commandline can get somewhat long (4-10 KB), but this should not be a problem in modern operating systems. An alternative would be to call pkg-config --modversion repeatedly for each package, but from quick testing this makes things much slower (0.02s vs 0.5s in my system).

ezyang · 2016-01-03T13:56:42Z

Have you tested on Windows? There's a fairly small maximum command line length which, if your command lines are really 10KB large, you are almost definitely going to hit.

garetxe · 2016-01-03T15:26:34Z

No, unfortunately I don't have access to a Windows machine where I could test this.

Nevertheless from reading the docs at
https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx
I was under the impression that the limit for CreateProcess in Windows is 32K, which should be fine.

I believe the limit you mention is for CMD.EXE, but from my reading of the code getDbProgramOutput does not use that, it directly uses CreateProcess under the hood. But I could well be wrong, one should check.

But using such a long command line is certainly somewhat brittle... I am not sure about the preferred design. I could easily chunk the calls to pkg-config --modversion into groups of 100, say. This should be well below the limit, I think. (For reference: in my system, Fedora 22, pkg-config --list-all lists 408 packages, for a command line to pkg-config --modversion of 5194 bytes)

ezyang · 2016-01-03T17:06:02Z

You are right, I misstated the limit. I think this should be fine!. LGTM.

(It would be nice if we could get some tests, but pkg-config seems a bit too difficult to actually mock in a useful way.)

On January 3, 2016 10:26:35 AM EST, "iñaki García Etxebarria" notifications@github.com wrote:

No, unfortunately I don't have access to a Windows machine where I
could test this.

Nevertheless from reading the docs at
https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx
I was under the impression that the limit for CreateProcess in
Windows is 32K, which should be fine.

I believe the limit you mention is for CMD.EXE, but from my reading
of the code getDbProgramOutput does not use that, it directly uses
CreateProcess under the hood. But I could well be wrong, one should
check.

But using such a long command line is certainly somewhat brittle... I
am not sure about the preferred design. I could easily chunk the calls
to pkg-config --modversion into groups of 100, say. This should be
well below the limit, I think. (For reference: in my system, Fedora 22,
pkg-config --list-all lists 408 packages, for a command line to
pkg-config --modversion of 5194 bytes)

Reply to this email directly or view it on GitHub:
#3023 (comment)

Sent from my Android device with K-9 Mail. Please excuse my brevity.

ezyang · 2016-01-03T23:43:52Z

CC @edsko and @kosmikus

garetxe · 2016-01-04T01:24:48Z

You are right, I misstated the limit. I think this should be fine!. LGTM.

Great, thanks for taking a look!

(It would be nice if we could get some tests, but pkg-config seems a bit too difficult to actually mock in a useful way.)

Indeed it would be nice. I took a look to the testing code in #2873, but I need to spend a bit more time with it to understand how that code works. In any case if you think that the basic approach to modification of the solver in the patch is OK I would be happy to write some tests, trivial as they may end up being (better than nothing at all, I suppose).

garetxe · 2016-01-04T02:46:33Z

The fact that in some (extremely unlikely, as discussed above, but possible in principle) cases the initial population of the pkg-config database may potentially fail was bothering me a little, so d46e363 addresses this by switching off the pkg-config check in the solver when this happens.

So in the normal case we have the check, but we revert to the previous situation if readPkgConfigDb fails.

ezyang · 2016-01-04T02:53:07Z

Sweet. Maybe make the error message be a little clearer about what the failure means? (Cabal will continue without solving for pkg-config constraints.)

garetxe · 2016-01-04T04:39:45Z

Sweet. Maybe make the error message be a little clearer about what the failure means? (Cabal will continue without solving for pkg-config constraints.)

Sure, done!

I also added some unit tests, and squashed all changes into a single commit, hopefully this makes review a little bit easier.

garetxe · 2016-01-09T15:46:15Z

There were some merge conflicts against HEAD, so I have rebased the patch.

Is there anything else you think I should improve in the patch? I would be happy to do so.

23Skidoo · 2016-01-11T16:16:54Z

@garetxe

I think I'll postpone merging this PR until the 1.24 branch is created. The 1.24 release is imminent and I don't want to introduce such large changes. Hope it's okay with you.

garetxe · 2016-01-11T17:48:39Z

@23Skidoo Sure, that is very reasonable! As long the functionality gets in eventually in some form I am happy :)

dcoutts · 2016-01-14T19:49:22Z

I've not reviewed the code, but in principle this is defiantly a good thing. I'd like to get the solver to take into account more of these things. The big one will be build-tools.

BardurArantsson · 2016-01-15T06:11:40Z

@23Skidoo Unless the branch really is imminent could you perhaps add a label to the issues which you want to postpone until post-branch? (Just so that I and others don't accidentally merge stuff if we miss one of your comments.)

23Skidoo · 2016-01-15T13:00:27Z

@BardurArantsson Sure, will do.

BardurArantsson · 2016-01-16T14:01:15Z

@23Skidoo Great, thanks!

kosmikus · 2016-03-05T06:24:16Z

I've briefly looked at the part of the PR that actually makes changes to the modular solver, and these look generally ok to me.

BardurArantsson · 2016-03-05T09:47:40Z

I'm going to just merge this based on the LGTMs, my own (possibly naïve) LGTM and the fact that 1.24 has been branched.

BardurArantsson · 2016-03-05T09:48:30Z

(Uh, hang on... I can't merge it -- there are conflicts. My bad.)

garetxe · 2016-03-05T16:51:24Z

I just rebased, it should merge cleanly now. Thanks for the reviews!

garetxe · 2016-03-05T17:25:19Z

Mmm, looking to the automatic checks that just ran one of them (appveyor) is failing, apparently due to the fact that it is in running in Windows, so it cannot find pkg-config, and an unexpected warning is printed (cabal itself should work fine even if pkg-config is missing, in this case we simply revert to the behavior previous to the patch). But if I add the warning to the expected output of the test the tests will fail in linux, or in general when pkg-config is available...

Is there a way of having pkg-config available for the Windows testing machines? Or a way of modifying the test so that the warning is ignored?

23Skidoo · 2016-03-05T19:38:37Z

@garetxe Windows binaries for pkg-config and its dependencies are available here; installation instructions are available [here(http://stackoverflow.com/a/22363820/242169). We can change the appveyor.yml script to download and install them during the setup phase.

23Skidoo · 2016-03-05T19:42:30Z

cabal-install/Distribution/Client/PkgConfigDb.hs

+        -- too long command line).
+        ioErrorHandler :: IOException -> IO PkgConfigDb
+        ioErrorHandler e = do
+          info verbosity ("Failed to query pkg-config, Cabal will continue without solving for pkg-config constraints: " ++ show e)


Please respect the 80-col line length limit.

Thanks, fixed!

23Skidoo · 2016-03-05T19:43:50Z

LGTM as well.

garetxe · 2016-03-05T19:44:51Z

@23Skidoo Thanks!

In any case I just updated the patch in the pull request to use info instead of warn when telling the user that pkg-config was not found, so it only appears in verbose mode. This should make the test pass.

This was probably the right thing to do in the first place anyway. It could be a bit annoying to see the message every time if one was interested in using cabal without having pkg-config around, which does make sense.

When solving, we now discard plans that would involve packages with a pkgconfig-depends constraint which is not satisfiable with the current set of installed packages (as listed by pkg-config --list-all). This fixes haskell#3016. It is possible (in principle, although it should be basically impossible in practice) that "pkg-config --modversion pkg1 pkg2... pkgN" fails to execute for various reasons, in particular because N is too large, so the command line becomes too long for the operating system limits. If this happens, revert to the previous behavior of accepting any install plan, regardless of any pkgconfig-depends constraints.

23Skidoo · 2016-03-05T19:53:55Z

Yes, I agree that using info is the right approach.

Make the solver aware of pkg-config constraints

23Skidoo · 2016-03-05T21:43:45Z

Merged, thanks!

garetxe · 2016-03-05T21:48:49Z

Great, thanks a lot!

garetxe force-pushed the master branch from 29fef36 to 4f645a2 Compare January 3, 2016 11:51

garetxe force-pushed the master branch from e9f8e48 to 8cc3241 Compare January 4, 2016 04:19

garetxe force-pushed the master branch 5 times, most recently from 8050fe7 to 3b465c5 Compare January 9, 2016 14:13

23Skidoo added the post-1.24 label Jan 15, 2016

23Skidoo added the cabal-install: solver label Jan 16, 2016

23Skidoo assigned kosmikus Jan 16, 2016

garetxe force-pushed the master branch 2 times, most recently from 34fa6e1 to 3af1c19 Compare March 5, 2016 16:35

garetxe force-pushed the master branch from 3af1c19 to aa2ec04 Compare March 5, 2016 19:38

23Skidoo reviewed Mar 5, 2016
View reviewed changes

garetxe force-pushed the master branch from aa2ec04 to c72aa8d Compare March 5, 2016 19:47

23Skidoo added a commit that referenced this pull request Mar 5, 2016

Merge pull request #3023 from garetxe/master

6c45ec9

Make the solver aware of pkg-config constraints

23Skidoo merged commit 6c45ec9 into haskell:master Mar 5, 2016

23Skidoo removed the post-1.24 label Mar 5, 2016

23Skidoo added this to the cabal-install 1.26 milestone Mar 5, 2016

ezyang modified the milestones: cabal-install 2.0, 2.0 (planned for next feature release) Sep 6, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make the solver aware of pkg-config constraints #3023

Make the solver aware of pkg-config constraints #3023

garetxe commented Jan 3, 2016

ezyang commented Jan 3, 2016

garetxe commented Jan 3, 2016

ezyang commented Jan 3, 2016

ezyang commented Jan 3, 2016

garetxe commented Jan 4, 2016

garetxe commented Jan 4, 2016

ezyang commented Jan 4, 2016

garetxe commented Jan 4, 2016

garetxe commented Jan 9, 2016

23Skidoo commented Jan 11, 2016

garetxe commented Jan 11, 2016

dcoutts commented Jan 14, 2016

BardurArantsson commented Jan 15, 2016

23Skidoo commented Jan 15, 2016

BardurArantsson commented Jan 16, 2016

kosmikus commented Mar 5, 2016

BardurArantsson commented Mar 5, 2016

BardurArantsson commented Mar 5, 2016

garetxe commented Mar 5, 2016

garetxe commented Mar 5, 2016

23Skidoo commented Mar 5, 2016

23Skidoo Mar 5, 2016

garetxe Mar 5, 2016

23Skidoo commented Mar 5, 2016

garetxe commented Mar 5, 2016

23Skidoo commented Mar 5, 2016

23Skidoo commented Mar 5, 2016

garetxe commented Mar 5, 2016

Make the solver aware of pkg-config constraints #3023

Make the solver aware of pkg-config constraints #3023

Conversation

garetxe commented Jan 3, 2016

ezyang commented Jan 3, 2016

garetxe commented Jan 3, 2016

ezyang commented Jan 3, 2016

ezyang commented Jan 3, 2016

garetxe commented Jan 4, 2016

garetxe commented Jan 4, 2016

ezyang commented Jan 4, 2016

garetxe commented Jan 4, 2016

garetxe commented Jan 9, 2016

23Skidoo commented Jan 11, 2016

garetxe commented Jan 11, 2016

dcoutts commented Jan 14, 2016

BardurArantsson commented Jan 15, 2016

23Skidoo commented Jan 15, 2016

BardurArantsson commented Jan 16, 2016

kosmikus commented Mar 5, 2016

BardurArantsson commented Mar 5, 2016

BardurArantsson commented Mar 5, 2016

garetxe commented Mar 5, 2016

garetxe commented Mar 5, 2016

23Skidoo commented Mar 5, 2016

23Skidoo Mar 5, 2016

Choose a reason for hiding this comment

garetxe Mar 5, 2016

Choose a reason for hiding this comment

23Skidoo commented Mar 5, 2016

garetxe commented Mar 5, 2016

23Skidoo commented Mar 5, 2016

23Skidoo commented Mar 5, 2016

garetxe commented Mar 5, 2016