nixpkgs stack uses different hpack than upstream binary release

[googleson78]

Describe the bug

The stack that's currently in nixpkgs uses hpack-0.35.2, the official stack binary release uses hpack-0.35.1

For more details, I think the core issue is basically the same as #84820.

I think the fix is to change the version of hpack which the stack in nixpkgs is built with from 0.35.2 (which is the version of the hpack package in nixpkgs) to 0.35.1.

Notify maintainers

@cdepillabout @expipiplus1 @maralorn @sternenseemann

[googleson78]

It would also be great to have some automation to make sure this doesn't regress. I guess that would involve downloading an official matching version stack binary and checking what the hpack version is there (via -v), if there is no better way to get the upstream hpack version.

[sternenseemann]

In what way does this issue present? Supposedly stack has become more resistant to this problem: commercialhaskell/stack#5210

It seems ridiculous that we'd have to track what stack binaries ship for all eternity.

[googleson78]

The concrete issue is that

0. I get stack from nixpkgs
1. My colleagues get stack from the official bindist
2. I add new source files
3. I run stack build
4. The .cabal file gets regenerated to include the newer files, but also potentially other changes happen, because my hpack version is different (at the very least, the hpack version comment changes)
5. I commit the .cabal file, as recommended by the issue you linked
6. Now my colleague pulls in my changes, they use an official stack release with a different hpack version, so they get arbitrary issues, depending on the different hpack version. At the very least they get some weird mismatch, or the file doesn't get regenerated, etc.

It seems ridiculous that we'd have to track what stack binaries ship for all eternity.

I don't really understand this - for dev facing tools, isn't it expected that nixpkgs would track the official releases? Otherwise this introduces reproducibility issues between getting your tools from nixpkgs and from official releases, hence I am now arbitrarily going to get breakage if I choose to use NixOS when collaborating with non-NixOS users (or I have to potentially apply some workarounds to be able to use official releases because official releases don't always work on NixOS).

[maralorn]

But the real problem here is not an incompatibility between stack and hpack versions. It is like with very many tools, that you using another hpack version than your coworkers does not work.

I am uncertain what the best solution for that is. But in general nixpkgs downgrading software because other distribution methods are not as fast is a bit unsatisfying.

Additionally, I find it mighty confusing, that the stack binaries are not build with hpack version from stackage. Using the hpack version from stackage LTS, which we do, is the most blessing one can generally achieve for a package version.

So all of this is me shouting like a five year old: “Not our fault!” The question is what to do about it.

Have you considered using an older nixpkgs version which uses the right hpack version?

Apparently the whole process is super sensitive to the hpack version, so if we downgrade hpack now we probably annoy other people.

If there is no better solution maybe we need to supply different hpack versions and stack with different hpack versions and the user can choose?

[googleson78]

But in general nixpkgs downgrading software because other distribution methods are not as fast is a bit unsatisfying.

I agree

Additionally, I find it mighty confusing, that the stack binaries are not build with hpack version from stackage.

I agree

If there is no better solution maybe we need to supply different hpack versions and stack with different hpack versions and the user can choose?

I would guess that if you bless the version that the official stack bindist ships, nobody would complain, except maybe a single instance of other nixpkgs stack users, after which it should be fine? I guess a more principled approach would be to instead live with this for now and wait until stack bindists bump hpack version, at which point start tracking the hpack version provided with bindists.

So all of this is me shouting like a five year old: “Not our fault!” The question is what to do about it.

Have you considered using an older nixpkgs version which uses the right hpack version?

Of course I can work (and actively am working) around this. And also I do not expect that miraculously software from nixpkgs will be bug for bug compatible with official bindists. But this feels like a relatively simple(perhaps? I do not know how easy it is to have an additional hpack version in nixpkgs) change to make, which results in actual observable incompatibility being reduced.

[cdepillabout]

In the past we have overridden the version of hpack used by stack to work around this issue. Just like @googleson78 is saying, it is very annoying to use stack from Nixpkgs with a team of people who otherwise get stack from upstream.

I guess a newer hpack version was recently released and it slipped through.

@googleson78 would you be able to send a PR overriding the hpack version used by stack so that it is in sync with upstream? If you could figure out a simple test to add in pkgs/test/haskell/, that would be ideal as well.

[googleson78]

I've created a small patch to adding a test. However, that requires an internet connection, and from what I understand, nixpkgs tests run in a sandbox, and that won't be available(?)

Should I:

• somehow allow connection to the stack official bindist urls in CI (not sure how to do this)
• instead manually hardcode a mapping stack version -> hpack version, which must have an entry for the stack version that's currently in nixpkgs, i.e. when a new stack version comes in, whoever is doing updates will be forced to update the mapping (not sure where to do this either)

Additionally, I couldn't find how to make stack build with a specific version of hpack - could you give me a pointer to where I could see an example/read up on this? Currently configuration-hackage2nix/main.yaml has - hpack == 0.35.0, but I don't seem to understand where that is used explicitly (if it is at all).

[cdepillabout]

from what I understand, nixpkgs tests run in a sandbox, and that won't be available

That's correct.

instead manually hardcode a mapping stack version -> hpack version, which must have an entry for the stack version that's currently in nixpkgs, i.e. when a new stack version comes in, whoever is doing updates will be forced to update the mapping (not sure where to do this either)

I like what you currently have in your commit, but you'll have to pin the upstream (non-nixpkgs) stack version with a commit hash (and ideally fetch it with something like fetchFromGitHub or fetchurl or something). Ideally you'd be able to write it so that when the stack version is bumped in Nixpkgs, the test will automatically start failing. Also, could you add yourself as a maintainer to the test, so that you'll be pinged when it starts failing? That would help with the maintenance of the test.

I couldn't find how to make stack build with a specific version of hpack - could you give me a pointer to where I could see an example/read up on this?

A lot of these types of overrides are in pkgs/development/haskell-modules/configuration-common.nix. Here's an example of overriding the version of a direct dependency for a package:

nixpkgs/pkgs/development/haskell-modules/configuration-common.nix

Line 655 in 98f3b08

descriptive = super.descriptive.override { aeson = self.aeson_1_5_6_0; };

You'll likely have to do something similar for hpack with stack.

[cdepillabout]

Currently master / haskell-updates provides stack-2.9.1 as the top-level stack attribute (even though the latest release of stack is 2.9.3), and it uses hpack-0.35.2.

Upstream stack uses the following hpack versions:

• stack-2.9.1: hpack-0.35.0
• stack-2.9.3: hpack-0.35.1

I've just pushed cbc30e4 in #223827, which drops stack's hpack dependency to 0.35.0, in-line with upstream stack. This should make sure people getting stack from Nixpkgs use a compatible hpack version with people that get stack from upstream.

I'm going to keep this issue open in case anyone is able to send a PR implementing some sort of test in Nixpkgs that upstream stack and our stack use the same hpack version.

[maralorn]

Just so I understand this right: If someone's team uses “stack-2.9.3: hpack-0.35.1” but they themselves get our “stack-2.9.1: hpack-0.35.0” do they still have this problem?

[googleson78]

Thanks @cdepillabout and sorry for not getting back to this earlier. I'll still have in mind trying to write a test for this.

Just so I understand this right: If someone's team uses “stack-2.9.3: hpack-0.35.1” but they themselves get our “stack-2.9.1: hpack-0.35.0” do they still have this problem?

It did somehow become an issue for my workplace, because this causes the original issue but in reverse - I was using a newer hpack(from nix) and someone was using the older one, causing some kind of issues for the other person. However, I currently can't dig out the exact complaints.

[cdepillabout]

If someone's team uses “stack-2.9.3: hpack-0.35.1” but they themselves get our “stack-2.9.1: hpack-0.35.0” do they still have this problem?

My understanding is that this is correct. If you're using stack + hpack in a team setting, you have to make sure everyone uses the same version of stack, compiled with the same version of hpack. (Although it might be the case that you only have to make sure the hpack version is the same, you don't all necessarily have to use the same version of stack.)

[sternenseemann]

What's the state of this issue?

[cdepillabout]

I fixed the underlying problem in cbc30e4. But we don't have any assurance that the same problem won't crop up again the next time Stack releases a new version.

I was leaving this issue open so that hopefully someone would send a PR adding some sort of test that the hpack version used by Stack in Nixpkgs is the same as the hpack version used by the upstream Stack release.

Although I don't have a good idea of how to write this test in such a way that we won't have to manually bump a version/URL/hash every time there is a new release of Stack, which is quite annoying. It is the git-annex problem all over again.

[cdepillabout]

I implemented the check for the upstream stack/hpack version in #243798