Creation of PHAR using Nix, in a reproducible environment #754

drupol · 2023-12-05T17:02:44Z

Hey there,

I saw the message from @sebastianbergmann on Mastodon (https://main.elk.zone/phpc.social/@sebastian/111528636893710027) and I thought... How about using Nix to do it?

This PR shows how it could be done using 2 different approaches:

By using the existing build-phar.sh and nixifying it, providing box without needing to install it with Composer and build the PHAR in a non isolated environment.
By using a flake.nix file

Run nix build .#build-phar-script to build the script to build the PHAR (result in result/ directory)
Run nix build .#phar to build the PHAR in isolation (result in result/ directory)
Run nix run github:drupol/BackwardCompatibilityCheck/creation-of-phar-using-nix to immediately run the app
from your workstation without installing it, in total isolation.
Run nix develop to create a development shell containing PHP, Composer and Box and the build-phar-script

Both methods work well, and are totally reproducible. My favourite approach is to use the second one which is more flexible and versatile, but anyway one or the other, the environment building the PHAR is the same... (pun intended!). Note, the PHAR is not reproducible, box doesn't produce reproducible PHAR files by default, unfortunately (read how to make it reproducible here)

I'm pretty sure this PR won't be merged, but I just wanted to see if this was doable, I hope this is going to give ideas to some PHP community members :)

It took me 25 minutes to do, mostly copy-pasting things here and there.

flake.lock

.github/workflows/release-phar.yml

.gitattributes

Ocramius

Need some more clarity on the degree of manual intervention on flake.nix: can't really manually change its contents at every release

flake.nix

renovate.json

build-phar.sh

composer.json

resign.php

scoper.inc.php

Using PHP-Scoper create issue with the `assert` function

Ocramius · 2023-12-06T09:49:11Z

flake.nix

+            # This only changes when `composer.lock` is updated
+            vendorHash = "sha256-LsrGmver7RyiI0/l2j6dZaqhFQf2OFyUOZb8xzFFEIA=";


Ok, so we're back at having the hash here: how would one approach a patch by Renovate where composer.lock is being touched? 🤔

Yeah... I can't basically get rid of it, I tried... and there's a reason to have it there.

Since this PR is quite messy, how about lively discuss this during an informal meeting at your best convenience?

Since this PR is quite messy, how about lively discuss this during an informal meeting at your best convenience?

Can probably do next week about this.

Meanwhile, I have an example of a repo that has a Cargo.lock and no hash: https://github.com/Ocramius/oci-srm-server-mock-rust/blob/383e9bd1e99fca42578bd885ae01ff7c74b163ed/flake.nix

Perhaps php.buildComposerProject to be improved? Or are we missing sha256 for downloaded vendor projects as part of Composer itself? I see that Cargo.lock has checksums, while we don't

Note: could be worked around by assuming --prefer-source to download via git, where we at least have a SHA1

Let's do that next week then, ping me on slack, matrix or even telegram when you're ready.

Regarding the builder, there might be room for improvements for sure, and what you said is actually something I tried to fix in NixOS/nixpkgs#225401

I wish I could have a mapping between the composer.lock file and the vendorHash... But I have the feeling that it would require to update Composer first. That's an interesting discussion for sure.

You can find the sources of the builder here: https://github.com/NixOS/nixpkgs/tree/master/pkgs/build-support/php

Feel free to suggest improvements, having you on board on this is definitely a big plus.

But I have the feeling that it would require to update Composer first.

If we only do source-based installs, we can rely on git for some basic shasum, since Composer's is currently useless.

For now, I'd be happy if in this patch the vendorHash could be generated inside the github action: that would be a sufficient approach to move forward, IMO

This was taken from PHPStan to reset the PHAR files timestamps.

flake.nix

Box 4.6.0 is now on `nixos-unstable` branch, there's no need to have an extra input of `master` branch.

remove the derivation using `buildComposerProject` and create a build-phar-script with Nix instead

Ocramius

🚢 thanks @drupol!

I know this isn't exactly what you wanted, but I'm also glad that we found a middle-ground here, and that we introduced nix and nix-flakes in the entire dev stack.

Happy that the discussions/requirements led to more understanding of what package maintainers need from this end too :-)

I'll now merge and try out the release script, thanks for the help!

Ocramius · 2023-12-20T10:21:11Z

Aw, still need to fix PHP 8.1 :D

Should be trivial, though: I'm on it.

Ocramius · 2023-12-20T12:59:29Z

The CI failure is due to an ancient build of the require-checker .phar.

Unsure why we wget it, but not a problem for this patch to solve.

Ocramius · 2023-12-20T13:12:12Z

Well, this is merged, but no phar got tagged :D

https://github.com/Roave/BackwardCompatibilityCheck/actions/workflows/release-phar.yml

I suspect the release being triggered by a bot prevents this task from being triggered:

BackwardCompatibilityCheck/.github/workflows/release-phar.yml

Lines 5 to 8 in 4f6c882

    
           on: 
        
             release: 
        
               types: 
        
                 - published

Ocramius · 2023-12-20T13:14:49Z

The publishing payload, meanwhile, taken from https://api.github.com/repos/roave/BackwardCompatibilityCheck/events :

  {
    "id": "34310185998",
    "type": "ReleaseEvent",
    "actor": {
      "id": 41898282,
      "login": "github-actions[bot]",
      "display_login": "github-actions",
      "gravatar_id": "",
      "url": "https://api.github.com/users/github-actions[bot]",
      "avatar_url": "https://avatars.githubusercontent.com/u/41898282?"
    },
    "repo": {
      "id": 108544542,
      "name": "Roave/BackwardCompatibilityCheck",
      "url": "https://api.github.com/repos/Roave/BackwardCompatibilityCheck"
    },
    "payload": {
      "action": "published",
      "release": {
        "url": "https://api.github.com/repos/Roave/BackwardCompatibilityCheck/releases/134622383",
        "assets_url": "https://api.github.com/repos/Roave/BackwardCompatibilityCheck/releases/134622383/assets",
        "upload_url": "https://uploads.github.com/repos/Roave/BackwardCompatibilityCheck/releases/134622383/assets{?name,label}",
        "html_url": "https://github.com/Roave/BackwardCompatibilityCheck/releases/tag/8.5.0",
        "id": 134622383,
        "author": {
          "login": "github-actions[bot]",
          "id": 41898282,
          "node_id": "MDM6Qm90NDE4OTgyODI=",
          "avatar_url": "https://avatars.githubusercontent.com/in/15368?v=4",
          "gravatar_id": "",
          "url": "https://api.github.com/users/github-actions%5Bbot%5D",
          "html_url": "https://github.com/apps/github-actions",
          "followers_url": "https://api.github.com/users/github-actions%5Bbot%5D/followers",
          "following_url": "https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}",
          "gists_url": "https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}",
          "starred_url": "https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}",
          "subscriptions_url": "https://api.github.com/users/github-actions%5Bbot%5D/subscriptions",
          "organizations_url": "https://api.github.com/users/github-actions%5Bbot%5D/orgs",
          "repos_url": "https://api.github.com/users/github-actions%5Bbot%5D/repos",
          "events_url": "https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}",
          "received_events_url": "https://api.github.com/users/github-actions%5Bbot%5D/received_events",
          "type": "Bot",
          "site_admin": false
        },
        "node_id": "RE_kwDOBnhCHs4IBiyv",
        "tag_name": "8.5.0",
        "target_commitish": "8.5.x",
        "name": "8.5.0",
        "draft": false,
        "prerelease": false,
        "created_at": "2023-12-20T13:08:55Z",
        "published_at": "2023-12-20T13:08:57Z",
        "assets": [

        ],
        "tarball_url": "https://api.github.com/repos/Roave/BackwardCompatibilityCheck/tarball/8.5.0",
        "zipball_url": "https://api.github.com/repos/Roave/BackwardCompatibilityCheck/zipball/8.5.0",
        "body": "### Release Notes for [8.5.0](https://github.com/Roave/BackwardCompatibilityCheck/milestone/59)\n\nFeature release (minor)\n\n### 8.5.0\n\n- Total issues resolved: **0**\n- Total pull requests resolved: **2**\n- Total contributors: **2**\n\n#### enhancement\n\n - [755: Fix: Consistently indent with 2 spaces](https://github.com/Roave/BackwardCompatibilityCheck/pull/755) thanks to @localheinz\n - [754: Creation of PHAR using Nix, in a reproducible environment](https://github.com/Roave/BackwardCompatibilityCheck/pull/754) thanks to @drupol\n",
        "mentions_count": 2,
        "mentions": [
          {
            "avatar_url": "https://avatars.githubusercontent.com/u/252042?v=4",
            "login": "drupol",
            "profile_name": "Pol Dellaiera",
            "profile_url": "https://github.com/drupol",
            "avatar_user_actor": true
          },
          {
            "avatar_url": "https://avatars.githubusercontent.com/u/605483?v=4",
            "login": "localheinz",
            "profile_name": "Andreas Möller",
            "profile_url": "https://github.com/localheinz",
            "avatar_user_actor": true
          }
        ],
        "short_description_html": "<h3>Release Notes for <a href=\"https://github.com/Roave/BackwardCompatibilityCheck/milestone/59\">8.5.0</a>\n</h3>\n<p>Feature release (minor)</p>\n<h3>8.5.0</h3>\n<ul>\n<li>Total issues resolved: <strong>0</strong>\n</li>\n<li>Total pull requests resolved: <strong>2</strong>\n</li>\n<li>Total contributors: <strong>2</strong>\n</li>\n</ul>\n<h4>enhancement</h4>\n<ul>\n<li>\n<a href=\"https://github.com/Roave/BackwardCompatibilityCheck/pull/755\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/Roave/BackwardCompatibilityCheck/pull/755/hovercard\">755: Fix: Consistently indent with 2 spaces</a> thank…</li>\n</ul>",
        "is_short_description_html_truncated": true
      }
    },
    "public": true,
    "created_at": "2023-12-20T13:08:57Z",
    "org": {
      "id": 3029050,
      "login": "Roave",
      "gravatar_id": "",
      "url": "https://api.github.com/orgs/Roave",
      "avatar_url": "https://avatars.githubusercontent.com/u/3029050?"
    }
  },

drupol · 2023-12-20T13:31:08Z

Let me have a look at the workflow at the end of the day

Ocramius · 2023-12-20T13:32:01Z

Identified that I need ORGANIZATION_ADMIN_TOKEN here, roughly:

BackwardCompatibilityCheck/.github/workflows/release-on-milestone-closed.yml

Line 24 in 4f6c882

"GITHUB_TOKEN": ${{ secrets.GITHUB_TOKEN }}

Ocramius · 2023-12-20T14:08:33Z

Now it did run, but failed :D

/home/runner/work/_temp/8083e12c-ba48-4ca7-b23b-3bba47cad80a.sh: line 1: nix run .#build-phar-script: command not found

https://github.com/Roave/BackwardCompatibilityCheck/actions/runs/7276447892

That's... curious

drupol added 2 commits December 5, 2023 17:30

chore: ext-json is enabled by default since PHP 8

0f143ff

fix creation of PHAR release

a8ffb14

drupol mentioned this pull request Dec 5, 2023

Phar release is not created #374

Open

Ocramius reviewed Dec 5, 2023

View reviewed changes

flake.lock Show resolved Hide resolved

Ocramius reviewed Dec 5, 2023

View reviewed changes

.github/workflows/release-phar.yml Outdated Show resolved Hide resolved

fix creation of PHAR release using a flake

ef57b1b

Ocramius reviewed Dec 5, 2023

View reviewed changes

.gitattributes Show resolved Hide resolved

renovate: support flake.lock updates

337614c

Ocramius added this to the 8.5.0 milestone Dec 5, 2023

Ocramius self-assigned this Dec 5, 2023

Ocramius added the enhancement label Dec 5, 2023

Ocramius requested changes Dec 5, 2023

View reviewed changes

flake.nix Outdated Show resolved Hide resolved

flake.nix Outdated Show resolved Hide resolved

flake.nix Outdated Show resolved Hide resolved

renovate.json Outdated Show resolved Hide resolved

simplification by creating a script that build the phar instead

430bb44

drupol changed the title ~~Creation of PHAR using Nix, in a reproducible way~~ Creation of PHAR using Nix, in a reproducible environment Dec 5, 2023

Ocramius reviewed Dec 5, 2023

View reviewed changes

build-phar.sh Outdated Show resolved Hide resolved

drupol added 2 commits December 5, 2023 22:34

remove obsolete build script since Nix is used

0b166ad

make PHAR reproducible

75d5d96

Ocramius requested changes Dec 5, 2023

View reviewed changes

composer.json Outdated Show resolved Hide resolved

resign.php Outdated Show resolved Hide resolved

scoper.inc.php Outdated Show resolved Hide resolved

drupol added 5 commits December 6, 2023 07:20

fix: minor typo in banner

6545903

fix: make sure the generated PHAR actually works

d732c9c

Using PHP-Scoper create issue with the `assert` function

remove not-needed dependency

b1ac9ad

update .gitattributes entry for infection.json.dist

7e720f3

use nix build .#phar in CI

9e2b47a

Ocramius reviewed Dec 6, 2023

View reviewed changes

drupol added 2 commits December 6, 2023 10:52

remove resign.php

b998178

This was taken from PHPStan to reset the PHAR files timestamps.

switch to Box 4.6.0

959fa81

Ocramius reviewed Dec 11, 2023

View reviewed changes

flake.nix Outdated Show resolved Hide resolved

drupol added 3 commits December 13, 2023 08:13

remove extra input

2b9925c

Box 4.6.0 is now on `nixos-unstable` branch, there's no need to have an extra input of `master` branch.

update flake.nix

4823995

remove the derivation using `buildComposerProject` and create a build-phar-script with Nix instead

build-script: add composer validate step

d9ecc6f

flake.lock: update

a860862

Ocramius approved these changes Dec 20, 2023

View reviewed changes

Reverting composer.lock to a version compatible with PHP 8.1.x

094d6cd

Ocramius merged commit 947b074 into Roave:8.5.x Dec 20, 2023
14 of 17 checks passed

drupol deleted the creation-of-phar-using-nix branch December 20, 2023 13:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Creation of PHAR using Nix, in a reproducible environment #754

Creation of PHAR using Nix, in a reproducible environment #754

drupol commented Dec 5, 2023 •

edited

Loading

Ocramius left a comment

Ocramius Dec 6, 2023

drupol Dec 6, 2023

Ocramius Dec 6, 2023

Ocramius Dec 6, 2023

drupol Dec 6, 2023

Ocramius Dec 6, 2023

Ocramius left a comment

Ocramius commented Dec 20, 2023 •

edited

Loading

Ocramius commented Dec 20, 2023

Ocramius commented Dec 20, 2023

Ocramius commented Dec 20, 2023

drupol commented Dec 20, 2023

Ocramius commented Dec 20, 2023

Ocramius commented Dec 20, 2023

		# This only changes when `composer.lock` is updated
		vendorHash = "sha256-LsrGmver7RyiI0/l2j6dZaqhFQf2OFyUOZb8xzFFEIA=";

Creation of PHAR using Nix, in a reproducible environment #754

Creation of PHAR using Nix, in a reproducible environment #754

Conversation

drupol commented Dec 5, 2023 • edited Loading

Ocramius left a comment

Choose a reason for hiding this comment

Ocramius Dec 6, 2023

Choose a reason for hiding this comment

drupol Dec 6, 2023

Choose a reason for hiding this comment

Ocramius Dec 6, 2023

Choose a reason for hiding this comment

Ocramius Dec 6, 2023

Choose a reason for hiding this comment

drupol Dec 6, 2023

Choose a reason for hiding this comment

Ocramius Dec 6, 2023

Choose a reason for hiding this comment

Ocramius left a comment

Choose a reason for hiding this comment

Ocramius commented Dec 20, 2023 • edited Loading

Ocramius commented Dec 20, 2023

Ocramius commented Dec 20, 2023

Ocramius commented Dec 20, 2023

drupol commented Dec 20, 2023

Ocramius commented Dec 20, 2023

Ocramius commented Dec 20, 2023

drupol commented Dec 5, 2023 •

edited

Loading

Ocramius commented Dec 20, 2023 •

edited

Loading