Better hashing of sources

[chrisburr]

Checklist

• I added a descriptive title
• I searched open requests and couldn't find a duplicate

What is the idea?

Support a content dependent hash of sources rather than only hashing the source file itself. This is how nixpkgs's fetchzip works and could also be used to have a stable hash for recipes using git_url/git_rev.

Why is this needed?

I occasionally see builds failing due to the source coming from an autogenerated git tarball/zip being different (most commonly on github, but the same applies to most git hosts). This is because the git archives are dependent on the git version used and are not intended to be stable or reproducible.

It would also be nice to have more resistance to changing tags if it was also applied to git_url/git_rev.

What should happen?

My first guess would be to have an key like sha256 in the sources section like content_sha256 which would trigger the content-based hash to be checked.

Additional Context

Mixpkgs solved this quite a long time ago with their fetchzip funcion so that would probably be a valuable source of ideas on the implementation.

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!

[jaimergp]

This would be handy if we ever start storing sources along the built artifacts. One thing to take care of is the local conda-build cache. sha256 is used to see if the artifact is already available, so I guess with content_sha256 we wouldn't be able to provide a cache. This would be a good option for github dynamic archives and others, but we don't need it for stable artifacts, which could still use sha256.

[wolfv]

We could also record a content hash in the "finalized sources" section of the "rendered recipe" for the new recipe format: conda/ceps#74

[ifitchet]

Just a thought on the YAML structure regarding content_* hashes. If the content is a function of the content and independent of the compression method then should content_md5 etc. be a child node of source rather than a (repeated) child node of each compressed file?

  - source:
    - content_md5: blah
    - fn: foo.tar.gz
       url: 
       md5: other-blah
    - fn: foo.zip
       url:
       md5: third-blah

(I doubt that the above is valid YAML but you get the idea.)