proposal: use zip format inside .a and .o files

[rsc]

I propose to change both Go package archives and Go object files to use the more standard zip archive format.
In contrast to ar archives, zip archives admit efficient random access to individual files within the archive
and also allow decisions about compression on a per-file basis.
The result for Go will be cleaner access to the parts of a package archive
and the ability later to add compression of individual parts
as appropriate.

See golang.org/design/14386-zip-package-archives for details.

[rsc]

Updated link to design doc in original post, but here it is for people getting email updates: https://golang.org/design/14386-zip-package-archives.

[dsnet]

I don't have too much of an opinion regarding new .a and .o files (since I don't deal with compiler/linker stuff).

As the one who's been touching archive/tar lately, I support using zip over tar. The tar "format" is pretty fragmented and essentially consists of variety of popular and incompatible extensions (of which, only a few have a specification). And of the ones that have a specification, the behavior of the tar utility that introduced a given format doesn't always match the specified behavior. In contrast (as mentioned), the zip format is well defined (even if it is somewhat complicated) and Go's implementation of zip is less buggy then tar, IMHO.

What are the extensions to archive/zip that @rogpeppe is adding? Does it have to deal with trying to extend zip.File.Open to return an io.ReadCloser that satisfies io.ReaderAt and/or io.Seeker under for the zip.Store method? As it is today, zip.File.Open does not return something that can be random-accessed (like the proposal assumes is possible).

Edit: I figured out that random-access on individual files can be achieved by a combination of zip.File.DataOffset and zip.File.CompressedSize64. Disregard my concern.

[bradfitz]

The proposal says both:

I propose to change both Go package archives and Go object files to use the more standard zip archive format.

and also:

Go uses its own object file format, which we refer to as Go object files; that format is unchanged by this proposal.

That sounds contradictory.

[minux]

so the Go 1.7 compiler and linker will have to vendor math/big, archive/zip, and potentially compress/flate? could we instead make cmd/dist copy the vendored code to pkg/bootstrap instead duplicating those packages in cmd/? Too bad we didn't introduce the vendoring experiment into Go 1.4, otherwise the solution will be really simple.

[rsc]

On Thu, Feb 18, 2016 at 9:29 PM, Minux Ma notifications@github.com wrote:

so the Go 1.7 compiler and linker will have to vendor
math/big, archive/zip, and potentially compress/flate?

could we instead make cmd/dist copy the vendored
code to pkg/bootstrap instead duplicating those
packages in cmd/? Too bad we didn't introduce the
vendoring experiment into Go 1.4, otherwise the
solution will be really simple.

Yes, I have a CL that already does this. cmd/dist already knows how to
rewrite imports; vendor directories would add very little.

Russ

[mwhudson]

I guess for a package containing assembly the assembler will produce .obj files directly and the go tool will simply add these to the zip file the compiler produced?

The abstract says "I propose to change both Go package archives and Go object files to use the more standard zip archive format." but is really mostly about package archives isn't it? I guess the .o files the compiler and assembler currently make are somewhat containerish, and the *.obj files will just contain the part that follows the \n!\n bit in the current .o files? (I wonder, for the sort of object file redesign I propose in #11123 maybe it would make sense to have the *.obj file be replaced by a directory in the zip containing files like symtable, strings, data, etc...)

[mdempsky]

Have we ruled out the possibility of just using the file system and storing compiled artifacts as individual files?

[adg]

@rsc ping!

[rsc]

This is a low-priority change for me. I don't see any objections here, so I'm assuming it's okay to implement. Whether I get to it in time for Go 1.8 remains to be seen.

[rsc]

This is a good idea, but right now we don't have the bandwidth to help redesign the object format and support the resulting churn in the code base. We should revisit the object format at some point, but it won't be soon. Marking this declined for now, with the expectation that we may come back to this later.