Discuss renaming lock.json and manifest.json

[bradleyfalzon]

Raising an issue to ensure this is tracked and closed.

As discussed in the Go Package Management mailing list (https://groups.google.com/forum/#!topic/go-package-management/et1qFUjrkP4), there were some thoughts about names and locations of the lock and manifest files. Broadly speaking:

• There was some concern with storing the files in the root repository directory, others have suggested other folders, such as within /vendor/. Storing in /vendor/ could cause issues with a VCS if that directory is also being ignored.
• I raised that if we are to store the files in generic directories, this could cause collisions with other package managers in use for other languages in the same project.

Personally, I don't mind which directory they are stored in, however, if they are generic directory names, to reduce the chance of name collisions, we should reconsider the names manifest.json and lock.json, such as gomanifest.json and golock.json?

I'm on the fence on this issue, just thought it should be raised formally.

[sdboyer]

Yeah, thanks for writing this up.

I raised that if we are to store the files in generic directories, this could cause collisions with other package managers in use for other languages in the same project.

This does concern me. I've always been annoyed that e.g. package.json's name doesn't have any obvious linkage to npm. Potential naming collisions with other systems are an issue as well.

Of course, WHAT we name it is...the mother of all bikesheds. It kinda brings us back to #25 (which we ultimately rolled back), except that it's even more permanent, because even if/when this tool hopefully becomes official, gets integrated into the toolchain and the name goes away...the manifest and lock files will need to retain the same names.

There was some concern with storing the files in the root repository directory, others have suggested other folders, such as within /vendor/. Storing in /vendor/ could cause issues with a VCS if that directory is also being ignored.

This one's a hard no. Please, let's just drop it right now. Manifest/lock files go in the root. I don't want to even give reasons, lest it invite even more bikeshedding than we're already risking with names.

[pbennett]

I think at a minimum, the prefix of the two files should be the same.
godep[-]lock.json, godep[-]manifest.json. That sort of thing.
manifest.json and lock.json are IMO far too generic and there's nothing to indicate they have anything to do with each other.

[zolotov]

+1 for less generic names for these files to avoid false positive results while determining whether the project is backed by dep tool. gomanifest/golock looks good to me.

[nathany]

This one's a hard no. Please, let's just drop it right now. Manifest/lock files go in the root. I don't want to even give reasons, lest it invite even more bikeshedding than we're already risking with names.

The other Go tools put these files in vendor/. I don't really care where they go, but the discrepancy is one of the first things I noticed. Unless there is a reason in a FAQ somewhere, I suspect the question will be raised multiple times. Guess we can wait and see.

[sdboyer]

@nathany that's fine - we can write up the reasons elsewhere. I just don't want to derail this issue.

[nathany]

Regarding the naming...

How important is it to have .json as the extension? Bundler (Ruby) and Hex (Elixir) use .lock as the extension for the lock file.

If the tool is going to be dep going forward (e.g. go dep ...), then how about dep.json? And perhaps dep.lock or dep.lock.json?

[nathany]

Manifest may not be the right name.

manifest |ˈmanəˌfest|
a document giving comprehensive details of a ship and its cargo and other contents, passengers, and crew for the use of customs officers.

Unlike other tools, dep treats imports in the code as the source of truth.

The file we've called manifest thus far really could (and probably should) be a human-editable file to wrangle with the solver, specifying constraints when the defaults aren't enough. It doesn't need to contain the full details, certainly not the transitive list of dependencies (that is the lock file's job).

[bradleyfalzon]

How important is it to have .json as the extension? Bundler (Ruby) and Hex (Elixir) use .lock as the extension for the lock file.

Important if:

manifest thus far really could (and probably should) be a human-editable file to wrangle with the solver

Then it could be easy to double-click a file to open an editor and the editor knows what the format is.

[joeshaw]

On the topic of renaming the manifest and lock files (which I support), here are uses of files named manifest.json and lock.json among existing Go files on GitHub:

https://github.com/search?utf8=%E2%9C%93&q=%22manifest.json%22&type=Code&ref=searchresults&l=go

https://github.com/search?utf8=%E2%9C%93&q=%22lock.json%22&type=Code&ref=searchresults&l=go

lock.json appears to not be an issue, but manifest.json likely would.

[nathany]

@bradleyfalzon Yes, the editor knowing the format could be of some benefit, especially if the manifest is human-editable.

A counter-argument is that if the files aren't intended for human consumption, the names and location (#207) could reflect that. E.g. deps.lock is a file that I'm not expected to look at or modify (it's an implementation detail).

[bradleyfalzon]

Yes agreed @nathany, my thought was something like godep.json and godep.lock (or along those lines)

[frou]

Regardless of their extension, IMO it is a no-brainer for their names to have a common prefix, to communicate that they are a pair of related files.

[tj]

Does there have to be two files? Similar prefix would be nice if we have to go with two, at least they'll sort together, gets a little gross having dozens of things like this in a project.

[pkieltyka]

I'd also prefer to see a single file, like dep.json as well. Or just put the files in ./vendor/, but I've heard thats been ruled out.

[ankitm123]

@tj and @pkieltyka, @sdboyer explains it pretty well in #281.
Having a lock file helps when working with a collaboration (the lock file locks it down to a specific version of a package). The composer package in php uses a similar pair of files to accomplish this: composer.json and composer.lock.

[tj]

I understand the use-case, just thought it might be worth considering not having two distinct files. Maybe that would be confusing I'm not sure, I don't see myself editing the manifest manually often if ever really, but I wouldn't think seeing a "lock" array or similar in there being too bad.

Going with deps.json / deps.lock (or singular) definitely gets my vote if it has to be two.

[pkieltyka]

I second that, I also understand the use-case and for the spirit of simplicity, I'd prefer to just have dep.json be the lock file and not edit it by hand unless I absolutely necessary. I would instead rely on the dep tool to manage it properly for me. But, I also second if there must be two files then dep.json and dep.lock feel the best so far :S

[sdboyer]

Right, so, single-file was more of a possibility earlier in the process, when we were imagining that the manifest and lock were both going to be managed by the tool. But we decided to move away from that model (#213 (comment)), as it was starting to make the commands and flags themselves look absurd. (What's the valency of -update? Is it the same as override? Do we need a separate remove command when all it does is delete something from the manifest? How do required/ignored packages get named?)

With that decision made, it means there's one file for humans, and one for machines; there's really no sane argument for condensing those into one file anymore.

I think the shared prefix idea is a good one - ensuring they're sorted out of the way helps with that. Maybe also using a capital leading letter (Manifest.json rather than manifest.json), so that it sorts separately from source files, and in with other meta files (e.g. README).

[sdboyer]

Also, point of order - at some point soon I'm gonna put a clock on this discussion, after which the committee will just make a fiat decision. Folks are welcome to throw suggestions in the meantime. I hope we can all understand that a question like "what do we name these files" could go in circles for years, so this is a time when it's better to just drop a hammer.

[pkieltyka]

I don't mean to come off negative here, but saying something like "more possible earlier in the process" doesn't jive with striving to make something great and lasting - everything should be possible during development (IMO), even if it means throwing code away, that is the point of design, implementation, evaluation, iteration.

The interface to the dep tool and its file structure will be fundamental for clear understanding, usage and adoption. I know people are offering up their time and thank you for that, and I wish I could wrap my head fully around the effort to offer a more informed opinion on the implementation details. But, I've been following many Go dependency projects for years, tried most of them, I even starting writing my own a while back along with a spec, and the single thing I've learnt is to design for a singular goal / perspective in mind and to not try to include every possible knob. For example, govendor is an amazing project but I feel it's cli is so flexible that its actually its detriment to its adoption. Focusing on the 80% of the daily usage from a developer experience perspective is the right way to go.

[sdboyer]

everything should be possible during development (IMO), even if it means throwing code away

Sure, I agree that everything is possible - when you first start. But as you progress with any project, you discover that you have to make tradeoffs. This is a tradeoff we chose to make.

and the single thing I've learnt is to design for a singular goal / perspective in mind and to not try to include every possible knob.

I don't think the two-file approach qualifies in the category of not having a singular goal, or including complexity/knobs beyond the minimum.

I hope, if you can find the time to investigate and wrap your head around some of the choices we've made, you'll see it's because they are tradeoffs made in pursuit of a clear goal, very much aimed at satisfying the 80% case.

[pksunkara]

FWIW I like the idea of dep.json and dep.lock described above.

[rnijveld]

An advantage of using godep.format (and godep.lock) (or their capitalized versions) over dep[s]?.format (and dep[s]?.lock) would be to allow new users to more quickly identify these files as being related to go code. I would imagine a project containing both package.json (npm) and dep.json might create some confusion.

Although this might just be an unlucky coincidence of this project being named dep (which might be confused as a generic 'dependencies', compared to cargo and rubygems which have a non-generic name in the context of dependency management).

I'd put them in the project root, as this allows the vendor directory to be gitignored in its entirety. I imagine that generally people will not want to commit their dependencies directly into their own source tree, although people coming from a C/C++ background may want to debate this.

[nathany]

It is interesting point that other package managers have a identifiable names that result in file names like Cargo.toml or yarn.lock. Not sure if "dep" will survive the merger into the go toolchain.

These all sound like good ideas to me:

• Capitalized to organize with README, separate from source code (like Cargo)
• File.<format> (human editable) and File.<lock> (machine generated)
• Use the standard extension for format (e.g. .json) for ease of opening the human editable file in an editor, not so much for the machine generated .lock file.
• Root of project so vendor can be ignored if desired
• Fairly easy to identify the files for Go dependencies (whether they contain "go" or become a well known name, such as the the name of a tool).

Just left with the hard part -- naming things:

Dep, Godep, Go, Gps, Hoard, Manifest, Vendor, etc.

[nathany]

Personally I rather like the simplicity of Go.<format> and Go.lock.

Though that may result in things other the dependency constraints finding their way into the Go.<format> file. Not sure if that's good or bad.

[nathany]

"In particular, I expect that the lock and manifest files will change in syntax and therefore file name during integration into the go command, so that it is possible for projects to work with both dep and the go command during a transition." - @rsc https://groups.google.com/forum/#!msg/golang-nuts/PaGu2s9knao/6quU08CTAgAJ

To leave open the possibility of backwards incompatible syntax changes, maybe the filenames decided here shouldn't be the final names.

It's either that or versioning the Manifest syntax 😵 (as with Vagrantfile).

[niondir]

docker-compose files are also versioned. But I see this more as something to add when we fail on backwards compatibility one day. Which will hopefully never happen.

Another reason for not putting stuff into /vendor is that a) it's hard to find and b) stuff like rm -rf /vendor && dep ensure gets much easier when the directory is clean. I would like to see the file next to package.json and personally don't like uppercase filenames.

I agree on @nathany conculsion of "good ideas".

Since the final name is still open I would like anything starting with go e.g. godeps.json and godeps.lock looks totally sane to me. I would avoid gps in the name, it will be a mess to google about it.

[rsc]

Please just keep the files with the names and formats they already have. They are perfectly fine for the dep experiment. They will probably change in the final go command integration based on that experience, and that's fine. But don't introduce unnecessary churn by changing them before that.

[nathany]

The format and therefore file extensions are already changing as per #119.

manifest.toml and lock.toml may introduce conflicts or confusion with multi-language repositories. The churn may not be absolutely necessary, but there are good reasons to move ahead, even if it may change again in the future.

[sdboyer]

We'll be going with Gopkg.toml and Gopkg.lock for this next iteration. Implementation will be done shortly in #342.

[davecheney]

Please please, no capitalisations.

…

On 6 Apr 2017, at 19:24, sam boyer ***@***.***> wrote: We'll be going with Gopkg.toml and Gopkg.lock for this next iteration. Implementation will be done shortly in #342. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[sdboyer]

I'm not a superfan of the initial capital, either. Makes me twitch thinking about it, in fact.

But, being that we do expect one more change in the future on integration into the toolchain, this is our last chance to try out the ideas discussed earlier in the thread: if we use an initial capital, will it tend to sort separately from actual source files - and how does that feel in practice?

[peterbourgon]

Sorry, capitalizations are a de facto standard for this class of file.

[sdboyer]

#experiment

[pksunkara]

When you are viewing a project in github (or other similar services) and editors, capitalised files appear first after the folders and dotfiles. The code which in golang is generally in lowercased files appear after them. An average user is not concerned about the dependencies of a package unless he specifically wants them. So, they should probably appear after the code.

But since, the first letter is g for these files, there is a high probability that lowercasing the files wouldn't have any effect at all on this particular issue.

So, I am okay with the capitalisation.

[davecheney]

Sorry, capitalizations are a de facto standard for this class of file.

@peterbourgon, I do not believe the de facto choice is to capitalise the first letter. Please consider this brief survey:

Java: ant, ivy, build.xml, maven: pom.xml
Rust: cargo, Cargo.toml
PHP: composer, composer.json
Javascript: npm, package.json, yarn, yarn.lock
Python: pip/easy_install, dependencies.txt
Haskell: cabal, a lower case file than ends in .cabal (oddly Haskell appears to prefer capitals for .hs source files)
R: packrat, packrat/packrat.{lock, opts}

[sdboyer]

It's true, I think it'd be tough to characterize this as the de facto pattern. Cargo's the only one there that does it (though also, the newest). The reasoning behind the capitalization is really the sorting within directory listings.

Again, we know these files are near-certain to change on their way into the toolchain. Let's please look at this as an opportunity to see how we feel about what, to me, seems like the intrinsic ickiness of a capital-led filename vis-a-vis the consequences for sorting.

[peterbourgon]

Yep, that's aggressive, I withdraw the characterization with apologies—though I observe Ruby and Gemfile. I also observe that this is conversation the distilled and purified essence of a bikeshed, withdraw from it completely, and encourage everyone else to do the same.

[adg]

Can we please end the tyranny of the lower case? Capital letters are letters too. I know this sounds like I'm joking, but I'm not.

If I have a file named Gopkg.something in my repo root it's not going to look like any of my actual project's files, just as README or LICENSE also do not. Huge benefit to me.

[davecheney]

If lower case was good enough for ken, it's good enough for me. But seriously; this tool has to work across windows, osx and linux, whose filesystems all handle case preservation differently. I'm not going to pull my "old man get off my lawn card", but there is a reason that mixed case and source control get along like two wet cats in a sack.

…

On Fri, Apr 7, 2017 at 9:51 AM, Andrew Gerrand ***@***.***> wrote: Can we please end the tyranny of the lower case? Capital letters are letters too. I know this sounds like I'm joking, but I'm not. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#168 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcAwYZEvG1CLYFtw2lI_8DAIlOIqRpks5rtXpygaJpZM4LunlS> .

[bketelsen]

piping in on the caps, please use lower case, case-sensitive, case-preserving, case-insensitive file systems ought to be all the reason in the world to agree with Dave on this one.

[krisnova]

Chiming in:

I think the system compatibility thing is enough to go with lowercase, and also am a fan of @rsc's point on changing this pending experimation results. Also if this does change later it's not the end of the world. Deprecating a pattern and making assumptions about what should happen when/if the old pattern is detected isn't going to be impossible to solve.

[adg]

Has this been a problem with Makefile, for example?

[davecheney]

Make accepts both Makefile and makefile, and a few others. https://www.gnu.org/software/make/manual/html_node/Makefile-Names.html

…

On Fri, 7 Apr 2017, 12:48 Andrew Gerrand ***@***.***> wrote: Has this been a problem with Makefile, for example? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#168 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA0e_QmqJx3m6GlV2QZt98TYwv-xVks5rtaQIgaJpZM4LunlS> .

[sdboyer]

Case sensitivity does not strike me as a relevant concern, as there is no planned workflow in which a user manually creates either a manifest or a lock. Yes, a user COULD manually create one, and if they did it wrong on a case-insensitive filesystem, committed it, and someone on a case-sensitive filesystem then checked it out, there'd be a problem. But that's a long series of conditionals, all ultimately coming back to a user workflow that the tool gives no real reason (or even opportunity) to engage in.

As Dave noted, make accepts makefile and Makefile - which, I think, is the real source of the difficulty there. But we're not talking about reading either Gopkg.toml OR gopkg.toml, as that that just makes the whole situation more ambiguous, helping no one.

So, again, I don't see a remotely normal workflow in which either the wrong or both capitalization schemes would be present in such a way that case sensitivity would end up mattering. It seems more like this only arises if someone is actively attempting to break the system.

Case preservation is something I hadn't considered, though. As far as I can think, the problematic vector here goes like this:

1. Developer 1 runs dep init command on a non-case-preserving filesystem, generating a gopkg.toml and gopkg.lock (or GOPKG.TOML and GOPKG.LOCK)
2. Developer 1 commits results into their SCM
3. Developer 2 checks out project onto a case-sensitive filesystem, which now fails to read the files
4. Developer 2 doesn't bother to notice, re-runs dep init and creates a proper Gopkg.toml and Gopkg.lock, and commits the results, creating a situation that will blow up on Developer 1's machine, and on Developer 3's case-insensitive filesystem.

This seems far-fetched, but not totally nuts...until I looked up which filesystems don't do case preservation. Wikipedia suggests that the most recent non-case preserving filesystem was FAT16, prior to the addition in 1994 of long filename support. So, someone would have to be actively developing - not merely compiling/running - Go code on a Windows NT 3.1 machine to create a problem with case preservation.

What have I missed?

[davecheney]

Windows' filesystem is the regular culprit. http://stackoverflow.com/questions/56022/prevent-file-casing-problems-in-subversion

…

On Fri, 7 Apr 2017, 12:56 sam boyer ***@***.***> wrote: Case sensitivity does not strike me as a relevant concern, as there is no planned workflow in which a user manually creates either a manifest or a lock. Yes, a user COULD manually create one, and if they did it wrong on a case-insensitive filesystem, committed it, and someone on a case-sensitive filesystem then checked it out, there'd be a problem. But that's a long series of conditionals, all ultimately dating back to a user workflow that the tool gives no real reason (or even opportunity) to engage in. As Dave noted, make accepts makefile and Makefile - which, I think, is the real source of the problem. But we're not talking about reading either Gopkg.toml OR gopkg.toml, as that that just makes the whole situation *more* ambiguous, and helps no one. So, again, I don't see a remotely normal workflow in which either the wrong or both capitalization schemes would be present in such a way that case sensitivity would end up mattering. It seems more like this only arises if someone is actively attempting to break the system. Case preservation is something I hadn't considered, though. As far as I can think, the problematic vector here goes like this: 1. Developer 1 runs dep init command on a non-case-preserving filesystem, generating a gopkg.toml and gopkg.lock (or GOPKG.TOML and GOPKG.LOCK) 2. Developer 1 commits results into their SCM 3. Developer 2 checks out project onto a case-sensitive filesystem, which now fails to read the files 4. Developer 2 doesn't bother to notice, re-runs dep init and creates a proper Gopkg.toml and Gopkg.lock, and commits the results, creating a situation that will blow up a case-insensitive filesystem later. This seems far-fetched, but not totally nuts...until I looked up which filesystems don't do case preservation. Wikipedia suggests <https://en.wikipedia.org/wiki/Case_preservation> that the most recent non-case preserving filesystem was FAT16, prior to the addition in 1994 <https://en.wikipedia.org/wiki/Long_filename> of long filename support. So, someone would have to be developing code on a Windows NT 3.1 machine to create a problem with case preservation. What have I missed? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#168 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA8rPyv4UScBtT8w-dtS38ms8Z2RNks5rtaX0gaJpZM4LunlS> .

[sdboyer]

That link has two developers manually naming files in a way that conflict with one another. As I described, I don't think that case is a concern here because there is no normal case, or really even an opportunity, for a user to manually create either the manifest or the lock.