cmd/go: reserve example/ and test/ as non-stdlib path prefixes

[bcmills]

The go command automatically fetches modules from network hosts, provided that those modules have a path whose first element includes a dot character (.).

First-elements that do not include a dot are, in general, reserved for the standard library and the Go toolchain (see #32819). Some names (std, cmd, all) have a special meaning no matter where they are used. Others (such as archive and debug) are currently standard-library prefixes but not packages, but could be made so in the future (as was done in the case of encoding).

Still others currently have no meaning, but may be assigned a special meaning in the future: for example, I would like to add mod to refer to the main module independent of its path (that would be a separate proposal).

The fact that import paths are required to contain a dot occasionally causes user consternation (#37554, #34592).

I propose that we explicitly reserve the following package paths for user code:

• for general use:
  • local
    • But not localhost, which we may at some point want to treat as a hostname.
• for documentation, tutorials, and bug reports:
  • example (along the lines of example.com, because it looks more like “something you should replace” than local does)
• for tests, especially tests of tools that manipulate modules (and thus need a “clean” module to run in):
  • test, which we already sort-of use:

    go/test/bench/go1/go.mod

    Line 1 in b1a783d

    module test/bench/go1

• for module-graph and package-import-graph examples:
  • main (because it is useful to indicate "the main module", and a package main is not importable anyway), and
  • any valid module path that consists of a single letter, because we already use those a lot in cmd/go tests.

CC @bronze1man @DisposaBoy @mvdan @jayconrod @matloob @thepudds

[mvdan]

Wouldn't main fall in the same category as, or perhaps be confused with, mod?

I'm overall in favor of this proposal. I seem to recall that @rogpeppe uses 0.0.0.0 as a module name, because it's a valid hostname and it contains dots, yet it can never be used to reach anything on the internet as it's not a usable IP. However, I have to admit that I too prefer using human-readable names like example or test, and I too have been doing it in tests.

From the tooling perspective, these rules are easy to implement, so that doesn't worry me either. The rules should be well documented, though, and probably alongside #32819.

Do you reckon that any sort of API would be useful for tools here? Similar to https://golang.org/pkg/go/token/#IsKeyword, imagine something like x/mod.IsLocal(path string) bool and x/mod.IsReserved(path string) bool. Their implementations would be pretty trivial, but the big advantage is that if we ever change the rules, we could easily bring the entire ecosystem in sync.

[bcmills]

Wouldn't main fall in the same category as, or perhaps be confused with, mod?

Maybe? I'd be fine with omitting it from the reserved names.

[bcmills]

Do you reckon that any sort of API would be useful for tools here?

Maybe? I'm not sure who would use them, though: I don't think we should explicitly reject other dotless paths at this point (we can break them if/when we need the name for something else, but if users are confident we'll never use their name I see little harm in letting it persist).

I guess it could be useful for linters..?

[bcmills]

(search.IsMetaPackage is what we currently use to enumerate the actually-meaningful non-package paths, but I don't think it covers builtin which is also somewhat special.)

[dmitshur]

I'll point out a cost of implementing this change, since I don't see it mentioned here so far, is that this will increase the amount of special cases. Special cases need to be documented, taught and learned by all Go users. Users who are not aware of the special cases and run into them accidentally may be surprised.

I think it's a good idea to evaluate how often each special case is expected to be used. For example, the package name main is a special case, and it is used very often, and adds more value than cost. The package name documentation is another special case, but it is used very infrequently, and has a higher cost than value (it was kept because of backwards compatibility reasons).

[jayconrod]

What would it mean to reserve these paths? I'm guessing a documented promise to never create std packages with those prefixes?

Should we allow dotless paths using reserved prefixes in the GOPROXY protocol?

[bcmills]

What would it mean to reserve these paths? I'm guessing a documented promise to never create std packages with those prefixes?

Yep. Specifically, we would apply the Go 1 compatibility policy to the non-existence of these paths in std.

[bcmills]

Should we allow dotless paths using reserved prefixes in the GOPROXY protocol?

Not at this time. (We could consider it as a future change, but since these paths explicitly do not have a single source of truth, I don't think they'd be a good fit.)

[slrz]

  * `example` (along the lines of `example.com`, because it looks more like “something you should replace” than `local` does)

What about jusing using example.com and the other existing DNS zones (example, invalid, test) reserved for test and documentation usage? If we already rely on the DNS, why not use its existing conventions for reserved names?

[dmitshur]

@slrz What you're suggesting is already possible, it does need any work to be completed. Or are you suggesting that as a reason this proposal should not be accepted?

[bcmills]

@slrz, IANA made the dubious decision to host an actual server on https://example.com.

While I do use those domains in examples myself, I could reasonably understand someone wanting to avoid them on security grounds: if those servers are ever compromised, they could serve go-import tags that point to malicious code.

[bronze1man]

Please leave reserve vendor for GOPATH , and remove the GOROOT/src/vendor .
I need to change the go source code when i want to upgrade golang version to make my private project works since version 1.13 (#34068 (comment))

[bcmills]

@bronze1man, vendor has had a special meaning since Go 1.6 (https://golang.org/doc/go1.6#go_command). The ship sailed on reserving that particular prefix a long time ago.

[thepudds]

Hi @bcmills, overall, I like the proposal.

Do you think the main use cases could be addressed by picking just one?

For example, maybe just pick local, or some other singular choice? That might help keep the number of special cases smaller.

People could still then use hierarchy underneath local to perhaps address the other ones you listed (e.g., local/test, local/example, local/main, or local/foo and local/bar if desired).

[bcmills]

@thepudds, just local could suffice. I suggested the others as well in the interest of maintaining semantic appropriateness.

For example, an “example” module in a tutorial is not really “local”. Someone following the tutorial might not understand local as a keyword that needs to be replaced with a non-local path prefix (or, really, any prefix that is not literally local), whereas the word example more clearly indicates a placeholder to be substituted by the reader.

[bronze1man]

Hi @bcmills, I like the proposal.
I think local and all single letter should be suffice.
Please do not make those reserve path prefixes can be relative resolve like vendor or ./x , it will make it difficult to find the actually package location.
GOPATH/src/local/ep and GOPATH/src/m/tmp and GOPATH/src/c/tmp looks local for me. And a single letter should be easy for different usage to end users.
Like:

• GOPATH/src/m for fast execute package from command line.
• GOPATH/src/c for project level config
• GOPATH/src/t for project level test
• GOPATH/src/l for private library only for this project.

[mewmew]

Just as a data point, I often use foo as a temporary local module path.

$ mkdir /tmp/foo
$ cd /tmp/foo
$ go mod init foo
$ go get github.com/llir/llvm/ir
$ cat go.mod 
module foo

go 1.14

require github.com/llir/llvm v0.3.0 // indirect

If prefixes were to be reserved, I'd echo @dmitshur's reflection that these special cases would need to be documented, and having many special cases increase the learning cost.

As such having a very clear rule, e.g. local can be use by whoever, where ever (with arbitrary sub-directories; local/foo). All other module paths where the first element does not contain a . character are reserved for the Go standard library/project.

[rsc]

Let's officially reserve example and test and stop there.
That is, we would commit that we will never add $GOROOT/src/{example,test}
to the main tree: there will not be packages with those names, nor packages
in subdirectories of those names.

Let's not do the single-letter thing. It's fine for us to do that ourselves,
and I certainly do 'echo module m >go.mod' every time I want to play with something,
but internal tests and ephemeral things don't need a long-term guarantee.

For things that need a long-term guarantee, it is reasonable to ask people to
use longer names, and I think example and test pretty much cover the use cases.

I don't understand the need for local, or else I disagree that it's a good use.
For a long-term use, it's better to use a domain name or other prefix you control.
For example inside Google all the paths begin with "google3/" (long story).
It's not a domain name, but it's also not a name that is going to end up in the
standard library and cause a conflict. Most company names have the same property,
so if you wanted to avoid using a domain name for some reason, your company
name is exceedingly unlikely to end up in the standard library unless it is a
generic word. More importantly, another company is exceedingly unlikely to use
the same name for their paths, so there's no problem when you need to start
sharing code with them (or buy them or are bought by them or whatever).
Similar problem happens sometimes with the reserved IP spaces.
(Even if we promise not to use local/, what if you use local/ for all your stuff
and you acquire a company that also uses local/ for all their stuff?
Better to use real prefixes for real code.)

[rsc]

Are there any objections to "Let's officially reserve example and test and stop there."?

/cc @bcmills

[mvdan]

Reserving test and example sgtm. I also agree that local isn't a clear benefit to my mind, partially because it can be easily confused with localhost.

@rsc one that you didn't include and didn't mention why is main. Could you clarify that bit?

I have to admit that main is a bit niche compared to test and example, though. I can see it being useful for testing cmd/go itself, since making the distinction between the main module and other modules can be very important to keep a test readable. However, one could imagine naming a module test/main, or example/main for an example, and I think we would accomplish the same goal. In a way, it mirrors Go's already present support for tests and examples.

[bcmills]

example and test seem like a reasonable enough start. (If we find that's not sufficient, it's easier to add reserved paths than to remove them.)

[dmitshur]

Are there any objections to "Let's officially reserve example and test and stop there."?

I'm open to being overruled on this, and I don't think this is a very significant change either way, but I would object at this moment because based on my evaluation of this proposal, the value it adds is too low compared to the cost of introducing this change.

The advantages of going with the suggested resolution, as I currently perceive, include:

• Some modules that don't need to be downloadable from the internet and want to have a very stort module path will be able to shorten the module path by 2 or more letters (e.g., module m.test -> module test or module example.com -> module example).

The costs of going with the suggested resolution, as I currently perceive, include:

• It creates a change that causes the addition of two Go-specific special cases when it comes to import paths that need to be learned and kept in memory over time.

  These special cases will affect and will need to be kept in mind both by developers and users of Go. Developers will need to remember not to use example or test as a prefix for a new Go standard library package (packages are added infrequently). Users will need to remember that example and test are safe to use, but sample or testing is not. Whenever someone uses this feature in a CL, the reviewers will be forced to learn about it. That means either this feature will never be used in modern practice (similar to the package documentation special case), or eventually all existing and future Go users will need to learn and remember this.

I think there would be a much stronger need for this proposal if the reserved TLDs .example, .test, .invalid, and .localhost did not exist. Given they do exist since 1999¹ and can be used by continuing to use the convention of using a URL for an import path, without introducing Go-specific special cases that need to be taught and learned, I think the incremental benefit offered by reserving "example" and "test" module path prefixes are not justified.

¹ https://tools.ietf.org/html/rfc2606#section-2

Edit: I want to clarify that I have considered this point:

The fact that import paths are required to contain a dot occasionally causes user consternation (#37554, #34592).

I think the proposed solution would benefit a small number of users by a small amount, and incur a small cost on a very large number of users. Because the benefit is so small, the left hand side of the equation doesn't seem to outweigh the right hand side by my current estimates.

[rsc]

My main hesitation about depending on the TLDs is that example.com was supposed to be the reserved name, and yet example.com has a web page. That means that go module examples using example.com make HTTPS requests. I don't trust the IETF not to create web pages for the other "reserved" TLDs.

The only thing we promise by reserving example/ and test/ is that we will never create those two directories in $GOROOT/src/. That is, there will never be a standard library package named example or example/x or test or test/x.

[dmitshur]

This is a documentation-only change,

Perhaps you meant this implicitly, but I think there should also be a test (or equivalent automated mechanism) that would detect if a future CL accidentally forgets about this promise. We shouldn't rely only on human reviewers for this.

[seankhliao]

given that these are reserved names, should a better error be returned? compare

» go build
main.go:3:8: package example is not in GOROOT (/home/arccy/sdk/gotip/src/example)

» go build
main.go:3:8: no required module provides package example.com; to add it:
	go get example.com

[marystern]

Sorry, this is a bit late, but I presume "local" would behave the same as "_" in #37554 (ie as I have been using for all my local-only modules for a while now)? I prefer the shorter version, but as long as it worked the same as "_" (ie no network lookups) I'd be content I believe.

[gopherbot]

Change https://golang.org/cl/359594 mentions this issue: _content/ref/mod: document dotless module paths

[gopherbot]

Change https://golang.org/cl/359576 mentions this issue: cmd/go: test that the reserved paths "example" and "test" are not used