Build error regarding h2quic

[wallacyyy]

When I try to build the server I'm receiving:

package github.com/lucas-clemente/quic-go/h2quic: 
cannot find package "github.com/lucas-clemente/quic-go/h2quic"

I noticed there was a recent change on the library renaming the directory from h2quic to http3:
Example here.

[francislavoie]

@marten-seemann what do you think should be done here?

[marten-seemann]

Nothing needs to be done here. Caddy has its vendored version of quic-go, so the current state of the quic-go repository is irrelevant.

[francislavoie]

Alright, thanks. Closing this then.

[MOZGIII]

I have the same issue when building from source. I don't see the vendored code in caddy.
Could you please re-open this issue?

I'm following the build instructions from the README.md

[JoanMartinezRipoll]

Same here, I am getting the exact same issue

[mholt]

We no longer use vendor, starting with 1.0 beta 1.

Are you in a GOPATH? Try setting export GO111MODULE=on and try again.

[elcore]

We no longer use vendor, starting with 1.0 beta 1.

Are you in a GOPATH? Try setting export GO111MODULE=on and try again.

Yes, this should fix this.

[wP99DhYrMMTYbfAvgiWm]

Good evening,

I am running into this issue as well. Setting GO111MODULE to on does allow the go get call to continue, but I then run into the error shown below:

# github.com/mholt/caddy/caddyhttp/markdown/summary
../pkg/mod/github.com/mholt/caddy@v0.11.5/caddyhttp/markdown/summary/render.go:24:5: cannot use (*renderer)(nil) (type *renderer) as type blackfriday.Renderer in assignment:
	*renderer does not implement blackfriday.Renderer (missing RenderFooter method)
../pkg/mod/github.com/mholt/caddy@v0.11.5/caddyhttp/markdown/summary/summary.go:26:44: too many arguments to conversion to blackfriday.Markdown: blackfriday.Markdown(input, renderer literal, 0)
# github.com/mholt/caddy/caddytls
../pkg/mod/github.com/mholt/caddy@v0.11.5/caddytls/setup.go:174:28: cannot use value (type "github.com/xenolf/lego/certcrypto".KeyType) as type "github.com/go-acme/lego/certcrypto".KeyType in assignment
../pkg/mod/github.com/mholt/caddy@v0.11.5/caddytls/setup.go:354:4: cannot use config.Manager.KeyType (type "github.com/go-acme/lego/certcrypto".KeyType) as type "github.com/xenolf/lego/certcrypto".KeyType in field value

Is there something else I should be doing or setting?

[mholt]

@wP99DhYrMMTYbfAvgiWm v0.11.5 does not use Go modules, so you'll need to use v1.0.0-beta1.

[wP99DhYrMMTYbfAvgiWm]

Thanks for the quick reply!

The strange thing is I am getting the code using go get -u github.com/mholt/caddy/caddy, which, as far as I know, fetches the latest version. In addition, if I try to run this without the environment variable mentioned above, I run into issues with the h2quic library, which should not exist in 0.11.5.

Disclaimer: As you can no doubt tell, I am fairly new to Go. Please forgive me for any stupid things I might say :D

[mholt]

Don't worry, I'm also new to this (Go modules, anyway) hence the beta period. :)

What does go list -u -m all show?

[mholt]

Huh, apparently the docs on modules are missing some critical information: https://github.com/golang/go/wiki/Modules#how-to-upgrade-and-downgrade-dependencies -- says

go get -u – Update all direct and indirect dependencies to latest minor or patch upgrades

But actually it ignores beta releases.

Edit: I just updated the wiki page so it at least says it now...

[wP99DhYrMMTYbfAvgiWm]

Huh, I would not have guessed that. Features, I suppose 😛.

I'm fine with running 0.11.5 for now, but unfortunately this still does not build due to the errors mentioned above (when using GO111MODULE=on) or due to the error shown below when not using this variable. Running go list -u -m all did not return anything. I should probably also mention then I'm attempting to build this for Docker (as part of an image for something else, so I can't use one of the existing Caddy images on Docker Hub).

package github.com/lucas-clemente/quic-go/h2quic: cannot find package "github.com/lucas-clemente/quic-go/h2quic" in any of:
	/usr/lib/go-1.12/src/github.com/lucas-clemente/quic-go/h2quic (from $GOROOT)
	/go/src/github.com/lucas-clemente/quic-go/h2quic (from $GOPATH)

Edit: I guess this makes sense. Even though I'm using an older version of Caddy, the h2quic library has still been updated. I guess this means we can't really build 0.11.5 from source anymore (for now?).

One other possible solution I came up with was to clone the entire Caddy repo, replace all instances of h2quick with http3 using sed and then attempting to build. Unfortunately I ran into issues when running go get -d ./... (before attempting to build) this way:

# cd /go/src/github.com/marten-seemann/qpack; git submodule update --init --recursive
Cloning into 'integrationtests/interop/qifs'...
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:qpackers/qifs.git' into submodule path 'integrationtests/interop/qifs' failed
package github.com/marten-seemann/qpack: exit status 128

(I am ignoring the possibility that it would still not build with the updated h2quic code for now... 👼)

[mholt]

If you check out v0.11.5, it should give you the vendor folder and use it, and it should build OK as far as I know.

[wP99DhYrMMTYbfAvgiWm]

I see. A final question then: how would I go about doing this?

I have attempted this using both go get -tags=v0.11.5 github.com/mholt/caddy/caddy and go get github.com/mholt/caddy/caddy@80dfb8b2a7f89b120a627bc4d866a1dc5ed3d92f, but both run into either the h2quic issue or the issues mentioned here.

Update: It is possible to get the beta version using the commit hash (using -tags does not work): go get -u github.com/mholt/caddy/caddy@657780bcdf1dae53269e4b024336bea6fb380e40:

go: finding github.com/mholt/caddy/caddy 657780bcdf1dae53269e4b024336bea6fb380e40
go: finding github.com/mholt/caddy 657780bcdf1dae53269e4b024336bea6fb380e40
go: downloading github.com/mholt/caddy v1.0.0-beta1

This still results in an error around markdown, but at least we now have the beta version!

go: extracting golang.org/x/text v0.3.0
# github.com/mholt/caddy/caddyhttp/markdown/summary
../pkg/mod/github.com/mholt/caddy@v1.0.0-beta1/caddyhttp/markdown/summary/render.go:24:5: cannot use (*renderer)(nil) (type *renderer) as type blackfriday.Renderer in assignment:
	*renderer does not implement blackfriday.Renderer (missing RenderFooter method)
../pkg/mod/github.com/mholt/caddy@v1.0.0-beta1/caddyhttp/markdown/summary/summary.go:26:44: too many arguments to conversion to blackfriday.Markdown: blackfriday.Markdown(input, renderer literal, 0)
# github.com/mholt/caddy/caddyhttp/httpserver
../pkg/mod/github.com/mholt/caddy@v1.0.0-beta1/caddyhttp/httpserver/tplcontext.go:278:14: undefined: blackfriday.HtmlRenderer
../pkg/mod/github.com/mholt/caddy@v1.0.0-beta1/caddyhttp/httpserver/tplcontext.go:280:11: undefined: blackfriday.EXTENSION_TABLES
../pkg/mod/github.com/mholt/caddy@v1.0.0-beta1/caddyhttp/httpserver/tplcontext.go:281:11: undefined: blackfriday.EXTENSION_FENCED_CODE
../pkg/mod/github.com/mholt/caddy@v1.0.0-beta1/caddyhttp/httpserver/tplcontext.go:282:11: undefined: blackfriday.EXTENSION_STRIKETHROUGH
../pkg/mod/github.com/mholt/caddy@v1.0.0-beta1/caddyhttp/httpserver/tplcontext.go:283:11: undefined: blackfriday.EXTENSION_DEFINITION_LISTS
../pkg/mod/github.com/mholt/caddy@v1.0.0-beta1/caddyhttp/httpserver/tplcontext.go:284:34: too many arguments to conversion to blackfriday.Markdown: blackfriday.Markdown([]byte(body), renderer, extns)

[elcore]

Hi,

I hope I can help you. Caddy v0.11.5 does not support modules. Because of this, these problems occur.

If you want to build Caddy v0.11.5, you will have to use these instructions. (GO111MODULES should be set to auto - export GO111MODULES=auto)

1. go get github.com/mholt/caddy
2. go get github.com/caddyserver/builds
3. cd $GOPATH/src/github.com/mholt/caddy/caddy
4. git checkout tags/v0.11.5
5. go run build.go

If you want to build Caddy v1.0.0-beta1, you will have to use these instructions.

1. export GO111MODULES=on This will not be necessary with Go 1.13 (it should be the default)
2. go get github.com/mholt/caddy
3. go get github.com/caddyserver/builds
4. cd $GOPATH/src/github.com/mholt/caddy/caddy
5. go run build.go

or

1. git clone https://github.com/mholt/caddy
2. cd caddy/caddy
3. go run build.go

[wP99DhYrMMTYbfAvgiWm]

Hi @elcore,

Thank you for the clear instructions, this helped me get it working! Much appreciated.

[MOZGIII]

I think this should be reflacted in the readme.

I ended up with the following:

export GO111MODULES=off

mkdir -p "$(go env GOPATH)/src/github.com/mholt/caddy"
cd "$(go env GOPATH)/src/github.com/mholt/caddy"
git clone https://github.com/mholt/caddy .
git checkout -f -b build 80dfb8b2a7f89b120a627bc4d866a1dc5ed3d92f

mkdir -p "$(go env GOPATH)/src/github.com/caddyserver/builds"
cd "$(go env GOPATH)/src/github.com/caddyserver/builds"
git clone https://github.com/caddyserver/builds .
git checkout -f -b build c62e2219460a8828970dad09212c3a4cec40b56c

cd "$(go env GOPATH)/src/github.com/mholt/caddy/caddy"
go run build.go

Why do we need to use this github.com/caddyserver/builds by the way?

[elcore]

We will upgrade the instructions, you just need to run export GO111MODULES=on beforehand, and the build will succeed. This is just temporary.

Why do we need to use this github.com/caddyserver/builds by the way?

To enable proper versioning of Caddy binaries.

[elcore]

Additionally, modules have the advantage that you can build them outside your GOPATH.

1. git clone https://github.com/mholt/caddy
2. cd caddy/caddy
3. go run build.go

[MOZGIII]

To enable proper versioning of Caddy binaries.

My question is: why would we want this code to be in a separate repo (github.com/caddyserver/builds)?
I makes no sense to keep it separate - it would've worked just as good if it was in a subdirectory in the same repo.

I'm not using modules because I need a very particular version of the caddy (commit hashes are for an older version in my real use case) - and used the vendoring setup back then.

I'm also donig git clone instead of a go get -d (or even go get as is the example), because it doesn't work if the latest version is "broken" (which is the case after switch to go modules for all the older Go versions or disabled GO111MODULES).

Of course, moving to the latest version could solve some of the issues - but it has other problems (with dependencies) that are blocking me from doing just that. I'll look into it later.

I'm just saying that for now I've came up with a workaround that's good enough for us - but this was not trivial. Please reconsider making caddy simply go-get-able.

[elcore]

I'm not using modules because I need a very particular version of the caddy (commit hashes are for an older version in my real use case) - and used the vendoring setup back then.

That is okay, you can use an older version of Caddy.

I'm also donig git clone instead of a go get -d (or even go get as is the example), because it doesn't work if the latest version is "broken" (which is the case after switch to go modules for all the older Go versions or disabled GO111MODULES).

It is not broken, go get github.com/mholt/caddy works

Of course, moving to the latest version could solve some of the issues - but it has other problems (with dependencies) that are blocking me from doing just that. I'll look into it later.

What is the issue?

I'm just saying that for now I've came up with a workaround that's good enough for us - but this was not trivial. Please reconsider making caddy simply go-get-able.

Caddy is go-get-able...

[MOZGIII]

In my case the behavior is pretty obvious. Remember, I need an older version of caddy. With modules enabled usual go get will pull v0.11.5 that has vendor dir, which will, however, be ignored, and go get will try to download the dependencies from the respective sources. And those won't be pinned by the mod file (and not from vendor dir). This obviously fails. The behavior is the same even if I do go get -d.
That's why I ended up doing git clone instead of go get -d, and then the good old git checkout <rev-i-want> to enforce proper version. This works like a charm because older caddy version still provides vendor directory - and everything that's needed for it to work is actually vendored, as opposed to just version-pinned. Personally I like vendoring more than version pinning, and mostly because it's much more easier to build some very old app version from just the app git repo. With pinning a lot of external services have to be up and provide old package versions - which is currently problematic for go.

By the way, it would be great if somebody notify go team about the issues we encounter here - as this is the very feedback they look on the new mod system.

[elcore]

With modules enabled usual go get will pull v0.11.5 that has vendor dir, which will, however, be ignored

That's the whole point, if you enable modules, vendor will be ignored.

[elcore]

So, if you want v0.11.5, you will have to disable modules or leave it in auto mode (see above)

[MOZGIII]

In my case the behavior is pretty obvious.

[MOZGIII]

So, if you want v0.11.5, you will have to disable modules or leave it in auto mode

Disabling modules will not help. You still have to do git clone, as go get (even with -d) will fail.

:/
The whole point of me writing all that text is to explain the behavior there. It's pretty easy to grasp if you know the rules of old vendoring and new modules system, however not everyone reads the release notes thoroughly.

[mholt]

The whole point of me writing all that text is to explain the behavior there. It's pretty easy to grasp if you know the rules of old vendoring and new modules system, however not everyone reads the release notes thoroughly.

Thank you for the explanation, which I find useful, as I am still learning modules here! 😇

To be fair, though, I've been referencing these resources for on and off for about a week:

• https://blog.golang.org/using-go-modules
• https://golang.org/cmd/go/
• https://github.com/golang/go/wiki/Modules

And not only are all those pages (the last 2 especially) huge walls of text, their docs are not totally consistent. For example, the wiki doesn't mention that prerelease tags are ignored (until I added a blurb to it myself). The go cmd docs say that go get will pull down the src folder, but neglects to mention that this only happens if GO111MODULE != on. That fact is instead very implicitly conveyed pages and pages away in a different section.

So it's all trial and error for me at this point. The docs have been mediocre and the transition has been frustrating for everyone. I appreciate everyone's patience.

[elcore]

Disabling modules will not help. You still have to do git clone

This works pretty well, you don't have to use git clone, just git checkout

If you want to build Caddy v0.11.5, you will have to use these instructions. (GO111MODULE should be set to auto - export GO111MODULE=auto)

1. go get github.com/mholt/caddy
2. go get github.com/caddyserver/builds
3. cd $GOPATH/src/github.com/mholt/caddy/caddy
4. git checkout tags/v0.11.5
5. go run build.go

[MOZGIII]

If that worked then our CI would not break, which it did. At the very step 1. @elcore, if you still insist that it works after reading my comment above - I find it hard to follow your reasoning.

[deanishe]

In my testing, I've seen wildly different, difficult-to-explain behavior when any and all of these variables change the slightest bit.

I feel much less stupid now.

[elcore]

@MOZGIII I hope this can help you.

Edit: Here I explicitly disabled modules, just to be sure.

[elcore]

I do not know, how your build environment (CI) is configured. I am using Ubuntu 18.10, with Go 1.12.4. If I can help you in any way, feel free to message me eldinhadzic [at] protonmail.com

[MOZGIII]

Are you starting from scratch? Empty GOPATH etc? We are building docker image with caddy, using golang:1.12-alpine.

[elcore]

Are you starting from scratch? Empty GOPATH etc? We are building docker image with caddy, using golang:1.12-alpine.

Yes, I am starting from scratch. On a side note, I would recommend you to update to Go 1.12.4

[MOZGIII]

Ok, let's go through what go get does currently with GO111MODULE=off. First of all, it doesn't crash - that's good, but it means something have changed since I last lloked into it. Nontheless.
It does load extra dependencies from the thirdparty repos, which is undesireable for the following reasons:

1. The versions of those dependencies are not pinned.
2. It's a waste of reources, since they're not gonig to be used (after we checkout to a tag with vendored versions).
3. They can break the build. In our current case it's not going to happen, but in another other it will due to stricter netwrok access rules and vendoring policy.

The worst this is what exactly is being attemped to be downloaded depends on the state of the current master branch of the caddy repo. And there was the case of this setup breaking the build, and there's no evidence that there are any measures taken to prevent that in the future.
Speaking of evidence, I don't think just providing and example of the command running successfully now proves that it's the right approach. At this point it looks like some kind of trolling from you, @elcore. I know you have something that works for you, but have to try and explain why it works and what are the potential downsides of the solution.
For us it still won't work, as we tried that, and now we know better, and will be using much more strightforward git cloning.

Regarding Go 1.12.4 - definetly a good idea. We already got our toolchain updated after we encountered this issue.

[francislavoie]

depends on the state of the cyrrent master branch of the caddy repo. And there was the case of this setup breaking the build, and there's no evidence that there are any measures taken to prevent that in the future.

Frankly, you shouldn't be relying on master for any kind of stability. That's effectively the development trunk of this repo. If you want stability you should always be using a tagged release or specific commits. It feels like you're asking for the Caddy repo to stifle progress in the name of stability in the master branch, but that's unrealistic.

I can definitely vouch for @elcore and that he's doing what he can to help you. He's not trolling.

We've also learned a lot here when it comes to go get and go mod. See the issue that @mholt opened on the Go upstream describing some of your complaints: golang/go#31529. go mod is still a new tool, and everyone's just trying to figure out how to use it properly.

With all that said, your feedback about this is definitely appreciated, it's helping uncover edge cases of the build tooling. Thanks for that.

[MOZGIII]

Ok, once again.

@francislavoie, go get with GO111MODULES=off relies on master (not latest tag as it does with modules on).
Therefore, the (@elcore's proposed) go get will load stuff from master before letting you checkout to the older commit you want.

My proposal for installing older version does not have that issue:

export GO111MODULES=off

mkdir -p "$(go env GOPATH)/src/github.com/mholt/caddy"
cd "$(go env GOPATH)/src/github.com/mholt/caddy"
git clone https://github.com/mholt/caddy .
git checkout -f -b build 80dfb8b2a7f89b120a627bc4d866a1dc5ed3d92f

mkdir -p "$(go env GOPATH)/src/github.com/caddyserver/builds"
cd "$(go env GOPATH)/src/github.com/caddyserver/builds"
git clone https://github.com/caddyserver/builds .
git checkout -f -b build c62e2219460a8828970dad09212c3a4cec40b56c

cd "$(go env GOPATH)/src/github.com/mholt/caddy/caddy"
go run build.go

That said, I'm unsubbing from this issue as I feel there's enough context by now for people to see my point of view on the issue.

One last note: maybe it's a good idea to re-add vendor dir, but with go mod vedoring instead of the old go dep. This would definetly be a good option for the transitioning period, while the non-suffixed tag is there. @mholt

[elcore]

Ok, let's go through what go get does currently with GO111MODULE=off. First of all, it doesn't crash - that's good

Great to hear that from you.

It does load extra dependencies from the thirdparty repos, which is undesireable for the following reasons

Your reasons are totally valid, it does pull some thirdparty repos over the network.

Speaking of evidence, I don't think just providing and example of the command running successfully now proves that it's the right approach.

I was trying to show you, that the build will be successful, if you follow instructions I have provided.

At this point it looks like some kind of trolling from you, @elcore. I know you have something that works for you [...].

I am very sorry, if you felt like I was trolling you! I was just trying to be helpful, the transition to modules is not as easy as it should have been. I do not expect anything in return, I am using my resources available to help this project and its community.

I even created a simple Docker container as a proof of concept a few minutes ago. https://github.com/elcore/caddy-docker

For us it still won't work, as we tried that, and now we know better, and will be using much more strightforward git cloning.

It is great that you have found a solution that works for you.

[elcore]

While we are at it, I will follow Matt´s advice, and give you some space to breath.

[ntgeralt]

same problem when docker build.
cannot find package "github.com/lucas-clemente/quic-go/h2quic"

Is there any quick solution?

[pascalandy]

Here is the answer and it's a hacky one:
https://github.com/swarmstack/caddy/blob/master/Dockerfile-no-stats#L50

[jasonwbarnett]

Hey everyone, I hope you can help. I ran into this issue recently and seeing it's been a couple of years later I'm hoping there may be some updates.

How to reproduce

With xcaddy

$ go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest
$ xcaddy build --with github.com/pyed/ipfilter

Without xcaddy

$ mkdir /tmp/buildenv
$ cd /tmp/buildenv

$ cat > main.go <<'eof'
package main

import (
	caddycmd "github.com/caddyserver/caddy/v2/cmd"

	// plug in Caddy modules here
	_ "github.com/caddyserver/caddy/v2/modules/standard"
	_ "github.com/pyed/ipfilter"
)

func main() {
	caddycmd.Main()
}
eof

$ go mod init caddy
$ go get -d -v github.com/caddyserver/caddy/v2
$ go get -d -v github.com/pyed/ipfilter

Error

github.com/pyed/ipfilter imports
	github.com/caddyserver/caddy/caddyhttp/httpserver imports
	github.com/lucas-clemente/quic-go/h2quic: cannot find module providing package github.com/lucas-clemente/quic-go/h2quic

Other Info

$ go version
go version go1.17 linux/amd64

[jasonwbarnett]

Well I just read here pyed/ipfilter#46 that github.com/pyed/ipfilter is not compatible with Caddy v2, that's probably the issue. 🤦

[Prakash-Sanchela]

git clone https://github.com/caddyserver/builds .
This step gives me error

go: module github.com/caddyserver/builds: git ls-remote -q origin in /root/go/pkg/mod/cache/vcs/315a5c435b97a9dd0d62d86b255f57ddc9d52bfd19d28da7f749688271cfdd7b: exit status 128: fatal: could not read Username for 'https://github.com': terminal prompts disabled

[mohammed90]

git clone https://github.com/caddyserver/builds .
This step gives me error

go: module github.com/caddyserver/builds: git ls-remote -q origin in /root/go/pkg/mod/cache/vcs/315a5c435b97a9dd0d62d86b255f57ddc9d52bfd19d28da7f749688271cfdd7b: exit status 128: fatal: could not read Username for 'https://github.com': terminal prompts disabled

You're replying to a closed 4-year old issue with insufficient data and invalid repository URL. Please start by following the proper steps of how to build caddy from source, then report failures through the appropriate channels.