x/pkgsite: local setup - tracking issue

[julieqiu]

Today, users can run pkgsite locally without setting up a local database by running:

go run cmd/frontend/main.go -direct_proxy -proxy_url=<your proxy URL>

See flags in cmd/frontend/main.go, and doc/frontend.md for docs.

Doing so allows users to view the package documentation, overview, imports, and licenses tabs. Search and other tabs on the package page are not supported in direct proxy mode.

However, there are cases when a user might want to run pkgsite on their local machine, and be able to view a private repository that is not available via a proxy.

Tasks related to this topic:

• x/pkgsite: run locally to view documentation of a pending CL #40159: run locally to view documentation of a pending CL
• x/pkgsite: cannot refresh documentation locally #54479
• render the standard library for GOROOT
• link to github documentation when possible?
• x/pkgsite: can not show dependencies with vendor directory #59542: add support for rendering vendor directories
• Display unexported symbols on request. (x/tools/cmd/godoc: show internal packages when explicitly requested #12092 proposal: x/tools/{godoc,cmd/godoc}: isolate, tag, and delete godoc #59056 (comment) )

Maybe:

• only render documentation for packages / units when they are accessed

This is an umbrella issue to discuss what features are needed for this to happen, so please comment on your use cases below. For discussions about hosting a private instance of pkgsite, including your own internal proxy, see #39827.

[achille-roussel]

Hello @julieqiu

If I can add color to the pain points of running locally, having to start a proxy isn't too much trouble, but the constraint of requiring https is a bigger barrier:

$ go run ./cmd/frontend -direct_proxy -proxy_url http://localhost:8888
config: {
    "AuthHeader": "",
    "ProxyURL": "https://proxy.golang.org",
    "IndexURL": "https://index.golang.org/index",
    "Port": "",
    "DebugPort": "",
    "ProjectID": "",
    "ServiceID": "",
    "VersionID": "",
    "ZoneID": "",
    "InstanceID": "",
    "LocationID": "us-central1",
    "QueueService": "",
    "GaeEnv": "",
    "GoogleTagManagerID": "",
    "AppMonitoredResource": {
        "type": "gae_app",
        "labels": {
            "module_id": "",
            "project_id": "",
            "version_id": "",
            "zone": ""
        }
    },
    "FallbackVersionLabel": "20200729t023455",
    "DBSecret": "",
    "DBUser": "postgres",
    "DBHost": "localhost",
    "DBPort": "5432",
    "DBName": "discovery-db",
    "DBSecondaryHost": "",
    "RedisCacheHost": "",
    "RedisCachePort": "6379",
    "RedisHAHost": "",
    "RedisHAPort": "6379",
    "UseProfiler": false,
    "Quota": {
        "QPS": 10,
        "Burst": 20,
        "MaxEntries": 1000,
        "RecordOnly": true,
        "AuthHeader": "",
        "AuthValues": null
    },
    "TeeproxyAuthValue": "",
    "TeeproxyForwardedHosts": null
}
2020/07/29 02:34:55 Error: proxy.New("http://localhost:8888"): scheme must be https (got http)
exit status 1

What do you think about relaxing this constraint?

[gopherbot]

Change https://golang.org/cl/245639 mentions this issue: internal/proxy: remove URL validation in New

[julieqiu]

@achille-roussel - done! Thanks for the feedback.

[MicahParks]

To run pkgsite in a docker container, it'd be good to customize what networking interface the program binds to.

Executables in docker containers can more easily publish their services on the host's networking interface if bound to 0.0.0.0. It would be possible to allow this configuration via a flag by changing these lines:

https://github.com/golang/pkgsite/blob/ef183200d192b20102bb1246a86ae11de2d0738e/cmd/frontend/main.go#L170
https://github.com/golang/pkgsite/blob/ef183200d192b20102bb1246a86ae11de2d0738e/cmd/frontend/main.go#L148

to use a variable that is assigned by a flag instead of the "localhost" string.

[achille-roussel]

We had to solve for this as well in the fork we maintain at https://github.com/segmentio/pkgsite:

diff --git a/cmd/frontend/main.go b/cmd/frontend/main.go
index 5773d8e..e81022b 100644
--- a/cmd/frontend/main.go
+++ b/cmd/frontend/main.go
@@ -42,6 +42,7 @@ var (
                "for direct proxy mode and frontend fetches")
        directProxy = flag.Bool("direct_proxy", false, "if set to true, uses the module proxy referred to by this URL "+
                "as a direct backend, bypassing the database")
+       httpAddr = flag.String("http", "localhost:8080", "address to listen for incoming requests on")
 )

 func main() {
@@ -167,7 +168,7 @@ func main() {
                middleware.Timeout(54*time.Second),
                middleware.Experiment(experimenter),
        )
-       addr := cfg.HostAddr("localhost:8080")
+       addr := cfg.HostAddr(*httpAddr)
        log.Infof(ctx, "Listening on addr %s", addr)
        log.Fatal(ctx, http.ListenAndServe(addr, mw(router)))
 }

Taking a closer look at the way the configuration is setup, it appears that if the PORT environment variable is set the server will be listening on all network interfaces (as the address will be :$PORT), so maybe that change was unnecessary:

// HostAddr returns the network on which to serve the primary HTTP service.
func (c *Config) HostAddr(dflt string) string {
	if c.Port != "" {
		return fmt.Sprintf(":%s", c.Port)
	}
	return dflt
}

[gopherbot]

Change https://golang.org/cl/260779 mentions this issue: cmd/frontend: add -local flag

[gopherbot]

Change https://golang.org/cl/290134 mentions this issue: cmd/pkgsite: add command

[gopherbot]

Change https://golang.org/cl/290135 mentions this issue: cmd/pkgsite: add -http flag

[akavel]

I was redirected here from an issue requesting a flag in godoc for showing internal packages (for use during internal/company development purposes, to make it possible to easily browse internal APIs) - #12092 (comment). Thus (per the request for use cases in the main comment above), that's a use case me and several other people expressed interest in. If godoc is seen as "in maintenance mode" (which was quite a surprise to me), it would be nice if the plan for the new replacement tool was outlined here (?) more clearly, or something. Currently, the situation & status may appear unclear to community members (like me).

[thediveo]

A major stumbling Block for us is the inferior Support of private modules and APIs in turn. In our past, a dev-system local godoc helped us, even if that required developers to check out the API repo. godoc already was problematic when viewed from a typical private GitHub and Gitlab use case, as you can't get any static pages to be hosted. gopkg seems to make the situation even worse, as it now requires more and more infrastructure and doesn't scale down.

[thediveo]

Adding to the injury of pkgsite is that it doesn't support hot reloading when the local documentation changes as part of writing, updating, and maintaining module documentation, before tagging and pushing to a public repository. godoc did.

[smoyer64]

@achille-roussel - how are you running your patched version of pkgsite? I'd like to (at a minimum) server the module in my current working directory. At this point, I'm missing godoc!

[thediveo]

#!/bin/bash
set -e

if ! command -v pkgsite &>/dev/null; then
    export PATH="$(go env GOPATH)/bin:$PATH"
    if ! command -v pkgsite &>/dev/null; then
        go install golang.org/x/pkgsite/cmd/pkgsite@master
    fi
fi

# In case the user hasn't set an explicit installation location, avoid polluting
# our own project...
NPMBIN=$(cd $HOME && npm bin)
export PATH="$NPMBIN:$PATH"
if ! command -v browser-sync &>/dev/null; then
    (cd $HOME && npm install browser-sync)
fi

if ! command -v nodemon &>/dev/null; then
    (cd $HOME && npm install nodemon)
fi

# https://stackoverflow.com/a/2173421
trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT

# https://mdaverde.com/posts/golang-local-docs
browser-sync start --port 6060 --proxy localhost:6061 --reload-delay 2000 --reload-debounce 5000 --no-ui --no-open &
PKGSITE=$(which pkgsite)
nodemon --signal SIGTERM --watch './**/*' -e go --exec "browser-sync --port 6060 reload && $PKGSITE -http=localhost:6061 ."

[cespare]

I tried using pkgsite as a local/private documentation viewer and found that it was a poor replacement for godoc currently. In addition to the issues already listed here (and #57742 and #50229) I noticed:

• The search functionality should be completely hidden if it is non-functional.
• The top banner should not appear if the webserver isn't serving pkg.go.dev. (This includes the non-functional search box and the link to go.dev.)
• It should be easy to hide the right-side column (with information about licenses, vulnerability reporting, etc) since that is generally irrelevant for local/private use cases

[gopherbot]

Change https://go.dev/cl/474295 mentions this issue: cmd/pkgsite: add multi-module support, invalidation, and search

[gopherbot]

Change https://go.dev/cl/475755 mentions this issue: cmd/pkgsite: hide irrelevant content in local mode; add quick links

[findleyr]

Hi folks. I've been using x/pkgsite/cmd/pkgsite recently and stumbled upon some of the problems cited here, so I added the following improvements:

• By default pkgsite (no arguments) hosts documentation for all main modules (those listed by go list -m). So you don't need to run it from a directory containing a go.mod file -- it can be run from a nested directory -- and it will honor go.work files.
• Added a basic search implementation. Quick-and-dirty, but works for me. No symbol search.
• Added links on the homepage for modules/directories being served.
• Documentation is re-generated when files change.
• Removed UI elements that don't make sense in local mode.

Screencast below.

I have additional plans to fix stdlib documentation to not require a network connection. Apart from that, is there anything else people need? I'd be willing to make additional changes now, while it is fresh in my mind.

[cespare]

@findleyr thanks very much for your effort here! This definitely covers the bulk of the issues that I encountered when trying to use pkgsite as a local or private doc server as a replacement for godoc.

I played around with it and for the most part everything looks great. I notice a couple of remaining regressions in comparison to local godoc:

1. Reloading (local) code is very slow. For example, I have a simple test package in a local module that I loaded a few times in godoc and pkgsite. Godoc seems to render the whole thing from scratch each time; it takes about 200-250ms to load the page regardless of whether the package was modified since the last load. Pkgsite takes about 150-180ms when the package has been loaded before (great!). But when I make some trivial change to the package, loading the page takes about 3s. In the pkgsite logs I see a bunch of messages about things in other packages (and a lot of mention of node_modules directories). I'm guessing that pkgsite reprocesses the whole module when there is a change. Is there any way to speed this up? Can I at least give pkgsite a hint about certain subtrees it should ignore (like anything named node_modules)? If I delete all the node_modules directories in my module then it takes about 2.5s so it definitely seems to be wasting effort on that, at least.

2. Code links are much less useful than godoc or pkg.go.dev. For example, consider https://pkg.go.dev/github.com/jordan-wright/email#NewEmail. If I click on the link in pkg.go.dev, it takes me to https://github.com/jordan-wright/email/blob/943e75fe5223/email.go#L63:

   

   If I click on the equivalent link in my local godoc, it shows me a rendered view of the code with the function highlighted. It's not as useful as GitHub, but it has some minimal syntax highlighting to differentiate comments and is fairly helpful as a quick reference:

   

   When I click the link in local pkgsite, I get a raw text file:

   

   (The #L61 fragment doesn't do anything in this context.)

   It would be great if local pkgsite could (a) link to GitHub or (b) provide a more helpful rendering of the code. If we had (a), then I understand that pkgsite doesn't know where to link to for private code, but ideally I could provide some of kind of "URL template" to make private code links work too.

   Finally, one last quirk I noticed with code links is that there are no links for stdlib code at all:

   

   Why are stdlib packages treated differently? We ought to be able to at least show the raw code for those as well too, right?

Lest this sound too critical, I want to emphasize that I'm really grateful for your work on this. Your changes take pkgsite about 90% of the way from where it was to where I was hoping it would be for my use cases. Even without any further improvements I imagine I could mitigate the issues I mentioned on my end (such as by automatically triggering rebuilds when the code changes for (1) or by post-processing the HTML output for (2)).

[findleyr]

Thanks for testing it out, @cespare! Responses below.

1. Reloading (local) code is very slow.

I was worried about this. Yes, I squeezed the reloading logic into the existing APIs, all of which operate on modules. I will revisit this and push the reloading logic down so that it can reload packages individually. This would also make the initial rendering faster. I'm a little concerned about the additional complexity this may cause; we'll see how this works out.

2. Code links are much less useful than godoc or pkg.go.dev.

Oh, interesting. I think it should be possible to fix this. I'll have a look.

3. Finally, one last quirk I noticed with code links is that there are no links for stdlib code at all

This is https://go.dev/issue/58923, which I'll fix. TL;DR: the standard library has special handling in pkgsite, which cmd/pkgsite doesn't touch. As a result, when navigating to standard library packages they are downloaded rather than read from GOROOT. I plan to fix this by making the standard library handled just like any other module, at which point links should work.

[hyangah]

@findleyr Thanks for working on this.
Do you mind updating the remaining task list included in the top comment (cc @julieqiu) ?

Also #59542 (handling vendored dependencies)

[findleyr]

@hyangah I'd already edited the top comment. Is there something else you want me to include there?

EDIT: Well, I've added #59542 to the list, which I thought would be a straightforward bug but in fact is related to being able to refresh at the package level: for vendored packages, we have no module dir.

[adonovan]

One other item we should add to the list is the ability to display unexported symbols on request. See for example:

• x/tools/cmd/godoc: show internal packages when explicitly requested #12092 corresponding feature request in godoc (which exists but is hard to discover)
• proposal: x/tools/{godoc,cmd/godoc}: isolate, tag, and delete godoc #59056 (comment) a request for this feature from a hold-out of x/tools/godoc

[tonglil]

Has anyone else gotten module versioning / browsing versions to work? Even with a local go proxy setup?