Web Risk Client App | Container & Go

Web Risk is the enterprise version of Google's Safe Browsing API that protects 5 Billion devices globally from dangerous URLs including phishing, malware, unwanted software, and social engineering.

This client implements the Web Risk Update API, which allows for URLs to be checked for badness via privacy-preserving and low-latency API. It works out-of-the-box via either Docker or Go.

This README provides a quickstart guide to running a client either with Docker or as Go binaries. It also serves as a reference implementation of the API. The GoDoc and API documentation in the .go source files provide more details on fine-tuning the parameters if desired.

Supported clients:

• wrserver runs a thin HTTP client that can query URLs via a POST request or a redirection endpoint that diverts bad URLs to a warning page. This is the client wrapped by Docker.
• wrlookup is a command line service that takes URLs from STDIN and outputs results to STDOUT. It can accept multiple URLs at a time on separate lines.

Supported blocklists:

• MALWARE
• UNWANTED_SOFTWARE
• SOCIAL_ENGINEERING
• SOCIAL_ENGINEERING_EXTENDED_COVERAGE

The client is originally forked from the Safebrowsing Go Client.

Enable Web Risk

To begin using Web Risk, you will need a GCP Account and a project to work in.

1. Enable the Web Risk API.

2. Create an API Key.

3. Enable Billing for your account and make sure it's linked to your project.

Install Docker and/or Go

To use the Container App, you will need Docker. To compile binaries from source or run tests install Go.

Docker Quickstart (recommended)

We have included a Dockerfile to accelerate and simplify onboarding. This container wraps the wrserver binary detailed below.

Clone and Build Container

Building the container is straightforward.

First, clone this repo into a local directory.

git clone https://github.com/google/webrisk && cd webrisk

Build the container. This will run all tests before compiling wrserver into a distroless container.

docker build --tag wr-container .

Run Container

We supply the APIKEY as an environmental variable to the container at runtime so that the API Key is not revealed as part of the docker file or in docker ps. This example also provides a port binding.

docker run -e APIKEY=XXXXXXXXXXXXXXXXXXXXXXX -p 8080:8080 wr-container

wrserver defaults to port 8080, but you can bind any port on the host machine. See the Docker documentation for details.

See Using wrserver below for how to query URLs or use the redirection endpoint.

Go Binary Quickstart | wrlookup example

The Go Client can be compiled and run directly without Docker. In this example we will use that to run the wrlookup binary that takes URLs from STDIN and outputs to STDOUT.

Before compiling from source you should install Go and have some familiarity with Go development. See here for a good place to get started.

Clone Source & Install Dependencies

To download and install this branch from the source, run the following commands.

First clone this repo into a local directory and switch to the webrisk directory.

git clone https://github.com/google/webrisk && cd webrisk

Next, install dependencies.

go install .

Build and Execute wrlookup

After installing dependencies, you can build and run wrlookup

go build -o wrlookup cmd/wrlookup/main.go

Run the binary and supply an API key.

./wrlookup -apikey=XXXXXXXXXXXXXXXXXXXXXXX

You should see some output similar to below as wrlookup starts up.

webrisk: 2023/01/27 19:36:46 database.go:110: no database file specified
webrisk: 2023/01/27 19:36:53 database.go:384: database is now healthy
webrisk: 2023/01/27 19:36:53 webrisk_client.go:492: Next update in 30m29s

wrlookup will take any URLs from STDIN. Test your configuration with a sample:

http://testsafebrowsing.appspot.com/s/social_engineering_extended_coverage.html #input
Unsafe URL: [SOCIAL_ENGINEERING_EXTENDED_COVERAGE] # output

Using wrserver

wrserver runs a WebRisk API lookup proxy that allows users to check URLs via a simple JSON API. This local API will use the API key supplied by the Docker container or the command line that runs the binary.

First start the wrserver by either running the container or binary.

To run in Docker:

docker run -e APIKEY=XXXXXXXXXXXXXXXXXXXXXXX -p 8080:8080 <container_name>

To run from a CLI, compile as wrlookup above and run:

./wrserver -apikey=XXXXXXXXXXXXXXXXXXXXXXX

With the default settings this will start a local server at 0.0.0.0:8080.

The server has a lightweight implementation of a Web Risk Lookup API-like endpoint at v1/uris:search. To use the local endpoint to check a URL, send a POST request to 0.0.0.0:8080/v1/uris:search with the a JSON body similar to the following.

{
  "uri":"http://testsafebrowsing.appspot.com/s/social_engineering_extended_coverage.html"
}

A sample cURL command:

curl -H 'Content-Type: application/json' \
	-d '{"uri":"http://testsafebrowsing.appspot.com/s/social_engineering_extended_coverage.html"}' \
	-X POST '0.0.0.0:8080/v1/uris:search'

See Sample URLs below to test the different blocklists.

wrserver also serves a URL redirector listening on /r?url=... which will show an interstitial for anything marked unsafe.

If the URL is safe, the client is automatically redirected to the target. Otherwise an interstitial warning page is shown as recommended by Web Risk.

Try some sample URLs:

http://0.0.0.0:8080/r?url=https://testsafebrowsing.appspot.com/s/social_engineering_extended_coverage.html
http://0.0.0.0:8080/r?url=https://testsafebrowsing.appspot.com/s/malware.html
http://0.0.0.0:8080/r?url=https://www.google.com/

Differences from Web Risk Lookup API

There are two significant differences between this local endpoint and the public v1/uris:search endpoint:

• The public endpoint accepts GET requests instead of POST requests.
• The local wrserver endpoint uses the privacy-preserving and lower latency Update API making it better suited for higher-demand use cases.

Sample URLs

For testing the blocklists, you can use the following URLs:

• Phishing / Social Engineering: https://testsafebrowsing.appspot.com/s/phishing.html
• Malware: https://testsafebrowsing.appspot.com/s/malware.html
• Unwanted Software: https://testsafebrowsing.appspot.com/s/unwanted.html
• Social Engineering Extended Coverage: https://testsafebrowsing.appspot.com/s/social_engineering_extended_coverage.html

Troubleshooting

4XX Errors

If you start the client without proper credentials or project set up, you will see an error similar to what is shown below on startup:

webrisk: 2023/01/27 19:36:13 database.go:217: ListUpdate failure (1): webrisk: unexpected server response code: 400

For 400 errors, this usually means the API key is incorrect or was not supplied correctly.

For 403 errors, this could mean the Web Risk API is not enabled for your project or your project does not have Billing enabled.

Configuration

Both wrserver (used by docker run) and wrlookup support several command line flags.

• apikey (required) -- Used to Authenticate requests with the Web Risk API. The API itself must also be enabled on the same project & be linked to a Billing account.

• threatTypes (optional) -- A comma-separated lists of different blocklists to load and check URLs against. Available options include MALWARE,UNWANTED_SOFTWARE,SOCIAL_ENGINEERING, SOCIAL_ENGINEERING_EXTENDED_COVERAGE. This arg will also accept ALL which is the default behavior.

• maxDiffEntries (optional) -- An int32 value that will set the max number of hash prefixes returned in a single diff request. This can be used in resource-bound environments to control bandwidth usage. The default value of 0 will result in this limit being ignored. Otherwise, this must be set to a positive integer which must be a power of 2 between 2 ^ 10 and 2 ^ 20.

• maxDatabaseEntries (optional) -- An in32 value that will set the upper boundary has prefixes to be returned from the API and stored locally. This can be used to limit the number of hash prefixes to be searched against. The default value of 0 will result in this limit being ignored. Otherwise, this must be set to a positive integer which must be a power of 2 between 2 ^ 10 and 2 ^ 20. Note: Setting this limit will decrease blocklist coverage.

About the Social Engineering Extended Coverage List

This is a newer blocklist that includes a greater range of risky URLs that are not included in the Safebrowsing blocklists shipped to most browsers. The extended coverage list offers significantly more coverage, but may have a higher number of false positives. For more details, see here.

WebRisk System Test

To perform an end-to-end test on the package with the WebRisk backend, run the following command after exporting your API key as $APIKEY:

go test github.com/google/webrisk -v -run TestWebriskClient