Skip to content

Latest commit

 

History

History
39 lines (27 loc) · 3.87 KB

README.org

File metadata and controls

39 lines (27 loc) · 3.87 KB

APISnoop Readme

A suite of tools to assist kubernetes test-writing and investigate test coverage.

Introduction

APISnoop is a collection of programs intended to run in a kubernetes cluster, that help with multiple aspects of conformance testing. These programs include:

snoopdb
a postgres database for running queries about the state of conformance coverage. This db starts up with audit events from a recent test run and the latest kubernetes api spec. This allows you to run queries to see exactly which endpoints in the spec are covered by exactly which tests. It can also be connected to auditlogger to create a more data-driven live-testing-environment.
auditlogger
a simple server that, when attached to a kubernetes cluster configured with an auditsink, will listen to all events in that cluster and insert them into the snoopdb’s testing tables. This allows you to perform mock tests against a cluster and then query the results immediately to ensure that your mock test is hitting the endpoints you intended.

We use snoopdb to generate the data for the site https://apisnoop.cncf.io.

APISnoop Conformance Gate

We use APISnoop currently in a prow job to check if a kubernetes release includes any new, conformance endpoints that do not have conformance tests. the intention is to help reduce technical debt, by ensuring all conformant endpoints have conformant coverage.

The testgrid for the job can be found here: https://testgrid.k8s.io/sig-arch-conformance#apisnoop-conformance-gate

How the job works

The job uses our snoopdb image. This image, on startup, runs through a set of sql initialization scripts including one that gathers any new, untested conformant endpoints. If the image detects it’s being run as part of prow, it will then do a simple check if its list of untested endpoints is greater than 0. If so, the job fails and we output the latest endpoints. Otherwise, it passes and the db shuts down, completing the job.

Snoopdb uses the following sources for determining tested conformant endpoints:

We define “new conformance endpoints” as the endpoints in the most recent openAPI spec that are stable, not deprecated, and are not included in the ineligible-endpoints.yaml.

We define “tested” to mean the endpoint shows up in the audit logs of the conformance test suite as a target of a conformance test(e.g. a test that is included in the above conformance.yaml)

The logic for this work is present among the sql files in snoopdb’s initdb. Mainly:

Learning more

To learn more about the database, you can check out its README: ./apps/snoopdb/README.md Or, for auditlogger:./apps/auditlogger/README.org