proposal: e2e test

[nic-chen]

Background

Manager api no longer interacts with admin api, but writes data directly to etcd.

At present, we do not have the e2e test cases that cover from dashboard, manager api to Apache APISIX.

Plan

1. e2e test is divided into two parts, front-end e2e and back-end e2e

2. Because the front-end only uses part of the interface of manager api, the front-end e2e only needs to ensure normal interaction with manager api

3. Back-end e2e, APISIX is started through docker, Go calls manager api to configure routing and other resources, then access APISIX for verification.

Details

Development language: Go

Test library: https://github.com/gavv/httpexpect

Test environment: Start with docker-compose, a 3-node ETCD cluster, and 2-node APISIX cluster (2 nodes need to be verified at the same time).

Since APISIX already has complete test cases, we plan to migrate the test cases under the APISIX t directory.

Because of the huge workload, I will first migrate the test cases under t/node/ and t/router/, and leave other use cases to others who have time, or migrate later when I have time to complete the coverage.

Test cases

1. route

• host:
  Configure the correct host and hosts through the manager api, including wildcard domain name, and verify the correctness of the route by accessing APISIX.
  Confirm that the manager api cannot configure host and hosts at the same time
  Make sure that the manager api cannot be configured with wrong hosts and hosts, including illegal strings, too high length, wrong data types, etc.

• invalid conf
  Confirm that the manager api verifies the correctness of each field. include:
  String: length check, regular expression check
  IP: v4 v6 format verification
  int, float, bool and other types check

• config route with not exist service_id or upstream_id

• remote_addr, remote_addrs: v4 v6 format verification

• uri, uris
  invalid value check
  empty value check

• vars: args, cookie, header, different operators
  check combinations of different types of vars and operators

• methods
  Wrong data type (not string array)
  Not in enumerated values

• priority
  Customize different priority checks
  Custom and default priority verification

• plugins
  Extract at least one plug-in from auth, limit, log, trace, metrics and other types respectively for verification

• plugin orchestration check (normal and invalid check)

• other fields: name, desc, create_time, update_time, labels, filter_func(can't verify in Go, TODO)

• not exist field
  Confirm that the manager api can refuse to configure undefined fields

• Verify routing correctness under multi routes

• router engine: check radixtree_uri first, radixtree_host_uri TODO

2. upstream

• key: remote_addr, query_string, arg_xxx
• hash_on: consumer, header, cookie, vars
• node: ipv6, domain, array nodes (struct)
• ewma: latency, frequency
• roundrobin:, weight
• timeout
• retries
• websocket
• health check: active, passive, healthy, unhealthy
• invalid conf

3. ssl

• cert + key + sni
• cert + key + snis
• certs + keys + snis
• wildcard domain name

4.consumer

• missing auth plugin
• two or more auth plugins
• with plugins

5. service

• invalid route
• route with empty service

6. global rules

• add
• update
• delete
• cover by route plugins

7. plugins merge

• route + service
• route + consumer
• route + service + consumer

---

The test cases only describe the route part in detail, and the other parts will be detailed in the issue later.

[moonming]

it will be best if add checkbox for each test case

[nic-chen]

it will be best if add checkbox for each test case

OK

[juzhiyuan]

Just added this issue to 2.1, feel free to adjust it.

[membphis]

Is there a small PR as a sample? For example, test the host part of route.

And we can run it in CI, confirm the manager works fine.

BTW, running e2e use cases, can we get the latest code coverage results?
Code coverage is a very important indicator for testing.

[nic-chen]

• Here is an example PR:
  test: add e2e test for field hosts in route api #612

• For the overall process of backend e2e testing, please refer to:
  https://github.com/apache/apisix-dashboard/pull/612/files#diff-22340e8e8ab51438af10cdd230be186d1ef57c49ca1a89ac51fe4f6034bb870f

• Initialization and login of manager api have been implemented in base.go:
  https://github.com/apache/apisix-dashboard/pull/612/files#diff-7fc06fc4ee2fd9aa89677944844b938da1f18530ebbd9a86b0869d93f46b3c29

• For specific test cases, please refer to:
  https://github.com/apache/apisix-dashboard/pull/612/files#diff-a51586806ed3d846236a041f0b2fadb57413d13b73b2d5199526d88c00f796a6

• Local debugging

# run APISIX and ETCD 

$ cd api/test/docker && docker-compose up -d

# run test

$ export APIX_ETCD_ENDPOINTS=127.0.0.1:2379,127.0.0.1:3379,127.0.0.1:4379

$ cd ../e2e && go test

[moonming]

please add tag for those small issues.

[nic-chen]

put it to next milestone to continue this job.
will create small issues to add test cases step by step.