Support TLS; Reduce the need of config.toml in integration tests

[kennytm]

What problem does this PR solve?

Fix #262.

What is changed and how it works?

1. Code changes:

   • Added the [security] section in the config to read CA, cert and key. These are used to construct the standard *tls.Config.
   • This config is used to:
     • Add a TLS layer on top of the web interface / metrics HTTP server
     • Encrypt gRPC communication with Importer and TiKV (via WithTransportCredentials)
     • Encrypt HTTP communication with PD and TiDB status port
     • Encrypt MySQL communication with TiDB (via the tls parameter in the DSN)
   • All of these operations are grouped into a struct common.TLS for simplified management. This struct mainly acts as an http.Client to fetch JSON objects, plus methods to produce options for securing gRPC and MySQL protocols.

2. Test changes:

   • The entire integration test is changed to use TLS for all communications.
   • The certs need be to pass into every tidb-lightning and tidb-lightning-ctl invocations. To simplify future changes, the run_lightning and run_lightning_ctl helper scripts now define most common settings in the command line.
   • Existing config.tomls are simplified to retain only the essential settings.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
  • check that web interface does serve HTTPS if required.

Side effects

Related changes

• Need to update the documentation
  • Describe the new config and CLI parameters
• Need to update the tidb-ansible repository
  • Copy the global TLS settings to config
• Need to be included in the release note
  • Note TLS support

[kennytm]

/run-all-tests

[kennytm]

PTAL @july2993 @3pointer

[july2993]

rest LGTM

[kennytm]

PTAL @july2993 @3pointer again thanks.

[july2993]

LGTM

[3pointer]

there are two kind of security, global security and tidb security, if pass different security here, it seems that importer and tikv will use global security and tidb, pd will use another. shoud tidb,pd,tikv use the same one?

[kennytm]

Only SQL connection uses [tidb.security] (corresponding to ssl-ca/cert/key in TiDB). Everything else should use [security] (corresponding to cluster-ssl-ca/cert/key in TiDB). If not it's a bug.

[3pointer]

LGTM