New Runner gRPC API

[NicolasMahe]

Related to #1764
Dependent on #1768 for the exists endpoint.

This PR adds the new Runner gRPC API that service must use.
The big difference between this API and the previous ones is this API is secure (requires authentication) and specifically design for Runners, so it is much easier to integrate.

The API is composed of 4 endpoints:

Register

The service must call this API first but only when it's ready to emit events and/or execute executions.
The service has to call this endpoint with the content of the env variable MESG_REGISTER_PAYLOAD (injected by the CLI) to get a credential token requires for the following endpoints.

Go example:

resp, _ := client.Register(context.Background(), &runner.RegisterRequest{
	Payload: os.Getenv("MESG_REGISTER_PAYLOAD"),
})
fmt.Println(resp.Token)

Once you got the token, the service need to pass it to the next endpoints in the gRPC metadata.
There are three ways to do this:

• You can use the gRPC helper WithPerRPCCredentials on the grpc DialOption of the client
• You can use the gRPC helper PerRPCCredentials on the CallOption of the request
• You can create the metadata object yourself and using the key mesg_credential_token

The first two options requires a struct that implement the interface credentials.PerRPCCredentials:

// credential is a structure that manage a token.
type credential struct {
	token string
}

// GetRequestMetadata returns the metadata for the request.
func (c *credential) GetRequestMetadata(context.Context, ...string) (map[string]string, error) {
	return map[string]string{
		runner.CredentialToken: c.token,
	}, nil
}

// RequireTransportSecurity tells if the transport should be secured.
func (c *credential) RequireTransportSecurity() bool {
	return false
}

Credential in client:

cred := &credential{
	token: resp.Token,
}
dialoptions := []grpc.DialOption{
	grpc.WithInsecure(),
	grpc.WithPerRPCCredentials(credential),
}
conn, _ := grpc.DialContext(context.Background(), os.Getenv("MESG_ENDPOINT"), dialoptions...)
client := runner.NewRunnerClient(conn)

Credential per request:

cred := &credential{
	token: token,
}
client.XXX(context.Background(), &runner.XXX{}, grpc.PerRPCCredentials(cred))

Event

This endpoint is used to emit events.
It accepts an event's key and data.
The credential token must be passed to this request.

Go example:

client.Event(context.Background(), &runner.EventRequest{
	Key: "test_service_ready",
	Data: &types.Struct{
		Fields: map[string]*types.Value{
			"msg": {
				Kind: &types.Value_StringValue{
					StringValue: "foo",
				},
			},
		},
	},
})

Execution

This endpoint is used to create a stream that will receive the execution the runner has to execute.
The credential token must be passed to this request.

Go example:

stream, _ := client.Execution(context.Background(), &runner.ExecutionRequest{})
for {
	exec, _ := stream.Recv()
	log.Printf("received execution %s %s\n", exec.TaskKey, exec.Hash)
}

Result

This endpoint is used to return the execution's result once the upcoming execution from the execution stream has been executed.
It accepts the execution's hash and the result.
The credential token must be passed to this request.

Go example:

client.Result(context.Background(), &runner.ResultRequest{
	ExecutionHash: exec.Hash,
	Result: &runner.ResultRequest_Outputs{
		Outputs: &types.Struct{
			Fields: map[string]*types.Value{
				"msg": {
					Kind: &types.Value_StringValue{
						StringValue: "foo",
					},
				},
			},
		},
	},
})

CLI

The CLI has to create and inject the env variable MESG_REGISTER_PAYLOAD that the service will use to register itself against the Engine.

This variable contains the data needed for the engine to register the runner as well as a signature that the engine verifies to check the authenticity of the runner by making sure the runner is created by the CLI.

The protobuf definition of this variable is:

// RegisterRequest is the request of the endpoint Register.
message RegisterRequestPayload {
  // signature of the value to verify authenticity.
  bytes signature = 1 [
    (gogoproto.moretags) = 'validate:"required"'
  ];

  // value contains what the engine actually needs to register the runner.
  Value value = 2 [
    (gogoproto.moretags) = 'validate:"required"',
    (gogoproto.nullable) = false
  ];

  // Value message
  message Value {
    // Service's hash to start the runner with.
    bytes serviceHash = 1 [
      (gogoproto.moretags) = 'validate:"required,hash"',
      (gogoproto.casttype) = "github.com/mesg-foundation/engine/hash.Hash"
    ];
  
    // Hash of the environmental variables to start the runner with.
    bytes envHash = 2 [
      (gogoproto.moretags) = 'validate:"omitempty,hash"',
      (gogoproto.casttype) = "github.com/mesg-foundation/engine/hash.Hash"
    ];
  }
}

The message Value is very close to the runner module MsgCreate. It contains the minimum of data so the engine can actually create, sign, and broadcast MsgCreate to register the runner on the network.

signature is the signature of the message Value (amino-json encoded) using the engine's account.

Finally, the whole RegisterRequestPayload message is also amino-json encoded to be injected in the env of the runner.

Here is the implementation of this system in the e2e runner test:

value := grpcrunner.RegisterRequestPayload_Value{
	ServiceHash: serviceHash,
	EnvHash:     instanceEnvHash,
}

encodedValue, _ := cdc.MarshalJSON(value)

signature, _, _ := kb.Sign(engineAccountName, engineAccountPassword, encodedValue)

payload := grpcrunner.RegisterRequestPayload{
	Signature: signature,
	Value:     value,
}
encodedPayload, _ := cdc.MarshalJSON(payload)

instanceEnv = append(instanceEnv, "MESG_REGISTER_PAYLOAD="+string(encodedPayload))

With MESG_REGISTER_PAYLOAD containing (without identation):

{
	"signature": "wpSfARExAzv5c4PivU7K++XpX1DMyoLaCmzCx7bYkwMJLfBSDbZMOngaG7tMrN+sogG2UuXkPcKDQMuhIxAQZA==",
	"value": {
		"serviceHash": "BoPaG69vP2ZoexqxFXkq94ZVfn9768GB8YCiYAdNWPju",
		"envHash": "J3PJvcXq3CXKEeP8jraYGUQSAj8g7cNxMNsMQgLqfJMW"
	}
}