Embedded etcd — not possible for my use case?

[peterbourgon]

This is a spiritual successor to #4531. Thanks to your tips, I'm able to connect a raft.Node to my net.PacketConn transport. Apparently, it needs a state machine which satisfies this interface:

// stateMachine is the behavior required by the raft.Node
// and should be implemented by e.g. an etcd server.
type stateMachine interface {
    applySnapshot(raftpb.Snapshot) error
    applyCommittedEntry(raftpb.Entry) error
}

My goal here is to provide the etcd API (v3) to my users, but in-process e.g. as a Go interface. (I am willing to do a gRPC connect to localhost, if necessary.) So, I expect etcdserver.Server to satisfy both of these things. But there are many options in the etcdserver.ServerConfig which I don't think I have answers for:

• I don't use a filesystem, so I don't have DataDir, DedicatedWALDir, MaxSnapFiles, MaxWALFiles — all of this has to do with my storage layer, which I don't expect the state machine to care about
• I don't use an HTTP transport, so I don't have ClientURLs, PeerURLs, PeerTLSInfo, and I think also not InitialPeerURLs, InitialClusterToken — again, don't understand why the state machine needs this knowledge
• I don't use your discovery mechanism, so I don't have DiscoveryURL, DiscoveryProxy

The fact that all of these options are ambiguously or not defined for my use case, leads me to believe that I'm doing something wrong. Or maybe that etcd can not be used in the way that I intend. Can you please give your opinion? Have I made some false assumptions? Is there another object I should be using? Or maybe I'm going about this the wrong way?

Thank you very much for your time!

[xiang90]

@peterbourgon I see a few separate things from this issue.

My goal here is to provide the etcd API (v3) to my users, but in-process e.g. as a Go interface. (I am willing to do a gRPC connect to localhost, if necessary.)

We plan to support this. But not high priority for now, check #4435.

I don't use your discovery mechanism, so I don't have DiscoveryURL, DiscoveryProxy

The discovery mechanism should not even be part of etcd.Server. We will move it out soon.

I don't use a filesystem, so I don't have DataDir, DedicatedWALDir, MaxSnapFiles, MaxWALFiles — all of this has to do with my storage layer, which I don't expect the state machine to care about

I don't use an HTTP transport, so I don't have ClientURLs, PeerURLs, PeerTLSInfo, and I think also not InitialPeerURLs, InitialClusterToken — again, don't understand why the state machine needs this knowledge

We are not going to change these two for etcd server interface. And etcdserver is the glue code, not the actual application state machine. This is what etcd designed for.

Or maybe that etcd can not be used in the way that I intend. Can you please give your opinion?

It definitely can. You should be able to reuse most of code in etcd:

v3 API: https://github.com/coreos/etcd/blob/master/etcdserver/v3demo_server.go (use your own server)

storage API: https://github.com/coreos/etcd/blob/master/storage/consistent_watchable_store.go#L50 (use your own storage)

So my suggestion is that you should build your own transportation layer, and maybe your own storage layer. And hook them up with your own Server implementation. You can share the same raft, server API and kv API implementation with etcd.

[xiang90]

@peterbourgon BTW, can you share what is your use case in more detail?

[peterbourgon]

Thank you for that detailed reply. Happy to provide more detail of my own.

I have this gossip library called mesh, it has many nice features I won't get in to here and exposes a specific gossipy API. On top of that I've built meshconn, which implements net.PacketConn (think of it as UDP) on top of mesh. So far so good.

Now I want to build some component that will take the meshconn as input and provide the etcd API as output. Right now it's raft-on-mesh prototype code. The important part is transport.go, which is probably not the right name for what it does. That's where I raft.StartNode and have a loop to drive it. I have basic implementation for the Raft requirements, but now I want to push entries into my state machine.

Right now I have a no-op implementation but in reality I want to create some coreos/etcd/something object, which implements this state machine interface (or something like it) and provides the etcd API (or something like it) on the other side. Because I like diagrams, here's one:

+----------+
| mesh     |
+----------+ +---------+
| meshconn | | Storage |
+----------+-+---------+ 
| raft.Node driver     |
| (transport.go)       |
+----------------------+ <- type stateMachine interface
| State machine        |
+----------------------+
| Translation layer?   |
+----------------------+ <- etcdserverpb/rpc.proto
    ^  ^  ^
    |  |  |
  etcd consumers

When two boxes are touching I expect them to know about and talk to each other, but otherwise not. So no connection between state machine and storage, for example. Maybe you can adjust the drawing if I got some expectation wrong ;) but this is what I expect. I'm trying to figure out what to plug into the "State machine" box. And given what you say here...

• I don't use a filesystem, so I don't have DataDir, DedicatedWALDir, MaxSnapFiles, MaxWALFiles — all of this has to do with my storage layer, which I don't expect the state machine to care about

• I don't use an HTTP transport, so I don't have ClientURLs, PeerURLs, PeerTLSInfo, and I think also not InitialPeerURLs, InitialClusterToken — again, don't understand why the state machine needs this knowledge

We are not going to change these two for etcd server interface. And etcdserver is the glue code, not the actual application state machine. This is what etcd designed for.

...I guess it is not EtcdServer. What is it?

It definitely can. You should be able to reuse most of code in etcd:
v3 API: https://github.com/coreos/etcd/blob/master/etcdserver/v3demo_server.go (use your own server)

I don't totally understand. Here you define some interfaces that mirror the V3 API (got that much) but you implement them on EtcdServer, which I can't use. This, plus...

So my suggestion is that you should build your own transportation layer, and maybe your own storage layer. And hook them up with your own Server implementation.

...leaves me a little confused. What do you mean by "use your own server"? I am trying to avoid reimplementing the core etcd business logic, a.k.a. the state machine. Is that possible?

EDIT: I had a sort of "a-ha" moment with contrib/raftexample. It doesn't work like I expected, but I think I understand it now. I'm digging in and will report back with my findings.

[xiang90]

I had a sort of "a-ha" moment with contrib/raftexample. It doesn't work like I expected, but I think I understand it now. I'm digging in and will report back with my findings.

I still want to reply since I got some extra time tonight :) Hope it can offer you more information anyway.

What is it

EtcdServer struct is more like an engine right now. It acts as transporter, API layer, and a applier that pulls committed entries out of raft to execute them.

Here you define some interfaces that mirror the V3 API (got that much) but you implement them on EtcdServer, which I can't use. This, plus...

Most of the implementation is not on etcdserver. https://github.com/coreos/etcd/blob/master/etcdserver/v3demo_server.go#L248-L643.

the state machine. Is that possible

The core business are in the storage pkg and the API part I mentioned above. The state machine (i would like to call it the applier since we have a pull model) part is trivial really, it just parses the raft entry and let the executor to execute them. The actual executor is the storage.KV.

[peterbourgon]

OK. Thanks to @xiang90 help I got the basic stuff working :) I will have more issues with more questions in the future, but for now I am very happy! Thank you.