API server overlaying topology - RFC

[binarylogic]

Now that #3225 is finalized, we need an RFC that proposed how a GraphQL server will be embedded into Vector. Specific topics to cover:

1. Will this be "always on"?
2. How this will be configured with configuration examples.
3. Where this will be integrated in the code.
4. How this aligns with the internal_metrics source, ensuring that we are aligning this work for future roadmap phases. Specifically, aligning on the data model -- working with the data coming out of the internal_metrics source.

[leebenson]

I've updated this title to reflect the API overlaying topology, rather than this being a GraphQL-specific concern.

This touches both configuration of the API, and a design decision around whether the API is part of topology, or aside from it. In either case, the API must observe it.

GraphQL-specific discussions were broken out to a separate RFC in #3648.

[leebenson]

I've settled on what I think is a good pattern for observing topology.

In earlier versions, the API restarted with toplogy, and would attempt to read from RunningTopology directly.

Now because the API persists across watched changes, the raw Config itself is stored statically behind an Arc<RwLock>, which updates when changes are made. This gives fast access to readers (the 90%+ case) and means later queries for config pick up the latest without the API needing to restart. It will always return the latest.

This answers the following queries:

Will this be "always on"?

Yes

How this will be configured with configuration examples.

No configuration is necessary, save for api.enabled = true in vector.toml to enable the API server.

Where this will be integrated in the code.

Directly in the schema, with the slim addition of updating a lazy_static! in the schema to store a copy of the current config.

How this aligns with the internal_metrics source, ensuring that we are aligning this work for future roadmap phases. Specifically, aligning on the data model -- working with the data coming out of the internal_metrics source.

N/A. Metrics are observed directly from a controller, and outside the scope of topology. The two worlds meet when augmenting topology types with statistical fields. But since an individual field resolver in GraphQL is simply an async fn, what we do in the body the resolver is up to us-- making it trivial to reference a metrics source from configuration without requiring any upfront overlap.

The path forward is now quite clear -- creating GraphQL schema to include types for sources, transforms and sinks, and provide relationships between each in the graph.

[leebenson]

Closing, since this is superceded by current work on vector top.

[jszwedko]

@leebenson

To clarify, for:

Will this be "always on"?

Yes

Do you mean the API server will be enabled by default? Or something else?

[leebenson]

Do you mean the API server will be enabled by default? Or something else?

No, to clarify, the API server still requires an explicit:

[api]
  enabled = true

... in vector.toml.

But once it's on, it remains on.

It did originally auto-reload with other changes to vector.toml, as part of topology. But @lukesteensen commented not to mix the API with topology, so it doesn't currently restart when making changes such as turning enabled = false or setting an explicit bind value.

Those changes require a hard restart of Vector.

[jszwedko]

Aha, gotcha 👍 Thanks for the clarification. I'm excited to see this work coming along!