Implement Discover mode #1
Conversation
Dockerfile
Outdated
| @@ -0,0 +1,19 @@ | |||
| # syntax=docker/dockerfile:1 | |||
|
|
|||
| ARG GO_VERSION=1.18rc1 | |||
There was a problem hiding this comment.
Should do 1.18.3, same for CI.
| "strings" | ||
| ) | ||
|
|
||
| // PlanetScaleSource defines a configured Airbyte Source for a PlanetScale database |
There was a problem hiding this comment.
Probably grep for uses of airbyte throughout and replace them.
| return tables, errors.Wrap(err, "unable to iterate table rows") | ||
| } | ||
|
|
||
| return tables, err |
There was a problem hiding this comment.
All errors have already been checked so this is a little misleading, just return tables, nil.
cmd/internal/types.go
Outdated
| func TableCursorToSerializedCursor(cursor *psdbconnect.TableCursor) (*SerializedCursor, error) { | ||
| d, err := codec.DefaultCodec.Marshal(cursor) | ||
| if err != nil { | ||
| return nil, errors.Wrap(err, "unable to marshal table cursor to save staate.") |
There was a problem hiding this comment.
*state, remove period for error chaining purposes.
cmd/internal/types.go
Outdated
| Metadata NodeMetadata `json:"metadata"` | ||
| } | ||
|
|
||
| func (s *Stream) GenerateMetadata() error { |
There was a problem hiding this comment.
It's a little confusing to figure out what this function does since it implicitly requires certain fields to be set on Stream which it then reads and does updates with internally. Consider passing key properties as an explicit argument.
cmd/singer-tap/main.go
Outdated
| execute(discoverMode, configFilePath, catalogFilePath, stateFilePath) | ||
| } | ||
|
|
||
| func execute(discoverMode bool, configFilePath, catalogFilePath, stateFilePath string) { |
There was a problem hiding this comment.
Have this return an error and then do the log/os.Exit in main.
This PR implements the Discover Mode for a singer tap as described in the specification.
A lot of the code here looks similar because its been purloined from PlanetScale's Airbyte source.
The key difference being that the types we return from the database access types are Singer types, and not Airbyte.
Key Points
stdout, and all log & error messages go tostderr, which is why you see the logger accept anoutand anstderrmetadataobject array on aStreamtype.TODO