Allow for custom requesters to be defined in low-code manifests #21001
Conversation
|
added test for this |
|
It think it is interesting to allow the custom implementations of RecordExtractor. I also remember us discussion that it could be interesting to reduce the number of custom implementations we have. I'm not sure I understand why |
We do sometimes ask to "promote" components that are generalizable. But we don't always have a set timeline for that. It very well could be, but as to not impact GL's velocity or if we haven't come to an exact decision on if we want to support it long term we tend to err on the side of caution. Once we allow it we'll always have to support it.
As mentioned above, probably worth a discussion, but also falls a little outside the scope of this change. I think there is potential use case for many of the more general components (retrievers, extractors, selectors, requester, error handlers) and extractors fit into that. But with caveat that we should be only going as far as the interface level. CustomRetrievers and not CustomSimpleRetriever. The less custom components the better, but in this case, it also does demonstrate why we need them on extractors because there are use cases we don't current cover and there might be more in the future. And also noting since it's relevant, source-monday was updated to use |
|
/publish-cdk dry-run=false
|
* Allow for custom requesters to be defined in low-code manifests * add test for custom requester component * bump versions and changelog
What
In the handwritten manifest, we're not currently allowing for developers to override the default HttpRequester. This wasn't a problem to this point because there were no cases of overriding HttpRequester. However in
source-monday, it is currently implementing a custom InterpolatedRequestOptionsProvider which is a component we are deprecating, so I'm advising that they instead use a CustomRequester which is why we need this now.More context as to the why:
https://airbytehq-team.slack.com/archives/C02U9R3AF37/p1672763295299529
How
Adds CustomRequester to the YAML manifest schema and regenerated the latest models.