Validate DO operations

[alasarr]

Any operation that require a dataset/geography in BQ download or enrichment must validate first that the requires datasets will be available in BQ.

So we're going to check this field for both (geography and dataset) before running an enrichment and before running a download

@cmongut please advise with the message

Blocked by: #1163

[alrocar]

Does this mean you can have a licensed dataset that does not exist in BQ? 🤔

[simon-contreras-deel]

Does this mean you can have a licensed dataset that does not exist in BQ?

I think the point is having datasets in BQ not validated or ready for "production".

@alasarr or @cmongut I have a question: a dataset or geography without the "available in bq" tag should appear in the catalog and we are not going to allow to work with it in enrichment and download. The question is: how should we do it in the subscription methods?

[alasarr]

Does this mean you can have a licensed dataset that does not exist in BQ?
No. they cannot be licensed, but the user could subscribe to it.

The question is: how do we should do in the subscription methods?

Users can subscribe to these kinds of datasets and we should receive the email. The backend won't license it but the email must be sent to bizops and data team to register the user's request. They will make sure to include the data in BQ and to update the catalog before licensing. The estimated_delivery_time of this kind of datasets is big enough to allow us to do it. It happens with dataset that are provided after-request by the data provider (customer asks for the dataset, we request it to the provider, we receive the data, we provide it to the customer).

@gonzaloriestra ☝️

[gonzaloriestra]

Right now we require the metadata to have available_in, estimated_delivery_days and subscription_list_price in order to subscribe or show subscription info: https://github.com/CartoDB/cartodb/blob/8f5420f0f942e2b42fe63163a29b2a766aa0d00b/app/controllers/carto/api/public/data_observatory_controller.rb#L146

What fields should we require then?