Create Ingest API Service in veda-backend

[ividito]

Background

Currently, veda-backend contains database, STAC, and raster tiling services, as well as standalone VPC infrastructure. Another repo, veda-stac-ingestor, contains an API for interacting with the ingestion process, both through direct insertion to a database and by submitting jobs to a pipeline that then interacts with the database.

We want to create a pattern in veda-backend that is comparable to cdk-pgstac, which also has an ingestor-api as its own module. This new service will centralize all database interactions within veda-backend, improving maintainability while also allowing the addition of any valid STAC object to the pgstac DB without necessarily requiring the object to contain VEDA/dashboard-specific rules defined in veda-stac-ingestor.

We also want to apply a DynamoDB serialization change that was implemented in cdk-pgstac - it makes sense to include this as part of this transition.

As a note - we aren't implementing cdk-pgstac within VEDA at this point, but the structure of that project aligns with our goals, and following that project's patterns could make it easier to transition to cdk-pgstac in the future.

AC

• Create a new ingest API service using a structure consistent with the current STAC and raster API services.
• Add infrastructure to manage our DynamoDB batch loading system
• Migrate all database and DynamoDB interactions (insertion, update, delete) from veda-stac-ingestor to the new ingest API service. Interactions with the ingestion pipeline are excluded from this migration.
• Apply DynamoDB serialization change to the new ingest API service.
• Using the new ingest API service, support adding any valid STAC object to the pgstac DB.
• Where possible, migrate unit tests from veda-stac-ingestor, and add them to veda-backend (the testing structure is somewhat different between the two repsitories)

Desired state (from https://github.com/NASA-IMPACT/veda-architecture/issues/215)

[ividito]

Current state - there is a PR (#174) up with functionality ported over from veda-stac-ingestor. Based on feedback, there's a draft PR(#191) open with some CDK restructuring to make our constructs more similar to what we have for our existing services, in order to maintain consistency in our repo.

In making this refactor, we are also evaluating eoapi-cdk's equivalent construct in the iac/eoapi-ingestor branch. We can't merge that in as-is just yet, since we have existing resources we want to use, as well as a handful of validations we would like to keep, but this is an opportunity to take a look at where the gaps are, as using that construct in the future could help improve reusability across projects and reduce complexity overall.