Prototyping: Create a test environment for a minimal data collector with queues, logging and divided work phases

[Piiit]

• Search for a real-world data collector (or migrate an existing one, should be simple) @rcavaliere @ohnewein @bertolla @RudiThoeni
• message queue
• divide work into downloading --> queue#1 --> enrichment --> queue#2 --> one target (Postgres)
• logging
• scheduling
• grafana

[rcavaliere]

@Piiit I would suggest to start with an easy and stable Data Collector, e.g. Parking Merano

[alexlanz]

• Setting up one managed ActiveMQ cluster on AWS with two queues - 2-3d
• Programming: 2-4d
  • Write a specific data collector (downloading)
  • Write a specific data enricher (enchriment)
  • Write a general data writer (save into PostgreSQL)
• Logging: out of the box with the new Docker setup - 0d
• Scheduling: out of the box with using the Spring Boot scheduler - 0d
• Grafana: dashboard for the queues - 2-3d