Skip to content

v2.2 - DataStage connector improvements

Compare
Choose a tag to compare
@cmgrote cmgrote released this 15 Sep 18:44

🎉 New features

  • Optimise the discovery of the oldest job since last sync, to apply the project limiters (previously ran across all jobs, ignoring limiters)
  • Ensures stage variables are bulk-retrieved as part of the up-front caching of jobs
  • Ensures that column-level lineage is traceable across both stage columns and stage variables
  • Forces lineage detection before trying to retrieve job details, to avoid potential race conditions between the connector polling and IGC calculating its own lineage details (to expose via its APIs)
  • Prefixes DataStage-owned qualifiedNames to be able to quickly bypass searches in IGC connector
  • Adds docs clarifying capabilities and limitations of the connectors
  • Extends IGC caching to the retrieval of full assets

🐞 Bug fixes

  • Ensures any cache misses of links and stage columns are individually looked up rather than skipped
  • Fixes issue with caching where a cache miss was not returning the information that was retrieved
  • Encodes URL path variables to ensure any special characters are appropriately handled against IGC's REST API
  • Fixes a potential NullPointerException when attempting to iterate through relationships that could be non-existent
  • Fixes issues with qualifiedNames generated for various DataStage objects, to ensure they are unique
  • Removes duplicative lineage mappings that were causing problems in certain stage-level jobs (ie. lookup stages)

🔨 Dependencies