ADAPT-1435: Add replacementTopics & previousTopics fields to track lineage of a deprecated topic

[aman-minz]

PR for ADAPT-1435 introduces new fields as below to track the lineage of a topic being deprecated.

NOTE: The metadata fields replacementTopics & previousTopics should be populated by the producers of a topic.

replacementTopics:

1. Whenever a topic is deprecated it is tedious for the consumers to track the new topic which should be used instead.
2. Consumers required a field where the replacement topic could be assigned by the producers.
3. Thus, replacementTopics field in metadata was conceived.
4. This field is a list as a deprecated topic could be broken into multiple topics.
5. Validations:
   • Accept only supported pattern of the topic name.
   • [Only on topics created after this fix] replacementTopics field must be set whenever the current topic is deprecated.

---

previousTopics:

1. This field alone was not enough to trace the lineage at one glance.
2. With replacementTopics, one could trace the lineage from old-to-new.
3. However, to find lineage in other direction (i.e. new-to-old), one has to traverse from the deprecated topics to the newest topic.
4. Therefore, another metadata field, previousTopics was conceived.
5. This field needs be populated in the new topic which is created to replace a deprecated topic.
6. This field is a list as it should contain all previous topics which were deprecated and replaced by the current topic.
7. Validations:
   • Accept only supported pattern of the topic name.

---

additionalValidations:

1. Sometimes additional validations are required which should not affect the existing topics.
2. In other words, the additional validations should be applied only on the new topics and not the existing/old topics.
3. To accomplish this, we had introduced cutoff-date environment variable. However, once a date is set, it is ideally never going to change.
4. Therefore, why not have a field in the metadata to decide whether to apply an additional validation or not.
5. additionalValidations is introduced to get rid of cutoff-date.
6. It contains a list of enums denoting the type of validations to apply.
7. Enum could be corresponding to a field or a custom validation which should be applied only on the new topics.
8. [Past] Existing topics before this feature will have nothing in additionalValidations. Therefore, no new additional validations will be applied on them.
9. [Present] New topics after this fix will have replacementTopics. Thus, corresponding field will be validated.
10. [Future] When new enum values are added, the additionalValidations field of only the new topics will be updated with all additionalValidations including new enums. Thus, new additional validations will be applied henceforth.

---

[olivierlacan]

Is this still planned?

[aman-minz]

Is this still planned?

Yes, @olivierlacan. However, we have few other PRs which needs to go before this. We will update you as soon as this is merged.