Lineage Ingestion workflow for Postgres database not applying inclusion/exclusion patterns

[mohammadalijafri]

Affected module
Lineage Ingestion workflow

Describe the bug
The database inclusion/exclusion patterns cannot be used to control the scope of an ingestion workflow running against a Postgres database. This also appears to be the case with schema level inclusion/exclusion patterns. Have not tried this on table level inclusion/exclusion.

To Reproduce

1. Create an Ingestion workflow for a Postges database
2. Under the "Database Filter Pattern" field "Includes", put anything that doesn't exist, eg. "nonexistent_database".
3. Alternatively, put some known database into the "Excludes" list, eg. "data_warehouse".
4. Run the ingestion

Expected behavior
Filters should be applied

• If inclusion is for something non-existent, then no databases should have lineage analysis , and record count should be zero.
• If the exclusion is for a database that exists, then that database should be ignored from lineage analysis

Actual behaviour
Filters are not being applied

• Despite instruction to include something that doesn't exist, all databases are being included, and with reporting records as successfully ingested.
• Despite being asked to exclude a known database, eg I can see the query history SQL fluff processor parsing queries on the unwanted database.

Version:

• OS: Ubuntu 20.04
• Python version:
• OpenMetadata version: 1.6.1
• OpenMetadata Ingestion package version: docker.getcollate.io/openmetadata/ingestion:1.6.1

Additional context
These filters work as anticipated on Metadata ingestion, but on Lineage ingestion they are not having the intended effect. When a metadata ingestion starts with filters defined and working, there is an intial debug log entry reporting what has been filtered out. However, there is no such logfile entry upon commencement of a lineage ingestion.