Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release/2.24.7 #25

Merged
merged 125 commits into from
Nov 27, 2024
Merged
Changes from 4 commits
Commits
Show all changes
125 commits
Select commit Hold shift + click to select a range
dea50a0
feat(pdf): Test Result PDF report
rboni-dk Sep 27, 2024
764bd12
misc(ui): enhancements to button, link and flex widgets
aarthy-dk Sep 26, 2024
bed5d1c
refactor(ui): rename "toolbar_select" widget to generic "select"
aarthy-dk Sep 26, 2024
15b3941
feat(ui): add empty_state widget
aarthy-dk Sep 26, 2024
2048bba
refactor(dialogs): move execution dialogs to separate files
aarthy-dk Sep 26, 2024
5c1de82
feat(ui): add empty state messaging and links/actions to all top-leve…
aarthy-dk Sep 26, 2024
40b24e9
misc(ui): disable buttons when not relevant and styling improvements
aarthy-dk Sep 26, 2024
4826fc9
style(ui): lint changed files
aarthy-dk Sep 26, 2024
0cf62ac
fix(ui): update empty state widget
aarthy-dk Sep 30, 2024
330aca1
feat(pdf): Improving the PDF report aesthetics
rboni-dk Oct 1, 2024
a575af2
Merge branch 'aarthy/empty-state' into 'enterprise'
aarthy-dk Oct 3, 2024
f8e26f3
refactor(ui): improvements to vanjs components to support page compon…
aarthy-dk Oct 3, 2024
dc968b5
refactor(ui): use page components in test runs and profiling runs
aarthy-dk Oct 3, 2024
0d749e9
refactor(ui): improve and abstract event handling from page components
aarthy-dk Oct 4, 2024
79c708f
fix(ui): handle height of page components responsively
aarthy-dk Oct 4, 2024
f082c0a
style: improve typing
aarthy-dk Oct 4, 2024
77f107e
feat(pdf): Refacoring the code and fixing the header
rboni-dk Oct 5, 2024
769681d
Merge branch 'aarthy/page-component' into 'enterprise'
Oct 7, 2024
ad724e7
mic(pdf): Addressing team feedback
rboni-dk Oct 8, 2024
2f6c43b
mic(pdf): Adding a progress bar to the download dialog
rboni-dk Oct 9, 2024
f0e2ae9
Merge branch 'enterprise' of gitlab.com:dkinternal/testgen/dataops-te…
rboni-dk Oct 9, 2024
7c1aa58
fix(upgrade): add where clause in sql update statements with joins
aarthy-dk Oct 9, 2024
6d9d966
Merge branch 'aarthy/fix-upgrade' into 'enterprise'
Oct 9, 2024
deaf648
fix(pdf): Fixing the data table header aligment plus some other impro…
rboni-dk Oct 10, 2024
0cfd448
Merge branch 'enterprise' of gitlab.com:dkinternal/testgen/dataops-te…
rboni-dk Oct 10, 2024
5696b06
fix(pdf): Addressing code review feedback
rboni-dk Oct 10, 2024
f1147b1
Merge branch 'tg-812' into 'enterprise'
rboni-dk Oct 10, 2024
e3a1311
fix(auth): handle cookies retrieved inconsistently
aarthy-dk Oct 10, 2024
e9121f6
Merge branch 'aarthy/cookies' into 'enterprise'
Oct 10, 2024
a86be02
fix(misc): Download dialog code cleanup (close, progress)
rboni-dk Oct 14, 2024
98319b0
Merge branch 'tg-812-dialog' into 'enterprise'
rboni-dk Oct 15, 2024
442f6e3
Merge branch 'main' into 'enterprise'
aarthy-dk Oct 15, 2024
65fa528
Merge branch 'main' into sync
aarthy-dk Oct 17, 2024
86a04df
Merge branch 'sync' into 'enterprise'
Oct 18, 2024
26d019d
astor/TG-806
FernandezAstor Oct 21, 2024
7555216
feat(ui): bind grid selection to query parameters
luis-dk Oct 23, 2024
eb3f71a
fix(ui): send the selection ID to AGgrid
luis-dk Oct 28, 2024
8b7b0c8
Merge branch 'tg-818-grid-selection' into 'enterprise'
Oct 29, 2024
6b16b53
TG-777: Inline SQL functions and remove profiling setup
Oct 31, 2024
acb1085
Merge branch 'TG-777' into 'enterprise'
diogodk Oct 31, 2024
8451e74
fix(overview): truncate tests percentage
luis-dk Oct 31, 2024
716ff8a
feat(overview): add sort dropdown for table groups list
luis-dk Oct 31, 2024
fd076da
feat(overview): add input field to filter table groups list
luis-dk Oct 31, 2024
69d490b
fix(overview): sort by latest activity date instead of dual sorting
luis-dk Nov 1, 2024
36da100
Merge branch 'tg-822-overview' into 'enterprise'
Nov 1, 2024
493d4c3
feat(connections): use the new wizard for setting up table groups
luis-dk Oct 17, 2024
4d11dc1
feat(ui): add a database flavor selector component
luis-dk Oct 22, 2024
25ac0f5
fix(ui): Add streamlit-pydantic to list of dependencies
luis-dk Oct 24, 2024
9ee6bd2
fix: python 3.10 compatibility and missing import
luis-dk Oct 24, 2024
3b84e6a
fix(connections): use the flavor form when parsing validation errors
luis-dk Oct 28, 2024
39aa35c
misc(ui): add white background to custom link
luis-dk Oct 29, 2024
5e0ec9a
misc(ui): add db flavor icons for azure sql and synapse
luis-dk Oct 29, 2024
a6e20a7
fix(connections): display cached value for private key file uploader
luis-dk Oct 31, 2024
372c12b
fix(ui): use index instead of value in flavor selector
luis-dk Nov 1, 2024
1a167b8
fix: sleep to protect against multiple reruns
luis-dk Nov 5, 2024
d25bce8
misc: remove qc schema creation logic
luis-dk Nov 5, 2024
10c612e
fix(profiling): add parenthesis to profiling issue criteria
luis-dk Nov 6, 2024
9a071ff
Merge branch 'issues-count-fix' into 'enterprise'
Nov 6, 2024
e64d20a
misc(profiling): remove qc schema creation logic
luis-dk Nov 7, 2024
b463905
Merge branch 'setup-wizard' into 'enterprise'
Nov 7, 2024
66ca55d
feat(pdf): Hygiene Issues report
rboni-dk Oct 29, 2024
a67078b
refactor(components): upgrade vanJS to latest version
aarthy-dk Nov 4, 2024
41b25c1
refactor(tooltip): support tooltip positions - add to button and tool…
aarthy-dk Nov 4, 2024
eee533c
feat(link): support opening links in new tabs
aarthy-dk Nov 4, 2024
ab08ba2
misc(summary): add legend colors to summary bar component
aarthy-dk Nov 4, 2024
58ca5a7
feat(components): add BoxPlot, PercentBar and FrequencyBars components
aarthy-dk Nov 4, 2024
6f1f595
feat(components): add Attribute, Card and EditableCard components
aarthy-dk Nov 4, 2024
b8cb63a
feat(components): add Input, Checkbox and RadioGroup components
aarthy-dk Nov 4, 2024
d87f69c
feat(components): add Tree component
aarthy-dk Nov 4, 2024
fd343eb
feat(components): add ColumnProfile component
aarthy-dk Nov 4, 2024
b1bf3ea
feat(ui): add Data Hierarchy page
aarthy-dk Nov 4, 2024
5ce267f
misc(ui): fixes and typing improvements
aarthy-dk Nov 4, 2024
9fa43f1
misc(ui): improve data hierarchy query performance
aarthy-dk Nov 6, 2024
26e23b5
Merge branch 'aarthy/data-hierarchy' into 'enterprise'
Nov 7, 2024
fc6cb63
fix(threshold error count fix): code fixes
FernandezAstor Nov 7, 2024
1113bcd
fix: threshold error count
FernandezAstor Nov 7, 2024
0069270
ci(docker): add git to dockerfile
aarthy-dk Nov 7, 2024
8f66c46
Merge branch 'fix-build' into 'enterprise'
Nov 7, 2024
c37ff38
Merge branch 'astor/TG-806' into 'enterprise'
FernandezAstor Nov 8, 2024
776d2b3
fix(profiling): add parenthesis to the formatted anomaly criteria
luis-dk Nov 7, 2024
0a97887
Merge branch 'profile-query-parentheses' into 'enterprise'
Nov 8, 2024
d919890
Merge branch 'enterprise' of gitlab.com:dkinternal/testgen/dataops-te…
rboni-dk Nov 11, 2024
9521759
misc(pdf): Code review feedback
rboni-dk Nov 11, 2024
4e063da
Merge branch 'tg-817' into 'enterprise'
rboni-dk Nov 11, 2024
f5ba879
feat(cli): add scoring infrastructure and default score roll-ups
cbloche Nov 8, 2024
c54266b
Tweaked Functional Datatypes
cbloche Nov 11, 2024
b623679
Tweaked Incremental Upgrade
cbloche Nov 11, 2024
bde97ca
Incremental upgrade fix
cbloche Nov 11, 2024
6603837
Merge branch 'chip/scoring_phase_1B' into 'enterprise'
Nov 11, 2024
b9b1743
fix(ui): Adding database icons to the python built package
rboni-dk Nov 12, 2024
6cbad51
Merge branch 'fix-ui-missing-assets' into 'enterprise'
rboni-dk Nov 12, 2024
bd32353
fix(ui): add table and column filters to hygiene issues and test results
aarthy-dk Nov 8, 2024
6116f3b
fix(ui): add dk favicon to image
aarthy-dk Nov 8, 2024
b8a94b3
feat(ui): add community and training links to header
aarthy-dk Nov 11, 2024
7b51a95
misc(ui): improve query performance with new fields
aarthy-dk Nov 12, 2024
49bda82
Merge branch 'aarthy/filters' into 'enterprise'
Nov 12, 2024
8a5593b
ci(tests): fix functional tests
aarthy-dk Nov 12, 2024
3a8d9e3
Merge branch 'fix-tests' into 'enterprise'
Nov 12, 2024
5be063c
fix(ui): error on login screen
aarthy-dk Nov 12, 2024
8390a16
feat(pdf): add link backs to pdf issue reports
aarthy-dk Nov 13, 2024
9c2848e
fix(runs): increase sleep before closing run dialogs
aarthy-dk Nov 13, 2024
7d78958
fix(ui): deep-linking on test results page
aarthy-dk Nov 13, 2024
6593ddf
fix(ui): update bind query js syntax
aarthy-dk Nov 13, 2024
e05385e
Merge branch 'qa-fixes' into 'enterprise'
Nov 13, 2024
726bd9a
fix(profiling): TG-774 - disable profiling dialog for multi-table res…
FernandezAstor Nov 14, 2024
fca069b
fix(functions): Strip out comments when loading templated functions
rboni-dk Nov 14, 2024
ff504be
Merge branch 'astor/TG-774' into 'enterprise'
Nov 14, 2024
973bb4a
fix(ui): bug fixes on test results and connection wizard
aarthy-dk Nov 13, 2024
a4729ef
fix(pdf): make file names unique when zipping multiple
aarthy-dk Nov 15, 2024
c8f1cd2
refactor: cleanup references to qc schema
aarthy-dk Nov 15, 2024
b31bed3
Merge branch 'qa-fixes' into 'enterprise'
Nov 15, 2024
da6e23a
fix(ui): bug in format duration utility
aarthy-dk Nov 18, 2024
c690125
fix(data-hierarchy): add selected query param to test issue link
aarthy-dk Nov 19, 2024
524218e
fix(sql): bug in source data dialog
aarthy-dk Nov 20, 2024
926b8a1
fix(ui): deep links intermittently don't work
aarthy-dk Nov 20, 2024
666b447
misc: change level of repetitive docker check log
aarthy-dk Nov 20, 2024
b2d5bda
Merge branch 'qa-fixes' into 'enterprise'
Nov 20, 2024
c55e04b
fix(sql): escape special characters in pattern match test
aarthy-dk Nov 21, 2024
bd64e67
Merge branch 'enterprise' of gitlab.com:dkinternal/testgen/dataops-te…
rboni-dk Nov 22, 2024
7b81575
Merge branch 'aarthy/pattern-match' into 'enterprise'
Nov 22, 2024
078e08c
Merge branch 'tg-830' into 'enterprise'
rboni-dk Nov 22, 2024
fbaa6be
fix(overview): error when profiling date exists but no test date
aarthy-dk Nov 22, 2024
637dce8
Merge branch 'aarthy/overview-bug' into 'enterprise'
Nov 23, 2024
374a5d5
fix(sql): special characters in pattern match test
aarthy-dk Nov 23, 2024
62386b3
release: 2.15.3 -> 2.24.7
aarthy-dk Nov 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
165 changes: 165 additions & 0 deletions testgen/ui/pdf/hygiene_issue_report.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
import pandas
from reportlab.lib import colors
from reportlab.lib.colors import HexColor
from reportlab.lib.enums import TA_CENTER
from reportlab.lib.styles import ParagraphStyle
from reportlab.platypus import CondPageBreak, KeepTogether, Paragraph, Table, TableStyle

from testgen.ui.pdf.dataframe_table import DataFrameTableBuilder
from testgen.ui.pdf.style import (
COLOR_GRAY_BG,
COLOR_GREEN_BG,
PARA_STYLE_CELL,
PARA_STYLE_FOOTNOTE,
PARA_STYLE_H1,
PARA_STYLE_INFO,
PARA_STYLE_MONO,
PARA_STYLE_TEXT,
PARA_STYLE_TITLE,
TABLE_STYLE_DEFAULT,
)
from testgen.ui.pdf.templates import DatakitchenTemplate
from testgen.ui.services.hygiene_issues_service import get_source_data

SECTION_MIN_AVAILABLE_HEIGHT = 120

CLASS_COLORS = {
"Definite": HexColor(0xE94D4A),
"Likely": HexColor(0xFC8F2A),
"Possible": HexColor(0xFCD349),
"Potential PII": HexColor(0xFC8F2A),
}

def build_summary_table(document, hi_data):

summary_table_style = TableStyle(
(
# All-table styles
("GRID", (0, 0), (-1, -1), 2, colors.white),
("BACKGROUND", (0, 0), (-1, -1), COLOR_GRAY_BG),

# Empty cells
("BACKGROUND", (2, 5), (-1, -1), colors.white),

# Header cells
*[
(cmd[0], *coords, *cmd[1:])
for coords in (
((2, 2), (2, 4)),
((0, 0), (0, -1))
)
for cmd in (
("FONT", "Helvetica-Bold"),
("ALIGN", "RIGHT"),
("BACKGROUND", COLOR_GREEN_BG),
)
],

# Layout
("SPAN", (1, 0), (3, 0)),

("SPAN", (1, 1), (4, 1)),

("SPAN", (3, 2), (4, 2)),
("SPAN", (3, 3), (4, 3)),
("SPAN", (3, 4), (4, 4)),
("SPAN", (3, 5), (4, 5)),


# Status cell
*[
(cmd[0], (4, 0), (4, 0), *cmd[1:])
for cmd in (
("BACKGROUND", CLASS_COLORS.get(hi_data["issue_likelihood"], COLOR_GRAY_BG)),
("ALIGNMENT", "CENTER"),
("VALIGN", "MIDDLE"),
)
],
),
parent=TABLE_STYLE_DEFAULT,
)


profiling_timestamp = pandas.to_datetime(hi_data["profiling_starttime"]).strftime("%Y-%m-%d %H:%M:%S")
summary_table_data = [
(
"Hygiene Issue",
(
Paragraph(f"<b>{hi_data['anomaly_name']}:</b>", style=PARA_STYLE_CELL),
Paragraph(hi_data["anomaly_description"], style=PARA_STYLE_CELL),
),
None,
None,
Paragraph(
hi_data["issue_likelihood"],
style=ParagraphStyle("likelihood", textColor=colors.white, fontSize=10, parent=PARA_STYLE_CELL, alignment=TA_CENTER),
),
),
(
"Detail",
Paragraph(
hi_data["detail"],
style=ParagraphStyle("detail", fontName="Helvetica-Bold", parent=PARA_STYLE_CELL),
),
),

("Database/Schema", hi_data["schema_name"], "Profiling Date", profiling_timestamp),
("Table", hi_data["table_name"], "Table Group", hi_data["table_groups_name"]),
("Column", hi_data["column_name"], "Disposition", hi_data["disposition"] or "No Decision"),
("Column Type", hi_data["column_type"]),
]

summary_table_col_widths = [n * document.width for n in (.15, .35, .15, .15, .20)]
return Table(summary_table_data, style=summary_table_style, hAlign="LEFT", colWidths=summary_table_col_widths)


def build_sample_data_content(document, sample_data_tuple):
sample_data_status, sample_data_msg, lookup_query, sample_data = sample_data_tuple
if sample_data_status in ("ND", "NA"):
yield Paragraph(sample_data_msg, style=PARA_STYLE_INFO)
elif sample_data_status == "ERR" or sample_data is None:
yield Paragraph("It was not possible to fetch the sample data this time.", style=PARA_STYLE_INFO)
else:
sample_data.columns = [col.replace("_", " ").title() for col in sample_data.columns]
df_table_builder = DataFrameTableBuilder(sample_data, document.width)
table_flowables = [df_table_builder.build_table(hAlign="LEFT")]
if df_table_builder.omitted_columns:
omitted_columns = ", ".join(df_table_builder.omitted_columns)
sample_data_msg = f"Note: The following columns were omitted from this table: {omitted_columns}"
if sample_data_msg:
table_flowables.append(Paragraph(sample_data_msg, style=PARA_STYLE_FOOTNOTE))

yield from df_table_builder.split_in_columns(table_flowables)


def build_sql_query_conntent(sample_data_tuple):
lookup_query = sample_data_tuple[2]
if lookup_query:
return Paragraph(lookup_query, PARA_STYLE_MONO)
else:
return Paragraph("No sample data lookup query registered for this issue.")


def get_report_content(document, hi_data):
yield Paragraph("TestGen Issue Report", PARA_STYLE_TITLE)
yield build_summary_table(document, hi_data)

yield CondPageBreak(SECTION_MIN_AVAILABLE_HEIGHT)
yield Paragraph("Suggested Action", style=PARA_STYLE_H1)
yield Paragraph(hi_data["suggested_action"], style=PARA_STYLE_TEXT)

sample_data_tuple = get_source_data(hi_data)

yield CondPageBreak(SECTION_MIN_AVAILABLE_HEIGHT)
yield Paragraph("Sample Data", PARA_STYLE_H1)
yield from build_sample_data_content(document, sample_data_tuple)

yield KeepTogether([
Paragraph("SQL Query", PARA_STYLE_H1),
build_sql_query_conntent(sample_data_tuple)
])


def create_report(filename, hi_data):
doc = DatakitchenTemplate(filename)
doc.build(flowables=list(get_report_content(doc, hi_data)))
87 changes: 87 additions & 0 deletions testgen/ui/services/hygiene_issues_service.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
import streamlit as st

from testgen.ui.services import database_service as db


def get_source_data(hi_data):
str_schema = st.session_state["dbschema"]
# Define the query
str_sql = f"""
SELECT t.lookup_query, tg.table_group_schema, c.project_qc_schema,
c.sql_flavor, c.project_host, c.project_port, c.project_db, c.project_user, c.project_pw_encrypted,
c.url, c.connect_by_url, c.connect_by_key, c.private_key, c.private_key_passphrase
FROM {str_schema}.target_data_lookups t
INNER JOIN {str_schema}.table_groups tg
ON ('{hi_data["table_groups_id"]}'::UUID = tg.id)
INNER JOIN {str_schema}.connections c
ON (tg.connection_id = c.connection_id)
AND (t.sql_flavor = c.sql_flavor)
WHERE t.error_type = 'Profile Anomaly'
AND t.test_id = '{hi_data["anomaly_id"]}'
AND t.lookup_query > '';
"""

def get_lookup_query(test_id, detail_exp, column_names):
if test_id in {"1019", "1020"}:
start_index = detail_exp.find("Columns: ")
if start_index == -1:
columns = [col.strip() for col in column_names.split(",")]
else:
start_index += len("Columns: ")
column_names_str = detail_exp[start_index:]
columns = [col.strip() for col in column_names_str.split(",")]
queries = [
f"SELECT '{column}' AS column_name, MAX({column}) AS max_date_available FROM {{TARGET_SCHEMA}}.{{TABLE_NAME}}"
for column in columns
]
sql_query = " UNION ALL ".join(queries) + " ORDER BY max_date_available DESC;"
else:
sql_query = ""
return sql_query

def replace_parms(str_query):
str_query = (
get_lookup_query(hi_data["anomaly_id"], hi_data["detail"], hi_data["column_name"])
if lst_query[0]["lookup_query"] == "created_in_ui"
else lst_query[0]["lookup_query"]
)
str_query = str_query.replace("{TARGET_SCHEMA}", lst_query[0]["table_group_schema"])
str_query = str_query.replace("{TABLE_NAME}", hi_data["table_name"])
str_query = str_query.replace("{COLUMN_NAME}", hi_data["column_name"])
str_query = str_query.replace("{DATA_QC_SCHEMA}", lst_query[0]["project_qc_schema"])
str_query = str_query.replace("{DETAIL_EXPRESSION}", hi_data["detail"])
str_query = str_query.replace("{PROFILE_RUN_DATE}", hi_data["profiling_starttime"])
if str_query is None or str_query == "":
raise ValueError("Lookup query is not defined for this Anomoly Type.")
return str_query

try:
# Retrieve SQL for customer lookup
lst_query = db.retrieve_data_list(str_sql)

# Retrieve and return data as df
if lst_query:
str_sql = replace_parms(str_sql)
df = db.retrieve_target_db_df(
lst_query[0]["sql_flavor"],
lst_query[0]["project_host"],
lst_query[0]["project_port"],
lst_query[0]["project_db"],
lst_query[0]["project_user"],
lst_query[0]["project_pw_encrypted"],
str_sql,
lst_query[0]["url"],
lst_query[0]["connect_by_url"],
lst_query[0]["connect_by_key"],
lst_query[0]["private_key"],
lst_query[0]["private_key_passphrase"],
)
if df.empty:
return "ND", "Data that violates Hygiene Issue criteria is not present in the current dataset.", str_sql, None
else:
return "OK", None, str_sql, df
else:
return "NA", "Source data lookup is not available for this Issue.", None, None

except Exception as e:
return "ERR", f"Source data lookup query caused an error:\n\n{e.args[0]}", None, None
12 changes: 5 additions & 7 deletions testgen/ui/views/dialogs/profiling_results_dialog.py
Original file line number Diff line number Diff line change
@@ -12,13 +12,11 @@
BUTTON_HELP = "Review profiling for highlighted column"


def view_profiling_button(button_container, str_table_name, str_column_name,
str_profile_run_id=None, str_table_groups_id=None):
with button_container:
if st.button(
BUTTON_TEXT, help=BUTTON_HELP, use_container_width=True
):
profiling_results_dialog(str_table_name, str_column_name, str_profile_run_id, str_table_groups_id)
def view_profiling_button(str_table_name, str_column_name, str_profile_run_id=None, str_table_groups_id=None):
if st.button(
BUTTON_TEXT, help=BUTTON_HELP, use_container_width=True
):
profiling_results_dialog(str_table_name, str_column_name, str_profile_run_id, str_table_groups_id)


@st.dialog(title="Profiling Results")
Loading