Merge pull request #14 from avohq/schema_to_table_works_in_snowflake_…

…and_bigquery

polish schema_to_table a little bit and run it against snowflake

CHANGELOG.md

@@ -2,9 +2,12 @@
 Features and bug fixes that have been applied but not released yet.
 
 
+# avo-audit v1.0.0
 
+## Fixes and Polish
 
-# avo-audit v1.0.0
+- `Schema_to_table` works against snowflake
+- `Schema_to_table` only queries tables instead of all relations in a schema
 
 ## Features
 

README.md

@@ -55,14 +55,12 @@ models:
 This test will by default check the past 15 days from yesterday, and it will fail if any rows are returned which indicates an anomaly in event volume data for any event in your raw data.
 
 
-
 ## Helper macro
 
-
 ## join_schema_to_table
 This macro is a helper macro for those who do not have all raw events in a single table, but instead in multiple tables in one dataset.
-It is intended to extract the key columns needed and merge all the tables into 1.
-This is something we use to be able to use `audit_event_volume`
+It is intended to extract the key columns needed and merge all the tables into one.
+This is something we use to be able to use `test_avo_audit_detect_event_count_anomaly`
 ```
 // schema/bigquery dataset
 {{
@@ -75,7 +73,7 @@ This is something we use to be able to use `audit_event_volume`
 }}
 ```
 > This macro is a work in progress and is intended to help joining together tables of single events into one table to automatically include all events.
-test_avo_audit_detect_event_count_anomaly can also instead be applied as a test to all event tables that you want to monitor instead.
+
 # Coming soon
 ### Issue types on the roadmap (not prioritized list):
 * Event missing on some platforms

macros/schema_to_table.sql

@@ -12,12 +12,12 @@
     {%- set required_columns = [] -%}
 
     {% for column in column_names %}
-      {% if column.name == event_name_column or column.name == event_version_column or column.name == event_date_column or column.name == event_source_column %}
+      {% if column.name|lower == event_name_column or column.name|lower == event_version_column or column.name|lower == event_date_column or column.name|lower == event_source_column %}
         {% do required_columns.append(column.name) %}
       {% endif %}
     {% endfor %}
 
-    {% if required_columns|length == 4 %}
+    {% if required_columns|length == 4 and event_relation.type == "table" %}
       {% do relations.append(event_relation) %}
     {% endif %}
   {% endfor %}
@@ -29,20 +29,26 @@
 {% macro join_schema_into_table(raw_event_schema, event_name_column, event_version_column, event_date_column, event_source_column) %}
 
 {% set event_relations = dbt_utils.get_relations_by_pattern(raw_event_schema, '%') %}
-{%- set relations = filter_event_tables(event_relations, event_name_column, event_version_column, event_date_column, event_source_column) -%}
+
+{%- set relations = avo_audit.filter_event_tables(event_relations, event_name_column, event_version_column, event_date_column, event_source_column) -%}
+
 
 {% for event_relation in relations %}
 
   select
-    ROW_NUMBER() over () as ID,
     {{event_name_column}} as {{event_name_column}},
     {{event_version_column}} as {{event_version_column}},
     {{event_date_column}} as {{event_date_column}},
     {{event_source_column}} as {{event_source_column}}
   from 
     {{event_relation}}
+  GROUP BY
+  {{event_name_column}},
+  {{event_version_column}},
+  {{event_date_column}},
+  {{event_source_column}}
   {% if not loop.last %}
-    UNION ALL 
+    UNION ALL
   {% endif %}
 
 {% endfor %}