/rad-analyst/ doesn't filter by analyst title or email

[lbeaufort]

/rad-analyst/ doesn't filter by analyst title or email

Example: https://api.open.fec.gov/v1/rad-analyst/?sort_nulls_last=false&page=1&api_key=DEMO_KEY&sort_hide_null=false&sort_null_only=false&per_page=20&title=superstar

This is because args.py doesn't match the filters in the resource. It's probably most user friendly to update the resource file to match args.

• Add automated tests
• Fix filter

[jason-upchurch]

From args.py:

rad_analyst = {
    'committee_id': fields.List(IStr, description=docs.COMMITTEE_ID),
    'analyst_id': fields.List(fields.Int(), description='ID of RAD analyst'),
    'analyst_short_id': fields.List(fields.Int(), description='Short ID of RAD analyst'),
    'telephone_ext': fields.List(fields.Int(), description='Telephone extension of RAD analyst'),
    'name': fields.List(fields.Str, description='Name of RAD analyst'),
    'email': fields.List(fields.Str, description='Email of RAD analyst'),
    'title': fields.List(fields.Str, description='Title of RAD analyst'),
    'min_assignment_update_date': fields.Date(description='Filter results for assignment updates made after this date'),
    'max_assignment_update_date': fields.Date(description='Filter results for assignment updates made before this date')
}

From rad_analyst.py (resource):

class RadAnalystView(ApiResource):

    model = models.RadAnalyst
    schema = schemas.RadAnalystSchema
    page_schema = schemas.RadAnalystPageSchema

    filter_fulltext_fields = [
        ('name', model.name_txt),
        ('analyst_email', model.analyst_email),
        ('analyst_title', model.analyst_title),
    ]
...

[jason-upchurch]

Initial api (non-local) test with swagger ui:

1. executed query with no analyst_email specified--returned count = 11,523
2. executed query with a particular analyst_email--returned count = 11,523

[edited to specify non-local test]

[jason-upchurch]

initial test of analyst_title v title:

• queried dev replica for count of title "Sr. Campaign Finance & Reviewing Analyst"-->4,188
• changed args.py to filter on analyst_title and used local env and swagger ui searching for "Sr. Campaign Finance & Reviewing Analyst" and returned 4,188 results.

[edited format]

[jason-upchurch]

changed resource to match args.py so that analyst_title is now title and updated model to have attribute model.title

[jason-upchurch]

[reopened ticket that was incorrectly closed]

[jason-upchurch]

potential similar issue related to @ in email address Casecommons/pg_search#224

[jason-upchurch]

in filters.py:

def filter_fulltext(query, kwargs, fields):
    from sqlalchemy.dialects import postgresql
    print('\nquery in\n')
    print(str(query.statement.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True})))
    print('\n')
    for key, column in fields:
        if kwargs.get(key):
            exclude_list = build_exclude_list(kwargs.get(key))
            include_list = build_include_list(kwargs.get(key))
            if exclude_list:
                filters = [
                    sa.not_(column.match(utils.parse_fulltext(value)))
                    for value in exclude_list
                ]
                query = query.filter(sa.and_(*filters))
            if include_list:
                filters = [
                    column.match(utils.parse_fulltext(value))
                    for value in include_list
                ]
                query = query.filter(sa.or_(*filters))
                print('\nquery out\n')
                print(str(query.statement.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True})))
                print('\n')
    return query

The filter_fulltext argument query is modified by the if include_list condition for email and generates the WHERE clause as:

WHERE public.ofec_rad_analyst_vw.analyst_email @@ to_tsquery(<analyst usr>:* & fec:* & gov:*')

From psql I verified that this query returns no results, but if I input a valid email address as a full string into the WHERE clause above and run again, the expected results are returned.

Next step: look at the sa.or_(*filters) logic

[jason-upchurch]

Moved email filter field to filter_multi_fields. Now requires users to correctly enter full email address of RAD Analyst (can enter multiple email addresses)

[jason-upchurch]

Tested various combinations of title in swagger ui (e.g., ".", "s", "sr.", "senior") to verify returns by title.