backend refactor #682

jmensch1 · 2020-06-09T18:30:18Z

This is pretty much a complete refactor of the backend, plus some new features. Here are the highlights:

A clearer file structure in the /server directory

At the top level we've got /api, which is the containerized python app. /jmeter contains performance tests, and /redis is the containerized redis service with configuration.

Within the /api folder, executables are separated from source code, and the source code itself is more modular and intuitive.

All environment settings in a single `.env` file.

All settings now live in one place -- the .env file -- which is also the only .gitignored file in the server. The docker-compose.yml file no longer contains any secrets, so it's in version control.

If we want to combine the frontend and backend environment variables, it will be easy to move them to the root of the project (although we'll also have to move the docker-compose.yml file there as well.)

Separation of route declarations from their implementation

# from /server/api/src/app.py 

routes = {
    '/': (
        ['GET'], R.index),

    '/status/api': (
        ['GET', 'HEAD'], R.status.api),

    '/status/sys': (
        ['GET', 'HEAD'], R.status.sys),

    '/status/db': (
        ['GET', 'HEAD'], R.status.db),

    '/servicerequest/<srnumber>': (
        ['GET'], R.request_detail),

    '/map/clusters': (
        ['POST'], R.map.clusters),

    '/map/heat': (
        ['POST'], R.map.heat),

    '/visualizations': (
        ['POST'], R.visualizations),

    '/comparison/frequency': (
        ['POST'], R.comparison.frequency),

    '/comparison/timetoclose': (
        ['POST'], R.comparison.timetoclose),

    '/comparison/counts': (
        ['POST'], R.comparison.counts),

    '/feedback': (
        ['POST'], R.feedback)}

Error handling (implementing #506) with Slack notifications

The server now returns 400 errors with helpful messages to the client if input is bad (e.g., required params are missing, or wrong datatypes are sent). Any 500 errors can be assumed to be a problem with the server.

If a slack webhook url is provided in the .env file, errors are sent to Slack with a stack trace and a description of the api call that generated the error.

A streamlined install process and a README

The onboarding process is much easier now, and there's a README that describes it. Basically it requires this (from the README):

install docker and docker compose on your machine
run chmod +x install.sh && ./install.sh from this directory
put a Socrata api token in your .env file
run docker-compose run api python bin/db_seed.py --years 2019,2020

Informative, colorful logging

An AWS deploy workflow

There's a new workflow for deploying to AWS whenever there's a push to dev.

Up to date with dev branch
Branch name follows guidelines
All PR Status checks are successful
Peer reviewed and approved

Any questions? See the getting started guide

adamkendis

This was a killer PR. The new structure is awesome - small files, pretty clear separation of concerns. The error handler and logging are great. Thanks for remembering to update the frontend to match updated api endpoints. Just had a few small questions but nothing blocking this from being merged in.

adamkendis · 2020-06-10T02:27:08Z

server/README.md

@@ -0,0 +1,102 @@
+## Getting Started


This is so good. I'll add front-end stuff to the README this week.

adamkendis · 2020-06-11T19:16:03Z

server/api/src/control/error_handler.py

+ERROR_CODES = Slack.ERROR_CODES
+
+
+class ErrorHandler(EH):


adamkendis · 2020-06-11T19:27:14Z

server/api/src/db/migrate/versions/v_1.py

@@ -0,0 +1,4 @@
+
+
+def migrate():


Is this (and v_2) for future functionality?

Yep, I stubbed out a migration concept, we'll see if it's actually useful. For now it doesn't do anything.

adamkendis · 2020-06-11T19:30:25Z

server/api/src/db/requests/add_years.py

        INSERT INTO requests SELECT * FROM stage
    """)
    log('\tInserted rows: {}'.format(inserted.rowcount))


-def add_years(years, rows_per_year=math.inf, batch_size=BATCH_SIZE):
+def add_years(years,
+              rows_per_year=-1,


What does -1 for rows_per_year do? No limit unless a positive int is passed for that arg?

yeah -1 just means get all the requests that Socrata has for the year

adamkendis · 2020-06-11T19:48:19Z

server/api/src/services/requests.py

+    # validate input
+    if not isinstance(fields, list) or len(fields) == 0:
+        raise Exception('fields must be provided')
+
+    if (
+        startDate is None or
+        endDate is None or
+        not isinstance(requestTypes, list) or
+        not isinstance(ncList, list)
+    ):
+        raise Exception('invalid filters')


Nice, is this new?

We might have been checking that before, but it wasn't set up to return a 400 error. It probably just resulted in a 500 error if you didn't include startDate/endDate.

jmensch1 added 2 commits June 9, 2020 11:02

backend refactor

d4a2d3a

updated location of requirements.txt file

8e57316

jmensch1 requested a review from adamkendis June 9, 2020 18:49

jmensch1 linked an issue Jun 9, 2020 that may be closed by this pull request

[FEAT] Handling client errors #506

Closed

1 task

workflow fix

885e3bc

adamkendis approved these changes Jun 11, 2020

View reviewed changes

adamkendis merged commit e73ad6e into dev Jun 11, 2020

adamkendis deleted the BACK-refactor branch June 11, 2020 20:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

backend refactor #682

backend refactor #682

jmensch1 commented Jun 9, 2020 •

edited

Loading

adamkendis left a comment

adamkendis Jun 10, 2020

adamkendis Jun 11, 2020

adamkendis Jun 11, 2020

jmensch1 Jun 11, 2020

adamkendis Jun 11, 2020

jmensch1 Jun 11, 2020

adamkendis Jun 11, 2020

jmensch1 Jun 11, 2020

		ERROR_CODES = Slack.ERROR_CODES


		class ErrorHandler(EH):

backend refactor #682

backend refactor #682

Conversation

jmensch1 commented Jun 9, 2020 • edited Loading

A clearer file structure in the /server directory

All environment settings in a single .env file.

Separation of route declarations from their implementation

Error handling (implementing #506) with Slack notifications

A streamlined install process and a README

Informative, colorful logging

An AWS deploy workflow

adamkendis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmensch1 commented Jun 9, 2020 •

edited

Loading

All environment settings in a single `.env` file.