Fixes v2 dev website dashboard not loading #1256

joshuayhwu · 2022-06-23T01:06:26Z

Resolve typeName key error by reloading older data when filtered table is empty

Up to date with dev branch
Branch name follows guidelines
All PR Status checks are successful
Peer reviewed and approved

Any questions? See the getting started guide

nichhk

Thanks for the debugging!

If there's really no data for the given time range, we should display that one way or another instead of loading data from a different time range. Is that possible?

joshuayhwu · 2022-06-23T02:26:38Z

Thanks for the debugging!

If there's really no data for the given time range, we should display that one way or another instead of loading data from a different time range. Is that possible?

It maybe possible to put the problematic code into a callback function so that whenever there is no data, we use the PreventUpdate() method (which should be available in older versions of dash as well...?) to stop the graph content from rendering.

From my understanding so far, the code breaks when there is no data because the pandas groupby method generates an empty dataframe, with columns that plotly fails to index. I will need to take a look at this again tomorrow.

nichhk · 2022-06-23T20:16:49Z

Gotcha. If it turns out that we can only fix this issue in the newer version of Dash, we can lump those together.

joshuayhwu · 2022-07-03T18:56:51Z

@nichhk this will probably the only one we need to merge now. Will make a new PR for combine dashboards.

nichhk

Some partial comments for now.

server/dash/dashboards/neighborhood_recent.py

nichhk

Thanks for the updates!

Can you also add screenshots showing how the graphs look on this PR?

server/dash/dashboards/neighborhood_recent.py

nichhk · 2022-07-08T20:57:32Z

server/dash/dashboards/neighborhood_recent.py


 # Populate the neighborhood dropdown
+fig = px.line()


Prefer to have more descriptive variable names, e.g., requests_over_time_line_fig.

server/dash/dashboards/neighborhood_recent.py

nichhk

Thanks! Just a few more readability/style comments.

In the future, I think we would greatly benefit from having function docstrings so that we know exactly what the arguments and return types are. But we can work on that in a different PR.

server/dash/dashboards/neighborhood_recent.py

nichhk

LGTM, thanks for working on this! Please add screenshots of the dashboard when there is no data.

joshuayhwu · 2022-07-12T04:46:58Z

LGTM, thanks for working on this! Please add screenshots of the dashboard when there is no data.

Still trying to work on that. Getting feedback is a slow process as dash/gunicorn is a little buggy - sometimes showing no data on the dashboard. Hot reloading is not that useful on this front as dashboards tend to be cached and takes some "waiting time" before it refreshes. Regardless, I will work on the function documentation before updating that.

nichhk · 2022-07-12T04:51:54Z

Ah gotcha, that sounds finicky. Eventually, we would want unit tests to test this sort of scenario for us. But for now, would it help to simply replace the call to the API here with something that just creates an empty dataframe, or something like that?

joshuayhwu · 2022-07-12T05:09:19Z

There are too many dependencies in the code that relies on the table columns, so what I did was basically hav date range in the future such that there is no data. The problem is a "successful reload" (i.e. refresh with code changes) takes too long becuaes of either 1) buggy gunicorn/dash interaction causing no data (happens when none of the calllback function gets fired), or 2) dashboards are cached so it's still using old code. Multiple manual refresh is necessary to get 1 "successful reload".

joshuayhwu · 2022-07-12T05:46:10Z

There seems to be some error popping up that I missed before. First line is "selected_council'" 's datatype, and second line is "start_date" 's datatype. Do you have any suggestions?

<class 'str'>
<class 'datetime.date'>
Exception on /_dash-update-component [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.7/site-packages/dash/dash.py", line 1078, in dispatch
    response.set_data(func(*args, outputs_list=outputs_list))
  File "/usr/local/lib/python3.7/site-packages/dash/dash.py", line 1009, in add_context
    output_value = func(*args, **kwargs)  # %% callback invoked %%
  File "/app/dashboards/neighborhood_recent.py", line 203, in update_text
    close_count = df.query(f"councilName == '{selected_council}' and closedDate >= '{start_date}'")['srnumber'].count()  # noqa
  File "/usr/local/lib/python3.7/site-packages/pandas/core/frame.py", line 3345, in query
    res = self.eval(expr, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pandas/core/frame.py", line 3475, in eval
    return _eval(expr, inplace=inplace, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pandas/core/computation/eval.py", line 346, in eval
    ret = eng_inst.evaluate()
  File "/usr/local/lib/python3.7/site-packages/pandas/core/computation/engines.py", line 127, in evaluate
    return self.expr()
  File "/usr/local/lib/python3.7/site-packages/pandas/core/computation/expr.py", line 794, in __call__
    return self.terms(self.env)
  File "/usr/local/lib/python3.7/site-packages/pandas/core/computation/ops.py", line 397, in __call__
    right = self.rhs(env)
  File "/usr/local/lib/python3.7/site-packages/pandas/core/computation/ops.py", line 399, in __call__
    return self.func(left, right)
  File "/usr/local/lib/python3.7/site-packages/pandas/core/ops/common.py", line 65, in new_method
    return method(self, other)
  File "/usr/local/lib/python3.7/site-packages/pandas/core/ops/__init__.py", line 370, in wrapper
    res_values = comparison_op(lvalues, rvalues, op)
  File "/usr/local/lib/python3.7/site-packages/pandas/core/ops/array_ops.py", line 241, in comparison_op
    return invalid_comparison(lvalues, rvalues, op)
  File "/usr/local/lib/python3.7/site-packages/pandas/core/ops/invalid.py", line 34, in invalid_comparison
    raise TypeError(f"Invalid comparison between dtype={left.dtype} and {typ}")
TypeError: Invalid comparison between dtype=float64 and str

I also added the docstrings for the functions in nc_recent. Will continue working on this tmr.

nichhk · 2022-07-13T00:43:34Z

Hmm I'm not sure. It looks like closedDate (within the df) is converted to a string, while start_date is a datetime object. The datetime object might be getting coerced to a float64. We can represent start_date as a string and try that.

joshuayhwu · 2022-07-13T16:17:56Z

Yeah the coercing seems to be the problem. The dataframe from the API gets one request with only created date and no close date (NaN) value, which the pd.to_datetime coerce as a float. The weird thing is that I have set the start date so far ahead of time and the API still return this request with a startdate close to today. I revert to your suggestion on using empty data frame.

I also update the last row assignment in the line chart dataframe using the following code

figure_df.loc[figure_df.shape[0]] = ...

If we simply use J, the same request type will be assigned on all date range (since the j value remains constant on the outer loop). But we want all request types to have value 0 in all date range.

For the pie chart, turns out using value 0 would just cause the chart to disappear altogether. I use 1 in this case.

Let me know what you think.

EDIT: on the other hand, we do need to figure out a way to deal with dataframe with createdDates only and no closedDates (incomplete requests). Any thoughts?

nichhk

Thanks Josh! Just a few comments about the docstring and then this should be good to go.

server/dash/dashboards/neighborhood_recent.py

joshuayhwu · 2022-07-14T04:08:34Z

Thanks for the review. Updated docstrings with args and returns.

nichhk

Thanks for your patience Josh! LGTM. Just a few comments. I approved the PR so that you can merge it after fixing the comments without waiting for another review from me.

server/dash/dashboards/neighborhood_recent.py

joshuayhwu added 2 commits June 22, 2022 17:55

Resolve typeName key error

ade1445

Resolve merge conflict

92963d2

joshuayhwu requested a review from nichhk June 23, 2022 01:06

joshuayhwu self-assigned this Jun 23, 2022

nichhk suggested changes Jun 23, 2022

View reviewed changes

joshuayhwu added 3 commits June 24, 2022 14:27

Resolve empty data edge case

07b5a84

Fill in dummy when no data available

1cc16ea

Set data to 0 when no data

158503b

nichhk suggested changes Jul 6, 2022

View reviewed changes

joshuayhwu added 3 commits July 7, 2022 18:11

Revert test date changes and add constnat

8981c17

Fix spacing issues

2b470da

Add flask request back

ddaca73

nichhk suggested changes Jul 8, 2022

View reviewed changes

joshuayhwu added 6 commits July 8, 2022 15:15

Add more descriptive names

3aadeaf

Callback return data

0ac2382

Fix indicies

324b679

Update checking condition

ef5fcc8

Update checking condiiton comment

0925e40

Pie chart value set to 0

76699ae

nichhk suggested changes Jul 11, 2022

View reviewed changes

Using enumerate

b72a404

nichhk approved these changes Jul 12, 2022

View reviewed changes

Added nc_recent func docstring

f72cb05

Resolve empty df issues

b35dd6a

nichhk suggested changes Jul 14, 2022

View reviewed changes

Add docstrings args returns

b72cf57

joshuayhwu requested a review from nichhk July 14, 2022 04:06

nichhk approved these changes Jul 14, 2022

View reviewed changes

Update docstrings

a1721a5

joshuayhwu merged commit a1b9e61 into dev Jul 15, 2022

joshuayhwu deleted the requestTypeColors branch July 15, 2022 00:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes v2 dev website dashboard not loading #1256

Fixes v2 dev website dashboard not loading #1256

joshuayhwu commented Jun 23, 2022 •

edited

Loading

nichhk left a comment

joshuayhwu commented Jun 23, 2022

nichhk commented Jun 23, 2022

joshuayhwu commented Jul 3, 2022

nichhk left a comment

nichhk left a comment

nichhk Jul 8, 2022

nichhk left a comment

nichhk left a comment

joshuayhwu commented Jul 12, 2022

nichhk commented Jul 12, 2022

joshuayhwu commented Jul 12, 2022

joshuayhwu commented Jul 12, 2022 •

edited

Loading

nichhk commented Jul 13, 2022

joshuayhwu commented Jul 13, 2022 •

edited

Loading

nichhk left a comment

joshuayhwu commented Jul 14, 2022

nichhk left a comment

Fixes v2 dev website dashboard not loading #1256

Fixes v2 dev website dashboard not loading #1256

Conversation

joshuayhwu commented Jun 23, 2022 • edited Loading

nichhk left a comment

Choose a reason for hiding this comment

joshuayhwu commented Jun 23, 2022

nichhk commented Jun 23, 2022

joshuayhwu commented Jul 3, 2022

nichhk left a comment

Choose a reason for hiding this comment

nichhk left a comment

Choose a reason for hiding this comment

nichhk Jul 8, 2022

Choose a reason for hiding this comment

nichhk left a comment

Choose a reason for hiding this comment

nichhk left a comment

Choose a reason for hiding this comment

joshuayhwu commented Jul 12, 2022

nichhk commented Jul 12, 2022

joshuayhwu commented Jul 12, 2022

joshuayhwu commented Jul 12, 2022 • edited Loading

nichhk commented Jul 13, 2022

joshuayhwu commented Jul 13, 2022 • edited Loading

nichhk left a comment

Choose a reason for hiding this comment

joshuayhwu commented Jul 14, 2022

nichhk left a comment

Choose a reason for hiding this comment

joshuayhwu commented Jun 23, 2022 •

edited

Loading

joshuayhwu commented Jul 12, 2022 •

edited

Loading

joshuayhwu commented Jul 13, 2022 •

edited

Loading