Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend mon_data_space.py to handle legacy DA monitor conventional data #167

Closed
EdwardSafford-NOAA opened this issue Dec 4, 2023 · 5 comments
Assignees

Comments

@EdwardSafford-NOAA
Copy link
Contributor

Description

Extend mon_data_space.py to load legacy DA monitor conventional data. The conventional data is (of course) slightly different from all others. It has 4 dimensions: data type (i.e. q120, 130, etc), region, vertical level, and time. The use of data type is unique among the DA monitor legacy data sources so support for that is needed.

Requirements/Acceptance Criteria:

Load legacy conventional DA monitor data and output time series plots.

Dependencies

None

@EdwardSafford-NOAA EdwardSafford-NOAA self-assigned this Dec 4, 2023
@EdwardSafford-NOAA
Copy link
Contributor Author

EdwardSafford-NOAA commented Dec 15, 2023

Naturally the conventional binary data time/vert files are a bit different from all the other legacy data. They have 4 dimensions: data type, region, level, and time. A given file has data for all available data types in that category -- (e.g. ges_q_stas.[yyyymmddhh] has all the q data (q120_0, q130_0, etc). That's the DataType dimension and I need to support data selection by DataType so a batch plot can step through data types in the same manner as channels.

I'm not quite there yet but I do have selection by data type working in single (non-batch) plots. Below is the time series plot for q120_0, region 1 (global), level all (0-2000mb). This matches the legacy plot, but I think maybe making the y axis a log scale would make sense given the wide spread.

q120 time

@EdwardSafford-NOAA
Copy link
Contributor Author

EdwardSafford-NOAA commented Dec 19, 2023

I've hit something of a snag. I'm trying to convert this plot into a batch plot, selecting data by data type (q120_0, q130_0, etc) in the same manner as channels or levels are selected during batch plotting. But I'm running into problems mapping the datatype string values to DataType number ('q120_0' to 1, 'q130_0' to 2, etc). I've got to dig back into channel and re-learn how non-continuous channels work. I'm beginning to think I messed up the dataset design here.

Update -- yup, that's the problem. mon_data_space.py explicitly loads the actual channel numbers as the values for the Channel dimension in the dataset. I need to do that for DataType dimension too and hope that using strings instead of ints won't cause problems. Learning, slowly.

EdwardSafford-NOAA added a commit to EdwardSafford-NOAA/eva that referenced this issue Dec 21, 2023
Add support for batch processing on DataType/dtype.
@EdwardSafford-NOAA
Copy link
Contributor Author

Batch processing by DataType is now working. This is the product of the prototype ConMon time series plot for q sources:

q120_0 time

EdwardSafford-NOAA added a commit to EdwardSafford-NOAA/eva that referenced this issue Dec 22, 2023
Change use of 'dtype' to 'datatype' to differentiate from np.dtype.
EdwardSafford-NOAA added a commit to EdwardSafford-NOAA/eva that referenced this issue Dec 22, 2023
Change use of 'dtype' to 'datatype' to differentiate from np.dtype.
EdwardSafford-NOAA added a commit to EdwardSafford-NOAA/eva that referenced this issue Dec 28, 2023
Add support for batch processing on DataType/dtype.
EdwardSafford-NOAA added a commit to EdwardSafford-NOAA/eva that referenced this issue Dec 28, 2023
Fix datatype use with station data.
EdwardSafford-NOAA added a commit to EdwardSafford-NOAA/eva that referenced this issue Dec 28, 2023
Merge branch 'develop' into feature/con-167
@EdwardSafford-NOAA
Copy link
Contributor Author

EdwardSafford-NOAA commented Dec 29, 2023

Question for @kevindougherty-noaa . I'm working on the ConMon vertical plots, example from eva is below. In the lower left plot the data values are all 0. If I don't explicitly set the xticks the xaxis puts 0 in the middle of the plot with negative and positive values to left and right of 0. I don't really want to show any negative xtick values, since that's not a possibility for counts. Is there a way in emcpy of forcing the xtick values to all be positive values without setting explicit xtick values?

q120_0 time(2)

EdwardSafford-NOAA added a commit to EdwardSafford-NOAA/eva that referenced this issue Dec 29, 2023
Add update for conmon vert plots.
EdwardSafford-NOAA added a commit to EdwardSafford-NOAA/eva that referenced this issue Jan 2, 2024
CoryMartin-NOAA pushed a commit that referenced this issue Jan 2, 2024
* Ref #167

Add support for batch processing on DataType/dtype.

* Ref #167

Fix datatype use with station data.

* Ref #167

Add update for conmon vert plots.

* Ref #167

Fix typo.
@EdwardSafford-NOAA
Copy link
Contributor Author

Closed via PR #169

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant