Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bugfix METreadnc does support reading netCDF data that only has level and latitude (rather than longitude and latitude and level) #247

Closed
5 of 23 tasks
bikegeek opened this issue Nov 29, 2023 · 1 comment
Closed
5 of 23 tasks

Bugfix METreadnc does support reading netCDF data that only has level and latitude (rather than longitude and latitude and level) #247

bikegeek opened this issue Nov 29, 2023 · 1 comment
Assignees
@bikegeek
Labels
component: METdataio priority: high High Priority requestor: METplus Team METplus Development Team required: FOR DEVELOPMENT RELEASE Required to be completed in the development release for the assigned project required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone type: bug Fix something that is not working
Milestone
METdataio-3.0.0

Comments

@bikegeek
Copy link
Collaborator

Describe the Problem

When reading in the following netCDF data (remove the .txt extension), which DOES NOT have a longitude, only latitude:
sa_test.nc.txt

The resulting pandas dataframe does not capture the level and latitude data into separate columns but instead creates an index comprised of the latitude and level:
xarray_to_dataframe

The Xarray method to_dataframe() does not correctly handle this input data.

Expected Behavior

When using the METdataio.METreadnc.util.read_netcdf package to reformat netCDF data with only latitude (and no corresponding longitude), no errors are produced but the resulting data frame does not have the expected data columns corresponding to the level and latitude.

On 'cheyenne', you will observe an AttributeError: 'DataFrame' object has no attribute 'append'. Did you mean: '_append'? while running the read_netcdf.py script. Note: the append method has been deprecated in later versions of pandas, so this should be replaced with concat.

Environment

Describe your runtime environment:
1. Machine: (e.g. HPC name, Linux Workstation, Mac Laptop) cheyenne
2. OS: (e.g. RedHat Linux, MacOS) GNU/ Linux
3. Software version number(s) Linux cheyenne5 4.12.14-95.51-default #1 SMP Fri Apr 17 08:14:12 UTC 2020 (c6bab98) x86_64 x86_64 x86_64 GNU/Linux (output from uname -a)

To Reproduce

Describe the steps to reproduce the behavior:
Clone the METcalcpy and METdataio source code
Set up a Python environment
*1. Go to 'METdataio/METreadnc/util
*2. Copy the netcdf file to a data directory and set the CALCPY_DATA env variable
*3. Set the PYTHONPATH to PATH-TO/METdataio/METreadnc:PATH-TO/METcalcpy:PATH-TO/METcalcpy/metcalcpy
*4. From the METdataio/METreadnc/util directory run the following:
python
Post relevant sample data following these instructions:
sa_test.nc.txt

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

  • Select engineer Minna Win
  • Select no scientist required

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Select Organization level Project for support of the current coordinated release
  • Select Repository level Project for development toward the next official release or add alert: NEED CYCLE ASSIGNMENT label
  • Select Milestone as the next bugfix version

Define Related Issue(s)

Consider the impact to the other METplus components.

Bugfix Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding Source.
  • Fork this repository or create a branch of main_<Version>.
    Branch name: bugfix_<Issue Number>_main_<Version>_<Description>
  • Fix the bug and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Add any new Python packages to the METplus Components Python Requirements table.
  • Push local changes to GitHub.
  • Submit a pull request to merge into main_<Version>.
    Pull request: bugfix <Issue Number> main_<Version> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s) and Development issues
    Select: Organization level software support Project for the current coordinated release
    Select: Milestone as the next bugfix version
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Complete the steps above to fix the bug on the develop branch.
    Branch name: bugfix_<Issue Number>_develop_<Description>
    Pull request: bugfix <Issue Number> develop <Description>
    Select: Reviewer(s) and Development issues
    Select: Repository level development cycle Project for the next official release
    Select: Milestone as the next official version
  • Close this issue.
@bikegeek bikegeek added alert: NEED ACCOUNT KEY Need to assign an account key to this issue priority: high High Priority type: bug Fix something that is not working requestor: METplus Team METplus Development Team required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone required: FOR DEVELOPMENT RELEASE Required to be completed in the development release for the assigned project component: METdataio labels Nov 29, 2023
@bikegeek bikegeek added this to the METdataio-3.0.0 milestone Nov 29, 2023
@bikegeek bikegeek self-assigned this Nov 29, 2023
@bikegeek bikegeek moved this from 🔖 Ready to 🏗 In progress in METplus-Analysis-6.0.0 Development Nov 29, 2023
@bikegeek
Copy link
Collaborator Author

bikegeek commented Dec 2, 2023

resetting the index of the resulting data frame seems to produce the expected data frame:

df = df.append(file_data.to_dataframe().reset_index(), sort=False)

bikegeek added a commit that referenced this issue Dec 2, 2023
@bikegeek
Issue #247 Reset the index of the dataframe that is converted from Xa…
d252615 
…rray.
bikegeek added a commit that referenced this issue Dec 2, 2023
@bikegeek
Issue #247 replace pandas append with pandas concat to address Future…
179b9bb 
…Warning
bikegeek added a commit that referenced this issue Dec 14, 2023
@bikegeek
Issue #247 files and tests for changes to readnc.py
449b105
bikegeek added a commit that referenced this issue Dec 14, 2023
@bikegeek
Issue #247 support for reading in a single file, list, or tuple of fi…
463813a 
…les. Requires user to specify yaml config file at command line
bikegeek added a commit that referenced this issue Dec 14, 2023
@bikegeek
Issue #247 Include pytests for readnc.py
8fac0d2
bikegeek added a commit that referenced this issue Dec 14, 2023
@bikegeek
Issue #247 Added xarray to requirements. Needed for METreadnc
acf58d9
bikegeek added a commit that referenced this issue Dec 18, 2023
@bikegeek
Issue #247 first draft of instructions for using read_netcdf in METre…
015d19f 
…adnc
bikegeek added a commit that referenced this issue Dec 18, 2023
@bikegeek
Issue #247 added read_nc to table of contents
101d72f
bikegeek added a commit that referenced this issue Dec 31, 2023
@bikegeek
Bugfix 247 readnc no lon (#249)
4214888 
* Issue #247 Reset the index of the dataframe that is converted from Xarray.

* Issue #247 replace pandas append with pandas concat to address FutureWarning

* Removed print statement

* Issue #247 files and tests for changes to readnc.py

* Issue #247 support for reading in a single file, list, or tuple of files.  Requires user to specify yaml config file at command line

* Issue #247 Include pytests for readnc.py

* Issue #247 Added xarray to requirements.  Needed for METreadnc

* Issue #247 first draft of instructions for using read_netcdf in METreadnc

* Issue #247 added read_nc to table of contents

* remove test_readnc.py to debug workflow

* debugging workflow

* updated content and fix RST syntax errors

* make captilization of Pandas consistent

* remove chdir to METreadnc for debugging

* revert to earlier version, add METreadnc to PYTHONPATH

* Add running of test_readnc.py

* forgot to chdir to the test directory in METreadnc

* modify import statement to address issues in Git workflow

* Update test_readnc.py

modify import

* Add step to retrieve METcalcpy

* suppress the numpy.ndarray warnings-these are not within our control

* skip tests that test the YAML config files.

* include netcdf4 under install dependencies

* Update unit_tests.yml

sync changes to test name change in METreformat
@bikegeek bikegeek moved this from 🏗 In progress to ✅ Done in METplus-Analysis-6.0.0 Development Jan 4, 2024
@bikegeek bikegeek closed this as completed Jan 4, 2024
@jprestop jprestop removed the alert: NEED ACCOUNT KEY Need to assign an account key to this issue label Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bikegeek bikegeek

Labels
component: METdataio priority: high High Priority requestor: METplus Team METplus Development Team required: FOR DEVELOPMENT RELEASE Required to be completed in the development release for the assigned project required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone type: bug Fix something that is not working
Projects
No open projects
METplus-Analysis-6.0.0 Development
Status: 🏁 Done
Milestone
METdataio-3.0.0
Development

No branches or pull requests
2 participants
@bikegeek @jprestop