IO AP sensing (Follow up of request #219 and #218) #220

davechris · 2024-08-30T09:06:15Z

implemented generic .tra parser
added a test
added documentation to changelog
solved the linting errors in apsensing.py sensornet.py and sensortran.py

And I run the tests:

hatch run format --> successful
hatch run test --> does still not work, but the command < pytest ./tests/ > worked OK, I guess. I do not know what to expect, the result was: 80 passed, 3 skipped, 1 xfailed, 1 xpassed, 22 warnings in 580.50s (0:09:40)
hatch run docs:build --> build succeeded, 35 warnings -- I guess this is OK?

…0. C320 was an arbitrary name given for the wellbore by the user

… .xml and .tra exported, which gives more metadata, readings of the instruments temperature sensors, and instrument computed log-ratio and attenuation.

added a check wheather a .tra file with identical timestamp exists in .xml directory. If yes it is used to: - import t_by_dts, log_ratio, loss - pt100 data (if it exists in file)

… NOT tested yet

changelog updated

…ng.py, sensornet.py and sensortran.py

BSchilperoort · 2024-08-30T14:16:06Z

Good to see you've found some time to work on this again! I've invited you as a collaborator, so the CI should run without me needing to click the button.

It seems the linter complains now due to a change in its default config.

davechris · 2024-08-30T14:23:52Z

Yeah, nice to be back :) It took me some time, as I had a bunch of responsibilities last semester. It seems, that I get a lot of time for my research in the coming months.

Ah, that linter-config-change expains the complains with the printout commands.

BSchilperoort

I had a look, didn't run the code yet (but I assumes that part is fine).

The CI is complaining, and you probably don't see those errors because your hatch environments use less new versions of packages. If you do hatch env prune you remove them and hatch will create new environments that should show the same errors as on the CI

tests/data/ap_sensing_2/readme_data_explanation.txt

tests/test_datastore.py

CHANGELOG.rst

src/dtscalibration/io/apsensing.py

Co-authored-by: Bart Schilperoort <b.schilperoort@gmail.com>

- changed detection algorithm of .tra files - added a double-ended calibration test of ap_sensing_2 data to test_datastore.py

…s-calibration into io_apsensing

… tested.

- load_tra_arrays flag - current .tra implementation is limited to in-memory reading only

…ation

davechris · 2024-09-09T15:31:35Z

The CI is complaining, and you probably don't see those errors because your hatch environments use less new versions of packages. If you do hatch env prune you remove them and hatch will create new environments that should show the same errors as on the CI

hatch env prune worked. So it seems that I can finally execute all hatch commands!

I started with hatch run format:
...\dtscalibration\lib\site-packages\bokeh\resources.py:363: error: Pattern matching is only supported in Python 3.10 and greater [syntax]
Found 1 error in 1 file (errors prevented further checking)

Either we need to change that pattern matching or require Python 3.10 in pyproject.toml:

line 26 --> requires python 3.10 (from 3.9)
line 131 delete "3.9" from array
line 173 target version py310 statt py39

then it continues to search for errors, finds 10, fixes 8. The two remaining are identical:
UP038 in assert isinstance(v, (list, tuple)) in src\dtscalibration\dts_accessor_utils.py
Very easy to fix (https://docs.astral.sh/ruff/rules/non-pep604-isinstance/)

The question is: Do you want to require python 3.10 or rather fix the first errror? - Can we tell the linter to ignore site packages?

BSchilperoort · 2024-09-10T14:42:37Z

...\dtscalibration\lib\site-packages\bokeh\resources.py:363: error: Pattern matching is only supported in Python 3.10 and greater [syntax] Found 1 error in 1 file (errors prevented further checking)

Oh that's an annoying bug. We probably have to run the CI at Python 3.9 to make it green. Bokeh has dropped support for 3.9. We don't use it, but it's being installed due to xarray[parallel] being in the dependencies, which then installs dask[complete] which needs bokeh.

Let's just ignore this error for now, that's less work 😉 It won't break any code anyway.

Can we tell the linter to ignore site packages

Not sure why it's checking it, I don't think it should.

I think the only thing to fix still is the docs, the build seems to fail. That's more important that the small mypy error. Do you think you can fix the notebooks that are failing? You should be able to build the docs locally with hatch run docs:build.

BSchilperoort · 2024-09-10T14:45:55Z

You can try changing the settings:

[tool.mypy]
ignore_missing_imports = true  # Preferably false, but matplotlib, scipy and statsmodels are missing typing stubs
python_version = "3.9"
exclude="*/site-packages/*"

davechris · 2024-09-10T14:56:11Z

Alright, I'll try that tomorrow.

BSchilperoort

CI is now green. If you can have a look at the delayed loading (into dask arrays) #220 (comment) and see if that still works or not (and add that info to docs), you can merge this PR.

- added explanation for .xml and .tra files - added an example of importing .xml + .tra data

…ation

davechris · 2024-09-12T09:16:05Z

I added a remark to document that issue in the docstring and in the docs notebook A3.

And I do not have the rights to merge a PR ;)

David Lah and others added 10 commits February 1, 2024 17:25

Device ID bug for APSensing fixed. Device ID is N4386B instead of C32…

3a6b5b2

…0. C320 was an arbitrary name given for the wellbore by the user

more ap-sensing test data added. Advantage compared to existing data:…

11db29d

… .xml and .tra exported, which gives more metadata, readings of the instruments temperature sensors, and instrument computed log-ratio and attenuation.

Merge branch 'dtscalibration:main' into main

85a1a8c

apsensing io to read both .xml and .tra files

f162003

added a check wheather a .tra file with identical timestamp exists in .xml directory. If yes it is used to: - import t_by_dts, log_ratio, loss - pt100 data (if it exists in file)

apsensing tra file parse updated based on Barts generic parser idea -…

2a38d29

… NOT tested yet

cleaned up comments and docstrings in apsensing.py

97b3e75

added test to test_datastore.py inside fuction test_read_apsensing_files

2099385

changelog updated

AP Sensing Device name changed in README.rst

2fe653e

corrected the linting errors by hatch run format in the files apsensi…

0ab33a6

…ng.py, sensornet.py and sensortran.py

Merge branch 'dtscalibration:main' into main

a98b31a

davechris requested a review from BSchilperoort September 4, 2024 11:22

BSchilperoort reviewed Sep 4, 2024

View reviewed changes

davechris and others added 12 commits September 9, 2024 13:24

Update CHANGELOG.rst

35148f9

Co-authored-by: Bart Schilperoort <b.schilperoort@gmail.com>

Update CHANGELOG.rst

b4ed021

Co-authored-by: Bart Schilperoort <b.schilperoort@gmail.com>

Update CHANGELOG.rst

79de203

Co-authored-by: Bart Schilperoort <b.schilperoort@gmail.com>

Update src/dtscalibration/io/apsensing.py printout language fix

d18f585

Co-authored-by: Bart Schilperoort <b.schilperoort@gmail.com>

- renamed ap sensing data explanation and changed it to markdown format

7840036

- changed detection algorithm of .tra files - added a double-ended calibration test of ap_sensing_2 data to test_datastore.py

Merge branch 'dtscalibration:main' into io_apsensing

9e746fc

Merge branch 'io_apsensing' of https://github.com/davechris/python-dt…

77c36b2

…s-calibration into io_apsensing

Bart's newly written append_to_data_vars_structure function added and…

0bb2fc4

… tested.

added flag to load .tra array data only when specified.

7ab23c0

automatic changes by hatch run format

20c9aff

added to docstring of io/apsensing.py

6213332

- load_tra_arrays flag - current .tra implementation is limited to in-memory reading only

Merge branch 'main' of https://github.com/davechris/python-dts-calibr…

ea32725

…ation

BSchilperoort added 3 commits September 11, 2024 09:06

Move CI linting to Python 3.9

7e21fd4

Ignore weird mypy error

c5e01f0

Fix broken notebook

f597541

BSchilperoort approved these changes Sep 11, 2024

View reviewed changes

David Lah added 3 commits September 12, 2024 10:33

updated docs/notebooks/A3Load_ap_sensing_files.ipynb

114e422

- added explanation for .xml and .tra files - added an example of importing .xml + .tra data

Merge branch 'io_apsensing'

7433406

Merge branch 'main' of https://github.com/davechris/python-dts-calibr…

ff61e9a

…ation

BSchilperoort merged commit 4a3aa8a into dtscalibration:main Sep 12, 2024
13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IO AP sensing (Follow up of request #219 and #218) #220

IO AP sensing (Follow up of request #219 and #218) #220

davechris commented Aug 30, 2024 •

edited

Loading

BSchilperoort commented Aug 30, 2024 •

edited

Loading

davechris commented Aug 30, 2024

BSchilperoort left a comment

davechris commented Sep 9, 2024 •

edited

Loading

BSchilperoort commented Sep 10, 2024 •

edited

Loading

BSchilperoort commented Sep 10, 2024

davechris commented Sep 10, 2024

BSchilperoort left a comment

davechris commented Sep 12, 2024

IO AP sensing (Follow up of request #219 and #218) #220

IO AP sensing (Follow up of request #219 and #218) #220

Conversation

davechris commented Aug 30, 2024 • edited Loading

BSchilperoort commented Aug 30, 2024 • edited Loading

davechris commented Aug 30, 2024

BSchilperoort left a comment

Choose a reason for hiding this comment

davechris commented Sep 9, 2024 • edited Loading

BSchilperoort commented Sep 10, 2024 • edited Loading

BSchilperoort commented Sep 10, 2024

davechris commented Sep 10, 2024

BSchilperoort left a comment

Choose a reason for hiding this comment

davechris commented Sep 12, 2024

davechris commented Aug 30, 2024 •

edited

Loading

BSchilperoort commented Aug 30, 2024 •

edited

Loading

davechris commented Sep 9, 2024 •

edited

Loading

BSchilperoort commented Sep 10, 2024 •

edited

Loading