Skip to content

cemsbv/pygef

Repository files navigation

PYGEF

Code style: black PyPi Version GitHub stars Coverage Status

Simple parser for *.gef files. These are ASCII based files used for soil properties measurements. Compatible with Python 3.9.

Recently added the parsing of xml boreholes file, the xml parsing is still in a preliminary phase, not all the files are supported. If you find a file that doesn't work with pygef, please make an issue about it or PR :)

Installation

Latest stable version:

$ pip install pygef

Cutting-edge version (might break):

$ pip install git+https://github.com/cemsbv/pygef.git

CPT files

>> > from pygef import read_cpt
>> >  # read gef and xml files
>> > cpt_data = read_cpt("./my-cpt.xml")
>> > cpt_data
CPTData: {'bro_id': 'CPT000000099543',
          'cone_diameter': 44,
          'cone_surface_area': 1500,
          'cone_surface_quotient': 0.67,
          'cone_to_friction_sleeve_distance': 100,
          'cone_to_friction_sleeve_surface_area': 22530,
          'cone_to_friction_sleeve_surface_quotient': 1.0,
          ...
              'zlm_pore_pressure_u3_after': None,
'zlm_pore_pressure_u3_before': None}
>> >  # access the underlying polars DataFrame under the `data` attribute
>> > cpt_data.data.head()
shape: (5, 9)
┌────────────┬───────┬───────────┬────────────┬─────┬────────────┬────────────┬────────────┬────────────┐
│ penetratiodepthelapsedTiconeResist ┆ ... ┆ inclinatioinclinatiolocalFrictfrictionRa │
│ nLength---meance       ┆     ┆ nNSnResultantiontio        │
│ ---f64------        ┆     ┆ ------------        │
│ f64        ┆       ┆ f64f64        ┆     ┆ i64i64f64f64        │
╞════════════╪═══════╪═══════════╪════════════╪═════╪════════════╪════════════╪════════════╪════════════╡
│ 0.00.0-9.99999
e-9.99999e5 ┆ ... ┆ -999999-999999-9.99999e5-9.99999e5 │
│            ┆       ┆ 5         ┆            ┆     ┆            ┆            ┆            ┆            │
├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤
│ 0.020.0211.02.708      ┆ ... ┆ 000.030.6        │
├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤
│ 0.040.03913.04.29       ┆ ... ┆ 000.0390.8        │
├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤
│ 0.060.05915.05.124      ┆ ... ┆ 000.0450.9        │
├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤
│ 0.080.07917.05.45       ┆ ... ┆ 000.0491.0        │
└────────────┴───────┴───────────┴────────────┴─────┴────────────┴────────────┴────────────┴────────────┘

Bore files

>> > from pygef import read_bore
>> >  # read gef and xml files
>> > bore_data = read_bore("./my-bore.xml")
>> > bore_data
BoreData: {'bore_hole_completed': True,
           'bore_rock_reached': False,
           'data': (13, 8),
           'delivered_location': Location(srs_name='urn:ogc:def:crs:EPSG::28992', x=158322.139, y=444864.706),
           'delivered_vertical_position_datum': 'nap',
           'delivered_vertical_position_offset': 10.773,
           'delivered_vertical_position_reference_point': 'maaiveld',
           'description_procedure': 'ISO14688d1v2019c2020',
           'final_bore_depth': 12.0,
           'final_sample_depth': 12.0,
           'research_report_date': datetime.date(2021, 10, 19)}
>> >  # access the underlying polars DataFrame under the `data` attribute
>> > bore_data.data.head()
shape: (5, 8)
┌────────────┬────────────┬────────────┬──────────┬────────────┬────────────┬────────────┬─────────┐
│ upper_bounlower_boungeotechniccolordispersed_organic_masand_mediasoil_di │
│ darydaryal_soil_na---inhomogenitter_conten_classst      │
│ ------mestrtynt_class------     │
│ f64f64---        ┆          ┆ ------strlist[f6 │
│            ┆            ┆ str        ┆          ┆ boolstr        ┆            ┆ 4]      │
╞════════════╪════════════╪════════════╪══════════╪════════════╪════════════╪════════════╪═════════╡
│ 0.01.0zwakGrindidonkergrfalsenietOrganimiddelgrof ┆ [0.2,   │
│            ┆            ┆ gZandijs      ┆            ┆ sch420
tot630u0.0,    │
│            ┆            ┆            ┆          ┆            ┆            ┆ m          ┆ ...     │
│            ┆            ┆            ┆          ┆            ┆            ┆            ┆ 0.0]    │
├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤
│ 1.01.1zwakGrindidonkergrfalsenietOrganimiddelgrof ┆ [0.2,   │
│            ┆            ┆ gZandijs      ┆            ┆ sch420
tot630u0.0,    │
│            ┆            ┆            ┆          ┆            ┆            ┆ m          ┆ ...     │
│            ┆            ┆            ┆          ┆            ┆            ┆            ┆ 0.0]    │
├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤
│ 1.12.0zwakZandigstandaarfalsenietOrganinull       ┆ [0.0,   │
│            ┆            ┆ eKleiMetGrdBruin   ┆            ┆ sch        ┆            ┆ 0.1,    │
│            ┆            ┆ ind        ┆          ┆            ┆            ┆            ┆ ...     │
│            ┆            ┆            ┆          ┆            ┆            ┆            ┆ 0.0]    │
├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤
│ 2.03.0zwakZandigstandaarfalsenietOrganinull       ┆ [0.0,   │
│            ┆            ┆ eKleidGrijs   ┆            ┆ sch        ┆            ┆ 0.0,    │
│            ┆            ┆            ┆          ┆            ┆            ┆            ┆ ...     │
│            ┆            ┆            ┆          ┆            ┆            ┆            ┆ 0.0]    │
├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤
│ 3.04.0zwakZandigdonkergrfalsenietOrganinull       ┆ [0.0,   │
│            ┆            ┆ eKleiijs      ┆            ┆ sch        ┆            ┆ 0.0,    │
│            ┆            ┆            ┆          ┆            ┆            ┆            ┆ ...     │
│            ┆            ┆            ┆          ┆            ┆            ┆            ┆ 0.0]    │
└────────────┴────────────┴────────────┴──────────┴────────────┴────────────┴────────────┴─────────┘

Plotting

from pygef import read_cpt, read_bore
from pygef.plotting import plot_cpt, plot_bore

# plot cpt file
plot_cpt(read_cpt("myfile.xml"))

# plot a bore file
plot_bore(read_bore("myfile.xml"))

Documentation

Build the docs:

python -m pip install --upgrade pip setuptools
pip install -r requirements.txt
pip install .

sphinx-build -b html docs public

Format

We format our code with black and isort.

black --config "pyproject.toml" .
isort --settings-path "pyproject.toml" .

Lint

To maintain code quality we use the GitHub super-linter.

To run the linters locally, run the following bash script from the root directory:

docker run \
--env VALIDATE_ALL_CODEBASE=false \
--env RUN_LOCAL=true \
--env VALIDATE_JSCPD=false \
--env VALIDATE_CSS=false \
--env VALIDATE_BASH=false \
--env VALIDATE_YAML=false \
--env VALIDATE_PYTHON_PYLINT=false \
--env VALIDATE_NATURAL_LANGUAGE=false \
--env VALIDATE_MARKDOWN=false \
--env LINTER_RULES_PATH=/ \
--env PYTHON_BLACK_CONFIG_FILE=pyproject.toml \
--env PYTHON_ISORT_CONFIG_FILE=pyproject.toml \
--env PYTHON_MYPY_CONFIG_FILE=pyproject.toml \
--env PYTHON_FLAKE8_CONFIG_FILE=.flake8 \
-v $(pwd):/tmp/lint ghcr.io/super-linter/super-linter:v6

UnitTest

Test the software with the use of coverage:

python -m pip install --upgrade pip setuptools
pip install -r requirements.txt
pip install .

coverage run -m pytest

Requirements

Requirements are autogenerated by pip-compile with python 3.9

pip-compile --extra=test --extra=docs --extra=lint --extra=map --output-file=requirements.txt pyproject.toml

To update the requirements within the defined ranges, run:

pip-compile --upgrade --extra=test --extra=docs --extra=lint --extra=map --output-file=requirements.txt pyproject.toml

Automatically upgrade your Polars code so it's compatible with future versions.

pip install -U polars-upgrade
polars-upgrade src/pygef tests --target-version=0.19.19