Support reading and writing all LAS 3.0 features

[kinverarity1]

LAS 3 specification: https://github.com/kinverarity1/lasio/blob/main/standards/LAS_3_File_Structure.pdf

Tasks:

• PR Rearrange reader #327 Re-arrange LASFile.read so that header sections are parsed before touching any data sections (see comments in Parse dates to datetime objects #1)
• Issue Parse comma-delimited ~ASCII sections #265 Write a reader to handle comma separated data sections and strings -> numpy record array (refer to discussion in Parse dates to datetime objects #1)
  • partly completed - see comment below
• PR Read data section as dataframe #424: Ensure LASFile can handle different data types per CurveItem (possible partial solution PR in Read data section as dataframe #424 which uses pd.DataFrame, although not pd.read_csv) - Update 26th April: resolved by Allow different data types per curve in data section reader #461
• Issue Use an accelerated numpy reader (np.genfromtxt) #446: Use an accelerated pandas reader e.g. pd.read_csv/pd.read_fwf where it is not needed for substitutions etc (refer to discussion in Parse dates to datetime objects #1) - Update 26th April: being worked on in Add a numpy engine for reading using numpy.genfromtxt() #452
• Issue Compile set of real-world LAS 3 files #513: Compile set of real-world LAS 3 files to provide targets for tests and feature implementation.
• Issue Link LAS 3 "data set" sections together #514: Finalize the API on LASFile for multiple linked Data + Definition sections (e.g. refer to discussion in this issue)
• Issue Parse LAS 3 format codes in header section lines #515: Add format to HeaderItem
• Issue TBA Add association to HeaderItem
• Issue TBA: Figure out how to handle runs (needs discussion)

Update May 2020: I will start to sketch out a roadmap for how to achieve this. I think once this is reasonably well tested we can do a version 1 release.

Goals:

• Assume all LAS files are version 3 for parsing, even if they have a VERS code of 2.0. Only avoid this if we have to for performance.
• Aim to improve reading performance times, it's really bad at the moment

Because I expect this work might require a broken branch for a while, let's merge into the las3-develop branch if we need to.

[kinverarity1]

Current error is not very graceful:

D:\study\2015\las_kgs_test\2014\1044761865.las
Traceback (most recent call last):
  File ".\basic_read_test.py", line 20, in <module>
    l = las_reader.read(fn)
  File "d:\work\dewnr\logging_software\las_reader\las_reader\__init__.py", line 5, in read
    return las.LASFile(file)
  File "d:\work\dewnr\logging_software\las_reader\las_reader\las.py", line 132, in __init__
    self.read(file, **kwargs)
  File "d:\work\dewnr\logging_software\las_reader\las_reader\las.py", line 149, in read
    self.well = reader.read_section('~W')
  File "d:\work\dewnr\logging_software\las_reader\las_reader\las.py", line 403, in read_section
    parser = SectionParser(section_name, version=self.version)
  File "d:\work\dewnr\logging_software\las_reader\las_reader\las.py", line 487, in __init__
    section_orders = ORDER_DEFINITIONS[self.version][self.section_name2]
KeyError: 3.0

[VelizarVESSELINOV]

For me two key interesting parts of LAS 3 are:

1. management of arrays (multi dimensional curves)
2. management of string curves (like geological markers)
3. management of reference in date-time

I will be very interesting to see this functionality available.

[kinverarity1]

This will be addressed in PR #106 and ultimately #61

Version number will be bumped to 1 when LAS 3 files can be read in.

[roliveira]

👍

I've been looking on the overall content of the package, specifically the parser and I would like to see what you think about the following suggestions:

• Extend HeaderItem to accept extra items to comply with the LAS 3.0 standard.
  Your current design is this:

 HeaderItem = namedlist("HeaderItem", ["mnemonic", "unit", "value", "descr"])

What I'm proposing is to include two extra items, "format" and "association":

 HeaderItem = namedlist("HeaderItem", ["mnemonic", "unit", "value", "description", "format", "association"])

Those items would be empty in case "VERS" is 1.2 or 2.0. I also think this would ease the support of datetime objects as suggested in #1.

• Drop the "curves" in favor of "definition" and keep a consistency with the 3.0 new naming convention:

~Section_Parameter, ~Section_Definition, and ~Section_Data

It would be some sort of always treat as 3.0 first approach...

Anyway, just my two cents to align my thoughts before doing some actual work.

[kinverarity1]

Hey, thanks for the help!

1 - Extend HeaderItem

Yes, sounds excellent.

2 - Drop the "curves" in favor of "definition"

Yep! Sounds good.

I had another read of the v3 spec. How does the below sound?

• We provide access to each section via a dictionary (maybe hybrid class) in LASFile.sections
• We provide a LASFile.data_sets dictionary (maybe hybrid class) which then (always) has keys 'Parameter', 'Definition', 'Data', the values of which are references to the relevant section in LASFile.sections.

e.g. for the example file in the LAS v3 spec:

LASFile.sections = {                Type                Example ID
    'Version':                  SectionItem()               1
    'Well':                     SectionItems()              2
    'Parameter':                SectionItems()              3
    'Curve':                    SectionItems()              4
    'Drilling_Parameter':       SectionItems()              5
    'Drilling_Definition':      SectionItems()              6
    'Drilling_Data':            np.ndarray / pd.DataFrame   7
    'Core_Parameter[1]':        SectionItems()              8
    'Core_Definition[1]':       SectionItems()              9
    'Core_Data[1]':             np.ndarray / pd.DataFrame   10
    'Core_Parameter[2]':        SectionItems()              11
    'Core_Definition[2]':       SectionItems()              12
    'Core_Data[2]':             np.ndarray / pd.DataFrame   13
    'Inclinometry_Parameter':   SectionItems()              14
    'Inclinometry_Definition':  SectionItems()              15
    'Inclinometry_Data':        np.ndarray / pd.DataFrame   16
    'Test_Parameter':           SectionItems()              17
    'Test_Definition':          SectionItems()              18
    'Test_Data':                np.ndarray / pd.DataFrame   19
    'Tops_Parameter':           SectionItems()              20
    'Tops_Definition':          SectionItems()              21
    'Tops_Data':                np.ndarray / pd.DataFrame   22
    'Perforation_Parameter':    SectionItems()              23
    'Perforation_Definition':   SectionItems()              24
    'Perforation_Data':         np.ndarray / pd.DataFrame   25
    'Ascii':                    np.ndarray / pd.DataFrame   26
}

and the data_sets variable for this would be:

LASFile.data_sets = {
    'ASCII': {
        'Parameter':            SectionItems()              3
        'Definition':           SectionItems()              4
        'Data':                 np.ndarray / pd.DataFrame   26
    },
    'Drilling': {
        'Parameter':            SectionItems()              5
        'Definition':           SectionItems()              6
        'Data':                 np.ndarray / pd.DataFrame   7
    },
    'Core': [
        {
            'Parameter':        SectionItems()              8
            'Definition':       SectionItems()              9
            'Data':             np.ndarray / pd.DataFrame   10
        },
        {
            'Parameter':        SectionItems()              11
            'Definition':       SectionItems()              12
            'Data':             np.ndarray / pd.DataFrame   13
        }
    ],
    'Inclinometry': {
        'Parameter':            SectionItems()              14
        'Definition':           SectionItems()              15
        'Data':                 np.ndarray / pd.DataFrame   16
    },
    'Test': {
        'Parameter':            SectionItems()              17
        'Definition':           SectionItems()              18
        'Data':                 np.ndarray / pd.DataFrame   19
    },
    'Tops': {
        'Parameter':            SectionItems()              20
        'Definition':           SectionItems()              21
        'Data':                 np.ndarray / pd.DataFrame   22
    },
    'Perforation': {
        'Parameter':            SectionItems()              23
        'Definition':           SectionItems()              24
        'Data':                 np.ndarray / pd.DataFrame   25
    }
}

One problem I can foresee with the above is the "Core" dataset. Maybe they should be retained as "Core[1]" and "Core[2]". I don't know... ?

This would be backwards-compatible with LAS v2 because the ~Parameter, ~Curve, and ~ASCII sections could always be considered part of the "ASCII" data set:

eg v2 pseudocode

LASFile.sections = {
    'Version':                  SectionItem()               1
    'Well':                     SectionItem()               2
    'Parameter':                SectionItem()               3
    'Curves':                   SectionItem()               4
    'ASCII':                    np.ndarray / pd.DataFrame   5
}

LASFile.data_sets = {
    'ASCII': {
        'Parameter':            SectionItem()               3
        'Definition':           SectionItem()               4
        'Data':                 np.ndarray / pd.DataFrame   5
    }
}

I think we should avoid calling the ~Parameter, ~Curve, ~ASCII set of sections the "Log" data set as the v3 spec sometimes does, because we might encounter a file with both the ~Parameter, ~Curve, ~ASCII AND the ~Log_Parameter, ~Log_Definition, and ~Log_Data sections.

(SectionItems() is a new class defined in PR #106)

At least two other major changes needed are:

3 - Add data definition types

Not all data is a CurveItem - we would also need ArrayItem and a "StringCurveItem". The latter could perhaps be achieved best with a wrapper around a striplog

4 - Datetime reference curves

This could be best managed by requiring pandas, which has excellent support for using date/time stamps as indexes.

[roliveira]

No problem, I'm glad you found useful! Continuing the discussion...

2 - Drop the "curves" in favor of "definition"

I agree with what you said.

Actually at first I idealised a dataset comprehending of a data and metadata entry. But while reading this I think that what you propose is closer to file contents and more likely to be understood. Like this I also agree on your read of the v3 spec. The concept of LASFile.data_sets fits just right with the standard!

I don't expect to find a lot of 'Core[1]' around there, and if so I don't see a (lot of) problem in keeping the name like that. Actually for me it looks exactly like you would be accessing the index 1 of the Core array, but I force myself to ignore that and read it as it's merely a name chosen by whoever created this particular file.

You actually grouped the 'Core[1]' and 'Core[2]' into a list of dict:

LASFile.data_sets = {
#...
    'Core': [
       {
           'Parameter':        SectionItems()              8
            'Definition':       SectionItems()              9
            'Data':             np.ndarray / pd.DataFrame   10
        },
        {
            'Parameter':        SectionItems()              11
            'Definition':       SectionItems()              12
            'Data':             np.ndarray / pd.DataFrame   13
        }
    ],
#...
}

But if we keep the 'Core[1]' and 'Core[2]' as two different datasets (that actually share the same metadata) it would be:

LASFile.data_sets = {
#...
    'Core[1]': {
            'Parameter':        SectionItems()              8
            'Definition':       SectionItems()              9
            'Data':             np.ndarray / pd.DataFrame   10
        },
    'Core[2]': {
            'Parameter':        SectionItems()              8
            'Definition':       SectionItems()              9
            'Data':             np.ndarray / pd.DataFrame   10
        },
#...
}

If I understood correctly that could also happen with any other section.

How are you planning to approach the different runs that could be present in the files? Like it is present in the sample_las3.0_spec.las file.

#...
#Run 1 Parameters
RUN_DEPTH.M 0, 1500 : Run 1 Depth Interval {F} | Run[1]
#...
#Run 2 Parameters
RUN_DEPTH.M 1500,2513 : Run 2 Depth Interval {F} | Run[2]
#...

3 - Add data definition types

You are right. However I'm not sure we need a particular type for it. For example the The ArrayItem could hold its dimensions via the ndarray shape and the StringCurveItem would be merely a CurveItem with {S} format. This would avoid wrappers for simplicity, only to have a solid base to start with. I'm not entirely familiar with striplog but it sounds promising.

4 - Datetime reference curves

Yes! If pandas eventually gets in as a requirement it's worth to try to leverage of the other packages that come along, such as the support for spreadsheets and so forth.

[kinverarity1]

FYI I'm happy to bring pandas in as a lasio requirement as part of this work. It's probably time.

[shakasaki]

Hello, I am wondering how this is progressing? I just got a hold of lasio and eagerly want to use it to read in some las v3.0 files from optical televiewer data. I attach part of one file here, with only 2 lines of data to avoid overloading you with data!

When trying to read with the current version of lasio I get the following error:

data_in = ls.read(r'example_header.las', ignore_header_errors=True)
Header section Curves regexp=~C was not found.
Found nonstandard LAS section: ~LOG_PARAMETER
Found nonstandard LAS section: ~LOG_DEFINITION
No data section (regexp='~A') found

The data section (as can be seen in the file) starts with
~LOG_DATA | LOG_DEFINITION
so it is obviously not found. For the rest, the errors do not really go away when I add the ignore-header flag.

Should I try this out with some other branch (i.e. las3.0 ? )

Thanks for your help and for setting this up!

example_file.zip

[kinverarity1]

Hi @shakasaki! Thanks for trying out lasio - sorry that it doesn't work yet for LAS 3.0. We are progressing, slowly, having today merged #327 which was the first step in the list above. No, the las3.0 branch does not have any improvements for you yet - we will continue to merge improvements to master unless they are breaking changes.

I expect your file may work prior to full LAS 3 support. Thank you very much for providing a example file, I'll test it out shortly and see what stands in the way of reading the data section in at least.

[shakasaki]

Hello @kinverarity1 and thanks for the response. Please do let me know if you can extract at least the data - that would be great for now. Or perhaps, do you know any resource where I can convert las3.0 to older versions (las 2.0 for example). Thank you for the help!

[kinverarity1]

@shakasaki using lasio master you should be able to use this to read the data section from your file: https://gist.github.com/kinverarity1/92f00b781472512349a9312d75fd4c33

It is very hack-y and will certainly break in the future as we add proper support for this kind of functionality, but it should get you by in the meantime. I wasn't sure how to parse values like 73.56.5 or 0.0.0, but can easily tweak this if you let me know how they should be handled 😎

[shakasaki]

@kinverarity1 Thank you so much and apologies for the late response. I have been doing fieldwork for the past few days. I really appreciate the effort you put into helping me out. I hope I can give back to this community somehow!

The values (e.g. 73.56.5) are RGB triples, that is, they denote a colored pixel. These data are from an optical televiewer. Actually the first value is a depth, followed by 360 RBG triples (one for each degree). I will try out the hack today and see how it works out. Thanks again

[kinverarity1]

No problem at all. You already have helped out the community by posting your example! 😁 I have adapted the notebook to parse the data as RGB triples - see comments on the gist.

[shakasaki]

Hello @kinverarity1

I have been using the code snipped you wrote to read in las 3.0 files and it works. However, when the file is too large (I have files with [198441 rows x 361 columns] ) the approach crashes.

I coded another hacky example using pandas and it can handle the large files, so I was wondering why this is, since I thought lasio also uses pandas internally.

Here is my approach:

import numpy as np
import pandas as pd # to read in dataframe
from subprocess import check_output # to find where the data starts in the file

file_name = 'input las file that is too large for lasio hack-y example'

# following line uses grep to find out where the data starts in the file
first_dataline = int(str(check_output(["grep", "-n", "~LOG_DATA", file_name]), 'utf-8').split(':')[0])

# read in only data, otherwise pandas crashes due to unicode characters
file_in = pd.read_csv(file_name, skiprows=first_dataline, header=None)

# create a depth array and an array to store RGB values as u integers
depth = file_in[0].to_numpy(dtype='float')
store_array = np.ndarray((file_in.shape[0], file_in.shape[1] -1,3),dtype='uint8')

# loop over each column in the ATV data
for col in range(360):
    temp = file_in[col+1].str.split('.', expand = True)
    store_array[:,col-1,:] = temp.apply(lambda x: pd.to_numeric(x, downcast='unsigned')).to_numpy(dtype='uint8')

The approach works well with the large files, but does not read in the header of course. Right now, i'm fine with it. Hopefully this insight can help in making lasio able to handle larger files (if that is actually a problem!)

[kinverarity1]

Thanks for the code! Lasio doesn't use pandas as a reader yet (see #1 and this thread for a discussion of the reasons) but we plan to switch to it soon. I'm glad you found a solution!

[donald-keighley]

I'm actually writing a LAS reader for R right now and have been struggling through LAS 3.0. Will be interested to see how you handle it.

[kinverarity1]

I completely forgot about this wiki page from ages ago:

Additions in LAS 3 to look out for:

• parameter zoning - values in HeaderItems can now be arrays e.g.

  NMAT_Depth[1].M 500,1500 : Neutron Matrix Depth interval {F}

• Extend HeaderItem to include format= and association=

  format should probably be left as is and then used when parsing data - special cases will be strings, datetimes, and DMS location data. Strings will need to be read as is, datetimes parsed to datetime objects, and DMS converted to decimal degrees.

  assocation should be a list of either the associated HeaderItem, or, if that doesn't exist, strings. It has to be a list because there can be multiples:

  NEUT .V/V : Neutron Porosity | MATR[1], MATR[2]

• Data section name. Ideally I think it's meant to be ~SECTNAME_Data | SECTDEF where again the | indicates the data section SECTNAME represents that defined by SECTDEF. But in practice I suspect this is not reliable, so probably best to assume SECTNAME == SECTDEF and just parse this pattern where it exists and use it to construct the LASFile.data_sets dictionary.

• Automatically turn lots of curves e.g. TT[1], TT[2], ... TT[n] into a single TT CurveItem with 2D data array.

[kinverarity1]

@dcslagel Regarding the Transform 21 hackathon - do you think perhaps we should convert (perhaps manually) this issue to a GitHub 'discussion'? I am getting lost in the discussion of the many different elements of what adds up to "LAS 3" support spread across multiple issues. It might work better as a discussion, with different threads for each element? I'm happy to do this tomorrow if you think it best.

[dcslagel]

Hi @kinverarity1, thought about a bit...

1. It makes sense to move this to a discussion since it is more of a general discussion than a specific task. However we would lose the Issue # which has been used to reference this in the paths. If that reference is of value then keeping it as an issue is okay too.

2. The really critical thing is that we breakdown LAS-3 to workable chunks in 'sub' issues. We probably should keep the list of sub-issues at the top of this issue/discussion as well as in the Hackathon Project.

[kinverarity1]

👍 I don't want to break links so I'll leave it here but clean up the body of the issue and make sure we have at least 'stub' issues for each individual part that needs doing.

Also, I'll change the title, since some version 3.0 files technically do now "work" in lasio, just not well.

[kinverarity1]

Update here: #452 was merged, so the next steps are making sure we can read comma-delimited data sections (#265) and then move on to the other LAS 3 issues which mostly relate to the header sections.

[dcslagel]

Although there is still some work to be done on Issue #265, the basic ability to read the comma-delimited data sections is merged to the main(master) branch via pull-request #485.

[ghost]

Hoping progress is still being made for handling the LAS 3.0 files. Is the 2 year gap in comments a bad sign?

[thirdplanetgeospatial]

Hoping progress is still being made for handling the LAS 3.0 files. Is the 2 year gap in comments a bad sign?

Sorry, GH account issues. Still interested in the LAS 3.0 progress.

[donald-keighley]

I haven't done anything. I remember looking at it and thinking it would require some pretty big changes and pose some backward compatibility problems.

Also, I'm an R user for the most part. I wrote my own package that will read las 3.0.

https://github.com/donald-keighley/lasr

The core of it is written in c++ so maybe it could be ported over? I don't know how to do that.

[donald-keighley]

Also how many las 3 files do you have to read? I've seen very few of these in the wild.

[kinverarity1]

Hey @thirdplanetgeospatial

Yep it's a bad sign. Although I'd love to get this working, I have no spare time at all for this. I also never (like zero times) see LAS 3 files outside of lasio development, as I don't work in oil & gas. My archive of files I work with daily is mostly LAS 1.2 😄

Happy to look at PRs which tackle specific parts of the problem here. But lasio has some poor design at its core, I suspect. I wrote it when I was learning Python. It would probably make more sense for someone to take lasio and significantly re-work it, or re-write a better approach from scratch. Or use R instead to use @donald-keighley's package (which is really good by the look of it!)

[thirdplanetgeospatial]

Also how many las 3 files do you have to read? I've seen very few of these in the wild.

Hey, thanks for the insight. Yeah, the only ones I've seen are relative to my particular use case. The North Carolina Groundwater Section provides LAS files for download from groundwater monitoring wells & they (at least some) are LAS 3.0.

[thirdplanetgeospatial]

Hey @thirdplanetgeospatial

Yep it's a bad sign. Although I'd love to get this working, I have no spare time at all for this. I also never (like zero times) see LAS 3 files outside of lasio development, as I don't work in oil & gas. My archive of files I work with daily is mostly LAS 1.2 😄

Happy to look at PRs which tackle specific parts of the problem here. But lasio has some poor design at its core, I suspect. I wrote it when I was learning Python. It would probably make more sense for someone to take lasio and significantly re-work it, or re-write a better approach from scratch. Or use R instead to use @donald-keighley's package (which is really good by the look of it!)

Ah, I see. Thanks for the background. Lasio actually works very well so far for the LAS viewer I built around it. The LAS 3.0 files are the only exception. Unfortunately, LAS 3.0 apparently exist to some extent in my world (see response to Donald Keighley). Anyway, thanks for the work you have done! Good luck with all.