Issues with pytables

[jeanconn]

What is the fallout from having the broken version 3.4.2 in skare3?

(Pytables issue 679)

[taldcroft]

I think the impact is that we cannot create indexed tables. But experience with agasc shows that Py3 can successfully read and use an indexed table created with Py2.

So for Ska3 1.0 (my working tag for the initial flight configured release), I think we can live with it. Any finalization work on the AGASC 1.7 products can be done with Py2. Sound reasonable?

[jeanconn]

I think so? I just wasn't sure that there weren't an read problems just yet as I hadn't reviewed the scope of the eng_archive, cmd_states, and agasc read-from-h5 tests with this bug in mind. I also wasn't sure if Py3 was actually using the index on files that had an index (I don't think we have any tests for that).

[jeanconn]

Also, apparently I haven't figured out how to add that index in py2 and read it in py3.

added to a test file

chunkshape := (4064,)
  autoIndex := True
  colindexes := {
    "AGASC_ID": Index(9, full, shuffle, zlib(1)).is_CSI=True,
    "DEC": Index(6, medium, shuffle, zlib(1)).is_CSI=False,
    "RA": Index(6, medium, shuffle, zlib(1)).is_CSI=False}

In [5]: 
Do you really want to exit ([y]/n)? y
Closing remaining open files: /fido.real/tmp/agasc1p7.h5... done

tried to read from py3

/fido.real/skare3/lib/python3.6/site-packages/tables/group.py:1187: UserWarning: problems loading leaf ``/data``::

  '<' not supported between instances of 'NoneType' and 'tuple'

The leaf will become an ``UnImplemented`` node.
  % (self._g_join(childname), exc))

[taldcroft]

I confirmed that https://github.com/sot/agasc/blob/master/create_mini_agasc_h5.py still works to create a 1.6 miniagasc that works with ska3. See ~aldcroft/git/agasc/miniagasc-py2.h5 for that file.

[jeanconn]

I'll try again.

[jeanconn]

Actually @malgosias, do you want to take this? The goal is to have an agasc1p7.h5 file that has the three indexes and loads successfully from py2 or py3. I don't know if my previous failure to add the indexes from py2 means I did something wrong or if there is a problem with the current agasc1p7.h5 file. If there is a problem then the process that we'd suggested to check the fits files might also be used to make the final flight version of the table (would still need added indexes per the lines in
create_mini_agasc_h5.py

[jeanconn]

When things are a little calmer, we might want to check if there's the problem that an h5 file created with 3.4.2 may have other defects (sot/agasc#25 (comment)) . If that's the case, there may be value in doing the source-build upgrade to pytables 3.4.4 (because having all h5 files that we make on the py3 side be possibly subtly broken seems undesired)

[javierggt]

Old issue. Context changed. All versions are different now.