RNTuple performance improvements advice needed

[JacekHoleczek]

I am trying to write a small python macro that should "demonstrate" that uproot is (almost) as fast as C++.

For test purposes, I have got a ROOT::Experimental::RNTuple created by ROOT 6.28/10 (its structure "mimics" the TTree from my other #1106 thread).

It seems I can't get it running.
(Well, until now, I have never played with any RNTuple.)

First, I created a conda-forge "scikit-hep" environment. It comes with Python 3.11.7 and an old Uproot 5.0.13 (can one ask someone to upgrade Uproot to the latest version in "Scikit-HEP"?).

Then, I created a conda-forge "uproot" environment. It comes with Python 3.12.1 and the newest Uproot 5.2.2 (but the "hist" cannot be added).

In both environments, I can:
t = uproot.open({'data/my_ntuple.root': 'hk'})

The "TQReal" is just a simple class with several members that keep C++ fundamental data types.

In both environments, when I try:
t['TQReal']
I get:
TypeError: 'Model_ROOT_3a3a_Experimental_3a3a_RNTuple' object is not subscriptable

Then, when I try:
tq_real = t.arrays(['TQReal.nhits', 'TQReal.pc2pe', 'TQReal.t0'])
Uproot 5.0.13 is able to retrieve them, but Uproot 5.2.2 terribly breaks.

Also, if I add "library='np'" in the above line, both Uproot versions break.

What now?

BTW. You might be interested in this recent ROOT Forum thread: RNTuple: Where are we now and what’s next?

[eduardo-rodrigues]

Hello @JacekHoleczek, I happen to try and follow this hence catched your comment

First, I created a conda-forge "scikit-hep" environment. It comes with Python 3.11.7 and an old Uproot 5.0.13 (can one ask someone to upgrade Uproot to the latest version in "Scikit-HEP"?).

Thank you for the interest. A new release is indeed due since a while and it's been on my to-do list. But then (core) work kicks in ...

I will do it today.

Note that we are always keen on getting contributions and contributors, across the org, hence you are more than welcome to create PR :-).

[JacekHoleczek]

Maybe one should also upgrade the "awkward" and "dask-awkward".

[eduardo-rodrigues]

I don't understand - as far as the current metapackage is concerned, meaning the packages it bundles, a pip install will now pick up all latest minor versions.

[JacekHoleczek]

Currently, in the "scikit-hep" environment I have "awkward 2.4.10" and "dask-awkward 2023.11.2", while in "uproot" environment I have "awkward 2.5.2" and "dask-awkward 2024.1.2".
Maybe when the Uproot gets upgraded also the "awkward" and "dask-awkward" will be automatically upgraded (I'll verify it once your merge request is processed and I get the new Uproot in "scikit-hep" from the conda-forge ).

[eduardo-rodrigues]

See https://github.com/scikit-hep/scikit-hep/releases/tag/2024.1.1 - you will now be picking up awkward 2.5.x and all that goes with it :-).

[JacekHoleczek]

Thanks. All these packages got upgraded in conda-forge.

[jpivarski]

Just for caution: the RNTuple format was recently updated and Uproot was updated to account for it (#962, #1004). Since the format is in active development, we're not making any attempt to be backward compatible for old RNTuple formats—not until RNTuple declares "1.0", anyway. Be sure to align your ROOT version with Uproot version.

Cc: @ariostas

[JacekHoleczek]

O.K. But how can I learn which ROOT versions some specific Uproot version supports? For example, I have here ROOT 6.26/10, 6.26/14, 6.28/06, 6.28/10, and 6.30/02 (on different Linux systems). It is written somewhere in Uproot "release notes"?

[jpivarski]

Uh... not really. Since this is on the development hot-path, we're just trying to keep the latest Uproot in line with the latest ROOT. We can't make more nuanced guarantees.

(Outside of RNTuple, the version support for TTrees was tested for all ROOT versions released 5 years before Uproot 1.0, which is 5 years ago now, so it should be an interval of validity of 10 years: see the uproot-sample-*.root files here. But that's not what you're interested in: you're interested in RNTuples.)

[jpivarski]

Okay, so now there's a new ROOT on the 6.30 line: 6.30.04.¹

Footnotes

1. I wonder if this means they're going to stop using slashes in version numbers and use only dots instead. That would make it easier to fit into systems that expect version numbers in a particular format, though I would always just convert a slash into a dot, anyway. ↩

[JacekHoleczek]

Nothing related to RNTuple is changed in the 6.30.04 (as compared to the 6.30.02).

I think I remember an "official" statement that (from now on) the "/" will be replaced with a ".".

[JacekHoleczek]

O.K. I "regenerated" my RNTuple test file with ROOT 6.30/02.

Now, Uproot 5.2.2 is able to get: tq_real = t.arrays(['TQReal.nhits', 'TQReal.pc2pe', 'TQReal.t0'])

Well, If I try to add "library='np'", I get:
TypeError: Model_ROOT_3a3a_Experimental_3a3a_RNTuple.arrays() got an unexpected keyword argument 'library'

I have some problems when using it (FieldNotFoundError: no field 'TQReal.nhits' in record with 1 fields):
tq_real['TQReal.nhits']

Why does the "t.arrays" call accept "'TQReal.nhits'" but then it fails?

This works, though (so I could use it in my "proof of concept" script):
tq_real['TQReal']['nhits']

But I still cannot get individual arrays (object is not subscriptable):
t['TQReal'].array()
t['TQReal.nhits'].array()

Any replacement?
I've only found a crazy way:
t.arrays(['A.B.C.D'])['A']['B']['C']['D']

Actually, it seems I have a much more serious problem:
tq_real_hits = t.arrays(['TQReal'])
returns:

File ~/micromamba/envs/scikit-hep/lib/python3.11/site-packages/uproot/models/RNTuple.py:507, in Model_ROOT_3a3a_Experimental_3a3a_RNTuple.arrays(self, filter_names, filter_typenames, entry_start, entry_stop, decompression_executor, array_cache)
    500     raise (
    501         RuntimeError(
    502             f"The key: {key} is missing both from the columns records and the alias columns."
    503         )
    504     )
    506 dtype_byte = self._column_records_dict[id]["rel_crs"][0].type
--> 507 content = self.read_col_pages(
    508     id, range(start_cluster_idx, stop_cluster_idx)
    509 )
    510 if dtype_byte == uproot.const.rntuple_col_type_to_num_dict["switch"]:
    511     kindex, tags = _split_switch_bits(content)
(...)

I suspect the problem is that the TQReal keeps ragged hits arrays (std::vector<FormatHit> which then have cable, T, and Q data members).

[jpivarski]

O.K. I "regenerated" my RNTuple test file with ROOT 6.30/02.

I confirm that this is the latest release of ROOT.

Well, If I try to add "library='np'", I get:
TypeError: Model_ROOT_3a3a_Experimental_3a3a_RNTuple.arrays() got an unexpected keyword argument 'library'

The Model_ROOT_3a3a_Experimental_3a3a_RNTuple.arrays function (do we have a better name for it than that? I'm surprised...) is not the TTree.arrays function; it doesn't support all of the same arguments. In particular, we only plan to support RNTuple → Awkward Array, since the RNTuple format is one-to-one with Awkward Arrays. Unlike TTree, there's no advantage to going directly to NumPy arrays; going through Awkward and ak.to_numpy is just as good.

As for the other errors, @ariostas is planning to take a look at the RNTuple code. This might be a good place to start.

[JacekHoleczek]

O.K. Many thanks for your help.

For the unexpected "library" argument. Maybe it would be a good idea to support the same user interface for TTree and RNTuple. Then, one could easily "reuse" scripts (of course, some arguments would be "ignored" when not needed).

[jpivarski]

FYI, there's a new version of ROOT, v6.28/12, which might have more RNTuple updates.

[JacekHoleczek]

I don't think they changed anything related to RNTuple in the 6.28/12 (as compared to the 6.28/10).

All development goes into the "master" branch, which will become 6.32 at one point (some changes may appear in any new 6.30 release, though).

UPDATE (2024.01.31): Nothing related to RNTuple is changed in the 6.30.04 (as compared to the 6.30.02).

[JacekHoleczek]

Well, if you want to play with these problems ... I created a tiny RNTuple with just 2 events.
With ROOT 6.30.02: my_ntuple.root.zip
With ROOT 6.30.04: my_ntuple.root.zip

[ariostas]

Hi @JacekHoleczek, would you be able to regenerate the RNTuple file or point me to the code that generates it? After #1191 we should be able to read most RNTuple files.