arrays dropping (or overwriting?) branches with similar names

[tomeichlersmith]

I have a TTree with branches that are all the same C++ class and I would like to read all of them into memory. I was able to generate a small enough file which I've just uploaded here if you wish to play with it: test.root.tar.gz

This is where I'm at version wise.

awkward==2.5.0
awkward-cpp==26
numpy==1.21.5
pandas==1.5.2
uproot==5.1.2

For those following along at home, I've loaded the TTree from the example file with uproot by doing

t = uproot.open({'test.root': 'performance/by_event'})

The shape of my TTree is pretty simple. As mentioned, each branch (except a completion-flag) is the same C++ class and so there is a lot of repetition.

In [3]: t.show()                                                                                                                                                                                                                              
name                 | typename                 | interpretation                                                                                                                                                                              
---------------------+--------------------------+-------------------------------                                                                                                                                                              
completed            | bool                     | AsDtype('bool')
__ALL__              | framework::performanc... | AsGroup(<TBranchElement '__...
__ALL__/start_time_  | int64_t                  | AsDtype('>i8')
__ALL__/duration_    | double                   | AsDtype('>f8')
MultiTry             | framework::performanc... | AsGroup(<TBranchElement 'Mu...
MultiTry/start_time_ | int64_t                  | AsDtype('>i8')
MultiTry/duration_   | double                   | AsDtype('>f8')
Recon                | framework::performanc... | AsGroup(<TBranchElement 'Re...
Recon/start_time_    | int64_t                  | AsDtype('>i8')
Recon/duration_      | double                   | AsDtype('>f8')

Both the np and ak libraries only bring two of the six branches into memory.

In [4]: t.arrays(library='ak').type.show()
4 * {
    completed: bool,
    start_time_: int64,
    duration_: float64
}

In [5]: t.arrays(library='np')
Out[5]: 
{'completed': array([ True, False,  True,  True]),
 'start_time_': array([1701965077841635851, 1701965077856929667, 1701965077857039110,
        1701965077857187765]),
 'duration_': array([7.22315e-04, 5.76170e-05, 8.84370e-05, 4.10540e-05])}

The pd library fails completely due to a key error.

In [6]: t.arrays(library='pd')
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[6], line 1
----> 1 t.arrays(library='pd')

File ~/.local/lib/python3.10/site-packages/uproot/behaviors/TBranch.py:904, in HasBranches.arrays(self, expressions, cut, filter_name, filter_typename, filter_branch, aliases, language, entry_start, entry_stop, decompression_executor, interpretation_executor, array_cache, library, ak_add_doc, how)
    897 del arrays
    899 expression_context = [
    900     (e, c) for e, c in expression_context if c["is_primary"] and not c["is_cut"]
    901 ]
    903 return _ak_add_doc(
--> 904     library.group(output, expression_context, how), self, ak_add_doc
    905 )

File ~/.local/lib/python3.10/site-packages/uproot/interpretation/library.py:872, in Pandas.group(self, arrays, expression_context, how)
    870 elif uproot._util.isstr(how) or how is None:
    871     arrays, names = _pandas_only_series(pandas, arrays, expression_context)
--> 872     return _pandas_memory_efficient(pandas, arrays, names)
    874 else:
    875     raise TypeError(
    876         f"for library {self.name}, how must be tuple, list, dict, str (for "
    877         "pandas.merge's 'how' parameter, or None (for one or more"
    878         "DataFrames without merging)"
    879     )

File ~/.local/lib/python3.10/site-packages/uproot/interpretation/library.py:798, in _pandas_memory_efficient(pandas, series, names)
    796             out = series[name].to_frame(name=name)
    797     else:
--> 798         out[name] = series[name]
    799     del series[name]
    800 if out is None:

KeyError: 'start_time_'

To me, this looks like an issue with how the branch-name-shortening is handling branches of similar/same types. It doesn't actually struggle with loading the data since I can instruct the library='np' option to stop worrying about the branch names and in this case we do see all of the branches.

In [7]: t.arrays(library='np', how=tuple)
Out[7]: 
(array([ True, False,  True,  True]),
 array([1701965077841635851, 1701965077856929667, 1701965077857039110,
        1701965077857187765]),
 array([7.22315e-04, 5.76170e-05, 8.84370e-05, 4.10540e-05]),
 array([1701965077841635851, 1701965077856929667, 1701965077857039110,
        1701965077857187765]),
 array([7.22315e-04, 5.76170e-05, 8.84370e-05, 4.10540e-05]),
 array([1701965077841635851, 1701965077856929667, 1701965077857039110,
        1701965077857187765]),
 array([7.22315e-04, 5.76170e-05, 8.84370e-05, 4.10540e-05]))

My questions are

• Are there special keywords that I can pass to arrays to avoid this overwriting behavior? I imagine some intelligent use of aliases would allow me to remap the branch names (with /) to names that won't clash (i.e. without the /).
• Is this a bug that could be resolved? Maybe fallback to un-shortened names if the shortened names clash?

[tomeichlersmith]

Without updating the uproot source, I can get around this issue by specifying the branches I wish to read with the expressions argument. For example,

In [14]: t.arrays(expressions=['__ALL__/duration_','MultiTry/duration_'], library='np')
Out[14]: 
{'__ALL__/duration_': array([7.22315e-04, 5.76170e-05, 8.84370e-05, 4.10540e-05]),
 'MultiTry/duration_': array([5.4085e-04, 5.5649e-05, 7.8560e-05, 3.5065e-05])}

We can get exactly what I'm interested in if we ask for the base TBranchElement branches explicitly.

In [18]: t.arrays(expressions=[k for k in t.keys() if '/' not in k]).type.show()
4 * {
    completed: bool,
    __ALL__: {
        start_time_: int64,
        duration_: float64
    },
    MultiTry: {
        start_time_: int64,
        duration_: float64
    },
    Recon: {
        start_time_: int64,
        duration_: float64
    }
}

In [19]: t.arrays(expressions=[k for k in t.keys() if '/' not in k], library='np')
Out[19]: 
{'completed': array([ True, False,  True,  True]),
 '__ALL__': {'start_time_': array([1701965077841635851, 1701965077856929667, 1701965077857039110,
         1701965077857187765]),
  'duration_': array([7.22315e-04, 5.76170e-05, 8.84370e-05, 4.10540e-05])},
 'MultiTry': {'start_time_': array([1701965077841635851, 1701965077856929667, 1701965077857039110,
         1701965077857187765]),
  'duration_': array([7.22315e-04, 5.76170e-05, 8.84370e-05, 4.10540e-05])},
 'Recon': {'start_time_': array([1701965077841635851, 1701965077856929667, 1701965077857039110,
         1701965077857187765]),
  'duration_': array([7.22315e-04, 5.76170e-05, 8.84370e-05, 4.10540e-05])}}

`library='pd'` still errors out though

In [20]: t.arrays(expressions=[k for k in t.keys() if '/' not in k], library='pd')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-20-0272b5a4f214> in ?()
----> 1 t.arrays(expressions=[k for k in t.keys() if '/' not in k], library='pd')

~/.local/lib/python3.10/site-packages/uproot/behaviors/TBranch.py in ?(self, expressions, cut, filter_name, filter_typename, filter_branch, aliases, language, entry_start, entry_stop, decompression_executor, interpretation_executor, array_cache, library, ak_add_doc, how)
    900             (e, c) for e, c in expression_context if c["is_primary"] and not c["is_cut"]
    901         ]
    902 
    903         return _ak_add_doc(
--> 904             library.group(output, expression_context, how), self, ak_add_doc
    905         )

~/.local/lib/python3.10/site-packages/uproot/interpretation/library.py in ?(self, arrays, expression_context, how)
    868             return {_rename(name, c): arrays[name] for name, c in expression_context}
    869 
    870         elif uproot._util.isstr(how) or how is None:
    871             arrays, names = _pandas_only_series(pandas, arrays, expression_context)
--> 872             return _pandas_memory_efficient(pandas, arrays, names)
    873 
    874         else:
    875             raise TypeError(

~/.local/lib/python3.10/site-packages/uproot/interpretation/library.py in ?(pandas, series, names)
    794                 out = pandas.Series(data=series[name]).to_frame(name=name)
    795             else:
    796                 out = series[name].to_frame(name=name)
    797         else:
--> 798             out[name] = series[name]
    799         del series[name]
    800     if out is None:
    801         return pandas.DataFrame(data=series, columns=names)

/usr/local/lib/python3.10/dist-packages/pandas/core/frame.py in ?(self, key, value)
   3964             self._setitem_frame(key, value)
   3965         elif isinstance(key, (Series, np.ndarray, list, Index)):
   3966             self._setitem_array(key, value)
   3967         elif isinstance(value, DataFrame):
-> 3968             self._set_item_frame_value(key, value)
   3969         elif (
   3970             is_list_like(value)
   3971             and not self.columns.is_unique

/usr/local/lib/python3.10/dist-packages/pandas/core/frame.py in ?(self, key, value)
   4119             self._set_item_mgr(key, arraylike)
   4120             return
   4121 
   4122         if len(value.columns) != 1:
-> 4123             raise ValueError(
   4124                 "Cannot set a DataFrame with multiple columns to the single "
   4125                 f"column {key}"
   4126             )

ValueError: Cannot set a DataFrame with multiple columns to the single column __ALL__

This still leaves the question about if this is a bug or if this is expected/understood behavior. I was expecting the results shown above to be the default behavior, but perhaps the implementation is too convoluted to do that?

[tomeichlersmith]

A more complete answer which is shorter and a little more refined is

t.arrays(expressions=t.keys(recursive=False))

for library='ak' (the default) or library='np', and

t.arrays(expressions=t.keys(filter_typename=lambda tn: '::' not in tn), library='pd')

for library='pd'.

[tomeichlersmith]

I think the actual bug is the fact that uproot silently ignores/overwrites certain branches when there is a name clash. Perhaps a warning can be printed or an exception thrown if there is a name clash? I cannot think of a situation where someone would want branches to be overwritten by sub-branches of another branch so I'd probably go for an exception.

[jpivarski]

No, we don't want identical subbranch names to overwrite each other. From what I see of your debugging, it looks like the subbranch names are the same but they're under branches with different names. Usually, ROOT files are made with an option set that gives the subbranches fully qualified names (that is, if start_time_ is under __ALL__, then it will be named __ALL__.start_time_). I think that choice is made by naming a branch with a trailing .: that's what tells ROOT to make fully qualified subbranch names.

But in the case that they aren't, then they ought to be distinguished. However, if someone uses the . to get fully qualified names, and we try to distinguish names by fully qualifying them, they would then be like __ALL__.__ALL__.start_time_. I don't know how we would detect whether the . option has been chosen, and it would be surprising behavior to have the extra qualification only turn on if there's a conflict. Not as surprising as losing a branch, though.

Do you see where in the code the subbranches are being clobbered? I'd be fine with fully qualifying everything, even if some people get longer names than they want. (It's easier to work around unexpectedly long names than missing branches.) Do all of the names enter this finalize function? Are they among the arrays that are passed to group?

uproot5/src/uproot/interpretation/library.py

Lines 836 to 886 in c5ff061

	def finalize(self, array, branch, interpretation, entry_start, entry_stop, options):
	pandas = self.imported
	index = _pandas_basic_index(pandas, entry_start, entry_stop)
	if (
	isinstance(array, numpy.ndarray)
	and array.dtype.names is None
	and len(array.shape) == 1
	and array.dtype != numpy.dtype(object)
	):
	return pandas.Series(array, index=index)
	try:
	interpretation.awkward_form(None)
	except uproot.interpretation.objects.CannotBeAwkward:
	pass
	else:
	array = _libraries[Awkward.name].finalize(
	array, branch, interpretation, entry_start, entry_stop, options
	)
	if isinstance(
	array.type.content, uproot.extras.awkward().types.NumpyType
	) and array.layout.minmax_depth == (1, 1):
	array = array.to_numpy()
	else:
	array = uproot.extras.awkward_pandas().AwkwardExtensionArray(array)
	return pandas.Series(array, index=index)
	def group(self, arrays, expression_context, how):
	pandas = self.imported
	if how is tuple:
	return tuple(arrays[name] for name, _ in expression_context)
	elif how is list:
	return [arrays[name] for name, _ in expression_context]
	elif how is dict:
	return {_rename(name, c): arrays[name] for name, c in expression_context}
	elif isinstance(how, str) or how is None:
	arrays, names = _pandas_only_series(pandas, arrays, expression_context)
	return _pandas_memory_efficient(pandas, arrays, names)
	else:
	raise TypeError(
	f"for library {self.name}, how must be tuple, list, dict, str (for "
	"pandas.merge's 'how' parameter, or None (for one or more"
	"DataFrames without merging)"
	)

[tomeichlersmith]

I was unaware of the ability to alter how ROOT makes the subbranch names, that makes sense then to strip the redundant prefix (before the /) in the full path name.

I do not see where in the code the subbranches are being clobbered, I expect (since I can see all of them when passing how=tuple), that the construction of the dict using the new names as keys is where this originates. Both NumPy and Awkward use the same procedure and don't check for clashes in the values returned by _rename.

uproot5/src/uproot/interpretation/library.py

Lines 874 to 875 in c5ff061

	elif how is dict:
	return {_rename(name, c): arrays[name] for name, c in expression_context}

The new names are retrieved from the passed expression_context which has its own construction path.

In any case, since the origin of expression_context is compilcated and there is another way to create branches that provides fully-specified names, I am ok with the current reading solution I was able to deduce and would just suggest that a check for clashes is inserted into the code above so that the user can be notified when deduced branch names are repeating themselves.

[tomeichlersmith]

I do have access to how the TTree is written, so I updated it to fully-specify the sub-branch names. I'm just reporting back here on what that looks like with the uproot version I've been using above.

There is not overlapping of branches anymore (as expected)

In [15]: t.arrays().type.show()
4 * {
    completed: bool,
    "__ALL__.start_time_": int64,
    "__ALL__.duration_": float64,
    "MultiTry.start_time_": int64,
    "MultiTry.duration_": float64,
    "Recon.start_time_": int64,
    "Recon.duration_": float64
}

The names are longer, but there is the / character that can be used for shortening if desired.

In [14]: t.keys()
Out[14]: 
['completed',
 '__ALL__.',
 '__ALL__./__ALL__.start_time_',
 '__ALL__./__ALL__.duration_',
 'MultiTry.',
 'MultiTry./MultiTry.start_time_',
 'MultiTry./MultiTry.duration_',
 'Recon.',
 'Recon./Recon.start_time_',
 'Recon./Recon.duration_']

Specifically requesting the "base" branches like I did before offers (what I think) a more natural shape for the data but the sub-branches in this case still have the long names which can probably be fixed by renaming record fields via ak.zip.

In [13]: t.arrays(
    ...:     expressions=t.keys(recursive=False),
    ...:     ).type.show()
4 * {
    completed: bool,
    "__ALL__.": {
        "__ALL__.start_time_": int64,
        "__ALL__.duration_": float64
    },
    "MultiTry.": {
        "MultiTry.start_time_": int64,
        "MultiTry.duration_": float64
    },
    "Recon.": {
        "Recon.start_time_": int64,
        "Recon.duration_": float64
    }
}

[JacekHoleczek]

Usually, ROOT files are made with an option set that gives the subbranches fully qualified names (...). I think that choice is made by naming a branch with a trailing .: (...)

ROOT never does it by default and it is not mandatory. If you want it, when creating a new branch, for some object which data members will be split into subbranches, you yourself have to explicitly set its name ending with a ".".

I don't know how we would detect whether the . option has been chosen, (...)

That is easy. The name of the "Branch" ("BranchElement") will end with a "." (because it is the original branch name that was set by the user when the branch was created).

{
  TObject o;
  TTree *t = new TTree("t", "t");
  t->Branch("dumb", &o);
  t->Branch("nice.", &o);
  t->Print();
}

[jpivarski]

Thanks! I should have said, "most of the ROOT files that I've encountered."

Since users can set the names of the top-most TBranches (directly below the TTree and not nested under any other TBranch), I suppose that it's a trailing dot (.) on this top-most TBranch, right?

We could, in Uproot, expand nested TBranch names to fully qualified names (by prepending parent names) only if the top-most parent name does not end in a trailing dot. That would effectively make every Pandas DataFrame look the same, whether the trailing dot was used in the ROOT-writing process or not.

[JacekHoleczek]

I suppose that it's a trailing dot (.) on this top-most TBranch, right?

Yes, as shown in my micro example (in my previous post).
Search for (all) occurrences of "dot" in the TTree class description (several methods mention it).

[jpivarski]

(I was checking to see that the top-most branch is indeed special—this isn't something we should look for in the middle of nested TBranches.)

Thanks!