Load track parameter constraints from external resource files

[pbutti]

So, basically in order to complete some sort of constrained alignment, we should load track parameter constraints from external sources. For example, if we want to correct p vs tanL trend, we could load the EoP correction binned in tanLambda.

We need:

• A way to read in a file containing the bins in a variable
• Apply the bin-by-bin correction to each track that falls in that bin.

The easiest idea is to use .txt files or .xml files to be loaded in java. Other suggestions?
@omar-moreno @normangraf @tomeichlersmith

Cheers,
PF

[bloodyyugo]

I have always just used a text file...

[tomeichlersmith]

Does java have an good xml parser? I just feel like a more rigid format will make it easier to maintain.

[pbutti]

I've written a basic JNA accessor to the root library.
https://github.com/pbutti/rootjna

This has the C wrappers to open TFiles and load histograms in memory. I can augment those by adding the GetBin and GetBinContent which should be everything we need to load corrections.

In my view this might be a nice tool.

[normangraf]

There is a native Java interface for reading simple root files available in freehep. I've not used it explicitly in a while, so you would have to check whether it works with the latest root files.

Details at java.freehep.org/lib/freehep/doc/root/index.html

[pbutti]

Hi Norman,

it doesn't work unfortunately. I found it yesterday and tried it before developing my own. I'm not able to run the basic example as it crashes when opening a file. It exists also a JNI implementation called jroot, but doesn't compile even if reverting to Java SDK 8 (the one used to compile the repo)

[pbutti]

I'm able to open / close a file and to read in an histogram and it's content.

org.hps.detector.svt:INFO loading default SVT conditions onto subdetector Tracker
org.hps.detector.svt:INFO setting up 40 SVT sensors
org.hps.detector.svt:INFO channel map has 24576 entries
org.hps.conditions.database:INFO Closing database connection.
org.hps.conditions.database:CONFIG Conditions system is now frozen and will not accept updates to detector or run.
org.hps.recon.ecal:CONFIG minLookbackEvents: 5
org.hps.recon.ecal:CONFIG maxLookbackEvents: 40
org.hps.recon.ecal:CONFIG maxLookbackTime:-1
PF::Opening File
TFile**		Example.root	
 TFile*		Example.root	
  KEY: TH1D	mainHistogram;1	mainHistogram
CHECK CHECK CHECK
GetEntries::1
Find Bin::11
Get BinContent of Bin 0.1::0.0
Find Bin::10
Get BinContent of Bin 0.1::100.0
org.hps.conditions.database:INFO Opening conditions db connection ...
url: jdbc:mysql://hpsdb.jlab.org:3306/hps_conditions
user: hpsuser
org.hps.conditions.database:INFO loading conditions set...
id: 3322
name: ecal_channels

I'll develop track parameter dependent constraints using this tool. If there is interest to add it to hps-java I can open a PR similar to what was done for accessing GBL via JNA.

[pbutti]

adding @cbravo135 as SW coord

[omar-moreno]

Does java have an good xml parser? I just feel like a more rigid format will make it easier to maintain.

Java does have a decent XML parser (JDOM Parser, StAX Parser) so this may be the simplest solution. However, being able to use 3D histograms and possibly perform some basic fits within hps-java makes the JNA/JNI solution much more attractive.

[omar-moreno]

My vote is for JNA/JNI.

[normangraf]

I would much prefer to get any needed corrections directly from the data on a track-by-track basis instead of introducing essentially arbitrary modifications.

[normangraf]

If we do decide to introduce a JNA/JNI solution, it should be off by default so we do not introduce any additional dependencies into our hps-java build system.

[pbutti]

Hi @normangraf,
These are 2 very fair comments.
The usage of Root histograms will allow us to store track-by-track corrections into histograms, retrieve them and apply them to track fits. I'm unsure which arbitrary modifications you are referring to. Perhaps you refer to fit those shapes and store the fit parameters that might feel hard to then visualize?

We already have JNA dependence from the GBL library, which is OFF by default. This will be introduced with the same spirit: targeted to alignment and off by default. I think this could be a good solution.

@omar-moreno
As we were discussing yesterday, I agree that dumping histograms into XML and parsing them back could be a good solution. However we can always extend the wrappers to add some other basic functionalities (if needed) while with XML I think we are pretty limited on what we can do.

Anyway this is just a suggestion. If this doesn't make it to master, I will still make an "alignment branch" with this in. I think Cam should have the final say on this as SW coordinator.

[normangraf]

Please proceed as this will provide new functionality that might be used in other ways. (I'll look into the freehep issue if I can ever find some "spare" time.) And, as you note, we already have the external dependencies as options. My concern was that sampling, for instance, an E/p correction from a distribution derived from an ensemble of tracks would not be as specific as that obtained from the energy of the cluster associated with that particular track.

[pbutti]

Hi Norman,
sounds good. If we manage to fix freehep that would be nice.

My idea is to use <E/p> for positive and negative tracks to correct the track fit and modify the track-by-track Chi2 instead of simply E/p on track-by-track basis.

Using the average, binned in phi/tanL, should give enough granularity to fix angle dependent biases and the advantage would be a stronger statistical power.
In principle using the average should reduce the track-by-track fluctuations and avoid applying possibly large biases to some particular tracks..

However, I am struggling to add additional motivations to why to prefer that method over just using E/p on track-by-track basis. I suppose that over a large sample of tracks, results would be the same.

[normangraf]

You might want to look at this package: https://github.com/chekanov/RootIOreader .

[normangraf]

I'm still not sure why you don't just read in some reference histograms and use them directly.

[pbutti]

Sorry, I missed your comment Norman.
I am not able to store 3D plots into Root so I can't create reference histograms..

[pbutti]

Regarding the package you pointed me to. In the readme there is written:
"This program shows how to read ROOT files (https://root.cern/) using Java. It illustrates problems with the compatibility of JDK versions. This example shows that compiling this code using JDK 1.5 creates a functional package that can read the file "Example.root". The JDK 1.5 compiled package can be executed on new versions of JDK 1.8 - JDK 15. However, the package compiled using JDK 1.6 (and above) creates a jar file that cannot be used for reading such files."

I am not sure if the scope of this package is to illustrate a compatibility issue. I went through the bug report and it's not clear to me if the package is functional.