some errors when using "merge_groups"

[zhangnaichi2020]

I encountered two types of error when processing data in bulk with larch in a python way (import modules in jupyter notebook).
I preprocess each spectrum in a way like:
"
from larch.io import read_ascii, merge_groups
from larch.xafs import pre_edge
CaCrO4_0 = read_ascii(./Cr_standards/CaCrO4_Cr_K_180.00000.xdi', labels='enc_energy, mono_energy, mono_bragg, step_pos, mono_x1vert, undulator_energy, undulator_gap, bragg_enc, enc_raw, time_pp, vfc01, vfc02, vfc03, vfc04, t1, t2')
CaCrO4_0.xdat = CaCrO4_0.data[0, :]
CaCrO4_0.ydat = np.log(CaCrO4_0.data[10, :] / CaCrO4_0.data[11, :])
CaCrO4_0.energy = CaCrO4_0.xdat
CaCrO4_0.mu = CaCrO4_0.ydat
pre_edge(CaCrO4_0, pre1=-200.00, pre2=-30.00, nvict=0.00, nnorm=None, norm1=150.00, norm2=-1.00)
"
then did merge_groups and pre_edge:
"
CaCrO4_merge = merge_groups([CaCrO4_0, CaCrO4_1, CaCrO4_2, CaCrO4_3, CaCrO4_4], master=CaCrO4_0, xarray='energy', yarray='mu', kind='cubic', trim=True)
pre_edge(CaCrO4_merge, pre1=-200.00, pre2=-30.00, nvict=0.00, nnorm=None, norm1=150.00, norm2=-1.00)
"
error1 take places in the merge_groups step and reported "ValueError: Expect x to not have duplicates";
some other spectra (like Cr2SO4_3_merge) reported error at pre_edge step with "ValueError: energy array must have at least 2 points", and the results of "Cr2SO4_3_merge.energy" is normal, but the results of "Cr2SO4_3_merge.mu" is "[nan nan nan ... nan nan nan]"

Do anyone know the reason of the two errors and how to solve them?

[newville]

@zhangnaichi2020 Can you provide a few of those data files?

[zhangnaichi2020]

sure, CaCrO4 and Cr_Goehetite reported error "ValueError('Expect x to not have duplicates')"; Cr2SO4_3 and CrCl3_6H2O reports error "ValueError('energy array must have at least 2 points')"
question.zip

[newville]

@zhangnaichi2020 Thanks... will investigate!

[zhangnaichi2020]

Is there any progress on this? I am not stressing at all. I have made effort, but unfortunately, the problem remains unresolved.
If there are any advancements, I would like to express my gratitude in advance~

[newville]

@zhangnaichi2020 I am able to read most of the data you provided with XAS Viewer and Larch 0.9.71.

The main issue you were seeing was due to very closely repeated or exactly repeated energy points. That is fixed in the latest version.

There is an additional problem for some of the datasets that you posted, depending on which channels you use. That is, the column labeled "vtc02" - which I think is meant to be the "Transmission Ion Chamber" -- has values that are exactly zero. I would guess this data should either not be used as transmission data but as fluorescence data (with vtc04 perhaps, but this was not clear to me), or that something went wrong with the data collection.

Basically, if you take mu = -log(itrans / i0), and itrans is zero, then mu is undefined. In fact, based on this data, I changed to code to use a "safe_log()" function that checks for this and puts a very small value in its place. That will allow the log to be taken, but it is not useful as XAS data -- those points would have to be removed. And reading in that data as XAS data will cause problems downstream.

So, I think that upgrading to 0.9.71 should help with reading that data, but I'm not sure there is really much else that we can do with that transmission data that has zeros.

[zhangnaichi2020]

Thank you for your answer. Inspired by your analysis, I discovered that the "ValueError: Expect x to not have duplicates" can be resolved by adding "sort_xafs" before "pre_edge". Additionally, I did not notice any zero values in the vct02 column before.
Thank you very much for your time and assistance~