Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

QL Specta: duplicated time bin with different data removed in add():unique #269

Open
nicHoch opened this issue Aug 25, 2022 · 1 comment
Labels
bug Something isn't working

Comments

@nicHoch
Copy link
Collaborator

nicHoch commented Aug 25, 2022

when processing this files:

32.zip

the pipeline will add data to previously created FITS files: 'solo_L0_stix-ql-spectra_0678153600_V01.fits'

in the add() method of the product one time bin gets removed as it is very very close (8binary seconds) to an existing one.
But the data (spectra, trigger, detector_index) within this "duplicated" time bins is very different.

needs to be investigated where this double time bin comes from and why it has different data.

was discovered during end2end testing fixes in: https://github.com/nicHoch/STIXCore/tree/end2end_fix
with forced IDB version 35

@nicHoch
Copy link
Collaborator Author

nicHoch commented Aug 25, 2022

records_array  = np.around(data['time'].as_float().value, 2)


# creates an array of indices, sorted by unique element
idx_sort = np.argsort(records_array)

# sorts records array so all unique elements are together 
sorted_records_array = records_array[idx_sort]

# returns the unique values, the index of the first occurrence of a value, and the count for each element
vals, idx_start, count = np.unique(sorted_records_array, return_counts=True, return_index=True)

# splits the indices into separate arrays
res = np.split(idx_sort, idx_start[1:])

#filter them with respect to their size, keeping only items occurring more than once
vals = vals[count > 1]
res = np.where(records_array == vals[0])

this will help to find the "almost identical" time bins

@samaloney samaloney added the bug Something isn't working label Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants