Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Very slow mseed read time #92

Open
tclements opened this issue Mar 3, 2022 · 0 comments
Open

Very slow mseed read time #92

tclements opened this issue Mar 3, 2022 · 0 comments

Comments

@tclements
Copy link
Contributor

I'm on SeisIO v1.2.1 and Julia 1.7.0

I have a miniseed file with over 3,000 channels. Loading the file on a windows machine with 32GB RAM is crashing with out of memory error. Ubuntu machine with 64 GB RAM can read the file but read_data is using a ton of memory and taking much longer than expected:

using SeisIO
@time read_data("mseed","39894823.ms")
194.283458 seconds (395.64 k allocations: 102.749 GiB, 99.57% gc time)
SeisData with 3192 channels (4 shown)
...

in comparison obspy/libmseed does well with this file

import obspy 
 %timeit obspy.read("39894823.ms")
621 ms ± 2.61 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

I'm having trouble attaching the file - it's larger than the file share limit (25MB).

Here's how I got it (requires AWS credentials):

using AWS
using AWSS3
aws = AWSConfig(region="us-west-2")
s3_get_file(aws,"scedc-pds","event_waveforms/2021/2021_357/39894823.ms","39894823.ms")

or can be downloaded with the awscli (does not require AWS credentials):

aws s3 cp --no-sign-request s3://scedc-pds/event_waveforms/2021/2021_357/39894823.ms .
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant