Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow "tsdb import" to read from STDIN #580

Merged
merged 1 commit into from
Mar 23, 2016
Merged

Conversation

hynd
Copy link
Contributor

@hynd hynd commented Oct 6, 2015

To facilitate streaming data from one OpenTSDB instance to another, avoiding intermediate files. If the import "path" is "-", it'll read from STDIN instead.

For example, something like this for migrating between two configs (with different salting or byte width configs etc):

tsdb scan --config=old.conf 1y-ago sum my.metric | tsdb import --skip-errors -- -

There may already be another way to achieve this?

A very quick check of the above got ~70k/s on an anaemic VM backed by a standalone Hbase (~85k unique metrics out of the 999981):

2015-10-06 21:23:42,263 INFO  [main] [TextImporter.importFile]: Processed - in 14855 ms, 999981 data points (67316.1 points/s)
2015-10-06 21:23:42,264 INFO  [main] [TextImporter.main]: Total: imported 999981 data points in 14.861s (67287.2 points/s)

(i wondered if TextImporter could also leverage aec67bf to improve its speed?)

@hynd
Copy link
Contributor Author

hynd commented Feb 9, 2016

Hey @manolama - what are the chances of this getting wangled into 2.2? Might help people with migrating data after upgrades...

@johann8384
Copy link
Member

Hey @manolama - perhaps this can go into 2.3?

@johann8384 johann8384 added this to the v2.3.0 milestone Mar 19, 2016
@johann8384 johann8384 merged commit c2d26cb into OpenTSDB:next Mar 23, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants