-
Notifications
You must be signed in to change notification settings - Fork 3.4k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* basics working, needs some more cleanup and more flags for config and a Dockerfile Signed-off-by: Edward Welch <edward.welch@grafana.com> * add label matchers start end time params and ability to remap the chunk to a different user Signed-off-by: Edward Welch <edward.welch@grafana.com> * adding dockerfile and makefile instructions Signed-off-by: Edward Welch <edward.welch@grafana.com> * push out some of the limits show errors from index query * for testing have docker container not run migrate command but instead stay running by tailing /dev/null * big overhaul to add parallelization * adding better stats * adding retry logic * tweaking retry logic * tweaking retry logic * work around changes in latest cortex Signed-off-by: Ed Welch <edward.welch@grafana.com> * fix up imports and changes since the last time this was merged added a readme fixed up the makefile added some comments * fixing some of my garbage * fixing more of my garbage * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update cmd/migrate/README.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * tabs not spaces Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
- Loading branch information
Showing
7 changed files
with
610 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
FROM golang:1.15.3 as build | ||
COPY . /src/loki | ||
WORKDIR /src/loki | ||
RUN make clean && make BUILD_IN_CONTAINER=false migrate | ||
|
||
FROM alpine:3.9 | ||
RUN apk add --update --no-cache ca-certificates | ||
COPY --from=build /src/loki/cmd/migrate/migrate /usr/bin/migrate | ||
#ENTRYPOINT [ "/usr/bin/migrate" ] | ||
CMD tail -f /dev/null |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,56 @@ | ||
# Loki Migrate Tool | ||
|
||
**WARNING: THIS TOOL IS NOT WELL TESTED, ALWAYS MAKE BACKUPS AND TEST ON LESS IMPORTANT DATA FIRST!** | ||
|
||
This is sort of a bare minimum code hooked directly into the store interfaces within Loki. | ||
|
||
Two stores are created, a source store and destination (abbreviated dest) store. | ||
|
||
Chunks are queried from the source store and written to the dest store, new index entries are created in the dest store as well. | ||
|
||
This _should_ handle schema changes and different schemas on both the source and dest store. | ||
|
||
You should be able to: | ||
|
||
* Migrate between clusters | ||
* Change tenant ID during migration | ||
* Migrate data between schemas | ||
|
||
All data is read and re-written (even when migrating within the same cluster). There are really no optimizations in this code for performance and there are much faster ways to move data depending on what you want to change. | ||
|
||
This is simple and because it uses the storage interfaces, should be complete and should stay working, but it's not optimized to be fast. | ||
|
||
There is however some parallelism built in and there are a few flags to tune this, `migrate -help` for more info | ||
|
||
This does not remove any source data, it only reads existing source data and writes to the destination. | ||
|
||
## Usage | ||
|
||
Build with | ||
|
||
``` | ||
make migrate | ||
``` | ||
|
||
or | ||
|
||
``` | ||
make migrate-image | ||
``` | ||
|
||
The docker image currently runs and doesn't do anything, it's intended you exec into the container and run commands manually. | ||
|
||
|
||
### Examples | ||
|
||
Migrate between clusters | ||
|
||
``` | ||
migrate -source.config.file=/etc/loki-us-west1/config/config.yaml -dest.config.file=/etc/loki-us-central1/config/config.yaml -source.tenant=2289 -dest.tenant=2289 -from=2020-06-16T14:00:00-00:00 -to=2020-07-01T00:00:00-00:00 | ||
``` | ||
|
||
Migrate tenant ID within a cluster | ||
|
||
``` | ||
migrate -source.config.file=/etc/loki-us-west1/config/config.yaml -dest.config.file=/etc/loki-us-west1/config/config.yaml -source.tenant=fake -dest.tenant=1 -from=2020-06-16T14:00:00-00:00 -to=2020-07-01T00:00:00-00:00 | ||
``` |
Oops, something went wrong.