Failing Migration: Drop (project, key, value) index on EventTag

[robinboening]

I was just updating sentry from v8.20.0 to current master. I am deploying to heroku.

Migrations from 0352 on were starting to run. When it came to 0365 it raised following error:

remote:  > sentry:0364_backfill_grouplink_from_groupcommitresolution
remote:  > sentry:0365_auto__del_index_eventtag_project_id_key_id_value_id
remote: FATAL ERROR - The following SQL query failed: DROP INDEX CONCURRENTLY sentry_eventtag_project_id_183f8a8ce88d19cb
remote: The error was: index "sentry_eventtag_project_id_183f8a8ce88d19cb" does not exist

I checked the database and the index sentry_eventtag_project_id_183f8a8ce88d19cb indeed does not exist, but instead its called sentry_eventtag_project_id_42979ba214ba3c43 and is for exactly the same columns (project_id, key_id, value_id).

I don't know how migrations in django work, so I don't really know how to debug further without getting into django more.

Is this an issue with the migration itself or how can I fix the issue for me?

Thanks for any help!

[mattrobenolt]

Hmmm. Weird. I’m not sure how that would have happened. Is your install really old by any chance? The easiest fix would be to just rename the current index.

ALTER INDEX sentry_eventtag_project_id_42979ba214ba3c43 RENAME TO sentry_eventtag_project_id_183f8a8ce88d19cb;

Should be sufficient to get it moving along so it can continue.

[robinboening]

Thanks for the quick response, @mattrobenolt.

I did what you suggested, but after running the migration again it tries to find the Index under a different name... every time I am running this migration it will change the hash in the name again:

remote: Running migrations for sentry:
remote:  - Migrating forwards to 0367_auto__chg_field_release_ref__chg_field_release_version.
remote:  > sentry:0365_auto__del_index_eventtag_project_id_key_id_value_id
remote: FATAL ERROR - The following SQL query failed: DROP INDEX CONCURRENTLY sentry_eventtag_project_id_51ff0f5e5f201c15
remote: The error was: index "sentry_eventtag_project_id_51ff0f5e5f201c15" does not exist

For me as a Python/Django/South dummy it looks as if its created on runtime due to this command:

...
db.create_index_name(u'sentry_eventtag', ['project_id', 'key_id', 'value_id'])
...

https://github.com/getsentry/sentry/blob/master/src/sentry/south_migrations/0365_auto__del_index_eventtag_project_id_key_id_value_id.py#L24

[mattrobenolt]

Umm, that should be computed deterministically.

$ sentry shell
Python 2.7.14 (default, Oct 13 2017, 19:36:01)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from south.db import db
>>> db.create_index_name(u'sentry_eventtag', ['project_id', 'key_id', 'value_id'])
u'sentry_eventtag_project_id_42979ba214ba3c43'
>>> db.create_index_name(u'sentry_eventtag', ['project_id', 'key_id', 'value_id'])
u'sentry_eventtag_project_id_42979ba214ba3c43'
>>> db.create_index_name(u'sentry_eventtag', ['project_id', 'key_id', 'value_id'])
u'sentry_eventtag_project_id_42979ba214ba3c43'
>>> db.create_index_name(u'sentry_eventtag', ['project_id', 'key_id', 'value_id'])
u'sentry_eventtag_project_id_42979ba214ba3c43'
>>> db.create_index_name(u'sentry_eventtag', ['project_id', 'key_id', 'value_id'])
u'sentry_eventtag_project_id_42979ba214ba3c43'
>>> db.create_index_name(u'sentry_eventtag', ['project_id', 'key_id', 'value_id'])
u'sentry_eventtag_project_id_42979ba214ba3c43'

So I have no idea why this would be problematic for you or why this behavior is different. I could suggest a fix to get past this, but I can only assume you're going to hit this problem again in the future. This is how lots of things are managed, and if somehow you have the wrong names being generated... any mutation depending on the auto generated names won't work. They need to be deterministic for most things to work.

If you open up sentry shell and try what I did, what do you get back?

[robinboening]

Thanks again.

I opened the sentry shell and did what you said.

Yes I am seeing a deterministic behaviour as long as I am in the same shell session. After closing the shell and starting again I get a new hash:

$ heroku run "sentry --config=sentry.conf.py shell"                                                                                                                            
Python 2.7.12 (default, Aug  3 2017, 22:43:55)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from south.db import db
>>> db.create_index_name(u'sentry_eventtag', ['project_id', 'key_id', 'value_id'])
u'sentry_eventtag_project_id_22bb94b22f303ead'
>>> db.create_index_name(u'sentry_eventtag', ['project_id', 'key_id', 'value_id'])
u'sentry_eventtag_project_id_22bb94b22f303ead'
>>> db.create_index_name(u'sentry_eventtag', ['project_id', 'key_id', 'value_id'])
u'sentry_eventtag_project_id_22bb94b22f303ead'

And then...

$ heroku run "sentry --config=sentry.conf.py shell"                                                                                                                            
Python 2.7.12 (default, Aug  3 2017, 22:43:55)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from south.db import db
>>> db.create_index_name(u'sentry_eventtag', ['project_id', 'key_id', 'value_id'])
u'sentry_eventtag_project_id_4d37cbda454695df'
>>> db.create_index_name(u'sentry_eventtag', ['project_id', 'key_id', 'value_id'])
u'sentry_eventtag_project_id_4d37cbda454695df'

Deploying this to heroku means I get a new hash on every deployment.

[mattrobenolt]

What's really odd is that the index you have in your database was named correctly and matches with what I'm generating locally. I'll dig into the South code to see how this is generated now because I'm curious. But there's definitely something wrong with what you're doing, I just don't know what.

[robinboening]

Interesting, in your shell you got the hash I mentioned in the beginning that I had in the database before I renamed it.

[robinboening]

Maybe important to know about my setup:

I am moving a sentry installation (v8.20.0) from a self hosted server to heroku. So I copied the database (pg_dump) from the old machine to heroku and deploy the current sentry master (I am not deploying the exact same version because of a different story with sentry-auth-google: getsentry/sentry-auth-google#23).