Add support for glob/regex filtering on NestedAsset list endpoint

[mvandenburgh]

Closes #1002.

Note that Postgres doesn't support querying with globs, but it does support querying with a regex. So, the approach I've taken to support globbing here is to convert the glob pattern to a regex pattern and then use that in the query. Python has a built-in function (fnmatch.translate) that converts a glob expression to a regex, but the regex syntax it uses is incompatible with Postgres's regex syntax. I found this 81-line library that does work with Postgres, so I've used that here.

[jjnesbitt]

Python has a built-in function (fnmatch.translate) that converts a glob expression to a regex, but the regex syntax it uses is incompatible with Postgres's regex syntax.

What exactly from the translation is incompatible with Postgres? I'd hope that any differences are minor and could be addressed directly. I'd rather not pull in an obscure 3rd party library if it's not necessary.

[mvandenburgh]

Python has a built-in function (fnmatch.translate) that converts a glob expression to a regex, but the regex syntax it uses is incompatible with Postgres's regex syntax.

What exactly from the translation is incompatible with Postgres? I'd hope that any differences are minor and could be addressed directly. I'd rather not pull in an obscure 3rd party library if it's not necessary.

fnmatch.translate('*.json') returns (?s:.*\.json)\Z while Postgres expects something like ^.*\.json$. I'm unsure of the differences between the two, although I've never seen the ?s \Z syntax from the first one before. I do know that support for regex varies across different languages. For example, apparently Python and JavaScript have slight differences in their regex syntaxes that make them incompatible. There's a surprisingly lack of information online about how to translate between the fnmatch format and postgres/Django format :(. I'll take another look

[jjnesbitt]

fnmatch.translate('*.json') returns (?s:.*\.json)\Z while Postgres expects something like ^.*\.json$. I'm unsure of the differences between the two, although I've never seen the ?s \Z syntax from the first one before. I do know that support for regex varies across different languages. For example, apparently Python and JavaScript have slight differences in their regex syntaxes that make them incompatible. There's a surprisingly lack of information online about how to translate between the fnmatch format and postgres/Django format :(. I'll take another look

It seems to me a lot of this has to do with regex modes and differences with chars used. For example, the ?:s is the python regex mode for include newlines in ., which doesn't really apply to us. Toying around with it for a bit, I got this regex replacement which should solve most of our problems

re.sub(r'\(\?s?m?:([^\)]+)\)\\Z', r'^\1$', glob)

This essentially removes the (?s:) and \\Z part from the string. It also catches the m (multiline) modifier).

[waxlamp]

fnmatch.translate('*.json') returns (?s:.*\.json)\Z while Postgres expects something like ^.*\.json$. I'm unsure of the differences between the two, although I've never seen the ?s \Z syntax from the first one before. I do know that support for regex varies across different languages. For example, apparently Python and JavaScript have slight differences in their regex syntaxes that make them incompatible. There's a surprisingly lack of information online about how to translate between the fnmatch format and postgres/Django format :(. I'll take another look

It seems to me a lot of this has to do with regex modes and differences with chars used. For example, the ?:s is the python regex mode for include newlines in ., which doesn't really apply to us. Toying around with it for a bit, I got this regex replacement which should solve most of our problems

re.sub(r'\(\?s?m?:([^\)]+)\)\\Z', r'^\1$', glob)

This essentially removes the (?s:) and \\Z part from the string. It also catches the m (multiline) modifier).

I don't love either including that library (even if it seems to be harmless) or this homebrew regex (even if we could write tests for it). The docs for fnmatch.translate imply that the globbing star is just converted to .*, while regular dots are converted to \.. (Not sure what happens to **, but we can defer that functionality for now.) Could we just do that?

[jjnesbitt]

I don't love either including that library (even if it seems to be harmless) or this homebrew regex (even if we could write tests for it). The docs for fnmatch.translate imply that the globbing star is just converted to .*, while regular dots are converted to \.. (Not sure what happens to **, but we can defer that functionality for now.) Could we just do that?

I think it's worth testing out at least. For the record, ** isn't handled by fnmatch, that's only handled by the glob module.

[mvandenburgh]

Agreed that @waxlamp's solution is cleaner. I tested it and it seems to work perfectly, I'll update this to do it that way 👍

[jjnesbitt]

Just one request otherwise looks good

[jjnesbitt]

@mvandenburgh it seems the test_zarr_rest_list_filter test failed? Have you seen that for yourself or is that just a fluke?

[mvandenburgh]

@mvandenburgh it seems the test_zarr_rest_list_filter test failed? Have you seen that for yourself or is that just a fluke?

I assumed it was a fluke because I'm not seeing it when I run tests locally, but it happened again when I reran the job 😕

[dandibot]

🚀 PR was released in v0.2.13 🚀