Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

redis connector table name to match exact one key #12464

Open
DeoLeung opened this issue May 18, 2022 · 1 comment
Open

redis connector table name to match exact one key #12464

DeoLeung opened this issue May 18, 2022 · 1 comment

Comments

@DeoLeung
Copy link

Hi,

I have table_names redis.table-names=a:app,a:sys
where in redis I have keys a:app:1, a:app:2, a:sys,
by enabling redis.key-prefix-schema-table=true, a:sys can't match any key, but a:app do.

select count(1) from "a:app"; -- 2
select count(1) from "a:sys"; -- 0
@polaris6
Copy link
Member

polaris6 commented May 19, 2022

In your case, redis.key-prefix-schema-table=true is set, so Trino scans Redis data via the a:sys:* wildcard, causes a:sys not to be scanned.

I think this requirement is reasonable, this situation is actually a key as a table, and we can do some work to support this requirement. I thought of two solutions:

The first solution

Similar to Iceberg's metadata file, we can support properties in the Redis table definition file. The property in the properties can be similar to tableIsAKey=true, which is not required by default and can be added by the user as needed. In this way, different scanning rules can be adapted according to the property tableIsAKey.
Example of the table definition file:

{
    "tableName": "...",
    "schemaName": "...",
    "key": {
        "dataFormat": "...",
        "fields": [
        ]
    },
    "value": {
        "dataFormat": "...",
        "fields": [
       ]
    },
    "properties": {
        "tableIsAKey": "true"
    }
}

Example of the Iceberg metadata file:

{
  "format-version" : 1,
  "schemas" : [ {
    "type" : "struct",
    "schema-id" : 0,
    "fields" : [ {
      "id" : 1,
      "name" : "ip",
      "required" : false,
      "type" : "string"
    }, {
      "id" : 2,
      "name" : "report_time",
      "required" : false,
      "type" : "string"
    }]
  } ],
  "properties" : {
    "preserve.snapshot.days" : "3",
    "preserve.snapshot.nums" : "10",
    "write.format.default" : "parquet",
    "write.metadata.delete-after-commit.enabled" : "true",
    "...": "..."
  },
  "...": "..."
}

The second solution

It is necessary to combine the pushdown of the key in this PR:
#12219

Users can write the following SQL to achieve this requirement (of course the PR needs some minor modifications).

select count(1) from "a:sys" where redis_key = "a:sys";

However, I think adding where filter conditions is a bit redundant, because the essence of this requirement is that a key is used as a table, so I prefer the first solution, adapting different scanning rules according to different property.

Hi @ebyhr, I would like to hear your suggestions, if possible, I will complete the corresponding development according to the first solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants