-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
redis connector table name to match exact one key #12464
Comments
In your case, I think this requirement is reasonable, this situation is actually a key as a table, and we can do some work to support this requirement. I thought of two solutions: The first solutionSimilar to Iceberg's metadata file, we can support properties in the Redis table definition file. The property in the properties can be similar to {
"tableName": "...",
"schemaName": "...",
"key": {
"dataFormat": "...",
"fields": [
]
},
"value": {
"dataFormat": "...",
"fields": [
]
},
"properties": {
"tableIsAKey": "true"
}
} Example of the Iceberg metadata file: {
"format-version" : 1,
"schemas" : [ {
"type" : "struct",
"schema-id" : 0,
"fields" : [ {
"id" : 1,
"name" : "ip",
"required" : false,
"type" : "string"
}, {
"id" : 2,
"name" : "report_time",
"required" : false,
"type" : "string"
}]
} ],
"properties" : {
"preserve.snapshot.days" : "3",
"preserve.snapshot.nums" : "10",
"write.format.default" : "parquet",
"write.metadata.delete-after-commit.enabled" : "true",
"...": "..."
},
"...": "..."
} The second solutionIt is necessary to combine the pushdown of the key in this PR: Users can write the following SQL to achieve this requirement (of course the PR needs some minor modifications). select count(1) from "a:sys" where redis_key = "a:sys"; However, I think adding where filter conditions is a bit redundant, because the essence of this requirement is that a key is used as a table, so I prefer the first solution, adapting different scanning rules according to different property. Hi @ebyhr, I would like to hear your suggestions, if possible, I will complete the corresponding development according to the first solution. |
Hi,
I have table_names
redis.table-names=a:app,a:sys
where in redis I have keys
a:app:1
,a:app:2
,a:sys
,by enabling
redis.key-prefix-schema-table=true
,a:sys
can't match any key, buta:app
do.The text was updated successfully, but these errors were encountered: