Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] SQL query doesn't support different date formats without date casting #2700

Open
rupal-bq opened this issue May 30, 2024 · 1 comment
Labels
bug Something isn't working v2.16.0 Issues targeting release v2.16.0

Comments

@rupal-bq
Copy link
Member

What is the bug?
Getting below error for user defined date format

"{\n  \"error\": {\n    \"reason\": \"Invalid SQL query\",\n    \"details\": \"date:03-Jan-24 in unsupported format, please use 'yyyy-MM-dd'\",\n    \"type\": \"SemanticCheckException\"\n  },\n  \"status\": 400\n}"

How can one reproduce the bug?
Steps to reproduce the behavior:

PUT testindex
{
  "mappings" : {
    "properties" :  {
      "release_date" : {
        "type" : "date",
        "format" : "dd-MMM-yy"
      }
    }
  }
}
PUT testindex/_doc/21 
{
  "release_date" : "03-Jan-24"
}
POST _plugins/_sql
{
"query":"select release_date from testindex where release_date='03-Jan-24'"
}

What is the expected behavior?
Query should work without date casting

What is your host/environment?

  • OS: [e.g. iOS]
  • Version [e.g. 22]
  • Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Add any other context about the problem.

@rupal-bq rupal-bq added bug Something isn't working untriaged labels May 30, 2024
@rupal-bq rupal-bq changed the title [BUG] SQL query doesn't allow different date formats without date casting [BUG] SQL query doesn't support different date formats without date casting May 30, 2024
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jun 18, 2024
…asting

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
@manasvinibs
Copy link
Member

We have one more similar issue reported from users when date in timestamp format in the SQL query is erroring out due to mismatch in the timestamp format between field mapping and the request query -

Sample query:

select count(*) from ami where State = 'available' and  Public = 'true' and CallerVersion = "2016-11-15 00:00:00";
select count(*) from ami where State = 'available' and  Public = 'true' and CallerVersion < "2016-11-15 00:00:00";

Error -

[INFO] 10f0cef3-4159-4c39-84e1-f4ba8b88be97 (Bobcat-29) com.amazonaws.fleetinsightsopensearchcommons.helper.ResponseConvertor: Original SQL response : {
  "error": {
    "reason": "Error occurred in OpenSearch engine: all shards failed",
    "details": "Shard[0]: OpenSearchParseException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]: [failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]]; nested: IllegalArgumentException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];\nShard[1]: OpenSearchParseException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]: [failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]]; nested: IllegalArgumentException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];\nShard[2]: OpenSearchParseException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]: [failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]]; nested: IllegalArgumentException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];\nShard[3]: OpenSearchParseException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]: [failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]]; nested: IllegalArgumentException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];\nShard[4]: OpenSearchParseException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]: [failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]]; nested: IllegalArgumentException[failed to parse date field [2016-11-15 00:00:00] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];\n\nFor more details, please send request for Json format to see the raw response from OpenSearch engine.",
    "type": "SearchPhaseExecutionException"
  },
  "status": 400
}

Root cause -
Currently for response JDBC format, when term/range query has timestamp field in the sql query, by default we are converting timestamps to epoch time in milliseconds when building query for Opensearch DSL which is causing the above exceptions. Instead for timestamp query string in epoch, we should send parsed formatted timestamp string as it is as part of the query similar to Date format.

Fixing this issue as part of this PR #2765

@manasvinibs manasvinibs added v2.16.0 Issues targeting release v2.16.0 and removed untriaged labels Jun 20, 2024
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jun 23, 2024
…asting

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 8, 2024
…asting

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 8, 2024
…e fields as part of Lucene query

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 9, 2024
…e fields as part of Lucene query

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 9, 2024
…e fields as part of Lucene query

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 9, 2024
…e fields as part of Lucene query

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 9, 2024
…e fields as part of Lucene query

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 11, 2024
…e fields as part of Lucene query

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 15, 2024
…e fields as part of Lucene query

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 15, 2024
…e fields as part of Lucene query

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 16, 2024
…e fields as part of Lucene query

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 16, 2024
…e fields as part of Lucene query

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 16, 2024
…e fields as part of Lucene query

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 16, 2024
…e fields as part of Lucene query

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 17, 2024
…e fields as part of Lucene query

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 17, 2024
…e fields as part of Lucene query

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 17, 2024
…e fields as part of Lucene query

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 18, 2024
…e fields as part of Lucene query

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 18, 2024
…e fields as part of Lucene query

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
penghuo pushed a commit that referenced this issue Jul 19, 2024
…e fields as part of Lucene query (#2762)

Github Issue - #2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 22, 2024
…e fields as part of Lucene query (opensearch-project#2762)

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
(cherry picked from commit 0fad56d)
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 22, 2024
…e fields as part of Lucene query (opensearch-project#2762)

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
(cherry picked from commit 0fad56d)
Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 23, 2024
…e fields as part of Lucene query (opensearch-project#2762)

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
(cherry picked from commit 0fad56d)
Signed-off-by: Manasvini B S <manasvis@amazon.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Jul 23, 2024
…e fields as part of Lucene query (opensearch-project#2762)

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
(cherry picked from commit 0fad56d)
Signed-off-by: Manasvini B S <manasvis@amazon.com>
ykmr1224 pushed a commit that referenced this issue Jul 23, 2024
…e fields as part of Lucene query (#2762) (#2849)

Github Issue - #2700


(cherry picked from commit 0fad56d)

Signed-off-by: Manasvini B S <manasvis@amazon.com>
opensearch-trigger-bot bot pushed a commit that referenced this issue Jul 23, 2024
…e fields as part of Lucene query (#2762) (#2849)

Github Issue - #2700

(cherry picked from commit 0fad56d)

Signed-off-by: Manasvini B S <manasvis@amazon.com>
(cherry picked from commit 02d57e0)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
ykmr1224 pushed a commit that referenced this issue Jul 24, 2024
…e fields as part of Lucene query (#2762) (#2849) (#2851)

Github Issue - #2700

(cherry picked from commit 0fad56d)


(cherry picked from commit 02d57e0)

Signed-off-by: Manasvini B S <manasvis@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
manasvinibs added a commit to manasvinibs/sql that referenced this issue Aug 14, 2024
…e fields as part of Lucene query (opensearch-project#2762)

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
jzonthemtn pushed a commit to jzonthemtn/sql that referenced this issue Aug 28, 2024
…e fields as part of Lucene query (opensearch-project#2762)

Github Issue - opensearch-project#2700

Signed-off-by: Manasvini B S <manasvis@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working v2.16.0 Issues targeting release v2.16.0
Projects
None yet
Development

No branches or pull requests

2 participants