Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing bwc java.time layer for 6.8 date_index_name ingest processor #58481

Closed
pgomulka opened this issue Jun 24, 2020 · 3 comments
Closed

Missing bwc java.time layer for 6.8 date_index_name ingest processor #58481

pgomulka opened this issue Jun 24, 2020 · 3 comments
Assignees
Labels
>bug :Core/Infra/Core Core issues without another label :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP needs:triage Requires assignment of a team area label Team:Core/Infra Meta label for core/infra team Team:Data Management Meta label for data/management team

Comments

@pgomulka
Copy link
Contributor

pgomulka commented Jun 24, 2020

looks like #34507 was not backported to 6.8
it was meant to be fixed by this #37407 but it missed IndexNameExpresionResolver class. It only fixed this in DateFormat (used by when creating an index name expression, but not resolving. This is used during simulate and on 'phase1' ingestion with index name processor)
it is not possible to use date formats with 8 prefix in date_index_name ingest processors
this works in simulate but does not work with ingest
steps to reproduce

curl --request PUT \
  --url http://localhost:9200/_ingest/pipeline/index_name \
  --header 'authorization: Basic ZWxhc3RpYzpwYXNzd29yZA==' \
  --header 'content-type: application/json' \
  --data '{
  "description": "Pipeline for routing data to specific index",
  "processors": [
    {
      "date_index_name": {
        "field": "my_timestamp",
        "date_rounding": "d",
        "index_name_prefix": "x-",
        "index_name_format": "8YYYY-w"
      }
    }
  ]
}'

simulate - ok

curl --request POST \
  --url http://localhost:9200/_ingest/pipeline/_simulate \
  --header 'authorization: Basic ZWxhc3RpYzpwYXNzd29yZA==' \
  --header 'content-type: application/json' \
  --data '{
  "pipeline": {
    "processors": [
      {
        "pipeline": {
          "name": "index_name"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
 				 "my_timestamp": "2020-08-10T01:01:01.000Z"
      }
    }
  ]
}'

simulate result:

{
  "docs": [
    {
      "doc": {
        "_index": "<x-{2020-33||/d{8YYYY-w|UTC}}>",
        "_type": "_type",
        "_id": "_id",
        "_source": {
          "my_timestamp": "2020-08-10T01:01:01.000Z"
        },
        "_ingest": {
          "timestamp": "2020-06-24T10:53:08.433437Z"
        }
      }
    }
  ]
}

ingestion fail

curl --request PUT \
  --url 'http://localhost:9200/test1/_doc/1?pipeline=index_name' \
  --header 'authorization: Basic ZWxhc3RpYzpwYXNzd29yZA==' \
  --header 'content-type: application/json' \
  --data '{
  "my_timestamp": "2020-08-10T01:01:01.000Z"
}'

ingestion failure result

{
  "error": {
    "root_cause": [
      {
        "type": "parse_exception",
        "reason": "failed to parse date field [2020-33] with format [8YYYY-w]"
      }
    ],
    "type": "parse_exception",
    "reason": "failed to parse date field [2020-33] with format [8YYYY-w]",
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "Parse failure at index [0] of [2020-33]"
    }
  },
  "status": 400
}

it works with joda

curl --request PUT \
  --url http://localhost:9200/_ingest/pipeline/index_name \
  --header 'authorization: Basic ZWxhc3RpYzpwYXNzd29yZA==' \
  --header 'content-type: application/json' \
  --data '{
  "description": "Pipeline for routing data to specific index",
  "processors": [
    {
      "date_index_name": {
        "field": "my_timestamp",
        "date_rounding": "d",
        "index_name_prefix": "x-",
        "index_name_format": "xxxx-w"
      }
    }
  ]
}'

simulate

curl --request POST \
  --url http://localhost:9200/_ingest/pipeline/_simulate \
  --header 'authorization: Basic ZWxhc3RpYzpwYXNzd29yZA==' \
  --header 'content-type: application/json' \
  --data '{
  "pipeline": {
    "processors": [
      {
        "pipeline": {
          "name": "index_name"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
 				 "my_timestamp": "2020-08-10T01:01:01.000Z"
      }
    }
  ]
}'

ingestion ok

curl --request PUT \
  --url 'http://localhost:9200/test1/_doc/1?pipeline=index_name' \
  --header 'authorization: Basic ZWxhc3RpYzpwYXNzd29yZA==' \
  --header 'content-type: application/json' \
  --data '{
  "my_timestamp": "2020-08-10T01:01:01.000Z"
}'

result

{
  "_index": "x-2020-33",
  "_type": "_doc",
  "_id": "1",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 0,
  "_primary_term": 1
}
@pgomulka pgomulka added >bug :Core/Infra/Core Core issues without another label :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP needs:triage Requires assignment of a team area label labels Jun 24, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Ingest)

@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Jun 24, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (:Core/Infra/Core)

@elasticmachine elasticmachine added the Team:Core/Infra Meta label for core/infra team label Jun 24, 2020
@pgomulka pgomulka self-assigned this Jun 24, 2020
pgomulka added a commit that referenced this issue Jul 2, 2020
backports #34507

closes #58481
also closes general parsing issue (no date processors) #58602
@pgomulka
Copy link
Contributor Author

pgomulka commented Jul 6, 2020

closed by #58503

@pgomulka pgomulka closed this as completed Jul 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug :Core/Infra/Core Core issues without another label :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP needs:triage Requires assignment of a team area label Team:Core/Infra Meta label for core/infra team Team:Data Management Meta label for data/management team
Projects
None yet
Development

No branches or pull requests

2 participants