Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add unit conversion ingest node processor #31737

Closed
inqueue opened this issue Jul 2, 2018 · 6 comments
Closed

Add unit conversion ingest node processor #31737

inqueue opened this issue Jul 2, 2018 · 6 comments
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >enhancement Team:Data Management Meta label for data/management team

Comments

@inqueue
Copy link
Member

inqueue commented Jul 2, 2018

Describe the feature:
Working with Filebeat modules, I have come across several instances where I need to convert seconds to millis and kilobytes/megabytes/gigabytes to bytes. While the script processor is available to perform such unit conversions, a dedicated processor for this task would facilitate the development of modules greatly by eliminating the need to write and rewrite them in painless. It might look something like this:

{
  "units": {
    "field": "duration_sec",
    "from": "seconds",
    "to": "milliseconds"
  }
}

Then use rename to adjust the field name accordingly:

{
  "rename": {
    "field": "duration_sec",
    "target_field": "duration_ms"
  }
}

Even better might a processor that accepts a conversion factor or possibly formula so it could be applied to any unit of measure; e.g., temperature, speed, energy, leagues, etc.

{
  "units": {
    "field": "duration_sec",
    "factor": .001,
    "target_field": "duration_ms"
  }
}
@inqueue inqueue added >enhancement :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP labels Jul 2, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra

@inqueue
Copy link
Member Author

inqueue commented Jul 2, 2018

Relates to #31244.

@jakelandis
Copy link
Contributor

jakelandis commented Jul 2, 2018

Would you expect all these types to need to convert numeric type to numeric type ?

Even better might a processor that accepts a conversion factor or possibly formula

That sounds alot like a general purpose math processor. Would something like this satisfy your use case(s) ?

{
  "math": {
    "field": "duration_sec",
    "operation" : "multiply"
    "by": 1000,
    "target_field": "duration_ms"
  }
}

It could be expanded (if needed) for user defined functions.

related: I just submitted a PR for converting human readable byte string to their byte value. (e.g 1kb -> 1024)

@rjernst
Copy link
Member

rjernst commented Jul 2, 2018

While the script processor is available to perform such unit conversions, a dedicated processor for this task would facilitate the development of modules greatly by eliminating the need to write and rewrite them in painless

Can you show what a script doing conversion looks like, for comparison against your other suggested examples here?

@ruflin
Copy link
Member

ruflin commented Jul 3, 2018

Linking #31244 here as in case we have such a processor, it should work well together with a potential duration type in this example.

@dakrone
Copy link
Member

dakrone commented May 8, 2024

This has been open for quite a while, and we haven't made much progress on this due to focus in other areas. For now I'm going to close this as something we aren't planning on implementing. We can re-open it later if needed.

@dakrone dakrone closed this as not planned Won't fix, can't repro, duplicate, stale May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >enhancement Team:Data Management Meta label for data/management team
Projects
None yet
Development

No branches or pull requests

6 participants