Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove strftime specifiers from the templating syntax #6254

Open
binarylogic opened this issue Jan 27, 2021 · 1 comment
Open

Remove strftime specifiers from the templating syntax #6254

binarylogic opened this issue Jan 27, 2021 · 1 comment
Labels
domain: config Anything related to configuring Vector domain: templating Anything related to templating Vector's configuration values type: enhancement A value-adding code change that enhances its existing functionality.

Comments

@binarylogic
Copy link
Contributor

The current support for strftime specifiers in the templating syntax was a quick hack to allow for date-based partitioning:

[sinks.backup]
  type = "aws_s3"
  bucket = "all_application_logs"
  key_prefix = "application_id={{ application_id }}/date=%F/"

The obvious limitation of this is the assumption that the timestamp field should be used to derive their formatted dates. What if a user wants to use another field?

Given that we decided against supporting the Remap syntax in #3836, we should do 1 of 2 things:

  1. Remove support for the strftime specifiers and require users to format dates in a remap transform.
  2. Reopen the discussion of supporting Remap functions in the templating syntax.

I lean towards 1 until we see more reasons to do 2.

@binarylogic binarylogic added type: enhancement A value-adding code change that enhances its existing functionality. domain: config Anything related to configuring Vector domain: vrl Anything related to the Vector Remap Language domain: templating Anything related to templating Vector's configuration values labels Jan 27, 2021
@jszwedko
Copy link
Member

jszwedko commented Feb 1, 2021

I think we need to address the metadata vs. event field separation before moving towards solution 1 here as otherwise it'll end up requiring users to write a "dummy key" in the remap transform that they probably will not want to actually appear in the event when it is written to S3 (in this example)

@JeanMertz JeanMertz removed the domain: vrl Anything related to the Vector Remap Language label Jun 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: config Anything related to configuring Vector domain: templating Anything related to templating Vector's configuration values type: enhancement A value-adding code change that enhances its existing functionality.
Projects
None yet
Development

No branches or pull requests

3 participants