Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: unexpected output when using an unknown property key in sdk.record.format.options #216

Open
alarbada opened this issue Nov 26, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@alarbada
Copy link
Contributor

Bug description

When specifying a property that doesn't exist in the passed record, conduit writes the whole record instead of erroring.

Steps to reproduce

Running this pipeline configuration should be enough to reproduce the error:

version: "2.2"
pipelines:
  - id: example-pipeline
    status: running
    name: "s3-to-file"
    connectors:
      - id: example
        type: source
        plugin: generator
        settings:
          rate: 1
          format.type: structured
          format.options.id: int
          format.options.name: string
          operations: create

      - id: file
        type: destination
        plugin: "file"
        settings:
          path: ./destination.txt
          sdk.record.format: template

          # This works as expected
          sdk.record.format.options: '
            {{ printf "%v" .Payload.After }}
          '

          # This makes the pipeline fail
          sdk.record.format.options: '
            {{ printif "%v" .Payload.After }}
          '

          # But this doesn't.
          sdk.record.format.options: '
            {{ printf "%v" .Payload.Afterrrr }}
          '
          
          # And it doesn't matter whether there was previous output written or
          # not, the whole record is still written:
          sdk.record.format.options: '
            {{ printf "%v" .Key }}
            {{ printf "%v" .SomePropThatDoesNotExist }}
          '

          # Not using yaml multilines, and the issue still happens
          sdk.record.format.options: '{{ printf "%v" .SomePropThatDoesNotExist }}'

Version

v0.12.2 linux/amd64

@alarbada alarbada added bug Something isn't working triage Needs to be triaged labels Nov 26, 2024
@alarbada
Copy link
Contributor Author

I have to note that because I was writing to a file, it was fairly easy to notice this behaviour. But if I was to write to another destination the errors (if any) would be much more difficult to track, since the destination doesn't output a warning or anything.

I don't know if this is really what we want, but at the very least there should a single warning (not per record, otherwise there could be many logs) telling me that there was an error executing the template.

I must note, that by default go templates do error when a property is not defined:
https://go.dev/play/p/2npYlakxMd3

@simonl2002 simonl2002 removed the triage Needs to be triaged label Dec 3, 2024
@simonl2002 simonl2002 moved this from Triage to Todo in Conduit Main Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Todo
Development

No branches or pull requests

2 participants