CSV Scanner not working in SFTP context : extraneous or missing \" in quoted-field"

[acornforth]

Tested with the same files locally using the csv input, works fine. Something about the records extracted from the csv is breaking when they are pulled from sftp. Not sure is this is something speicif to the SFTP server we are connecting to.

I have already googled for issues, and a found some instance of this error occurring for example when line endings are /r, not /r/n or /n. but i checked this and it doesn't appear to be the cause in our case.

Not sure if this is a benthos issue directly or something to do with the underlying encoding/csv package

config below:

input:
  sftp:
    address: '${SFTP_HOST}:22'
    credentials:
      username: '${SFTP_USER}'
      password: '${SFTP_PASSWORD}'
      private_key_file: '/app/.ssh/id_rsa'
    paths:
      - '/*.csv'
    auto_replay_nacks: true
    scanner:
      csv:
        - parse_header_row: true
    delete_on_finish: false
    watcher:
      enabled: true
      minimum_age: 20s
      poll_interval: 60s
      cache: memory

buffer:
  none: {}
output:
  stdout: {}

cache_resources:
  - label: memory
    memory:
      default_ttl: 7200s
  - label: redis
    redis:
      url: 'redis://${REDIS_HOST}:${REDIS_PORT}'
      prefix: benthos_sftpw_cache_

[mihaitodor]

Hey @acornforth 👋 Thanks for reporting this! Would you mind doing a quick test and set lazy_quotes: true in the csv scanner? It should look like this (not sure why you had it as an array):

scanner:
  csv:
    parse_header_row: true
    lazy_quotes: true

I think the error you're hitting is due to this code in the parser.

PS: Converting to a discussion as per #2026.

[acornforth]

Hi @mihaitodor thanks for the reply.

It's not actually an array in my config, that a mistake in copying the config to the discussion.

I'm more than happy to try this with lazy_quotes: true, It wasn't required to set that when testing locally with csv input. The code you linked to does seem suspect though.

I'll give it a try and come back to you with the results