Encrypt or hide ES credentials in td-agent configuration file

[parvvam]

Describe the bug

We want to encrypt the elasticsearch password mentioned in the td-agent configuration.

To Reproduce

<match index name.**>
@type elasticsearch_dynamic
host
port 9200
logstash_format true
index_name ${'indexname'+'.'+Date.parse(record['Time']).strftime('%W%Y')}
time_key Time
logstash_format false
flush_interval 10s
user elastic
password
scheme https
client_cert /path-to-certificate
client_key /path-to-cert-key
ssl_version TLSv1_2
ssl_verify false

Expected behavior

Encrypted elasticsearch password in the conf block

Your Environment

- Fluentd version: 1.14.3
- TD Agent version: td-agent 4.3.0
- Operating system: Ubuntu 18.04
- Kernel version: 4.15.0-192-generic

Your Configuration

####
## Output descriptions:
##


# Treasure Data (http://www.treasure-data.com/) provides cloud based data
# analytics platform, which easily stores and processes data from td-agent.
# FREE plan is also provided.
# @see http://docs.fluentd.org/articles/http-to-td
#
# This section matches events whose tag is td.DATABASE.TABLE
<match td.*.*>
  @type tdlog
  @id output_td
  apikey YOUR_API_KEY

  auto_create_table
  <buffer>
    @type file
    path /var/log/td-agent/buffer/td
  </buffer>

  <secondary>
    @type file
    path /var/log/td-agent/failed_records
  </secondary>
</match>

## match tag=debug.** and dump to console
<match debug.**>
  @type stdout
  @id output_stdout
</match>

####
## Source descriptions:
##

## built-in TCP input
## @see http://docs.fluentd.org/articles/in_forward
<source>
  @type forward
  @id input_forward
</source>

## built-in UNIX socket input
#<source>
#  type unix
#</source>

# HTTP input
# POST http://localhost:8888/<tag>?json=<json>
# POST http://localhost:8888/td.myapp.login?json={"user"%3A"me"}
# @see http://docs.fluentd.org/articles/in_http
<source>
  @type http
  @id input_http
  port 8888
</source>

## live debugging agent
<source>
  @type debug_agent
  @id input_debug_agent
  bind 127.0.0.1
  port 24230
</source>

####
## Examples:
##

## File input
## read apache logs continuously and tags td.apache.access
#<source>
#  @type tail
#<source>
#  @type tail
#  @id input_tail
#  <parse>
#    @type apache2
#  </parse>
#  path /var/log/httpd-access.log
#  tag td.apache.access
#</source>

## File output
## match tag=local.** and write to file
#<match local.**>
#  @type file
#  @id output_file
#  path /var/log/td-agent/access
#</match>

## Forwarding
## match tag=system.** and forward to another td-agent server
#<match system.**>
#  @type forward
#  @id output_system_forward
#
#  <server>
#    host 192.168.0.11
#  </server>
#  # secondary host is optional
#  <secondary>
#    <server>
#      host 192.168.0.12
#    </server>
#  </secondary>
#</match>

## Multiple output
## match tag=td.*.* and output to Treasure Data AND file
#<match td.*.*>
#  @type copy
#<match td.*.*>
#  @type copy
#  @id output_copy
#  <store>
#    @type tdlog
#    apikey API_KEY
#    auto_create_table
#    <buffer>
#      @type file
#      path /var/log/td-agent/buffer/td
#    </buffer>
#  </store>
#  <store>
#    @type file
#    path /var/log/td-agent/td-%Y-%m-%d/%H.log
#  </store>
#</match>
<match indexname.**>
   @type elasticsearch_dynamic
   host hostname
   port 9200
   include_tag_key true
   tag_key @log_name
   logstash_format true
   index_name ${'index name'+'.'+Date.parse(record['Time']).strftime('%m%Y')}
   time_key Time
   logstash_format false
   flush_interval 10s
   user username
   password password
   scheme https
   client_cert /path-to-certificate
   client_key /path-to-certificate-key
   ssl_version TLSv1_2
   ssl_verify false
</match>

Your Error Log

NA

Additional context

We want to encrypt the password mentioned in the configuration block, instead of placing it in cleartext

[rama534]

Yes, i'm also looking for the same.

Team, any updates on this query?

[ashie]

There is no plan to support such feature in Fluentd core for now, but there is a work around: fluent/fluent-plugin-sql#106 (comment)