How to encrypt password ?

[harsh288]

How to encrypt the password, because of security reasons we can't keep in a text file

host rdb_host
  database rdb_database
  adapter mysql2_or_postgresql_or_etc
  username myusername
  password mypassword

[kenhys]

How about using https://docs.fluentd.org/quickstart/faq#how-can-i-use-environment-variables-to-configure-parameters-dynamically?

[harsh288]

Hello @kenhys Sorry for replying late, I thought this is a dead forum, wasn't expecting any reply over here, the environment is the last option to me but still, it is a security threat.

Also one more issue I faced recently was if you specify a host like hostName/sqlInstanceName it doesn't work, in one server there is a possibility of having more than one server instances.

@repeatedly @kenhys @frsyuki @cosmo0920 @ashie @ganmacs

[harsh288]

@kenhys

Can we write some custom encryption logic inside
fluent-plugin-sql for the password where the password will be in an encrypted format in the config file and the custom logic will decrypt first and then it will pass it to the SQL DB.

[harsh288]

Hello @repeatedly @kenhys @frsyuki @cosmo0920 @ashie @ganmacs

Can you please help, this is such a big security loophole.

[harsh288]

@kenhys Do we have any working solution? This issue has been escalated so need some workaround ASAP.

[ashie]

Generally speaking, encrypting the password here doesn't make sense.
You should set the permission of reading the config file only for the file owner.
Even when encryption is implemented, if an attacker can read the file, the attacker will be also able to decrypt the password same way with the fluentd process. This is the why no one reply to your question.

Writing plain text credentials in the config file isn't fluent-plugin-sql specific, most application use same way such as Rails applications.

If you have a special reason that you are able to make the encryption meaningful, please describe it.
For example, if entering password manually is allowed on every starting your Fluentd process, encrypting the credentials make sense.

[harsh288]

@ashie Yes we do have to enter the password manually at every start of the Fluentd process, more than that, for us, no matter if it is a system password, DB password, or any other passwords, it can't be in plain text, that is our company security policy, we don't care about attackers right now or any other scenario you described, it just that it has to be encrypted, so I request you to please help me with a possible solution as soon as possible.

[ashie]

Rails 5.2 or later supports encrypted credentials by ActiveSupport::EncryptedConfiguration.
(A master key is required even though using it.)

We might be able to introduce similar feature if its really required by many users.

[harsh288]

@ashie Request you to please introduce that.

[KairenP]

I was under the impression this feature would be available, I started browsing for some syntax to enable encryption and unfortunately I didn't find anything so when I checked the code it was completely missing.

I upvote for this feature, can we have in next version soon?

[Harrier007]

I think we can use "config/credentials.yml.enc" of Rails for encrypted password, though looking for a solution, elastic cache and other plugins have the same feature but not for SQL and Oracle plugin.

[harsh288]

Hello @ashie
Could you please share details, will this feature be implemented? if yes, then what will be the deadline?
We have just one week left and most of the testing needs to be completed mid-week so any help on this will be of great help.

[ashie]

We don't have actual plan for it yet.
Although I'm positive for implementing this feature than before, not yet have higher priority.
In addition, if we decide to implement it, we'll add this feature to Fluentd core, not only for this plugin.

BTW I think your requirement can be realized without modifying both this plugin and Fluentd core.
For example:

1. Prepare a ruby script like this as /etc/fluentd/mycredential.rb (please replace key/salt/path with yours):

#!/usr/bin/env ruby

require 'openssl'
require 'fileutils'

class MyCredential
  PATH = ENV["MY_CREDENTIAL_PATH"] || "/etc/fluentd/credential.txt"
  ENCRYPTED_PATH = PATH + ".enc"
  KEY = ENV["MY_CREDENTIAL_KEY"]
  SALT = ENV["MY_CREDENTIAL_SALT"] || "a9bab8f6-5db7-4693-81e8-93951d2c2468"

  def self.encrypt
    data = File.read(PATH)

    enc = OpenSSL::Cipher.new("AES-256-CBC")
    enc.encrypt
    key_iv = OpenSSL::PKCS5.pbkdf2_hmac_sha1(KEY, SALT, 2000,
                                             enc.key_len + enc.iv_len)
    enc.key = key_iv[0, enc.key_len]
    enc.iv = key_iv[enc.key_len, enc.iv_len]

    encrypted_data = ""
    encrypted_data << enc.update(data)
    encrypted_data << enc.final

    encrypted_data
  end

  def self.decrypt(field=nil)
    data = File.read(ENCRYPTED_PATH)

    dec = OpenSSL::Cipher.new("AES-256-CBC")
    dec.decrypt
    key_iv = OpenSSL::PKCS5.pbkdf2_hmac_sha1(KEY, SALT, 2000,
                                             dec.key_len + dec.iv_len)
    dec.key = key_iv[0, dec.key_len]
    dec.iv = key_iv[dec.key_len, dec.iv_len]

    decrypted_data = ""
    decrypted_data << dec.update(data)
    decrypted_data << dec.final

    if field == :user
      decrypted_data.split(/\R/)[0]
    elsif field == :pass
      decrypted_data.split(/\R/)[1]
    else
      decrypted_data
    end
  end

  def self.encrypt_file
    data = encrypt
    File.open(ENCRYPTED_PATH, "wb") do |f|
      f.write(data)
    end
    FileUtils.chmod(0660, ENCRYPTED_PATH)
    FileUtils.rm_f(PATH)
  end

  def self.decrypt_file
    data = decrypt
    File.open(PATH, "wb") do |f|
      f.write(data)
    end
    FileUtils.chmod(0660, PATH)
    FileUtils.rm_f(ENCRYPTED_PATH)
  end
end

if $0 == __FILE__
  case ARGV[0]
  when "encrypt"
    MyCredential.encrypt_file
  when "decrypt"
    MyCredential.decrypt_file
  else
    puts "Unknown command: #{ARGV[0]}"
  end
end

2. Prepare /etc/fluentd/credential.txt like this

myuser
mypassword

3. Encrypt the credential.txt (/etc/fluentd/credential.txt.enc will be created and /etc/fluentd/credential.txt will be deleted):

$ MY_CREDENTIAL_KEY=mykey ruby /etc/fluentd/mycredential.rb encrypt

4. Prepare fluent.conf like this:

  username "#{require('/etc/fluentd/mycredential'); MyCredential.decrypt(:user)}"
  password "#{MyCredential.decrypt(:pass)}"
  ...

5. Run fluentd with the environment variable MY_CREDENTIAL_KEY=mykey

[harsh288]

@ashie Thanks a lot for taking efforts and providing a solution, that worked for me.

[harsh288]

Hello @ashie

username "#{require('C:/opt/td-agent/etc/td-agent/mycredential'); MyCredential.decrypt(:user)}"
password "#{MyCredential.decrypt(:pass)}"

I have the above configuration in the .conf file

When I run td-agent from the command prompt it works however when I run from Windows service, it doesn't work, any idea??
It is not even generating logs in td-agent.log file

[ashie]

Although I don't try it by myself, if storing the master key to registry is acceptable for you, it seems that you can set environment variables for Fluentd service by registry.

• Add a key HKLM\SYSTEM\CurrentControlSet\Services\flunentdwinsvc\Environment with REG_MULTI_SZ type
• Set MY_CREDENTIAL_KEY=mykey as the value of it
• Set permission for the key by registry editor (Edit -> Permission) to restrict showing it by users

[harsh288]

Hi @ashie I missed to update, the above issue was resolved, ruby was expecting the path in forward slashes instead of backward, as if now I have kept the key in a static format, and I'm just running the command once "ruby /etc/fluentd/mycredential.rb encrypt" from Powershell in our custom build setup.

Thanks for the suggestion for storing in registry however we are providing our custom setup to the customer where I don't think we can edit permission, even though we save in registry how to get it either in fluentd or ruby .rb file

[harsh288]

Hello @ashie

We are going to use the latest version of td-agent & fluentd, so are you planning to add this feature?