out_forward plugin can set targets dynamically via service discovery plugin. #2541

ganmacs · 2019-08-05T08:37:09Z

Which issue(s) this PR fixes:
Fixes #756 ?

What this PR does / why we need it:

I Added service_discovery plugin which sets config of servers in out_forwardy dynamically.
In this PR, I supported sd_file plugins and sd_static plugin.
sd_file checks the specified file periodically and update its targets.
sd_static is same as <server> configuration. So the following configurations are completely the same.

<match test>
  @type forward

  <server>
    host 127.0.0.1
    port 24224
  </server>
<match>

<match test>
  @type forward

  <service_discovery>
    @type static
    host 127.0.0.1
    port 24224
  </service_discovery>
<match>

Docs Changes:

Needed for service discovery plugin.

Release Note:

out_forward plugin can set targets dynamically via service discovery plugin.

repeatedly

Should we create service_discovery helper for supporting create/configure/manage routines?

repeatedly · 2019-08-06T12:06:45Z

lib/fluent/plugin/sd_file.rb

+
+module Fluent
+  module Plugin
+    class SdFile < ServiceDiscovery


Use FileServiceDiscovery better

57d0ef0 fixed

repeatedly · 2019-08-06T12:19:14Z

lib/fluent/plugin/sd_file.rb

+          )
+        end
+      rescue KeyError => e
+        raise Fluent::ConfigError, e


Maybe, KeyError's error message is not user friendly.
Adding more information is better.

Added information about what config needs
61a05cd
b3f4ee905

repeatedly · 2019-08-06T12:23:25Z

lib/fluent/plugin/sd_static.rb

+
+module Fluent
+  module Plugin
+    class SdStatic < ServiceDiscovery


StaticServiceDiscovery is better

57d0ef0 fixed

repeatedly · 2019-08-06T12:34:44Z

lib/fluent/plugin/sd_static.rb

+      def configure(conf)
+        super
+
+        @services << ServiceDiscovery::Service.new(:static, @host, @port, @name, @weight, @standby, @username, @password, @shared_key)


Does this plugin support only one server unlike out_forward?

Good catch.. changed to be able to handle multiple services.
be04758

repeatedly · 2019-08-06T12:55:53Z

lib/fluent/plugin/service_discovery/round_robin_balancer.rb

+          end
+        end
+
+        def select_node


select_node naming is ok?
This seems out_forward's one. select_service is better than select_node?

Right. this is out_forward's name. changed to select_service

6a97db7

ganmacs · 2019-08-07T07:25:50Z

Should we create service_discovery helper for supporting create/configure/manage routines?

fixed 9883173

repeatedly · 2019-08-07T17:00:27Z

lib/fluent/plugin/out_forward.rb

@@ -138,6 +139,10 @@ class ForwardOutput < Output
      config_param :weight, :integer, default: 60
    end

+    config_section :service_discovery do


This define should be done via helper. See parser helper implementation.

fixed 54e8d53

repeatedly · 2019-08-07T17:13:31Z

lib/fluent/plugin_helper/service_discovery.rb

+      # @param configurations [Hash] hash which must has discivery_service type and its configuration like `{ type: :static, conf: <Fluent::Config::Element> }`
+      # @param load_balancer [Object] object which has two methods #rebalance and #select_service
+      # @param custom_build_method [Proc]
+      def create_service_discovery_manager(title, configurations:, load_balancer: nil, custom_build_method: nil, interval: 3)


service_disovery_create_manager is better to follow other plugin helpers.
helpers uses its name as function prefix.

fixed 46e6e0c

repeatedly · 2019-08-07T17:14:42Z

lib/fluent/plugin_helper/service_discovery.rb

+        @_plugin_helper_service_discovery_title = title
+        @_plugin_helper_service_discovery_iterval = interval
+
+        @discovery_manager = Fluent::PluginHelper::ServiceDiscovery::Manager.new(


unlike other helpers, this helper doesn't allow multiple instance, right?

You mean multiple @dicovery_mangers? if so, yes.
This manager instance should be one per output/input plugin. the manager has multiple sd plugins.

repeatedly · 2019-08-07T17:18:46Z

lib/fluent/plugin/service_discovery/discovery_message.rb

+module Fluent
+  module Plugin
+    class ServiceDiscovery # already loaded
+      DiscoveryMessage = Struct.new(:type, :service) do


Exposing this struct is needed? ServiceDiscovery.service_in/service_out also looks good for me.
With this approach, we can move this implementation into service_discovery.rb and users don't need to require additional file.

fixed 4ba7c7a

repeatedly · 2019-08-08T05:20:07Z

@cosmo0920 Do you need to review this? You link this PR from elasticsearch plugin.

cosmo0920 · 2019-08-09T04:48:04Z

I'm considering to use service discovery plugin in elasticsearch plugin.
This PR looks good to me. 👍

@cosmo0920 Do you need to review this? You link this PR from elasticsearch plugin.

Curently, no. The above link is just a note.

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

add sd_file Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

if @rr is greater than the size of @weight_array by reducing weight_array number, node can be nil Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

repeatedly · 2019-10-01T09:38:34Z

Finally merged :)

repeatedly · 2019-12-12T05:05:14Z

@ganmacs Could you update fluentd document for service discovery plugin?

ganmacs self-assigned this Aug 5, 2019

ganmacs added the enhancement Feature request or improve operations label Aug 5, 2019

cosmo0920 mentioned this pull request Aug 5, 2019

Elastic Failover Instance Configuration incase the primary is down, so that fluentd can output the data uken/fluent-plugin-elasticsearch#576

Closed

ganmacs force-pushed the sds branch 3 times, most recently from 87a21da to d1294ee Compare August 6, 2019 06:07

ganmacs marked this pull request as ready for review August 6, 2019 06:07

ganmacs requested a review from repeatedly August 6, 2019 06:08

repeatedly reviewed Aug 6, 2019

View reviewed changes

repeatedly reviewed Aug 7, 2019

View reviewed changes

ganmacs force-pushed the sds branch from be04758 to 4ba7c7a Compare August 8, 2019 03:48

repeatedly added the feature request *Deprecated Label* Use enhancement label in general label Sep 3, 2019

ganmacs added 15 commits September 12, 2019 12:27

Setup for service_discovery plugin

5780b9f

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

wrap load_balancer with sd_manager

8d1e4ab

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

Extract logic building node as build_node

17ae60a

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

Regard server directive as static config in service_discovery

070b3bb

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

handle multiple plugin

ce870c4

add sd_file Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

Add sd_file config

48d4045

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

Fix rr size bug

a62b22f

if @rr is greater than the size of @weight_array by reducing weight_array number, node can be nil Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

Move sd_manager and fix misc

051360e

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

Add examples

067f6d1

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

Add test for service_discovery

c761f94

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

Add stats

bb74342

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

Add default load_balancer

8e362a3

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

Use StatWatcher

6656823

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

use const to avoid typo

fcb95ee

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

change method name

248834d

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

ganmacs added 10 commits September 12, 2019 12:27

Delete unnecessary require

85f0974

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

Fix log message

3548e72

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

Use consistent name with other plugins

694745a

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

Use name which is independent with out_forward

0a02646

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

Use more descriptive error

fd14d0f

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

Add sd helper

bf7bb75

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

StaticServiceDiscovery can handle multiple services

1575927

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

move config_section into helper

f626ede

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

service_disovery_create_manager is better

1059337

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

move descovery_message into service_discovery

6e9d913

Signed-off-by: Yuta Iwama <ganmacs@gmail.com>

ganmacs force-pushed the sds branch from 4ba7c7a to 6e9d913 Compare September 12, 2019 03:28

repeatedly merged commit a14af30 into fluent:master Oct 1, 2019

ganmacs deleted the sds branch October 1, 2019 09:38

bootjp mentioned this pull request Oct 8, 2019

support DNS SRV record by RFC2782 #2496

Closed

nokute78 mentioned this pull request Oct 18, 2019

Quick Start is broken #2659

Closed

ganmacs mentioned this pull request Dec 12, 2019

Add sd fluent/fluentd-docs-gitbook#124

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

out_forward plugin can set targets dynamically via service discovery plugin. #2541

out_forward plugin can set targets dynamically via service discovery plugin. #2541

ganmacs commented Aug 5, 2019 •

edited

Loading

repeatedly left a comment

repeatedly Aug 6, 2019

ganmacs Aug 7, 2019

repeatedly Aug 6, 2019

ganmacs Aug 7, 2019

repeatedly Aug 6, 2019

ganmacs Aug 7, 2019

repeatedly Aug 6, 2019

ganmacs Aug 7, 2019

repeatedly Aug 6, 2019

ganmacs Aug 7, 2019

ganmacs commented Aug 7, 2019

repeatedly Aug 7, 2019

ganmacs Aug 8, 2019

repeatedly Aug 7, 2019

ganmacs Aug 8, 2019

repeatedly Aug 7, 2019

ganmacs Aug 8, 2019

repeatedly Aug 7, 2019

ganmacs Aug 8, 2019

repeatedly commented Aug 8, 2019

cosmo0920 commented Aug 9, 2019

repeatedly commented Oct 1, 2019

repeatedly commented Dec 12, 2019

out_forward plugin can set targets dynamically via service discovery plugin. #2541

out_forward plugin can set targets dynamically via service discovery plugin. #2541

Conversation

ganmacs commented Aug 5, 2019 • edited Loading

repeatedly left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ganmacs commented Aug 7, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

repeatedly commented Aug 8, 2019

cosmo0920 commented Aug 9, 2019

repeatedly commented Oct 1, 2019

repeatedly commented Dec 12, 2019

ganmacs commented Aug 5, 2019 •

edited

Loading