ingest-geoip file in docker config dir

[maozi07]

• Docker image used
  docker.elastic.co/elasticsearch/elasticsearch-oss:6.0.0
• Operating System
  macOS

I found ingest-geoip dir in elasticsearch config dir /usr/share/elasticsearch/config add there is 2 file
-rw-rw---- 1 elasticsearch root 29M Nov 10 19:50 GeoLite2-City.mmdb.gz
-rw-rw---- 1 elasticsearch root 1.2M Nov 10 19:50 GeoLite2-Country.mmdb.gz
I wonder what are they for ;and without them the docker image can't run
But I download the elasticsearch source code version (6.0) , there no these files

[dliappis]

This is because the ingest-geoip and ingest-user-agent plugins are prebundled in the image, to assist with the getting started experience together with beats (related issue: (see #48).

[maozi07]

Thank you ! But I think we should keep the image simple.Anyway thanks your work

[ewlarson]

Hey! This is really awkward. Plugins should be optional.

This practice is really problematic if you intend to only use ES and not Beats. Why bundle Beats dependencies here? I don't care about the getting started experience of Beats at all. NONE.

It's especially sad if you plan to copy in your own configuration in docker-compose.yml:

...
    volumes:
      - esdata1:/usr/share/elasticsearch/data
      - /gimlet-elasticsearch/config:/usr/share/elasticsearch/config
...

Because, without warning (or expectation!) you'll end up with these ingest-geoip errors in your container logs:

[2017-12-08T22:18:04,096][INFO ][o.e.p.PluginsService     ] [es-node1] loaded plugin [x-pack]
[2017-12-08T22:18:07,450][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [es-node1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config/ingest-geoip

SO! Please consider properly divorcing these projects. Plugins should be plugins. And, please update your documentation for Configuring Elasticsearch with Docker so people can know they're are some mines here.

[nuest]

This breaks my configuration, too. I tried to solve this by disabling the plugin at startup with

ingest-geoip.enabled: false

but can't comfirm this works due to other issues. Hope it helps someone else.

[jarpy]

I don't care about the getting started experience of Beats at all. NONE.

No shouting, please! We're all just trying to do our best here. Specifically, in this case, we're just trying to optimise for common cases. A lot of people do want geo-ip, and want it without having to build a custom image.

Now to your problem...

Your docker-compose.yml is mounting all of /usr/share/elasticsearch/config (the directory) into the container, but does it need to? Does it resolve the issue for you if you instead specifically mount just the elasticsearch.yml file, as demonstrated in the docs. Are there a lot of files you need to customise? If not, mounting the entire directory might be overkill.

[ewlarson]

Sorry for shouting.

I just find it particularly maddening that these plugins come along for the ride here -- on the "official Elasticsearch Docker image from Elastic" -- expressly added for the ease of Beats adopters. A plugin that self-describes its GeoIP lookup process as "surprisingly expensive."

Thanks for the config suggestions, but I'm not pursuing this image any further. An ES only image would be nice someday.

[nuest]

@ewlarson It would be great if you can share here which image you eventually decided to use.

[jeff4747]

Your docker-compose.yml is mounting all of /usr/share/elasticsearch/config (the directory) into the container, but does it need to?

Whether or not @ewlarson does, I need to. I'm attempting to run the docker image in a Kubernetes cluster, and that requires specifying the configuration via mounting a configMap into the config directory. And that overwrites the ingest-geoip configuration.

So, I had to make my own docker image in order to work around this "feature".

Also, ingest-geoip is useless to people in my situation. All the IPs are fake because it's in a virtual network.

Perhaps you could create a separate "with plugins" docker image for your common cases instead of sticking plugins in the main docker image. That allows people to easily run with your plugins, easily run without your plugins, or to easily build a new image with only the plugins that they want.

Dockerfile for anyone else running into this problem:

FROM docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.4
RUN bin/elasticsearch-plugin remove ingest-geoip --purge

[fxdgear]

This is also causing issues when creating a docker-volume to share the keystore across nodes.

By default Elasticsearch looks for the keystore in the config directory, and the keystore must be in config directory.

[dliappis]

Note that ingest-geoip (and ingest-useragent) are going to be a module rather than a plugin starting with 6.7.0 and elastic/elasticsearch#36949 is bringing some improvements that will help when bind-mounting the config directory.

Docs have also been updated to document the upcoming change in elastic/elasticsearch#36966

[rjernst]

As @dliappis pointed out, 6.7.0 now includes the geoip and useragent plugins as modules, built into elasticsearch. While this is the opposite of the original ask of this issue, the ease for these commonly asked for features is important, and the inclusion of these modules is now something outside of docker.

Additionally, maintenance of the docker files for elasticsearch has moved to the elasticsearch repo. As we will be archiving this repository, I am going to close this issue.