Elasticsearch

A simple, no-nonsense Elasticsearch library for Elixir. Highlights include:

• No DSLs. Interact directly with the Elasticsearch JSON API.
• Zero-downtime index (re)building. Via Mix.Tasks.Elasticsearch.Build task.
• Dev Tools. Helpers for running Elasticsearch as part of your supervision tree during development.

The Fork

This package is a fork from the upstream.

It has been modified to use the CarReq HTTP adapter.

CarReq is a cars.com HTTP adapter that has been optimized for cars.com use cases and tooling.

Installation

Add elasticsearch to your list of dependencies in mix.exs:

def deps do
  [
    {:elasticsearch, git: "git@github.com:carsdotcom/elasticsearch-elixir", tag: "~> 7.0"}
  ]
end

Then, create an Elasticsearch.Cluster in your application:

defmodule MyApp.ElasticsearchCluster do
  use Elasticsearch.Cluster, otp_app: :my_app
end

Once you have created your cluster, add it to your application's supervision tree:

children = [
  MyApp.ElasticsearchCluster
]

Finally, you can issue requests to Elasticsearch using it.

Elasticsearch.get(MyApp.ElasticsearchCluster, "/_cat/health")

Configuration

See the annotated example configuration below.

config :my_app, MyApp.ElasticsearchCluster,
  # The URL where Elasticsearch is hosted on your system
  url: "http://localhost:9200",

  # If your Elasticsearch cluster uses HTTP basic authentication,
  # specify the username and password here:
  username: "username",
  password: "password",

  # If you want to mock the responses of the Elasticsearch JSON API
  # for testing or other purposes, you can inject a different module
  # here. It must implement the Elasticsearch.API behaviour.
  api: Elasticsearch.API.HTTP,

  # Customize the library used for JSON encoding/decoding.
  json_library: Poison, # or Jason

  # You should configure each index which you maintain in Elasticsearch here.
  # This configuration will be read by the `mix elasticsearch.build` task,
  # described below.
  indexes: %{
    # This is the base name of the Elasticsearch index. Each index will be
    # built with a timestamp included in the name, like "posts-5902341238".
    # It will then be aliased to "posts" for easy querying.
    posts: %{
      # This file describes the mappings and settings for your index. It will
      # be posted as-is to Elasticsearch when you create your index, and
      # therefore allows all the settings you could post directly.
      settings: "priv/elasticsearch/posts.json",

      # This store module must implement a store behaviour. It will be used to
      # fetch data for each source in each indexes' `sources` list, below:
      store: MyApp.ElasticsearchStore,

      # This is the list of data sources that should be used to populate this
      # index. The `:store` module above will be passed each one of these
      # sources for fetching.
      #
      # Each piece of data that is returned by the store must implement the
      # Elasticsearch.Document protocol.
      sources: [MyApp.Post],

      # When indexing data using the `mix elasticsearch.build` task,
      # control the data ingestion rate by raising or lowering the number
      # of items to send in each bulk request.
      bulk_page_size: 5000,

      # Likewise, wait a given period between posting pages to give
      # Elasticsearch time to catch up.
      bulk_wait_interval: 15_000, # 15 seconds

      # By default bulk indexing uses the "create" action. To allow existing
      # documents to be replaced, use the "index" action instead.
      bulk_action: "create"
    }
  }

Specifying HTTPoison Options

config :my_app, MyApp.ElasticsearchCluster,
  default_options: [
    timeout: 5_000,
    recv_timeout: 5_000,
    hackney: [pool: :pool_name]
  ]

Protocols and Behaviours

Elasticsearch.Store

Your app must provide a Store module, which will fetch data to upload to Elasticsearch. This module must implement the Elasticsearch.Store behaviour.

The example below uses Ecto, but you can implement the behaviour on top of any persistence layer.

defmodule MyApp.ElasticsearchStore do
  @behaviour Elasticsearch.Store

  import Ecto.Query

  alias MyApp.Repo

  @impl true
  def stream(schema) do
    Repo.stream(schema)
  end

  @impl true
  def transaction(fun) do
    {:ok, result} = Repo.transaction(fun, timeout: :infinity)
    result
  end
end

Elasticsearch.Document

Each result returned by your store must implement the Elasticsearch.Document protocol.

defimpl Elasticsearch.Document, for: MyApp.Post do
  def id(post), do: post.id
  def routing(_), do: false
  def encode(post) do
    %{
      title: post.title,
      author: post.author
    }
  end
end

Elasticsearch.API

You can plug in a different module to make API requests, as long as it implements the Elasticsearch.API behaviour.

This can be used in test mode, for example:

# config/test.exs
config :my_app, MyApp.ElasticsearchCluster,
  api: MyApp.ElasticsearchMock

Your mock can then stub requests and responses from Elasticsearch.

defmodule MyApp.ElasticsearchMock do
  @behaviour Elasticsearch.API

  @impl true
  def request(_config, :get, "/posts/1", _data, _opts) do
    {:ok, %HTTPoison.Response{
      status_code: 404,
      body: %{
        "status" => "not_found"
      }
    }}
  end
end

Elasticsearch.API.AWS

As AWS does not provide credentials' based http authentication, you can use the Elasticsearch.API.AWS module if you want to use AWS Elasticsearch Service with AWS Signature V4 signed HTTP connections.

To use this, just add sigaws to your dependencies and add this to your configuration:

# Add to deps 
def deps do
  [          
    # ...
    {:sigaws, ">= 0.0.0"}
  ]
end

# config/prod.exs
config :my_app, MyApp.ElasticsearchCluster,
  api: Elasticsearch.API.AWS,
  default_options: [
    aws: [
      region: "us-east-1",
      service: "es",
      access_key: "aws_access_key_id",
      secret: "aws_secret_access_key"
    ]
  ]

Indexing

Bulk

Use the mix elasticsearch.build task to build indexes using a zero-downtime, hot-swap technique with Elasticsearch aliases.

# This will read the `indexes[posts]` configuration seen above, to build
# an index, `posts-123123123`, which will then be aliased to `posts`.
$ mix elasticsearch.build posts --cluster MyApp.ElasticsearchCluster

See the docs on Mix.Tasks.Elasticsearch.Build and Elasticsearch.Index for more details.

Individual Documents

Use Elasticsearch.put_document/3 to upload a document to a particular index.

# MyApp.Post must implement Elasticsearch.Document
Elasticsearch.put_document(MyApp.ElasticsearchCluster, %MyApp.Post{}, "index-name")

To remove documents, use Elasticsearch.delete_document/3:

Elasticsearch.delete_document(MyApp.ElasticsearchCluster, %MyApp.Post{}, "index-name")

Querying

You can query Elasticsearch the post/3 function:

# Raw query
Elasticsearch.post(MyApp.ElasticsearchCluster, "/posts/_search", '{"query": {"match_all": {}}}')

# Using a map
Elasticsearch.post(MyApp.ElasticsearchCluster, "/posts/_search", %{"query" => %{"match_all" => %{}}})

See the official Elasticsearch documentation for how to write queries.

Dev Tools

This package provides two utilities for developing with Elasticsearch:

• mix elasticsearch.install: A mix task to install Elasticsearch and Kibana to a folder of your choosing.

• Elasticsearch.Executable. Use this to start and stop Elasticsearch as part of your supervision tree.

  children = [
    worker(Elasticsearch.Executable, [
      "Elasticsearch",
      "./vendor/elasticsearch/bin/elasticsearch", # assuming elasticsearch is in your vendor/ dir
      9200
    ], id: :elasticsearch),
    worker(Elasticsearch.Executable, [
      "Kibana",
      "./vendor/kibana/bin/kibana", # assuming kibana is in your vendor/ dir
      5601
    ], id: :kibana)
  ]

Elasticsearch 5.x Support

As of version 0.3.0 of this client library, multiple document types are not supported, because support for these was removed in Elasticsearch 6.x. You can still use this library with Elasticsearch 5.x, but you must design your indexes in the Elasticsearch 6.x style.

Read more about this in Elasticsearch's guide, "Removal of Mapping Types".

Documentation

Hex Documentation

Run mix docs to generate local documentation.

Contributing

To contribute code to this project, you'll need to:

1. Fork the repo
2. Clone your fork
3. Run bin/setup
4. Create a branch
5. Commit your changes
6. Open a PR