Logstash and Elastic Cloud on Kubernetes Announcement

[flexitrev]

Overview

We are announcing the Technical Preview of Logstash on Elastic Cloud on Kubernetes (ECK). Enabling users to deploy and manage Logstash instances on Kubernetes clusters. Taking advantage of Kubernetes’ powerful scaling and management capabilities, users can handle large volumes of data while simplifying the complexities of managing a distributed system. Using Kubernetes, users can more easily scale up and down their Logstash deployments based on processing needs. This ensures your data processing pipeline can handle large volumes of data without having wasted capacity.

The initial release includes everything users need to deploy Logstash instances on Kubernetes including documentation, stack monitoring, pipeline definition and support for references to Elasticsearch instances. This functionality is currently in Technical Preview, as we test and add features. We are pleased to make the deployment of Logstash on Kubernetes available under the Basic license which is Free and open to all customers. Advanced features, such as auto-scaling, may require an enterprise license in the future.

Supported Functionality

We have added a Custom Resource Definition (CRD) for Logstash to ECK, supporting the necessary parameters to deploy Logstash on Kubernetes clusters. The Logstash operator supports the deployment, configuration and management of Logstash 8.6+.

Included Features:

• Pipeline Definitions
• All options available in pipelines.yml - workers, batch size, etc
• Support for pipeline reload without pod restart
• ElasticsearchRef
• Stack Monitoring (requires Logstash 8.7.0+)
• Limited Plugin support
• Elasticsearch plugins
• Beats/Agent Input
• Kafka input/output
• TCP input/output
• HTTP input//output
• Most filters with the exception of aggregate and JDBC streaming/static

Limitations

By making it available to users as a Technical Preview, we hope to provide the community with much desired functionality and solicit feedback on feature development. A technical preview means functionality can change without deprecation warnings. Be sure to pay close attention to ECK release notes.

The following features are under consideration, but will not be available in the initial release.

• Persistent Volume management for DLQ/PQs
• Automatic configuration and ingestion from Beats/Agent instances running inside the K8s cluster
• Support for special egress rules
• Autoscaling
• Additional Plugin Support
• S3 input/output
• Redis input/output
• UDP input/output

Where can I get it?

We plan on releasing with the next Elastic stack release, in late May. The logstash operator, updated CRD and user facing documentation will be available as part of the Elastic Cloud on Kubernetes for anyone to download. We want to hear back from the community about their experience deploying and managing Logstash with Kubenetes, please use comments below, or file issues against the cloud-on-k8s repository.

[asazallesmilner]

How does this interact with the licensing of resources managed by the operator in enterprise clusters and how that effects how many ERU's we need.

[flexitrev]

How does this interact with the licensing of resources managed by the operator in enterprise clusters and how that effects how many ERU's we need.

The current Elastic subscription agreement specifically excludes Logstash from billable enterprise software, which means Logstash deployments are not counted towards enterprise resource units

[flexitrev]

How does this interact with the licensing of resources managed by the operator in enterprise clusters and how that effects how many ERU's we need.

The current Elastic subscription agreement specifically excludes Logstash from billable enterprise software, which means Logstash deployments are not counted towards enterprise resource units

The Elastic subscription agreement has been updated to distinguish between ECK and all other deployments. This is to be consistent with the ECK pricing strategy. Under the new agreement, ECK managed Logstash resources are not excluded from Billable software. The Opex savings made available by the Logstash operator, through the ease of deployment, management and scaling more than offset the cost of ERUs associated with ECK managed Logstash resources.