Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't Upgrade from SELKS 5 to SELKS 6 -- Elasticsearch and Moloch errors #297

Open
P-Sandusky opened this issue Feb 23, 2021 · 14 comments
Open

Can't Upgrade from SELKS 5 to SELKS 6 -- Elasticsearch and Moloch errors #297

P-Sandusky opened this issue Feb 23, 2021 · 14 comments

Comments

@P-Sandusky
Copy link

I have a SELKS installation here that I've successfully upgraded to SELKS 5, but now it's not making the transition to SELKS 6. Suricata and Scirius seem to have made the change, but Logstash hangs when I run the upgrade scripts, and Elasticsearch fails outright. Once Elastic fails, Moloch doesn't upgrade.

I've tried killing Logstash's processes directly, and that got the install script to run. Elastic promptly quit. At first, I noticed a number of what appeared to be Python dependencies missing, so I ran the dependency install script. At the most recent attempt, Logstash would not halt, so I did not see if installing the dependencies had an impact.

Here is my most recent health output:

selks-user@SELKS:~$ sudo selks-health-check_stamus
● suricata.service - LSB: Next Generation IDS/IPS
Loaded: loaded (/etc/init.d/suricata; generated)
Active: active (running) since Tue 2021-02-23 14:08:21 EST; 35min ago
Docs: man:systemd-sysv-generator(8)
Process: 846 ExecStart=/etc/init.d/suricata start (code=exited, status=0/SUCCESS)
Tasks: 14 (limit: 4915)
Memory: 1.4G
CGroup: /system.slice/suricata.service
└─909 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -v --user=logstash

Feb 23 14:08:21 SELKS systemd[1]: Starting LSB: Next Generation IDS/IPS...
Feb 23 14:08:21 SELKS suricata[846]: Starting suricata in IDS (af-packet) mode... done.
Feb 23 14:08:21 SELKS systemd[1]: Started LSB: Next Generation IDS/IPS.
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2021-02-23 14:22:54 EST; 20min ago
Docs: https://www.elastic.co
Process: 3897 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 3897 (code=exited, status=1/FAILURE)

Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: at org.elasticsearch.cli.Command.main(Command.java:79)
Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: For complete error details, refer to the log at /var/log/elasticsearch/elasticsearch.log
Feb 23 14:22:54 SELKS systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Feb 23 14:22:54 SELKS systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Feb 23 14:22:54 SELKS systemd[1]: Failed to start Elasticsearch.
● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
Active: deactivating (stop-sigterm) since Tue 2021-02-23 14:12:36 EST; 31min ago
Main PID: 497 (java)
Tasks: 55 (limit: 4915)
Memory: 985.1M
CGroup: /system.slice/logstash.service
└─497 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyO…

Feb 23 14:43:23 SELKS logstash[497]: [2021-02-23T14:43:23,532][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info…
Feb 23 14:43:26 SELKS logstash[497]: [2021-02-23T14:43:26,752][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES inst…
Feb 23 14:43:26 SELKS logstash[497]: [2021-02-23T14:43:26,923][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES inst…
Feb 23 14:43:28 SELKS logstash[497]: [2021-02-23T14:43:28,605][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info…
Feb 23 14:43:31 SELKS logstash[497]: [2021-02-23T14:43:31,759][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES inst…
Feb 23 14:43:31 SELKS logstash[497]: [2021-02-23T14:43:31,932][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES inst…
Feb 23 14:43:33 SELKS logstash[497]: [2021-02-23T14:43:33,698][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info…
Feb 23 14:43:36 SELKS logstash[497]: [2021-02-23T14:43:36,766][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES inst…
Feb 23 14:43:36 SELKS logstash[497]: [2021-02-23T14:43:36,942][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES inst…
Feb 23 14:43:38 SELKS logstash[497]: [2021-02-23T14:43:38,777][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info…
Hint: Some lines were ellipsized, use -l to show in full.
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2021-02-23 14:22:54 EST; 20min ago
Docs: https://www.elastic.co
Main PID: 4121 (node)
Tasks: 11 (limit: 4915)
Memory: 159.4M
CGroup: /system.slice/kibana.service
└─4121 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kiba…

Feb 23 14:22:54 SELKS systemd[1]: Started Kibana.
● evebox.service - EveBox Server
Loaded: loaded (/lib/systemd/system/evebox.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2021-02-23 14:08:20 EST; 35min ago
Main PID: 491 (evebox)
Tasks: 9 (limit: 4915)
Memory: 19.2M
CGroup: /system.slice/evebox.service
└─491 /usr/bin/evebox server

Feb 23 14:08:20 SELKS systemd[1]: Started EveBox Server.
Feb 23 14:08:20 SELKS evebox[491]: 2021-02-23 14:08:20 INFO evebox::version: This is EveBox version 0.12.0 (rev: ba9d586); x86_64-unknown-linux-musl
Feb 23 14:08:20 SELKS evebox[491]: 2021-02-23 14:08:20 INFO evebox::server::main: Using temporary in-memory configuration database
Feb 23 14:08:20 SELKS evebox[491]: 2021-02-23 14:08:20 INFO evebox::sqlite::configrepo: Initializing SQLite database
Feb 23 14:08:20 SELKS evebox[491]: 2021-02-23 14:08:20 INFO evebox::sqlite::configrepo: Updating SQLite database to schema version 1
Feb 23 14:08:20 SELKS evebox[491]: 2021-02-23 14:08:20 ERROR evebox::server::main: Failed to get Elasticsearch version, things may not work righ…s error 111)
Feb 23 14:08:20 SELKS evebox[491]: 2021-02-23 14:08:20 INFO evebox::server::main: Starting server on 127.0.0.1:5636, tls=false
Hint: Some lines were ellipsized, use -l to show in full.
● molochviewer-selks.service - Moloch Viewer
Loaded: loaded (/etc/systemd/system/molochviewer-selks.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2021-02-23 14:14:39 EST; 29min ago
Process: 2085 ExecStart=/bin/sh -c /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini >> /data/moloch/logs/viewer.log 2>&1 (code=exited, status=1/FAILURE)
Main PID: 2085 (code=exited, status=1/FAILURE)

Feb 23 14:14:39 SELKS systemd[1]: molochviewer-selks.service: Service RestartSec=1min 30s expired, scheduling restart.
Feb 23 14:14:39 SELKS systemd[1]: molochviewer-selks.service: Scheduled restart job, restart counter is at 4.
Feb 23 14:14:39 SELKS systemd[1]: Stopped Moloch Viewer.
Feb 23 14:14:39 SELKS systemd[1]: molochviewer-selks.service: Start request repeated too quickly.
Feb 23 14:14:39 SELKS systemd[1]: molochviewer-selks.service: Failed with result 'exit-code'.
Feb 23 14:14:39 SELKS systemd[1]: Failed to start Moloch Viewer.
● molochpcapread-selks.service - Moloch Pcap Read
Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2021-02-23 14:14:21 EST; 29min ago
Process: 2082 ExecStart=/bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/ >> /data/moloch/logs/capture.log 2>&1 (code=exited, status=1/FAILURE)
Main PID: 2082 (code=exited, status=1/FAILURE)

Feb 23 14:14:21 SELKS systemd[1]: molochpcapread-selks.service: Service RestartSec=1min 30s expired, scheduling restart.
Feb 23 14:14:21 SELKS systemd[1]: molochpcapread-selks.service: Scheduled restart job, restart counter is at 4.
Feb 23 14:14:21 SELKS systemd[1]: Stopped Moloch Pcap Read.
Feb 23 14:14:21 SELKS systemd[1]: molochpcapread-selks.service: Start request repeated too quickly.
Feb 23 14:14:21 SELKS systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'.
Feb 23 14:14:21 SELKS systemd[1]: Failed to start Moloch Pcap Read.
scirius RUNNING pid 4135, uptime 0:20:46
ii elasticsearch 7.11.1 amd64 Distributed RESTful search engine built for the cloud
ii elasticsearch-curator 5.8.3 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices.
ii evebox 1:0.12.0 amd64 no description given
ii kibana 7.11.1 amd64 Explore and visualize your Elasticsearch data
ii kibana-dashboards-stamus 2020122001 amd64 Kibana 6 dashboard templates.
ii logstash 1:7.11.1-1 amd64 An extensible logging pipeline
hi moloch 2.2.3-1 amd64 Moloch Full Packet System
ii scirius 3.5.0-3 amd64 Django application to manage Suricata ruleset
ii suricata 1:2021012201-0stamus0 amd64 Suricata open source multi-thread IDS/IPS/NSM system.
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs tmpfs 798M 8.7M 790M 2% /run
/dev/sda1 ext4 484G 12G 448G 3% /
tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
tmpfs tmpfs 798M 0 798M 0% /run/user/1000
selks-user@SELKS:~$
@pevma
Copy link
Member

pevma commented Feb 23, 2021 via email

@P-Sandusky
Copy link
Author

I used the one in /SELKS5-SELKS6/. No luck.

@P-Sandusky
Copy link
Author

Here's the most recent (two?) runs from Elasticsearch.log:

[2021-02-23T14:08:40,776][INFO ][o.e.n.Node ] [SELKS] version[7.11.1], pid[849], build[default/deb/ff17057114c2199c9c1bbecc727003a907c0db7a/2021-02-15T13:44:09.394032Z], OS[Linux/4.19.0-14-amd64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2021-02-23T14:08:40,800][INFO ][o.e.n.Node ] [SELKS] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2021-02-23T14:08:40,801][INFO ][o.e.n.Node ] [SELKS] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-16348176929129466504, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3988m, -Xmx3988m, -XX:MaxDirectMemorySize=2090860544, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2021-02-23T14:08:50,572][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [aggs-matrix-stats]
[2021-02-23T14:08:50,573][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [analysis-common]
[2021-02-23T14:08:50,574][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [constant-keyword]
[2021-02-23T14:08:50,574][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [flattened]
[2021-02-23T14:08:50,575][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [frozen-indices]
[2021-02-23T14:08:50,575][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [ingest-common]
[2021-02-23T14:08:50,576][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [ingest-geoip]
[2021-02-23T14:08:50,576][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [ingest-user-agent]
[2021-02-23T14:08:50,577][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [kibana]
[2021-02-23T14:08:50,578][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [lang-expression]
[2021-02-23T14:08:50,578][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [lang-mustache]
[2021-02-23T14:08:50,584][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [lang-painless]
[2021-02-23T14:08:50,585][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [mapper-extras]
[2021-02-23T14:08:50,587][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [mapper-version]
[2021-02-23T14:08:50,587][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [parent-join]
[2021-02-23T14:08:50,588][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [percolator]
[2021-02-23T14:08:50,589][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [rank-eval]
[2021-02-23T14:08:50,590][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [reindex]
[2021-02-23T14:08:50,591][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [repositories-metering-api]
[2021-02-23T14:08:50,591][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [repository-url]
[2021-02-23T14:08:50,592][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [search-business-rules]
[2021-02-23T14:08:50,592][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [searchable-snapshots]
[2021-02-23T14:08:50,593][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [spatial]
[2021-02-23T14:08:50,594][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [systemd]
[2021-02-23T14:08:50,595][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [transform]
[2021-02-23T14:08:50,596][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [transport-netty4]
[2021-02-23T14:08:50,599][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [unsigned-long]
[2021-02-23T14:08:50,600][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [vectors]
[2021-02-23T14:08:50,600][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [wildcard]
[2021-02-23T14:08:50,601][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-aggregate-metric]
[2021-02-23T14:08:50,601][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-analytics]
[2021-02-23T14:08:50,601][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-async]
[2021-02-23T14:08:50,602][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-async-search]
[2021-02-23T14:08:50,602][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-autoscaling]
[2021-02-23T14:08:50,602][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ccr]
[2021-02-23T14:08:50,603][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-core]
[2021-02-23T14:08:50,603][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-data-streams]
[2021-02-23T14:08:50,603][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-deprecation]
[2021-02-23T14:08:50,603][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-enrich]
[2021-02-23T14:08:50,604][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-eql]
[2021-02-23T14:08:50,604][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-fleet]
[2021-02-23T14:08:50,604][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-graph]
[2021-02-23T14:08:50,605][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-identity-provider]
[2021-02-23T14:08:50,605][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ilm]
[2021-02-23T14:08:50,605][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ingest]
[2021-02-23T14:08:50,606][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-logstash]
[2021-02-23T14:08:50,606][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ml]
[2021-02-23T14:08:50,606][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-monitoring]
[2021-02-23T14:08:50,606][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ql]
[2021-02-23T14:08:50,607][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-rollup]
[2021-02-23T14:08:50,607][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-runtime-fields]
[2021-02-23T14:08:50,607][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-security]
[2021-02-23T14:08:50,608][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-sql]
[2021-02-23T14:08:50,608][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-stack]
[2021-02-23T14:08:50,608][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-voting-only-node]
[2021-02-23T14:08:50,609][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-watcher]
[2021-02-23T14:08:50,610][INFO ][o.e.p.PluginsService ] [SELKS] no plugins loaded
[2021-02-23T14:08:50,753][INFO ][o.e.e.NodeEnvironment ] [SELKS] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [447.3gb], net total_space [483.2gb], types [ext4]
[2021-02-23T14:08:50,762][INFO ][o.e.e.NodeEnvironment ] [SELKS] heap size [3.8gb], compressed ordinary object pointers [true]
[2021-02-23T14:08:51,570][INFO ][o.e.n.Node ] [SELKS] node name [SELKS], node ID [voxRMqkTTqGjtgSKpUG0FA], cluster name [elasticsearch], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2021-02-23T14:09:02,384][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [SELKS] [controller/1147] [Main.cc@117] controller (64 bit): Version 7.11.1 (Build b7aec245e3d54f) Copyright (c) 2021 Elasticsearch BV
[2021-02-23T14:09:03,919][INFO ][o.e.x.s.a.s.FileRolesStore] [SELKS] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-02-23T14:09:07,930][INFO ][o.e.t.NettyAllocator ] [SELKS] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-02-23T14:09:08,164][INFO ][o.e.d.DiscoveryModule ] [SELKS] using discovery type [zen] and seed hosts providers [settings]
[2021-02-23T14:09:09,493][INFO ][o.e.g.DanglingIndicesState] [SELKS] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-02-23T14:09:10,327][INFO ][o.e.n.Node ] [SELKS] initialized
[2021-02-23T14:09:10,327][INFO ][o.e.n.Node ] [SELKS] starting ...
[2021-02-23T14:09:10,374][INFO ][o.e.x.s.c.PersistentCache] [SELKS] persistent cache index loaded
[2021-02-23T14:09:10,652][INFO ][o.e.t.TransportService ] [SELKS] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2021-02-23T14:09:11,090][ERROR][o.e.b.Bootstrap ] [SELKS] Exception
java.lang.IllegalStateException: The index [[logstash-fileinfo-2021.02.22/3gFBI1G8So-ctURgcbl92A]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:108) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:82) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:236) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) [elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.1.jar:7.11.1]
at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.1.jar:7.11.1]
[2021-02-23T14:09:11,100][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [SELKS] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: The index [[logstash-fileinfo-2021.02.22/3gFBI1G8So-ctURgcbl92A]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.1.jar:7.11.1]
at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.1.jar:7.11.1]
Caused by: java.lang.IllegalStateException: The index [[logstash-fileinfo-2021.02.22/3gFBI1G8So-ctURgcbl92A]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:108) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:82) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:236) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.1.jar:7.11.1]
... 6 more
[2021-02-23T14:09:11,107][INFO ][o.e.n.Node ] [SELKS] stopping ...
[2021-02-23T14:09:11,135][INFO ][o.e.n.Node ] [SELKS] stopped
[2021-02-23T14:09:11,136][INFO ][o.e.n.Node ] [SELKS] closing ...
[2021-02-23T14:09:11,159][INFO ][o.e.n.Node ] [SELKS] closed
[2021-02-23T14:09:11,162][INFO ][o.e.x.m.p.NativeController] [SELKS] Native controller process has stopped - no new native processes can be started
[2021-02-23T14:22:35,579][INFO ][o.e.n.Node ] [SELKS] version[7.11.1], pid[3897], build[default/deb/ff17057114c2199c9c1bbecc727003a907c0db7a/2021-02-15T13:44:09.394032Z], OS[Linux/4.19.0-14-amd64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2021-02-23T14:22:35,585][INFO ][o.e.n.Node ] [SELKS] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2021-02-23T14:22:35,585][INFO ][o.e.n.Node ] [SELKS] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-11326985180863640013, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3988m, -Xmx3988m, -XX:MaxDirectMemorySize=2090860544, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2021-02-23T14:22:40,236][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [aggs-matrix-stats]
[2021-02-23T14:22:40,237][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [analysis-common]
[2021-02-23T14:22:40,237][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [constant-keyword]
[2021-02-23T14:22:40,238][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [flattened]
[2021-02-23T14:22:40,238][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [frozen-indices]
[2021-02-23T14:22:40,238][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [ingest-common]
[2021-02-23T14:22:40,239][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [ingest-geoip]
[2021-02-23T14:22:40,239][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [ingest-user-agent]
[2021-02-23T14:22:40,240][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [kibana]
[2021-02-23T14:22:40,240][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [lang-expression]
[2021-02-23T14:22:40,240][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [lang-mustache]
[2021-02-23T14:22:40,241][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [lang-painless]
[2021-02-23T14:22:40,241][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [mapper-extras]
[2021-02-23T14:22:40,242][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [mapper-version]
[2021-02-23T14:22:40,242][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [parent-join]
[2021-02-23T14:22:40,243][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [percolator]
[2021-02-23T14:22:40,243][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [rank-eval]
[2021-02-23T14:22:40,244][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [reindex]
[2021-02-23T14:22:40,244][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [repositories-metering-api]
[2021-02-23T14:22:40,245][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [repository-url]
[2021-02-23T14:22:40,246][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [search-business-rules]
[2021-02-23T14:22:40,246][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [searchable-snapshots]
[2021-02-23T14:22:40,247][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [spatial]
[2021-02-23T14:22:40,247][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [systemd]
[2021-02-23T14:22:40,247][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [transform]
[2021-02-23T14:22:40,248][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [transport-netty4]
[2021-02-23T14:22:40,248][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [unsigned-long]
[2021-02-23T14:22:40,249][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [vectors]
[2021-02-23T14:22:40,249][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [wildcard]
[2021-02-23T14:22:40,250][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-aggregate-metric]
[2021-02-23T14:22:40,250][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-analytics]
[2021-02-23T14:22:40,250][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-async]
[2021-02-23T14:22:40,251][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-async-search]
[2021-02-23T14:22:40,251][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-autoscaling]
[2021-02-23T14:22:40,251][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ccr]
[2021-02-23T14:22:40,252][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-core]
[2021-02-23T14:22:40,252][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-data-streams]
[2021-02-23T14:22:40,252][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-deprecation]
[2021-02-23T14:22:40,253][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-enrich]
[2021-02-23T14:22:40,253][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-eql]
[2021-02-23T14:22:40,253][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-fleet]
[2021-02-23T14:22:40,254][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-graph]
[2021-02-23T14:22:40,254][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-identity-provider]
[2021-02-23T14:22:40,255][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ilm]
[2021-02-23T14:22:40,255][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ingest]
[2021-02-23T14:22:40,256][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-logstash]
[2021-02-23T14:22:40,256][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ml]
[2021-02-23T14:22:40,257][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-monitoring]
[2021-02-23T14:22:40,257][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ql]
[2021-02-23T14:22:40,257][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-rollup]
[2021-02-23T14:22:40,258][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-runtime-fields]
[2021-02-23T14:22:40,258][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-security]
[2021-02-23T14:22:40,258][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-sql]
[2021-02-23T14:22:40,259][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-stack]
[2021-02-23T14:22:40,259][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-voting-only-node]
[2021-02-23T14:22:40,260][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-watcher]
[2021-02-23T14:22:40,261][INFO ][o.e.p.PluginsService ] [SELKS] no plugins loaded
[2021-02-23T14:22:40,353][INFO ][o.e.e.NodeEnvironment ] [SELKS] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [447.1gb], net total_space [483.2gb], types [ext4]
[2021-02-23T14:22:40,354][INFO ][o.e.e.NodeEnvironment ] [SELKS] heap size [3.8gb], compressed ordinary object pointers [true]
[2021-02-23T14:22:40,629][INFO ][o.e.n.Node ] [SELKS] node name [SELKS], node ID [voxRMqkTTqGjtgSKpUG0FA], cluster name [elasticsearch], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2021-02-23T14:22:48,732][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [SELKS] [controller/4081] [Main.cc@117] controller (64 bit): Version 7.11.1 (Build b7aec245e3d54f) Copyright (c) 2021 Elasticsearch BV
[2021-02-23T14:22:50,215][INFO ][o.e.x.s.a.s.FileRolesStore] [SELKS] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-02-23T14:22:52,161][INFO ][o.e.t.NettyAllocator ] [SELKS] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-02-23T14:22:52,288][INFO ][o.e.d.DiscoveryModule ] [SELKS] using discovery type [zen] and seed hosts providers [settings]
[2021-02-23T14:22:53,236][INFO ][o.e.g.DanglingIndicesState] [SELKS] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-02-23T14:22:54,171][INFO ][o.e.n.Node ] [SELKS] initialized
[2021-02-23T14:22:54,171][INFO ][o.e.n.Node ] [SELKS] starting ...
[2021-02-23T14:22:54,220][INFO ][o.e.x.s.c.PersistentCache] [SELKS] persistent cache index loaded
[2021-02-23T14:22:54,351][INFO ][o.e.t.TransportService ] [SELKS] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2021-02-23T14:22:54,801][ERROR][o.e.b.Bootstrap ] [SELKS] Exception
java.lang.IllegalStateException: The index [[logstash-2021.02.22/lgzF808KSneDDI6whnU_4w]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:108) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:82) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:236) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) [elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.1.jar:7.11.1]
at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.1.jar:7.11.1]
[2021-02-23T14:22:54,810][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [SELKS] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: The index [[logstash-2021.02.22/lgzF808KSneDDI6whnU_4w]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.1.jar:7.11.1]
at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.1.jar:7.11.1]
Caused by: java.lang.IllegalStateException: The index [[logstash-2021.02.22/lgzF808KSneDDI6whnU_4w]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:108) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:82) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:236) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.1.jar:7.11.1]
... 6 more
[2021-02-23T14:22:54,816][INFO ][o.e.n.Node ] [SELKS] stopping ...
[2021-02-23T14:22:54,846][INFO ][o.e.n.Node ] [SELKS] stopped
[2021-02-23T14:22:54,847][INFO ][o.e.n.Node ] [SELKS] closing ...
[2021-02-23T14:22:54,879][INFO ][o.e.n.Node ] [SELKS] closed
[2021-02-23T14:22:54,883][INFO ][o.e.x.m.p.NativeController] [SELKS] Native controller process has stopped - no new native processes can be started
selks-user@SELKS:$

@P-Sandusky
Copy link
Author

Not to spam overmuch, but... Running updates directly from apt didn't set things to rights, either.

Earlier, Elastic was giving some indexing errors. I found an earlier issue (#238) that I hoped would get around the problem, but there hasn't been much happiness to be found where Elastic is concerned.

@P-Sandusky
Copy link
Author

Here's what I'm getting when I run the install script (I killed Logstash before I ran this, else it would hang):

selks-user@SELKS:/SELKS/scripts$ ls -l
total 12
drwxr-xr-x 2 root root 4096 Feb 22 17:04 SELKS3-SELKS4
drwxr-xr-x 2 root root 4096 Feb 22 17:04 SELKS4-SELKS5
drwxr-xr-x 2 root root 4096 Feb 22 17:04 SELKS5-SELKS6
selks-user@SELKS:/SELKS/scripts$ cd SELKS5-SELKS6
selks-user@SELKS:/SELKS/scripts/SELKS5-SELKS6$ ls -l
total 20
-rw-r--r-- 1 root root 102 Feb 22 17:04 README.rst
-rwxr-xr-x 1 root root 12853 Feb 22 17:04 SN-S5-S6-Upgrade.sh
selks-user@SELKS:/SELKS/scripts/SELKS5-SELKS6$ sudo ./SN-S5-S6-Upgrade.sh

  • (( 0 != 0 ))
  • mkdir -p /opt/selks/preupgrade/elasticsearch/etc/default /opt/selks/preupgrade/elasticsearch/etc/elasticsearch
  • mv /etc/alternatives/desktop-background /opt/selks/preupgrade
    mv: cannot stat '/etc/alternatives/desktop-background': No such file or directory
  • /bin/systemctl stop elasticsearch
  • /bin/systemctl stop kibana
  • /bin/systemctl stop logstash
  • /bin/systemctl stop suricata
  • /usr/bin/supervisorctl stop scirius
    scirius: stopped
  • '[' -f /etc/apt/sources.list.d/elastic-6.x.list ']'
  • cat
  • '[' -f /etc/apt/sources.list.d/curator5.list ']'
  • mv /etc/apt/sources.list.d/curator5.list /opt/selks/preupgrade/curator5.list.orig
  • cat
  • '[' -f /etc/nginx/sites-available/default ']'
  • rm -rf /etc/nginx/sites-enabled/default
  • '[' -f /etc/nginx/sites-available/selks5.conf ']'
  • rm -rf /etc/nginx/sites-available/selks5.conf
  • rm -rf /etc/nginx/sites-enabled/selks5.conf
  • '[' -f /etc/nginx/sites-available/selks5.conf ']'
  • cat
  • ln -s /etc/nginx/sites-available/selks6.conf /etc/nginx/sites-enabled/selks6.conf
    ln: failed to create symbolic link '/etc/nginx/sites-enabled/selks6.conf': File exists
  • /bin/systemctl restart nginx
    Job for nginx.service failed because the control process exited with error code.
    See "systemctl status nginx.service" and "journalctl -xe" for details.
  • '[' -f /etc/logstash/conf.d/logstash.conf ']'
  • mv /etc/logstash/conf.d/logstash.conf /opt/selks/preupgrade/logstash.conf.orig
  • cat
  • '[' -f /etc/logstash/elasticsearch6-template.json ']'
  • cat
  • '[' -f /etc/apt/sources.list.d/selks5.list ']'
  • cat
  • wget -qO - http://packages.stamus-networks.com/packages.selks6.stamus-networks.com.gpg.key
  • apt-key add -
    OK
  • /bin/systemctl stop kibana
  • '[' -f /usr/lib/systemd/system/elasticsearch.service ']'
  • cp /usr/lib/systemd/system/elasticsearch.service /opt/selks/preupgrade/elasticsearch.service.orig
  • cp -r /etc/elasticsearch/elasticsearch.keystore /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/jvm.options /etc/elasticsearch/jvm.options.d /etc/elasticsearch/log4j2.properties /etc/elasticsearch/role_mapping.yml /etc/elasticsearch/roles.yml /etc/elasticsearch/users /etc/elasticsearch/users_roles /opt/selks/preupgrade/elasticsearch/etc/
  • cp /etc/default/elasticsearch /opt/selks/preupgrade/elasticsearch/etc/default/
  • sed -i s/stretch/buster/g /etc/apt/sources.list
  • apt-get update
    Hit:1 http://security.debian.org/debian-security buster/updates InRelease
    Hit:2 http://packages.stamus-networks.com/selks4/debian stretch InRelease
    Hit:3 https://packages.elastic.co/curator/5/debian9 stable InRelease
    Ign:4 https://artifacts.elastic.co/packages/5.x/apt stable InRelease
    Hit:5 http://files.evebox.org/evebox/debian stable InRelease
    Hit:6 http://packages.stamus-networks.com/selks4/debian-kernel stretch InRelease
    Hit:7 http://packages.stamus-networks.com/selks6/debian buster InRelease
    Hit:8 https://artifacts.elastic.co/packages/7.x/apt stable InRelease
    Hit:9 http://debian.csail.mit.edu/debian buster InRelease
    Hit:10 http://packages.stamus-networks.com/selks6/debian-kernel buster InRelease
    Hit:11 http://debian.csail.mit.edu/debian buster-updates InRelease
    Hit:12 https://artifacts.elastic.co/packages/5.x/apt stable Release
    Reading package lists... Done
    W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12
    W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12
    W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12
    W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12
    W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2
    W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2
    W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2
    W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2
    W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13
    W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13
    W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13
    W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13
    W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2
    W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2
    W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2
    W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2
    W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12
    W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12
    W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12
    W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12
    W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13
    W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13
    W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13
    W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13
  • DEBIAN_FRONTEND=noninteractive
  • apt-get -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef -y dist-upgrade
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    Calculating upgrade... Done
    0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
  • exit_status=0
  • [[ 0 -ne 0 ]]
  • chown root:elasticsearch /etc/default/elasticsearch
  • '[' -f /usr/lib/systemd/system/elasticsearch.service.dpkg-new ']'
  • '[' -f /etc/default/elasticsearch.dpkg-new ']'
  • chown -R kibana /usr/share/kibana/optimize/
    chown: cannot access '/usr/share/kibana/optimize/': No such file or directory
  • /bin/systemctl restart elasticsearch
    Job for elasticsearch.service failed because the control process exited with error code.
    See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
  • /bin/systemctl restart kibana
  • /usr/share/logstash/bin/logstash-plugin install logstash-filter-geoip
    Using bundled JDK: /usr/share/logstash/jdk
    OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
    Validating logstash-filter-geoip
    Installing logstash-filter-geoip
    Installation successful
  • /bin/systemctl restart logstash
  • chown logstash -R /data/nsm/
  • /bin/systemctl restart suricata
  • /usr/bin/supervisorctl restart scirius
    scirius: ERROR (not running)
    scirius: started
  • sleep 30
  • apt-get update
    Hit:1 http://debian.csail.mit.edu/debian buster InRelease
    Hit:2 http://packages.stamus-networks.com/selks4/debian stretch InRelease
    Hit:3 https://packages.elastic.co/curator/5/debian9 stable InRelease
    Hit:4 http://debian.csail.mit.edu/debian buster-updates InRelease
    Ign:5 https://artifacts.elastic.co/packages/5.x/apt stable InRelease
    Hit:6 http://packages.stamus-networks.com/selks4/debian-kernel stretch InRelease
    Hit:7 http://files.evebox.org/evebox/debian stable InRelease
    Hit:8 http://packages.stamus-networks.com/selks6/debian buster InRelease
    Hit:9 http://security.debian.org/debian-security buster/updates InRelease
    Hit:10 https://artifacts.elastic.co/packages/7.x/apt stable InRelease
    Hit:11 http://packages.stamus-networks.com/selks6/debian-kernel buster InRelease
    Hit:12 https://artifacts.elastic.co/packages/5.x/apt stable Release
    Reading package lists... Done
    W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12
    W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12
    W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12
    W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12
    W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2
    W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2
    W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2
    W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2
    W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13
    W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13
    W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13
    W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13
    W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2
    W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2
    W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2
    W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2
    W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12
    W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12
    W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12
    W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12
    W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13
    W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13
    W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13
    W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13
  • apt-get -y install elasticsearch-curator
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    elasticsearch-curator is already the newest version (5.8.3).
    0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
  • mv /opt/selks/delete-old-logs.sh /opt/selks/preupgrade/delete-old-logs.sh
  • cat
  • mkdir -p /opt/molochtmp
  • cd /opt/molochtmp/
  • apt-get -y install libwww-perl libjson-perl libyaml-dev libcrypto++6
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    libcrypto++6 is already the newest version (5.6.4-8).
    libjson-perl is already the newest version (4.02000-1).
    libwww-perl is already the newest version (6.36-2).
    libyaml-dev is already the newest version (0.2.1-1).
    0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
  • wget https://files.molo.ch/builds/ubuntu-18.04/moloch_2.2.3-1_amd64.deb
    --2021-02-23 15:14:42-- https://files.molo.ch/builds/ubuntu-18.04/moloch_2.2.3-1_amd64.deb
    Resolving files.molo.ch (files.molo.ch)... 52.85.79.27, 52.85.79.30, 52.85.79.126, ...
    Connecting to files.molo.ch (files.molo.ch)|52.85.79.27|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 82686856 (79M) [application/x-debian-package]
    Saving to: ‘moloch_2.2.3-1_amd64.deb’

moloch_2.2.3-1_amd64.deb 100%[=============================================================================>] 78.86M 10.9MB/s in 7.4s

2021-02-23 15:14:50 (10.7 MB/s) - ‘moloch_2.2.3-1_amd64.deb’ saved [82686856/82686856]

  • dpkg -i moloch_2.2.3-1_amd64.deb
    dpkg: warning: files list file for package 'python-minimal' missing; assuming package has no files currently installed
    dpkg: warning: files list file for package 'python2-minimal' missing; assuming package has no files currently installed
    (Reading database ... 203043 files and directories currently installed.)
    Preparing to unpack moloch_2.2.3-1_amd64.deb ...
    Unpacking moloch (2.2.3-1) over (2.2.3-1) ...
    Setting up moloch (2.2.3-1) ...
    READ /data/moloch/README.txt and RUN /data/moloch/bin/Configure
  • cd /opt/
  • rm /opt/molochtmp -r
  • apt-mark hold moloch
    moloch set on hold.
  • echo '0 3 * * * root ( /data/moloch/db/db.pl http://127.0.0.1:9200 expire daily 14 )'
  • sed -i 's/ELASTICSEARCH_VERSION = 6/ELASTICSEARCH_VERSION = 7/g' /etc/scirius/local_settings.py
  • sed -i 's/KIBANA_VERSION = 6/KIBANA_VERSION = 7/g' /etc/scirius/local_settings.py
  • sed -i 's/KIBANA_INDEX = "kibana-int"/KIBANA_INDEX = ".kibana"/g' /etc/scirius/local_settings.py
  • sed -i 's/KIBANA6_DASHBOARDS_PATH = "/opt/selks/kibana6-dashboards/"/KIBANA6_DASHBOARDS_PATH = "/opt/selks/kibana7-dashboards/"/g' /etc/scirius/local_settings.py
  • echo 'ELASTICSEARCH_KEYWORD = "keyword"'
  • echo 'USE_MOLOCH = True'
  • echo 'MOLOCH_URL = "http://localhost:8005"'
  • /usr/bin/supervisorctl restart scirius
    scirius: stopped
    scirius: started
  • curl -XDELETE 'http://localhost:9200/.kibana*'
    curl: (7) Failed to connect to localhost port 9200: Connection refused
  • /bin/systemctl restart kibana
  • sleep 20
  • selks-first-time-setup_stamus
    START of first time setup script - Tue 23 Feb 2021 03:16:58 PM EST

Setting up sniffing interface

Please supply a network interface(s) to set up SELKS Suricata IDPS thread detection on
0: ens32
1: ens33
2: lo
Please type in interface or space delimited interfaces below and hit "Enter".
Example: eth1
OR
Example: eth1 eth2 eth3

Configure threat detection for INTERFACE(S):
ens33

The supplied network interface(s): ens33

DONE!
FPC - Full Packet Capture. Suricata will rotate and delete the pcap captured files.
FPC_Retain - Full Packet Capture with having Moloch's pcap retention/rotation. Keeps the pcaps as long as there is space available.
None - disable packet capture

  1. FPC
  2. FPC_Retain
  3. NONE
    Please choose an option. Type in a number and hit "Enter" 2
    Enable Full Pcacket Capture with pcap retaining

Starting Moloch DB set up

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 9200: Connection refused
Traceback (most recent call last):
File "bin/manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/core/management/init.py", line 364, in execute_from_command_line
utility.execute()
File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/core/management/init.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/usr/share/python/scirius/local/lib/python2.7/site-packages/rules/management/commands/kibana_reset.py", line 38, in handle
self.kibana_reset()
File "/usr/share/python/scirius/local/lib/python2.7/site-packages/rules/es_data.py", line 1972, in kibana_reset
self._create_kibana_mappings()
File "/usr/share/python/scirius/local/lib/python2.7/site-packages/rules/es_data.py", line 1852, in _create_kibana_mappings
if not self.client.indices.exists('.kibana'):
File "/usr/share/python/scirius/local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/share/python/scirius/local/lib/python2.7/site-packages/elasticsearch/client/indices.py", line 268, in exists
return self.transport.perform_request("HEAD", _make_path(index), params=params)
File "/usr/share/python/scirius/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 358, in perform_request
timeout=timeout,
File "/usr/share/python/scirius/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 224, in perform_request
raise ConnectionError("N/A", str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7ff516b25710>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7ff516b25710>: Failed to establish a new connection: [Errno 111] Connection refused)
Dashboards loading set up job failed...Exiting...

Exited with ERROR

FINISH of first time setup script - Tue 23 Feb 2021 03:17:58 PM EST

Exited with FAILED
Full log located at - /opt/selks/log/selks-first-time-setup_stamus.log
Press enter to continue

selks-user@SELKS:~/SELKS/scripts/SELKS5-SELKS6$

@pevma
Copy link
Member

pevma commented Feb 24, 2021

It seems Elastic is not up?

@P-Sandusky
Copy link
Author

Still isn't. Now it's showing as "dead."

I tried a fresh install from the 6.0 build, hoping to put back anything that might be missing and to start again as fresh as possible. Here's the latest health check report:

selks-user@SELKS:~$ sudo selks-health-check_stamus -l
● suricata.service - LSB: Next Generation IDS/IPS
Loaded: loaded (/etc/init.d/suricata; generated)
Active: active (exited) since Tue 2021-02-23 20:49:01 EST; 11h ago
Docs: man:systemd-sysv-generator(8)
Process: 865 ExecStart=/etc/init.d/suricata start (code=exited, status=0/SUCCESS)

Feb 23 20:49:01 SELKS systemd[1]: Starting LSB: Next Generation IDS/IPS...
Feb 23 20:49:01 SELKS suricata[865]: Starting suricata in IDS (af-packet) mode... done.
Feb 23 20:49:01 SELKS systemd[1]: Started LSB: Next Generation IDS/IPS.
● elasticsearch.service - LSB: Starts elasticsearch
Loaded: loaded (/etc/init.d/elasticsearch; generated)
Active: inactive (dead)
Docs: man:systemd-sysv-generator(8)
● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2021-02-23 20:49:00 EST; 11h ago
Main PID: 516 (java)
Tasks: 58 (limit: 4915)
Memory: 1000.5M
CGroup: /system.slice/logstash.service
└─516 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=…

Feb 24 08:29:50 SELKS logstash[516]: [2021-02-24T08:29:50,891][WARN ][logstash.outputs.elasticsearch][main] Attempted to re…
Feb 24 08:29:51 SELKS logstash[516]: [2021-02-24T08:29:51,021][WARN ][logstash.outputs.elasticsearch][main] Attempted to re…
Feb 24 08:29:55 SELKS logstash[516]: [2021-02-24T08:29:55,894][WARN ][logstash.outputs.elasticsearch][main] Attempted to re…
Feb 24 08:29:56 SELKS logstash[516]: [2021-02-24T08:29:56,026][WARN ][logstash.outputs.elasticsearch][main] Attempted to re…
Feb 24 08:30:00 SELKS logstash[516]: [2021-02-24T08:30:00,898][WARN ][logstash.outputs.elasticsearch][main] Attempted to re…
Feb 24 08:30:01 SELKS logstash[516]: [2021-02-24T08:30:01,030][WARN ][logstash.outputs.elasticsearch][main] Attempted to re…
Feb 24 08:30:05 SELKS logstash[516]: [2021-02-24T08:30:05,901][WARN ][logstash.outputs.elasticsearch][main] Attempted to re…
Feb 24 08:30:06 SELKS logstash[516]: [2021-02-24T08:30:06,035][WARN ][logstash.outputs.elasticsearch][main] Attempted to re…
Feb 24 08:30:10 SELKS logstash[516]: [2021-02-24T08:30:10,904][WARN ][logstash.outputs.elasticsearch][main] Attempted to re…
Feb 24 08:30:11 SELKS logstash[516]: [2021-02-24T08:30:11,040][WARN ][logstash.outputs.elasticsearch][main] Attempted to re…
Hint: Some lines were ellipsized, use -l to show in full.
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2021-02-23 20:49:11 EST; 11h ago
Docs: https://www.elastic.co
Process: 892 ExecStart=/usr/share/kibana/bin/kibana --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/kibana.pid (code=exited, status=200/CHDIR)
Main PID: 892 (code=exited, status=200/CHDIR)

Feb 23 20:49:11 SELKS systemd[1]: kibana.service: Service RestartSec=3s expired, scheduling restart.
Feb 23 20:49:11 SELKS systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3.
Feb 23 20:49:11 SELKS systemd[1]: Stopped Kibana.
Feb 23 20:49:11 SELKS systemd[1]: kibana.service: Start request repeated too quickly.
Feb 23 20:49:11 SELKS systemd[1]: kibana.service: Failed with result 'exit-code'.
Feb 23 20:49:11 SELKS systemd[1]: Failed to start Kibana.
Unit evebox.service could not be found.
● molochviewer-selks.service - Moloch Viewer
Loaded: loaded (/etc/systemd/system/molochviewer-selks.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2021-02-23 20:55:02 EST; 11h ago
Process: 1031 ExecStart=/bin/sh -c /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini >> /data/moloch/logs/viewer.log 2>&1 (code=exited, status=200/CHDIR)
Main PID: 1031 (code=exited, status=200/CHDIR)

Feb 23 20:55:02 SELKS systemd[1]: molochviewer-selks.service: Service RestartSec=1min 30s expired, scheduling restart.
Feb 23 20:55:02 SELKS systemd[1]: molochviewer-selks.service: Scheduled restart job, restart counter is at 4.
Feb 23 20:55:02 SELKS systemd[1]: Stopped Moloch Viewer.
Feb 23 20:55:02 SELKS systemd[1]: molochviewer-selks.service: Start request repeated too quickly.
Feb 23 20:55:02 SELKS systemd[1]: molochviewer-selks.service: Failed with result 'exit-code'.
Feb 23 20:55:02 SELKS systemd[1]: Failed to start Moloch Viewer.
● molochpcapread-selks.service - Moloch Pcap Read
Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2021-02-23 20:55:01 EST; 11h ago
Process: 1029 ExecStart=/bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/ >> /data/moloch/logs/capture.log 2>&1 (code=exited, status=127)
Main PID: 1029 (code=exited, status=127)

Feb 23 20:55:01 SELKS systemd[1]: molochpcapread-selks.service: Service RestartSec=1min 30s expired, scheduling restart.
Feb 23 20:55:01 SELKS systemd[1]: molochpcapread-selks.service: Scheduled restart job, restart counter is at 4.
Feb 23 20:55:01 SELKS systemd[1]: Stopped Moloch Pcap Read.
Feb 23 20:55:01 SELKS systemd[1]: molochpcapread-selks.service: Start request repeated too quickly.
Feb 23 20:55:01 SELKS systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'.
Feb 23 20:55:01 SELKS systemd[1]: Failed to start Moloch Pcap Read.
/usr/bin/selks-health-check_stamus: line 30: /usr/bin/supervisorctl: No such file or directory
rc elasticsearch 7.11.1 amd64 Distributed RESTful search engine built for the cloud
ii elasticsearch-curator 5.8.3 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices.
rc evebox 1:0.12.0 amd64 no description given
rc kibana 7.11.1 amd64 Explore and visualize your Elasticsearch data
ii kibana-dashboards-stamus 2020122001 amd64 Kibana 6 dashboard templates.
ii logstash 1:7.11.1-1 amd64 An extensible logging pipeline
rc scirius 3.5.0-3 amd64 Django application to manage Suricata ruleset
rc suricata 1:2021012201-0stamus0 amd64 Suricata open source multi-thread IDS/IPS/NSM system.
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs tmpfs 798M 21M 778M 3% /run
/dev/sda1 ext4 484G 23G 437G 5% /
tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
tmpfs tmpfs 798M 0 798M 0% /run/user/1000
selks-user@SELKS:~$

@pevma
Copy link
Member

pevma commented Feb 24, 2021

Ok - so if you tail the elasticsearch logs there should be some info > /var/log/elasticsearch/elasticsearch.log ?

@P-Sandusky
Copy link
Author

Elastic hasn't even built out a folder for itself in /var/log. Kibana and Logstash have folders, but Elastic has not generated one.

@P-Sandusky
Copy link
Author

P-Sandusky commented Feb 24, 2021 •
edited by pevma
Loading

I had to install an earlier version of Elastic and then upgrade it to the latest version to get it to behave itself semi-appropriately. Now it's failing again, but it has generated a log file. Here's what I got:

selks-user@SELKS:~$ sudo tail -l var/log/elasticsearch/elasticsearch.log
tail: cannot open 'var/log/elasticsearch/elasticsearch.log' for reading: No such file or directory
selks-user@SELKS:~$ sudo tail -l  /var/log/elasticsearch/elasticsearch.log
        at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.1.jar:7.11.1]
        ... 6 more
[2021-02-24T16:38:29,805][INFO ][o.e.n.Node               ] [SELKS] stopping ...
[2021-02-24T16:38:29,838][INFO ][o.e.n.Node               ] [SELKS] stopped
[2021-02-24T16:38:29,839][INFO ][o.e.n.Node               ] [SELKS] closing ...
[2021-02-24T16:38:29,872][INFO ][o.e.n.Node               ] [SELKS] closed
[2021-02-24T16:38:29,876][INFO ][o.e.x.m.p.NativeController] [SELKS] Native controller process has stopped - no new native processes can be started
selks-user@SELKS:~$

@pevma
Copy link
Member

pevma commented Feb 25, 2021

It seems incomplete - can you tail the last 200 lines please?

@P-Sandusky
Copy link
Author

Here's something a little more complete...

selks-user@SELKS:~$ sudo tail -200 /var/log/elasticsearch/elasticsearch.log
[2021-02-24T16:34:40,469][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [transport-netty4]
[2021-02-24T16:34:40,470][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [unsigned-long]
[2021-02-24T16:34:40,470][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [vectors]
[2021-02-24T16:34:40,471][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [wildcard]
[2021-02-24T16:34:40,471][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-aggregate-metric]
[2021-02-24T16:34:40,471][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-analytics]
[2021-02-24T16:34:40,472][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-async]
[2021-02-24T16:34:40,472][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-async-search]
[2021-02-24T16:34:40,473][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-autoscaling]
[2021-02-24T16:34:40,473][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ccr]
[2021-02-24T16:34:40,474][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-core]
[2021-02-24T16:34:40,474][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-data-streams]
[2021-02-24T16:34:40,474][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-deprecation]
[2021-02-24T16:34:40,475][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-enrich]
[2021-02-24T16:34:40,475][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-eql]
[2021-02-24T16:34:40,476][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-fleet]
[2021-02-24T16:34:40,476][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-graph]
[2021-02-24T16:34:40,476][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-identity-provider]
[2021-02-24T16:34:40,477][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ilm]
[2021-02-24T16:34:40,477][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ingest]
[2021-02-24T16:34:40,478][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-logstash]
[2021-02-24T16:34:40,478][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ml]
[2021-02-24T16:34:40,478][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-monitoring]
[2021-02-24T16:34:40,479][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ql]
[2021-02-24T16:34:40,479][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-rollup]
[2021-02-24T16:34:40,480][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-runtime-fields]
[2021-02-24T16:34:40,480][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-security]
[2021-02-24T16:34:40,480][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-sql]
[2021-02-24T16:34:40,481][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-stack]
[2021-02-24T16:34:40,481][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-voting-only-node]
[2021-02-24T16:34:40,481][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-watcher]
[2021-02-24T16:34:40,482][INFO ][o.e.p.PluginsService     ] [SELKS] no plugins loaded
[2021-02-24T16:34:40,619][INFO ][o.e.e.NodeEnvironment    ] [SELKS] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [435.5gb], net total_space [483.2gb], types [ext4]
[2021-02-24T16:34:40,620][INFO ][o.e.e.NodeEnvironment    ] [SELKS] heap size [3.8gb], compressed ordinary object pointers [true]
[2021-02-24T16:34:42,016][INFO ][o.e.n.Node               ] [SELKS] node name [SELKS], node ID [voxRMqkTTqGjtgSKpUG0FA], cluster name [elasticsearch], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2021-02-24T16:34:54,040][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [SELKS] [controller/1262] [Main.cc@117] controller (64 bit): Version 7.11.1 (Build b7aec245e3d54f) Copyright (c) 2021 Elasticsearch BV
[2021-02-24T16:34:55,539][INFO ][o.e.x.s.a.s.FileRolesStore] [SELKS] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-02-24T16:34:58,809][INFO ][o.e.t.NettyAllocator     ] [SELKS] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-02-24T16:34:58,953][INFO ][o.e.d.DiscoveryModule    ] [SELKS] using discovery type [zen] and seed hosts providers [settings]
[2021-02-24T16:35:00,031][INFO ][o.e.g.DanglingIndicesState] [SELKS] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-02-24T16:35:01,359][INFO ][o.e.n.Node               ] [SELKS] initialized
[2021-02-24T16:35:01,360][INFO ][o.e.n.Node               ] [SELKS] starting ...
[2021-02-24T16:35:01,441][INFO ][o.e.x.s.c.PersistentCache] [SELKS] persistent cache index loaded
[2021-02-24T16:35:01,776][INFO ][o.e.t.TransportService   ] [SELKS] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2021-02-24T16:35:03,218][ERROR][o.e.b.Bootstrap          ] [SELKS] Exception
java.lang.IllegalStateException: The index [[logstash-dns-2021.02.22/w_xhgPARSamErzGJlcQQEA]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:108) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:82) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:236) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.1.jar:7.11.1]
[2021-02-24T16:35:03,232][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [SELKS] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: The index [[logstash-dns-2021.02.22/w_xhgPARSamErzGJlcQQEA]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.1.jar:7.11.1]
Caused by: java.lang.IllegalStateException: The index [[logstash-dns-2021.02.22/w_xhgPARSamErzGJlcQQEA]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:108) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:82) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:236) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.1.jar:7.11.1]
        ... 6 more
[2021-02-24T16:35:03,240][INFO ][o.e.n.Node               ] [SELKS] stopping ...
[2021-02-24T16:35:03,268][INFO ][o.e.n.Node               ] [SELKS] stopped
[2021-02-24T16:35:03,268][INFO ][o.e.n.Node               ] [SELKS] closing ...
[2021-02-24T16:35:03,295][INFO ][o.e.n.Node               ] [SELKS] closed
[2021-02-24T16:35:03,298][INFO ][o.e.x.m.p.NativeController] [SELKS] Native controller process has stopped - no new native processes can be started
[2021-02-24T16:38:12,347][INFO ][o.e.n.Node               ] [SELKS] version[7.11.1], pid[3238], build[default/deb/ff17057114c2199c9c1bbecc727003a907c0db7a/2021-02-15T13:44:09.394032Z], OS[Linux/4.19.0-14-amd64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2021-02-24T16:38:12,353][INFO ][o.e.n.Node               ] [SELKS] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2021-02-24T16:38:12,354][INFO ][o.e.n.Node               ] [SELKS] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-14297637706770411412, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3988m, -Xmx3988m, -XX:MaxDirectMemorySize=2090860544, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [aggs-matrix-stats]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [analysis-common]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [constant-keyword]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [flattened]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [frozen-indices]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [ingest-common]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [ingest-geoip]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [ingest-user-agent]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [kibana]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [lang-expression]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [lang-mustache]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [lang-painless]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [mapper-extras]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [mapper-version]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [parent-join]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [percolator]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [rank-eval]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [reindex]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [repositories-metering-api]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [repository-url]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [search-business-rules]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [searchable-snapshots]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [spatial]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [systemd]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [transform]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [transport-netty4]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [unsigned-long]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [vectors]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [wildcard]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-aggregate-metric]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-analytics]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-async]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-async-search]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-autoscaling]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ccr]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-core]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-data-streams]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-deprecation]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-enrich]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-eql]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-fleet]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-graph]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-identity-provider]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ilm]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ingest]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-logstash]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ml]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-monitoring]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ql]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-rollup]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-runtime-fields]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-security]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-sql]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-stack]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-voting-only-node]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-watcher]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] no plugins loaded
[2021-02-24T16:38:17,795][INFO ][o.e.e.NodeEnvironment    ] [SELKS] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [435.4gb], net total_space [483.2gb], types [ext4]
[2021-02-24T16:38:17,796][INFO ][o.e.e.NodeEnvironment    ] [SELKS] heap size [3.8gb], compressed ordinary object pointers [true]
[2021-02-24T16:38:18,026][INFO ][o.e.n.Node               ] [SELKS] node name [SELKS], node ID [voxRMqkTTqGjtgSKpUG0FA], cluster name [elasticsearch], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2021-02-24T16:38:25,224][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [SELKS] [controller/3424] [Main.cc@117] controller (64 bit): Version 7.11.1 (Build b7aec245e3d54f) Copyright (c) 2021 Elasticsearch BV
[2021-02-24T16:38:26,122][INFO ][o.e.x.s.a.s.FileRolesStore] [SELKS] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-02-24T16:38:27,651][INFO ][o.e.t.NettyAllocator     ] [SELKS] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-02-24T16:38:27,746][INFO ][o.e.d.DiscoveryModule    ] [SELKS] using discovery type [zen] and seed hosts providers [settings]
[2021-02-24T16:38:28,543][INFO ][o.e.g.DanglingIndicesState] [SELKS] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-02-24T16:38:29,355][INFO ][o.e.n.Node               ] [SELKS] initialized
[2021-02-24T16:38:29,356][INFO ][o.e.n.Node               ] [SELKS] starting ...
[2021-02-24T16:38:29,423][INFO ][o.e.x.s.c.PersistentCache] [SELKS] persistent cache index loaded
[2021-02-24T16:38:29,554][INFO ][o.e.t.TransportService   ] [SELKS] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2021-02-24T16:38:29,794][ERROR][o.e.b.Bootstrap          ] [SELKS] Exception
java.lang.IllegalStateException: The index [[logstash-http-2021.02.22/GCm01ePnSx6Ypg84hnKdxQ]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:108) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:82) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:236) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.1.jar:7.11.1]
[2021-02-24T16:38:29,799][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [SELKS] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: The index [[logstash-http-2021.02.22/GCm01ePnSx6Ypg84hnKdxQ]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.1.jar:7.11.1]
Caused by: java.lang.IllegalStateException: The index [[logstash-http-2021.02.22/GCm01ePnSx6Ypg84hnKdxQ]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:108) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:82) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:236) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.1.jar:7.11.1]
        ... 6 more
[2021-02-24T16:38:29,805][INFO ][o.e.n.Node               ] [SELKS] stopping ...
[2021-02-24T16:38:29,838][INFO ][o.e.n.Node               ] [SELKS] stopped
[2021-02-24T16:38:29,839][INFO ][o.e.n.Node               ] [SELKS] closing ...
[2021-02-24T16:38:29,872][INFO ][o.e.n.Node               ] [SELKS] closed
[2021-02-24T16:38:29,876][INFO ][o.e.x.m.p.NativeController] [SELKS] Native controller process has stopped - no new native processes can be started
selks-user@SELKS:~$

@P-Sandusky
Copy link
Author

Does the Elasticsearch migration assistant run as part of the upgrade scripts?

@pevma
Copy link
Member

pevma commented Feb 25, 2021

So that here is really the issue it seems -
was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
You need to migrate form ES 5 to ES 6 to ES7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pevma @P-Sandusky