Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Elasticsearch can't start on my laptop and on some ci nodes #77126

Closed
nik9000 opened this issue Sep 1, 2021 · 5 comments · Fixed by #77128
Closed

Elasticsearch can't start on my laptop and on some ci nodes #77126

nik9000 opened this issue Sep 1, 2021 · 5 comments · Fixed by #77128
Labels
>bug :Core/Infra/Node Lifecycle Node startup, bootstrapping, and shutdown Team:Core/Infra Meta label for core/infra team

Comments

@nik9000
Copy link
Member

nik9000 commented Sep 1, 2021

Elasticsearch version (bin/elasticsearch --version): 54d4737

OS version (uname -a if on a Unix-like system):

Linux porco 5.12.10-arch1-1 #1 SMP PREEMPT Thu, 10 Jun 2021 16:34:50 +0000 x86_64 GNU/Linux

Steps to reproduce:
Anything that starts Elasticsearch fails. ./gradlew run should do it.

Provide logs (if relevant):

[2021-09-01T09:35:13,897][INFO ][o.e.p.PluginsService     ] [runTask-0] loaded module [x-pack-watcher]
[2021-09-01T09:35:13,898][INFO ][o.e.p.PluginsService     ] [runTask-0] no plugins loaded
[2021-09-01T09:35:13,902][ERROR][o.e.b.Bootstrap          ] [runTask-0] Exception
java.security.AccessControlException: access denied ("java.io.FilePermission" "/sys/fs/cgroup/user.slice/user-1000.slice/session-2.scope/cpu.stat" "read")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
        at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
        at java.lang.SecurityManager.checkRead(SecurityManager.java:747) ~[?:?]
        at sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:245) ~[?:?]
        at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:133) ~[?:?]
        at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:146) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:217) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:375) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:426) ~[?:?]
        at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) ~[?:?]
        at java.nio.file.Files.newInputStream(Files.java:160) ~[?:?]
        at java.nio.file.Files.newBufferedReader(Files.java:2916) ~[?:?]
        at java.nio.file.Files.readAllLines(Files.java:3396) ~[?:?]
        at java.nio.file.Files.readAllLines(Files.java:3436) ~[?:?]
        at org.elasticsearch.monitor.os.OsProbe.readCgroupV2CpuStats(OsProbe.java:618) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.monitor.os.OsProbe.getCgroupV2CpuStats(OsProbe.java:598) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:650) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:848) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:855) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.ml.MachineLearning.additionalSettings(MachineLearning.java:650) ~[?:?]
        at org.elasticsearch.plugins.PluginsService.updatedSettings(PluginsService.java:199) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.node.Node.<init>(Node.java:338) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.node.Node.<init>(Node.java:270) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:225) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:225) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:365) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:158) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114) [elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:123) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
[2021-09-01T09:35:13,906][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [runTask-0] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.security.AccessControlException: access denied ("java.io.FilePermission" "/sys/fs/cgroup/user.slice/user-1000.slice/session-2.scope/cpu.stat" "read")
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:171) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:158) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114) ~[elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:123) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/sys/fs/cgroup/user.slice/user-1000.slice/session-2.scope/cpu.stat" "read")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
        at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
        at java.lang.SecurityManager.checkRead(SecurityManager.java:747) ~[?:?]
        at sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:245) ~[?:?]
        at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:133) ~[?:?]
        at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:146) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:217) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:375) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:426) ~[?:?]
        at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) ~[?:?]
        at java.nio.file.Files.newInputStream(Files.java:160) ~[?:?]
        at java.nio.file.Files.newBufferedReader(Files.java:2916) ~[?:?]
        at java.nio.file.Files.readAllLines(Files.java:3396) ~[?:?]
        at java.nio.file.Files.readAllLines(Files.java:3436) ~[?:?]
        at org.elasticsearch.monitor.os.OsProbe.readCgroupV2CpuStats(OsProbe.java:618) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.monitor.os.OsProbe.getCgroupV2CpuStats(OsProbe.java:598) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:650) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:848) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:855) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.ml.MachineLearning.additionalSettings(MachineLearning.java:650) ~[?:?]
        at org.elasticsearch.plugins.PluginsService.updatedSettings(PluginsService.java:199) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.node.Node.<init>(Node.java:338) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.node.Node.<init>(Node.java:270) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:225) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:225) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:365) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        ... 6 more
@nik9000 nik9000 added >bug :Core/Infra/Node Lifecycle Node startup, bootstrapping, and shutdown labels Sep 1, 2021
@elasticmachine elasticmachine added the Team:Core/Infra Meta label for core/infra team label Sep 1, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (Team:Core/Infra)

@nik9000
Copy link
Member Author

nik9000 commented Sep 1, 2021

here is an example in CI:

»    ↓ errors and warnings from /dev/shm/elastic+elasticsearch+master+intake+multijob+rest-compat/modules/ingest-user-agent/build/testclusters/yamlRestCompatTest-0/logs/yamlRestCompatTest.log ↓
» [2021-09-01T13:22:11,968][ERROR][o.e.b.Bootstrap          ] [yamlRestCompatTest-0] Exception
»  java.security.AccessControlException: access denied ("java.io.FilePermission" "/sys/fs/cgroup/system.slice/runsvdir.service/cpu.stat" "read")
»  	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
»  	at java.security.AccessController.checkPermission(AccessController.java:897) ~[?:?]
»  	at java.lang.SecurityManager.checkPermission(SecurityManager.java:322) ~[?:?]
»  	at java.lang.SecurityManager.checkRead(SecurityManager.java:661) ~[?:?]
»  	at sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:255) ~[?:?]
»  	at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:143) ~[?:?]
»  	at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:156) ~[?:?]
»  	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:217) ~[?:?]
»  	at java.nio.file.Files.newByteChannel(Files.java:371) ~[?:?]
»  	at java.nio.file.Files.newByteChannel(Files.java:422) ~[?:?]
»  	at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) ~[?:?]
»  	at java.nio.file.Files.newInputStream(Files.java:156) ~[?:?]
»  	at java.nio.file.Files.newBufferedReader(Files.java:2839) ~[?:?]
»  	at java.nio.file.Files.readAllLines(Files.java:3330) ~[?:?]
»  	at java.nio.file.Files.readAllLines(Files.java:3370) ~[?:?]
»  	at org.elasticsearch.monitor.os.OsProbe.readCgroupV2CpuStats(OsProbe.java:618) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.monitor.os.OsProbe.getCgroupV2CpuStats(OsProbe.java:598) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:650) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:848) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:855) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.monitor.os.OsService.<init>(OsService.java:39) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.monitor.MonitorService.<init>(MonitorService.java:33) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.node.Node.<init>(Node.java:443) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.node.Node.<init>(Node.java:270) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:225) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:225) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:365) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:158) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114) [elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:123) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» [2021-09-01T13:22:11,978][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [yamlRestCompatTest-0] uncaught exception in thread [main]
»  org.elasticsearch.bootstrap.StartupException: java.security.AccessControlException: access denied ("java.io.FilePermission" "/sys/fs/cgroup/system.slice/runsvdir.service/cpu.stat" "read")
»  	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:171) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:158) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114) ~[elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:123) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/sys/fs/cgroup/system.slice/runsvdir.service/cpu.stat" "read")
»  	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
»  	at java.security.AccessController.checkPermission(AccessController.java:897) ~[?:?]
»  	at java.lang.SecurityManager.checkPermission(SecurityManager.java:322) ~[?:?]
»  	at java.lang.SecurityManager.checkRead(SecurityManager.java:661) ~[?:?]
»  	at sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:255) ~[?:?]
»  	at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:143) ~[?:?]
»  	at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:156) ~[?:?]
»  	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:217) ~[?:?]
»  	at java.nio.file.Files.newByteChannel(Files.java:371) ~[?:?]
»  	at java.nio.file.Files.newByteChannel(Files.java:422) ~[?:?]
»  	at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) ~[?:?]
»  	at java.nio.file.Files.newInputStream(Files.java:156) ~[?:?]
»  	at java.nio.file.Files.newBufferedReader(Files.java:2839) ~[?:?]
»  	at java.nio.file.Files.readAllLines(Files.java:3330) ~[?:?]
»  	at java.nio.file.Files.readAllLines(Files.java:3370) ~[?:?]
»  	at org.elasticsearch.monitor.os.OsProbe.readCgroupV2CpuStats(OsProbe.java:618) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.monitor.os.OsProbe.getCgroupV2CpuStats(OsProbe.java:598) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:650) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:848) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:855) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.monitor.os.OsService.<init>(OsService.java:39) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.monitor.MonitorService.<init>(MonitorService.java:33) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.node.Node.<init>(Node.java:443) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.node.Node.<init>(Node.java:270) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:225) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:225) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:365) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	... 6 more
»   ↓ last 40 non error or warning messages from /dev/shm/elastic+elasticsearch+master+intake+multijob+rest-compat/modules/ingest-user-agent/build/testclusters/yamlRestCompatTest-0/logs/yamlRestCompatTest.log ↓
» [2021-09-01T13:22:05.011365949Z] [BUILD] Configuring custom cluster specific distro directory: /dev/shm/elastic+elasticsearch+master+intake+multijob+rest-compat/modules/ingest-user-agent/build/testclusters/yamlRestCompatTest-0/distro/8.0.0-INTEG_TEST
» [2021-09-01T13:22:05.019369935Z] [BUILD] Setting up 1 additional config files

@rjernst
Copy link
Member

rjernst commented Sep 1, 2021

@pugnascotia It looks like cgroups are using symlinks? We may need a recursive permission on /sys/fs/cgroup

@nik9000
Copy link
Member Author

nik9000 commented Sep 1, 2021

@pugnascotia has reverted the v2 cgroup handling in 33c4129 to unbreak me and the rare CI machines that break on it.

@pugnascotia
Copy link
Contributor

@rjernst oh man - I totally meant to add that. Epic facepalm.

@pugnascotia pugnascotia mentioned this issue Sep 1, 2021
Merged
@nik9000 nik9000 mentioned this issue Sep 1, 2021
Merged
pugnascotia added a commit that referenced this issue Sep 3, 2021
@pugnascotia
Handle cgroups v2 in OsProbe (#77128)
d238374 
Closes #76812. Closes #77126.

OsProbe was only capable of handle cgroup data in the v1 format.
However, Debian 11 uses cgroups v2 by default, and Elasticsearch isn't
capable of reporting any cgroup information. Therefore, add support for
the v2 layout.

Note that we have to open access to all of /sys/fs/cgroup because with
cgroups v2, the files we need are in an unpredictably location.
pugnascotia added a commit to pugnascotia/elasticsearch that referenced this issue Sep 6, 2021
@pugnascotia
Handle cgroups v2 in OsProbe (elastic#77128)
f034688 
Closes elastic#76812. Closes elastic#77126.

OsProbe was only capable of handle cgroup data in the v1 format.
However, Debian 11 uses cgroups v2 by default, and Elasticsearch isn't
capable of reporting any cgroup information. Therefore, add support for
the v2 layout.

Note that we have to open access to all of /sys/fs/cgroup because with
cgroups v2, the files we need are in an unpredictably location.
@pugnascotia pugnascotia mentioned this issue Sep 6, 2021
Merged
elasticsearchmachine pushed a commit that referenced this issue Sep 6, 2021
@pugnascotia
Handle cgroups v2 in OsProbe (#77128) (#77306)
cf8ae49 
* Handle cgroups v2 in `OsProbe` (#77128)

Closes #76812. Closes #77126.

OsProbe was only capable of handle cgroup data in the v1 format.
However, Debian 11 uses cgroups v2 by default, and Elasticsearch isn't
capable of reporting any cgroup information. Therefore, add support for
the v2 layout.

Note that we have to open access to all of /sys/fs/cgroup because with
cgroups v2, the files we need are in an unpredictably location.

* Handle a max memory value of 'max' (#77289)

* Handle a max memory value of 'max'

* Update docs/changelog/77289.yaml

* Delete 77289.yaml

* Fixes to backport

* Fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>bug :Core/Infra/Node Lifecycle Node startup, bootstrapping, and shutdown Team:Core/Infra Meta label for core/infra team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Handle cgroups v2 in OsProbe pugnascotia/elasticsearch
4 participants
@nik9000 @rjernst @pugnascotia @elasticmachine