keystore value not available in configuration processing #11055
Comments
LeeDr
changed the title
|
The fact that logstash keystore lowercases keys seems different than beats. I'm not sure about Elasticsearch and Kibana. But I don't think that changed from 7.3.0 to 7.3.1. But it could possibly be that the interpreting of the variables in the config file changed to be case sensitive. that would explain the key/value pair being added (lowercase) and not being found (uppercase). |
|
@LeeDr, the Logstash keystore implementation is different than ES, Beats, and Kibana and has always converted keys to lowercase before storing them. Could you share your Docker configuration because in following your steps above outside of Docker, everything works as expected. |
|
@danhermann the docker configuration in this repo (in stack-docker folder) One option is to reproduce this issue using the integration-test repo.
if [ -z "${MANIFEST}" ]; then
# MANIFEST=snapshots; BRANCH=7.3
# MANIFEST=staging; BUILD=7.3.1-dd7e8fea
MANIFEST=artifacts; BUILD=7.3.0
fi
if [ -z "${MANIFEST}" ]; then
# MANIFEST=snapshots; BRANCH=7.3
MANIFEST=staging; BUILD=7.3.1-dd7e8fea
# MANIFEST=artifacts; BUILD=7.3.0
fi
|
|
Can I get an update as to if this is a blocker? I need to know if we need a fix and a BC build. |
|
@clintandrewhall, I haven't been able to isolate the problem, yet, but there's a reasonable possibility it would cause any Logstash installations with configuration variables to fail to start. Because of that, I'm going to revert the commit that introduced these changes so we can get a 7.3.1 BC without them. |
|
The revert is here: #11056 |
|
Labeling this as a blocker for tracking purposes. Once merged to 7.3.1 I'll build a new BC. Thanks! |
|
@clintandrewhall, this has been merged to the |
|
FYI, I just reproduced this issue outside of a docker environment (CentOS 7 with rpm package install). Aug 19 19:04:06 localhost logstash: ERROR: Failed to load settings file from "path.settings". Aborting... path.setting=/etc/logstash, exception=LogStash::ConfigurationError, message=>Cannot evaluate But I can see that I did add that ELASTIC_PASSWORD key to the keystore. Also I confirmed it not a case problem. I changed logstash.yml to try the lowercase form of the key and it still failed the same way; Aug 19 19:10:01 localhost logstash: ERROR: Failed to load settings file from "path.settings". Aborting... path.setting=/etc/logstash, exception=LogStash::ConfigurationError, message=>Cannot evaluate |
|
I believe this is the same problem as with any other config files in logsatsh. when the config folder is elsewhere (like |
|
@jsvd From this error message, it looks like it is using the correct Aug 19 19:04:06 localhost logstash: ERROR: Failed to load settings file from "path.settings". Aborting... path.setting=/etc/logstash, exception=LogStash::ConfigurationError, message=>Cannot evaluate ${ELASTIC_PASSWORD}. Replacement variable ELASTIC_PASSWORD is not defined in a Logstash secret store or as an Environment entry and there is no default value given. |
|
Ohhh, it's logstash not finding the keystore file? The message doesn't make it clear if it did find a keystore or not. |
The same docker compose I've been running for a few releases has just broken sometime between 7.3.0 and the latest 7.3.1 BC builds. It seems like the value added to the keystore isn't in the environment when the config or yml file is loaded. I don't know if it has anything to do with it being a docker image or not. I can try to reproduce it outside of docker.
I don't know if it's related or not, but this is one of the only changes in 7.3.1 that fits the timeframe of when this broke; #11043
logstash.conf
logstash.yml
The scripts are setting a value in the logstash keystore like this;
And that step appears to work OK;
But when logstash starts it fails with errors indicating it doesn't see that keystore key/value;
"Unable to configure plugins: Cannot evaluate
${ELASTIC_PASSWORD}. Replacement variableELASTIC_PASSWORDis not defined in a Logstash secret store or an environment entry and there is no default value given.",The text was updated successfully, but these errors were encountered: