[Ingest Manager]: Data Stream not generated for standalone Agent on kibana.

[dikshachauhan-qasource]

Kibana version:
7.11 BC5 Kibana Cloud environment

Host OS and Browser version:
Windows 10, All

Preconditions

1. 7.11 BC5 Kibana Cloud environment should be available.

BUILD 37827
COMMIT d801a7bb08e17368584d00fbd97a5d0006285b51
Artifact link: https://staging.elastic.co/7.11.0-903dc0b6/downloads/beats/elastic-agent/elastic-agent-7.11.0-windows-x86_64.zip

2. Default policy having system integration should exist.[No change in namespace].

Steps to reproduce:

1. Login to Kibana cloud environment.
2. Go to Add Agents flyout. Go To 'Run Standalone' tab.
3. Download elastic-agent.yml And update username/password with kibana cloud credentials.
4. RDP to endpoint. Replace elastic-agent.yml at extracted elastic-agent folder.
5. Open Powershell <run ./elastic-agent.exe install command.
6. Say no to enroll to fleet.

Expected Result:
Data Stream should be available for standalone Agent on kibana.

Logs:
elastic-agent-json.log
elastic-agent-watcher-json.log

Screenshots:

However on running './elastic-agent version' command, Below error is generated:

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[dikshachauhan-qasource]

@manishgupta-qasource Please Review

[manishgupta-qasource]

Reviewed & assigned to @EricDavisX

[ph]

Where you using the default configuration?

[dikshachauhan-qasource]

Hi @ph

Yes we have opted here for default policy.

Please find elastic-agent.yml attached here for more details.

[ph]

I've edited the file and created a gist https://gist.github.com/ph/fffc82a3a9e5670b0f0449a4316225a3

This configuration was taken from Fleet and put locally on disk?

Going to add it to triage we need to investigate it.

Looking at the screenshot it seems the elastic agent is not running.

[EricDavisX]

I believe I reproduced this on a linux host in 7.11 latest snapshot, and I see the Agent process running:
[zeus@mainqa-atlcolo-10-0-7-165 elastic-agent-7.11.0-SNAPSHOT-linux-x86_64]$ ps ax | grep elastic
2152 ? Ssl 0:01 /opt/Elastic/Agent/elastic-agent
2159 ? Zs 0:00 [elastic-agent]
2167 pts/1 R+ 0:00 grep --color=auto elastic

I'm curious what a process is about.

netstat call shows there are no connections to Elasticsearch:
[root@mainqa-atlcolo-10-0-7-165 elastic-agent-7.11.0-SNAPSHOT-linux-x86_64]# netstat -antp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 910/sshd
tcp 0 0 127.0.0.1:6789 0.0.0.0:* LISTEN 2152/elastic-agent
tcp 0 0 10.0.7.165:47674 147.75.83.237:80 TIME_WAIT -
tcp 0 0 10.0.7.165:38864 76.8.18.7:80 TIME_WAIT -
tcp 0 36 10.0.7.165:22 192.168.118.46:49862 ESTABLISHED 1799/sshd: zeus [pr
tcp6 0 0 :::22 :::* LISTEN 910/sshd

[dikshachauhan-qasource]

Hi @EricDavisX

Today, we have faced same issue on installing standalone agent using non-default policy on mac catalina OS_sip enabled on 7.11 BC5 kibana cloud build. On running below command we see Agent process running:
ps -ef | gre p 'elastic-agent'
0 541 1 0 2:53AM ?? 0:02.27 /Library/Elastic/Agent/elastic- agent
0 542 541 0 2:53AM ?? 0:00.00 (elastic-agent)
502 566 501 0 2:56AM ttys000 0:00.00 grep elastic-agent
zeus@mainqa-atlcolo-10-0-6-242 elastic-agent-7.11.0-darwin-x86_64 %

Please let us know if anything else is required.

Thanks
QAS

[EricDavisX]

The automated coverage planned for this is: elastic/e2e-testing#217

[ph]

@michalpristas Can you confirm the behavior here? I am surprised that it doesn't work.

[michalpristas]

i see this: Failed to read configuration, error: could not emit configuration: could not create the AST from the configuration: missing field accessing 'inputs' (source:'/Library/Elastic/Agent/elastic-agent.yml')

when i remove last 2 inputs and keep just first one it works looking into why

[michalpristas]

seems related to conditions, when condition is present it starts misbehaving
i think it removes whole inputs when condition fails

[michalpristas]

cc @blakerouse is it intended to do so?
using same config with test is running ok. digging deeper to see where it fails

edit:
we're failing here with conditions

c, err := config.LoadFiles(filename)
require.NoError(t, err)

m, err := c.ToMapStr() // HERE

the problem here is that it takes config as loaded from disk and in condition it looks for dynamic path of host.platform it tries to resolve it here. but the path does not exist as it is not injected at this point.
we either need to

• inject agent data here somehow
• skip config and parse yaml into map directly

i need to look around the code to see which versions is the best for us.

[michalpristas]

Injecting Agent Info at load seems viable.
I can inject paths and ECS metadata what worries me is that host.platform is not ECS compatible.

Do we have a list of variables we use in this conditions?
cc @nchaulet @fearful-symmetry

[blakerouse]

@michalpristas constraints isn't a valid key for constraints so by changing constraint to constraints you just disabled the constraint.

I think this is related to the same bug here: #23685

There seems to be a core issue with constraints, which only works because of dynamic inputs. It's all related code, and follows the same code path.

[dikshachauhan-qasource]

Hi @EricDavisX

Today, while performing testing on P1 testcases, we have observed that data streaming is not available for 7.10.2 standalone windows Agent as well on 7.11 BC 8 Kibana cloud build.

Build details:

BUILD 37897
COMMIT 3f71ce7177a41e067ddb1e670ec4ace5f6d4f5fe
Artifact link: https://www.elastic.co/downloads/past-releases/elastic-agent-7-10-2

Logs:
elastic-agent-json.log

elastic-agent(yml).txt

Screenshot:

Thanks
QAS

[EricDavisX]

With the timing of 7.11.1 BC1 and the partial build-update BC2 I'm not sure if this is in 7.11.1