Ingest processor fails if host.ip contains an array of IPs
#46193
Comments
rwaight
added
>bug
:Data Management/Ingest Node
|
Pinging @elastic/es-core-features |
|
So the ip addresses are then provided as json array in the source field (the |
|
I think an acceptable implementation would be to loop over addresses, and simply decorate the event based on the first IP that has a match in the DB. The most likely scenario this will happen is in situations specifically like |
|
I am opting for a geoip feature where the ingest would iterates over an array of IPs and enrich the document's geoip field with an array of geoip information. (Furthermore opening the possibility to point to point paths in visualizations. As well opening up the possibility for computed searches shortest paths, triangulations, etc..) |
This change lets you use array of IPs in addition to string in geoip processor source field. It will set array containing geoip data for each element in source, unless first_only parameter option is enabled, then only first found will be returned. Closes elastic#46193
Set `netinfo.enabled: false` until `elastic/elasticsearch#46193` is resolved. Update version.
|
Hi @probakowski, thank you for opening the PR! Would it be possible to backport to |
Comment out the `output.elasticsearch.pipeline: geoip-info` setting until `elastic/elasticsearch#46193` is resolved.
|
@rwaight Now that 7.5 is released I doubt there will be any further releases of the 7.4 branch |
|
Thanks for clarifying this with me, @jbaiera! I'll keep an eye on this for release in |
* Allow list of IPs in geoip ingest processor This change lets you use array of IPs in addition to string in geoip processor source field. It will set array containing geoip data for each element in source, unless first_only parameter option is enabled, then only first found will be returned. Closes #46193
* Allow list of IPs in geoip ingest processor This change lets you use array of IPs in addition to string in geoip processor source field. It will set array containing geoip data for each element in source, unless first_only parameter option is enabled, then only first found will be returned. Closes elastic#46193
* Allow list of IPs in geoip ingest processor This change lets you use array of IPs in addition to string in geoip processor source field. It will set array containing geoip data for each element in source, unless first_only parameter option is enabled, then only first found will be returned. Closes #46193
* Allow list of IPs in geoip ingest processor This change lets you use array of IPs in addition to string in geoip processor source field. It will set array containing geoip data for each element in source, unless first_only parameter option is enabled, then only first found will be returned. Closes elastic#46193
|
I'm still seeing a similar issue in 7.6.0. |
|
@robertdeheer would you mind sharing exact error/stacktrace? and it works as expected |
|
Yes, the code you provided worked. I realized that my issue is probably
related to locating hosts in a private network and visualizing them in
kibana. That’s a different issue.
Thanks,
Rob
…On Tue, Mar 10, 2020 at 12:49 PM Przemko Robakowski < ***@***.***> wrote:
@robertdeheer <https://github.com/robertdeheer> would you mind sharing
exact error/stacktrace?
I've just tested it on 7.6.0 with:
PUT _ingest/pipeline/testpipe
{
"processors": [
{
"geoip": {
"field": "ip",
"target_field": "geo",
"ignore_missing": true
}
}
]
}
POST _ingest/pipeline/testpipe/_simulate
{
"docs": [
{
"_source": {
"ip": ["192.168.1.1", "151.101.2.217"]
}
}
]
}
and it works as expected
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#46193?email_source=notifications&email_token=AD22AX3N46Q6XSOIOJZHFZLRG2KWJA5CNFSM4ISPSR6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOM4PGQ#issuecomment-597280666>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AD22AXY3V5W6LO3ZZTUF5DLRG2KWJANCNFSM4ISPSR6A>
.
|
Elasticsearch version: 7.3.1 running on Elastic Cloud
Description of the problem including expected versus actual behavior:
When attempting to enrich events with geoIP information in beats, with
netinfo.enabled: truedefined, the Elasticsearch Ingest processor fails ifhost.ipcontains an array of IPs. The expected behavior is that an array of IPs can be ingested into Elasticsearch. The current workaround is to disable netinfo (i.e.,netinfo.enabled: false)Steps to reproduce:
Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.
Configure adding the geoip-info ingest pipeline in order to enrich events with geoIP information
Configure the Elasticsearch output to use the
geoip-infopipeline (i.e.output.elasticsearch.pipeline: geoip-info)Configure a top-level processor in *beats (this impacts auditbeat, filebeat, heartbeat, metricbeat, packetbeat, and winlogbeat) to add geo information, with
netinfo.enabled: trueconfiguration from `packetbeat.yml`
Provide logs (if relevant):
more of the same `Bulk item insert failed` logs
When not in DEBUG mode, the only response generated is:
The text was updated successfully, but these errors were encountered: