Lens Migration of OBS integrations

[rameshelastic]

This is the issue that tracks the preliminary list of integrations that will be migrated to Lens from an infrastructure observability perspective. There is a list of phases for this lens migration work, it can be found from below tracker issue:

Phases and issues tracker #6489

Issue tracking the Lens Migration from obs service integrations.

Completed Integrations

• HAproxy [O11y][HAProxy] Lens Migration #5584
• Kafka [O11y][Kafka] Lens Migration #5591
• MySQL [O11y][MySQL] Lens Migration #5709
• Oracle [O11y][Oracle] Lens Migration #5753
• WebSphere Application Server [O11y][WebSphere Application Server] Lens Migration #5575
• CoreDNS [O11y][CoreDNS] Lens Migration #5708
• IIS [O11y][IIS] Lens Migration #5707
• Apache Spark [O11y][Apache Spark] Lens Migration #5491
• ActiveMQ [O11y][ActiveMQ] Lens Migration #5516
• Cassandra [O11y][Cassandra] Lens Migration #5672
• Citrix ADC [O11y][Citrix ADC] Lens Migration #5562 (Not relevant. All viz in lens already)
• Couchbase [O11y][Couchbase] Lens Migration #5609
• CouchDB [O11y][CouchDB] Lens Migration #5564
• Hadoop [O11y][Hadoop] Lens Migration #5504
• IBM MQ [O11y][IBM MQ] Lens Migration #5494
• Nagios XI [O11y][Nagios XI] Lens Migration  #5567
• Oracle WebLogic [O11y][Oracle WebLogic] Lens Migration #5547
• PHP-FPM [O11y][PHP-FPM] Lens Migration #5520
• Salesforce [O11y][Salesforce] Lens Migration #5579
• Spring Boot [O11y][Spring Boot] Lens Migration #5529
• Apache [O11y][Apache] Lens Migration #5608
  • Validate the changes in the PR is taken care.
• CockroachDB [O11y][CockroachDB] Lens Migration #5543
• etcd [O11y][ETCD] Lens Migration #5542
• InfluxDB [O11y][InfluxDb] Lens Migration #5505
• Memcached [O11y][Memcached] Lens Migration #5563 (Not relevant. All viz in lens already)
• MongoDB [O11y][MongoDB] Lens Migration #5541
• Microsoft SQL Server [O11y][Microsoft SQL Server] Lens Migration  #5710
• PostgreSQL [O11y][PostgreSQL] Lens Migration #5701
• Nginx [O11y][Nginx] Lens Migration  #5713
• RabbitMQ [O11y][RabbitMQ] Lens migration #5535
• Redis [O11y][Redis] Lens Migration #5583
• Redis Enterprise [O11y][Redis Enterprise] Lens Migration  #5674
• Traefik [O11y][Traefik] Lens Migration #5692
• Vsphere [O11y][Vsphere] Lens Migration #5676
• Zookeeper [O11y][ZooKeeper] Lens Migration #5690

Issues Identified

Lens/Kibana Issues

• Lens doesn't support number formatting. (For example 20000 -> 20k)
  [Lens] Add more number format options kibana#72711
  • Impact : [ Severity : Blocker, Workaround : No ]
• Not able to convert value into human readable format ( Seconds / Minutes / Hours )
  [Lens] Support the duration formatter kibana#153378
  • Impact : [ Severity : Blocker, Workaround : No ]
  • Comment : Kibana team has provided the workaround. We tried the solution and discussed it with @SubhrataK but it was not user-friendly. So we have provided this comment and asked the kibana team to provide this feature in the lens.
• The text type field is not supported in table type visualizations of Lens.
  Lens only shows keyword fields, not text fields kibana#96576
  • Impact : [ Severity : Blocker, Workaround : No]
  • Comment : As discussed with @agithomas, removing the text field is not the idle option. Hence we will keep the visualization as it is.
• Not having feature to set last bucket value
  [Lens] Not having feature to set last bucket value kibana#153326
  • Impact : [ Severity : Critical, Workaround : Yes ]
  • Comment : As discussed with @SubhrataK for this issue, we are good with the metrics chart to convert into the lens. If metrics charts contains differences which uses the date histogram we can convert them to line charts.
• The TagCloud visualizations are not supported in the lens.
  [Lens] Add word cloud support kibana#95542
  • Impact : [ Severity : Critical, Workaround : Yes (Create horizontal bar chart[reference]])
  • Comment : As per this comment, we have converted tag cloud visualization to horizontal bar chart
• Splitting a chart into multiple smaller charts by a breakdown dimension is not supported in the lens.
  [Lens] Support small multiples / facet / trellis for XY and Partition charts kibana#72063
  • Impact : [ Severity : Critical, Workaround : Yes(Convert to table chart)]
  • Comment : As discussed with @SubhrataK, we have converted split chart to table chart.
• Facing issue related to aggregation based gauge chart to lens gauge chart while migrating.
  [Lens Gauge chart] Gauge chart is not showing value if it exceeds than the provided limit like aggregation based gauge chart kibana#153010
  • Impact : [ Severity : Major, Workaround : Yes (Converting gauge to metric)]
  • Comment : As discussed with @SubhrataK,, we can convert gauge charts to metric. Also Kibana team will work on gauge charts as per this comment
• Gradient and rainbow color palettes are not available in lens and won't be available in the future versions as well. Hence, the closest color palettes (positive and negative) have been used.
  [Lens] Support for Gradient and Rainbow color palette in Lens kibana#152699
  • Impact : [ Severity : Low, Workaround : Yes (Use alternative color palettes)]
  • Comment : As discussed with @agithomas, we will keep the closest color palettes while doing the lens migration.
• TSVB visualization doesn’t show negative values in UI. But the lens shows both negative and positive values. (This will be fixed in 8.8)
  The chart do not render negative values elastic-charts#135
  • Impact : [ Severity : Low, Workaround : This will be fixed in 8.8 ]
  • Comment : This has been identified as a better functionality. In lens user can see both values negative and positive.
• Support multiple metrics in one chart using lens
  [Lens] Support multiple metrics in one chart using lens kibana#153324
  • Impact : [ Severity : Low, Workaround : Yes (Convert into multiple metrics chart) ]
  • Comment : We are able to convert this visualization to single metric charts. So as a workaround, we have converted into single metrics charts.
• In input control visualization, we can make dependency filters that strictly depend on each other but in new controls, dependency filtering is possible but filters are not required to be strict. As per the comment, this is the expected behavior of new controls.
  • Impact : [ Severity : Low, Workaround : No (Expected Behaviour) ]

Integration Issues

• Kafka Update layout of dashboards
  [O11y][Kafka] Update layout of dashboards #7148
• CoreDNS dashboard visualizations are showing error
  [O11y][CoreDNS] Visualizations are showing error #5518
  • Impact : [ Severity : Critical, Workaround : No ]
• The InfluxDB dashboard panel contains an error.
  [InfluxDB] Shard disk size panel from Status metrics dashboard contains error #5506
  • Impact : [ Severity : Critical, Workaround : No]
• SpringBoot dashboards contains meta field (_id)
  [O11y][SpringBoot] Update _id field in SpringBoot dashboard #5871
  • Impact : [ Severity : Critical, Workaround : As mentioned in the documentation of _id, the content of _id field can be copied in another field using ingest pipeline and the new field can be used in visualizations.]

  Note: When user will upgrade to the newer version, spring_boot.audit_events.document_id field will be added only to the new incoming documents. The data that is already present in indices will not get updated. Hence, the Audit Events [Logs Spring Boot] visualization will show documents which have spring_boot.audit_events.document_id field present (documents that have been ingested after upgrading to the latest version). Reindexing can be recommended in case the user wish to see the old information following the change (adoption of document_id) as the new field to store the values presently stored under _id. Please refer to the troubleshooting section of README.

• Salesforce dashboards contains meta field (_id)
  [O11y][Salesforce] Update _id field in Salesforce dashboard #5876
  • Impact : [ Severity : Critical, Workaround : As mentioned in the documentation of _id, the content of _id field can be copied in another field using ingest pipeline and the new field can be used in visualizations.]

  Note: When user will upgrade to the newer version, salesforce.login.document_id field will be added only to the new incoming documents. The data that is already present in indices will not get updated. Hence, the Changes made in the setup [Logs Salesforce] visualization will show documents which have salesforce.login.document_id field present (documents that have been ingested after upgrading to the latest version). Reindexing can be recommended in case the user wish to see the old information following the change (adoption of document_id) as the new field to store the values presently stored under _id. Please refer to the troubleshooting section of README.(https://github.com/elastic/integrations/tree/main/packages/salesforce#readme).

Migration stats

Name of the package	No of dashboards	Migration Percentage	Total Panels	Failed Panels	Comment	Before Migration Kibana version	After Migration Kibana version
Apache Spark	1	100%	20	0		8.1.0	8.1.0
ActiveMQ	5	100%	28	0		8.2.0	8.2.0
Cassandra	2	100%	23	0		7.15.0 8.0.0	8.3.0
Citrix ADC	-	-	-	-	-	-	-
Couchbase	4	100%	36	0		8.3.0	8.4.0
CouchDB	1	100%	8	0		8.3.0	8.4.0
Hadoop	4	100%	27	0		8.1.0	8.3.0
IBM MQ	4	97%	38	1	#5662	8.2.0	8.5.0
Nagios XI	4	100%	27	0		8.2.0	8.2.0
Oracle WebLogic	3	100%	27	0		8.4.0	8.4.0
PHP-FPM	2	100%	11	0		8.1.0	8.4.0
Salesforce	4	100%	34	0		8.5.0	8.5.0
Spring Boot	4	100%	18	0		8.1.0	8.1.0
WebSphere Application Server	4	100%	20	0		8.1.0	8.4.0
Apache	2	100%	20	0	elastic/kibana#153378	8.0.0	8.3.0
CockroachDB	1	100%	7	0		7.14.0 8.0.0	8.4.0
etcd	1	100%	8	0		7.15.0 8.0.0	8.3.0
CoreDNS	1	100%	4	0		8.0.0	8.0.0
HAproxy	7	100%	33	0		8.0.0	8.6.0
IIS	5	100%	55	0		8.3.0	8.5.0
InfluxDB	2	100%	27	0		8.3.0	8.3.0
Kafka	2	100%	15	0		7.14.0 8.0.0	8.6.0
Memcached	-	-	-	-	-	-	-
MongoDB	2	100%	11	0		7.14.0 8.0.0	8.1.0
MySQL	2	100%	21	0		7.14.0 8.0.0	8.6.0
Microsoft SQL Server	4	100%	24	0		8.3.0	8.4.0
Oracle	6	100%	66	0		8.3.0	8.6.0
PostgreSQL	3	100%	13	0		7.14.0 8.0.0	8.4.0
Nginx	3	100%	23	0		8.0.0	8.5.0
RabbitMQ	1	100%	4	0		7.14.0 8.0.0	8.0.0
Redis	3	100%	17	0		7.14.0 8.0.0	8.3.0
Redis Enterprise	1	100%	12	0		8.2.0	8.3.0
Traefik	1	100%	6	0	elastic/kibana#124279	8.0.0	8.0.0
Vsphere	2	100%	12	0		7.15.0 8.0.0	8.7.0
Zookeeper	1	100%	5	0		7.14.0 8.0.0	8.2.0

Backlog

Name of the Package	Dashboard Name - Visualisation Name	Link to the issue	Issue category
InfluxDB	[InfluxDB Metrics] Status metrics - Shard disk size [InfluxDB Metrics]	#5506	Integration
CoreDNS	[Filebeat CoreDNS] Overview - Top Domains [CoreDNS] [Filebeat CoreDNS] Overview - Unique Domains [CoreDNS]	#5518	Integration
Cassandra	[Metrics Cassandra] Overview - Disk Usage [Metrics Cassandra] [Metrics Cassandra] Overview - Cache [Metrics Cassandra] [Metrics Cassandra] Overview - Heap Memory [Metrics Cassandra]  [Metrics Cassandra] Overview - Read Latency [Metrics Cassandra]  [Metrics Cassandra] Overview - Write Latency [Metrics Cassandra] [Metrics Cassandra] Overview - Range Slice Latency [Metrics Cassandra]  [Metrics Cassandra] Overview - Tasks [Metrics Cassandra] [Metrics Cassandra] Overview - CAS Read/Write Latency [Metrics Cassandra] [Metrics Cassandra] Overview - Read/Write Failures [Metrics Cassandra] [Metrics Cassandra] Overview - Dropped Message [Metrics Cassandra] [Metrics Cassandra] Overview - Read/Write Count [Metrics Cassandra] [Metrics Cassandra] Overview - Storage Exception [Metrics Cassandra] [Metrics Cassandra] Overview - Unavailable Requests [Metrics Cassandra] [Metrics Cassandra] Overview - Request Timeouts [Metrics Cassandra]	#5531	Integration
Apache	[Metrics Apache] Overview - Uptime [Metrics Apache]	elastic/kibana#153378	Lens / Kibana
Redis	[Metrics Redis] Overview - Keyspaces [Metrics Redis] [Logs Redis] Logs Overview - Logs level and roles breakdown [Logs Redis]	#5671 #5773	Integration
IBM MQ	[Logs IBM MQ] Overview of error log - Top 5 Errors [Logs IBM MQ]	#5662	Lens/ Kibana
Hadoop	[Metrics Hadoop] Application - The number of virtual cores the application has allocated [Metrics Hadoop] [Metrics Hadoop] Application - The amount of memory the application has allocated [Metrics Hadoop]	#5628	Integration
IIS	Revisit colors used in panels	#5743	Integration
Cassandra	Update service.address field	#5766	Integration
MongoDB	Revisit dashboards	#5846	Integration
Vsphere	Change format of value to bytes	#5820	Integration
PostgreSQL	Add label in Y-axis	#5797	Integration
IIS	Change format of value to bytes	#5796	Integration
Nginx	Add label in Y-axis	#5795	Integration
Redis	Add description in Panels	#5723	Integration
SpringBoot	Update _id field in visualizations	#5871	Integration
Salesforce	Update _id field in visualizations	#5876	Integration
HAProxy	Re-visit Traffic volume [Metrics HAProxy] panel from [Metrics HAProxy] Frontend dashboard	#5881	Integration
ActiveMQ	[Logs ActiveMQ] Audit - Audit Account Tag Cloud [Logs ActiveMQ]	elastic/kibana#95542	Lens/ Kibana
Oracle WebLogic	[Logs Oracle WebLogic] Logs Overview - Request Endpoint [Logs Oracle WebLogic]	elastic/kibana#95542	Lens/ Kibana
OBS IntegrationS	Re-visit gauge chart visualizations	elastic/kibana#155131	Lens/ Kibana
SpringBoot	Update meta fields in visualization	#6012	Lens/ Kibana
Salesforce	Update meta fields in visualization	#6065	Lens/ Kibana

[milan-elastic]

Please find an Observability integration list that we are planning to Migrate to the Lens type visualization below.

Integration	Maps	Search	Lens	Visualization	Tag Cloud
ActiveMQ	0	2	0	26	1
ApacheSpark	0	0	2	18	0
Cassandra	0	1	0	22	0
Citrix ADC	0	0	28	0	0
Couchbase	0	0	25	11	0
CouchDB	0	0	0	8	0
Hadoop	0	0	6	21	0
IBM MQ	0	1	0	37	0
Nagios XI	0	1	21	5	0
Oracle WebLogic	0	0	16	11	0
PHP-FPM	0	0	10	1	0
Salesforce	2	0	11	2	0
Spring Boot	0	0	15	3	0
WebSphere Application Server	0	0	17	3	0
Apache	1	1	0	18	0
CockroachDB	0	0	0	7	0
CoreDNS	0	0	0	4	0
etcd	0	0	0	8	0
HAProxy	1	0	0	32	0
IIS	1	0	0	55	0
InfluxDB	0	0	0	27	0
Kafka	0	2	0	11	0
Memcached	0	0	7	0	0
MongoDB	0	2	0	9	0
MySQL	0	1	0	21	0
Microsoft SQLserver	0	3	13	8	0
Nginx	1	2	0	20	0
Oracle	0	0	0	66	0
PostgreSQL	0	3	0	13	0
RabbitMQ	0	0	0	4	0
Redis	0	2	0	15	0
Redis Enterprise	0	0	0	12	0
Traefik	1	0	0	5	0
VSphere	0	0	0	12	0
Zookeeper	0	0	0	5	0

Also, for more insights, challenges and overall approach find the attached document here: https://docs.google.com/document/d/18H7rvQkB5FVOql58HnzRx4azUiDAGGFY5fGTKmkxXIo/edit#

[muthu-mps]

@milan-elastic - Are we including the control panel(deprecated) update change in dashboards as part of lens migration ?

[milan-elastic]

@milan-elastic - Are we including the control panel(deprecated) update change in dashboards as part of lens migration ?

@muthu-mps , Input controls are deprecated which we are migrating into Lens control panels.