Add analysis components and mapping types to the usage API. #51031

jpountz · 2020-01-15T11:08:07Z

Knowing about used analysis components and mapping types would be incredibly
useful in order to know which ones may be deprecated or should get more love.

Some field types also act as a proxy to know about feature usage of some APIs
like the percolator or completion fields types for percolation and the
completion suggester, respectively.

Knowing about used analysis components and mapping types would be incredibly useful in order to know which ones may be deprecated or should get more love. Some field types also act as a proxy to know about feature usage of some APIs like the `percolator` or `completion` fields types for percolation and the completion suggester, respectively.

elasticmachine · 2020-01-15T11:08:10Z

Pinging @elastic/es-core-features (:Core/Features/Stats)

jimczi

I left some minor comments regarding the extracted components but it looks great !

jimczi · 2020-01-15T12:35:31Z

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/oss/IndexUsageTransportAction.java

+        for (IndexMetaData indexMetaData : state.metaData()) {
+            MappingMetaData mappingMetaData = indexMetaData.mapping();
+            if (mappingMetaData != null) {
+                populateFieldTypesFromObject(mappingMetaData.sourceAsMap(), usedFieldTypes);


Should we also extract the analyzers set on the fields ? This could be useful to check the usage of the pre-built analyzers ?

jimczi · 2020-01-15T12:36:35Z

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/oss/IndexUsageTransportAction.java

+            aggregateAnalysisTypes(tokenFilterSettings.values(), usedTokenFilters);
+
+            Map<String, Settings> analyzerSettings = indexSettings.getGroups("index.analysis.analyzer");
+            aggregateAnalysisTypes(analyzerSettings.values(), usedAnalyzers);


Should we also extract the pre-built tokenizer, filters and char_filters from custom analyzers ?

martijnvg

+1 great to have this kind of information available!

jpountz · 2020-01-15T15:22:52Z

@jimczi I added built-in analysis components.

jimczi

I added built-in analysis components.

LGTM, thanks!

We added tracking of index feature usage in elastic#51031 but due to some copy and paste errors the test fails on some seeds. This fixes those errors.

We added tracking of index feature usage in #51031 but due to some copy and paste errors the test fails on some seeds. This fixes those errors.

We added tracking of index feature usage in elastic#51031 but due to some copy and paste errors the test fails on some seeds. This fixes those errors.

We added tracking of index feature usage in #51031 but due to some copy and paste errors the test fails on some seeds. This fixes those errors.

…51031) Knowing about used analysis components and mapping types would be incredibly useful in order to know which ones may be deprecated or should get more love. Some field types also act as a proxy to know about feature usage of some APIs like the `percolator` or `completion` fields types for percolation and the completion suggester, respectively.

We added tracking of index feature usage in elastic#51031 but due to some copy and paste errors the test fails on some seeds. This fixes those errors.

codebrain · 2020-04-16T06:46:02Z

This doesn't appear to be in 7.7.0, running against latest snapshot (build-50109232) I get the following:

Create an index with the following settings:

PUT https://127.0.0.1:9200/text-index
{"settings":{"analysis":{"char_filter":{"c":{"mappings":["a => b"],"type":"mapping"}}}}}

Then calling GET https://127.0.0.1:9200/_xpack/usage gives the following:

HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 7569

{
  "flattened" : {
    "available" : true,
    "enabled" : true,
    "field_count" : 2
  },
  "security" : {
    "available" : true,
    "enabled" : true,
    "realms" : {
      "file" : {
        "name" : [
          "file1"
        ],
        "available" : true,
        "cache" : [
          {
            "size" : 1
          }
        ],
        "size" : [
          2
        ],
        "enabled" : true,
        "order" : [
          0
        ]
      },
      "ldap" : {
        "available" : false,
        "enabled" : false
      },
      "native" : {
        "name" : [
          "native1"
        ],
        "available" : true,
        "cache" : [
          {
            "size" : 0
          }
        ],
        "size" : [
          0
        ],
        "enabled" : true,
        "order" : [
          2
        ]
      },
      "saml" : {
        "available" : false,
        "enabled" : false
      },
      "kerberos" : {
        "available" : false,
        "enabled" : false
      },
      "oidc" : {
        "available" : false,
        "enabled" : false
      },
      "active_directory" : {
        "available" : false,
        "enabled" : false
      },
      "pki" : {
        "available" : false,
        "enabled" : false
      }
    },
    "roles" : {
      "native" : {
        "size" : 0,
        "fls" : false,
        "dls" : false
      },
      "dls" : {
        "bit_set_cache" : {
          "count" : 0,
          "memory" : "0b",
          "memory_in_bytes" : 0
        }
      },
      "file" : {
        "size" : 3,
        "fls" : false,
        "dls" : false
      }
    },
    "role_mapping" : {
      "native" : {
        "size" : 0,
        "enabled" : 0
      }
    },
    "ssl" : {
      "http" : {
        "enabled" : true
      },
      "transport" : {
        "enabled" : true
      }
    },
    "token_service" : {
      "enabled" : true
    },
    "api_key_service" : {
      "enabled" : true
    },
    "audit" : {
      "enabled" : false
    },
    "ipfilter" : {
      "http" : false,
      "transport" : false
    },
    "anonymous" : {
      "enabled" : false
    },
    "fips_140" : {
      "enabled" : false
    }
  },
  "monitoring" : {
    "available" : true,
    "enabled" : true,
    "collection_enabled" : false,
    "enabled_exporters" : {
      "local" : 1
    }
  },
  "vectors" : {
    "available" : true,
    "enabled" : true,
    "dense_vector_fields_count" : 0,
    "sparse_vector_fields_count" : 0,
    "dense_vector_dims_avg_count" : 0
  },
  "watcher" : {
    "available" : false,
    "enabled" : true,
    "execution" : {
      "actions" : {
        "_all" : {
          "total" : 0,
          "total_time_in_ms" : 0
        }
      }
    },
    "watch" : {
      "input" : {
        "_all" : {
          "total" : 0,
          "active" : 0
        }
      },
      "trigger" : {
        "_all" : {
          "total" : 0,
          "active" : 0
        }
      }
    },
    "count" : {
      "total" : 0,
      "active" : 0
    }
  },
  "analytics" : {
    "available" : true,
    "enabled" : true,
    "stats" : [
      {
        "boxplot_usage" : 0,
        "cumulative_cardinality_usage" : 0,
        "string_stats_usage" : 0,
        "top_metrics_usage" : 0
      }
    ]
  },
  "ml" : {
    "available" : false,
    "enabled" : true,
    "jobs" : {
      "_all" : {
        "count" : 0,
        "detectors" : {
          "total" : 0.0,
          "min" : 0.0,
          "avg" : 0.0,
          "max" : 0.0
        },
        "created_by" : { },
        "model_size" : {
          "total" : 0.0,
          "min" : 0.0,
          "avg" : 0.0,
          "max" : 0.0
        },
        "forecasts" : {
          "total" : 0,
          "forecasted_jobs" : 0
        }
      }
    },
    "datafeeds" : {
      "_all" : {
        "count" : 0
      }
    },
    "data_frame_analytics_jobs" : {
      "_all" : {
        "count" : 0
      }
    },
    "inference" : {
      "ingest_processors" : {
        "_all" : {
          "num_docs_processed" : {
            "max" : 0,
            "sum" : 0,
            "min" : 0
          },
          "pipelines" : {
            "count" : 0
          },
          "num_failures" : {
            "max" : 0,
            "sum" : 0,
            "min" : 0
          },
          "time_ms" : {
            "max" : 0,
            "sum" : 0,
            "min" : 0
          }
        }
      },
      "trained_models" : {
        "_all" : {
          "count" : 0
        }
      }
    },
    "node_count" : 1
  },
  "ilm" : {
    "policy_count" : 4,
    "policy_stats" : [
      {
        "phases" : {
          "hot" : {
            "min_age" : 0,
            "actions" : [
              "rollover"
            ]
          },
          "delete" : {
            "min_age" : 7776000000,
            "actions" : [
              "delete"
            ]
          }
        },
        "indices_managed" : 0
      },
      {
        "phases" : {
          "delete" : {
            "min_age" : 604800000,
            "actions" : [
              "delete"
            ]
          }
        },
        "indices_managed" : 0
      },
      {
        "phases" : {
          "hot" : {
            "min_age" : 0,
            "actions" : [
              "rollover"
            ]
          }
        },
        "indices_managed" : 0
      },
      {
        "phases" : {
          "hot" : {
            "min_age" : 0,
            "actions" : [
              "rollover"
            ]
          },
          "delete" : {
            "min_age" : 7776000000,
            "actions" : [
              "delete"
            ]
          }
        },
        "indices_managed" : 0
      }
    ]
  },
  "slm" : {
    "available" : true,
    "enabled" : true
  },
  "voting_only" : {
    "available" : true,
    "enabled" : true
  },
  "frozen_indices" : {
    "available" : true,
    "enabled" : true,
    "indices_count" : 0
  },
  "logstash" : {
    "available" : false,
    "enabled" : true
  },
  "ccr" : {
    "available" : false,
    "enabled" : true,
    "follower_indices_count" : 0,
    "auto_follow_patterns_count" : 0
  },
  "graph" : {
    "available" : false,
    "enabled" : true
  },
  "sql" : {
    "available" : true,
    "enabled" : true,
    "features" : {
      "having" : 0,
      "subselect" : 0,
      "limit" : 0,
      "orderby" : 0,
      "where" : 0,
      "groupby" : 0,
      "join" : 0,
      "command" : 0,
      "local" : 0
    },
    "queries" : {
      "cli" : {
        "total" : 0,
        "paging" : 0,
        "failed" : 0
      },
      "rest" : {
        "total" : 0,
        "paging" : 0,
        "failed" : 0
      },
      "canvas" : {
        "total" : 0,
        "paging" : 0,
        "failed" : 0
      },
      "odbc" : {
        "total" : 0,
        "paging" : 0,
        "failed" : 0
      },
      "jdbc" : {
        "total" : 0,
        "paging" : 0,
        "failed" : 0
      },
      "odbc32" : {
        "total" : 0,
        "paging" : 0,
        "failed" : 0
      },
      "odbc64" : {
        "total" : 0,
        "paging" : 0,
        "failed" : 0
      },
      "translate" : {
        "count" : 0
      },
      "_all" : {
        "total" : 0,
        "paging" : 0,
        "failed" : 0
      }
    }
  },
  "enrich" : {
    "available" : true,
    "enabled" : true
  },
  "transform" : {
    "available" : true,
    "enabled" : true
  },
  "spatial" : {
    "available" : true,
    "enabled" : true
  },
  "rollup" : {
    "available" : true,
    "enabled" : true
  },
  "eql" : {
    "available" : true,
    "enabled" : false
  }
}

Notice there is no index property in the response JSON.

jpountz · 2020-04-16T21:57:52Z

@codebrain Good catch, it's been superseded by #51138. I removed the version labels and will remove from the release notes tomorrow.

It's been superseded by elastic#51138.

jpountz · 2020-04-17T11:55:08Z

Here is the PR #55387.

It's been superseded by #51138.

jpountz added >enhancement :Data Management/Stats Statistics tracking and retrieval APIs labels Jan 15, 2020

jimczi reviewed Jan 15, 2020

View reviewed changes

martijnvg approved these changes Jan 15, 2020

View reviewed changes

Review

5c6278f

jimczi approved these changes Jan 15, 2020

View reviewed changes

Merge branch 'master' into usage_mappings_analysis

5214ae1

jpountz added v7.7.0 v8.0.0 labels Jan 15, 2020

jpountz merged commit af59ad7 into elastic:master Jan 15, 2020

jpountz deleted the usage_mappings_analysis branch January 15, 2020 17:07

jpountz mentioned this pull request Jan 17, 2020

Move information about usage of field types and analysis components to cluster stats #51138

Closed

nik9000 added a commit to nik9000/elasticsearch that referenced this pull request Jan 17, 2020

Fix flaky usage tracking test

3f82005

We added tracking of index feature usage in elastic#51031 but due to some copy and paste errors the test fails on some seeds. This fixes those errors.

nik9000 mentioned this pull request Jan 17, 2020

Fix flaky usage tracking test #51169

Merged

nik9000 added a commit that referenced this pull request Jan 17, 2020

Fix flaky usage tracking test (#51169)

bae12cb

We added tracking of index feature usage in #51031 but due to some copy and paste errors the test fails on some seeds. This fixes those errors.

nik9000 mentioned this pull request Jan 17, 2020

Fix flaky usage tracking test (backport of #51169) #51179

Merged

nik9000 added a commit that referenced this pull request Jan 17, 2020

Fix flaky usage tracking test (#51169) (#51179)

977b53a

We added tracking of index feature usage in #51031 but due to some copy and paste errors the test fails on some seeds. This fixes those errors.

codebrain mentioned this pull request Apr 1, 2020

7.7.0 meta ticket (Part 2) elastic/elasticsearch-net#4533

Closed

codebrain mentioned this pull request Apr 14, 2020

7.7.0 meta ticket elastic/elasticsearch-net#4525

Closed

38 tasks

jpountz removed v7.7.0 v8.0.0 labels Apr 16, 2020

jpountz added a commit to jpountz/elasticsearch that referenced this pull request Apr 17, 2020

Remove elastic#51031 from the changelog.

e125dd2

It's been superseded by elastic#51138.

jpountz added a commit that referenced this pull request Apr 18, 2020

Remove #51031 from the changelog. (#55387)

9fbb340

It's been superseded by #51138.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add analysis components and mapping types to the usage API. #51031

Add analysis components and mapping types to the usage API. #51031

jpountz commented Jan 15, 2020

elasticmachine commented Jan 15, 2020

jimczi left a comment

jimczi Jan 15, 2020

jimczi Jan 15, 2020

martijnvg left a comment

jpountz commented Jan 15, 2020

jimczi left a comment

codebrain commented Apr 16, 2020 •

edited

Loading

jpountz commented Apr 16, 2020

jpountz commented Apr 17, 2020

Add analysis components and mapping types to the usage API. #51031

Add analysis components and mapping types to the usage API. #51031

Conversation

jpountz commented Jan 15, 2020

elasticmachine commented Jan 15, 2020

jimczi left a comment

Choose a reason for hiding this comment

jimczi Jan 15, 2020

Choose a reason for hiding this comment

jimczi Jan 15, 2020

Choose a reason for hiding this comment

martijnvg left a comment

Choose a reason for hiding this comment

jpountz commented Jan 15, 2020

jimczi left a comment

Choose a reason for hiding this comment

codebrain commented Apr 16, 2020 • edited Loading

jpountz commented Apr 16, 2020

jpountz commented Apr 17, 2020

codebrain commented Apr 16, 2020 •

edited

Loading