-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add crawler metrics into the stats metricset for Enterprise Search #28790
Conversation
CI is expected to fail since the latest Enterprise Seaerch snapshot used for integration testing does not have the crawler stats code (since it is very old due to the breakdown of the unified snapshot process). |
Pinging @elastic/integrations (Team:Integrations) |
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
💚 Flaky test reportTests succeeded. 🤖 GitHub commentsTo re-run your PR in the CI, just comment with:
|
/test |
…figs to align with recent deprecations
Ok, this is ready for review/merging |
@@ -9,4 +9,4 @@ COPY docker-entrypoint-dependencies.sh /usr/local/bin/ | |||
ENTRYPOINT ["tini", "--", "/usr/local/bin/docker-entrypoint-dependencies.sh"] | |||
|
|||
HEALTHCHECK --interval=1s --retries=300 --start-period=60s \ | |||
CMD curl --user elastic:changeme --fail --silent http://localhost:3002/api/as/v1/internal/health | |||
CMD curl --user elastic:changeme --fail --silent http://localhost:3002/api/ent/v1/internal/health |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The old app-search scoped API has been deprecated and removed from 8.0
- "secret_management.encryption_keys=[4a2cd3f81d39bf28738c10db0ca782095ffac07279561809eecc722e0c20eb09]" | ||
- "kibana.host=http://localhost:5601" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is needed to make Enterprise Search start, we don't actually need Kibana there
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work! Just minor spelling suggestions.
@@ -19,5 +19,5 @@ func init() { | |||
// AssetEnterprisesearch returns asset data. | |||
// This is the base64 encoded zlib format compressed contents of module/enterprisesearch. | |||
func AssetEnterprisesearch() string { | |||
return "eJzsnEuP47gRx+/9KQp9SoB0B/u69GGByewGs4vMYIGZzR6CQEuRZZtjidTwYbf30wckJZu2KFmSZXeQdB8WCz+qfv9isVik6HmANe6eAIVBVSmuUSNRdHUHYLgp8Anuf9y/BR/9e/d3AAoLJBqfIEdD7gA0GsPFUj/Bv+61Lu7/AvcrY6r7f98BMNRU8cpwKZ7g+zsAgJZNKCWzBd4BLDgWTD/5jz2AICUm6dyf2VX4BEslbVW/knDl/n4/NfA7UCkM4UJDiUZxqoEIBiskhVkBFwupSuKswEKqNm1tOEaNcWlhtUGVWcvZ/s0Gd427rVTx60fQb8N34ddff/rBOzcrhB8Log2ngb0xD1YjA6L9JxgxBLSRCtPEjy3IoLWFF0fT/Z0OdBK6PZwt66exilHcf4/e6I5Vy/U7qY03cAhWC0ZIdrCfJNig0lyKJMRpRAaor83FiXRiIBWOo5DYMkfVers/MCPIgn2wmoulj5rGkgjDafMJ93qAT0cuhs0tL1i2Inp1KfAbsIJ/sRhMgjPZM64VoWuyPDO0lZIUtZ5raGtzFwxtxVNhCECFFMtzMfqlJojrQwuTC22IoB2xiXFsZXiJjxrpLFTB3CxkC15gjlECnnKlhq8F9i5d0x0cljkyhgz+Xntqwx3+uoYVzg7t2UB2BjMa4gGsvWgKtSHKZFRakYroeMgPoYTIBbgR1z2EsCIMjIQcGwxkwCy617QsEbjWFvUwIc5ZprmgmLlVMastdiTweFmfXPp6+z7uzkeftNo9/OnhK+ALELhBdVD55/7a9HlTTqtLP//zfXuJhQHFJ73Qwdg67fw3S4lvQ4wEZUVnp9Qzy5fddWfQ/P6ZbAgsicrJEoHKokDqJ3nd102cyQdDM86YT9KQoll65SKJLhVwsZHUVyodJWJIMzfbVtha1IZq8aW+TAVlqhpf7XWFwrgE8G1Dh6ypUhoZJZZS7TKr4yX/FH5Qyrz3lqDL0pD8WCGpMi64ecx3BicHNKxJT9Bn5HQ1q8D5DdMu3/kYuunIRbDyOCwpvABn4yUEzMFOZVlyY24u4G3j12O4wneJipI835j/PXkO5F2DMIxeSJG91CT4IMXDfiJwUtSl4dK02kt6qdTa69oDzKRM5p+RmqxCwbhYZgsuSMH/IDMvb993fAjgB66rguzCUQWpKiWfeUkMRmthQNS+392uOF1BTAlcQ03fI3nfIq4UEpYK+YhF4lMw0rPRg6G9hFUKZ4rx22ArClzBN9goHpgOjGCZVDMbT/AwEqskz/PsnpCs47CAT/Go/3AzqNmG1Nuryn1nS7Rr3dEMBDauA8pqS1fpEOsAAlVIHC0R7K+ularhk5I64PfZqMi2aB0tDdx3vA1fnrb32Eq1RnXhtGwIuo0N2qhLWWSa/5Fq/2D0qP0WWLxVcFYH5g+hhm/mQThsxoPRJj5DSTaEFyQvZodp7LZ5GtfaEBMP46zH0N74Y/T5vmNoKoXwW5WJh4UuDaqC0L33g0HQNtc7bbBsI3VhxWifZZ750/3LJk8f4WeZ1w8QkoR9lDHplnDDO9Nkah4dc+rGS32oVG8hO0tfi1Gq9fUZ62W/2EGOjnUspRtzx9Lff3aNewv0bwrJmsmtSLG61tK5So06nBn5mJlhgaariMCQ4La4DwEOxgMwF37Z8//vs/ZMUGPIhS2KKyE606B3gl5OyQVVWKIw5FqwkYeZmCtUJdeaS9GVtJcyRx4mMsctwMzVNKxx3rK+rIris1GEmixYzDpYz/O2mD8aYqxOPw1xPo3ct1Zex6UFoafHgkmp8NshxkNaLTh6TKl3V0pLZ3pgyxUTfbFoMWNYtXrpucC4wXLqchmTctbZG16MyFLt4VmisPmisqzcqtC1/boULqzkjZPTM+zRuJqukNniurh7JyNx93PW5i9Rf7TNH15r0GsN6iR9rUH/PzWosnnB9eqmBaj2+Vp7XmtPi/S19vzv1p79A1ljTkvIwJO/nwSVpUuwd58+/QIKv1jUprmBMvasrz6b6dhEj9ifHlMdzPqtKfcXiqfuT6/7SOuEV4OsUIRLRv13Cm/8oOk9eealLSNyKkUdmrSI8xImrpd+es0i6vQR1OTBGKFk/wQejSuEl6X9h2AEjCKLBadd0xAGprpCinxz87sAp8Pgv7iHae4CzJ5HGsWtb3OklTqQq6k8HtOsQpVdcH10tOI3G1RkiZeMboUKNFIpJm/y9wP9XyN/4JBPkn4Yeb82Z8yqrisdI0rNm+VS4ZIYPF74G+MzLLQleZ7rvuY/pFg6utNu4ECbvqA5dnFFIuZCbjLlysjasIzhZi7qj4YIRhQDhht+2GJ2itiQwmLnVd8BWqI7/pUUOjkLL8nqYDVco9GQK7lGAf5RYr4Ln9FhV+2cTM30r57nacqS/Us9MzUsuOB6hQy23KyAOK+xQNZaUSam1NcvoubrK6n55kXUfHMlNd++iJpvr6TmuxdR891Mao5OgibeuznUK//MFWR4GrywwhdatwOuYXJC186cYOGkoD4J8rczeIUFF+gF8sQvWcZu41EsucCMoTZK7lA9dl20Hf4bu9MhijUggw0n4UHuiecQ3BnWl5Lw4pZCgr/Z8GsHmb8BeTsVR25nE7NwsWHXUtEcmNYXK4KzwH5mIldKMktN8mc6o+dzbSz8Uifqpnt+XsqwKuSudLubkROWVFWmj3+0NgS9jV9V0Y+Fk/zjeqTjI9Qwu9Od6pmKf+bYLQKvnTTF0P8LA4nAdsZy21yNmSWirYs288T1NLJSLTMtraLTDz/ORLglRKrlw5Yz9P8chfts7X904NtqKsU3xOCtFdVuhwr6TwAAAP//L5rceg==" | |||
return "eJzsXE2P4zbSvvevKPRpBkhPpj98SB8C5J3k3STIBMFmsjksFgollm2OKVLhh7udX78gKcmyPmzJltSL3e5DEIztqucpFqvIYpE3sMHdI6AwqDLFNGokKllfARhmOD7C9XflR/Cr/+z6CkAhR6LxEWI05ApAozFMrPQj/PNaa379BVyvjcmu/3UFQFEnimWGSfEIX18BADRkQiqp5XgFsGTIqX70X7sBQVJsRef+zC7DR1gpabP8X1pUub8/6gL+gEQKQ5jQkKJRLNFABIU1Em7WwMRSqpQ4KbCUqok2F1yFWoWbcKsNqshaRssPC7gb3D1JVf33A9Afwm/ht99++NYrN2uE7zjRhiUBeyEerEYKRPtvUGIIaCMVtiN+1wAZuDbgVa3p/uoD3Qq6OZwN6XVbVaG4/x580G2rhurvpTZewN5YDTBC0r38VgRbVJpJ0QqibpEe7HNxVUeqCWgzx4FJbBqjanx83DADkAX5YDUTK281jSkRhiXFN9y/B/DtlquCjS3jNFoTvb4U8DdgBfvTYhAJTuSRcc1IsiGrE0ObKZmg1mMNbS7ugqHNWJsZAiAuxeqUjX7JEVTjQwMmE9oQkXTYpgrHZoal+E5jMgqqIG4UZEvGMcaKA9ZxtQ1fA9j37THdgcM0RkqRwv/nmprg9n9dwwonh/akITuNWRniHliPQlOoDVEmSqQVbRYdDvLnEELkEtyI6yMIYU0oGAkxFjCQArXo/k3LFIFpbVH3I+KURZqJBCOXFaNcYocDD6f1ybmvl+/t7nQco5arhzc3t8CWIHCLas/y7fHY9HmbnheXfvzHx2aKhR7Bpz3RwdA47fQXqcQvQ4wEZUXnSunILF91x51e8/tHsiWwIiomK4REco6Jn+T5uu7MmbwXNOKM+SQN4UXqlctW6FIBE1uZ+EilK44Y3MzNtjU2klpfLj7Up21GOZeNj/Y6Q2GcA/hlQwetc6kUNFJMpdpFVldTfh18L5f56CVBl6Q+/rFGkkVMMPMu3hk826AhJz3CMSH1bJaB0xumXbzzNnTTkYkg5V0/p/AEnIyXIDAG9kSmKTNmdgIfCr0ehgt8l7BIyfPM+D+S54C8axD6oRdSRC81CX6W4qacCIzwPDRc6lYlpZdyrZJXCWAkZjL+jImJMhSUiVW0ZIJw9hcZOb193fElgG+ZzjjZhVIFyTIln1lKDFZyYYCo/Xr3ac2SNVRRAtOQoz9CuVwirhUS2mbyAUniUxByZKMHfdcSVikcycYfgqyK4TjbYsG4pztQgmkrm9HwBA0DYaXkeZzdE5JN1SzgXbyy/nAzqNiG5NurzP3miWi3dEfTE7BxK6AolzTJCjE3ICQKiUNLBP3SLaVy8K2UOsCX3qjIE2+UlnruOz6EH5+393iSaoPqwmlZIOgW1mujLiWPNPurbfkHg0ft94DFSwUntaf/kMSw7TgQ9pvxILSwT18kW8I4ifnoYAq5TTyFam2IqQ7jqGVoL/xd5fvHytCJFMJvVc4sFjo3yDhJSu17gaBtrHfaYNqE1AWrCu2zjCNf3b9s8hxD+FnG+QFCK8JjKKtInwgzrNNNzvWjQ5y60JIXlfItZGfoa2CUajM9xjzt8x3E6LAORenG3GE5vv7sGvcG0P9TSDZUPok2rG5p6VS1jTqcGPkqZoocTVcQgT7GbeDeGzgID4CZ8GnP/7/32hNGrYJcWs4nguhEg96J5HKUTCQKUxSGTAW2omEkzBmqlGnNpOhy2ksxVzScibm6BBg5moYc5yXry6IoPhtFEhMFiVEH1tN4G5h/NcRY3X4a4nQaWS6tPI9LA8KRNRac5Qq/723cZ6kFB8eUejeRWzrRPZdcVUR/WrQYUcwaa+mxgDGD6bnpsoqU0c614cUQadvy8CSisPlKZJq5rNC1/boUXMjkhZJ6DXswXJ2skVo+LdxSyUC45Zy18UvEH23jm9cY9BqDOpG+xqD/nRiU2ZgzvZ41AOU6X2PPa+xpIH2NPf+9sac8kDWmHkJ6Vv5+EIlMnYN9/+nTL6DwT4vaFB0oQ2t9eW2mYxM9YH96iGov1m9NmW8oPnd/Ou2RVg2vBpmhCE1Gx3sKZz5o+kieWWrTCvJEitw07SROUzgzX/rpNQqp+hHU2YMxgEl5Ao/GBcLL3P7nIASMIsslS7qmIfR0dYUJsu3svQD1YfA/LMEUvQCj+5FGMXc3RztTB2QylodjGmWoogvaRwcz/maLiqzwktHNUIHGRIqzN/nlQP/H0O855GdR34+8z80RtaqrpWNAqPlmtVK4IgYPE38hfIREm5Lnsfo1f5Ji5dDVVwN7tO0NmkOTKxIxFuTCUyaGrA2NKG7HQv2rIYISRYHilu23mJ0ktoRb7Gz17cGl0uOfSaFbZ+ElXh2khjYaDbGSGxTgjxLjXfiODrtqp+RcT799HmdR1rp+yWemhiUTTK+RwhMzayBOa5UgbWSUM13q7kXY3E3E5v5F2NxPxObhRdg8TMRm8SJsFiOxOagEndl3s49X/swVZDgNXlrhA63bAedgYpJsnDhBQ6UgrwT57gyWIWcCPUHWcpNl6DYexYoJjChqo+QO1buuRtv+d+zqQ1TlgBS2jISD3JrmYNwR8ktKGJ+TSNA3GvxcQeQ7IOdjcaB2NDJLZxs6FYuiYJo3VgRlAXsLttYZfVGfaf8pnaupNNiNNYFXXMatNZUBS6m/eRlAMeNyl6IwN0+MYvnmQHHykJM4uxrnfh4VsfpoNuh9TuL7bMsdjbZpStSuBLwndOkBSd7VP2o9uO7SSZVLj9OALqxH+3UngZrsOnoI4U1x287/Qr89jV7bxP12acftJzvBwKyJgTXZYtCPtI+hQ7yZFGYe0g7R9ilRVt+SqAPqV5+UFG84bpFX6gNjhYKMrFBHW6bZVFcSvAbINRTlmjIKl3ltX6EbUJSyiuuIcC6fJgL/299/0pArqGOnVrn5RJlO5BbVbhQyFAU7waV3SG4lQzFmF3D5or6dpyh2oJBoKS4N7YQrJHQXacSuy0Ywylz2hhDSwFIWQ4uJJRrzD4HiDbUZZ+FWNbxBkqz9B2zvyVLwHUiR4NsBp5pUpoSJaMm4QXV8sEekGvQ4ksQGkkSAFRvhxjBAGtZoLJXy9wOVNDLp7LQYkYHfRZaKv2RiSzijX1qhbZZJf7epADPkTJwzsYmMlBFF7JpSI7I4cLn8PYuQUHxLA3CWsqNJpY5fyCBuNhfKQZNSs1uaEjBkBVL5GUKMUSy2ZkijSWUYo0QKg8L4mwyz0xIBSnCoHEmtSHmiMEyM1VEi6Vi3MA7OCRRqy01eVy0jcbwryqlO76VB+Pru/fvrSQ3vKd29f78vP4OL+MOXBp0U7t/fzkDh/v3tlBTuZqFwNx2Fh1kc6WFKR3qYxZEepnSkh1kc6WFaR7qfhcL9lBQeZqHwMCWFxSwUFhNSuJ0lIt1OGZHuZpnOd1NO57uvZqHw1XQUFrOktsWUqW0xS2pbTJnaFrOktsWUqW0xS2pbTJnaFrOktsUIqe3o1i1cLTj5xMfA4lvR3Ot2xsVJ3l/7106b53eHf70PRRRLiRr3skZ7HebgKCSYviwmnmJTva2jWl/SmxbtGpPNHq3HkJdfsjXRwx5hibrfjOpF4PSRSPlmzf5Rn/KBiPzNlvyUc2ADSPdLOHDmsd/+eZ3We1KjefmJR3jgAs/x07J6RHxwqwvetLbsh6M0yKTWLOb4tjfjeY8I625zyK9+MHgOiROvAY3Oo3wl6M1SIb6tMeomcNWGPlOS2sS0PhQ6uP0gFxbeCm07r2s2KnefjZ/qOCBZFunDZ3P7QG/Cz7LKc+Wt+IedLx5e4gr9Re0B54RbnLj4UwGeKym6ObqaDjpt+VQ8zjGKRRtPfYxj17plpVpFWlqVnH/94oSFG0SkWoWelKI8nusfbPgmm0yxLTE4N6NcbV9C/w4AAP//FhlRvw==" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know how to verify this machine-generated string. I trust it's machine-generated. :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, it is the content from the yaml file auto-generated into a go file by make update
x-pack/metricbeat/module/enterprisesearch/stats/_meta/fields.yml
Outdated
Show resolved
Hide resolved
x-pack/metricbeat/module/enterprisesearch/stats/_meta/fields.yml
Outdated
Show resolved
Hide resolved
x-pack/metricbeat/module/enterprisesearch/stats/_meta/fields.yml
Outdated
Show resolved
Hide resolved
x-pack/metricbeat/module/enterprisesearch/stats/_meta/fields.yml
Outdated
Show resolved
Hide resolved
Installed a spellchecker into my vscode, fixed all spelling errors. |
…28790) (#28852) * Add crawler metrics into the stats metricset for Enterprise Search * Adjust ent-search docker testing setup to use 8.0 images + update configs to align with recent deprecations * Spelling, etc fixes * Better field description (cherry picked from commit c80dc4f) Co-authored-by: Oleksiy Kovyrin <oleksiy@kovyrin.net>
…in-the-package-binareis * upstream/master: allows disable pod events enrichment with deployment name (elastic#28521) Remove Docker input from Filebeat (elastic#28817) [breaking] Make default_field: false the default for all fields (elastic#28596) Osquerybeat: Improve osquery client connect code (elastic#28848) Add crawler metrics into the stats metricset for Enterprise Search (elastic#28790) Remove the now deprecated appsearch module from metricbeat (elastic#28850) Remove Beat generators (elastic#28816) chore: upload files to Google Storage when they exist (elastic#28836) Revert "chore(ci): disable E2E tests in Beats (elastic#28715)" (elastic#28812) Deprecate generating custom Beats (elastic#28814) [Metricbeat] upgrade flatbuffers to 1.12.1 (elastic#28094) Osquerybeat: Fix restart flags after previously bad config (elastic#28827) Force ECS and JSON logging for libbeat/logp (elastic#28573) Filebeat: Error on startup for unconfigured module (elastic#28818) Deprecate log input in favour of filestream (elastic#28623) Fix some spelling mistakes (elastic#28080)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM 👍
…in-the-package-binareis * upstream/master: allows disable pod events enrichment with deployment name (elastic#28521) Remove Docker input from Filebeat (elastic#28817) [breaking] Make default_field: false the default for all fields (elastic#28596) Osquerybeat: Improve osquery client connect code (elastic#28848) Add crawler metrics into the stats metricset for Enterprise Search (elastic#28790) Remove the now deprecated appsearch module from metricbeat (elastic#28850) Remove Beat generators (elastic#28816) chore: upload files to Google Storage when they exist (elastic#28836) Revert "chore(ci): disable E2E tests in Beats (elastic#28715)" (elastic#28812) Deprecate generating custom Beats (elastic#28814) [Metricbeat] upgrade flatbuffers to 1.12.1 (elastic#28094) Osquerybeat: Fix restart flags after previously bad config (elastic#28827) Force ECS and JSON logging for libbeat/logp (elastic#28573) Filebeat: Error on startup for unconfigured module (elastic#28818) Deprecate log input in favour of filestream (elastic#28623) Fix some spelling mistakes (elastic#28080)
What does this PR do?
This PR adds more metrics to the stats metricset for Enterprise Search. The new metrics have been added to Enterprise Search in https://github.com/elastic/ent-search/pull/5192 and will be available in 8.0.0.
Why is it important?
Users need these new metrics to be able to reason about the health of their crawler deployments.
Checklist
CHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.Author's Checklist
How to test this PR locally
Related issues