Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] CohereServiceUpgradeIT testCohereEmbeddings {upgradedNodes=1} failing #116974

Open
elasticsearchmachine opened this issue Nov 18, 2024 · 4 comments
Labels
low-risk An open issue or test failure that is a low risk to future releases :ml Machine learning Team:ML Meta label for the ML team >test-failure Triaged test failures from CI

Comments

@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Nov 18, 2024

Build Scans:

Reproduction Line:

./gradlew ":x-pack:plugin:inference:qa:rolling-upgrade:v8.14.3#bwcTest" -Dtests.class="org.elasticsearch.xpack.application.CohereServiceUpgradeIT" -Dtests.method="testCohereEmbeddings {upgradedNodes=1}" -Dtests.seed=1868AB709715B87A -Dtests.bwc=true -Dtests.locale=ta-SG -Dtests.timezone=Europe/Saratov -Druntime.java=23

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

Issue Reasons:

  • [8.x] 2 consecutive failures in step 8.14.3_bwc
  • [8.x] 4 consecutive failures in step 8.13.4_bwc
  • [8.x] 6 failures in test testCohereEmbeddings {upgradedNodes=1} (4.2% fail rate in 143 executions)
  • [8.x] 2 failures in step 8.14.3_bwc (50.0% fail rate in 4 executions)
  • [8.x] 4 failures in step 8.13.4_bwc (100.0% fail rate in 4 executions)
  • [8.x] 3 failures in pipeline elasticsearch-periodic (100.0% fail rate in 3 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

@elasticsearchmachine elasticsearchmachine added :ml Machine learning >test-failure Triaged test failures from CI labels Nov 18, 2024
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch 8.x

Mute Reasons:

  • [8.x] 4 failures in test testCohereEmbeddings {upgradedNodes=1} (0.9% fail rate in 448 executions)
  • [8.x] 4 failures in step 8.16.1_bwc-snapshots (4.0% fail rate in 100 executions)
  • [8.x] 3 failures in pipeline elasticsearch-pull-request (6.0% fail rate in 50 executions)

Build Scans:

elasticsearchmachine added a commit that referenced this issue Nov 18, 2024
@elasticsearchmachine elasticsearchmachine added Team:ML Meta label for the ML team needs:risk Requires assignment of a risk label (low, medium, blocker) labels Nov 18, 2024
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/ml-core (Team:ML)

@davidkyle
Copy link
Member

The mixed cluster failed to form due to one of the nodes missing a required feature

Caused by: java.lang.IllegalStateException: Node gX0XxROcRlmfqMEXlwfZcw is missing required features [semantic_text.delete_fix]

Low risk as the issue in the test, just need to figure out how to fix it

@davidkyle davidkyle added low-risk An open issue or test failure that is a low risk to future releases and removed needs:risk Requires assignment of a risk label (low, medium, blocker) labels Nov 20, 2024
elasticsearchmachine pushed a commit that referenced this issue Dec 13, 2024
#118664

```
- class: org.elasticsearch.xpack.application.CohereServiceUpgradeIT
  method: testRerank {upgradedNodes=1}
  issue: #116973
- class: org.elasticsearch.xpack.application.CohereServiceUpgradeIT
  method: testCohereEmbeddings {upgradedNodes=1}
  issue: #116974
- class: org.elasticsearch.xpack.application.CohereServiceUpgradeIT
  method: testCohereEmbeddings {upgradedNodes=2}
  issue: #116975

- class: org.elasticsearch.xpack.application.OpenAiServiceUpgradeIT
  method: testOpenAiEmbeddings {upgradedNodes=1}
  issue: #118156
- class: org.elasticsearch.xpack.application.HuggingFaceServiceUpgradeIT
  method: testElser {upgradedNodes=1}
  issue: #118127
- class: org.elasticsearch.xpack.application.OpenAiServiceUpgradeIT
  method: testOpenAiCompletions {upgradedNodes=1}
  issue: #118162
- class: org.elasticsearch.xpack.application.OpenAiServiceUpgradeIT
  method: testOpenAiCompletions {upgradedNodes=2}
  issue: #118163
- class: org.elasticsearch.xpack.application.OpenAiServiceUpgradeIT
  method: testOpenAiEmbeddings {upgradedNodes=2}
  issue: #118204

- class: org.elasticsearch.xpack.application.HuggingFaceServiceUpgradeIT
  method: testHFEmbeddings {upgradedNodes=1}
  issue: #118197
```
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch 8.x

Mute Reasons:

  • [8.x] 4 consecutive failures in step 8.13.4_bwc
  • [8.x] 2 consecutive failures in pipeline elasticsearch-periodic
  • [8.x] 5 failures in test testCohereEmbeddings {upgradedNodes=1} (3.7% fail rate in 136 executions)
  • [8.x] 4 failures in step 8.13.4_bwc (100.0% fail rate in 4 executions)
  • [8.x] 3 failures in pipeline elasticsearch-periodic (100.0% fail rate in 3 executions)

Build Scans:

elasticsearchmachine added a commit that referenced this issue Dec 14, 2024
maxhniebergall added a commit to maxhniebergall/elasticsearch that referenced this issue Dec 16, 2024
elastic#118664

```
- class: org.elasticsearch.xpack.application.CohereServiceUpgradeIT
  method: testRerank {upgradedNodes=1}
  issue: elastic#116973
- class: org.elasticsearch.xpack.application.CohereServiceUpgradeIT
  method: testCohereEmbeddings {upgradedNodes=1}
  issue: elastic#116974
- class: org.elasticsearch.xpack.application.CohereServiceUpgradeIT
  method: testCohereEmbeddings {upgradedNodes=2}
  issue: elastic#116975

- class: org.elasticsearch.xpack.application.OpenAiServiceUpgradeIT
  method: testOpenAiEmbeddings {upgradedNodes=1}
  issue: elastic#118156
- class: org.elasticsearch.xpack.application.HuggingFaceServiceUpgradeIT
  method: testElser {upgradedNodes=1}
  issue: elastic#118127
- class: org.elasticsearch.xpack.application.OpenAiServiceUpgradeIT
  method: testOpenAiCompletions {upgradedNodes=1}
  issue: elastic#118162
- class: org.elasticsearch.xpack.application.OpenAiServiceUpgradeIT
  method: testOpenAiCompletions {upgradedNodes=2}
  issue: elastic#118163
- class: org.elasticsearch.xpack.application.OpenAiServiceUpgradeIT
  method: testOpenAiEmbeddings {upgradedNodes=2}
  issue: elastic#118204

- class: org.elasticsearch.xpack.application.HuggingFaceServiceUpgradeIT
  method: testHFEmbeddings {upgradedNodes=1}
  issue: elastic#118197
```
maxhniebergall pushed a commit to maxhniebergall/elasticsearch that referenced this issue Dec 16, 2024
elasticsearchmachine pushed a commit that referenced this issue Dec 17, 2024
```
- class: org.elasticsearch.xpack.application.OpenAiServiceUpgradeIT
  method: testOpenAiEmbeddings {upgradedNodes=1}
  issue: #118156
- class: org.elasticsearch.xpack.application.HuggingFaceServiceUpgradeIT
  method: testHFEmbeddings {upgradedNodes=1}
  issue: #118197
- class: org.elasticsearch.xpack.application.OpenAiServiceUpgradeIT
  method: testOpenAiCompletions {upgradedNodes=2}
  issue: #118163
- class: org.elasticsearch.xpack.application.OpenAiServiceUpgradeIT
  method: testOpenAiCompletions {upgradedNodes=1}
  issue: #118162
- class: org.elasticsearch.xpack.application.HuggingFaceServiceUpgradeIT
  method: testElser {upgradedNodes=1}
  issue: #118127
- class: org.elasticsearch.xpack.application.CohereServiceUpgradeIT
  method: testCohereEmbeddings {upgradedNodes=1}
  issue: #116974
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
low-risk An open issue or test failure that is a low risk to future releases :ml Machine learning Team:ML Meta label for the ML team >test-failure Triaged test failures from CI
Projects
None yet
Development

No branches or pull requests

2 participants