Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Add custom rule parameters to force time shift #110974

Merged
merged 11 commits into from
Jul 25, 2024

Conversation

valeriy42
Copy link
Contributor

@valeriy42 valeriy42 commented Jul 17, 2024

This PR extends the schema of the detection rule API by adding a new parametrizable action force_time_shift and a new property params to hold the parameter total_shift_amount as signed long in seconds to control the behavior of the time shift.

A valid schema looks like this:

POST _ml/anomaly_detectors/my_job/_update
{
	"detectors": {
		"detector_index": 0,
		"custom_rules": [
			// update the detector with a custom rule that forces a time shift of 1 hour back starting now
			{
				"actions": [
					"force_time_shift"
				],
				"params": {
					"force_time_shift": {
"time_shift_amount": 3600}
					}
				},
				"conditions": [{
					"applies_to": "time",
					"operator": "gt",
					"value": "now"
				},
{
					"applies_to": "time",
					"operator": "lt",
					"value": "now+bucket_span"
				},

]
			},
			...

Execution of the detection rule action force_time_shift will shift the time inside the anomaly detector by a specified amount. This is useful, e.g. to quickly adapt to the daylight saving time events that are known beforehand.

This PR is accompanied by changes in Kibana elastic/kibana#188710 and the native ml-cpp code elastic/ml-cpp#2690.

The integration tests between Java and C++ parts of the force-time shift action will be implemented in a separate PR.

@elasticsearchmachine
Copy link
Collaborator

Hi @valeriy42, I've created a changelog YAML for you.

…#110582)

DO NOT MERGE THIS INTO `main`!

This PR enables the storage of data and configuration of an anomaly detection job in files so it can be reproduced using the `autodetect` process without Elasticsearch.

To enable the storage, specify `keep_job_data` parameter in the `custom_settings` parameter of the job config:

```json
  "custom_settings": {
    "keep_job_data": "true"
    } 
```

Now, start the job and watch for a log message with the autodetect command similar to the following:

```bash
[2024-06-19T16:03:38,248][INFO ][o.e.x.m.j.p.a.NativeAutodetectProcessFactory] [Elastic-MBP.fritz.box] Autodetect process command: [./autodetect, --lengthEncodedInput, --maxAnomalyRecords=500, --validElasticLicenseKeyConfirmed=true, --config=/var/folders/_j/gcj6z4b950bdzpw7_fzrmpf40000gn/T/elasticsearch-12668972032551307591/config10764979302390040373.json, --logPipe=/var/folders/_j/gcj6z4b950bdzpw7_fzrmpf40000gn/T/elasticsearch-12668972032551307591/autodetect_test-2_log_45530, --input=/var/folders/_j/gcj6z4b950bdzpw7_fzrmpf40000gn/T/elasticsearch-12668972032551307591/autodetect_test-2_input_45530, --inputIsPipe, --output=/var/folders/_j/gcj6z4b950bdzpw7_fzrmpf40000gn/T/elasticsearch-12668972032551307591/autodetect_test-2_output_45530, --outputIsPipe, --persist=/var/folders/_j/gcj6z4b950bdzpw7_fzrmpf40000gn/T/elasticsearch-12668972032551307591/autodetect_test-2_persist_45530, --persistIsPipe, --namedPipeConnectTimeout=10]
```
and
```bash
[2024-06-19T15:29:08,640][INFO ][o.e.x.m.p.w.LengthEncodedWriter]  Opening file: /var/folders/_j/gcj6z4b950bdzpw7_fzrmpf40000gn/T/elasticsearch-12668972032551307591/autodetect_test-2_input_45530 for writing.
```

Copy the config file, the persist file from the first message, and the input file from the second message.
@valeriy42 valeriy42 marked this pull request as ready for review July 24, 2024 12:23
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/ml-core (Team:ML)

@elasticsearchmachine elasticsearchmachine added the Team:ML Meta label for the ML team label Jul 24, 2024
@valeriy42 valeriy42 requested review from davidkyle and removed request for davidkyle July 25, 2024 14:36
@valeriy42 valeriy42 merged commit f755e80 into elastic:main Jul 25, 2024
15 checks passed
@valeriy42 valeriy42 deleted the force-time-shift branch July 25, 2024 14:59
valeriy42 added a commit to elastic/kibana that referenced this pull request Jul 26, 2024
This PR follows up elastic/elasticsearch#110974
and extends the schema for custom rules.

A valid schema looks like this:
```http
POST _ml/anomaly_detectors/my_job/_update
{
	"detectors": {
		"detector_index": 0,
		"custom_rules": [
			// update the detector with a custom rule that forces a time shift of 1 hour back starting now
			{
				"actions": [
					"force_time_shift"
				],
				"params": {
					"force_time_shift": {
"time_shift_amount": 3600}
					}
				},
				"conditions": [{
					"applies_to": "time",
					"operator": "gt",
					"value": "now"
				},
{
					"applies_to": "time",
					"operator": "lt",
					"value": "now+bucket_span"
				},

]
			},
			...
```

---------

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement :ml Machine learning Team:ML Meta label for the ML team v8.16.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants