[Question] How to import rules from this repository in Kibana?

[vedard]

Hi Elastic, thanks you for opening this repository to the public.

I was wondering if it was possible to fork this repository, add some of our environment specific rules and import them in our Kibana instance.

It would improve our workflow:

• We could write tests for our rules
• We would keep a history of our rules
• We could frequently update Elastic's rules (only need to pull from upstream)

I see there is a build-release command, but I'm not sure if I can import the generated package in Kibana or if it's only when you release a new version of Kibana

Thanks.

[rw-access]

💯

glad to see you've done some sleuthing! I think we have most of the pieces in place for this — you probably noticed the wrapper around the Kibana API as well.

For our initial release of the repository, we intentionally kept this out of scope. We just needed to get something released quickly and wanted to have a path for users like yourself to contribute directly to the released detection engine.

This is a near-term priority for us. @brokensound77 and I are figuring out the details but most of the pieces are there. We know there is a need for many organizations like to synchronize their rules to/from git. Or at least to pull out of the platform and push back.

It's hard to commit to a timeline, especially as we start wrapping up a release. But I think it's fair to say this is a "near-term" goal.

We'll keep you posted!

[vedard]

This is good news! Thank you for the quick response.

[vedard]

Hi rw-access,

I'm really happy to see you've added the kibana-upload command. I've been trying it out, and I think it would be really convenient:

• to be able to update a rule without having to delete the previous version.
• to be able to upload all rules at once, this one is easy, i was able to do it by allowing the toml-files arguments to be None:

  @click.argument("toml-files", nargs=-1, default=None)

Are these features you are planning to implement?

[SHolzhauer]

I am interested in trying to pick this issue up; But let's first have a look at what it is that's actually needed.

We need a way to

• Update existing, custom, rules in Kibana,
• Update existing, elastic, rules in Kibana between stack releases,
• Create new, custom, rules in Kibana
• Create new, elastic, rules in Kibana between stack releases
• Do above with multiple rules at once

This process should

• be possible to automate (CI/CD)
• validate rules (python -m detection_rules test functionality ?)
• only update/create valid rules

This FR should not

• interfere with/change existing commands

I am purposefully leaving the original first requirement from @vedard out to scope the FR down.

Looking forward to hearing what everyone thinks?

[rw-access]

@vedard does this work now for multiple upload?
The current command looks like this which should be working:

detection-rules/detection_rules/eswrap.py

Line 231 in 464d5e6

@click.argument("toml-files", nargs=-1, required=True)

@brokensound77 have you looked at this?

[vedard]

@rw-access it does if you want to specify one or multiple files, but it would be nice to have an option upload all rules at once.

I was able to do it by set None as the default value for the argument, but it could another option like --all-rules.

[brokensound77]

This is similar to how it is done for import-rules

detection-rules/detection_rules/main.py

Lines 50 to 56 in 140091e

	@root.command('import-rules')
	@click.argument('infile', type=click.Path(dir_okay=False, exists=True), nargs=-1, required=False)
	@click.option('--directory', '-d', type=click.Path(file_okay=False, exists=True), help='Load files from a directory')
	def import_rules(infile, directory):
	"""Import rules from json, toml, or Kibana exported rule file(s)."""
	rule_files = glob.glob(os.path.join(directory, '**', '*.*'), recursive=True) if directory else []
	rule_files = sorted(set(rule_files + list(infile)))

The glob pattern could be updated to *.toml, but the onus would still be on the user to ensure that a directory contains only valid rule files

[rw-access]

I'm hesitant to make it this easy to automatically upload all of the rules with this workaround. It's beyond the initial scope of the command, and needs a more complete solution in place. This command is mostly to for testing, and there are no safeguards to make sure we don't create duplicate rules, and if you run the command twice, you get all of those rules as duplicates again.

I think these are good use cases brought up by this issue, but they need a more complete workflow that kibana-upload just doesn't provide.

I'm wondering if it would help with scoping to create another issue to track the complete workflow for synchronizing to/from a fork of this repo to a Kibana instance, or if this issue should be renamed

[SHolzhauer]

After some thought I think it's a good idea to create a new issue (e.g: Functionality to sync between a git repository and Kibana) and determine the exact functionality, properties etc required to make that work in that issue.

As to this specific issue, the original question/issue has been answered solved; My proposal would be to close this issue and reference it in the "new" issue.

Decision is up to OP and you @rw-access

[vedard]

I agree, the original question was answered. Feel free to close this issue when opening the new one.