Skip to content
This repository has been archived by the owner on May 12, 2021. It is now read-only.

METRON-2092: Config UI does not require you to set a grok timestamp field by default #1393

Closed
wants to merge 2 commits into from

Conversation

ruffle1986
Copy link
Contributor

Contributor Comments

Link to the original ASF Jira: https://issues.apache.org/jira/browse/METRON-2092

Issue

A Grok parser requires a timestampField attribute. Currently the UI allows the user to add a new Grok parser without any timestampField. When you start the parser, nothing happens and there are no error messages.

The UI should either default the timestampField attribute to timestamp or pop up an error message when the user tries to save a grok parser with no timestampField attribute.

Testing

  1. Go to metron config
  2. Try to add a new sensor which has a Grok parser
  3. Make sure the TIMESTAMP field is there
  4. It should have the default value timestamp
  5. Try to remove it completely and save the sensor (you should not be able)
  6. Try to save when it has a value timestamp or something else (it should work)

It should be the same in case of editing the sensor. If there's no timestamp field value, there should be an error message below the input field.

Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.

In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:

For all changes:

  • Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
  • Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
  • Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

  • Have you included steps to reproduce the behavior or problem that is being changed or addressed?

  • Have you included steps or a guide to how the change may be verified and tested manually?

  • Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:

    mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh 
    
  • Have you written or updated unit tests and or integration tests to verify your changes?

  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

  • Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

  • Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:

    cd site-book
    mvn site
    

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

@sardell
Copy link
Contributor

sardell commented May 16, 2019

@ruffle1986 When testing, I'm not seeing the timestamp field when creating a new Grok parser. What do I need to do to 'Make sure the TIMESTAMP field is there'? Maybe I need to take additional steps besides clicking on the '+' button in the bottom right corner and choosing Grok as my parser type.

Screen Shot 2019-05-16 at 5 53 44 PM

@ruffle1986
Copy link
Contributor Author

ruffle1986 commented May 17, 2019

@sardell weird. for me, it's there. looks like a caching issue. There's no additional step that I know about.

Screen Shot 2019-05-17 at 9 49 57

@sardell
Copy link
Contributor

sardell commented May 17, 2019

@ruffle1986 You were right, it must've been something weird with my cache (or maybe it was just user error 😳). It works as described for me now when testing locally on full dev. While I can verify that the UI works, I would like another committer who is more familiar with the timestampField to verify that an unformatted string value is correct. It just seems weird to me that a timestamp field wouldn't enforce timestamp formatting, and would default to a string value of "timestamp." Again, this probably stems from my own ignorance of that field and what it is used for on the backend.

+1 pending another committer verifying that this is indeed how the field should be.

@mmiklavc
Copy link
Contributor

mmiklavc commented Jun 4, 2019

@sardell This comes from the grok pattern used. The value is an indirection to another variable that references the actual timestamp the user wants to use. For example, in full dev:

YAF

  • "timestampField":"start_time"
  • "dateFormat":"yyyy-MM-dd HH:mm:ss.S" - need this in order for the parser to translate the non-timestamp-fromatted date/time into a proper numeric timestamp.
  • You can find the grok expressions loaded in HDFS at /patterns/yaf. They're also staged from the RPM installs on the local file system at $METRON_HOME/patterns. Anyhow, here's what YAF's Grok patterns file looks like:
YAF_TIME_FORMAT %{YEAR:UNWANTED}-%{MONTHNUM:UNWANTED}-%{MONTHDAY:UNWANTED}[T ]%{HOUR:UNWANTED}:%{MINUTE:UNWANTED}:%{SECOND:UNWANTED}
YAF_DELIMITED %{YAF_TIME_FORMAT:start_time}\|%{YAF_TIME_FORMAT:end_time}\|%{SPACE:UNWANTED}%{BASE10NUM:duration}\|%{SPACE:UNWANTED}%{BASE10NUM:rtt}\|%{SPACE:UNWANTED}%{INT:protocol}\|%{SPACE:UNWANTED}%{IP:ip_src_addr}\|%{SPACE:UNWANTED}%{INT:ip_src_port}\|%{SPACE:UNWANTED}%{IP:ip_dst_addr}\|%{SPACE:UNWANTED}%{INT:ip_dst_port}\|%{SPACE:UNWANTED}%{DATA:iflags}\|%{SPACE:UNWANTED}%{DATA:uflags}\|%{SPACE:UNWANTED}%{DATA:riflags}\|%{SPACE:UNWANTED}%{DATA:ruflags}\|%{SPACE:UNWANTED}%{WORD:isn}\|%{SPACE:UNWANTED}%{DATA:risn}\|%{SPACE:UNWANTED}%{DATA:tag}\|%{GREEDYDATA:rtag}\|%{SPACE:UNWANTED}%{INT:pkt}\|%{SPACE:UNWANTED}%{INT:oct}\|%{SPACE:UNWANTED}%{INT:rpkt}\|%{SPACE:UNWANTED}%{INT:roct}\|%{SPACE:UNWANTED}%{INT:app}\|%{GREEDYDATA:end_reason}

Notice the start_time field - %{YAF_TIME_FORMAT:start_time}

For an original message "original_string": "2019-06-03 20:38:27.000|2019-06-03 20:38:27.000| 0.000| 0.000| 6| 192.168.138.158|49189| 62.75.195.236| 80| A| 0| 0| 0|9dfb1927|00000000|000|000| 1| 40| 0| 0| 0|idle ",, this results in a Metron timestamp field published to the index as "timestamp": 1559594307000

Squid

  • "timestampField": "timestamp"
  • No need for dateFormat bc the timestamp is in the right format for Metron OOTB.
SQUID_DELIMITED %{NUMBER:timestamp}[^0-9]*%{INT:elapsed} %{IP:ip_src_addr} %{WORD:action}/%{NUMBER:code} %{NUMBER:bytes} %{WORD:method} %{NOTSPACE:url}[^0-9]*(%{IP:ip_dst_addr})?

Does that answer your question?

@sardell
Copy link
Contributor

sardell commented Jun 4, 2019

@mmiklavc That makes sense. I figured the name of the field might have been misleading me, but I wanted to check before merging. You answered my question. Thanks!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants