-
Notifications
You must be signed in to change notification settings - Fork 512
METRON-2327 - Support for SOLR time-based arrays #1572
base: feature/METRON-2088-support-hdp-3.1
Are you sure you want to change the base?
METRON-2327 - Support for SOLR time-based arrays #1572
Conversation
…TML converter some grief
…er producing nicer html
You should create a new JIRA not reuse 2088 please. |
metron-platform/metron-solr/metron-solr-common/src/main/scripts/create_configset.sh
Show resolved
Hide resolved
…com/apache/metron into metron-2327_solr_timebased_routing
…the sample schema files
Interesting enough, when I ran the integration tests on my laptop, I got the following error:
|
# test for presence of datetime field in schema collection | ||
DQT='"' | ||
DATETIME_SCHEMA="<field name=${DQT}datetime${DQT} type=${DQT}datetime${DQT}" | ||
grep --quiet --fixed-strings $DATETIME_SCHEMA $METRON_HOME/config/schema/$1/schema.xml; rc=$? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see this when I run the create_configset
script...
[root@solrtra-5 schema]# pwd
/usr/metron/0.7.2/config/schema
[root@solrtra-5 schema]# ../../bin/create_configset.sh bro1
grep: name="datetime": No such file or directory
grep: type="datetime": No such file or directory
adding: solrconfig.xml (deflated 72%)
adding: schema.xml (deflated 88%)
{
"responseHeader":{
"status":0,
"QTime":96}}
Configset bro1 successfully uploaded
Looks like the regex pattern needs fixing?
|
||
1. Add to the sensor parser config json field the following transformation: | ||
``` | ||
"fieldTransformations" : [{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm guessing the markdown changed some of the double quotes.. this should be something like....
{
"parserClassName":"org.apache.metron.parsers.bro.BasicBroParser",
"sensorTopic":"bro",
"parserConfig": {
"fieldTransformations" : [{
"input" : [ ]
,"transformation" : "STELLAR"
,"output" : [ "datetime" ]
,"config" : {
"datetime" : "DATE_FORMAT(\"yyyy-MM-dd'T'HH:mm:ss.SSSX\",timestamp)"
}
}]
}
}
Assuming the following values: | ||
* SOLR_HOST: Host SOLR is installed on | ||
|
||
* ALIAS_NAME: Name of the new alias |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In my case, I created the alias name as bro1
but I notice the indices are not being written. In the solr admin UI, I see the name of the collection as bro1_2019-12-03
.
Can you provide an example for "name of the new alias" ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this case the alias is what Metron is configured to read and write from - "bro1". What SOLR does behind the scenes is actually route the event to be stored in the collection "bro1_2019-12-03". If the event had been collected the next day, then it may be stored in the collection "bro1_2019-12-04". Both events are still retrievable when searching for them via the alias "bro1".
# | ||
METRON_VERSION=${project.version} | ||
METRON_HOME=/usr/metron/$METRON_VERSION | ||
ZOOKEEPER=${ZOOKEEPER:-localhost:2181} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to source /etc/default/metron
before setting ZOOKEEPER. This step will fail in a multinode deployment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And in addition, we need to do something like this before invoking the script so that the ZK variable is set properly with the /solr
suffix.
The README can be updated with this info.
# | ||
METRON_VERSION=${project.version} | ||
METRON_HOME=/usr/metron/$METRON_VERSION | ||
ZOOKEEPER=${ZOOKEEPER:-localhost:2181} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And in addition, we need to do something like this before invoking the script so that the ZK variable is set properly with the /solr
suffix.
The README can be updated with this info.
|
||
Then the following command will create a time-routed alias: | ||
``` | ||
curl "http://$SOLR_HOST:8983/solr/admin/collections?action=CREATEALIAS\ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor nit... can we remove the spaces at the beginning of this block of code? Copy-pasting this will result in this command to fail.
Also, in my case I had to set the numShards
to 1 for the command to complete successfully. Not sure if that is something to do with the way my solr is configured.
``` | ||
${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh --bootstrap-server $BROKERLIST --topic $PARSER_NAME | ||
``` | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be good to include the step to start the parser topology ? :)
Contributor Comments
Updated schemas and instructions on how to enable SOLR time based arrays.
Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
For all changes:
For documentation related changes:
Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via
site-book/target/site/index.html
:Have you ensured that any documentation diagrams have been updated, along with their source files, using draw.io? See Metron Development Guidelines for instructions.