This project is a Java Application powered by VertX which listens to Github Push Events.
Each time something is pushed on bblfr_data, this application:
- fetch https://github.com/brownbaglunch/bblfr_data/blob/gh-pages/baggers.js
- split
data
inbaggers
andcities
java beans - enrich
baggers.cities
with geo location point and image of the associated city - index
baggers
in elasticsearch. The index name is timestamped. - move or create alias
bblfr
on the index name - remove the old indices
Data are then available in elasticsearch:
It also monitors Pull Requests and validate that the json file is still correct.
To run it, you need to have Maven installed.
git clone https://github.com/brownbaglunch/java-webhook.git
cd java-webhook
mvn clean install
java -jar target/java-webhook-1.0-SNAPSHOT-fat.jar
It should says:
23:24:08,233 INFO [f.b.w.ElasticsearchVerticle] Starting Elasticsearch Client
23:24:08,233 INFO [f.b.w.WebhookVerticle] Starting HTTP server on port 8080
23:24:09,104 INFO [f.b.w.WebhookVerticle] HTTP server started on port 8080
23:24:09,918 INFO [f.b.w.s.ElasticsearchClientManager] Elasticsearch client started and connected to a cluster running version 6.1.1
Then just POST to http://localhost:8080/:
curl -XPOST localhost:8080/
And you should see:
23:24:12,916 INFO [f.b.w.WebhookVerticle] Signature has not been verified. Probably Dev Mode.
23:24:12,916 INFO [f.b.w.WebhookVerticle] Reading data from https://raw.githubusercontent.com/brownbaglunch/bblfr_data/gh-pages/baggers.js
23:24:16,832 INFO [f.b.w.ElasticsearchVerticle] Let's index 243 speakers and 39 cities.
23:24:19,323 INFO [f.b.w.ElasticsearchVerticle] Indexed in elasticsearch: OK
If you want to push to another cluster, you need to start the application with:
ROOT=https://raw.githubusercontent.com/brownbaglunch/bblfr_data \
BRANCH=gh-pages \
SOURCE=/baggers.js \
TARGET=https://username:password@yourcluster.found.io:9243 \
TOKEN=hhdquiHdsuqiudshqhiudhuebefbbcbzbczib \
java -jar target/java-webhook-1.0-SNAPSHOT-fat.jar
TOKEN
value is the one you defined in Github Hooks.
It can be null
(default) in development mode.
If you want to change network settings, change PORT
system variables:
PORT=9000 java -jar target/java-webhook-1.0-SNAPSHOT-fat.jar
By default, it will listen on 0.0.0.0
, port 8080
.
If you just want to check that the service is running, call:
curl -XGET localhost:8080
You will get back:
{
"message" : "Vert.x is running!"
}
If you want to run the process of reading data from Github, then sending that to elasticsearch, you need to call:
curl -XPOST localhost:8080/
In production (when TOKEN
is set), you need to provide the right SHA1 signature like:
curl -XPOST localhost:8080/_stop -H "X-Hub-Signature: sha1=22202b35f1482c1a8d3d0c3f6b3c46307792d409" -d '{
// Your content here. The signature depends on it.
}'
You will get back:
{
"speakers" : 243,
"cities" : 39,
"with_failures" : false
}
In production (when TOKEN
is set), you need to provide the right SHA1 signature like:
curl -XPOST localhost:8080/_stop -H "X-Bblfr-Key: sha1=22202b35f1482c1a8d3d0c3f6b3c46307792d409"
In development mode, just call:
curl -XPOST localhost:8080/_stop
Connect to your Clever-cloud console. Create your Java application and define your variables in the Console:
ROOT=https://raw.githubusercontent.com/brownbaglunch/bblfr_data
BRANCH=gh-pages
SOURCE=/baggers.js
TARGET=http://bblfr:password@localhost:9200
TOKEN=12345678
PORT=8080
Note that PORT
must be 8080
.
Add clever as a git remote repository (change ID
below):
git remote add clever git+ssh://git@push.par.clever-cloud.com/app_ID.git
Deploy!
git push -u clever master
Et voilà!
This software is licensed under the Apache 2 license, quoted below.
Copyright 2018 Brownbaglunch.fr
Licensed under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.