GCP workshop source code
- create vm
- ssh to vm
gcloud beta compute ssh --zone "us-central1-a" "instance-1" --project "serverlessdays20"
- install dependencies
sudo apt install git
sudo apt install python3
git clone https://github.com/redvg/serverlessdays20
cd serverlessdays20
cd simulation
sudo apt install python3-pip
pip3 install -v requests
pip3 install aiohttp
- run simulation
chmod +x ./run.sh
./run.sh {SIMULATION_ID} {NUMBER_OF_DEVICES} {NUMBER_OF_SIGNALS_FROM_DEVICE} {ENDPOINT}
or
python3 simulate.py --endpoint=https://europe-west1-serverlessdays20.cloudfunctions.net/consumer --simulation_id=green --number_of_devices=4 --number_of_signals=100
- create with http trigger
- use main.py & requirements.txt
https://cloud.google.com/functions/docs/monitoring/error-reporting#functions-errors-log-python
- enable firestore in native mode
- create
events
collection
https://cloud.google.com/firestore/docs/firestore-or-datastore
https://firebase.google.com/docs/firestore/quotas
- add web sdk via ui
- enable google sign in
- add localhost to authorized domains
- change firestore security rules
- run locally
cd webapp
npm i
npm start
- feed
firebase.utils.js
with firebase wed sdk App.js
forFirestoreProvider
component andwithFirebaseAuth
higher-order componentDashboard.js
pure stream observerChart.js
client-side stream aggregator
components: https://material-ui.com
- build
cd appengine
make build
- deploy
git clone https://github.com/redvg/serverlessdays20
cd appengine
cd app
gcloud app deploy
- add
domain
to firebase
https://cloud.google.com/appengine/docs/flexible/python/quickstart
https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/appengine/flexible/hello_world/app.yaml
https://cloud.google.com/appengine/docs/flexible/custom-runtimes/build
- create topic
events
https://cloud.google.com/pubsub/architecture
- create dataset
events
- create table
records
, use schema from cloud function
- create bucket
events-etl
- create job from
pubsub topic to bigquery
template, use pubsub & bigquery & cloud storage vars
https://beam.apache.org/
https://cloud.google.com/pubsub/docs/pubsub-dataflow
template:
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
https://cloud.google.com/dataflow/docs/guides/templates/provided-streaming#cloudpubsubtobigquery
https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/main/java/com/google/cloud/teleport/templates/PubSubToBigQuery.java