Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

Usermanagement documentation (Nifi) + Tensorflow use case #130

Merged
merged 18 commits into from
May 4, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
140 changes: 124 additions & 16 deletions doc/USERMANAGEMENT.md

Large diffs are not rendered by default.

Binary file added doc/images/installation/Team1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/images/installation/Team2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/images/installation/Teams.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/images/installation/create.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/images/installation/grey.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/images/installation/sign-in.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/images/installation/users.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/images/logos/tensorflow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 2 additions & 1 deletion examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,5 @@ This section contains various usage examples for FADI:

* [basic example](/USERGUIDE.md) with batch ingestion
* [streaming ingestion](examples/kafka/README.md) with streaming ingestion with the help of the [Apache Kafka](https://kafka.apache.org) message broker
* [on-demand compute environments](examples/binderhub/README.md) with [BinderHub](https://binderhub.readthedocs.io/en/latest/)
* [on-demand compute environments](examples/binderhub/README.md) with [BinderHub](https://binderhub.readthedocs.io/en/latest/)
* [Tensorflow example](examples/tensorflow/README.md) for image classification
101 changes: 101 additions & 0 deletions examples/tensorflow/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# Tensorflow simple use case

This is a simple [Tensorflow](https://www.tensorflow.org/) and [Jupyter](https://jupyter.readthedocs.io/en/latest/) use case with FADI for image classification using the [MNIST dataset](http://yann.lecun.com/exdb/mnist/).


Before starting, change the environment:
* Click on `Control panel`
* Click on `Stop my server`
* Finally, click on `Start server`, choose `tensorflow environment` and click on `Spawn`.

![Jupyter web interface](./images/Tensorflow.png)


* Import TensorFlow:

```python
from __future__ import absolute_import, division, print_function, unicode_literals


import tensorflow as tf
```
* Load and prepare the [MNIST](http://yann.lecun.com/exdb/mnist/) dataset. Convert the samples from integers to floating-point numbers:

```python
mnist = tf.keras.datasets.mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
```
* Build the `tf.keras.Sequential` model by stacking layers. Choose an optimizer and loss function for training:

```python
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10)
])
```
* For each example the model returns a vector of "[logits](https://developers.google.com/machine-learning/glossary#logits)" or "[log-odds](https://developers.google.com/machine-learning/glossary#log-odds)" scores, one for each class.:

```python
predictions = model(x_train[:1]).numpy()
predictions
```
![Jupyter web interface](./images/Tensorflowusecase.png)

* The `tf.nn.softmax` function converts these logits to "probabilities" for each class:

```python
tf.nn.softmax(predictions).numpy()
```
![Jupyter web interface](./images/tensor2.png)

* The `losses.SparseCategoricalCrossentropy` loss takes a vector of logits and a True index and returns a scalar loss for each example.

```python
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
```
* This loss is equal to the negative log probability of the the true class: It is zero if the model is sure of the correct class.

This untrained model gives probabilities close to random (1/10 for each class), so the initial loss should be close to -tf.log(1/10) ~= 2.3.

```python
loss_fn(y_train[:1], predictions).numpy()
```
![Jupyter web interface](./images/tensor3.png)

```python
model.compile(optimizer='adam',
loss=loss_fn,
metrics=['accuracy'])
```
* The `Model.fit` method adjusts the model parameters to minimize the loss:

```python
model.fit(x_train, y_train, epochs=5)
```

![Jupyter web interface](./images/tensor4.png)

* The `Model.evaluate` method checks the models performance, usually on a "[Validation-set](https://developers.google.com/machine-learning/glossary#validation-set)".

```python
model.evaluate(x_test, y_test, verbose=2)
```
![Jupyter web interface](./images/tensor5.png)

* The image classifier is now trained to ~98% accuracy on this dataset, If you want your model to return a probability, you can wrap the trained model, and attach the softmax to it:

```python
probability_model = tf.keras.Sequential([
model,
tf.keras.layers.Softmax()
])
```

```python
probability_model(x_test[:5])
```
![Jupyter web interface](./images/tensor6.png)
Binary file added examples/tensorflow/images/Tensorflow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/tensorflow/images/Tensorflowusecase.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/tensorflow/images/tensor2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/tensorflow/images/tensor3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/tensorflow/images/tensor4.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/tensorflow/images/tensor5.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/tensorflow/images/tensor6.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.