Skip to content

Commit

Permalink
Added Recall and F1 scores to IoU Statistics
Browse files Browse the repository at this point in the history
This involved calculating the number of false negatives in the IoU Matrix scores functions
  • Loading branch information
JacobGlennAyers committed Feb 22, 2021
1 parent f916a0c commit 4d55bab
Showing 2 changed files with 101 additions and 43 deletions.
108 changes: 75 additions & 33 deletions Microfaune_Local_Score_Package_Tutorial.ipynb
Original file line number Diff line number Diff line change
@@ -916,7 +916,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:5 out of the last 8 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f979095fd30> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
"WARNING:tensorflow:5 out of the last 8 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f85280bde50> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
]
},
{
@@ -940,7 +940,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Graph with Human Label"
"### Graph with Human Labelling"
]
},
{
@@ -952,7 +952,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:6 out of the last 9 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f97915325e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
"WARNING:tensorflow:6 out of the last 9 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f852848c700> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
]
},
{
@@ -976,7 +976,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Graph with Both Automated and Human Labels "
"### Graph with Both Automated and Human Labels \n",
" Orange ==> True Positive\n",
" Red ==> False Negative\n",
" Yellow ==> False Positive\n",
" White ==> True Negative"
]
},
{
@@ -990,7 +994,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:7 out of the last 10 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f979062b700> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
"WARNING:tensorflow:7 out of the last 10 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f84fedc6820> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
]
},
{
@@ -1417,8 +1421,11 @@
" <th>FOLDER</th>\n",
" <th>IN FILE</th>\n",
" <th>TRUE POSITIVE</th>\n",
" <th>FALSE NEGATIVE</th>\n",
" <th>FALSE POSITIVE</th>\n",
" <th>PRECISION</th>\n",
" <th>RECALL</th>\n",
" <th>F1</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
@@ -1428,6 +1435,9 @@
" <td>ScreamingPiha2.wav</td>\n",
" <td>1</td>\n",
" <td>4</td>\n",
" <td>4</td>\n",
" <td>0.2</td>\n",
" <td>0.2</td>\n",
" <td>0.2</td>\n",
" </tr>\n",
" </tbody>\n",
@@ -1438,8 +1448,8 @@
" FOLDER IN FILE \\\n",
"0 /home/jacob/Acoustic-Species-Identification/pa... ScreamingPiha2.wav \n",
"\n",
" TRUE POSITIVE FALSE POSITIVE PRECISION \n",
"0 1 4 0.2 "
" TRUE POSITIVE FALSE NEGATIVE FALSE POSITIVE PRECISION RECALL F1 \n",
"0 1 4 4 0.2 0.2 0.2 "
]
},
"execution_count": 14,
@@ -1471,10 +1481,15 @@
"text": [
"20190622_210000.WAV\n",
"20190623_222000.WAV\n",
"Division by zero setting precision, recall, and f1 to zero\n",
"BlackFacedAntbird1.wav\n",
"Division by zero setting precision, recall, and f1 to zero\n",
"HowlerMonkey1.WAV\n",
"Division by zero setting precision, recall, and f1 to zero\n",
"20190624_152000.WAV\n",
"ScreamingPiha2.wav\n"
"Division by zero setting precision, recall, and f1 to zero\n",
"ScreamingPiha2.wav\n",
"Division by zero setting precision, recall, and f1 to zero\n"
]
},
{
@@ -1501,8 +1516,11 @@
" <th>FOLDER</th>\n",
" <th>IN FILE</th>\n",
" <th>TRUE POSITIVE</th>\n",
" <th>FALSE NEGATIVE</th>\n",
" <th>FALSE POSITIVE</th>\n",
" <th>PRECISION</th>\n",
" <th>RECALL</th>\n",
" <th>F1</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
@@ -1511,48 +1529,66 @@
" <td>/home/jacob/Acoustic-Species-Identification/pa...</td>\n",
" <td>20190622_210000.WAV</td>\n",
" <td>1</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>0.3333</td>\n",
" <td>0.2</td>\n",
" <td>0.25</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>/home/jacob/Acoustic-Species-Identification/pa...</td>\n",
" <td>20190623_222000.WAV</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>0.3333</td>\n",
" <td>3</td>\n",
" <td>0.0000</td>\n",
" <td>0.0</td>\n",
" <td>0.00</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>/home/jacob/Acoustic-Species-Identification/pa...</td>\n",
" <td>BlackFacedAntbird1.wav</td>\n",
" <td>0</td>\n",
" <td>4</td>\n",
" <td>7</td>\n",
" <td>0.0000</td>\n",
" <td>0.0</td>\n",
" <td>0.00</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>/home/jacob/Acoustic-Species-Identification/pa...</td>\n",
" <td>HowlerMonkey1.WAV</td>\n",
" <td>0</td>\n",
" <td>3</td>\n",
" <td>10</td>\n",
" <td>0.0000</td>\n",
" <td>0.0</td>\n",
" <td>0.00</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>/home/jacob/Acoustic-Species-Identification/pa...</td>\n",
" <td>20190624_152000.WAV</td>\n",
" <td>0</td>\n",
" <td>7</td>\n",
" <td>3</td>\n",
" <td>0.0000</td>\n",
" <td>0.0</td>\n",
" <td>0.00</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>/home/jacob/Acoustic-Species-Identification/pa...</td>\n",
" <td>ScreamingPiha2.wav</td>\n",
" <td>1</td>\n",
" <td>4</td>\n",
" <td>0.2000</td>\n",
" <td>0</td>\n",
" <td>5</td>\n",
" <td>5</td>\n",
" <td>0.0000</td>\n",
" <td>0.0</td>\n",
" <td>0.00</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
@@ -1567,13 +1603,13 @@
"4 /home/jacob/Acoustic-Species-Identification/pa... 20190624_152000.WAV \n",
"5 /home/jacob/Acoustic-Species-Identification/pa... ScreamingPiha2.wav \n",
"\n",
" TRUE POSITIVE FALSE POSITIVE PRECISION \n",
"0 1 2 0.3333 \n",
"1 1 2 0.3333 \n",
"2 0 7 0.0000 \n",
"3 0 10 0.0000 \n",
"4 0 3 0.0000 \n",
"5 1 4 0.2000 "
" TRUE POSITIVE FALSE NEGATIVE FALSE POSITIVE PRECISION RECALL F1 \n",
"0 1 4 2 0.3333 0.2 0.25 \n",
"1 0 1 3 0.0000 0.0 0.00 \n",
"2 0 4 7 0.0000 0.0 0.00 \n",
"3 0 3 10 0.0000 0.0 0.00 \n",
"4 0 7 3 0.0000 0.0 0.00 \n",
"5 0 5 5 0.0000 0.0 0.00 "
]
},
"execution_count": 15,
@@ -1582,7 +1618,7 @@
}
],
"source": [
"stats_df = dataset_IoU_Statistics(automated_df,manual_df,threshold = 0.5)\n",
"stats_df = dataset_IoU_Statistics(automated_df,manual_df,threshold = 0.8)\n",
"stats_df"
]
},
@@ -1620,24 +1656,30 @@
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>TRUE POSITIVE</th>\n",
" <th>FALSE NEGATIVE</th>\n",
" <th>FALSE POSITIVE</th>\n",
" <th>PRECISION</th>\n",
" <th>RECALL</th>\n",
" <th>F1</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>3</td>\n",
" <td>28</td>\n",
" <td>0.0968</td>\n",
" <td>1</td>\n",
" <td>24</td>\n",
" <td>30</td>\n",
" <td>0.0323</td>\n",
" <td>0.04</td>\n",
" <td>0.0357</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" TRUE POSITIVE FALSE POSITIVE PRECISION\n",
"0 3 28 0.0968"
" TRUE POSITIVE FALSE NEGATIVE FALSE POSITIVE PRECISION RECALL F1\n",
"0 1 24 30 0.0323 0.04 0.0357"
]
},
"execution_count": 16,
@@ -1666,7 +1708,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:8 out of the last 11 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f9790441670> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
"WARNING:tensorflow:8 out of the last 11 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f84fe396790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
]
},
{
@@ -1697,7 +1739,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:9 out of the last 12 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f97c752f9d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
"WARNING:tensorflow:9 out of the last 12 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f85283a2a60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
]
},
{
@@ -1726,7 +1768,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:9 out of the last 11 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f9792fe89d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
"WARNING:tensorflow:9 out of the last 11 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f8529fe5a60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
]
},
{
@@ -1771,7 +1813,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"/home/jacob/Desktop/EngineersForExploration/Bioacoustics/Automated_Audio_Labelling_System_AID/microfaune_local_score.py:561: SettingWithCopyWarning: \n",
"/home/jacob/Desktop/EngineersForExploration/Bioacoustics/Automated_Audio_Labelling_System_AID/microfaune_local_score.py:574: SettingWithCopyWarning: \n",
"A value is trying to be set on a copy of a slice from a DataFrame.\n",
"Try using .loc[row_indexer,col_indexer] = value instead\n",
"\n",
@@ -2220,14 +2262,15 @@
"20190622_210000.WAV\n",
"20190623_222000.WAV\n",
"BlackFacedAntbird1.wav\n",
"HowlerMonkey1.WAV\n"
"HowlerMonkey1.WAV\n",
"20190624_152000.WAV\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/jacob/Desktop/EngineersForExploration/Bioacoustics/Automated_Audio_Labelling_System_AID/microfaune_local_score.py:625: SettingWithCopyWarning: \n",
"/home/jacob/Desktop/EngineersForExploration/Bioacoustics/Automated_Audio_Labelling_System_AID/microfaune_local_score.py:650: SettingWithCopyWarning: \n",
"A value is trying to be set on a copy of a slice from a DataFrame.\n",
"Try using .loc[row_indexer,col_indexer] = value instead\n",
"\n",
@@ -2239,7 +2282,6 @@
"name": "stdout",
"output_type": "stream",
"text": [
"20190624_152000.WAV\n",
"ScreamingPiha2.wav\n"
]
}
Loading

0 comments on commit 4d55bab

Please sign in to comment.