Skip to content

Commit

Permalink
Merge pull request #19 from UCSD-E4E/master
Browse files Browse the repository at this point in the history 
Added Recall and F1 scores to IoU Statistics
  • Loading branch information
@JacobGlennAyers
JacobGlennAyers authored Feb 22, 2021
2 parents 712d5be + 4d55bab commit aced641
Show file tree
Hide file tree
Showing 2 changed files with 101 additions and 43 deletions.
108 changes: 75 additions & 33 deletions Microfaune_Local_Score_Package_Tutorial.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -916,7 +916,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:5 out of the last 8 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f979095fd30> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
"WARNING:tensorflow:5 out of the last 8 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f85280bde50> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
]
},
{
Expand All @@ -940,7 +940,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Graph with Human Label"
"### Graph with Human Labelling"
]
},
{
Expand All @@ -952,7 +952,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:6 out of the last 9 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f97915325e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
"WARNING:tensorflow:6 out of the last 9 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f852848c700> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
]
},
{
Expand All @@ -976,7 +976,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Graph with Both Automated and Human Labels "
"### Graph with Both Automated and Human Labels \n",
" Orange ==> True Positive\n",
" Red ==> False Negative\n",
" Yellow ==> False Positive\n",
" White ==> True Negative"
]
},
{
Expand All @@ -990,7 +994,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:7 out of the last 10 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f979062b700> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
"WARNING:tensorflow:7 out of the last 10 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f84fedc6820> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
]
},
{
Expand Down Expand Up @@ -1417,8 +1421,11 @@
" <th>FOLDER</th>\n",
" <th>IN FILE</th>\n",
" <th>TRUE POSITIVE</th>\n",
" <th>FALSE NEGATIVE</th>\n",
" <th>FALSE POSITIVE</th>\n",
" <th>PRECISION</th>\n",
" <th>RECALL</th>\n",
" <th>F1</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
Expand All @@ -1428,6 +1435,9 @@
" <td>ScreamingPiha2.wav</td>\n",
" <td>1</td>\n",
" <td>4</td>\n",
" <td>4</td>\n",
" <td>0.2</td>\n",
" <td>0.2</td>\n",
" <td>0.2</td>\n",
" </tr>\n",
" </tbody>\n",
Expand All @@ -1438,8 +1448,8 @@
" FOLDER IN FILE \\\n",
"0 /home/jacob/Acoustic-Species-Identification/pa... ScreamingPiha2.wav \n",
"\n",
" TRUE POSITIVE FALSE POSITIVE PRECISION \n",
"0 1 4 0.2 "
" TRUE POSITIVE FALSE NEGATIVE FALSE POSITIVE PRECISION RECALL F1 \n",
"0 1 4 4 0.2 0.2 0.2 "
]
},
"execution_count": 14,
Expand Down Expand Up @@ -1471,10 +1481,15 @@
"text": [
"20190622_210000.WAV\n",
"20190623_222000.WAV\n",
"Division by zero setting precision, recall, and f1 to zero\n",
"BlackFacedAntbird1.wav\n",
"Division by zero setting precision, recall, and f1 to zero\n",
"HowlerMonkey1.WAV\n",
"Division by zero setting precision, recall, and f1 to zero\n",
"20190624_152000.WAV\n",
"ScreamingPiha2.wav\n"
"Division by zero setting precision, recall, and f1 to zero\n",
"ScreamingPiha2.wav\n",
"Division by zero setting precision, recall, and f1 to zero\n"
]
},
{
Expand All @@ -1501,8 +1516,11 @@
" <th>FOLDER</th>\n",
" <th>IN FILE</th>\n",
" <th>TRUE POSITIVE</th>\n",
" <th>FALSE NEGATIVE</th>\n",
" <th>FALSE POSITIVE</th>\n",
" <th>PRECISION</th>\n",
" <th>RECALL</th>\n",
" <th>F1</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
Expand All @@ -1511,48 +1529,66 @@
" <td>/home/jacob/Acoustic-Species-Identification/pa...</td>\n",
" <td>20190622_210000.WAV</td>\n",
" <td>1</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>0.3333</td>\n",
" <td>0.2</td>\n",
" <td>0.25</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>/home/jacob/Acoustic-Species-Identification/pa...</td>\n",
" <td>20190623_222000.WAV</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>0.3333</td>\n",
" <td>3</td>\n",
" <td>0.0000</td>\n",
" <td>0.0</td>\n",
" <td>0.00</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>/home/jacob/Acoustic-Species-Identification/pa...</td>\n",
" <td>BlackFacedAntbird1.wav</td>\n",
" <td>0</td>\n",
" <td>4</td>\n",
" <td>7</td>\n",
" <td>0.0000</td>\n",
" <td>0.0</td>\n",
" <td>0.00</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>/home/jacob/Acoustic-Species-Identification/pa...</td>\n",
" <td>HowlerMonkey1.WAV</td>\n",
" <td>0</td>\n",
" <td>3</td>\n",
" <td>10</td>\n",
" <td>0.0000</td>\n",
" <td>0.0</td>\n",
" <td>0.00</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>/home/jacob/Acoustic-Species-Identification/pa...</td>\n",
" <td>20190624_152000.WAV</td>\n",
" <td>0</td>\n",
" <td>7</td>\n",
" <td>3</td>\n",
" <td>0.0000</td>\n",
" <td>0.0</td>\n",
" <td>0.00</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>/home/jacob/Acoustic-Species-Identification/pa...</td>\n",
" <td>ScreamingPiha2.wav</td>\n",
" <td>1</td>\n",
" <td>4</td>\n",
" <td>0.2000</td>\n",
" <td>0</td>\n",
" <td>5</td>\n",
" <td>5</td>\n",
" <td>0.0000</td>\n",
" <td>0.0</td>\n",
" <td>0.00</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
Expand All @@ -1567,13 +1603,13 @@
"4 /home/jacob/Acoustic-Species-Identification/pa... 20190624_152000.WAV \n",
"5 /home/jacob/Acoustic-Species-Identification/pa... ScreamingPiha2.wav \n",
"\n",
" TRUE POSITIVE FALSE POSITIVE PRECISION \n",
"0 1 2 0.3333 \n",
"1 1 2 0.3333 \n",
"2 0 7 0.0000 \n",
"3 0 10 0.0000 \n",
"4 0 3 0.0000 \n",
"5 1 4 0.2000 "
" TRUE POSITIVE FALSE NEGATIVE FALSE POSITIVE PRECISION RECALL F1 \n",
"0 1 4 2 0.3333 0.2 0.25 \n",
"1 0 1 3 0.0000 0.0 0.00 \n",
"2 0 4 7 0.0000 0.0 0.00 \n",
"3 0 3 10 0.0000 0.0 0.00 \n",
"4 0 7 3 0.0000 0.0 0.00 \n",
"5 0 5 5 0.0000 0.0 0.00 "
]
},
"execution_count": 15,
Expand All @@ -1582,7 +1618,7 @@
}
],
"source": [
"stats_df = dataset_IoU_Statistics(automated_df,manual_df,threshold = 0.5)\n",
"stats_df = dataset_IoU_Statistics(automated_df,manual_df,threshold = 0.8)\n",
"stats_df"
]
},
Expand Down Expand Up @@ -1620,24 +1656,30 @@
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>TRUE POSITIVE</th>\n",
" <th>FALSE NEGATIVE</th>\n",
" <th>FALSE POSITIVE</th>\n",
" <th>PRECISION</th>\n",
" <th>RECALL</th>\n",
" <th>F1</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>3</td>\n",
" <td>28</td>\n",
" <td>0.0968</td>\n",
" <td>1</td>\n",
" <td>24</td>\n",
" <td>30</td>\n",
" <td>0.0323</td>\n",
" <td>0.04</td>\n",
" <td>0.0357</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" TRUE POSITIVE FALSE POSITIVE PRECISION\n",
"0 3 28 0.0968"
" TRUE POSITIVE FALSE NEGATIVE FALSE POSITIVE PRECISION RECALL F1\n",
"0 1 24 30 0.0323 0.04 0.0357"
]
},
"execution_count": 16,
Expand Down Expand Up @@ -1666,7 +1708,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:8 out of the last 11 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f9790441670> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
"WARNING:tensorflow:8 out of the last 11 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f84fe396790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
]
},
{
Expand Down Expand Up @@ -1697,7 +1739,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:9 out of the last 12 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f97c752f9d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
"WARNING:tensorflow:9 out of the last 12 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f85283a2a60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
]
},
{
Expand Down Expand Up @@ -1726,7 +1768,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:9 out of the last 11 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f9792fe89d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
"WARNING:tensorflow:9 out of the last 11 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f8529fe5a60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
]
},
{
Expand Down Expand Up @@ -1771,7 +1813,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"/home/jacob/Desktop/EngineersForExploration/Bioacoustics/Automated_Audio_Labelling_System_AID/microfaune_local_score.py:561: SettingWithCopyWarning: \n",
"/home/jacob/Desktop/EngineersForExploration/Bioacoustics/Automated_Audio_Labelling_System_AID/microfaune_local_score.py:574: SettingWithCopyWarning: \n",
"A value is trying to be set on a copy of a slice from a DataFrame.\n",
"Try using .loc[row_indexer,col_indexer] = value instead\n",
"\n",
Expand Down Expand Up @@ -2220,14 +2262,15 @@
"20190622_210000.WAV\n",
"20190623_222000.WAV\n",
"BlackFacedAntbird1.wav\n",
"HowlerMonkey1.WAV\n"
"HowlerMonkey1.WAV\n",
"20190624_152000.WAV\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/jacob/Desktop/EngineersForExploration/Bioacoustics/Automated_Audio_Labelling_System_AID/microfaune_local_score.py:625: SettingWithCopyWarning: \n",
"/home/jacob/Desktop/EngineersForExploration/Bioacoustics/Automated_Audio_Labelling_System_AID/microfaune_local_score.py:650: SettingWithCopyWarning: \n",
"A value is trying to be set on a copy of a slice from a DataFrame.\n",
"Try using .loc[row_indexer,col_indexer] = value instead\n",
"\n",
Expand All @@ -2239,7 +2282,6 @@
"name": "stdout",
"output_type": "stream",
"text": [
"20190624_152000.WAV\n",
"ScreamingPiha2.wav\n"
]
}
Expand Down
Loading

0 comments on commit aced641

Please sign in to comment.