Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/add visualization component #78

Merged
merged 7 commits into from
Dec 19, 2023
Merged

Feature/add visualization component #78

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
6bd58d5
wip
denysgerasymuk799 Dec 18, 2023
d5893af
Cleaned unnecessary files
denysgerasymuk799 Dec 19, 2023
fe7fbf4
Merge pull request #75 from DataResponsibly/feature/add_visualization…
denysgerasymuk799 Dec 19, 2023
ed7ab57
Merge branch 'virny_demo' into feature/add_visualization_component
denysgerasymuk799 Dec 19, 2023
933100c
wip1
denysgerasymuk799 Dec 19, 2023
986438d
Resolved merge conflicts
denysgerasymuk799 Dec 19, 2023
d8f64d0
Resolved merge conflicts
denysgerasymuk799 Dec 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 18 additions & 37 deletions docs/examples/Multiple_Models_Interface_For_Incremental_ML.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,8 @@
],
"metadata": {
"collapsed": false
}
},
"id": "d8505edef7184a5a"
},
{
"cell_type": "markdown",
Expand All @@ -156,7 +157,8 @@
],
"metadata": {
"collapsed": false
}
},
"id": "c84bae470652be94"
},
{
"cell_type": "markdown",
Expand All @@ -173,7 +175,8 @@
],
"metadata": {
"collapsed": false
}
},
"id": "8b41b746e152c76f"
},
{
"cell_type": "code",
Expand All @@ -195,7 +198,8 @@
],
"metadata": {
"collapsed": false
}
},
"id": "a878d125e8bfaf4d"
},
{
"cell_type": "code",
Expand All @@ -207,7 +211,8 @@
],
"metadata": {
"collapsed": false
}
},
"id": "53d2fcd40c862014"
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -307,7 +312,8 @@
],
"metadata": {
"collapsed": false
}
},
"id": "8feb498942cc2a8c"
},
{
"cell_type": "code",
Expand All @@ -318,7 +324,8 @@
],
"metadata": {
"collapsed": false
}
},
"id": "7915190e0847f1a7"
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -581,7 +588,8 @@
],
"metadata": {
"collapsed": false
}
},
"id": "ca3fe31f0515a973"
},
{
"cell_type": "code",
Expand All @@ -602,7 +610,8 @@
],
"metadata": {
"collapsed": false
}
},
"id": "dfc57f1870ed71d1"
},
{
"cell_type": "code",
Expand Down Expand Up @@ -647,34 +656,6 @@
")"
]
},
{
"cell_type": "markdown",
"id": "55e6ce42",
"metadata": {},
"source": [
"Create an analysis report. It includes correspondent visualizations and details about your result metrics."
]
},
{
"cell_type": "code",
"execution_count": 59,
"id": "5a3811ff",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": "<IPython.core.display.Markdown object>",
"text/markdown": "App saved to ./docs/examples/results/reports/COMPAS_Without_Sensitive_Attributes_Metrics_Report_20230812__223906.html"
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"visualizer.create_html_report(report_type=ReportType.MULTIPLE_RUNS_MULTIPLE_MODELS,\n",
" report_save_path=os.path.join(ROOT_DIR, \"results\", \"reports\"))"
]
},
{
"cell_type": "code",
"execution_count": 59,
Expand Down
39 changes: 0 additions & 39 deletions docs/examples/Multiple_Models_Interface_Use_Case.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1096,45 +1096,6 @@
")"
]
},
{
"cell_type": "markdown",
"id": "55e6ce42",
"metadata": {
"is_executing": true
},
"source": [
"Create an analysis report. It includes correspondent visualizations and details about your result metrics."
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "5a3811ff",
"metadata": {
"ExecuteTime": {
"end_time": "2023-10-21T20:58:36.148703Z",
"start_time": "2023-10-21T20:58:35.395033Z"
}
},
"outputs": [
{
"ename": "AttributeError",
"evalue": "module 'datapane' has no attribute 'Report'",
"output_type": "error",
"traceback": [
"\u001B[0;31m---------------------------------------------------------------------------\u001B[0m",
"\u001B[0;31mAttributeError\u001B[0m Traceback (most recent call last)",
"Cell \u001B[0;32mIn[27], line 1\u001B[0m\n\u001B[0;32m----> 1\u001B[0m \u001B[43mvisualizer\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mcreate_html_report\u001B[49m\u001B[43m(\u001B[49m\u001B[43mreport_type\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mReportType\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mMULTIPLE_RUNS_MULTIPLE_MODELS\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m 2\u001B[0m \u001B[43m \u001B[49m\u001B[43mreport_save_path\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mos\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mpath\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mjoin\u001B[49m\u001B[43m(\u001B[49m\u001B[43mROOT_DIR\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;124;43m\"\u001B[39;49m\u001B[38;5;124;43mresults\u001B[39;49m\u001B[38;5;124;43m\"\u001B[39;49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;124;43m\"\u001B[39;49m\u001B[38;5;124;43mreports\u001B[39;49m\u001B[38;5;124;43m\"\u001B[39;49m\u001B[43m)\u001B[49m\u001B[43m)\u001B[49m\n",
"File \u001B[0;32m~/UCU/4course_2term/Bachelor_Thesis/Code/Virny/virny/custom_classes/metrics_visualizer.py:480\u001B[0m, in \u001B[0;36mMetricsVisualizer.create_html_report\u001B[0;34m(self, report_type, report_save_path)\u001B[0m\n\u001B[1;32m 475\u001B[0m \u001B[38;5;28;01mif\u001B[39;00m report_type \u001B[38;5;241m==\u001B[39m ReportType\u001B[38;5;241m.\u001B[39mMULTIPLE_RUNS_MULTIPLE_MODELS:\n\u001B[1;32m 476\u001B[0m boxes_and_whiskers_plot \u001B[38;5;241m=\u001B[39m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mcreate_boxes_and_whiskers_for_models_multiple_runs(\n\u001B[1;32m 477\u001B[0m metrics_lst\u001B[38;5;241m=\u001B[39m[\u001B[38;5;124m'\u001B[39m\u001B[38;5;124mStd\u001B[39m\u001B[38;5;124m'\u001B[39m, \u001B[38;5;124m'\u001B[39m\u001B[38;5;124mIQR\u001B[39m\u001B[38;5;124m'\u001B[39m, \u001B[38;5;124m'\u001B[39m\u001B[38;5;124mJitter\u001B[39m\u001B[38;5;124m'\u001B[39m, \u001B[38;5;124m'\u001B[39m\u001B[38;5;124mLabel_Stability\u001B[39m\u001B[38;5;124m'\u001B[39m, \u001B[38;5;124m'\u001B[39m\u001B[38;5;124mAccuracy\u001B[39m\u001B[38;5;124m'\u001B[39m, \u001B[38;5;124m'\u001B[39m\u001B[38;5;124mTPR\u001B[39m\u001B[38;5;124m'\u001B[39m, \u001B[38;5;124m'\u001B[39m\u001B[38;5;124mTNR\u001B[39m\u001B[38;5;124m'\u001B[39m, \u001B[38;5;124m'\u001B[39m\u001B[38;5;124mFPR\u001B[39m\u001B[38;5;124m'\u001B[39m, \u001B[38;5;124m'\u001B[39m\u001B[38;5;124mFNR\u001B[39m\u001B[38;5;124m'\u001B[39m]\n\u001B[1;32m 478\u001B[0m )\n\u001B[0;32m--> 480\u001B[0m \u001B[43mdp\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mReport\u001B[49m(\u001B[38;5;124m\"\u001B[39m\u001B[38;5;124m# Fairness and Stability Report\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n\u001B[1;32m 481\u001B[0m general_desc,\n\u001B[1;32m 482\u001B[0m \n\u001B[1;32m 483\u001B[0m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124m## Model Composed Metrics\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n\u001B[1;32m 484\u001B[0m composed_metrics_desc,\n\u001B[1;32m 485\u001B[0m dp\u001B[38;5;241m.\u001B[39mDataTable(\u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mmodels_composed_metrics_df),\n\u001B[1;32m 486\u001B[0m \n\u001B[1;32m 487\u001B[0m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124m## Boxes and Whiskers Plot Based On Multiple Models Runs\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n\u001B[1;32m 488\u001B[0m boxes_and_whiskers_plot_desc,\n\u001B[1;32m 489\u001B[0m dp\u001B[38;5;241m.\u001B[39mPlot(boxes_and_whiskers_plot),\n\u001B[1;32m 490\u001B[0m \n\u001B[1;32m 491\u001B[0m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124m## Overall Fairness and Stability Model Metrics Comparison\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n\u001B[1;32m 492\u001B[0m overall_metrics_desc,\n\u001B[1;32m 493\u001B[0m dp\u001B[38;5;241m.\u001B[39mPlot(fairness_overall_metrics_bar_chart, responsive\u001B[38;5;241m=\u001B[39m\u001B[38;5;28;01mFalse\u001B[39;00m),\n\u001B[1;32m 494\u001B[0m dp\u001B[38;5;241m.\u001B[39mPlot(variance_overall_metrics_bar_chart, responsive\u001B[38;5;241m=\u001B[39m\u001B[38;5;28;01mFalse\u001B[39;00m),\n\u001B[1;32m 495\u001B[0m \n\u001B[1;32m 496\u001B[0m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124m## Fairness and Stability Interactive Bar Chart\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n\u001B[1;32m 497\u001B[0m individual_metrics_interactive_bar_chart_desc,\n\u001B[1;32m 498\u001B[0m dp\u001B[38;5;241m.\u001B[39mPlot(interactive_bar_chart),\n\u001B[1;32m 499\u001B[0m \n\u001B[1;32m 500\u001B[0m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124m## Model Ranks Based On Group Fairness and Stability Metrics\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n\u001B[1;32m 501\u001B[0m model_ranked_heatmap_desc,\n\u001B[1;32m 502\u001B[0m dp\u001B[38;5;241m.\u001B[39mPlot(model_rank_heatmap, responsive\u001B[38;5;241m=\u001B[39m\u001B[38;5;28;01mFalse\u001B[39;00m),\n\u001B[1;32m 503\u001B[0m \n\u001B[1;32m 504\u001B[0m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124m## Total Ranks Sum For Group Fairness and Stability Metrics\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n\u001B[1;32m 505\u001B[0m overall_model_ranked_heatmap_desc,\n\u001B[1;32m 506\u001B[0m dp\u001B[38;5;241m.\u001B[39mPlot(total_model_rank_heatmap, responsive\u001B[38;5;241m=\u001B[39m\u001B[38;5;28;01mFalse\u001B[39;00m),\n\u001B[1;32m 507\u001B[0m )\u001B[38;5;241m.\u001B[39msave(path\u001B[38;5;241m=\u001B[39mos\u001B[38;5;241m.\u001B[39mpath\u001B[38;5;241m.\u001B[39mjoin(report_save_path, report_filename))\n\u001B[1;32m 508\u001B[0m \u001B[38;5;28;01melse\u001B[39;00m:\n\u001B[1;32m 509\u001B[0m dp\u001B[38;5;241m.\u001B[39mReport(\u001B[38;5;124m\"\u001B[39m\u001B[38;5;124m# Fairness and Stability Report\u001B[39m\u001B[38;5;124m\"\u001B[39m,\n\u001B[1;32m 510\u001B[0m general_desc,\n\u001B[1;32m 511\u001B[0m \n\u001B[0;32m (...)\u001B[0m\n\u001B[1;32m 531\u001B[0m dp\u001B[38;5;241m.\u001B[39mPlot(total_model_rank_heatmap, responsive\u001B[38;5;241m=\u001B[39m\u001B[38;5;28;01mFalse\u001B[39;00m),\n\u001B[1;32m 532\u001B[0m )\u001B[38;5;241m.\u001B[39msave(path\u001B[38;5;241m=\u001B[39mos\u001B[38;5;241m.\u001B[39mpath\u001B[38;5;241m.\u001B[39mjoin(report_save_path, report_filename))\n",
"\u001B[0;31mAttributeError\u001B[0m: module 'datapane' has no attribute 'Report'"
]
}
],
"source": [
"visualizer.create_html_report(report_type=ReportType.MULTIPLE_RUNS_MULTIPLE_MODELS,\n",
" report_save_path=os.path.join(ROOT_DIR, \"results\", \"reports\"))"
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down
Loading