Gsoc2021 - Phase 3 #54

akshitpatel01 · 2021-08-14T12:56:49Z

No description provided.

…wn, jump to page on clicking a row in table

DvdMgr · 2021-08-20T09:49:56Z

sem/dashboard/README.md

@@ -0,0 +1,36 @@
+# Overview of Dashboard features
+The objective of this interactive dashboard is to allow users to visualize the logs genearted by ns-3 efficiently. 


Suggested change

The objective of this interactive dashboard is to allow users to visualize the logs genearted by ns-3 efficiently.

The objective of this interactive dashboard is to allow users to visualize the logs generated by ns-3 efficiently.

DvdMgr · 2021-08-20T09:50:28Z

sem/dashboard/README.md

+### The layout and features of the dashboard:
+Starting from top, the dashboard is divided into the following sections:
+###### Dropdown filters along with a time filter:
+All the filter values in the dropdown are populated (according to the log file passed) as the dashboard first initializes. The dropdown filters have an inbuilt search box as well 'Select All' and 'Deselect All' buttons. A few points to note about the filters:


Suggested change

All the filter values in the dropdown are populated (according to the log file passed) as the dashboard first initializes. The dropdown filters have an inbuilt search box as well 'Select All' and 'Deselect All' buttons. A few points to note about the filters:

All the filter values in the dropdown are populated (according to the log file passed) as the dashboard first initializes. The dropdown filters have an inbuilt search box as well as 'Select All' and 'Deselect All' buttons. A few points to note about the filters:

DvdMgr · 2021-08-20T09:51:06Z

sem/dashboard/README.md

+Starting from top, the dashboard is divided into the following sections:
+###### Dropdown filters along with a time filter:
+All the filter values in the dropdown are populated (according to the log file passed) as the dashboard first initializes. The dropdown filters have an inbuilt search box as well 'Select All' and 'Deselect All' buttons. A few points to note about the filters:
+- These filters act as global flters for both the graph and the table.


Suggested change

- These filters act as global flters for both the graph and the table.

- These filters act as global filters for both the graph and the table.

DvdMgr · 2021-08-20T09:51:20Z

sem/dashboard/README.md

+###### Dropdown filters along with a time filter:
+All the filter values in the dropdown are populated (according to the log file passed) as the dashboard first initializes. The dropdown filters have an inbuilt search box as well 'Select All' and 'Deselect All' buttons. A few points to note about the filters:
+- These filters act as global flters for both the graph and the table.
+- Any unspecified filters are assmed to be 'free' and can take any value. For example, if 'context=[0,1]', 'function=['f1']' and 'Upper Time Limit:1', then the logs with (context=0 or context=1) and (function=f1) and (timestamp<1) will be displayed.


Suggested change

- Any unspecified filters are assmed to be 'free' and can take any value. For example, if 'context=[0,1]', 'function=['f1']' and 'Upper Time Limit:1', then the logs with (context=0 or context=1) and (function=f1) and (timestamp<1) will be displayed.

- Any unspecified filters are assumed to be 'free' and can take any value. For example, if 'context=[0,1]', 'function=['f1']' and 'Upper Time Limit:1', then the logs with (context=0 or context=1) and (function=f1) and (timestamp<1) will be displayed.

DvdMgr · 2021-08-20T09:52:12Z

sem/dashboard/README.md

+- If no component is selected from the component filter, then the severity class filter acts as an independent filter.
+
+###### The Graph:
+This is a line graph which allows users to easily analyze logs with resepect to a time axis. A few points to note about the graph:


Suggested change

This is a line graph which allows users to easily analyze logs with resepect to a time axis. A few points to note about the graph:

This is a line graph which allows users to easily analyze logs with respect to a time axis. A few points to note about the graph:

DvdMgr · 2021-08-20T09:52:54Z

sem/dashboard/README.md

+  - Hold Ctrl and drag the mouse to pan the graph.
+
+###### The Table:
+The table allows users to effectivel visualize the logs in a tabular way. A few points to note about the table:


Suggested change

The table allows users to effectivel visualize the logs in a tabular way. A few points to note about the table:

The table allows users to effectively visualize the logs in a tabular way. A few points to note about the table:

DvdMgr · 2021-08-20T09:53:26Z

sem/dashboard/README.md

+###### The Table:
+The table allows users to effectivel visualize the logs in a tabular way. A few points to note about the table:
+- The 'Show entries' dropdown menu allows the users to select the number of logs to be displayed on each page.
+- The search box above the table allows the users to search in sepecific columns. the searchable columns can be selected/deselected using the 'Search columns' dropdown as per need.


Suggested change

- The search box above the table allows the users to search in sepecific columns. the searchable columns can be selected/deselected using the 'Search columns' dropdown as per need.

- The search box above the table allows the users to search in specific columns. the searchable columns can be selected/deselected using the 'Search columns' dropdown as needed.

DvdMgr · 2021-08-20T09:54:04Z

sem/dashboard/README.md

+- The 'Show entries' dropdown menu allows the users to select the number of logs to be displayed on each page.
+- The search box above the table allows the users to search in sepecific columns. the searchable columns can be selected/deselected using the 'Search columns' dropdown as per need.
+- The search box is specific to the table and has no effect on the graph.
+- Clicking on a row in the table jumps to the page where that particular row is diplayed in the table. For example, if user searches for 'xyz' and logs are displayed based on this search query, then on clicking any row, the table will jump to the page (in the original table without the search query) which contains that particular row. 


Suggested change

- Clicking on a row in the table jumps to the page where that particular row is diplayed in the table. For example, if user searches for 'xyz' and logs are displayed based on this search query, then on clicking any row, the table will jump to the page (in the original table without the search query) which contains that particular row.

- Clicking on a row in the table jumps to the page where that particular row is displayed in the table. For example, if user searches for 'xyz' and logs are displayed based on this search query, then on clicking any row, the table will jump to the page (in the original table without the search query) which contains that particular row.

DvdMgr · 2021-08-20T09:54:26Z

sem/dashboard/README.md

+- The search box above the table allows the users to search in sepecific columns. the searchable columns can be selected/deselected using the 'Search columns' dropdown as per need.
+- The search box is specific to the table and has no effect on the graph.
+- Clicking on a row in the table jumps to the page where that particular row is diplayed in the table. For example, if user searches for 'xyz' and logs are displayed based on this search query, then on clicking any row, the table will jump to the page (in the original table without the search query) which contains that particular row. 
+- All the columns of the table (except extended_context) can be sorted either in ascending or descending order. 


Suggested change

- All the columns of the table (except extended_context) can be sorted either in ascending or descending order.

- All the columns of the table (except for extended_context) can be sorted either in ascending or descending order.

mattia-lecci

Good job as always!

I installed it in my home pc and was missing the following dependencies:

flask (installed 2.0.1)
tinydb-smartcache (installed 2.0.0)

They could be added to the .toml file as already discussed

Furthermore, when hovering over the plot it would be nice to also get time and context information, even if it's redundant with the information from the plot's axis, and even the index.

mattia-lecci · 2021-08-20T14:22:28Z

examples/logging_example2.py

@@ -31,18 +31,21 @@
    'STA1_x': 5.0,
    'stepsTime': 1,
    'minPower': 0,
-    'steps': 2,


is this necessary?

This is just to ensure that the example does not take a very long time to run. The default steps value in the ns-3 example is set to 200. According to me, that might take quite a long time to run. What do you guys think?

sorry for not being clear, i meant: why did this change from 2 to [2]? It's the only element set as a list.
If it's possible to make this dictionary uniform (either all lists or all values) i'd go for it

examples/logging_example2.py

mattia-lecci · 2021-08-20T14:25:27Z

examples/logging_example2.py

 }
-# log_components = 'NS_LOG="PowerAdaptationDistance=all:FrameExchangeManager=info"'
-
+# log_components = 'NS_LOG="PowerAdaptationDistance=debug:ParfWifiManager=info"'


i suggest to show the two formats in the same configuration

mattia-lecci · 2021-08-20T14:27:03Z

examples/logging_example2.py

-    logging.wipe_results(db, db_path)
+    db, data_dir = logging.process_logs(log_path[0])
+    print('Filtered Logs:')
+    print(logging.filter_logs(db,


maybe too verbose?

Mattia, can you please check if you have the latest version of the example?

this is the output that i get running the script:

[{'index': 50667, 'time': 0.800403247, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 50734, 'time': 0.800725281, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 50801, 'time': 0.801074315, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 50863, 'time': 0.801459349, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 50930, 'time': 0.801817383, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 50997, 'time': 0.802184417, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51064, 'time': 0.802623451, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51131, 'time': 0.803026485, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51198, 'time': 0.803438519, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51265, 'time': 0.803868553, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51332, 'time': 0.804217587, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51394, 'time': 0.804602621, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51461, 'time': 0.804951655, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51528, 'time': 0.805318689, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51595, 'time': 0.805676723, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51657, 'time': 0.806052757, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51724, 'time': 0.806392791, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51791, 'time': 0.806723825, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51858, 'time': 0.807162859, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51925, 'time': 0.807619893, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51992, 'time': 0.808004927, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 52054, 'time': 0.808344961, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 52121, 'time': 0.808792995, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 52188, 'time': 0.809169029, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 52255, 'time': 0.809626063, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 52322, 'time': 0.809948097, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}]

My point is: does a user care about this output when running the example or is it just interested into the dashboard?
Or should we separate this in a third example showing the logging/filtering programmatic API, instead?

My point was just to demonstrate the API output but I agree with your point. I probably wouldn't make a new example just for this. Maybe keep the print statement and just comment it?

sem/dashboard/README.md

sem/dashboard/templates/dashboard.html

sem/dashboard/views.py

mattia-lecci · 2021-08-20T14:51:28Z

sem/logging.py

@@ -92,6 +94,7 @@ def parse_logs(log_file):
    # Note: '^$' - Ensures that the entire regex matches the entire log line.
    regex = re.compile(r'^' + time_re + context_extended_context_re + component_function_arguments_re + severity_class_message_re + r'$')

+    idx = 0
    with open(log_file) as f:
        for log in f:


one of python's best practices is to avoid creating manual counters when using loops.

Suggested change

for log in f:

for idx, log in enumerate(f):

I initially thought of this but we have to increment the index only if the log line is parsed correctly. If the control goes here
if groups is None: warnings.warn("Log format is not consistent with prefix_all. Skipping log '%s'" % log, RuntimeWarning, stacklevel=2) continue

we do not increment the index.

Okay, i get it now, thank you for the explanation.
Though, I would suggest the opposite: in this way row number and log index would be consistent, and a user would easily be able to find any given line if he needs to.
On the other hand, having the index and the row number as possibly different might be annoying for a user.

What do you guys think?

I did not quite understand what approach did you referred to as the opposite. Can you please elaborate a little? Also, I would like to just point out that the index that we store in the database is not visible to the user at any point. It is just to make searching for the log entries easier and faster(as the index are always sorted).

sem/utils.py

akshitpatel01 · 2021-08-22T13:59:22Z

I installed it in my home pc and was missing the following dependencies:
flask (installed 2.0.1)
tinydb-smartcache (installed 2.0.0)
They could be added to the .toml file as already discussed

Thanks! I verified these on my system as well.

akshitpatel01 and others added 24 commits August 14, 2021 18:06

Add dashboard table

5fe68b2

Small fix

8b3bb26

Small fix

b265028

Clean views.py

19194ce

Enable component filter

b59167b

Create README.md

aa884ab

Added interactive plot

fc2bbc8

Enabled filters for graph and added hover labels

73a0f43

Fixed a bug in hover data shown by plot

a37d13c

Add spinning wheel, link dashboard with SEM, labels for filter dropdo…

5e19788

…wn, jump to page on clicking a row in table

Fixed jitter logs

2898966

Improved hover labels, Added dropdown to set searchable columns

17f3164

Increase point size

fec1f3a

Uncommented Buffering spinner

dce7fe1

Update logging_example2.py

f458889

Fixed Ajax error, added drag zoom

f274fd4

Deleted root dashboard

f022faa

Small fix and add dashboard image

d2a73ac

Update Readme.md

cb88733

Small fixes

57ecadc

Restructured dashboard with proper names

c0b4fef

Cleaned up code

89738d2

Added a TODO

db26ebb

Set theme jekyll-theme-cayman

6918ef4

DvdMgr reviewed Aug 20, 2021

View reviewed changes

mattia-lecci reviewed Aug 20, 2021

View reviewed changes

akshitpatel01 added 4 commits August 22, 2021 17:49

Fixed row-click to jump to row-page

e0e054e

Fixed typos in README.md

04e5da3

Fixed typos in README.md

8e7cfe7

Updated ns-3 path in the logging examples

12e8ada

akshitpatel01 added 5 commits August 22, 2021 18:34

Updated README.md

74bcfd4

Updated README.md

fafd77d

Small fixes

ece7f76

Removed views.py

9f1e539

Update dependencies

12146fa

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gsoc2021 - Phase 3 #54

Gsoc2021 - Phase 3 #54

akshitpatel01 commented Aug 14, 2021

DvdMgr Aug 20, 2021

DvdMgr Aug 20, 2021

DvdMgr Aug 20, 2021

DvdMgr Aug 20, 2021

DvdMgr Aug 20, 2021

DvdMgr Aug 20, 2021

DvdMgr Aug 20, 2021

DvdMgr Aug 20, 2021

DvdMgr Aug 20, 2021

mattia-lecci left a comment

mattia-lecci Aug 20, 2021

akshitpatel01 Aug 22, 2021

mattia-lecci Aug 23, 2021

mattia-lecci Aug 20, 2021

mattia-lecci Aug 20, 2021

akshitpatel01 Aug 22, 2021

mattia-lecci Aug 23, 2021

mattia-lecci Aug 23, 2021

akshitpatel01 Aug 28, 2021

mattia-lecci Aug 20, 2021

akshitpatel01 Aug 22, 2021

mattia-lecci Aug 23, 2021

akshitpatel01 Aug 28, 2021

akshitpatel01 commented Aug 22, 2021

		@@ -0,0 +1,36 @@
		# Overview of Dashboard features
		The objective of this interactive dashboard is to allow users to visualize the logs genearted by ns-3 efficiently.

	All the filter values in the dropdown are populated (according to the log file passed) as the dashboard first initializes. The dropdown filters have an inbuilt search box as well 'Select All' and 'Deselect All' buttons. A few points to note about the filters:
	All the filter values in the dropdown are populated (according to the log file passed) as the dashboard first initializes. The dropdown filters have an inbuilt search box as well as 'Select All' and 'Deselect All' buttons. A few points to note about the filters:

	- These filters act as global flters for both the graph and the table.
	- These filters act as global filters for both the graph and the table.

	- Any unspecified filters are assmed to be 'free' and can take any value. For example, if 'context=[0,1]', 'function=['f1']' and 'Upper Time Limit:1', then the logs with (context=0 or context=1) and (function=f1) and (timestamp<1) will be displayed.
	- Any unspecified filters are assumed to be 'free' and can take any value. For example, if 'context=[0,1]', 'function=['f1']' and 'Upper Time Limit:1', then the logs with (context=0 or context=1) and (function=f1) and (timestamp<1) will be displayed.

	This is a line graph which allows users to easily analyze logs with resepect to a time axis. A few points to note about the graph:
	This is a line graph which allows users to easily analyze logs with respect to a time axis. A few points to note about the graph:

	The table allows users to effectivel visualize the logs in a tabular way. A few points to note about the table:
	The table allows users to effectively visualize the logs in a tabular way. A few points to note about the table:

	- The search box above the table allows the users to search in sepecific columns. the searchable columns can be selected/deselected using the 'Search columns' dropdown as per need.
	- The search box above the table allows the users to search in specific columns. the searchable columns can be selected/deselected using the 'Search columns' dropdown as needed.

	- Clicking on a row in the table jumps to the page where that particular row is diplayed in the table. For example, if user searches for 'xyz' and logs are displayed based on this search query, then on clicking any row, the table will jump to the page (in the original table without the search query) which contains that particular row.
	- Clicking on a row in the table jumps to the page where that particular row is displayed in the table. For example, if user searches for 'xyz' and logs are displayed based on this search query, then on clicking any row, the table will jump to the page (in the original table without the search query) which contains that particular row.

	- All the columns of the table (except extended_context) can be sorted either in ascending or descending order.
	- All the columns of the table (except for extended_context) can be sorted either in ascending or descending order.

Gsoc2021 - Phase 3 #54

Are you sure you want to change the base?

Gsoc2021 - Phase 3 #54

Conversation

akshitpatel01 commented Aug 14, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattia-lecci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akshitpatel01 commented Aug 22, 2021