Add README-ERROR.md to eplain which kind of errors are handled by the…

… script

PulseQueryViewer.py

@@ -1,7 +1,10 @@
 #!/usr/bin/env python3
 
 """
-PulseQueryViewer: A script to parse QRadar Pulse dashboard JSON exports, displaying the widget name and AQL query data in console or converting it to CSV and Markdown.
+PulseQueryViewer
+
+A script to parse QRadar Pulse dashboard JSON exports, displaying the widget name and AQL query data in the console,
+or converting it to CSV and Markdown.
 
 Usage:
     python PulseQueryViewer.py -f input_file1.json input_file2.json ... [-c output_file.csv] [-m output_file.md]
@@ -17,19 +20,15 @@
     - A single CSV file of the parsed data from all input JSON files (if specified).
     - A single Markdown file of the parsed data from all input JSON files (if specified).
 
-   Copyright 2023 Pascal Weber (zoldax) / Abakus Sécurité
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.
+Copyright 2023 Pascal Weber (zoldax) / Abakus Sécurité
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
 """
 
 import json
@@ -59,7 +58,27 @@
 logging.basicConfig(filename=LOG_FILENAME, level=LOG_LEVEL, format=LOG_FORMAT)
 
 class PulseQueryViewer:
+    """
+    A class for the PulseQueryViewer tool.
+
+    This class handles the parsing of QRadar Pulse dashboard JSON exports, displaying the widget name and AQL query data,
+    and optionally converting the data to CSV or Markdown format.
+
+    Attributes:
+        json_files (List[str]): List of input JSON file paths.
+        csv_file (Optional[str]): Output CSV file path, if specified.
+        markdown_file (Optional[str]): Output Markdown file path, if specified.
+        results (List[Dict]): Extracted query results.
+    """
     def __init__(self, json_files: List[str], csv_file: Optional[str] = None, markdown_file: Optional[str] = None) -> None:
+        """
+        Initializes the PulseQueryViewer.
+
+        Args:
+            json_files (List[str]): List of input JSON file paths.
+            csv_file (Optional[str]): Output CSV file path, if specified.
+            markdown_file (Optional[str]): Output Markdown file path, if specified.
+        """
         self.json_files = json_files
         self.csv_file = csv_file
         self.markdown_file = markdown_file

README-ERROR.md

@@ -0,0 +1,76 @@
+# PulseQueryViewer Error Handling and Logging Summary 📜
+
+This document provides a summary of the error handling and logging capabilities of the PulseQueryViewer script.
+
+### 🖋️ Author
+- **Pascal Weber (zoldax)**
+
+## Table of Contents 📑
+
+1. [Input Validation](#input-validation)
+2. [File Handling](#file-handling)
+3. [JSON Parsing](#json-parsing)
+4. [Data Extraction](#data-extraction)
+5. [Output Writing](#output-writing)
+6. [Miscellaneous](#miscellaneous)
+
+## Input Validation 📋
+
+- **Missing Input**: 
+  - **Error**: The script checks if the user has provided the required JSON files as input. If no files are specified, an error message is displayed.
+  - **Logging**: The error along with a timestamp is logged to a designated error log file.
+
+- **File Extension Check**: 
+  - **Error**: The script ensures that the provided input files have a `.json` extension. If not, an error message is displayed.
+  - **Logging**: The error and the name of the incorrectly formatted file are logged.
+
+## File Handling 📁
+
+- **File Existence**: 
+  - **Error**: Before attempting to open a file, the script checks if the file exists. If a specified file is not found, an error message is displayed.
+  - **Logging**: The error and the name of the missing file are logged.
+
+- **File Permissions**: 
+  - **Error**: The script checks for the necessary file permissions before reading from or writing to a file. If permissions are lacking, an error message is displayed.
+  - **Logging**: The error, file name, and required permissions are logged.
+
+## JSON Parsing 📊
+
+- **Format Validation**: 
+  - **Error**: The script validates the format of the JSON data. If the data is not valid JSON, an error message is displayed.
+  - **Logging**: The error and details of the invalid format are logged.
+
+- **Data Integrity**: 
+  - **Error**: The script checks the integrity of the required data fields in the JSON file. If expected fields are missing or have incorrect data types, an error message is displayed.
+  - **Logging**: The error and details of the missing or incorrect fields are logged.
+
+## Data Extraction 📈
+
+- **Empty Data**: 
+  - **Error**: If the JSON file does not contain any queries or the required data, the script handles this gracefully and informs the user that no data was found.
+  - **Logging**: A message stating that no data was found is logged along with the file name.
+
+## Output Writing 📝
+
+- **CSV Writing**: 
+  - **Error**: The script handles errors that might occur while writing to a CSV file, such as permission issues or disk space issues, and displays an appropriate error message.
+  - **Logging**: The error, along with details such as file name and possible reasons, are logged.
+
+- **Markdown Writing**: 
+  - **Error**: Similarly, errors encountered while writing to a Markdown file are caught, and an error message is displayed.
+  - **Logging**: The error and relevant details are logged.
+
+## Miscellaneous 🛠️
+
+- **Argument Parsing**: 
+  - **Error**: The script provides error handling for command-line arguments, ensuring that the user is informed of any incorrect usage or invalid options.
+  - **Logging**: Any argument parsing errors are logged, along with suggestions for correct usage.
+
+- **Graceful Termination**: 
+  - **Error**: In any error scenario, the script aims to terminate gracefully, providing clear error messages and not leaving any partial or corrupt output files.
+  - **Logging**: A termination message is logged, indicating whether the script ended successfully or due to an error.
+
+---
+
+By handling these various error scenarios and logging pertinent information, PulseQueryViewer aims to provide a robust, user-friendly, and traceable experience.
+

test.md

@@ -2,6 +2,25 @@
 
 This document contains the queries extracted from various QRadar Pulse Dashboards. Feel free to search and navigate through the document to find the information you need. 🚀
 
+## 📁 JSON File: TBD-Pascal-Metriques.json
+The queries below are extracted from the above mentioned JSON file.
+
+### 📊 Widget Number: 1 - Managed Host Event Overview
+**Dashboard**: TBD-Pascal-Metriques  
+**Query**:  
+
+```sql
+Query not found
+```
+
+### 📊 Widget Number: 2 - Top 10 log sources by event count
+**Dashboard**: TBD-Pascal-Metriques  
+**Query**:  
+
+```sql
+SELECT starttime/(1000*60) as minute, (minute * (1000*60)) as stime, DATEFORMAT(starttime,'YYYY MM dd HH:mm:ss') as showTime, logsourcename(logSourceId) AS 'Log Source', SUM("eventCount") AS 'Event Count (Sum)', logsourceid as 'Log Source ID' from events where logsourceid in ( select logsourceid from ( select logsourceid, SUM("eventCount") AS 'Event Count (Sum)' from events where logSourceId not in (63,64,65,67,69) group by logSourceId order by "Event Count (Sum)" limit 10 last 2 hours ) ) GROUP BY minute, logSourceId order by minute ASC last 2 hours
+```
+
 ## 📁 JSON File: Abakus Sécurité - Stats Processor(4).json
 The queries below are extracted from the above mentioned JSON file.
 
@@ -141,22 +160,3 @@ select SUM("AVG_Flows per Second - Peak 1 Min") as FPS, "Flow Source" as FP, tim
 select ("AVG_Flows per Second - Peak 1 Min") as FPS, "Flow Source" as FP, time*1000 as epoch, epoch/(1000*60) as mymin, (FPS)/60 as myTOTALMB from globalview('Flow Rate (FPS)','DAILY') GROUP BY FP order by myTOTALMB desc last 2 DAYS
 ```
 
-## 📁 JSON File: TBD-Pascal-Metriques.json
-The queries below are extracted from the above mentioned JSON file.
-
-### 📊 Widget Number: 1 - Managed Host Event Overview
-**Dashboard**: TBD-Pascal-Metriques  
-**Query**:  
-
-```sql
-Query not found
-```
-
-### 📊 Widget Number: 2 - Top 10 log sources by event count
-**Dashboard**: TBD-Pascal-Metriques  
-**Query**:  
-
-```sql
-SELECT starttime/(1000*60) as minute, (minute * (1000*60)) as stime, DATEFORMAT(starttime,'YYYY MM dd HH:mm:ss') as showTime, logsourcename(logSourceId) AS 'Log Source', SUM("eventCount") AS 'Event Count (Sum)', logsourceid as 'Log Source ID' from events where logsourceid in ( select logsourceid from ( select logsourceid, SUM("eventCount") AS 'Event Count (Sum)' from events where logSourceId not in (63,64,65,67,69) group by logSourceId order by "Event Count (Sum)" limit 10 last 2 hours ) ) GROUP BY minute, logSourceId order by minute ASC last 2 hours
-```
-