Merge branch 'master' into chaplin-auto-run

contrib/README.md

@@ -0,0 +1,36 @@
+# Contrib
+
+This folder contains various scripts, templates and pieces contributed from the community.
+
+## Disclaimer
+
+None of the content of this folder is maintained by the Timesketch-dev team.
+Using it is at your own risk.
+
+# Content
+
+## deploy_timesketch.ps1
+
+This PowerShell script automates the deployment of Timesketch on a system. It checks for existing Timesketch installations, Docker service, and Timesketch containers to prevent conflicts. The script sets up necessary directories, configures parameters, and downloads the required configuration files. It then modifies these files to establish connections with OpenSearch, Redis, and Postgres. Finally, it provides instructions on how to start Timesketch and create a user.
+
+## deploy_timesketch.sh
+
+This Bash script automates the deployment of Timesketch, a digital forensic timeline analysis tool. It checks for prerequisites like root access, Docker, and Docker Compose, then sets up necessary directories and fetches configuration files. The script configures Timesketch parameters, including connections to a PostgreSQL database, OpenSearch instance, and Redis. It can also start the Timesketch containers and guide the user to create a new Timesketch user.
+
+## gcs_importer.py
+
+This Python script facilitates the import of forensic timeline data from Google Cloud Storage (GCS) into Timesketch. It listens for messages on a Google Cloud Pub/Sub topic, automatically downloading and indexing Plaso files from GCS when Turbinia processes are completed. The script creates or uses existing Timesketch sketches and timelines to organize the imported data, enhancing the automation of forensic analysis workflows.
+
+## timesketch-importer.sh
+
+This Bash script automates importing forensic timeline data into Timesketch. It monitors a specified directory for new files with extensions `.plaso, .csv, or .jsonl`, commonly used for storing timeline data. When a new file is detected, it automatically imports the data into Timesketch using the `tsctl` command. This script simplifies the process of adding new data to Timesketch for analysis.
+
+## nginx.conf
+
+This Nginx configuration file sets up a reverse proxy for Timesketch, routing incoming HTTP requests to the appropriate backend servers. It defines two server blocks: one for the main Timesketch application (/) and another for the legacy interface (/legacy/).  The configuration includes settings for client maximum body size, proxy buffering, and request timeouts to optimize performance.  Additionally, it sets necessary headers to ensure proper communication between the proxy and the backend servers.    
+
+## timesketch-importer.conf / timesketch-importer.service
+
+This configuration file defines the settings for the Timesketch Importer script. It specifies the directory that the script should monitor for new Plaso, CSV, or JSONL files to import into Timesketch.
+
+This systemd service file configures the Timesketch Importer script to run as a service. It defines the service description, start command, and restart behavior.  This allows the importer to run automatically in the background and restart if it fails, ensuring continuous monitoring and importing of forensic timeline data.

docs/changelog/2023-03.md

@@ -6,7 +6,7 @@ hide:
 
 ## Add "events add" to timesketch-cli-client
 
-The new Timesketch CLI client feature allows users to add events to their Timesketch sketch using the `events add` command. The command takes three arguments: `message, date, timestamp-desc` and an optional argument for `attributes`. `message` is a short description of the event, `date` is the date and time the event occured (ISO formatted), and the attributes are a comma-separated list of key-value pairs that will be associated with the event.
+The new Timesketch CLI client feature allows users to add events to their Timesketch sketch using the `events add` command. The command takes three arguments: `message, date, timestamp-desc` and an optional argument for `attributes`. `message` is a short description of the event, `date` is the date and time the event occurred (ISO formatted), and the attributes are a comma-separated list of key-value pairs that will be associated with the event.
 
 For example, the following command would add an event to the Timesketch 
 

docs/developers/analyzer-development.md

@@ -13,7 +13,7 @@ analyzer.
 
 Timesketch analyzers are programs that run when new data is indexed, e.g. when
 you upload a new plaso storage file or when adding an existing index to a
-sketch. They can also be triggered manaully for a specific timeline from the
+sketch. They can also be triggered manually for a specific timeline from the
 Analyzer tab in the UI after the index is finished.
 You have access to a simple API that makes searching, commenting, tagging etc
 easy. Everything you can do in the UI you can do programmatically.

docs/developers/frontend-development.md

@@ -62,7 +62,7 @@ This will spawn a listener on port `5001`. Point your browser to `http://localho
 dev credentials, and you should be redirected to the main Timesketch page. All code changes in `.vue` files will
 be instantly picked up.
 
-## Frontend-ng developement
+## Frontend-ng development
 
 When developing the `frontend-ng` you use the VueJS frontend server. Changes will be picked up automatically
 as soon as a `.vue` file is saved without having to rebuild the frontend or even refresh your browser.

docs/guides/admin/install.md

@@ -155,3 +155,9 @@ Congratulations, your Timesketch system is operational and ready to use.
 ### Set up users
 
 After system is set up, look at [here](/guides/admin/admin-cli/) to add users.
+
+# Timesketch K8s Installation
+
+Please refer to
+[OSDFIR Infrastructure](https://github.com/google/osdfir-infrastructure)
+repository for official installation instructions.

docs/guides/admin/scaling-and-limits.md

@@ -19,7 +19,7 @@ These are the limitations:
 ## OpenSearch indices limitation
 
 In the past, every timeline in a sketch was a dedicated OpenSearch Index. In larger installations, Timesketch hit the number of maximum open shards OpenSearch could handle.
-Therefor a design [https://github.com/google/timesketch/issues/1567](change) was made to tackle those limitations
+Therefore a design [https://github.com/google/timesketch/issues/1567](change) was made to tackle those limitations
 
 - There are maximum number (1500) of shards that can be opened.
 - There are limitations with Lucene (which OpenSearch uses) and then OpenSearch itself, see https://www.elastic.co/guide/en/app-search/current/limits.html and maximum sizes of HTTP requests, hence when Timesketch uploads files they are split up, to avoid HTTP limitations.

docs/guides/admin/troubleshooting.md

@@ -10,7 +10,7 @@ Please provide as much detailed information as possible, keep in mind that:
 * we cannot look into your thoughts or on your systems;
 * we cannot easily isolate errors if you keep changing your test environment.
 
-If you report a bug or ask for help in Slack, please provide as much information as possible, error messsages, log lines, stacktraces, screenshots, videos etc.
+If you report a bug or ask for help in Slack, please provide as much information as possible, error messages, log lines, stacktraces, screenshots, videos etc.
 
 ## Troubleshooting playbook
 

docs/guides/analyzers/feature_extraction.md

@@ -135,7 +135,7 @@ name:
     mapping:                Type: list[dict] | REQUIRED
                             A list of dicts that define the new attribute name
                             and the string index of the event to extract the
-                            value from. Additonally it can also contain an
+                            value from. Additionally it can also contain an
                             alias list to add multiple attributes with
                             the same value but different names.
 
@@ -149,7 +149,7 @@ name:
 
         aliases:            Type: list[str] | OPTIONAL
                             A list of aliases to add additionally to the
-                            offical name of the attribute. This can be used
+                            official name of the attribute. This can be used
                             to add different field names matching individual
                             field name ontologies. E.g. srcIP, domain, etc.
 ```

docs/guides/analyzers/sigma_analyzer.md

@@ -17,7 +17,7 @@ Reasons might be because:
 ## Which rules should be deployed
 
 It is not recommended to deploy all rules from https://github.com/SigmaHQ/sigma as it is impossible for the Timesketch project to ensure that all rules produce valid OpenSearch Queries.
-Instead pick the rules you verified the format of your logs allign and you expect hits.
+Instead pick the rules you verified the format of your logs align and you expect hits.
 
 ## Troubleshooting
 

docs/guides/user/sigma.md

@@ -146,7 +146,7 @@ Field mappings like:
       default: xml_string
 ```
 
-Are interpreted depending on the selected product in the rule. If the product in the rule is `linux` the Selector `TargetFilename` in a rule would be tranlated to `filename:"foobar"`. If the product is anything else, e.g. `Windows` it would be `xml_string:"foobar"`
+Are interpreted depending on the selected product in the rule. If the product in the rule is `linux` the Selector `TargetFilename` in a rule would be translated to `filename:"foobar"`. If the product is anything else, e.g. `Windows` it would be `xml_string:"foobar"`
 
 ## Test data
 

end_to_end_tests/upload_test.py

@@ -230,5 +230,109 @@ def test_large_upload_csv_over_flush_limit(self):
         events = sketch.explore("data_type:73kcsv", as_pandas=True, max_entries=100000)
         self.assertions.assertEqual(len(events), 73251)
 
+    def test_datetime_out_of_normal_range_in_csv(self):
+        """Test uploading a file with events from way back and some
+        in a distant future. This test can reveal edge cases that might occur
+        when tools produce a "fake" datetime value"""
+
+        rand = str(random.randint(0, 10000))
+        sketch = self.api.create_sketch(
+            name=f"datetime_out_of_normal_range_in_csv_{rand}"
+        )
+        self.sketch = sketch
+        file_path = "/usr/local/src/timesketch/tests/test_events/validate_time_out_of_range.csv"  # pylint: disable=line-too-long
+        self.import_timeline(file_path, index_name=rand, sketch=sketch)
+        timeline = sketch.list_timelines()[0]
+        # check that timeline was uploaded correctly
+        self.assertions.assertEqual(timeline.name, file_path)
+        self.assertions.assertEqual(timeline.index.name, str(rand))
+        self.assertions.assertEqual(timeline.index.status, "ready")
+
+        # Search for the very old event
+        search_obj = search.Search(sketch)
+        search_obj.query_string = "data_type:csv_very_old_event"
+        search_obj.commit()
+        self.assertions.assertEqual(len(search_obj.table), 1)
+        self.assertions.assertEqual(
+            "1601-01-01" in str(search_obj.table["datetime"]), True
+        )
+
+        # Search for future event check if datetime value is in the result
+        search_obj2 = search.Search(sketch)
+        search_obj2.query_string = "data_type:csv_very_future_event"
+        search_obj2.commit()
+        self.assertions.assertEqual(len(search_obj2.table), 1)
+        self.assertions.assertEqual(
+            "2227-12-31" in str(search_obj2.table["datetime"]), True
+        )
+
+    def test_csv_different_timestamps(self):
+        """Test uploading a timeline with different precision of timestamps."""
+
+        # create a new sketch
+        rand = str(random.randint(0, 10000))
+        sketch = self.api.create_sketch(name=f"csv_different_timestamps_{rand}")
+        self.sketch = sketch
+
+        file_path = "/tmp/timestamptest.csv"
+
+        with open(file_path, "w", encoding="utf-8") as file_object:
+            file_object.write(
+                '"message","timestamp","datetime","timestamp_desc","data_type"\n'
+            )
+            string = (
+                '"total precision","123456789",'
+                '"2024-07-24T10:57:02.877297Z","Write time","timestamptest"\n'
+            )
+            file_object.write(string)
+            string = (
+                '"ISO8601","1331698658276340",'
+                '"2015-07-24T19:01:01+00:00","Write time","timestamptest"\n'
+            )
+            file_object.write(string)
+            string = (
+                '"Wrong epoch","123456",'
+                '"2015-07-24 19:01:01","Write time","timestamptest fail"\n'
+            )
+            file_object.write(string)
+            string = '"no_datetime","123456","","Write time","no_datetime"\n'
+            file_object.write(string)
+            string = (
+                '"Notimestamp","",'
+                '"2015-07-24 19:01:01","Write time","no_timestamp"\n'
+            )
+            file_object.write(string)
+            string = (
+                '"Accurate_timestamp","1331712840499027",'
+                '"2015-07-24 19:01:01","Write time","Accurate_timestamp"\n'
+            )
+            file_object.write(string)
+
+        self.import_timeline("/tmp/timestamptest.csv", index_name=rand, sketch=sketch)
+        os.remove(file_path)
+
+        timeline = sketch.list_timelines()[0]
+        # check that timeline was uploaded correctly
+        self.assertions.assertEqual(timeline.name, file_path)
+        self.assertions.assertEqual(timeline.index.name, str(rand))
+        self.assertions.assertEqual(timeline.index.status, "ready")
+
+        search_obj = search.Search(sketch)
+        search_obj.query_string = "data_type:timestamptest"
+        search_obj.commit()
+        self.assertions.assertEqual(len(search_obj.table), 3)
+
+        # check that the number of events is correct with a different method
+        events = sketch.explore("data_type:timestamptest", as_pandas=True)
+        self.assertions.assertEqual(len(events), 3)
+
+        # check that events with no timestamp
+        events = sketch.explore("data_type:no_timestamp", as_pandas=True)
+        self.assertions.assertEqual(len(events), 1)
+
+        # check number of events with no datetime
+        events = sketch.explore("data_type:no_datetime", as_pandas=True)
+        self.assertions.assertEqual(len(events), 1)
+
 
 manager.EndToEndTestManager.register_test(UploadTest)

importer_client/python/timesketch_import_client/data/formatter.yaml

@@ -47,7 +47,7 @@
 #
 #       columns - a list of columns that should be present in the data file
 #                 for this to be a match on. It should be noted that all the
-#                 columns need to be present and no extra columnns should be
+#                 columns need to be present and no extra columns should be
 #                 in the log file. If there may be extra columns then use
 #                 columns_subset instead.
 #

notebooks/README.md

@@ -0,0 +1,4 @@
+# Notebooks
+
+This folder contains various Jupyter notebooks to show how interaction with the
+Timesketch API works.