Version upgrade of influxDB for 1.x to 2.x #27

NexFlare · 2023-06-27T18:29:19Z

Updates

New InfluxClient added to support 2.x
INIT env variables added to docker compose file to intial setup of influx db
Telegraf version upgrade
Env variables added in circleCI config
output.conf updated with outputs.influxdb_v2

Updates - New InfluxClient added to support 2.x - INIT env variables added to docker compose file to intial setup of influx db - Telegraf version upgrade - Env variables added in circleCI config - output.conf updated with outputs.influxdb_v2

sivanov-nuodb · 2023-06-30T11:13:54Z

.circleci/config.yml

@@ -10,11 +10,11 @@ commands:
      - run:
          name : Take a nap
          command: |
-            sleep 15
+            sleep 120


Why is this sleep for? Generally, it's better to wait for the condition to become true instead of adding random delays that are unlikely to work in all situations.

conf/nuodb.conf

docker-compose.yml

test_requirements.txt

sivanov-nuodb · 2023-06-30T12:15:14Z

tests/nuodb_test_class.py

@@ -34,15 +35,36 @@ def assert_await(fxn, timeout=120, interval=10):

    if last_assert:
        raise last_assert
+
+def getClient():
+     influxToken = os.getenv("INFLUXDB_TOKEN", 'none')


The test should fail fast if this is not set as there is no way to authenticate.

Should I provide token as the default value ?

If INFLUXDB_TOKEN environment variable is not found, we should just fail the test.

The test will fail automatically with the unauthenticated error message.

sivanov-nuodb · 2023-06-30T12:20:51Z

tests/stats_test.py

-from nuodb_test_class import *
+from .nuodb_test_class import *


Why this is needed?

tests/stats_test.py

docker-compose.yml

adriansuarez · 2023-07-01T02:12:19Z

Dockerfile

-    echo "deb https://repos.influxdata.com/debian stretch stable" > /etc/apt/sources.list.d/influxdb.list && \
+    # influxdata-archive_compat.key GPG Fingerprint: 9D539D90D3328DC7D6C8D3B9D8FF8E1F7DF8B07E
+    curl -s https://repos.influxdata.com/influxdata-archive_compat.key > influxdata-archive_compat.key && \
+    echo '393e8779c89ac8d958f81f942f9ad7fb82a25e133faddaf92e15b16e6ac9ce4c influxdata-archive_compat.key' | sha256sum -c && cat influxdata-archive_compat.key | gpg --dearmor | tee /etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg > /dev/null && \


Where is this 393e8779c89ac8d958f81f942f9ad7fb82a25e133faddaf92e15b16e6ac9ce4c value coming from and why do we have to hardcode it?

https://docs.influxdata.com/telegraf/v1.21/introduction/installation/#installation Provides these values

adriansuarez · 2023-07-01T02:15:34Z

Dockerfile

+    # influxdata-archive_compat.key GPG Fingerprint: 9D539D90D3328DC7D6C8D3B9D8FF8E1F7DF8B07E
+    curl -s https://repos.influxdata.com/influxdata-archive_compat.key > influxdata-archive_compat.key && \
+    echo '393e8779c89ac8d958f81f942f9ad7fb82a25e133faddaf92e15b16e6ac9ce4c influxdata-archive_compat.key' | sha256sum -c && cat influxdata-archive_compat.key | gpg --dearmor | tee /etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg > /dev/null && \
+    echo 'deb [signed-by=/etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg] https://repos.influxdata.com/debian stable main' | tee /etc/apt/sources.list.d/influxdata.list && \


Do we need to see the output of this command? Otherwise, just redirect it to the file without using tee.

adriansuarez · 2023-07-01T02:20:48Z

conf/outputs.conf

-# [[outputs.file]]
-#   ## Files to write to, "stdout" is a specially handled file.
-#   files = ["stdout"]
+  ## Destination bucket to write into.
+  bucket = "${INFLUXDB_BUCKET}"

-#   ## Data format to output.
-#   ## Each data format has its own unique set of configuration options, read
-#   ## more about them here:
-#   ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
-#   data_format = "json"
+  ## The value of this tag will be used to determine the bucket.  If this
+  ## tag is not set the 'bucket' option is used as the default.
+  bucket_tag = "db_tag"

-# [[outputs.prometheus_client]]
-#   ## Address to listen on.
-#   listen = ":9273"


I think the commented out plugins were here to serve as an example, and we should keep it (unless they don't work anymore).

adriansuarez · 2023-07-01T02:21:25Z

docker-compose.yml

@@ -30,11 +30,21 @@ services:
    command: ["nuodocker", "start", "te", "--db-name", "hockey", "--server-id", "nuoadmin1",
          "--servers-ready-timeout", "60"]
  influxdb:
-    image: influxdb:1.8
+    image: influxdb:2.7.1


Can we drop the patch version and just use 2.7? The tag <major>.<minor> tag will be set to the latest patch release within that release stream.

adriansuarez · 2023-07-01T02:25:25Z

scripts/setup_influx.sh

@@ -0,0 +1,6 @@
+#! /bin/bash


Can you change this to

#!/bin/sh

It doesn't look like we have use any bash-specific syntax, so might as well avoid the dependency.

adriansuarez · 2023-07-01T02:28:30Z

tests/nuodb_test_class.py

+    query = f"""
+    import \"influxdata/influxdb/schema\"
+
+    schema.measurements(bucket: \"{bucket}\")
+    """


Is this a Python3-ism? I'm guessing the f prefix is what causes bucket to be injected into the format string without have to write .format(bucket=bucket).

Yes. F strings are sort of improved way of string formatting. It was introduced in Python 3.6

adriansuarez · 2023-07-01T02:31:27Z

tests/nuodb_test_class.py

+
+def getMeasurenment(client, bucket):
+    query = f"""
+    import \"influxdata/influxdb/schema\"


Are the quotes here and and below escaped for InfluxDB or for Python? When I test this, it looks like the escape character is actually gratuitous, since the multi-line string is delimited by """ and " is just a normal character:

>>> print(""" ... "test" ... """) "test" >>> print(""" ... \"test\" ... """) "test"

sivanov-nuodb · 2023-07-06T08:38:58Z

README.md

+DOCKER_INFLUXDB_INIT_BUCKET=[your_value]
+DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=[your_value]
+```
+


We need to also document the environment variables expected by nuodb-collector docker container in section Running NuoDB Collector.

INFLUXDB_TOKEN=... INFLUXDB_BUCKET=... INFLUXDB_ORG=...

README.md

docker-compose.yml

adriansuarez · 2023-07-07T19:49:49Z

.circleci/config.yml

@@ -14,7 +14,7 @@ commands:
      - run:
          name : Configure and run tests
          command: |
-            pip install -r test_requirements.txt
+            pip3 install -r test_requirements.txt
            pytest --junitxml=test_results/result.xml
            echo "Done"


Can you try setting the environment variable PYTHONUNBUFFERED=1 and reverting the change to start up YCSB?

See https://docs.python.org/3/using/cmdline.html#envvar-PYTHONUNBUFFERED for more info.

sivanov-nuodb

Looks good to me.
Thanks for processing this change!

NexFlare requested review from adriansuarez and sivanov-nuodb June 27, 2023 18:29

Version upgrade of influxDB for 1.x to 2.x

7844003

Updates - New InfluxClient added to support 2.x - INIT env variables added to docker compose file to intial setup of influx db - Telegraf version upgrade - Env variables added in circleCI config - output.conf updated with outputs.influxdb_v2

NexFlare force-pushed the hal/influx branch from fce9ee5 to 7844003 Compare June 27, 2023 19:08

AGARWAL Harsh Mohan (INTERN) added 2 commits June 28, 2023 10:19

Telegraf version bump

fc1f840

Log time change

2ff5b94

sivanov-nuodb reviewed Jun 30, 2023

View reviewed changes

docker-compose.yml Show resolved Hide resolved

adriansuarez reviewed Jul 1, 2023

View reviewed changes

AGARWAL Harsh Mohan (INTERN) added 5 commits July 3, 2023 11:03

YCSB added in docker compose

9aabf96

Code changes according to pr comments

e7f7080

Readme file updated

6a60022

Import 🐛 fix

5fee108

Readme crlf to lf change

63afc60

sivanov-nuodb reviewed Jul 6, 2023

View reviewed changes

README.md Show resolved Hide resolved

sivanov-nuodb reviewed Jul 6, 2023

View reviewed changes

docker-compose.yml Outdated Show resolved Hide resolved

AGARWAL Harsh Mohan (INTERN) added 4 commits July 6, 2023 13:04

Readme file updated, env varibales added in docker_compose script

1d2181d

Docker compose background added

96befaf

Unauthorized 🐛 fix

718f850

Unauthorized issue fix

f6d8e94

NexFlare requested a review from adriansuarez July 6, 2023 17:34

Removed tee command from docker file

c702289

adriansuarez reviewed Jul 7, 2023

View reviewed changes

AGARWAL Harsh Mohan (INTERN) added 2 commits July 11, 2023 09:25

YCSB image removed, pythonUnBuffered flag added

2618767

PythonUnBuffered flag added in docker-compose file

1aa73cf

sivanov-nuodb approved these changes Jul 18, 2023

View reviewed changes

NexFlare merged commit 0fcc9a2 into master Aug 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Version upgrade of influxDB for 1.x to 2.x #27

Version upgrade of influxDB for 1.x to 2.x #27

NexFlare commented Jun 27, 2023

sivanov-nuodb Jun 30, 2023

adriansuarez Jul 1, 2023

sivanov-nuodb Jun 30, 2023

NexFlare Jul 5, 2023

sivanov-nuodb Jul 5, 2023

NexFlare Jul 6, 2023

sivanov-nuodb Jun 30, 2023

adriansuarez Jul 1, 2023 •

edited

Loading

NexFlare Jul 5, 2023

adriansuarez Jul 1, 2023 •

edited

Loading

adriansuarez Jul 1, 2023

adriansuarez Jul 1, 2023 •

edited

Loading

adriansuarez Jul 1, 2023

adriansuarez Jul 1, 2023

NexFlare Jul 5, 2023

adriansuarez Jul 1, 2023 •

edited

Loading

sivanov-nuodb Jul 6, 2023 •

edited

Loading

adriansuarez Jul 7, 2023

sivanov-nuodb left a comment

		from nuodb_test_class import *
		from .nuodb_test_class import *

Version upgrade of influxDB for 1.x to 2.x #27

Version upgrade of influxDB for 1.x to 2.x #27

Conversation

NexFlare commented Jun 27, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adriansuarez Jul 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adriansuarez Jul 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adriansuarez Jul 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adriansuarez Jul 1, 2023 • edited Loading

Choose a reason for hiding this comment

sivanov-nuodb Jul 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sivanov-nuodb left a comment

Choose a reason for hiding this comment

adriansuarez Jul 1, 2023 •

edited

Loading

adriansuarez Jul 1, 2023 •

edited

Loading

adriansuarez Jul 1, 2023 •

edited

Loading

adriansuarez Jul 1, 2023 •

edited

Loading

sivanov-nuodb Jul 6, 2023 •

edited

Loading