Write the final setup part and tweak source code

BME680Setup.md

@@ -7,7 +7,7 @@
     - This will create a folder `bsec_bme680_linux`, with the two most important files there being `bsec_bme680.c` which is the code that reads from the sensor via the BSEC library, and `make.sh` which compiles this code to create the executable.
     - `cd` into the `bsec_bme680_linux` folder for the following steps.
 
-2. Follow the setup instructions on (https://github.com/alexh-name/bsec_bme680_linux).
+2. Follow the setup instructions on here: https://github.com/alexh-name/bsec_bme680_linux. Some notes:
     - To copy Bosh's BSEC library's Zip to the Zero, if you're using an headless install, use something like `scp bsec_1-4-7-4_generic_release.zip pi@192.168.1.123:/home/pi/` (replacing the IP address of your Zero). `scp` is pre-installed on Windows 10 now, like `ssh`, and stands for "Securely Copy Files".
     - Remember to create a folder `src` inside `bsec_bme680_linux/` on the Zero
     - Also remember to `chmod +x make.sh` for it to be executable.

DeviceUploadData.md

@@ -1,10 +1,51 @@
 # Get sensor readings in CSV format and upload to Azure
 
+## Change C code that does the sensor readings to write CSV files
 
+The original code from https://github.com/alexh-name/bsec_bme680_linux prints out sensor the readings to stdout (screen). Replace the file from that repo with my version [here](https://github.com/lokijota/Azure-RPi-BME680/blob/master/bsec_bme680.c). The changes I made were:
 
+- modified the `output_ready()` function to generate files named YYYYNNDD-HHMMSS.csv in a `data` subfolder. Remeber to create this folder;
+- at the top a `temp_offset` constant is defined with value `5.0f`. This value is being added to the temperature measurements (which are in Celcius), to compensate for the temperature of the Raspberry/Power supply/etc. I changed this value to `4.0f` as I believe I have better isolation -- do edit to your own situation;
+- also at the head of the file there's an `int` variable valled `i2c_address` declared, with default value `BME680_I2C_ADDR_PRIMARY` (corresponding to I2C address 76). In my case I have address 77, so needed to change this to `BME680_I2C_ADDR_SECONDARY`;
+- changed call `bsec_iot_loop` near the end to checkpoint the internal state of the sensor every 4 hours (i.e., changed the last parameter from `10000` to `4800`).
 
+After this, compile the file again by calling `./make.sh` as before. Remember to create a `data` folder as a subfolder from where you'll be running the compiled `bsec_bme680`, or you'll a friendly `Segmentation fault` when you run it.
 
-Also -- for another page, not here:
-- change the checkpointing interval
-- every 3 seconds
-- change 5ºC to 4ºC ??
+Now type `./bsec_bme680 &` to run the application in the background. If you change into the data folder, you'll start seeing the csv files being generated every 3 seconds.
+
+## Upload the CSV files to Azure
+
+I decided to separate the capture of the sensor readings from the uploading to Azure for a couple of reasons:
+- If there's any networking issue, you won't run the risk of losing readings. Additionally, while the time to push a reading to the cloud is non-deterministic, writing to a file is much more so, so you can keep a predictable reading every 3 seconds;
+- It's much simpler to install/use the IoT Hub Client SDK for Python than it is to figure out how the CMake-based compilation process of the IoT Hub Client SDK for C works. Hence, I'll be using Python for the upload.
+
+The steps to follow are:
+
+1. Create a file on the Pi called `scoop_up_data.py` with the contents you find [here](https://github.com/lokijota/Azure-RPi-BME680/blob/master/scoop_up_data.py). This code uses the  version 1.4.4 of the Azure IotHub Device Client SDK (which you installed [previously](BME680Setup.md)).
+2. Edit the file to change the value of the `iothub_connstring` variable. This is a string looking like `"HostName=NAME_OF_YOUR_IOTHUB.azure-devices.net;DeviceId=NAME_OF_YOUR_DEVICE_IN_THE_IOTHUB;SharedAccessKey=LOTS_OF_ALFANUM_CHARACTERS"` which you can get from the Azure portal.
+3. To do test run, call `python3 scoop_up_data.py ./data/`. This will upload all your already captured CSV files to the Azure IoTHub, in cronological order, and print out something like:
+
+```
+pi@rpi0:~/bsec_bme680_linux $ python3 scoop_up_data.py ./data/
+Starting iothub client
+Reading files from /home/pi/bsec_bme680_linux/data:
+1 - /home/pi/bsec_bme680_linux/data/20200131-233324.csv
+2 - /home/pi/bsec_bme680_linux/data/20200131-233327.csv
+...
+378 - /home/pi/bsec_bme680_linux/data/20200131-235215.csv
+Files uploaded: 378
+
+```
+
+4. The filed in the `data` folder are renamed, with an `uploaded_` prefix added. E.g., `20200131-235215.csv` becomes `uploaded_20200131-235215.csv`. You'll need to clear up these files later.
+5. Now that this process has been tested, you need to run the previous command on a schedule with a cron job. Do do this, run `crontab -e`, pick an editor (nano may be the simplest one), and add the following at the end:
+
+`* * * * * /usr/bin/python3 /home/pi/bsec_bme680_linux/scoop_up_data.py /home/pi/bsec_bme680_linux/data/`
+
+This will execute the command every minute, and upload the readings (typically 20 at a time, considering they are recorded every 3 seconds).
+
+## TBD
+
+**TBD** - how to set up your Azure IotHub -- create it and an IoT Device. Add a new instructions step. Maybe link to this: https://docs.microsoft.com/en-gb/learn/modules/remotely-monitor-devices-with-azure-iot-hub/2-create-iot-hub-device-id?pivots=csharp
+
+**TBD** avoid race condition as per here: https://www.cyberciti.biz/faq/how-to-run-cron-job-every-minute-on-linuxunix/

README.md

@@ -2,7 +2,7 @@
 
 This is a sample of using the Bosch BME680 sensor connected to a Raspberry Pi Zero and uploading/processing data in Azure. I'm using Adafruit's breakout (https://www.adafruit.com/product/3660) and using it to gather information including temperature, humidity and air quality.
 
-*Note: This project is Work in Progress as of 23/01/2020. I'm actively working on it and will be adding daily.*
+*Note: This project is Work in Progress as of 31/01/2020. I'm actively working on it and will be adding daily.*
 
 **To-add: Add photo of it with the connections**
 
@@ -29,12 +29,12 @@ Bosch has a library (compiled C) to handle their BME680 sensor, including propri
 
 ## Base setup instructions (if you're starting from scratch)
 
-Start by following these instructions to set up the Zero: [Basic Device Setup](DeviceSetup.md) . If you already have one running, just remember to enable I2C. After this, you'll need to install some aditional libraries in the device, related either to the BME680 sensor or to Azure. To do this, follow the steps here: [Libraries setup](BME680Setup.md) . When you finish these two, you'll have a C application installed that prints out the sensor's readings, and all the required dependencies for what follows.
+Start by following these instructions to set up the Zero: [Basic Device Setup](DeviceSetup.md) . If you already have one running, just remember to enable I2C. After this, you'll need to install some additional libraries in the device, related either to the BME680 sensor or to Azure. To do this, follow the steps here: [Libraries setup](BME680Setup.md) . When you finish these two, you'll have a C application compiled that prints out the sensor's readings to the screen, and all the required dependencies for what follows.
 
 ## Get formatted sensor readings and push them to Azure
 
-After the setup is done, you'll need to make make changes to make sure that:
-- the readings are captured in a format simple to process in Azure (I picked CSV, JSON would be another good option)
+After the setup is done, you'll need to make make changes so that:
+- the readings are captured in a format simple to process in Azure (I picked CSV, but JSON would be another good option)
 - have code to send the readings to Azure IoT Hub.
 
 To do this, follow [these steps](DeviceUploadData.md)

bsec_bme680.c

@@ -1,5 +1,6 @@
 /* Copyright (C) 2017 alexh.name */
 /* I2C code by twartzek 2017 */
+/* Minor edits by Lokijota to write out CSV to a file instead of printing readings to the screen */
 
 /*
  * Read the BME680 sensor with the BSEC library by running an endless loop in
@@ -28,13 +29,12 @@
 /* definitions */
 
 #define DESTZONE "TZ=Europe/Berlin"
-#define temp_offset (5.0f)
-#define sample_rate_mode (BSEC_SAMPLE_RATE_LP)
+#define temp_offset (4.0f)
+#define sample_rate_mode (BSEC_SAMPLE_RATE_LP) // Sample rate in case of Low Power Mode = 0.33333f
 
 int g_i2cFid; // I2C Linux device handle
 
-int i2#_address = BME600_I2C_ADDR_SECONDARY; // was BME680_I2C_ADDR_PRIMARY - 76,secundary is 77
-int i2c_address = BME680_I2C_ADDR_PRIMARY;
+int i2c_address = BME680_I2C_ADDR_SECONDARY; // was BME680_I2C_ADDR_PRIMARY - 76,secundary is 77
 char *filename_state = "bsec_iaq.state";
 char *filename_config = "bsec_iaq.config";
 
@@ -358,7 +358,7 @@ int main()
    * sensor settings.
    * State is saved every 10.000 samples, which means every 10.000 * 3 secs
    * = 500 minutes (depending on the config).
-   * NOTAJOTA: CHANGED TO 4 hours = 4800
+   * Lokijota: Changed TO 4 hours = 4800
    *
    */
   bsec_iot_loop(_sleep, get_timestamp_us, output_ready, state_save, 4800);

scoop_up_data.py

@@ -19,7 +19,7 @@
 
 #get a list of files from the folder, sorted alfabetically
 listOfFiles = sorted(os.listdir(res.dataFolder))
-pattern = "*.csv"
+pattern = "2*.csv" # must start with a '2' of year 2000, otherwise uploaded_ files are re-sent
 count = 0
 
 for entry in listOfFiles: