Pandas outputs warning when calling dataframe.append in flux_csv_parser._prepare_data_frame

[jonashoechst]

Specifications

• Client Version: 1.36.0
• InfluxDB Version: 2.7

Code sample to reproduce problem

query_api.query_data_frame(query)

Expected behavior

The query should be executed without warnings.

Actual behavior

Pandas outputs a FutureWarning:

/usr/local/lib/python3.11/site-packages/influxdb_client/client/flux_csv_parser.py:256: FutureWarning:
The behavior of array concatenation with empty entries is deprecated. In a future version, this will no longer exclude empty items when determining the result dtype. To retain the old behavior, exclude the empty entries before the concat operation.

This is related to the this code snippet:

https://github.com/influxdata/influxdb-client-python/blob/aa6e5c4aad1aaf004195c175ebe424ac6040c2e3/influxdb_client/client/flux_csv_parser.py#L243C9-L256

Additional info

In the recent pandas version handling of concatenation of empty dataframes changed, see pandas-dev/pandas#39122. This should be handled else wise to reflect the changes in pandas.

[Sciator]

I wasn't able to replicate this issue. I have tried to run this with 2.0.3 and 2.1.1 pandas with no warning.
Could you provide more information (query, data etc.) ?

[bednar]

@Sciator, this warning is a part of the Pandas 2.1. For more info see: - pandas-dev/pandas#52532 - pandas-dev/pandas#39122 - https://github.com/pandas-dev/pandas/blob/0cbe41ad0bab42fa6fc8694adcf6db3804447740/pandas/core/dtypes/concat.py#L115

…

On 26. 10. 2023, at 13:35, Sciator ***@***.***> wrote: I wasn't able to replicate this issue. I have tried to run this with 2.0.3 and 2.1.1 pandas with no warning. Could you provide more information (query, data etc.) ? — Reply to this email directly, view it on GitHub <#613 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADPDYJENWP2Z4PVRRNUCLTYBJDIPAVCNFSM6AAAAAA46NILZKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBQHE2DAMJTHE>. You are receiving this because you are subscribed to this thread.

[ivankudibal]

pls, it would be great if we have data and the query to reproduce, before we can go on with this issue

[oskarwilliams]

I am seeing this issue as well, do you still need a query to repro this?

[bednar]

Hi @oskarwilliams, the query and possibly an example of your data in the database would be helpful… Thanks.

[dave-cz]

Python 3.12
Pandas 2.2.3
InfluxDB 2.7.5
influxdb_client 1.47.0

Query results from InfluxDB Data Explorer
query(1).csv

Code:

query_api.query_data_frame(query, data_frame_index=['time'])

query:

from(bucket: "model")
|> range(start: 2024-10-27T23:00:00Z, stop: 2024-11-03T22:00:00.000001Z)
|> filter(fn: (r) => r["_measurement"] == "sp_el.J612")
|> aggregateWindow(every: 1h, fn: last, timeSrc: "_start", timeDst: "time", createEmpty: true)
|> pivot(rowKey: ["time"], columnKey: ["_measurement", "_field"], valueColumn: "_value")`

flux_csv_parser.py l.258
df = pd.concat([self._data_frame.astype(_temp_df.dtypes), _temp_df])

>> self._data_frame

Empty DataFrame
Columns: [result, table, _start, _stop, sp_el.J612_value]
Index: []

>> _temp_df
                            result  table                    _start                            _stop  sp_el.J612_value
time                                                                                                                  
2024-10-27 23:00:00+00:00  _result      0 2024-10-27 23:00:00+00:00 2024-11-03 22:00:00.000001+00:00               NaN
2024-10-28 00:00:00+00:00  _result      0 2024-10-27 23:00:00+00:00 2024-11-03 22:00:00.000001+00:00               NaN
...
2024-10-29 22:00:00+00:00  _result      0 2024-10-27 23:00:00+00:00 2024-11-03 22:00:00.000001+00:00               NaN
2024-10-29 23:00:00+00:00  _result      0 2024-10-27 23:00:00+00:00 2024-11-03 22:00:00.000001+00:00               2.0
2024-10-30 00:00:00+00:00  _result      0 2024-10-27 23:00:00+00:00 2024-11-03 22:00:00.000001+00:00               2.0
...
2024-10-31 22:00:00+00:00  _result      0 2024-10-27 23:00:00+00:00 2024-11-03 22:00:00.000001+00:00               2.0
2024-10-31 23:00:00+00:00  _result      0 2024-10-27 23:00:00+00:00 2024-11-03 22:00:00.000001+00:00               NaN
...
2024-11-03 22:00:00+00:00  _result      0 2024-10-27 23:00:00+00:00 2024-11-03 22:00:00.000001+00:00               NaN

_temp_df.to_csv('temp_df.csv')

temp_df.csv

WARNING    warnings                  C:\my_project\venv\Lib\site-packages\influxdb_client\client\flux_csv_parser.py:258: FutureWarning:
The behavior of array concatenation with empty entries is deprecated. In a future version, this will no longer exclude empty items when determining the result dtype. To retain the old behavior, exclude the empty entries before the concat operation.

I see

>>> df.equals(_temp_df)
True

so possible solution can look like

if self._data_frame.empty:
    df = _temp_df
else:
    df = pd.concat([self._data_frame.astype(_temp_df.dtypes), _temp_df])

but I'm not sure about consequences across all codes