Google cloud storage csv file write with pandas empty lines

[P00L]

Hi,
Seems to be a difference in creating file local with respect to google cloud storage

import pandas as pd

df = pd.DataFrame({'a': [1, 2], 'b': [1, 2]})
df.to_csv('filename.csv')
df.to_csv('gs://BUCKET_NAME/filename.csv')

filename.csv

,a,b
0,1,1
1,2,2

gs://BUCKET_NAME/filename.csv

,a,b

0,1,1

1,2,2

gs://BUCKET_NAME/filename.csv file seems to add a new line for each inserted line.

python version 3.6.4
gcsfs version 0.3.0
pandas version 0.25.1

[TomAugspurger]

What platform are you on?

[P00L]

What platform are you on?

I'm using windows local environment

[TomAugspurger]

Hmm yeah that's what I guessed. Possibly something with universal line endings. Do you have time to check whether the issue is in pandas or gcsfs?

…

On Fri, Sep 13, 2019 at 7:15 AM Paolo Fusari ***@***.***> wrote: What platform are you on? I'm using windows local environment — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#176>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKAOIXOFQKSONCMRKGDAP3QJN76ZANCNFSM4IWN4RDA> .

[martindurant]

Wasn't there an issue with text mode and line endings? I thought we had dealt with that, so perhaps getting latest fsspec will fix.

[TomAugspurger]

Yeah, there were various issues around that. It only affected when a file-object was passed in, but perhaps that's what pandas is doing internally.

…

On Fri, Sep 13, 2019 at 8:41 AM Martin Durant ***@***.***> wrote: Wasn't there an issue with text mode and line endings? I thought we had dealt with that, so perhaps getting latest fsspec will fix. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#176>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKAOIRGHIEVN4OH4WQQPOLQJOKAHANCNFSM4IWN4RDA> .

[P00L]

Hmm yeah that's what I guessed. Possibly something with universal line endings. Do you have time to check whether the issue is in pandas or gcsfs?
…
On Fri, Sep 13, 2019 at 7:15 AM Paolo Fusari @.***> wrote: What platform are you on? I'm using windows local environment — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#176?email_source=notifications&email_token=AAKAOITOJS7QOJ2XNZIOGPLQJN76ZA5CNFSM4IWN4RDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6U24WQ#issuecomment-531213914>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKAOIXOFQKSONCMRKGDAP3QJN76ZANCNFSM4IWN4RDA .

I'll try to make the test this week. I also make a test creating the file on Google App Engine and the file was created correctly without empty lines.

[P00L]

Hmm yeah that's what I guessed. Possibly something with universal line endings. Do you have time to check whether the issue is in pandas or gcsfs?
…
On Fri, Sep 13, 2019 at 7:15 AM Paolo Fusari @.***> wrote: What platform are you on? I'm using windows local environment — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#176?email_source=notifications&email_token=AAKAOITOJS7QOJ2XNZIOGPLQJN76ZA5CNFSM4IWN4RDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6U24WQ#issuecomment-531213914>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKAOIXOFQKSONCMRKGDAP3QJN76ZANCNFSM4IWN4RDA .

I tried to write a file line by line only with gcsfs and worked correctly without empty lines

[TomAugspurger]

I think pandas-dev/pandas#21406 is the relevant pandas issue.

What happens with

import pandas as pd

df = pd.DataFrame({'a': [1, 2], 'b': [1, 2]})
df.to_csv('filename.csv')
df.to_csv('gs://BUCKET_NAME/filename.csv', newline="\n")

[P00L]

I think pandas-dev/pandas#21406 is the relevant pandas issue.

What happens with

import pandas as pd

df = pd.DataFrame({'a': [1, 2], 'b': [1, 2]})
df.to_csv('filename.csv')
df.to_csv('gs://BUCKET_NAME/filename.csv', newline="\n")

Sorry for the delay.
I tried the following code with line_terminator instead of newline and both file where correctly written without empty lines.

import pandas as pd

df = pd.DataFrame({'a': [1, 2], 'b': [1, 2]})
df.to_csv('filename.csv')
df.to_csv('gs://BUCKET_NAME/filename.csv', line_terminator="\n")

[martindurant]

OK to close, then?

[P00L]

Yes, thanks