Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add UTF-8 BOM to CSV exports for Excel users #2959

Closed
grantfitzsimmons opened this issue Feb 9, 2023 · 6 comments · Fixed by #5204
Closed

Add UTF-8 BOM to CSV exports for Excel users #2959

grantfitzsimmons opened this issue Feb 9, 2023 · 6 comments · Fixed by #5204
Assignees
Labels
1 - Request Improvements or extensions to existing behavior 2 - Exporting Data Issues that are related to exporting data to DwC, GBIF, IPT, Web Portal, etc. SeparationFrom6
Milestone

Comments

@grantfitzsimmons
Copy link
Member

Excel seems to not use UTF-8 by default and requires major system changes to work

https://answers.microsoft.com/en-us/msoffice/forum/all/excel-read-csv-set-utf-8-as-default-for-all-csv/62eb4068-fc70-4f9b-9bd7-c904713beaf0

https://en.wikipedia.org/wiki/Byte_order_mark#UTF-8

Any idea why I am seeing degree symbols like this when exporting lat1text and long1text from Specify 7 via a query? I presume it has something to do with UTF8 incompatibility. Any idea how I can correct it post-process? Is this a bug?

image

Requested By: Many users

@grantfitzsimmons grantfitzsimmons added 1 - Request Improvements or extensions to existing behavior pri:unknown labels Feb 9, 2023
@grantfitzsimmons
Copy link
Member Author

This could be another export button or option so that Excel users can know that the data will appear in the correct format

@specifysoftware
Copy link

This issue has been mentioned on Specify Community Forum. There might be relevant details there:

https://discourse.specifysoftware.org/t/export-as-csv-with-alternative-delimiter-e-g-semicolon/1003/19

@FedorSteeman
Copy link

What is the status of this feature?

@grantfitzsimmons grantfitzsimmons added the 2 - Exporting Data Issues that are related to exporting data to DwC, GBIF, IPT, Web Portal, etc. label Jul 22, 2024
@grantfitzsimmons grantfitzsimmons added this to the Grant's issue list milestone Jul 22, 2024
@grantfitzsimmons
Copy link
Member Author

@FedorSteeman This has not yet been implemented, so data still must be imported into Excel using the 'Get Data' functions to import CSVs with the appropriate encoding.

The initial controversy surrounding the use of a Byte Order Mark (BOM) for UTF-8 encoding was due to the fact that it is "not required nor recommended" by the Unicode standard. However, Microsoft Excel requires a BOM to correctly identify a CSV file as UTF-8 instead of the local compatibility locale (i.e., ANSI).

Either we will need to release an update that provides the option to add BOM to CSV exports (my preferred approach) or we will need to wait for Microsoft to improve the mechanism Excel uses when opening CSV files as other spreadsheet applications have (Apple Numbers, Google Sheets, LibreOffice Calc, etc.). I will raise this issue again with the developers, but our timeline for the remainder of the year is quite full. As a result, development on a solution is unlikely to begin until early next year if our team does start work on this feature.

Thank you for your message!

@FedorSteeman
Copy link

FedorSteeman commented Jul 26, 2024

@grantfitzsimmons I cannot stress enough how large of a stumbling block this is for getting users comfortable with using Specify7 over Specify6. Since the latter can directly export to M$ Excel format, end users perceive this as a major shortcoming of the web-based version.

@grantfitzsimmons
Copy link
Member Author

@FedorSteeman We have a new PR opened to resolve this now: #5204

@CarolineDenis CarolineDenis modified the milestones: 7.9.9, 7.9.7 Aug 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
1 - Request Improvements or extensions to existing behavior 2 - Exporting Data Issues that are related to exporting data to DwC, GBIF, IPT, Web Portal, etc. SeparationFrom6
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

5 participants