generated zip archives now support utf8 file names

[DeepDiver1975]

refs #1086, #930, #578

File/folder names:
中文blah中文blah.txt
undaçao.doc
öäüß.txt

Currently known issues/observations:

• Chinese characters are not displayed within my Gnome Archive Tools
• On Windows XP with built in zip support the file and folder names are totally screwed

Next steps:

• either generate zips depending on the client os
• or normalize the file names to ASCII 😧

@owncloud/core-developers
I need reviewers and tester on Windows, Mac and Linux

@skodak thanks a lot for relicensing your code!
Can I ask you to add a comment on this PR where you state again that you agree to relicense? THX

[ghost]

--- Original Nachricht ---
Betreff: Re: Relicensing of zip_archive::fix_utf8_flags() under AGPL3
Von: Petr Skoda info@skodak.org
An: "Thomas Müller" thomas.mueller@ownCloud.org
Datum: 08.01.2013 1:08

Hello,

sure, I do not mind dual licensing it under GPL3 and AGPL3, I give you the permission to reuse and adopt my code that hacks around utf8 limitations in standard zip extension for PHP.

Petr Skoda

On 7. 1. 2013, at 23:46, Thomas Müller thomas.mueller@ownCloud.org wrote:

Hi Petr,

I'm a open source developer within the ownCloud project.

While searching for a solution to the well known zip and utf8 filename issue
I found your code within the moodle project.

As ownCloud uses AGPL3 license and your code has been released under
GPL3 I cannot simply reuse your code right away.

Can I kindly ask you to grant me the permission to reuse and adopt your code
under the AGPL3 license?

Thanks a lot,

Thomas Müller

ownCloud
Your Cloud, Your Data, Your Way!

[DeepDiver1975]

PLEASE DO NOT MERGE - LICENSING HAS TO BE CLARIFIED!

[skodak]

Hello, I hope the code is going to work fine for you. If you find any problems or discover some new way to improve it please ping me. Ciao.

[karlitschek]

👍 @skodak licensed this as MIT so this is ready for merging. If we find another reviewers of course.

[DeepDiver1975]

Summing reviewers .....
@butonic @Raydiation @LukasReschke @bartv2 .... maybe? THX

[dragotin]

The code uses fseek and others which are subject of doubts about big file handling, ie. file with size > 2GB. It should be tested with files > 2GB and > 4GB to verify it works, at least on 64 bit (PHP-) platforms.

[DeepDiver1975]

@dragotin good point - even I doubt that is makes sense to generate zips pf that size 😄

[dragotin]

@DeepDiver1975 sure, it does not make sense to have that large files in general but people do... Users don't think in terms of file sizes, they just do stuff. The thing is that these overflow errors cause all funny side effect errors which are hard to track down, because you usually do not get a warning or error somewhere in the logfiles. So its better to keep in mind...

[DeepDiver1975]

sure - I'll test this code with big files.
FYI: there is a max zip size which can be defined by the ownCloud admin - default is 800MB

[dragotin]

On 16.01.2013 10:19, Thomas Müller wrote:

sure - I'll test this code with big files.
Thanks.

FYI: there is a max zip size which can be defined by the ownCloud admin - default is 800MB
Thats very good. Maybe it should simply be forbidden to extend this
value over 2GB. That makes it secure for all cases.

[jacotec]

Just want to add that also common ZIP software like 7-zip got the special chars broken in OC generated ZIP files, not only Windows internal zip support. (OC 4.5.6)

[butonic]

still broken. I locally merged master and downloaded a folder as zip. not only are utf8 symbols replaced with ?? but subfolders get pulled to the root folder of the zip. with the following folder structure:

+ tozip
  + empty folder
  + ö ä ü ß
  | + öä+
  + 中文blah中文blah
  | + äöü#+
  + undaçao.doc
  + öäüß.txt
  + 中文blah中文blah.txt

first of all the 'empty folder' does not show up in the zip at all
the dir structure of the zip looks like this (looking at it with gnome archive viewer):

+ tozip.zip
  + tozip
  | + unda?????ao.doc
  | + ?????????????blah?????????????blah.txt
  | + ?????????????blah?????????????blah.txt
  | + ??????????????????????.txt
  + ?????????????blah?????????????blah
  | + ?????????????????#+
  + ????????
    + ????+

@DeepDiver1975 rebase please

[nicolaskarp]

Still broken for me as well with the current release :

• Server OS : Linux / kernel 3.2.0-33
• OwnCloud 5
• Browser : Firefox 19.0.2
• Client OS : Windows 7

The issue is happening when I try to download several files (zip) :

For Instance :

Filename with accent :  Carte_Identité_Recto.jpg
Inside the zip after unzip : Carte_Identit+®_Recto.jpg 

When I download only a file, it's working.

Nic

[mahiuchun]

For Gnome Archive Tools, aka, File Roller, the solution is to install p7zip as the issue of UTF-8 zip is caused by Info-Zip (patch it a little bit can also help, which is being done in some distro now).

For Mac and Windows, I think recent versions of third-party software should all be fine with UTF-8 zip.

[karlitschek]

What is the status here?

[PVince81]

Just observed the issue on OC6 beta3.

Please rebase.

[mahiuchun]

Ubuntu is going to fix its unzip of 12.04-13.04 for utf8 file names issue (13.10 is already okay). It is in SRU verification phase now. Can anyone help the verification?

http://pad.lv/1199239

[PVince81]

Going to try and rebase this onto master to revive it a bit.

[PVince81]

Here you go: mega-rebase onto master.

I've tested it and it still doesn't work, which, by reading the previous comments, seem to never have worked properly... at least we can use this as a base to continue experimenting.

[PVince81]

Trying to debug this: the file name workaround only seem to operate on the first file in the zip. There is still hope 😄

[ghost]

Test failed.
Refer to this link for build results: https://ci.owncloud.org/job/pull-request-analyser/2327/

[PVince81]

That should explain those numbers: http://unix.stackexchange.com/a/14727
I think I'll just use them as is as they may contain the correct attribute flags for Windows as well.

[PVince81]

Ladies and gentlemen, please help testing this: download this ZIP file generated with this fix and let us know whether you were able to
1) See file names without encoding issues
and
2) Extract it with correct file names

https://owncloud.vincentpetry.net/public.php?service=files&t=6955b2fa7fc20fc76d2b94fb6ed433fc&download

• Linux unzip 6.00
• Ark (KDE)
• Gnome Fileroller
• P7Zip (Linux)
• WinZIP
• WinRAR WinXP
• WinRAR Win7
• 🚫 7Zip WinXP
• 7Zip Win7
• 7Zip Win8
• 🚫 Windows' ZIP directory WinXP
• 🚫 Windows' ZIP directory Win7
• Windows' ZIP directory Win8
• Mac (not sure how it's called)
• other tools... feel free to add

[georgehrke]

Works fine on mac:

[PVince81]

Summonning the original reporters: @Hausmarke @ricardoar7 @bureautranslations @RealRancor @PKduck @frisi

See #1117 (comment) if you'd like to help testing ZIP with UTF-8 characters.

[frisi]

i'm on kde too @PVince81. ark works fine, as does unzip in console.

[ghost]

Test failed.
Refer to this link for build results: https://ci.owncloud.org/job/pull-request-analyser/2333/

[georgehrke]

@PVince81 and the folder with the Chinese characters is the third and not the second folder :P

[PVince81]

@georgehrke it was the second one I created until I decided to test umlauts as well 😉

[DeepDiver1975]

[DeepDiver1975]

on WinXP the content is not properly displayed using 7zip and integrated zip feature in explorer
will retest on win8 soon ...

[PVince81]

From my research it looks like Windows expects the file name to be encoded based on the system's encoding, not UTF-8. Which means we'd need to detect Windows and encode using the current locale... which means that the zip files wouldn't be the same according to which OS it has been created from... which might be acceptable if the purpose of the zip file is only to download multiple files that are going to be extracted locally anyway.

Needs more research...

[futal]

Apparently, not all zip implementations have UTF8 support so the standard specifies a "Unicode Path Extra Field" understood by most popular software. See evanmiller/mod_zip#3

The base filename should then default to a "most likely" encoding, possibly with transliteration and accent stripping from the original filenames for better compatibility. Different solutions for PHP are discussed at http://stackoverflow.com/questions/1284535/php-transliteration

[PVince81]

@futal thanks for the info.
In the current fix we set the utf-8 bit but it doesn't seem to be enough.
I haven't found any info that says that the "Unicode Path Extra Field" solution also works in Windows (Winzip, Winrar, 7zip).

To implement this we'd need to rewrite the whole file and insert the relevant fields. The current workarounds were only about replacing bytes in the existing zip files generated by PHP.

[PVince81]

According to http://support.microsoft.com/kb/2704299 there is a slight chance that the test file generated above can be extracted properly on Windows 7. Would be good if someone could test this 😄

[futal]

I have tested the following files with Windows 7 Pro 64 SP1:

• Encoding.zip from generated zip archives now support utf8 file names #1117 (comment)
• Broken_encoding.zip from generated zip archives now support utf8 file names #1117 (comment)
• ziponefile.zip from generated zip archives now support utf8 file names #1117 (comment)
• ziponefile_broken from generated zip archives now support utf8 file names #1117 (comment)
• zipfixed.zip from generated zip archives now support utf8 file names #1117 (comment)

For all these zip files, filenames were good with 7zip 9.22beta and wrong with Windows Explorer.

[PVince81]

@futal cool, thanks.
Can someone test with WinZIP, WinRAR on WinXP/Win7 ?

[georgehrke]

I'll test winrar on win7 this evening

[georgehrke]

Works fine with WinRAR on Windows 7 Pro

[georgehrke]

maybe we should check winrar on xp too

[ghost]

Test failed.
Refer to this link for build results: https://ci.owncloud.org/job/pull-request-analyser/2344/

[PVince81]

After discussing with @DeepDiver1975 we decided to use ZipStreamer instead as it will not only fix the utf-8 file names issues but also significantly improve download performance.

See #6893

[PepeN]

Hello,

I have the same problem.

Server OC 6.0.3RC1 (CentOS release 6.5 (i686) httpd-2.2.15-30.el6.centos.i686 php-5.4.27-1.el6.remi.i686)
Clients Windows 2003 IE 8, Chrome, 7zip 9.20 x64

[DeepDiver1975]

@PepeN this issue has been fixed for the upcoming OC7