Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How difficult it is to make appdmg cross-platform? #14

Open
szwacz opened this issue Nov 22, 2013 · 52 comments
Open

How difficult it is to make appdmg cross-platform? #14

szwacz opened this issue Nov 22, 2013 · 52 comments

Comments

@szwacz
Copy link
Contributor

szwacz commented Nov 22, 2013

Hello again.

This lib could be intergallactically awesome if dependent only on pure-js node libraries, so it can run on every OS node is running on. How difficult it is to make it this way?
You mentioned previously building .DS_Dtore.
What are the other problems to address?

If this is relatively easy I might be interested in contributing to this effort.

@LinusU
Copy link
Owner

LinusU commented Nov 22, 2013

Information regarding the .DS_Store file format:

Roadmap:

  • Find SetFile replacement

We use SetFile to enable the icon-flag. Using a pure js implementation here would be awesome just in it self since SetFile is distributed with Xcode. This is only necessary for icon-support.

  • Find hdiutil replacement

We need to be able to create a dmg file from a folder, this shouldn't be too hard. hdiutil is distributed with vanilla OS X so this is only needed for support of other platforms.

  • Find bless replacement

We use the bless command to make sure that the Finder opens the image after mounting it. I would guess that this program really just sets some bits. Should be fairly easy to implement in pure js.

  • Generate .DS_Store file

This is, AFAIT, the hardest part. Right now we use apple script to open up the finder, rearrange the icons and set the background. Then sleep for some seconds which makes Finder save the changes to the .DS_Store file.

The best way to do this would is writing the file directly from the program, but we need to figure out how the file format works and what bits and bytes we need to set in order to accomplish the following:

  • Set icon size
  • Set view to icon view
  • Hide toolbar, sidebar, etc.
  • Set the background

@LinusU
Copy link
Owner

LinusU commented Nov 22, 2013

I'm going to start investigating SetFile now, since that would benefit the application at large, and report back here later.

@szwacz
Copy link
Contributor Author

szwacz commented Nov 22, 2013

I don't know if .DS_Store is actually that scary, afterall you don't have to understand the whole spec of it, you don't want to parse it, you just want to put some values in predefined binnary :)

When I read about DMG images themselves it looks like more of a hedache http://newosxbook.com/DMG.html
Since this is fully proprietary standard it is only reverse-engineered to other platforms. There is something for linux: http://serverhorror.wordpress.com/2011/02/26/create-dmg-images-for-os-x-on-linux/ but I can find nothing for windows.

@LinusU
Copy link
Owner

LinusU commented Nov 22, 2013

Ohh, I actually thought that dmg images was just a raw binary file with a file system... I have some reading up to do :)

@LinusU
Copy link
Owner

LinusU commented Nov 22, 2013

The SetFile is a nightmare as well, but it seems like it's just really an extended attributes of 32 bytes (com.apple.FinderInfo).

This is the output on a clean folder with only the custom icon applied.

xattr -p com.apple.FinderInfo /tmp/test1607
00 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

This project might have some relevant info as well: https://code.google.com/p/profuse/wiki/XattrUtility

@LinusU
Copy link
Owner

LinusU commented Nov 22, 2013

Fun fun fun, I think I actually have a way to do SetFile cross-platform.

> x = require('xattr')
{ list: [Function], set: [Function] }
> x.list
[Function]
> x.list.toString()
'function () { [native code] }'
> x.list('/tmp/test1607')
{ 'com.apple.FinderInfo': '\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0004\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000' }
> x.list('/tmp/test1607')['com.apple.FinderInfo']
'\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0004\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000'
> var target = x.list('/tmp/test1607')['com.apple.FinderInfo']
undefined
> target
'\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0004\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000'
> x.set('/tmp/test1734', 'com.apple.FinderInfo', target)
true
> 
  • /tmp/test1607 is a folder which I have used SetFile on
  • /tmp/test1734 is a folder I made with mkdir

@LinusU
Copy link
Owner

LinusU commented Nov 22, 2013

Okay, so I don't know why this works (in that I don't know what the different bits mean) but this should create a buffer that can be written to the com.apple.FinderInfo xattr.

var buf = new Buffer(32);
buf.fill(0);
buf.writeUInt8(4, 8);
return buf.toString('binary');

@LinusU
Copy link
Owner

LinusU commented Nov 22, 2013

Seems like most of the things we need for .DS_Store is documented on the mozilla wiki.

Structure fwi0 type blob is Finder window information. Known length is 0x10 (16). The data is first four two-byte values representing the top, left, bottom, and right edges of the rect defining the content area of the window. The next four bytes represent the view of the window, "icnv" is icon view. The next four bytes are unknown.

Structure fwvh type shor is Finder window vertical height. If present, it overrides the height defined by the rect in fwi0. The Finder seems to create these (at least on 10.4) even though it will do the right thing for window height with only an fwi0 around, perhaps this is because the stored height is weird when accounting for toolbars and status bars.

Structure icvt type shor is icon view text label (filename) size.

Structure icvo type blob is icon view options. Known length 0x12 (18), first 4 bytes "icvo", then 8 unknown bytes (flags?), then 2 bytes corresponding to the selected icon view size, then 4 unknown bytes (0x6e 6f 6e 65) (the text "none", guess that this is the "keep arranged by" setting?)

Structure Iloc type blob is icon location for the last-identified file. Length is 0x10 (16), two 4-byte values representing the horizontal and vertical positions of the icon's center (not top-left). (Then, 6 bytes 0xff and 2 bytes 0?) For the purposes of the center, the icon only is taken into account, not any label. The icon's size comes from the icvo blob.

Structure BKGD type blob known length 0x0c (12) is for the background. It contains a reference to another strucutre, the first four bytes are PctB, followed by four bytes indicating the length of the referenced pict structure (same as the pict's length), then 00 00 00 13 (always the same?)

Structure pict type blob, length dependent on contents. This is for the background, along with BKGD. The contents: first, two empty bytes (00 00) followed by 4 bytes giving the length of the entire structure again, same as the length following "pictblob" in the header. Then, 4 bytes, 00 02 00 00. Then, 1 byte giving the length of the volume name. Then, 31 bytes to hold the volume name, unused bytes are 00...

This seems easy! 🍻

@LinusU
Copy link
Owner

LinusU commented Nov 23, 2013

I tried looking at the FinderInfo after running bless and most 00 where replaced by 20, no idea why thought 🎊

xattr -p com.apple.FinderInfo /tmp/test1734
20 20 20 20 20 20 20 20 04 20 20 20 20 20 20 20
20 20 20 20 00 00 00 00 20 20 20 20 20 20 20 20

GetFileInfo shows no difference with or without bless...

@LinusU LinusU mentioned this issue Dec 4, 2013
@LinusU
Copy link
Owner

LinusU commented Jan 2, 2014

I have now stopped using seticonflag so this should be one step closer to success!

@al45tair
Copy link

The FinderInfo data is documented (see the man page for getattrlist on a Mac OS X system), so there’s really no need to guess at its contents.

FWIW, I’ve already got portable Python code for reading/writing/editing .DS_Store files (see the ds_store module on PyPI), and semi-portable code for generating the Alias record you need to set background images (see mac_alias on PyPI). There are some issues with making the alias code fully portable (namely it needs to be able to extract HFS+ catalog node IDs from the filesystem somehow in order to generate the alias record correctly).

There’s also my dmgbuild tool, which is based on the above; that currently uses SetFile, though I have in mind that I might ditch that some time soon.

@LinusU
Copy link
Owner

LinusU commented Mar 14, 2014

Cool, I'm gonna have a look when I have time and updated my code.

Had a quick look at your documentation, it seems to be well done, keep up the good work! Feel free to contact me at any time if you need any help or want to discuss some file format :)

@adam-lynch
Copy link

This would be unbelievable!

@felicienfrancois
Copy link

For hdutil replacement, maybe hfsutils can do the job.
http://serverhorror.wordpress.com/2011/02/26/create-dmg-images-for-os-x-on-linux/

hfsutils is licensed under GPLv2 license and seems to have ports for windows:
http://www.mars.org/home/rob/proj/hfs/

@felicienfrancois
Copy link

There is also https://github.com/planetbeing/libdmg-hfsplus which supports HFS+

It seems to be buildable for both linux and windows: https://github.com/planetbeing/libdmg-hfsplus/blob/master/CMakeLists.txt

@stevenvachon
Copy link

+1

@guidograzioli
Copy link

You can also create a dmg (uncompressed tough) on linux, with an iso9660/udf hybrid filesystem, with the command:

# genisoimage -V Name -D -R -apple -no-pad -o Name.dmg directory

@al45tair
Copy link

@guidograzioli There’s nothing special about uncompressed DMGs — they’re just raw disk image files, so you can equally create one using dd and the appropriate mkfs command (you don’t need to use genisoimage). You can even mount uncompressed DMG files (assuming you have the correct filesystem drivers in the kernel) using mount -oloop /path/to/disk/image.dmg /mount/point.

It’s also worth pointing out that the Mac is able to handle a number of filesystems, besides HFS+, out of the box, including FAT, NTFS (read-only) and ExFAT as well as a variant of UFS. There’s nothing stopping you from creating DMGs using those filesystems; modern Mac software doesn’t really use many of the special features of HFS+ anyway, and most users won’t notice and don’t care anyway, so you won’t be losing much by doing that, and no need for an HFS driver.

@bundyo
Copy link

bundyo commented Jun 30, 2015

I managed to build a working dmg under Linux, using genisoimage and the dmg tool from this repo to compress it (its image building doesn't work, but compresses fine).

Turned out since OS X and Linux are both unix based, you only need to have in the root a hidden .background folder with background.png in it for the image, the said .DS_Store with the background specified (which I built on a real OS X in order to use it), a .VolumeIcon.icns holding the icons and a normal symlink called Applications to /Applications. And of course the .app folder. Something like this:

image/ -> | .background/     -> | background.png
          | MyApp.app/       -> | ...
          | Applications     ~ /Applications
          | .DS_Store
          | .VolumeIcon.icns

Then I used genisoimage like this (where image/ is the dmg root above):

genisoimage -D -V "App Name" -no-pad -r -apple -file-mode 0755 -o ./my_temp.dmg image/

And dmg like this:

dmg dmg ./my_temp.dmg ./my_app.dmg

However afterwards I've hit another roadblock - codesign, so I moved my build to an OS X machine. :(

I hope this helps.

@LinusU
Copy link
Owner

LinusU commented Jun 30, 2015

@bundyo The .DS_Store can be generated even on linux with node-ds-store. The link to /Application is usually an alias instead of a symlink, but I guess that maybe it doesn't really matter...

@al45tair
Copy link

@LinusU It’s better to use a symlink rather than an alias for /Applications IMO. That’s what we’ve always done.

@bundyo
Copy link

bundyo commented Jun 30, 2015

Thanks. @LinusU I basically did the same the first time, though with GHex :)

@bundyo
Copy link

bundyo commented Jun 30, 2015

Is there an alternative to codesign, aside from ldid, which AFAIK doesn't work on Linux or for OS X apps?

@AlicanC
Copy link

AlicanC commented Oct 7, 2015

Dropping util.sh() should also be in the checklist.

You can use fs-extra's copy() for util.sh('cp', ['-R', resolvePath(entry.path), finalPath], cb).

@LinusU
Copy link
Owner

LinusU commented Dec 3, 2015

No problem, thank you. I'll have a look this weekend

@kevinmartin
Copy link

@LinusU , what is missing for cross platform support?

@LinusU
Copy link
Owner

LinusU commented Dec 3, 2015

  • A way to make an .dmg file on Linux/Windows
  • macos-alias for Linux/Windows (should be easy)
  • bless replacement (might not be that important)

I think that should be it, there was some work by someone working on packaging electron apps. I think that they had solved it.

@stevenvachon

This comment has been minimized.

@LinusU

This comment has been minimized.

@medfreeman
Copy link

I wonder if dmg generation could work with exFat (works on all platforms RW), using dd and mkfs on linux, and dd,FileDisk on windows (both are under GPL) with DiskPart. I'll try this mid-january.

@thanpolas
Copy link

Is there a way to not throw an error on npm install so as to be able to build our apps on CI environments which typically use ubuntu?

No appdmg method will be called on those environments.

What do you do today to test your apps that require appdmg?

@jakubzitny
Copy link

@thanpolas add it to optionalDependencies

"optionalDependencies": {
  "appdmg": "^0.3.5"
}

@johnhaley81
Copy link

@jakubzitny doesn't work if you shrinkwrap :( npm/npm#2679

@BenjaminVanRyseghem
Copy link

@thanpolas maybe a bit late, but what I did was to put appdmg as an optional dependency.

it now looks like

  "devDependencies": {
    // ...
  },
  "optionalDependencies": {
    "appdmg": "^0.3.5"
  },
  "dependencies": {
    // ...
  }

@0181532686cf4a31163be0bf3e6bb6732bf
Copy link

0181532686cf4a31163be0bf3e6bb6732bf commented Sep 3, 2016

@LinusU What's the current status of this? Is electron-userland/electron-builder#14 did help in any way?

@develar
Copy link
Contributor

develar commented Sep 3, 2016

@lyssdod You can sign app only on macOS. And unsigned app is blocked. So, for what you need to build DMG?

@0181532686cf4a31163be0bf3e6bb6732bf

@develar I'm trying to setup a multiplatform build via Jenkins using electron-builder. I also have a spare mac which can be used for signing. Would it be simpler to build mac version on mac?

@develar
Copy link
Contributor

develar commented Sep 3, 2016

@lyssdod Please see https://github.com/electron-userland/electron-builder/wiki/Multi-Platform-Build

I also have a spare mac which can be used for signing. Would it be simpler to build mac version on mac?

Will be simpler and more robust :) Yes, for example, in my company we do use separate servers to sign products, but only due to security reasons. So, I suggest just use mac to build version for mac. Or use Travis/CircleCI.

@0181532686cf4a31163be0bf3e6bb6732bf

@develar Thanks!

@develar
Copy link
Contributor

develar commented Oct 10, 2016

@gbaumgart ...and then your app will be blocked in any case because unsigned ;) If you can use unsigned app (i.e. it is ok for your users) — well, just use zip or tar.gz And even more — Apple recommends to sign even DMG since macOS Sierra (electron-builder doesn't sign DMG files yet, but definitely it will be fixed this autumn).

@quorten
Copy link

quorten commented Nov 15, 2016

By the way, we have the source code for bless right here:

https://opensource.apple.com/source/bless/bless-105/

Thus, if there are any doubts about the functionality of bless, we can look through the source code to verify the behavior. Indeed, from looking at the source myself, some parts of the code merely set/clear words to perform the desired operations.

@mjmacleod
Copy link

I wonder if isign might be the missing peice of the puzzle here, to replace codesign.
sauce-archives/isign#88 - currently only works for iOS apps, but it seems likely it could be modified to work with normal osx apps as well.

@chunta

This comment has been minimized.

@MadfishDT
Copy link

MadfishDT commented Jan 9, 2020

case of dmg for OSX, I have made dmg file generator server in linux system and serve this dmg file to client. according to previous comment, I used genisoimage, libdmg that is modified by me, ds-store npm that is modified by me. use theses things, you can make dmg is compressed and designed in linux system. hmm, windows system have some file system problem.

@o0101
Copy link

o0101 commented Oct 24, 2020

You can build dmg on Deb/Ubuntu via:

sudo apt install hfsprogs
dd if=/dev/zero of=./$name.dmg bs=1M count=16 status=progress
mkfs.hfsplus -v Install ./$name.dmg

Does this help?

Source: https://askubuntu.com/questions/1117461/how-do-i-create-a-dmg-file-on-linux-ubuntu-for-macos

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests