add new StorageDriver: rawblock #522

laijs · 2017-03-01T16:16:20Z

The driver creates raw-file-block for every images/valume.
It likes vfs driver in the operation creating an image from
the parent. Both drivers do the operation by copying, but
rawblock driver tries to copy the parent by reflink.
So the copy-on-write is supported for the rawblock driver
if the backing(host) filesystem supports copy-on-write.

If the backing filesystem is btrfs, rawblock driver
supports copy-on-write now. If the backing filesystem
is xfs, the rawblock driver will test whether it is
reflink-enabled, if true, rawblock driver supports
copy-on-write.

rawblock driver vs vfs driver:
  hypervisor handles block better than dir(9p, sharefs).
  rawblock driver supports copy-on-write

rawblock driver vs devicemapper:
  loopback-based devicemapper is too ugly.
  rawblock driver (will) use loopback only when manipulate
    the filesystem in host.

The rawblock driver is also the base feature prepared for
dax support and pages-shared across VMs.

Signed-off-by: Lai Jiangshan <jiangshanlai@gmail.com>

laijs · 2017-03-01T16:19:15Z

# hyperctl info
Images: 4
Containers: 0
PODs: 1
Storage Driver: rawblock
  Backing Filesystem: xfs
  Support Copy-On-Write: false
  Block Filesystem: xfs
  Block Size: 10GB
...

gnawux · 2017-03-01T16:45:43Z

daemon/storage.go

+}
+
+func (s *RawBlockStorage) RemoveVolume(podId string, record []byte) error {
+	return nil


Should we remove the volume file here? Or is it created in the Pod dir?

the code was copied from other drivers. but it seems the volume file should be removed.

gnawux · 2017-03-01T16:47:33Z

daemon/storage.go

+}
+
+func (s *RawBlockStorage) InjectFile(src io.Reader, mountId, target, baseDir string, perm, uid, gid int) error {
+	return fmt.Errorf("unsupported")


The hyperd inject resolv.conf file with InjectFile by default. And we have inject file cases in CI scripts. I think we need this to be implemented finally.

gnawux · 2017-03-01T16:55:39Z

storage/graphdriver/rawblock/driver.go

+	// check if they are running over btrfs or xfs
+	switch fsMagic {
+	case graphdriver.FsMagicBtrfs: // support
+	case graphdriver.FsMagicXfs: // TODO: check support(201609) and use ioctl(FICLONE) instead of cp --reflink


This commit supports btrfs only, right?

it supports all host filesystem. only btrfs supports cow. xfs supports cow soon.

gnawux · 2017-03-01T16:59:20Z

storage/graphdriver/rawblock/driver.go

+
+	options = joinMountOptions(options, label.FormatMountLabel("", mountLabel))
+
+	//if err := mount.Mount(d.block(id), d.mnt(id), d.blockFs, options); err != nil {


Dose this work? or just use exec for debug?

we need to add loopback related code when use mount.Mount(). using mount commandline is much simpler. The 'Put()' function forgot to detach the loopdev. it needs to be fixed.

gnawux · 2017-03-01T17:13:37Z

storage/graphdriver/rawblock/driver.go

+		return d.createBaseBlock(id, mountLabel)
+	}
+	// TODO: use syscall directly, cp --reflink supports btrfs only currently
+	if out, err := exec.Command("cp", "-a", "--reflink=auto", d.block(parent), d.block(id)).CombinedOutput(); err != nil {


Won't this failed if it is not on btrfs?

"--reflink=auto", it always tries with ioctl(BTRFS_IOC_CLONE) and will fallback to normal copy(read-write) on non-btrfs. so we need to use directly syscall for xfs.

ioctl(4, BTRFS_IOC_CLONE, 0x3) = -1 ENOTTY (Inappropriate ioctl for device) fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 read(3, "..."..., 65536) = 252 write(4, "..."..., 252) = 252

Got it. Thank you

BTRFS_IOC_CLONE has been pulled to generic vfs layer as FICLONE, so ioctl(BTRFS_IOC_CLONE) should work for all file systems that support file clone.

bergwolf · 2017-03-02T00:44:46Z

daemon/storage.go

+	}
+	spec.Source = block
+	spec.Format = "xfs"
+	spec.Fstype = "raw"


Format="raw" && Fstype ="xfs"? Also set spec.Fstype according to rawblockDriver.blockFs.

rawblockDriver.blockFs can't be fetched here, we can use 'probefs()' like dm.

bergwolf · 2017-03-02T00:48:06Z

daemon/storage.go

+
+func (s *RawBlockStorage) CreateVolume(podId string, spec *apitypes.UserVolume) error {
+	block := filepath.Join(s.RootPath(), "volumes", fmt.Sprintf("%s-%s", podId, spec.Name))
+	if out, err := exec.Command("truncate", fmt.Sprintf("--size=%d", storage.DEFAULT_DM_VOL_SIZE), block).CombinedOutput(); err != nil {


call rawblockDriver.CreateVolume() here? otherwise need mkfs?

bergwolf · 2017-03-02T00:54:41Z

daemon/storage.go

+func (*RawBlockStorage) CleanUp() error { return nil }
+
+func (s *RawBlockStorage) PrepareContainer(containerId, sharedDir string) (*runv.VolumeDescription, error) {
+	devFullName := filepath.Join(s.RootPath(), "blocks", containerId)


Add a separator before containerId, like "blocks_"?

bergwolf · 2017-03-02T04:08:42Z

storage/graphdriver/rawblock/driver.go

+		return d.createBaseBlock(id, mountLabel)
+	}
+	// TODO: use syscall directly, cp --reflink supports btrfs only currently
+	if out, err := exec.Command("cp", "-a", "--reflink=auto", d.block(parent), d.block(id)).CombinedOutput(); err != nil {


BTRFS_IOC_CLONE has been pulled to generic vfs layer as FICLONE, so ioctl(BTRFS_IOC_CLONE) should work for all file systems that support file clone.

bergwolf · 2017-03-02T04:12:33Z

storage/graphdriver/rawblock/driver.go

+func (d *Driver) Get(id, mountLabel string) (string, error) {
+	d.Lock()
+	defer d.Unlock()
+	active := d.active[id]


d.active does not seem reliable upon daemon restart. e.g., after daemon restarts, d.active[id] is zero and damon tries to mount the device again, which will fail if a device is mounted before.

Possibly we can use xattr to save the refcount in the raw block file, and retrieve it in Get()/Put() when d.active[id] is zero.

The driver creates raw-file-block for every images/valume. It likes vfs driver in the operation creating an image from the parent. Both drivers do the operation by copying, but rawblock driver tries to copy the parent by reflink. So the copy-on-write is supported for the rawblock driver if the backing(host) filesystem supports copy-on-write. If the backing filesystem is btrfs, rawblock driver supports copy-on-write now. If the backing filesystem is xfs, the rawblock driver will test whether it is reflink-enabled, if true, rawblock driver supports copy-on-write. rawblock driver vs vfs driver: hypervisor handles block better than dir(9p, sharefs). rawblock driver supports copy-on-write rawblock driver vs devicemapper: loopback-based devicemapper is too ugly. rawblock driver (will) use loopback only when manipulate the filesystem in host. The rawblock driver is also the base feature prepared for dax support and pages-shared across VMs. Signed-off-by: Lai Jiangshan <jiangshanlai@gmail.com>

bergwolf · 2017-03-02T08:32:40Z

lgtm!

gnawux · 2017-03-02T08:55:53Z

travis build failed, looks like download lvm source failed.

gnawux · 2017-03-02T09:15:18Z

@laijs we need change the lvm download link to: ftp://sources.redhat.com/pub/lvm2/LVM2.2.02.131.tgz

or newer version:
ftp://sources.redhat.com/pub/lvm2/LVM2.2.02.168.tgz

The fedora site has retired.

gnawux reviewed Mar 1, 2017

View reviewed changes

bergwolf reviewed Mar 2, 2017

View reviewed changes

laijs force-pushed the rawblock branch 2 times, most recently from 54b0074 to 5e39011 Compare March 2, 2017 08:00

laijs force-pushed the rawblock branch from 5e39011 to 6cfe2c1 Compare March 2, 2017 08:06

gnawux merged commit 7f8de6b into hyperhq:master Mar 2, 2017

laijs deleted the rawblock branch June 21, 2017 06:51

gnawux mentioned this pull request Dec 3, 2018

deprecation of devicemapper: need path forward for block devices + Kata kata-containers/runtime#943

Closed

bergwolf mentioned this pull request Mar 26, 2019

Introduce the rawblock snapshotter containerd/containerd#3130

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add new StorageDriver: rawblock #522

add new StorageDriver: rawblock #522

laijs commented Mar 1, 2017 •

edited

Loading

laijs commented Mar 1, 2017

gnawux Mar 1, 2017

laijs Mar 1, 2017

gnawux Mar 1, 2017

gnawux Mar 1, 2017

laijs Mar 1, 2017

gnawux Mar 1, 2017

laijs Mar 1, 2017

gnawux Mar 1, 2017

laijs Mar 1, 2017

gnawux Mar 1, 2017

bergwolf Mar 2, 2017

bergwolf Mar 2, 2017

laijs Mar 2, 2017

bergwolf Mar 2, 2017

bergwolf Mar 2, 2017

bergwolf Mar 2, 2017

bergwolf Mar 2, 2017

bergwolf Mar 2, 2017

bergwolf commented Mar 2, 2017

gnawux commented Mar 2, 2017

gnawux commented Mar 2, 2017


		options = joinMountOptions(options, label.FormatMountLabel("", mountLabel))

		//if err := mount.Mount(d.block(id), d.mnt(id), d.blockFs, options); err != nil {

add new StorageDriver: rawblock #522

add new StorageDriver: rawblock #522

Conversation

laijs commented Mar 1, 2017 • edited Loading

laijs commented Mar 1, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bergwolf commented Mar 2, 2017

gnawux commented Mar 2, 2017

gnawux commented Mar 2, 2017

laijs commented Mar 1, 2017 •

edited

Loading