Refactor: Use the netlink library instead of exec'ing out to ip

CHANGELOG.md

@@ -2,6 +2,62 @@
 
 # Changelog
 
+## v0.5.0-alpha.1, 06/08/2019
+
+This is the first prerelease in the `v0.5.x` series. Please try it out, and also note we now have arm64 builds!
+
+### New Features
+
+-  Add new binary:  `ignited` ([#264](https://github.com/weaveworks/ignite/pull/264), [@luxas](https://github.com/luxas))
+-  Add ARM64 support ([#173](https://github.com/weaveworks/ignite/pull/173), [@luxas](https://github.com/luxas))
+-  Add new command: `ignite exec` ([#232](https://github.com/weaveworks/ignite/pull/232), [@BenTheElder](https://github.com/BenTheElder))
+-  Add Manifest Directory support (like kubelet's Static Pods) ([#234](https://github.com/weaveworks/ignite/pull/234), [@twelho](https://github.com/twelho))
+-  Support directories as well with the `--copy-files` flag ([#271](https://github.com/weaveworks/ignite/pull/271), [@twelho](https://github.com/twelho))
+-  Implement read-write GitOps ([#241](https://github.com/weaveworks/ignite/pull/241), [@twelho](https://github.com/twelho))
+
+### API Changes
+
+-  Add the initial v1alpha2 API types ([#250](https://github.com/weaveworks/ignite/pull/250), [@twelho](https://github.com/twelho))
+
+### Enhancements
+
+-  Automatically generate the release notes ([#283](https://github.com/weaveworks/ignite/pull/283), [@luxas](https://github.com/luxas))
+-  Structured logging across the application; add logging support to `ignite-spawn` ([#247](https://github.com/weaveworks/ignite/pull/247), [@twelho](https://github.com/twelho))
+-  Extract watcher, batcher and monitor into pkg/util ([#245](https://github.com/weaveworks/ignite/pull/245), [@luxas](https://github.com/luxas))
+-  Robust recursive FileWatcher support using `notify` ([#265](https://github.com/weaveworks/ignite/pull/265), [@twelho](https://github.com/twelho))
+-  Document developer meetings ([#272](https://github.com/weaveworks/ignite/pull/272), [@dholbach](https://github.com/dholbach))
+-  Enable testing in CI, fix the Makefile and tidy ([#280](https://github.com/weaveworks/ignite/pull/280), [@luxas](https://github.com/luxas))
+-  Create/use a runtime interface instead of direct calls to Docker ([#211](https://github.com/weaveworks/ignite/pull/211), [@twelho](https://github.com/twelho))
+-  Add structured validation for the API types ([#216](https://github.com/weaveworks/ignite/pull/216), [@luxas](https://github.com/luxas))
+-  Add Strategic Merge Patch support to the storage ([#225](https://github.com/weaveworks/ignite/pull/225), [@luxas](https://github.com/luxas))
+-  Improve vulnerability scanning of Docker image ([#239](https://github.com/weaveworks/ignite/pull/239), [@DieterReuter](https://github.com/DieterReuter))
+-  CNI networking cleanup support, Docker client robustness improvements ([#111](https://github.com/weaveworks/ignite/pull/111), [@twelho](https://github.com/twelho))
+-  Support checksum-based Cache invalidation, improve cache's object handling ([#227](https://github.com/weaveworks/ignite/pull/227), [@twelho](https://github.com/twelho))
+-  Rename GitStorage into ManifestStorage ([#226](https://github.com/weaveworks/ignite/pull/226), [@luxas](https://github.com/luxas))
+-  Client and Storage rework: Recognize multiple API groups ([#221](https://github.com/weaveworks/ignite/pull/221), [@luxas](https://github.com/luxas))
+-  Create internal API types, and use them ([#215](https://github.com/weaveworks/ignite/pull/215), [@luxas](https://github.com/luxas))
+
+### Bug Fixes
+
+-  Ensure the directory for `godoc2md` ([#231](https://github.com/weaveworks/ignite/pull/231), [@BenTheElder](https://github.com/BenTheElder))
+-  Run `gofmt` first after generating code ([#236](https://github.com/weaveworks/ignite/pull/236), [@BenTheElder](https://github.com/BenTheElder))
+-  Fix image root permissions ([#249](https://github.com/weaveworks/ignite/pull/249), [@praseodym](https://github.com/praseodym))
+-  Separate graph generation from `make tidy`, add make target docs ([#233](https://github.com/weaveworks/ignite/pull/233), [@twelho](https://github.com/twelho))
+
+### Documentation
+
+-  Add a Read The Docs website: `ignite.readthedocs.org` ([#246](https://github.com/weaveworks/ignite/pull/246), [@dholbach](https://github.com/dholbach))
+-  Documentation updates and clarifications for the New Storage implementation ([#242](https://github.com/weaveworks/ignite/pull/242), [@twelho](https://github.com/twelho))
+-  Index awesome doc ([#276](https://github.com/weaveworks/ignite/pull/276), [@dholbach](https://github.com/dholbach))
+-  Update docs links ([#268](https://github.com/weaveworks/ignite/pull/268), [@dholbach](https://github.com/dholbach))
+-  Add Google Group to docs for Calendar and permissions ([#248](https://github.com/weaveworks/ignite/pull/248), [@stealthybox](https://github.com/stealthybox))
+-  Docs fix: Remove duplicate bracket ([#212](https://github.com/weaveworks/ignite/pull/212), [@silenceshell](https://github.com/silenceshell))
+-  Docs fix: Update the command for deleting all VMs ([#201](https://github.com/weaveworks/ignite/pull/201), [@curx](https://github.com/curx))
+-  Docs fix: Duplicate bracket ([#218](https://github.com/weaveworks/ignite/pull/218), [@silenceshell](https://github.com/silenceshell))
+-  Docs fix: ID is in `.metadata.uid`, not `.metadata.name` ([#219](https://github.com/weaveworks/ignite/pull/219), [@silenceshell](https://github.com/silenceshell))
+-  Add an awesome-ignite list for ignite ([#270](https://github.com/weaveworks/ignite/pull/270), [@luxas](https://github.com/luxas))
+-  Changed --kernel to --kernel-image for accuracy ([#217](https://github.com/weaveworks/ignite/pull/217), [@paavan98pm](https://github.com/paavan98pm))
+
 ## v0.4.2, 16/07/2019
 
 The second patch release for the `v0.4.x` release stream.

docs/releases/v0.5.0-alpha.1.md

@@ -0,0 +1,56 @@
+## v0.5.0-alpha.1, 06/08/2019
+
+This is the first prerelease in the `v0.5.x` series. Please try it out, and also note we now have arm64 builds!
+
+### New Features
+
+-  Add new binary:  `ignited` ([#264](https://github.com/weaveworks/ignite/pull/264), [@luxas](https://github.com/luxas))
+-  Add ARM64 support ([#173](https://github.com/weaveworks/ignite/pull/173), [@luxas](https://github.com/luxas))
+-  Add new command: `ignite exec` ([#232](https://github.com/weaveworks/ignite/pull/232), [@BenTheElder](https://github.com/BenTheElder))
+-  Add Manifest Directory support (like kubelet's Static Pods) ([#234](https://github.com/weaveworks/ignite/pull/234), [@twelho](https://github.com/twelho))
+-  Support directories as well with the `--copy-files` flag ([#271](https://github.com/weaveworks/ignite/pull/271), [@twelho](https://github.com/twelho))
+-  Implement read-write GitOps ([#241](https://github.com/weaveworks/ignite/pull/241), [@twelho](https://github.com/twelho))
+
+### API Changes
+
+-  Add the initial v1alpha2 API types ([#250](https://github.com/weaveworks/ignite/pull/250), [@twelho](https://github.com/twelho))
+
+### Enhancements
+
+-  Automatically generate the release notes ([#283](https://github.com/weaveworks/ignite/pull/283), [@luxas](https://github.com/luxas))
+-  Structured logging across the application; add logging support to `ignite-spawn` ([#247](https://github.com/weaveworks/ignite/pull/247), [@twelho](https://github.com/twelho))
+-  Extract watcher, batcher and monitor into pkg/util ([#245](https://github.com/weaveworks/ignite/pull/245), [@luxas](https://github.com/luxas))
+-  Robust recursive FileWatcher support using `notify` ([#265](https://github.com/weaveworks/ignite/pull/265), [@twelho](https://github.com/twelho))
+-  Document developer meetings ([#272](https://github.com/weaveworks/ignite/pull/272), [@dholbach](https://github.com/dholbach))
+-  Enable testing in CI, fix the Makefile and tidy ([#280](https://github.com/weaveworks/ignite/pull/280), [@luxas](https://github.com/luxas))
+-  Create/use a runtime interface instead of direct calls to Docker ([#211](https://github.com/weaveworks/ignite/pull/211), [@twelho](https://github.com/twelho))
+-  Add structured validation for the API types ([#216](https://github.com/weaveworks/ignite/pull/216), [@luxas](https://github.com/luxas))
+-  Add Strategic Merge Patch support to the storage ([#225](https://github.com/weaveworks/ignite/pull/225), [@luxas](https://github.com/luxas))
+-  Improve vulnerability scanning of Docker image ([#239](https://github.com/weaveworks/ignite/pull/239), [@DieterReuter](https://github.com/DieterReuter))
+-  CNI networking cleanup support, Docker client robustness improvements ([#111](https://github.com/weaveworks/ignite/pull/111), [@twelho](https://github.com/twelho))
+-  Support checksum-based Cache invalidation, improve cache's object handling ([#227](https://github.com/weaveworks/ignite/pull/227), [@twelho](https://github.com/twelho))
+-  Rename GitStorage into ManifestStorage ([#226](https://github.com/weaveworks/ignite/pull/226), [@luxas](https://github.com/luxas))
+-  Client and Storage rework: Recognize multiple API groups ([#221](https://github.com/weaveworks/ignite/pull/221), [@luxas](https://github.com/luxas))
+-  Create internal API types, and use them ([#215](https://github.com/weaveworks/ignite/pull/215), [@luxas](https://github.com/luxas))
+
+### Bug Fixes
+
+-  Ensure the directory for `godoc2md` ([#231](https://github.com/weaveworks/ignite/pull/231), [@BenTheElder](https://github.com/BenTheElder))
+-  Run `gofmt` first after generating code ([#236](https://github.com/weaveworks/ignite/pull/236), [@BenTheElder](https://github.com/BenTheElder))
+-  Fix image root permissions ([#249](https://github.com/weaveworks/ignite/pull/249), [@praseodym](https://github.com/praseodym))
+-  Separate graph generation from `make tidy`, add make target docs ([#233](https://github.com/weaveworks/ignite/pull/233), [@twelho](https://github.com/twelho))
+
+### Documentation
+
+-  Add a Read The Docs website: `ignite.readthedocs.org` ([#246](https://github.com/weaveworks/ignite/pull/246), [@dholbach](https://github.com/dholbach))
+-  Documentation updates and clarifications for the New Storage implementation ([#242](https://github.com/weaveworks/ignite/pull/242), [@twelho](https://github.com/twelho))
+-  Index awesome doc ([#276](https://github.com/weaveworks/ignite/pull/276), [@dholbach](https://github.com/dholbach))
+-  Update docs links ([#268](https://github.com/weaveworks/ignite/pull/268), [@dholbach](https://github.com/dholbach))
+-  Add Google Group to docs for Calendar and permissions ([#248](https://github.com/weaveworks/ignite/pull/248), [@stealthybox](https://github.com/stealthybox))
+-  Docs fix: Remove duplicate bracket ([#212](https://github.com/weaveworks/ignite/pull/212), [@silenceshell](https://github.com/silenceshell))
+-  Docs fix: Update the command for deleting all VMs ([#201](https://github.com/weaveworks/ignite/pull/201), [@curx](https://github.com/curx))
+-  Docs fix: Duplicate bracket ([#218](https://github.com/weaveworks/ignite/pull/218), [@silenceshell](https://github.com/silenceshell))
+-  Docs fix: ID is in `.metadata.uid`, not `.metadata.name` ([#219](https://github.com/weaveworks/ignite/pull/219), [@silenceshell](https://github.com/silenceshell))
+-  Add an awesome-ignite list for ignite ([#270](https://github.com/weaveworks/ignite/pull/270), [@luxas](https://github.com/luxas))
+-  Changed --kernel to --kernel-image for accuracy ([#217](https://github.com/weaveworks/ignite/pull/217), [@paavan98pm](https://github.com/paavan98pm))
+

go.mod

@@ -33,9 +33,12 @@ require (
 	github.com/sirupsen/logrus v1.4.2
 	github.com/spf13/cobra v0.0.5
 	github.com/spf13/pflag v1.0.3
+	github.com/vishvananda/netlink v1.0.0
+	github.com/vishvananda/netns v0.0.0-20190625233234-7109fa855b0f // indirect
 	github.com/weaveworks/flux v0.0.0-20190704153721-8292179855e1
 	github.com/whilp/git-urls v0.0.0-20160530060445-31bac0d230fa // indirect
 	golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4
+	golang.org/x/net v0.0.0-20190620200207-3b0461eec859 // indirect
 	golang.org/x/sys v0.0.0-20190616124812-15dcb6c0061f
 	golang.org/x/time v0.0.0-20190308202827-9d24e82272b4 // indirect
 	google.golang.org/grpc v1.21.1 // indirect

go.sum

@@ -249,6 +249,10 @@ github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXf
 github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
 github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
+github.com/vishvananda/netlink v1.0.0 h1:bqNY2lgheFIu1meHUFSH3d7vG93AFyqg3oGbJCOJgSM=
+github.com/vishvananda/netlink v1.0.0/go.mod h1:+SR5DhBJrl6ZM7CoCKvpw5BKroDKQ+PJqOg65H/2ktk=
+github.com/vishvananda/netns v0.0.0-20190625233234-7109fa855b0f h1:nBX3nTcmxEtHSERBJaIo1Qa26VwRaopnZmfDQUXsF4I=
+github.com/vishvananda/netns v0.0.0-20190625233234-7109fa855b0f/go.mod h1:ZjcWmFBXmLKZu9Nxj3WKYEafiSqer2rnvPr0en9UNpI=
 github.com/whilp/git-urls v0.0.0-20160530060445-31bac0d230fa h1:rW+Lu6281ed/4XGuVIa4/YebTRNvoUJlfJ44ktEVwZk=
 github.com/whilp/git-urls v0.0.0-20160530060445-31bac0d230fa/go.mod h1:2rx5KE5FLD0HRfkkpyn8JwbVLBdhgeiOb2D2D9LLKM4=
 github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
@@ -272,6 +276,8 @@ golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3 h1:0GoQqolDA55aaLxZyTzK/Y2eP
 golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190613194153-d28f0bde5980 h1:dfGZHvZk057jK2MCeWus/TowKpJ8y4AmooUzdBSR9GU=
 golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20190620200207-3b0461eec859 h1:R/3boaszxrf1GEUWTVDzSKVwLmSJpwZ1yqXm8j0v2QI=
+golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=

hack/cobra.go

@@ -1,10 +1,10 @@
 package main
 
 import (
+	"fmt"
 	"log"
 	"os"
 	"os/exec"
-	"fmt"
 
 	"github.com/spf13/cobra"
 	"github.com/spf13/cobra/doc"
@@ -20,7 +20,7 @@ func main() {
 	}
 
 	cmds := map[string]*cobra.Command{
-		"ignite": ignitecmd.NewIgniteCommand(os.Stdin, os.Stdout, os.Stderr),
+		"ignite":  ignitecmd.NewIgniteCommand(os.Stdin, os.Stdout, os.Stderr),
 		"ignited": ignitedcmd.NewIgnitedCommand(os.Stdin, os.Stdout, os.Stderr),
 	}
 

pkg/container/network.go

@@ -7,8 +7,8 @@ import (
 
 	"github.com/pkg/errors"
 	log "github.com/sirupsen/logrus"
+	"github.com/vishvananda/netlink"
 	"github.com/weaveworks/ignite/pkg/constants"
-	"github.com/weaveworks/ignite/pkg/util"
 	"k8s.io/apimachinery/pkg/util/wait"
 )
 
@@ -117,19 +117,31 @@ func bridge(iface *net.Interface) (*DHCPInterface, error) {
 	tapName := constants.TAP_PREFIX + iface.Name
 	bridgeName := constants.BRIDGE_PREFIX + iface.Name
 
-	if err := createTAPAdapter(tapName); err != nil {
+	handle, err := netlink.NewHandle()
+	if err != nil {
+		return nil, err
+	}
+
+	tuntap, err := createTAPAdapter(handle, tapName)
+	if err != nil {
+		return nil, err
+	}
+
+	bridge, err := createBridge(handle, bridgeName)
+	if err != nil {
 		return nil, err
 	}
 
-	if err := createBridge(bridgeName); err != nil {
+	if err = handle.LinkSetMaster(tuntap, bridge); err != nil {
 		return nil, err
 	}
 
-	if err := connectAdapterToBridge(tapName, bridgeName); err != nil {
+	link, err := netlink.LinkByName(iface.Name)
+	if err != nil {
 		return nil, err
 	}
 
-	if err := connectAdapterToBridge(iface.Name, bridgeName); err != nil {
+	if err = handle.LinkSetMaster(link, bridge); err != nil {
 		return nil, err
 	}
 
@@ -147,6 +159,11 @@ func takeAddress(iface *net.Interface) (*net.IPNet, bool, error) {
 		return nil, true, fmt.Errorf("interface %s has no address", iface.Name)
 	}
 
+	handle, err := netlink.NewHandle()
+	if err != nil {
+		return nil, false, errors.Wrapf(err, "failed to acquire handle on network namespace")
+	}
+
 	for _, addr := range addrs {
 		var ip net.IP
 		var mask net.IPMask
@@ -169,7 +186,17 @@ func takeAddress(iface *net.Interface) (*net.IPNet, bool, error) {
 			continue
 		}
 
-		if _, err := util.ExecuteCommand("ip", "addr", "del", ip.String(), "dev", iface.Name); err != nil {
+		link, err := netlink.LinkByName(iface.Name)
+		if err != nil {
+			return nil, false, errors.Wrapf(err, "failed to get interface by name %s", iface.Name)
+		}
+
+		delAddr, err := netlink.ParseAddr(addr.String())
+		if err != nil {
+			return nil, false, errors.Wrapf(err, "failed to parse address from stringified ip %s", addr.String())
+		}
+
+		if err = handle.AddrDel(link, delAddr); err != nil {
 			return nil, false, errors.Wrapf(err, "failed to remove address from interface %s", iface.Name)
 		}
 
@@ -184,28 +211,31 @@ func takeAddress(iface *net.Interface) (*net.IPNet, bool, error) {
 	return nil, false, fmt.Errorf("interface %s has no valid addresses", iface.Name)
 }
 
-func createTAPAdapter(tapName string) error {
-	if _, err := util.ExecuteCommand("ip", "tuntap", "add", "mode", "tap", tapName); err != nil {
-		return err
+func createTAPAdapter(handle *netlink.Handle, tapName string) (*netlink.Tuntap, error) {
+	la := netlink.NewLinkAttrs()
+	la.Name = tapName
+	tuntap := &netlink.Tuntap{
+		LinkAttrs: la,
+		Mode:      netlink.TUNTAP_MODE_TAP,
 	}
-
-	return setLinkUp(tapName)
-}
-
-func createBridge(bridgeName string) error {
-	if _, err := util.ExecuteCommand("ip", "link", "add", "name", bridgeName, "type", "bridge"); err != nil {
-		return err
+	if err := netlink.LinkAdd(tuntap); err != nil {
+		return nil, err
 	}
-
-	return setLinkUp(bridgeName)
-}
-
-func setLinkUp(adapterName string) error {
-	_, err := util.ExecuteCommand("ip", "link", "set", adapterName, "up")
-	return err
+	if err := netlink.LinkSetUp(tuntap); err != nil {
+		return nil, err
+	}
+	return tuntap, nil
 }
 
-func connectAdapterToBridge(adapterName, bridgeName string) error {
-	_, err := util.ExecuteCommand("ip", "link", "set", adapterName, "master", bridgeName)
-	return err
+func createBridge(handle *netlink.Handle, bridgeName string) (*netlink.Bridge, error) {
+	la := netlink.NewLinkAttrs()
+	la.Name = bridgeName
+	bridge := &netlink.Bridge{LinkAttrs: la}
+	if err := netlink.LinkAdd(bridge); err != nil {
+		return nil, err
+	}
+	if err := netlink.LinkSetUp(bridge); err != nil {
+		return nil, err
+	}
+	return bridge, nil
 }

third_party/forked/github.com/weaveworks/flux/git/url.go

@@ -4,7 +4,7 @@ import (
 	"fmt"
 	"net/url"
 
-	"github.com/whilp/git-urls"
+	giturls "github.com/whilp/git-urls"
 )
 
 // Remote points at a git repo somewhere.