Multinet

The goal of Multinet is to provide a fast, controlled and resource-efficient way to boot large-scale SDN topologies. It builds on the Mininet project to emulate SDN networks via multiple isolated topologies, each launched on a separate machine, and all connected to the same controller.

Multinet has been verified with the Lithium release of the OpenDaylight controller, where we managed to boot and connect a topology of 3000+ OVS OF 1.3 switches to a single controller instance in less than 10 minutes. The controller was running on a moderate-sized VM (8 VCPUs, 32GB memory) and the multinet topology over 10 small-sized VMs (1 VCPU, 4GB memory each).

Why isolated topologies?

The main motivation behind Multinet was to be able to stress an SDN controller in terms of its switch scalable limits. In this context, Multinet contents itself to booting topologies that are isolated from each other, without really caring to be interconnected, as we believe this policy is simple and good enough to approximate the behavior of large-scale realistic SDN networks and their interaction with the controller. If creating large-scale interconnected topologies is your primary concern, then you might want to look at other efforts such as Maxinet or the Cluster Edition Prototype of Mininet. Instead, Multinet clearly emphasizes on creating scalable pressure to the controller and provides options to control certain aspects that affect the switches-controller interaction, such as the way these are being connected during start-up.

Why multiple VMs?

The cost to boot a large Mininet topology on a single machine grows exponentially with the number of switches. To amortize this cost, we opted to scale out and utilize multiple VMs to spawn multiple smaller topologies in parallel. Eventually, one of the key questions that we try to answer through Multinet is: what is the best time to boot-up a topology of S Mininet switches with the least amount of resources?

Features

• Large-scale SDN networks emulation, using multiple isolated Mininet topologies distributed across multiple VMs
• Controllable boot-up of switches in groups of configurable size and configurable intermediate delay. This enables studying different policies of connecting large-scale topologies to the controller.
• Centralized and RESTful control of topologies via a master-worker architecture
• Well-known topology types offered out-of-the-box (disconnected, linear, ring, mesh)
• Smooth integration with custom topologies created via the high-level Mininet API, provided they have slightly modified their build method

Getting Started

Environment setup

To use Multinet you should have a distributed environment of machines configured as follows:

• Software dependencies:
  • Python 2.7
  • bottle, requests and paramiko Python packages
  • a recent version of Mininet (we support 2.2.1rc)
  • Mausezahn, tool for network traffic generation.
• Connectivity:
  • the machines should be able to communicate with each other
  • the machines should have SSH connectivity

The above software dependencies are installed inside a virtualenv (isolated Python environment), which is created from the deploy/provision.sh script which is responsible for the environment setup. In the next section we demonstrate how to prepare such an environment using Vagrant to provision and boot multiple VMs and docker to provision multiple containers.

Environment setup using Vagrant

You can use Vagrant to setup a testing environment quickly. Using the provided Vagrantfile you can boot a configurable number of fully provisioned VMs in a private network and specify their IP scheme.

Under the deploy/vagrant directory we provide scripts and Vagrantfiles to automatically setup a distributed environment of VMs to run Multinet. The steps for this are:

1. Provision the base box from which VMs will be instantiated:

   [user@machine multinet/]$ cd deploy/vagrant/base/

   If you sit behind a proxy, edit the http_proxy variable in the Vagrantfile. Then start provisioning:

   [user@machine multinet/deploy/vagrant/base]$ vagrant up

   When the above command finishes, package the base box that has been created:

   [user@machine multinet/deploy/vagrant/base]$ vagrant package --output mh-provisioned.box
   [user@machine multinet/deploy/vagrant/base]$ vagrant box add mh-provisioned mh-provisioned.box
   [user@machine multinet/deploy/vagrant/base]$ vagrant destroy

   For more info on Vagrant box packaging take a look at this guide

2. Configure the VMs:

   [user@machine multinet/]$ cd deploy/vagrant/packaged_multi/

   Edit the Vagrantfile according to your preferences. For example:

   http_proxy = ''  # if you sit behind a corporate proxy, provide it here
   mh_vm_basebox = 'mh-provisioned' # the name of the Vagrant box we created in step 2
   mh_vm_ram_mini = '2048'  # RAM size per VM
   mh_vm_cpus_mini = '2'    # number of CPUs per VM
   num_multinet_vms = 10    # total number of VMs to boot
   mh_vm_private_network_ip_mini = '10.1.1.70'  # the first IP Address in the mininet VMs IP Address range

   Optional Configuration If you need port forwarding from the master guest machine to the host machine, edit these variables inside the Vagrantfile:

   forwarded_ports_master = [] # A list of the ports the guest VM needs
                               # to forward
   forwarded_ports_host = []   # The host ports where the guest ports will be
                               # forwarded to (1 - 1 correspondence)
   # Example:
   #   port 3300 from master VM will be forwarded to port 3300 of
   #   the host machine
   #   port 6634 from master VM will be forwarded to port 6635 of
   #   the host machine
   #   forwarded_ports_master = [3300, 6634]
   #   forwarded_ports_host = [3300, 6635]

3. Boot the VMs:

   [user@machine multinet/deploy/vagrant/packaged_multi]$ vagrant up

You should now have a number of interconnected VMs with all the dependencies installed.

Environment setup using Docker

In order to create a docker container we must first create an image which will be used as a base for creating one or more docker containers. For the creation of a docker image we provide a dockerfile

1. Install docker: docker installation guides

2. Creation of docker image (without proxy settings):

   [user@machine multinet/]$ cd deploy/docker/no_proxy/
   [user@machine multinet/deploy/docker/no_proxy/]$ sudo docker build -t multinet_image .

   After this step when you run the command

   [user@machine multinet/deploy/docker/no_proxy/]$ sudo docker images

   You should see something like the following output

   REPOSITORY          TAG                   IMAGE ID            CREATED             SIZE
   <repo_name>         multinet_image        a75c906f03c7        1 minute ago        1.72 GB

3. Create containers from the created image: Open 2 terminals and execute the following command in order to create 2 docker containers

   [user@machine]$ sudo docker run -it <repo_name>:multinet_image /bin/bash

   After running the above commands on each terminal you should see the command prompt of the container. It should be something like the following

   root@cfb6dccfc41d:/#

   Multinet, inside a docker container, is under path /opt/multinet

   root@cfb6dccfc41d:/#cd /opt/multinet

The 2 containers are interconnected you can get the ip address information if you run the command

root@cfb6dccfc41d:/opt/multinet# ifconfig

The default docker network for the containers is 172.17.0.0/16. Use the IP addresses of docker containers in the configuration file for the "master_ip": and "worker_ip_list":. See next in the document in the configuration section of multinet. For more information about docker container networks visit the link Understand Docker container networks

Configuration

To start using Multinet, the first step is to clone the Multinet repository on your local machine. This machine is supposed to act as the client machine to Multinet, and is denoted in the following examples with the [user@machine] prompt.

Once you have the repository checked out, you may proceed with some configuration.

Deployment configuration

Edit the configuration file to your IP sceme:

[user@machine multinet/]$ vim config/config.json

{
  "master_ip" : "10.1.1.80",
  "master_port": 3300,
  "worker_ip_list": ["10.1.1.80", "10.1.1.81"],
  "worker_port": 3333,

  "deploy": {
    "multinet_base_dir": "/home/vagrant/multinet",
    "ssh_port": 22,
    "username": "vagrant",
    "password": "vagrant"
  }
}

• master_ip is the IP address of the machine where the master will run
• master_port is the port where the master listens for REST requests from the user or any external client application
• worker_ip_list is the list with the IPs of all machines where workers will be created to launch topologies
• worker_port is the port where each worker listens for REST requests from the master
• multinet_base_dir is the location where the Multinet repo was cloned on the client machine
• ssh_port is the port where the master and worker machines listen for SSH connections
• username, password are the credentials used to access via SSH the master and worker machines

Topology configuration

Edit the configuration file to the desired topology features:

{
  "topo": {
    "controller_ip_address":"10.1.1.39",
    "controller_of_port":6653,
    "switch_type":"ovsk",
    "topo_type":"linear",
    "topo_size":30,
    "group_size":3,
    "group_delay":100,
    "hosts_per_switch":2,
    "traffic_generation_duration_ms":60000,
    "interpacket_delay_ms":5000
  }
}

• controller_ip_address is the IP address of the machine where the SDN controller will run
• controller_of_port is the port where the controller listens for OpenFlow messages
• switch_type is the type of soft switch used for the emulation (supported types: ovsk for OVS OF1.3 switches, user for CPqD OF1.3 switches)
• topo_type is the type of topologies to be booted on every worker node (out of the box supported types: linear, mesh, ring, disconnected)
• topo_size is the size of topologies to be booted on every worker node
• group_size, group_delay are the parameters defining the gradual bootup groups (see section below)
• hosts_per_switch is the number of hosts connected to each switch of the topology
• traffic_generation_duration_ms is the amount of time in milliseconds, during which PACKET_IN's with ARP payload, will be transmitted.
• interpacket_delay_ms is connected to the traffic_generation_duration_ms and is the interval between consecutive PACKET_IN's, with ARP payload, sent from a particular Multinet worker.

Deployment

The goal of this phase is to deploy the Multinet master and worker nodes on a set of up and running physical or virtual machines that satisfy the conditions mentioned above. The provided deploy script automates the process of copying the required files on each machine and starting the master and worker REST servers. The deployment process assumes the existence of the /opt/venv_multinet directory created by deploy/provsision.sh desdribed in Environment setup section.

Run the deploy script from the client machine to copy the necessary files and start the master and the workers:

[user@machine /opt/multinet/]$./bin/venv_handler_master.sh /opt/multinet /opt/multinet/bin/deploy /opt/multinet/config/config.json

Initialize Multinet topology

This step initializes the distributed topologies without connecting them to the controller. It creates every necessary component of the topology, such as switches, links, hosts, etc.

Run the following command from the client machine:

[user@machine /opt/multinet/]$./bin/venv_handler_master.sh /opt/multinet opt/multinet/bin/handlers/init_topos /opt/multinet/config/config.json

The above will send an init command to every worker node concurrently, and an identical Mininet topology will be booted on every worker machine. If all topologies are initialized successfully you should get a 200 OK output message on the client machine. If any topology fails to initialize successfully, an error message will be printed.

Start Multinet topology

In this step the initialized topologies are being connected to the SDN controller. Multinet provides control over the way that the topologies are being connected to the controller, allowing gradual connection in a group-wise fashion. The rationale behind this is to prevent the controller from being overwhelmed at cases where a large number of switches are being exposed to it at once. As a result, the gradual start up enables investigating different policies of connecting large-scale topologies to the controller.

The group_size and group_delay configuration options control the way that the distributed topologies are being connected to the controller. The connection process proceeds in intervals as follows: at every step, group_size switches from every worker node will be connected to the controller, and after that, a delay of group_delay milliseconds will follow before proceeding to the next interval. Note that each worker node will execute the gradual booting process independent from the others, without employing any kind of intermediate synchronization.

To connect a Multinet topology after it has been initialized, run the following command from the client machine:

[user@machine /opt/multinet/]$./bin/venv_handler_master.sh /opt/multinet /opt/multinet/bin/handlers/start_topos /opt/multinet/config/config.json

The above will send a start command to every worker node in parallel and initiate the gradual connections of the topologies. In case we want to impose serialization to this process, we can use the flag --serial-requests. In this way all the requests from the master to the workers to start their topology will be send one by one serially.

[user@machine /opt/multinet/]$./bin/venv_handler_master.sh /opt/multinet /opt/multinet/bin/handlers/start_topos /opt/multinet/config/config.json --serial-requests

If all topologies are connected successfully you should get a 200 OK output message on the client machine. If any topology fails to connect, an error message will be printed.

Interact with Multinet topology

Get the number of switches

To query Multinet for the number of booted switches on each worker node, run the following command from the client machine:

[user@machine multinet/]$ ./bin/venv_handler_master.sh /opt/multinet /opt/multinet/bin/handlers/get_switches /opt/multinet/config/config.json

If the distributed topologies have been successfully booted, you should get a 200 OK message and the number of switches booted on each worker node.

Get the number of installed flows on switches of the topology

To query Multinet for the number of all installed flows on topology switches on each worker, we can use the following command:

[user@machine /opt/multinet/]$ ./bin/venv_handler_master.sh /opt/multinet /opt/multinet/bin/handlers/get_flows /opt/multinet/config/config.json

With this command on each switch we get a dump of its flows and we count them. For each worker we add the different counts of switches flows and we get the total installed flows for all the switches on the worker node. We return the per Multinet worker total installed flows.

Do a pingall operation

To perform a "pingall" operation on every worker node in parallel run the following command from the client machine:

[user@machine /opt/multinet/]$ ./bin/venv_handler_master.sh /opt/multinet /opt/multinet/bin/handlers/pingall /opt/multinet/config/config.json

If the operation runs on a successfully booted topology you should synchronously get a 200 OK response code and the pingall output should be logged. Note that a pingall operation may take a long time to complete if the topology has many hosts.

Trigger host visibility

When connecting a Multinet topology to the OpenDaylight controller, the hosts are not made automatically visible by the L2 switch plugin, but rather when they generate traffic. To trigger host visibility, we have opted to perform a dummy ping from each host to the controller, which fires a PACKET_IN transmission.

To do this, run the following command from the client machine:

[user@machine /opt/multinet/]$ ./bin/venv_handler_master.sh /opt/multinet /opt/multinet/bin/handlers/detect_hosts /opt/multinet/config/config.json

If all the topologies are booted successfully you should synchronously get a 200 OK response code. Note that a detect_hosts operation may take a long time to complete if the topology has many hosts.

Stop Multinet topology

To stop a Multinet topology run the following command from the client machine:

[user@machine /opt/multinet/]$ ./bin/venv_handler_master.sh /opt/multinet /opt/multinet/bin/handlers/stop_topos /opt/multinet/config/config.json

The above will send a stop command to every worker node in parallel and destroy the topologies. If all the topologies are destroyed successfully, you should synchronously get a 200 OK output message.

Clean machines from Multinet installation

A dedicated script exist to revert the Multinet deployment. To clean all the Multinet machines simply run:

[user@machine /opt/multinet/]$ ./bin/venv_handler_master.sh /opt/multinet /opt/multinet/bin/cleanup /opt/multinet/config/config.json

Generate PACKET_IN events with ARP payload

Multinet has the capability to generate traffic from switches to the controller. This traffic consists of PACKET_IN events, which contain in their Data field, ARP payload. This operation requires to have defined 2, as a minimum value of hosts_per_switch, in the topo section of the configuration file. For more information see in the Configuration section above.

If we asume the case that we use OpenDaylight controller with a proper configuration, to connect our multinet topology, the above mentioned PACKET_IN traffic will have as a result the provoke of FLOW_MOD's, transmitted from the controller to the switches. Controller must have the following plugins installed:

• odl-restconf-all
• odl-openflowplugin-flow-services
• odl-l2switch-switch

Aditionally we must change the values of the following configuration keys in the mentioned XML configuration files of the controller:

• <controller root directory>/etc/opendaylight/karaf/54-arphandler.xml
  • Inside the above mentioned file there is a configuration option, the is-proactive-flood-mode. We must set its value to false:

       <is-proactive-flood-mode>false</is-proactive-flood-mode>

• <controller root directory>/etc/opendaylight/karaf/58-l2switchmain.xml
  • inside the above mentioned file we have two configuration options, the reactive-flow-idle-timeout and reactive-flow-hard-timeout. We must set the minimum value for these configuration keys, which is equal to 1:

       <reactive-flow-idle-timeout>1</reactive-flow-idle-timeout>
       <reactive-flow-hard-timeout>1</reactive-flow-hard-timeout>

If the XML configuration files do not exist, we must start and stop the controller once to provoke the generation of these files.

The operational result of the above configuration is presented in the following diagram:

In order to use the PACKET_IN generation capability, the following command must be executed:

[user@machine /opt/multinet/]$ ./bin/venv_handler_master.sh /opt/multinet /opt/multinet/bin/handlers/traffic_gen /opt/multinet/config/config.json

System Architecture

The end goal of Multinet is to deploy a set of Mininet topologies over multiple machines and have them connected to the same controller simultaneously. To make this possible, every switch must have a unique DPID to avoid naming collisions in the controller's housekeeping mechanism, and to achieve this, Multinet automatically assigns a proper DPID offset to each Mininet topology.

The local Mininet topologies are identical in terms of size, structure and configuration, and they are all being handled in the same fashion simultaneously. For example, during start up, topologies are being created simultaneously on the worker machines and populated in the same way. To relieve the end user from having to manage each topology separately, we have adopted a master-worker model for centralized control.

The master process acts as the Multinet front-end that accepts REST requests from the client machine. At the same time, it orchestrates the pool of workers. On a user request, the master creates a separate command for each local topology which it dispatches simultaneously to the workers. Each worker process controls a local Mininet topology. It accepts commands from the master via REST, applies them to its topology, and responds back with a result status. Every worker is unaware of the topologies being operated from other workers. The master is responsible to collect partial results and statuses from all workers and reply to the user as soon as it has a complete global view.

For resource efficiency and speed, it is preferable to create each worker along with its topology on a separate machine.

Code Design

Code structure

Path	Description
.travis.yml	Travis CI job
bin/	Binaries
bin/handlers/	Command Line Handlers
bin/cleanuph	Cleanup script to reset the Multinet machines environment
bin/deploy	Automation script to copy and start the master and the workers in the Multinet machines
config/	Configuration file for the handlers, the deployment and the master
figs/	Figures needed for documentation
multi/	Module containing the Master / Worker REST servers
net/	Module containing the Mininet related functionality
net/multinet.py	Class inheriting from the core Mininet with added / modified functionality
net/topologies.py	example topologies
test	basic functionality tests
travis-jobs	Travis CI machine provisioning helper scripts
util/	Utility modules
vagrant/	Vagrantfiles for fast provisioning of a running environment

Interacting with the master programmatically

Via REST

To make easier the communication between a client application and the master node, we augmented it with a REST API. The client application can issue the POST requests shown below to interact with Multinet programmatically. In essence, the command line handlers presented in the previous sections are wrapper scripts to those POST requests.

• Initialize Multinet topology

  @bottle.route('/init', method='POST')

  When making a POST request to the init endpoint, you must also send a JSON body with the following format (also see the Initialize Multinet topology section)

  {
        "controller_ip_address":"10.1.1.39",
        "controller_of_port":6653,
        "switch_type":"ovsk",
        "topo_type":"linear",
        "topo_size":30,
        "group_size":3,
        "group_delay":100,
        "hosts_per_switch":2,
        "traffic_generation_duration_ms":60000,
        "interpacket_delay_ms":5000
  }

• Start Multinet topology

  @bottle.route('/start', method='POST')

• Get the number of switches

  @bottle.route('/get_switches', method='POST')

• Perform a pingall in each topology

  @bottle.route('/ping_all', method='POST')

• Make the hosts visible

  @bottle.route('/detect_hosts', method='POST')

• Stop the topologies

  @bottle.route('/stop', method='POST')

• Generate PACKET_IN traffic

  @bottle.route('/generate_traffic', method='POST')

Via Python

You can also utilize the wrappers from the multinet_requests module for the same purpose. Example:

1. For initialization

   # Send a POST request to the master 'init' endpoint
   
   topo_data= {
       "controller_ip_address":"10.1.1.39",
       "controller_of_port":6653,
       "switch_type":"ovsk",
       "topo_type":"linear",
       "topo_size":30,
       "group_size":3,
       "group_delay":100,
       "hosts_per_switch":2,
       "traffic_generation_duration_ms":60000,
       "interpacket_delay_ms":5000
   }
   
   multinet_requests.master_cmd(master_ip,
                                master_port,
                                'init',
                                data=topo_data)

2. And for any other operation

   # Send a POST request to any master endpoint
   # The endpoint is specified by the 'opcode' parameter
   
   opcode = choose_one_of(['start', 'get_switches', 'ping_all', 'detect_hosts', 'stop'])
   multinet_requests.master_cmd(master_ip, master_port, opcode)

Core components

• Multinet class
  • Extends the Mininet class.
  • Adds a dpid offset during the switch creation phase to distinguish between the switches in different instances.
  • Inserts the notion of gradual switch bootup, inserting some idle time (group_delay) between the bootup of groups of switches (group_size)
• worker
  • creates its own Multinet instance
  • creates a REST API that wraps the exposed methods of that instance.
• master
  • exposes a REST API to the end user.
  • broadcasts the commands to the workers
  • aggregates the responses and returns a summary response to the end user

Adding your own topologies

To be able to plug any topology created with the high level Mininet API to Multinet, modify the build method to conform with the following method signature:

# k is the number of switches
# n is the number of hosts per switch
# dpid is the dpid offset
def build(self, k=2, n=1, dpid=1, **_opts):

1. Create a topology with the Mininet high level API, for example

## mytopo.py
class MyTopo(Topo):
      "Disconnected topology of k switches, with n hosts per switch."

      def build(self, k=2, n=1, dpid=1, **_opts):
          """
          k: number of switches
          n: number of hosts per switch
          dpid: the dpid offset (to enable distributed topology creation)
          """
          self.k = k
          self.n = n

          for i in xrange(k):
              # Add switch
              switch = self.addSwitch(genSwitchName(i, dpid))
              # Add hosts to switch
              for j in xrange(n):
                  host = self.addHost(genHostName(i, j, dpid, n))
                  self.addLink(host, switch)

2. Add the new topology to the Multinet.TOPOS dictionary

   # worker.py
   import mytopo ...
   Multinet.TOPOS['mytopo'] = mytopo.MyTopo
   MININET_TOPO = Multinet( ... )

   # or from inside multinet.py
   TOPOS = {
      'linear': ...
      'mytopo': mytopo.MyTopo
   }