Skip to content

Category: Mininet

TCP BBR Congestion Control on Mininet

In this post, I demonstrate some benefits of using BBR congestion control and illustrate how easy it is to adopt it by using Mininet as an example. I’m excited to share this post with you guys because it’s been a while since I’ve made a tutorial and I love breakthrough innovations like this.

This post is divided into three sections: Background on BBR, Tutorial and Technical challenges.

Background on BBR

TCP BBR has significantly increased throughput and reduced latency on Google’s internal backbone networks. From this  a great resource:

TCP BBR is rate-based rather than window-based; that is, at any one time, TCP BBR sends at a given calculated rate, instead of sending new data in response to each received ACK. In particular, TCP BBR does not directly link the sending of new data to the receipt of ACKs, and so, strictly speaking, is not actually a sliding-windows implementation. Therefore, we cannot properly talk about winsize or cwnd. Instead, we talk about the number of packets in flight, which is the rate times RTTactual, with the understanding that this number may vary with conditions.

Basically, BBR estimates bandwidth by keeping track of goodput: if an increase in the sender rate does not increase the observed goodput, it assumes that’s the available bandwidth. It is reasonably effective in doing so and that way it provides minimal queueing in the network.

TCP’s throughput is inversely proportional to RTT and most TCP implementations cause additional delays, in consequence, TCP by itself can never reach 100% utilization. BBR changes that, that’s why it’s such an impressive accomplishment.

Quick start

Open Source is great because it allows innovation to be deployed much faster, BBR is already implemented in the Linux kernel and using Mininet you can test it right away.

I’m a long time fan of the website: reproducing network research from Stanford. I leveraged most of the Mininet code for this experiment from there.

Now let’s get to it!! This tutorial assumes you have vagrant and git. If you don’t, don’t panic, follow this link. To start you will need to set up the VM. I took care of all the dependencies for you. If you want to inspect what I’m doing take a look at the mininet role in the ansible folder.

git clone https://github.com/castroflavio/bbr-replication/
git checkout vagrant
vagrant up

This should take 10 min to complete. After it’s done proceed

vagrant ssh
cd mininet
sudo ./figure5.sh all

After around 30 seconds the experiment should be done and you can exit the VM:

exit
open figure5_mininet/figure5_mininet.png

This should open the following figure:figure5_mininet

The figure compares the latency on TCP BBR and TCP CUBIC (less is better). And as you can see BBR reduces the latency from ~150ms to ~50ms(66%) on the average case and from 400ms to 50ms (87%) on the worst case. This is crazy!

Technical challenges

The first technical challenge is finding a linux kernel that implements BBR, and it turns out it’s implemented on 4.9 so look out for that. The second challenge was to implement the BBR pacing mechanism, it was mentioned on the CS244 website but I did not understand it at first.

BBR requires a mechanism to control the sender rate and it leverages tc ( traffic control ) module from linux. I knew about tc but I didn’t know it was such a powerful tool. After some research on linux queueing mechanisms, I found that BBR requires the fq (Fair queueing) queueing discipline because it uses that to rate control the sender. It turns out Mininet did not support fq for some reason, and I had to change a couple lines of code to add support for it.

Conclusion

TCP has been around for decades and for decades people have been trying to improve it. At first, TCP congestion control mechanism literally saved the internet, now I’m gonna be bold and say that BBR by providing a “queueless” congestion control is saving latency-sensitive applications. It really is a big deal. I highly encourage you to try it out, the least you should do is check the following article: Increase your linux server Internet speed with TCP BBR congestion control.

For future reference:

Leave a Comment

Easiest way to install OpenVswitch and Mininet on Ubuntu 12.04

I’ve been struggling trying to set up the OVS 2.0 with Mininet 2.2 on Ubuntu 12.04.

Apparently the installation script from Mininet is not fully working on Ubuntu 12.04.

What did I do to fix it?

The easiest way to install it is to use the Mininet installation script and then do a small fix.

I did this exactly to install:

sudo apt-get update
sudo apt-get install -y git
git clone git://github.com/mininet/mininet
cd mininet
git checkout -b 2.2.0 2.2.0
util/install.sh -nf
util/install.sh -V 2.3.0

After this you will some weird errors. It should look like this:

Setting up openvswitch-switch (2.3.0-1) ...
 * Inserting openvswitch module
 * /etc/openvswitch/conf.db does not exist
 * Creating empty database /etc/openvswitch/conf.db
 * Starting ovsdb-server
2015-01-02T00:57:50Z|00002|stream_unix|WARN|/usr/local/var/run/openvswitch/db.sock: connection failed (No such file or directory)
2015-01-02T00:57:50Z|00003|reconnect|WARN|unix:/usr/local/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory)
2015-01-02T00:57:51Z|00004|stream_unix|WARN|/usr/local/var/run/openvswitch/db.sock: connection failed (No such file or directory)

You should interrupt the script now with ctrl+C. Next do this:

cd ~/openvswitch/openvswitch-2.3.0/
sudo -s
ovsdb-tool create /usr/local/etc/openvswitch/conf.db vswitchd/vswitch.ovsschema
ovsdb-server -v --remote=punix:/usr/local/var/run/openvswitch/db.sock \
--remote=db:Open_vSwitch,Open_vSwitch,manager_options \
--private-key=db:Open_vSwitch,SSL,private_key \
--certificate=db:Open_vSwitch,SSL,certificate \
--pidfile --detach --log-file
ovs-vsctl --no-wait init
ovs-vswitchd --pidfile --detach
ovs-vsctl show

This partially solves the problem. Let’s go ahead and test it: sudo mn --test pingall.

If you reboot you are going to have the same error. What you can do is to run this everytime you reboot:

cd ~/openvswitch/openvswitch-2.3.0/
sudo -s
ovsdb-tool convert /usr/local/etc/openvswitch/conf.db vswitchd/vswitch.ovsschema
ovsdb-server -v --remote=punix:/usr/local/var/run/openvswitch/db.sock \
--remote=db:Open_vSwitch,Open_vSwitch,manager_options \
--private-key=db:Open_vSwitch,SSL,private_key \
--certificate=db:Open_vSwitch,SSL,certificate \
--pidfile --detach --log-file
ovs-vsctl --no-wait init
ovs-vswitchd --pidfile --detach
ovs-vsctl show

If you find a best solution or this does not work for you, please report me your issue. I will do my best to help you!

Regards,

Leave a Comment

Setting up OpenVswitch and Mininet 2.2 on Ubuntu 14.04

In this tutorial I’ll briefly describe how to perform a standard installation of Mininet. This tutorial should work on Ubuntu 14.04.

sudo apt-get update
sudo apt-get install -y git
git clone git://github.com/mininet/mininet
cd mininet
git checkout -b 2.2.0 2.2.0
util/install.sh -nfv

Do install.sh -h to explore additional options.

3 Comments