How to Write and Debug C Extension Modules

The CPython interpreter allows us implement modules in C for performance critical code or to interface with external libraries while presenting users with a high level Python API. This tutorial will teach you how to leverage the power of C in your Python projects.

We will start by explaining the C representation of Python objects and how to manipulate them from within C. We will then move on to implementing functions in C for use in Python. We will discuss reference counting and correct exception handling. We will also talk about how to package and build your new extension module so that it may be shared on PyPI. (We will only be covering building extension modules on GNU/Linux and OSX, not Windows).

After the break, we will show how to implement a new type in C. This will cover how to hook into various protocols and properly support cyclic garbage collection. We will also discuss techniques for debugging C extension modules with gdb using the CPython gdb extension.

Install Steps

Prior to the tutorial, attendees need to install a git, a C89 compatible C compiler, and gdb. OSX users will also need to install brew.

For the C compiler: I recommend gcc, but clang works as well. The tutorial will be using gdb specific features so gdb is reqiured. Other C debuggers like lldb will not work.

Once all of the system packages have been installed, run the following commands in a terminal:

$ git clone --recursive https://github.com/llllllllll/c-extension-tutorial.git
$ cd c-extension-tutorial
$ source etc/setup-env

The setup-env script will attempt to download the needed dependencies, compile a debug version of CPython 3.6, and create a local virtual env with this new interpreter. This ensures that everyone is using the same version of CPython with the same compile time flags.

The setup-env should print a lot of stuff to the terminal. You can ignore most of it but the last line should be:

Environment is setup correctly!

Viewing the Tutorial

The tutorial is structured as a sphinx project. This allows the tutorial to be viewed from a standard browser or hosted online.

The material can be viewed in a browser by opening tutorial/build/html/index.html, for example:

$ ${BROWSER} tutorial/build/html/index.html

Install Steps (Alternate)

If you had significant difficulty getting your development environment set up in the recommended manner (see above), you may prefer to try working in a virtual machine.

First install Vagrant 1.9 and Virtualbox 5.1:

https://www.vagrantup.com/downloads.html

https://www.virtualbox.org/wiki/Downloads

Then run the following commands in your terminal:

$ git clone --recursive https://github.com/llllllllll/c-extension-tutorial.git
$ cd c-extension-tutorial/etc
$ vagrant plugin install vagrant-vbguest
$ vagrant up --provider=virtualbox

If all went well Vagrant will finish successfully with the message "c_extension_tutorial VM provisioned successfully". Be patient, it'll take a while to provision the VM. Don't proceed until you see the success message.

Vagrant should eventually bring up a Virtualbox GUI with a login prompt. Log in with user vagrant, password vagrant. The c-extension-tutorial directory is mounted at /home/vagrant/c-extension-tutorial. You can build the project in the VM by running the following commands in the VM terminal:

$ cd ~/c-extension-tutorial
$ source etc/setup-env

Because c-extension-tutorial is a shared folder, you will see updates propagate to your host folder. You can browse the documentation generated by the VM build in your host machine's web browser (see previous section).

If the vagrant-vbguest plugin fails to install you may be missing the Ruby development files on your host machine. If you are running Windows you may have additional difficulties. See:

https://github.com/dotless-de/vagrant-vbguest

If for whatever reason you need to blow away your VM and start over, run the following commands on your host machine:

$ cd c-extension-tutorial/etc
$ vagrant destroy