Creating a simple kernel for IPython
The architecture that has been developed for IPython and that will be the core of Project Jupyter is becoming increasingly language independent. The decoupling between the client and kernel makes it possible to write kernels in any language. The client communicates with the kernel via socket-based messaging protocols. Thus, a kernel can be written in any language that supports sockets.
However, the messaging protocols are complex. Writing a new kernel from scratch is not straightforward. Fortunately, IPython 3.0 brings a lightweight interface for kernel languages that can be wrapped in Python.
This interface can also be used to create an entirely customized experience in the IPython notebook (or another client application such as the console). Normally, Python code has to be written in every code cell; however, we can write a kernel for any domain-specific language. We just have to write a Python function that accepts a code string as input (the contents of the code cell), and sends text or rich data as output. We can also easily implement code completion and code inspection.
We can imagine many interesting interactive applications that go far beyond the original use cases of IPython. These applications might be particularly useful for nonprogrammer end users such as high school students.
In this recipe, we will create a simple graphing calculator. The calculator is transparently backed by NumPy and matplotlib. We just have to write functions as y = f(x)
in a code cell to get a graph of these functions.
This recipe has been tested on the development version of IPython 3.0. It should work on the final version of IPython 3.0 with no or minimal changes. We give all references about wrapper kernels and messaging protocols at the end of this recipe.
Note
Warning: This recipe works only on IPython >= 3.0!
- First, we create a
plotkernel.py
file. This file will contain the implementation of our custom kernel. Let's import a few modules:Note
Be sure to put the code in steps 1-6 in an external text file named plotkernel.py
, rather than in the notebook's input!
- We write a function that returns a base64-encoded PNG representation of a matplotlib figure:
- Now, we write a function that parses a code string, which has the form
y = f(x)
, and returns a NumPy function. Here, f
is an arbitrary Python expression that can use NumPy functions: - For our new wrapper kernel, we create a class that derives from
Kernel
. There are a few metadata fields we need to provide: - In this class, we implement a
do_execute()
method that takes code as input and sends responses to the client: - Finally, we add the following lines at the end of the file:
- Our kernel is ready! The next step is to indicate to IPython that this new kernel is available. To do this, we need to create a kernel spec
kernel.json
file and put it in ~/.ipython/kernels/plot/
. This file contains the following lines:The plotkernel.py
file needs to be importable by Python. For example, we could simply put it in the current directory.
- In IPython 3.0, we can launch a notebook with this kernel from the IPython notebook dashboard. There is a drop-down menu at the top right of the notebook interface that contains the list of available kernels. Select the Plot kernel to use it.
- Finally, in a new notebook backed by our custom plot kernel, we can simply write the mathematical equation,
y = f(x)
. The corresponding graph appears in the output area. Here is an example:
We will give more details about the architecture of IPython and the notebook in Chapter 3, Mastering the Notebook. We will just give a summary here. Note that these details might change in future versions of IPython.
The kernel and client live in different processes. They communicate via messaging protocols implemented on top of network sockets. Currently, these messages are encoded in JSON, a structured, text-based document format.
Our kernel receives code from the client (the notebook, for example). The do_execute()
function is called whenever the user sends a cell's code.
The kernel can send messages back to the client with the self.send_response()
method:
- The first argument is the socket, here, the IOPub socket
- The second argument is the message type, here,
stream
, to send back standard output or a standard error, or display_data
to send back rich data - The third argument is the contents of the message, represented as a Python dictionary
The data can contain multiple MIME representations: text, HTML, SVG, images, and others. It is up to the client to handle these data types. In particular, the HTML notebook client knows how to represent all these types in the browser.
The function returns execution results in a dictionary.
In this toy example, we always return an ok
status. In production code, it would be a good idea to detect errors (syntax errors in the function definitions, for example) and return an error status instead.
All messaging protocol details can be found at the links given at the end of this recipe.
Wrapper kernels can implement optional methods, notably for code completion and code inspection. For example, to implement code completion, we need to write the following method:
This method is called whenever the user requests code completion when the cursor is at a given cursor_pos
location in the code cell. In the method's response, the cursor_start
and cursor_end
fields represent the interval that code completion should overwrite in the output. The matches
field contains the list of suggestions.
These details might have changed by the time IPython 3.0 is released. You will find all up-to-date information in the following references: