Creating a simple kernel for IPython

The architecture that has been developed for IPython and that will be the core of Project Jupyter is becoming increasingly language independent. The decoupling between the client and kernel makes it possible to write kernels in any language. The client communicates with the kernel via socket-based messaging protocols. Thus, a kernel can be written in any language that supports sockets.

However, the messaging protocols are complex. Writing a new kernel from scratch is not straightforward. Fortunately, IPython 3.0 brings a lightweight interface for kernel languages that can be wrapped in Python.

This interface can also be used to create an entirely customized experience in the IPython notebook (or another client application such as the console). Normally, Python code has to be written in every code cell; however, we can write a kernel for any domain-specific language. We just have to write a Python function that accepts a code string as input (the contents of the code cell), and sends text or rich data as output. We can also easily implement code completion and code inspection.

We can imagine many interesting interactive applications that go far beyond the original use cases of IPython. These applications might be particularly useful for nonprogrammer end users such as high school students.

In this recipe, we will create a simple graphing calculator. The calculator is transparently backed by NumPy and matplotlib. We just have to write functions as y = f(x) in a code cell to get a graph of these functions.

Getting ready

This recipe has been tested on the development version of IPython 3.0. It should work on the final version of IPython 3.0 with no or minimal changes. We give all references about wrapper kernels and messaging protocols at the end of this recipe.

How to do it...

Note

Warning: This recipe works only on IPython >= 3.0!

1. First, we create a plotkernel.py file. This file will contain the implementation of our custom kernel. Let's import a few modules:

   Note

   Be sure to put the code in steps 1-6 in an external text file named plotkernel.py, rather than in the notebook's input!

   from IPython.kernel.zmq.kernelbase import Kernel
   import numpy as np
   import matplotlib.pyplot as plt
   from io import BytesIO
   import urllib, base64

2. We write a function that returns a base64-encoded PNG representation of a matplotlib figure:

   def _to_png(fig):
       """Return a base64-encoded PNG from a 
       matplotlib figure."""
       imgdata = BytesIO()
       fig.savefig(imgdata, format='png')
       imgdata.seek(0)
       return urllib.parse.quote(
           base64.b64encode(imgdata.getvalue()))

3. Now, we write a function that parses a code string, which has the form y = f(x), and returns a NumPy function. Here, f is an arbitrary Python expression that can use NumPy functions:

   _numpy_namespace = {n: getattr(np, n) 
                       for n in dir(np)}
   def _parse_function(code):
       """Return a NumPy function from a string 'y=f(x)'."""
       return lambda x: eval(code.split('=')[1].strip(),
                             _numpy_namespace, {'x': x})

4. For our new wrapper kernel, we create a class that derives from Kernel. There are a few metadata fields we need to provide:

   class PlotKernel(Kernel):
       implementation = 'Plot'
       implementation_version = '1.0'
       language = 'python'  # will be used for
                            # syntax highlighting
       language_version = ''
       banner = "Simple plotting"

5. In this class, we implement a do_execute() method that takes code as input and sends responses to the client:

   def do_execute(self, code, silent,
                  store_history=True,
                  user_expressions=None,
                  allow_stdin=False):
   
       # We create the plot with matplotlib.
       fig = plt.figure(figsize=(6,4), dpi=100)
       x = np.linspace(-5., 5., 200)
       functions = code.split('\n')
       for fun in functions:
           f = _parse_function(fun)
           y = f(x)
           plt.plot(x, y)
       plt.xlim(-5, 5)
   
       # We create a PNG out of this plot.
       png = _to_png(fig)
   
       if not silent:
           # We send the standard output to the client.
           self.send_response(self.iopub_socket,
               'stream', {
                   'name': 'stdout', 
                   'data': 'Plotting {n} function(s)'. \
                               format(n=len(functions))})
   
           # We prepare the response with our rich data
           # (the plot).
           content = {
               'source': 'kernel',
   
               # This dictionary may contain different
               # MIME representations of the output.
               'data': {
                   'image/png': png
               },
   
               # We can specify the image size
               # in the metadata field.
               'metadata' : {
                     'image/png' : {
                       'width': 600,
                       'height': 400
                     }
                   }
           }        
   
           # We send the display_data message with the
           # contents.
           self.send_response(self.iopub_socket,
               'display_data', content)
   
       # We return the execution results.
       return {'status': 'ok',
               'execution_count': self.execution_count,
               'payload': [],
               'user_expressions': {},
              }

6. Finally, we add the following lines at the end of the file:

   if __name__ == '__main__':
       from IPython.kernel.zmq.kernelapp import IPKernelApp
       IPKernelApp.launch_instance(kernel_class=PlotKernel)

7. Our kernel is ready! The next step is to indicate to IPython that this new kernel is available. To do this, we need to create a kernel spec kernel.json file and put it in ~/.ipython/kernels/plot/. This file contains the following lines:

   {
    "argv": ["python", "-m",
             "plotkernel", "-f",
             "{connection_file}"],
    "display_name": "Plot",
    "language": "python"
   }

   The plotkernel.py file needs to be importable by Python. For example, we could simply put it in the current directory.

8. In IPython 3.0, we can launch a notebook with this kernel from the IPython notebook dashboard. There is a drop-down menu at the top right of the notebook interface that contains the list of available kernels. Select the Plot kernel to use it.
9. Finally, in a new notebook backed by our custom plot kernel, we can simply write the mathematical equation, y = f(x). The corresponding graph appears in the output area. Here is an example:
   

   Example of our custom plot wrapper kernel

How it works...

We will give more details about the architecture of IPython and the notebook in Chapter 3, Mastering the Notebook. We will just give a summary here. Note that these details might change in future versions of IPython.

The kernel and client live in different processes. They communicate via messaging protocols implemented on top of network sockets. Currently, these messages are encoded in JSON, a structured, text-based document format.

Our kernel receives code from the client (the notebook, for example). The do_execute()function is called whenever the user sends a cell's code.

The kernel can send messages back to the client with the self.send_response() method:

• The first argument is the socket, here, the IOPub socket
• The second argument is the message type, here, stream, to send back standard output or a standard error, or display_data to send back rich data
• The third argument is the contents of the message, represented as a Python dictionary

The data can contain multiple MIME representations: text, HTML, SVG, images, and others. It is up to the client to handle these data types. In particular, the HTML notebook client knows how to represent all these types in the browser.

The function returns execution results in a dictionary.

In this toy example, we always return an ok status. In production code, it would be a good idea to detect errors (syntax errors in the function definitions, for example) and return an error status instead.

All messaging protocol details can be found at the links given at the end of this recipe.

There's more...

Wrapper kernels can implement optional methods, notably for code completion and code inspection. For example, to implement code completion, we need to write the following method:

def do_complete(self, code, cursor_pos):
    return {'status': 'ok',
            'cursor_start': ...,
            'cursor_end': ...,
            'matches': [...]}

This method is called whenever the user requests code completion when the cursor is at a given cursor_pos location in the code cell. In the method's response, the cursor_start and cursor_end fields represent the interval that code completion should overwrite in the output. The matches field contains the list of suggestions.

These details might have changed by the time IPython 3.0 is released. You will find all up-to-date information in the following references:

• Wrapper kernels, available at http://ipython.org/ipython-doc/dev/development/wrapperkernels.html
• Messaging protocols, available at http://ipython.org/ipython-doc/dev/development/messaging.html
• KernelBase API reference, available at http://ipython.org/ipython-doc/dev/api/generated/IPython.kernel.zmq.kernelbase.html