In this article by Roger Ye, the author of the book Android System Programming, introduces two challenges present in most of the embedded Linux system programming that you need to resolve before you can boot up your system. These two challenges are:

• How to load your kernel and ramdisk image?
• Where do you store your filesystem?

(For more resources related to this topic, see here.)

This is true for Android systems as well. After you got a development board, you have to build the bootloader first and flash it to the storage on the board before you can move to the next step. After that, you have to build the kernel, ramdisk, and filesystem. You have to repeat this tedious build, flash, and test process again and again. In this process, you need to use special tools to flash various images for the development board.

Many embedded system developers want to get rid of the process of flashing images so that they can concentrate on the development work itself. Usually, they use two techniques PXE boot and NFS filesystem. If you search "Android NFS" on the Internet, you can find many articles or discussions about this topic.

I don't have a development board on my hand, so I will use VirtualBox as a virtual hardware board to demonstrate how to boot a system using PXE bootloader and NFS as filesystem.

To repeat the same process in this article, you need to have the following hardware and software environment.

• A computer running Ubuntu 14.04 as the host environment
• VirtualBox Version 5.1.2 or above
• A virtual machine running Android x86vbox
• A virtual machine running Ubuntu 14.04 as PXE server (optional)

Android x86vbox is a ROM that I developed in the book Android System Programming. You can download the ROM image at the following URL:

https://sourceforge.net/projects/android-system-programming/files/android-7/ch14/ch14.zip/download

After you download the preceding ZIP file, you can find a list of files here:

• initrd.img: This is the modified ramdisk image from open source project android-x86
• kernel: NFS-enabled Android kernel for device x86vbox
• ramdisk.img: ramdisk for the Android boot
• ramdisk-recovery.img: ramdisk for the recovery boot
• update-android-7.1.1_r4_x86vbox_ch14_r1.zip: OTA update image of x86vbox, you can install this image using recovery

Setting up a PXE boot environment

What is PXE? PXE means Preboot eXecution Environment. Before we can boot a Linux environment, what we need is to find a way to load kernel and ramdisk to the system memory. This is one of the major tasks performed by most of Linux bootloader. The bootloader usually fetches kernel and ramdisk from a kind of storage device, such as flash storage, harddisk, or USB. It can also be retrieved from a network connection. PXE is a method that we can boot a device with LAN connection and a PXE-capable network interface controller (NIC).

As shown in the following diagram, PXE uses DHCP and TFTP protocols to complete the boot process. In a simplest environment, a PXE server is setup as both DHCP and TFTP server. The client NIC obtains the IP address from DHCP server and uses TFTP protocol to get the kernel and ramdisk images to start the boot process.

I will demonstrate how to prepare a PXE-capable ROM for VirtualBox virtio network adapter so we can use this ROM to boot the system via PXE. You will also learn how to set up a PXE server which is the key element in the PXE boot setup. In VirtualBox, it also includes a built-in PXE server. We will explore this option as well.

Preparing PXE Boot ROM

Even though PXE boot is supported by VirtualBox, but the setup is not consistent on different host platforms. You may get error message like PXE-E3C - TFTP Error - Access Violation during the boot. This is because the PXE boot depends on LAN boot ROM. When you choose different network adapters, you may get different test results. To get a consistent test result, you can use the LAN boot ROM from Etherboot/gPXE project. gPXE is an open source (GPL) network bootloader. It provides a direct replacement for proprietary PXE ROMs, with many extra features such as DNS, HTTP, iSCSI, and so on. There is a page at gPXE project website about how to set up LAN boot ROM for VirtualBox:

http://www.etherboot.org/wiki/romburning/vbox

The following table is a list of network adapters supported by VirtualBox.

Since paravirtualized network has better performance in most of the situation, we will explore how to support PXE boot using virtio-net network adapter.

Downloading and building the LAN boot ROM

There may be LAN boot ROM binary image available on the Internet, but it is not provided at gPXE project. We have to build from source code according to the instructions from gPXE project website.

Let's download and build the source code using the following commands.

$ git clone git://git.etherboot.org/scm/gpxe.git

$ cd gpxe/src

$ make bin/1af41000.rom  # for virtio 1af4:1000  

Fixing up the ROM image

Before the ROM image can be used, the ROM image has to be updated due to VirtualBox have the following requirements on ROM image size:

• Size must be 4K aligned (that is, a multiple of 4096)
• Size must not be greater than 64K

Let's check the image size first and make sure it is not larger than 65536 bytes (64K):

$ ls -l bin/1af41000.rom | awk '{print $5}'

62464  

We can see that it is less than 64K. Now, we have to pad the image file to a 4K boundary. We can do this using the following commands.

$ python

>>> 65536 - 62464             # Calculate padding size

3072

>>> f = open('bin/1af41000.rom', 'a')

>>> f.write(' ' * 3072)      # Pad with zeroes  

We can check the image file size again.

$ ls -l 1af41000.rom | awk '{print $5}'

65536  

As we can see, the size is 64K now.

Configuring the virtual machine to use the LAN boot ROM

To use this LAN boot ROM, we can use command VBoxManage to update VirtualBox settings. We use the following command to set the LanBootRom path:

$ VBoxManage setextradata $VM_NAME VBoxInternal/Devices/pcbios/0/Config/LanBootRom /path/to/1af41000.rom  

Replace $VM_NAME with your VM's name. If you use global as $VM_NAME then all VMs will use the gPXE LAN boot ROM.

To remove the above configuration, you just have to reset the path value as below.

$ VBoxManage setextradata $VM_NAME VBoxInternal/Devices/pcbios/0/Config/LanBootRom  

You can also check the current configuration using the below command:

$ VBoxManage getextradata $VM_NAME VBoxInternal/Devices/pcbios/0/Config/LanBootRom

Value: /path/to/1af41000.rom 

If you don't want to build LAN boot ROM yourself, you can use the one that I posted at:

https://sourceforge.net/projects/android-system-programming/files/android-7/ch14/1af41000.rom/download

Setting up PXE boot environment

With a proper PXE ROM installed, we can set up the PXE on the host now. Before we setup a PXE server, we need to think about the network connections. There are three ways a virtual machine in VirtualBox can connect to the network.

• Bridged network: Connect to the same physical network as the host. It looks like the virtual machine connects to the same LAN connection as the host.
• Host only network: Connect to a virtual network, which is only visible by the virtual machine and the host. In this configuration, the virtual machine cannot connect to outside network, such as Internet.
• NAT network: Connect to the host network through NAT. This is the most common choice. In this configuration, the virtual machine can access to the external network, but the external network cannot connect to the virtual machine directly. For an example, if you set up a FTP service on the virtual machine, the computers on the LAN of the host cannot access this FTP service. If you want to publish this service, you have to use port forwarding setting to do this.

With these concepts in mind, if you want to use a dedicated machine as the PXE server, you can use bridged network in your environment. However, you must be very careful using this kind of setup. This is usually done by the IT group in your organization, since you cannot setup a DHCP server on the LAN without affecting others. We won't use this option here.

The host only network is actually a good choice for this case, because this kind of network is an isolated network configuration. The network connection only exists between the host and the virtual machine. The problem is we cannot access to the outside network. We will configure two network interfaces to our virtual machine instance one host only network for the PXE boot and one NAT network to access Internet. We will see this configuration later.

In VirtualBox, it also has a built-in PXE server in NAT network. With this option, we don't need to setup PXE server by ourselves.

We will explain how to set up our own PXE boot environment first and then explain how to use the built-in PXE server of VirtualBox.

As we can see in the following figure, we have two virtual machines pxeAndroid and PXE Server in our setup. The upper part PXE Server is optional. If we use the built-in PXE server, both PXE server and NFS server will be on the development host.

Let's look at how to set up our own PXE server first. To setup a PXE boot environment, we need to install a TFTP and DHCP server. I assume that you can set up a Linux virtual machine by yourself. I will use Ubuntu as an example here. In your environment, you have to create two virtual machines.

• A PXE server with a host only network interface
• A virtual machine to boot Android with a host only network interface and a NAT network interface

Setting up TFTP server

We can install tftp server in the PXE server using the following command:

$ sudo apt-get install tftpd-hpa  

After the tftp server is installed, we need to set up PXE boot configuration in the folder /var/lib/tftpboot. We can use the following command to start tftp server.

$ sudo service tftpd-hpa restart 

Configuring DHCP server

Once tftp server is installed, we need to install a DHCP server. We can install DHCP server using the following command.

$ sudo apt-get install isc-dhcp-server  

After install the DHCP server, we have to add the following lines into the DHCP server configuration file at /etc/dhcp/dhcpd.conf.

subnet 192.168.56.0 netmask 255.255.255.0 { 
        range 192.168.56.10 192.168.56.99; 
        filename "pxelinux.0"; 
}

We use the IP address range 192.168.56.x for the host only subnet, since this is the default range after we create a host-only network in VirtualBox. There may be more than one host only network configured in your VirtualBox environment. You may want to check the right host only network configuration that you want to use and set the above configuration file according to the host only network setup.

Configuring and testing the PXE boot

After we set up the PXE server, we can create a virtual machine instance to test the environment. We will demonstrate this using Ubuntu 14.04 as the host environment. The same setup can be duplicated to Windows or OS X environment as well. If you use a Windows environment, you have to set up the NFS server inside the PXE server. The Windows host cannot support NFS.

Setting up the Android Virtual Machine

Let's create a virtual machine called pxeAndroid in VirtualBox first. After start VirtualBox, we can click on  the button New to create a new virtual machine as shown in the following screenshot:

We call it pxeAndroid and choose Linux as the type of virtual machine. We can just follow the wizard to create this virtual machine with suitable configuration. After the virtual machine is created, we need to make a few changes to the settings.

The first one need to be changed is the network configuration as I mentioned before we need both NAT and host Only connections. We can click on the name of virtual machine pxeAndroid first and then click on the button Settings to change the settings. Select the option Network in the left-hand side, as we can see from the following screen:

We select the Adapter 1 and it is default to NAT network. We need to change the Adapter Type to Paravirtualized Network (virtio-net) since we will use the PXE ROM that we just built. The NAT network can connect to the outside network. It supports port forwarding so that we can access certain services in the virtual machine. The one that we need to set up here is the ADB service. We need to use ADB to debug the pxeAndroid device later. We can set up the port forwarding for ADB as follows:

Now, we can select Adapter 2 to set up a host-only network as the following figure:

We choose the adapter as Host-only Adapter and Adapter Type as Paravirtualized Network (virtio-net).

Next, we can click on the System option to set the boot order so the default boot order is to boot from the network interface as the following figure:

Configuring pxelinux.cfg

Before we can test the virtual machine we just setup, we need to specify in the configuration file to let the PXE boot to know where to find the kernel and ramdisk images.

The PXE boot process is something like this.

1. When the virtual machine pxeAndroid power on, the client will get the IP address through DHCP.
2. After the DHCP configuration is found, the configuration includes the standard information such as IP address, subnet mask, gateway, DNS, and so on. In addition, it also provides the location of TFTP server and the filename of a boot image. The name of boot image is usually called pxelinux.0 as we can see in the previous section when we set up DHCP server. The name of boot image is vmname.pxe for the built-in PXE boot environment. Where the vmname should be the name of virtual machine. For example, it is pxeAndroid.pxe for our virtual machine.
3. The client contacts TFTP server to obtain the boot image.
4. TFTP server sends the boot image (pxelinux.0 or vmname.pxe), and the client executes it.
5. By default, the boot image searches the pxelinux.cfg directory on TFTP server for boot configuration files.
6. The client downloads all the files it needs (kernel, ramdisk, or root filesystem) and then loads them.
7. The target machine pxeAndroid reboots.

In the above step 5, the boot image searches the boot configuration files in the following steps:

1. First, it searches for the boot configuration file that is named according to the MAC address represented in lower case hexadecimal digits with dash separators. For example, for the MAC address 08:00:27:90:99:7B, it searches for the file 08-00-27-90-99-7b.
2. Then, it searches for the configuration file using the IP address (of the machine that is being booted) in upper case hexadecimal digits. For example, for the IP address 192.168.56.100, it searches for the file C0A83864.
3. If that file is not found, it removes one hexadecimal digit from the end and tries again. However, if the search is still not successful, it finally looks for a file named default (in lower case).

For example, if the boot filename is /var/lib/tftpboot/pxelinux.0, the Ethernet MAC address is 08:00:27:90:99:7B, and the IP address is 192.168.56.100, the boot image looks for file names in the following order:

/var/lib/tftpboot/pxelinux.cfg/08-00-27-90-99-7b

/var/lib/tftpboot/pxelinux.cfg/C0A83864

/var/lib/tftpboot/pxelinux.cfg/C0A8386

/var/lib/tftpboot/pxelinux.cfg/C0A838

/var/lib/tftpboot/pxelinux.cfg/C0A83

/var/lib/tftpboot/pxelinux.cfg/C0A8

/var/lib/tftpboot/pxelinux.cfg/C0A

/var/lib/tftpboot/pxelinux.cfg/C0

/var/lib/tftpboot/pxelinux.cfg/C

/var/lib/tftpboot/pxelinux.cfg/default  

The boot image pxelinux.0 is part of an open source project syslinux. We can get the boot image and the menu user interface from Syslinux project using the following commands:

$ sudo apt-get install syslinux  

After Syslinux is installed, pxelinux.0 can be copied to the TFTP root folder as .

$ cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot/pxelinux.0  

To have a better user interface, we can copy menu.c32 to the TFTP folder as well.

$ cp /usr/lib/syslinux/menu.c32 /var/lib/tftpboot/menu.c32

pxelinux.cfg/default

Now, we will look at how to configure the boot configuration file pxelinux.cfg/default. In our setup, it looks like the following code snippet:

prompt 1 
default menu.c32 
timeout 100 
 
label 1. NFS Installation (serial port) - x86vbox 
menu x86vbox_install_serial 
kernel x86vbox/kernel 
append ip=dhcp console=ttyS3,115200 initrd=x86vbox/initrd.img root=/dev/nfs rw androidboot.hardware=x86vbox INSTALL=1 DEBUG=2 SRC=/x86vbox ROOT=192.168.56.1:/home/sgye/vol1/android-6/out/target/product qemu=1 qemu.gles=0 
 
label 2. x86vbox (ROOT=/dev/sda1, serial port) 
menu x86vbox_sda1 
kernel x86vbox/kernel 
append ip=dhcp console=ttyS3,115200 initrd=x86vbox/initrd.img androidboot.hardware=x86vbox DEBUG=2 SRC=/android-x86vbox ROOT=/dev/sda1 
... 

The syntax in the boot configuration file can be found at the following URL from Syslinux project:

http://www.syslinux.org/wiki/index.php?title=SYSLINUX

In the mentioned configuration file that we use, we can see the following commands and options:

• prompt: It will let the bootloader know whether it will show a LILO-style "boot:" prompt. With this command line prompt, you can input the option directly. All the boot options define by the command label.
• default: It defines the default boot option.
• timeout: If more than one label entry is available, this directive indicates how long to pause at the boot: prompt until booting automatically, in units of 1/10 s. The timeout is cancelled when any key is pressed, the assumption being the user will complete the command line. A timeout of zero will disable the timeout completely. The default is 0.
• label: A human-readable string that describes a kernel and options. The default label is linux, but you can change this with the DEFAULT keyword.
• kernel: The kernel file that the boot image will boot.
• append: The kernel command line, which can be passed to the kernel during the boot.

In this configuration file, we show two boot options. In the first option, we can boot to a minimum Linux environment using NFS root filesystem. We can install the x86vbox images from that environment to hard disk. The source location of installation is your AOSP build output folder. In the second option, we can boot x86vbox from disk partition /dev/sda1. After the x86vbox image is installed on the partition /dev/sda1, the Android system can be started using the second option.

Using VirtualBox internal PXE booting with NAT

VirtualBox provides a built-in support for PXE boot using NAT network. We can also set up PXE boot using this built-in facility. There are a few minor differences between the built-in PXE and the one that we set up in the PXE server.

• The built-in PXE uses the NAT network connection while the PXE server uses host only network connection.
• TFTP root is at /var/lib/tftpboot for the normal PXE setup while the built-in TFTP root is at $HOME/.VirtualBox/TFTP on Linux or %USERPROFILE%.VirtualBoxTFTP on Windows.
• Usually, the default boot image name is pxelinux.0, but it is vmname.pxe for the VirtualBox built-in PXE. For example, if we use pxeAndroid as virtual machine name, we have to make a copy of pxelinux.0 and name it pxeAndroid.pxe under the VirtualBox TFTP root folder.

If you choose to use the built-in PXE support, you don't have to create a PXE server by yourself. This is the recommended test environment to simplify the test process.

Setting up serial port for debugging

The reason that we want to boot Android using PXE and NFS is that we want to use a very simple bootloader and find an easier way to debug the system. In order to see the debug log, we want to redirect the debug output from the video console to a serial port so that we can separate the graphic user interface from the debug output. We need to do two things in order to meet our goals.

The Linux kernel debug message can be re-directed to a specific channel using kernel command-line arguments. We specify this in PXE boot configuration with option console=ttyS3,115200. This is defined in pxelinux.cfg/default as follows:

label 1. NFS Installation (serial port) - x86vbox 
menu x86vbox_install_serial 
kernel x86vbox/kernel 
append ip=dhcp console=ttyS3,115200 initrd=x86vbox/initrd.img root=/dev/nfs rw androidboot.hardware=x86vbox INSTALL=1 DEBUG=2 SRC=/x86vbox ROOT=192.168.56.1:/home/sgye/vol1/android-6/out/target/product qemu=1 qemu.gles=0 

We will explain the details about kernel parameters in the option append later. The next thing is that we need to create a virtual serial port so that we can connect to. We configure this in the virtual machine settings page as shown in the following screen:

We use a host pipe to simulate the virtual serial port. We can set the path as something like /tmp/pxeAndroid_p.

The mapping between COMx to /dev/ttySx can be found here:

/dev/ttyS0 - COM1 
/dev/ttyS1 - COM2 
/dev/ttyS2 - COM3 
/dev/ttyS3 - COM4 

To connect to the host pipe, we can use a tool like minicom in Linux or putty in Windows. If you don't have minicom installed, you can install and configure minicom as shown in the host environment:

$ sudo apt-get install minicom  

To setup minicom, we can use the following command:

$ sudo minicom -s  

After minicom start, select Serial port setup, and set Serial Device as unix#/tmp/pxeAndroid_p. Once this is done, select Save setup as dfl and Exit from minicom as shown in the following screenshot. Now, we can connect to the virtual serial port using minicom:

After we made all the changes for the configuration, we can power on the virtual machine and test it. We should be able to see the following boot up screen:

We can see from the preceding screenshot that the virtual machine loads the file pxelinux.cfg/default and wait on the boot prompt. We are ready to boot from PXE ROM now.

Build AOSP images

To build the x86vbox images in this article, we can retrieve the source code using the following commands:

$ repo init https://github.com/shugaoye/manifests -b android-7.1.1_r4_ch14_aosp
$ repo sync  

After the source code is ready for use, we can set the environment and build the system as shown here:

$ . build/envsetup.sh
$ lunch x86vbox-eng
$ make -j4  

To build initrd.img, we can run the following command.

$ make initrd USE_SQUASHFS=0  

We can also build an OTA update image which can use recovery to install it.

$ cd device/generic/x86vbox
$ make dist  

NFS filesystem

Since I am discussing about Android system programming, I will assume you know how to build Android images from AOSP source code. In our setup, we will use the output from the AOSP build to boot the Android system in VirtualBox. They are not able to be used by VirtualBox directly. For example, the system.img can be used by emulator, but not VirtualBox. VirtualBox can use the standard virtual disk images in VDI, VHD, or VMDK formats, but not the raw disk image as system.img.

In some open source projects, such as the android-x86 project, the output is an installation image, such as ISO or USB disk image formats. With an installation image, it can be burnt to CD ROM or USB drive. Then, we can boot VirtualBox from CD ROM or USB to install the system just like how we install Windows on our PC. It is quite tedious and not efficient to use this method, when we are debugging a system. As a developer, we want a simple and quick way that we can start the debugging immediately after we build the system.

The method that we will use here is to boot the system using NFS filesystem. The key point is that we will treat the output folder of AOSP build as the root filesystem directly so that we can boot the system using it without any additional work.

If you are an embedded system developer, you may be used this method in your work already. When we work on the initial debugging phase of an embedded Linux system, we often use NFS filesystem as a root filesystem. With this method, we can avoid to flash the images to the flash storage every time after the build.

Preparing the kernel

To support NFS boot, we need a Linux kernel with NFS filesystem support. The default Linux kernel for Android doesn't have NFS boot support. In order to boot Android and mount NFS directory as root filesystem, we have to re-compile Linux kernel with the following options enabled:

CONFIG_IP_PNP=y 
CONFIG_IP_PNP_DHCP=y 
CONFIG_IP_PNP_BOOTP=y 
CONFIG_IP_PNP_RARP=y 
CONFIG_USB_USBNET=y 
CONFIG_USB_NET_SMSC95XX=y 
CONFIG_USB=y 
CONFIG_USB_SUPPORT=y 
CONFIG_USB_ARCH_HAS_EHCI=y 
CONFIG_NETWORK_FILESYSTEMS=y 
CONFIG_NFS_FS=y 
CONFIG_NFS_V3=y 
CONFIG_NFS_V3_ACL=y 
CONFIG_ROOT_NFS=y 

The kernel source code used in this article is a modified version by me for the book Android System Programming. You can find the source code at the following URL:

https://github.com/shugaoye/goldfish

We can get the source code using the following command:

$ git clone https://github.com/shugaoye/goldfish -b android-7.1.1_r4_x86vbox_ch14_r

We can use menuconfig to change the kernel configuration or copy a configuration file with NFS support.

To configure kernel build using menuconfig, we can use the following commands:

$ . build/envsetup.sh
$ lunch x86vbox-eng
$ make -C kernel O=$OUT/obj/kernel ARCH=x86 menuconfig  

We can also use the configuration file with NFS enable from my GitHub directly. We can observe the difference between this configuration file and the default kernel configuration file from android-x86 project as shown here:

$ diff kernel/arch/x86/configs/android-x86_defconfig ~/src/android-x86_nfs_defconfig
    216a217
    > # CONFIG_SYSTEM_TRUSTED_KEYRING is not set
    1083a1085
    > CONFIG_DNS_RESOLVER=y
    1836c1838
    < CONFIG_VIRTIO_NET=m
    ---
    > CONFIG_VIRTIO_NET=y
    1959c1961
    < CONFIG_E1000=m
    ---
    > CONFIG_E1000=y
    5816a5819
    > # CONFIG_ECRYPT_FS is not set
    5854,5856c5857,5859
    < CONFIG_NFS_FS=m
    < CONFIG_NFS_V2=m
    < CONFIG_NFS_V3=m
    ---
    > CONFIG_NFS_FS=y
    > CONFIG_NFS_V2=y
    > CONFIG_NFS_V3=y
    5858c5861
    < # CONFIG_NFS_V4 is not set
    ---
    > CONFIG_NFS_V4=y
    5859a5863,5872
    > CONFIG_NFS_V4_1=y
    > CONFIG_NFS_V4_2=y
    > CONFIG_PNFS_FILE_LAYOUT=y
    > CONFIG_PNFS_BLOCK=y
    > CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org"
    > # CONFIG_NFS_V4_1_MIGRATION is not set
    > CONFIG_NFS_V4_SECURITY_LABEL=y
    > CONFIG_ROOT_NFS=y

    > # CONFIG_NFS_USE_LEGACY_DNS is not set
    > CONFIG_NFS_USE_KERNEL_DNS=y
    5861,5862c5874,5875
    < CONFIG_GRACE_PERIOD=m
    < CONFIG_LOCKD=m
    ---
    > CONFIG_GRACE_PERIOD=y
    > CONFIG_LOCKD=y
    5865c5878,5880
    < CONFIG_SUNRPC=m
    ---
    > CONFIG_SUNRPC=y
    > CONFIG_SUNRPC_GSS=y
    > CONFIG_SUNRPC_BACKCHANNEL=y
    5870a5886
    > # CONFIG_CIFS_UPCALL is not set
    5873a5890
    > # CONFIG_CIFS_DFS_UPCALL is not set
    6132c6149,6153
    < # CONFIG_KEYS is not set
    ---
    > CONFIG_KEYS=y
    > # CONFIG_PERSISTENT_KEYRINGS is not set
    > # CONFIG_BIG_KEYS is not set
    > # CONFIG_ENCRYPTED_KEYS is not set
    > # CONFIG_KEYS_DEBUG_PROC_KEYS is not set
    6142a6164
    > # CONFIG_INTEGRITY_SIGNATURE is not set
    6270a6293
    > # CONFIG_ASYMMETRIC_KEY_TYPE is not set
    6339a6363
    > CONFIG_ASSOCIATIVE_ARRAY=y
    6352a6377
    > CONFIG_OID_REGISTRY=y

We can copy this configuration file and use it to build Linux kernel as shown here:

$ cp ~/src/android-x86_nfs_defconfig out/target/product/x86/obj/kernel/.config
$ . build/envsetup.sh
$ lunch x86vbox-eng
$ make -C kernel O=$OUT/obj/kernel ARCH=x86 

After the build, we can copy the kernel and ramdisk files to the TFTP root at /var/lib/tftpboot/x86vbox or $HOME/.VirtualBox/TFTP/x86vbox.

Setting up NFS server

After we prepare the Android kernel, we need to setup a NFS server on our development host so that we can mount to the NFS folders exported by our NFS server. We can check whether the NFS server is already installed or not using the following command:

$ dpkg -l | grep nfs  

If the NFS server is not installed, we can install it using the following command:

$ sudo apt-get install nfs-kernel-server  

Once we have a NFS server ready, we need to export our root filesystem through NFS. We will use the AOSP build output folder as we mentioned previously. We can add the following line to the configuration file /etc/exports.

$AOSP/out/target/product/ *(rw,sync,insecure,no_subtree_check,async) 

After that, we execute the following command to export the folder $AOSP/out/target/product. You need to replace $AOSP to the absolute path in your setup.

$ sudo exportfs -a 

Configuring PXE boot menu

We can use PXE boot ROM to support the boot path like a real Android device. As we know that Android device can boot to three different modes, they are the bootloader mode, the recovery mode and the normal start up.

With PXE boot ROM, we can easily support the same and more. By configuring the file pxelinux.cfg/default, we can allow x86vbox to boot in different paths. We will configure multiple boot paths here.

Booting to NFS installation

We can boot the system to an installation mode so that we can borrow the installation script from android-x86 project to install x86vbox images to the virtual hard disk.

label 1. NFS Installation (serial port) - x86vbox 
menu x86vbox_install_serial 
kernel x86vbox/kernel 
append ip=dhcp console=ttyS3,115200 initrd=x86vbox/initrd.img root=/dev/nfs rw androidboot.hardware=x86vbox INSTALL=1 DEBUG=2 SRC=/x86vbox ROOT=192.168.56.1:$AOSP/out/target/product 

In this configuration, we use the NFS-capable kernel from TFTP folder, such as $HOME/.VirtualBox/TFTP/x86vbox/kernel. The ramdisk image initrd.img is also stored in the same folder. Both files under TFTP folder can actually be the symbol links to the AOSP output. In this case, we don't have to copy them after the build.

We use the following three options to configure the NFS boot.

• ip=dhcp: Use DHCP to get IP address from DHCP server. The DHCP server can be the built-in DHCP server of VirtualBox or the one that we set up previously.
• root=/dev/nfs: Use NFS boot.
• ROOT=10.0.2.2:$AOSP/out/target/product: The root is the AOSP output folder in the development host. If we use the built-in PXE, the IP address 10.0.2.2 is the default host IP address in the NAT network. It could be changed using the VirtualBox configuration.

We want to monitor the debug output so we set the console to the virtual serial port that we configured previously as console=ttyS3,115200. We can use a host pipe to connect to it using minicom.

We set three kernel parameters using by the android-x86 init script and installation script.

• INSTALL=1: Tells the init script that we want to install the system.
• DEBUG=2: This will bring us to the debug console during the boot process.
• SRC=/x86vbox: This is the directory for the android root filesystem.

Finally, the option androidboot.hardware=x86vbox is passed to the Android init process to tell which init script to run. In this case, the device init script init.x86vbox.rc will be executed.

In our PXE boot menu, we can add another configuration for the installation without option console=ttyS3,115200. In this case, all debug output will print on the screen which is the default standard output.

Booting to hard disk

We can have another option as shown to boot the system from hard disk after we install the system using the previous configuration.

label 2. x86vbox (ROOT=/dev/sda1, serial port) 
menu x86vbox_sda1 
kernel x86vbox/kernel 
append ip=dhcp console=ttyS3,115200 initrd=x86vbox/initrd.img androidboot.hardware=x86vbox DEBUG=2 SRC=/android-x86vbox ROOT=/dev/sda1 

In the preceding configuration, we use device /dev/sda1 as root and we don't have the option INSTALL=1. With this configuration, the virtual machine will boot to Android system from hard disk /dev/sda1 and the debug output will print to virtual serial port.

We can configure another similar configuration which prints the debug output to the screen.

Booting to recovery

With PXE boot menu, we can configure the system to boot to recovery as well. We can see the following configuration:

label 5. x86vbox recovery (ROOT=/dev/sda2) 
menu x86vbox_recovery 
kernel x86vbox/kernel 
append ip=dhcp console=ttyS3,115200 initrd=x86vbox/ramdisk-recovery.img androidboot.hardware=x86vbox DEBUG=2 SRC=/android-x86vbox ROOT=/dev/sda2 

Note the difference here is that we use recovery ramdisk instead of initrd.img. Since recovery is a self-contained environment, we can set variable ROOT to another partition as well. We can use recovery to install an OTA update image. With PXE boot, you can explore many different possibilities to play with various boot methods and images.

With all this setup, we can boot to PXE boot menu as the following screenshot:

We can select an option from the PXE boot menu above to boot to a debug console as shown here:

From the preceding debug output, we can see that the virtual machine obtains the IP address 10.0.2.15 from DHCP server 10.0.2.2. The NFS root is found at IP address 192.168.56.1, which is the development host. It uses a different IP address range is because we use two network interfaces in our configuration. We use a NAT network interface which has the IP address range in 10.0.2.x and a host-only network interface which has the IP address range in 192.168.56.x. The IP address 10.0.2.2 is the IP address of the development host in NAT network while IP address 192.168.56.1 is the IP address of the development host in host only network.

In this setup, we use the VirtualBox built-in PXE support so both DHCP and TFTP server are on the NAT network interface. If we use a separate PXE server, both DHCP and TFTP server will be on the host only network interface.

Summary

In this article, you learned a debugging method with the combination of PXE boot and NFS root filesystem. This is a common practice in the embedded Linux development world. We try to use the similar setup for the Android system development. As we can see this setup can make the development and debugging process more efficiently. We can use this setup to remove the dependency of bootloader. We can also reduce the time to flash or provision the build images to the device. Even though we did all the exploration in VirtualBox, you can reuse the same method in your hardware board development as well.

Resources for Article:

Further resources on this subject:

• Setting up Development Environment for Android Wear Applications [article]
• Practical How-To Recipes for Android [article]
• Optimizing Games for Android [article]