Go to http://opencv.org/downloads.html and click on the download link for the latest version of OpenCV for iOS. Specifically, we require OpenCV 3.1 or a later version. The download's filename is opencv2.framework.zip. Unzip it to get the framework file, opencv2.framework. Later, we will add this framework to our iOS application projects; we will import its header files using the following code:

#import <opencv2/core.hpp>

This imports the core module's header file from opencv2.framework. The import statement will vary according to the module's name.

We will try to get and build all of OpenCV's modules. Broadly, this process will consist of the following four steps:

Now, let's discuss the details as we walk through the steps. To obtain OpenCV's latest source code, we can use Git, an open source version control tool. We already installed Git as part of the Xcode Command Line Tools. OpenCV's standard and extra modules are hosted in two repositories on GitHub, an online repository hosting service. To download the standard modules' source code to <opencv_source_path>, run the following command:

$ git clone https://github.com/Itseez/opencv.git <opencv_source_path>

Similarly, to download the extra modules' source code to <opencv_contrib_source_path>, run the following command:

$ git clone https://github.com/Itseez/opencv_contrib.git <opencv_contrib_source_path>

OpenCV's source code comes with build scripts for various platforms. The iOS build script takes two arguments—the build path and the opencv_contrib source path. Run the script in the following manner:

$ ./<opencv_source_path>/platforms/ios/build_framework.py <opencv_contrib_build_path> --contrib <opencv_contrib_source_path>

Read the script's output to see whether it failed to build any modules. Remember that opencv_contrib contains experimental modules from various authors, and some authors might not test their modules for iOS compatibility. For example, the following output shows a compilation error in the saliency module (modules/saliency):

** BUILD FAILED **


The following build commands failed:
  CompileC /Users/Joe/SDKs/OpenCV/fork_build_ios/build/iPhoneOS-armv7/modules/saliency/OpenCV.build/Release-iphoneos/opencv_saliency_object.build/Objects-normal/armv7/FilterTIG.o /Users/Joe/SDKs/OpenCV/fork_contrib/modules/saliency/src/BING/FilterTIG.cpp normal armv7 c++ com.apple.compilers.llvm.clang.1_0.compiler
(1 failure)
('Child returned:', 65)

If we do not require the problematic module, we may simply delete its source subfolder in <opencv_contrib_source_path>/modules, and then rerun build_framework.py. For example, to avoid building the saliency module, we may delete <opencv_contrib_source_path>/modules/saliency.

On the other hand, if we do require the problematic module, first somebody must modify its source code so that it successfully compiles and runs for iOS. Patching opencv_contrib is beyond the scope of this book, but if you are skilled in C++ programming, I encourage you to try it sometime. Alternatively, you may decide to file an issue report at https://github.com/Itseez/opencv_contrib/issues and wait for the module's authors to respond.

When build_framework.py works properly, it prints ** INSTALL SUCCEEDED **, and creates the framework file at <opencv_contrib_build_path>/opencv2.framework. Later, we will add this framework to our iOS application projects; we will import its header files using the following code:

#import <opencv2/xphoto.hpp>

This imports the xphoto module's header file from opencv2.framework. The import statement will vary according to the module's name.

As the extra modules are less stable than the standard modules, we may want to make them optional in our code. By enclosing the optional code inside a preprocessor condition, we can easily disable or re-enable it in order to test the effect. Consider the following example:

#ifdef WITH_OPENCV_CONTRIB
#import <opencv2/xphoto.hpp>
#endif

If we want to use opencv2_contrib, we will edit the Xcode project settings to add WITH_OPENCV_CONTRIB as a preprocessor definition. Then, in the preceding example, the xphoto.hpp headers will be imported in our code. Detailed steps to create a preprocessor definition are provided later in this chapter, in the Configuring the project section.

Open Xcode. Click on the Create new Xcode project button or select the File | New | Project… menu item. Now, a dialog asks you to choose a project template. Select iOS | Application | Single View Application, as shown in the following screenshot:

Single View Application is the simplest template as it just creates an empty GUI with no special navigational structure. Click on the Next button to confirm the selection. Now, a dialog asks you to pick a few project settings. Fill out the form as shown in the following screenshot:

Let's review the items in the form:

Click on the Next button to confirm the project options. Now, a file chooser dialog asks you to pick a folder for the project. Pick any location, which we will refer to as <app_project_path>.

Optionally, you may enable the Create Git repository checkbox if you want to put the project under version control using Git. Click on the Create button. Now, Xcode creates and opens the project.

Use Finder or Terminal to copy files to the following locations:

Go back to Xcode to view the project. Navigate to the File | Add Files to "CoolPig"… menu item. Now, Xcode opens a file chooser dialog. Select opencv2.framework and click on the Add button. Repeat the same steps for CoolPig/Piggy.png. Note that these files appear in the project navigator pane, which is the leftmost section of the Xcode window. In this pane, drag Piggy.png to the CoolPig | Supporting Files group. When you are finished, the navigator pane should look similar to the following screenshot:

First, let's configure our app to run in fullscreen mode with no status bar. Select the CoolPig project file at the top of the navigator pane. Now, select the General tab in the editor area, which is the central part of the Xcode window. Find the Deployment Info group, and enable the Hide status bar and Requires full screen checkboxes, as shown in the following screenshot:

The status bar and fullscreen settings are stored in the app's Info.plist file. Select CoolPig | CoolPig | Info.plist in the navigator pane. Now, in the editor area, note that the UIRequiresFullscreen and Status bar is initially hidden properties both have the YES value. However, we still need to add another property to ensure that the status bar will not appear. Hover over the last item in the list, and click on the + button to insert a new property. Enter View controller-based status bar appearance as the property's key and set its value to NO, as shown in the following screenshot:

Next, let's link the project with additional frameworks. OpenCV depends on two of Apple's frameworks called CoreGraphics.framework and UIKit.framework. Optionally, for optimizations, OpenCV can also use a third Apple framework called Accelerate.framework.

Select the CoolPig project file in the navigator pane and then select the Build Phases tab in the editor area. Find the Link Binary With Libraries group. Click on the + button, select Accelerate.framework from the dialog, and click on the Add button. Repeat these steps for CoreGraphics.framework and UIKit.framework. Now, the editor area should look similar to the following screenshot:

Now, the linker will be able to find OpenCV's dependencies. However, we need to change another setting to ensure that the compiler will understand the C++ code in OpenCV's header files. Open the Build Settings tab in the editor area and find the Apple LLVM 7.0 - Language group. Set the value of the Compile Sources As item to Objective-C++, as seen in the following screenshot:

We have just one more thing to configure in the Build Settings tab. Remember that we consider the opencv2_contrib modules to be an optional dependency of our projects, as described earlier in the Making the extra modules optional in our code section. If we did build opencv2.framework with these modules and if we do want to use their functionality, let's create a preprocessor definition, WITH_OPENCV_CONTRIB. Find the Apple LLVM 7.0 - Preprocessing group. Edit Preprocessor Macros | Debug and Preprocessor Macros | Release to add the WITH_OPENCV_CONTRIB text. Now, the settings should look like the following screenshot:

As a final, optional step in the configuration, you may want to set the app's icon. Select CoolPig | CoolPig | Assets.xcassets in the project navigator pane. Assets.xcassets is a bundle, which may contain several variants of the icon for different devices and different contexts (the Home screen, Spotlight searches, and the Settings menu).

Click on the AppIcon list item in the editor area and then drag and drop an image file into each square of the AppIcon grid. If the image's size is incorrect, Xcode will notify you so that you may resize the image and try again. Once you have added your images, the editor area might look similar to the following screenshot:

Now, our project is fully configured and we are ready to design its graphical user interface (GUI). Xcode comes with a built-in tool called Interface Builder, which enables us to arrange GUI elements, connect them to variables and events in our code, and even define the transitions between scenes (or informally, screens). Remember that CoolPig's GUI is just a fullscreen image. However, even our simple GUI has a transition between a static loading screen (where the image does not change color) and dynamic main screen (where the image changes color every two seconds). Let's first configure the loading screen and then the main screen.

Select CoolPig | CoolPig | LaunchScreen.storyboard in the navigator pane. This file is a storyboard, which stores the configuration of a set of scenes (or a single scene in this case). A scene hierarchy appears in the editor area. Navigate to View Controller Scene | View Controller | View. A blank view appears on the right-hand side of the editor area, as seen in the following screenshot:

Let's add an image view inside the empty view. Notice the list of available GUI widgets in the lower-right corner of the Xcode window. This area is called the library pane. Scroll through the library pane's contents. Find the Image View item and drag it to the empty view. Now, the editor area should look like this:

Drag the corners of the highlighted rectangle to make the image view fill its parent view. The result should look like this:

We still need to take a further step to ensure that the image view scales up or down to match the screen size on all devices. Click on the Pin button in the toolbar at the bottom of the editor area. The button's icon looks like a rectangle pinned between two lines. Now, a pop-up menu appears with the title Add New Constraints. Constraints define a widget's position and size relative to other widgets.

Specifically, we want to define the image view's margins relative to its parent view. To define a margin on every side, click on the four I-shaped lines that surround the square. They turn red. Now, enter 0 for the top and bottom values and -20 for the left and right values. Some iOS devices have built-in horizontal margins, and our negative values ensure that the image extends to the screen's edge even on these devices. The following screenshot shows the settings:

Click on the Add 4 Constraints button to confirm these parameters.

Finally, we want to show an image! Look at the inspector pane, which is in the top-right area of the Xcode window. Here, we can configure the currently selected widget. Select the Attributes tab. Its icon looks like a slider. From the Image drop-down list, select Piggy.png. From the Mode drop-down list, select Aspect Fill. This mode ensures that the image will fill the image view in both dimensions, without appearing stretched. The image may appear cropped in one dimension. Now, the editor area and inspector pane should look similar to the following screenshot:

So far, we have completed the loading screen's layout. Now, let's turn our attention to the main screen. Select CoolPig | CoolPig | Main.storyboard in the project navigator. This storyboard, too, has a single scene. Select its view. Add an image view and configure it in exactly the same way as the loading screen's image view. Later, in the Connecting an interface element to the code section, we will connect this new image view to a variable in our code.

As part of the Single View Application project template, Xcode has already created the following code files for us:

For CoolPig, we simply need to modify ViewController.m. Select CoolPig | CoolPig | ViewController.m in the project navigator. The code appears in the editor area. At the beginning of the code, let's add more #import statements to include the header files for several OpenCV modules, as seen in the following code:

#import <opencv2/core.hpp>
#import <opencv2/imgcodecs/ios.h>
#import <opencv2/imgproc.hpp>

#ifdef WITH_OPENCV_CONTRIB
#import <opencv2/xphoto.hpp>
#endif

#import "ViewController.h"

We will need to generate random numbers to create the image's random tint. For convenience, let's define the following macro, which generates a 64-bit floating-point number in the range of 0 to 1:

#define RAND_0_1() ((double)arc4random() / 0x100000000)

The remainder of ViewController.m deals with the private interface and implementation of the ViewController class. Elsewhere, in ViewController.h, the class is declared as follows:

@interface ViewController : UIViewController
@end

Note that ViewController is a subclass of UIViewController, which is an important class in the iOS SDK. UIViewController manages the life cycle of a set of views and provides reasonable default behaviors as well as many methods that may override these defaults. If we develop applications according to the model-view-controller (MVC) pattern, then UIViewController is the controller or coordinator, which enforces good separation between the platform-specific view or GUI and platform-independent model or "business logic".

Let's turn our attention back to the private interface of ViewController in ViewController.m. The class keeps the original image and updated image as member variables. They are instances of OpenCV's cv::Mat class, which can represent any kind of image or other multidimensional data. ViewController also has a reference to the image view where we will display the image. Another of the class's properties is an NSTimer object, which will fire a callback every two seconds. Finally, the class has a method, updateImage, which will be responsible for displaying a new random variation of the image. Here is the code for ViewController's private interface:

@interface ViewController () {
  cv::Mat originalMat;
  cv::Mat updatedMat;
}

@property IBOutlet UIImageView *imageView;
@property NSTimer *timer;

- (void)updateImage;

@end

Now, let's implement the methods of the ViewController class. It inherits many methods from its parent class, UIViewController, and we could override any of these. First, we want to override the viewDidLoad method, which runs when the scene is loaded from its storyboard. Typically, this is an appropriate time to initialize the view controller's member variables. Our implementation of viewDidLoad will begin by loading Piggy.png from file and converting it to OpenCV's RGB format. If the image was not originally grayscale and OpenCV's extra photo module is available, we will use a function from this module to adjust the white balance. Finally, we will start a timer to invoke our updateImage method every two seconds. Here is our code for viewDidLoad:

@implementation ViewController

- (void)viewDidLoad {
  [super viewDidLoad];
  
  // Load a UIImage from a resource file.
  UIImage *originalImage =
      [UIImage imageNamed:@"Piggy.png"];
  
  // Convert the UIImage to a cv::Mat.
  UIImageToMat(originalImage, originalMat);
  
  switch (originalMat.type()) {
    case CV_8UC1:
      // The cv::Mat is in grayscale format.
      // Convert it to RGB format.
      cv::cvtColor(originalMat, originalMat,
          cv::COLOR_GRAY2RGB);
      break;
    case CV_8UC4:
      // The cv::Mat is in RGBA format.
      // Convert it to RGB format.
      cv::cvtColor(originalMat, originalMat,
          cv::COLOR_RGBA2RGB);
#ifdef WITH_OPENCV_CONTRIB
      // Adjust the white balance.
      cv::xphoto::autowbGrayworld(originalMat,
          originalMat);
#endif
      break;
    case CV_8UC3:
      // The cv::Mat is in RGB format.
#ifdef WITH_OPENCV_CONTRIB
      // Adjust the white balance.
      cv::xphoto::autowbGrayworld(originalMat, originalMat);
#endif
      break;
    default:
      break;
  }
  
  // Call an update method every 2 seconds.
  self.timer = [NSTimer scheduledTimerWithTimeInterval:2.0
      target:self selector:@selector(updateImage)
      userInfo:nil repeats:YES];
}

Now, let's implement the updateImage helper method. It will multiply each color channel by a random floating-point number. The following table describes the effects of multiplying various channels by a coefficient, k:

The following code generates the random color, multiplies it together with the original image, and displays the result in the image view:

- (void)updateImage {
  // Generate a random color.
  double r = 0.5 + RAND_0_1() * 1.0;
  double g = 0.6 + RAND_0_1() * 0.8;
  double b = 0.4 + RAND_0_1() * 1.2;
  cv::Scalar randomColor(r, g, b);
  
  // Create an updated, tinted cv::Mat by multiplying the
  // original cv::Mat and the random color.
  cv::multiply(originalMat, randomColor, updatedMat);
  
  // Convert the updated cv::Mat to a UIImage and display
  // it in the UIImageView.
  self.imageView.image = MatToUIImage(updatedMat);
}

@end

We have implemented all the custom logic of CoolPig in just 50 lines of code! The project template, storyboard, iOS SDK, and OpenCV provide many useful abstractions and thus enable us to focus on writing concise, application-specific code.

Let's connect the image view in Main.Storyboard to the imageView property in ViewController.m. Open Main.Storyboard in the project navigator, hold command and click on View Controller in the scene hierarchy. A dialog with a dark background appears. Right-click on the Piggy.png image view in the scene hierarchy and drag it to the circle beside Outlets | imageView in the dark dialog box, as shown in the following screenshot:

Release the mouse button to complete the connection. Close the dark dialog box.

We are ready to build the app and run it in an iOS simulator or on an iOS device. First, if you want to use an iOS device, connect it to the Mac via a USB cable. The first time you connect a device, Xcode's top toolbar might show a progress bar and message, Processing symbol files. Wait for the message to disappear. Now, click on the CoolPig drop-down menu in Xcode's top toolbar and select the device or simulator that you want to use, such as Devices | Joseph's iPad or iOS Simulators | iPad Pro. Click on the Run button. Its icon is the standard triangular play symbol. Xcode builds the app, copies it to the device or simulator, and then launches it. Watch the pig change colors! For example, the app might look like this on an iPad Mini device:

Congratulations! We have built and run our first iOS application, including OpenCV for image processing and a pig for artistic reasons.

Many iOS devices have a built-in flash, which consists of a white LED light on the back of the device. Camera apps may activate the flash during photo capture, especially if the scene is dimly lit. Other apps may activate the flash for a long duration so that it acts as a flashlight or torch to help the user see. With only a single LED, the built-in flash may provide insufficient or uneven illumination in some circumstances.

If you need stronger or more evenly distributed illumination, or if your iOS device lacks a built-in flash altogether, you may want to purchase an external flash. Depending on the design, the external flash may mount as part of a case or may plug into the iOS device's audio jack. Typically, the external flash will have multiple white LEDs arranged in a line, grid, or ring. The latter design is called a ring flash.

Alternatively, in a controlled environment, you may set up any kind of lighting anywhere you please and you do not need to rely on the iOS device as a power source. Even a pair of well-placed desk lamps can greatly enhance the clarity and beauty of a scene. Normally, it is best to illuminate the subject from multiple angles to prevent shadows. Do not shine a light directly into the camera and do not illuminate the background more brightly than the subject, as these types of lighting tend to give the subject a very murky appearance with low contrast in the foreground. Sometimes, murky light can be artistically interesting, but it is not good for computer vision.

A conventional photo camera has a threaded mount where the user may screw in the head of a tripod. Of course, an iOS device has no threaded mount. If we want to use a tripod with a standard screw, we may purchase an adapter that consists of a threaded mount and clip to hold the iOS device. Alternatively, we may purchase a tripod that has a built-in clip instead of a standard screw. Regardless of the type of mount, we also need to consider the following characteristics of the tripod:

Typically, a small, lightweight tripod might cost between $10 and $30. This kind is often marketed as a mini or travel tripod. A tripod is a useful but optional accessory for all chapters in this book.

If you do not have a tripod or there is nowhere to place it, you may want to experiment with makeshift forms of stabilization. For example, if you want to monitor a room or hallway, you can tape the iOS device to a wall or ceiling. Be careful to choose tape that you can remove cleanly, without damaging the device's screen.

A lens attachment or add-on lens is an additional optical unit that sits in front of the iPhone or iPad's built-in lens. Typically, the attachment is designed for the rear camera and its mount may consist of a magnet, clip, or case. The types of add-on lenses include the following:

Typically, a lens attachment might cost between $20 and $50. The sharpness of the optics can vary greatly, so try to compare reviews before you choose a product. A fisheye attachment could be a fun accessory for our photographic work in Chapter 2, Capturing, Storing, and Sharing Photos. A macro, close-up, or microscope attachment could be useful for our work with small objects in Chapter 5, Classifying Coins and Commodities. Generally, you can experiment with any lens attachment in any chapter's project.