Working with Cloudera Spark images

Cloudera is a company that was founded in 2008 by ex-employees of Google, Yahoo!, Oracle, and Facebook. It was an early adopter of open source technologies like Apache Hadoop when it was still fresh from the oven; as a matter of a fact, the author of Hadoop itself joined the company shortly thereafter. Today, Cloudera sells licenses for a broad array of open source products, mostly from the Apache Software Foundation, and also provides consulting services.

In this recipe, we will look at a free virtual image from Cloudera that we can use to learn how to use the newest technologies supported by the company. 

Getting ready

To go through this recipe, you will need a working installation of a VirtualBox, a free virtualization tool from Oracle.

Here are the instructions for installing VirtualBox:
On Windows: https://www.htpcbeginner.com/install-virtualbox-on-windows/ 
On Linux: https://www.packtpub.com/books/content/installing-virtualbox-linux
On Mac: https://www.youtube.com/watch?v=lEvM-No4eQo.

To run the VMs, you will need:

• A 64-bit host; Windows 10, macOS, and most of the Linux distributions are 64-bit systems
• A minimum 4 GB of RAM dedicated for the VM, thus a system with a minimum of 8 GB of RAM is required

No other prerequisites are required.

How to do it...

To begin with, in order to download the Cloudera QuickStart VM:

1. Go to https://www.cloudera.com/downloads/quickstart_vms/5-12.html.
2. Select VirtualBox as your platform from the dropdown on the right, and click on the Get it now button.
3. A window to register will show up; fill it in as appropriate and follow the instructions on the screen:

Note, that it is a 6 GB+ download, so it may take a while. 

4. Once downloaded, open the VirtualBox.
5. Go to File | Import appliance, click on the button next to the path selection, and find the .ovf file (it should be accompanied by the .vmdk file, which is appropriate for the version you just downloaded).

On macOS, the image is automatically decompressed upon downloading. On Windows and Linux, you might need to unzip the archive first.

You should see a progress bar that is similar to this one:

Once imported, you should see a window like this:

6. If you now click on Start, you should see a new window pop up, and Cloudera VM (that is built on CentOS) should start booting up. Once done, a window similar to the following one should show up on your screen:

How it works...

There is really not much to configure: Cloudera QuickStart VM has everything you need to get going. As a matter of fact, this is a much simpler solution for Windows users than installing all the necessary environments. However, at the time of writing this book, it only comes with Spark 1.6.0:

Nothing, however, can stop you from upgrading to Spark 2.3.1 by following either the Installing Spark from sources or Installing Spark from binaries recipes we presented earlier in this book.