Testing Spark applications
There are many ways to try to test your Spark code, depending on whether it's Java (you can do basic JUnit tests to test non-Spark pieces) or ScalaTest for your Scala code. You can also do full integration tests by running Spark locally or on a small test cluster. Another awesome choice from Holden Karau is using Spark-testing base. You probably know that there is no native library for unit testing in Spark as of yet. Nevertheless, we can have the following two alternatives to use two libraries:
- ScalaTest
- Spark-testing base
However, before starting to test your Spark applications written in Scala, some background knowledge about unit testing and testing Scala methods is a mandate.
Testing Scala methods
Here, we will see some simple techniques for testing Scala methods. For Scala users, this is the most familiar unit testing framework (you can also use it for testing Java code and soon for JavaScript). ScalaTest supports a number of different testing styles, each designed...
