Integration Tests

Goals

Distinguish between unit tests and integration tests.
Configure the Maven Failsafe plugin for integration testing.
Use REST Assured for testing a RESTful API on a deployed server.

Concepts

integrate
integration testing
system under test (SUT)

Library

Dependencies

io.rest-assured:rest-assured:rest-assured (scope: test)
org.apache.maven.plugins:maven-failsafe-plugin:2.20.1 (plugin)

Preview

Preparation

Lesson

You've been creating unit tests for quite a while. Unit tests are essential for testing an API implementation, independent of the other components in the system. The unit tests are grouped by class, as this is the fundamental unit of organization in an object-oriented program, but the individual tests operate on the contracts of the individual methods of the class, guaranteeing that each method adheres to its API contract.

Unit tests are effective when they cover as many of the possible code paths as possible, from the “happy path” to edge cases, ensuring high code coverage. Unit tests should be ran as often as possible, preferable with continuous integration. This is why unit tests must need to be fast as possible, to prevent holding up development while unit tests execute.

Integration Testing

But as effective as unit tests are, and as well-defined as an API may be, it is hard to guarantee that one unit will integrate with other units in a system. It is one thing to test a car's engine extensively against its specification, as well as put the wheel design through a rigorous series of quality assurance checks. But once the engine, the wheels, and the other parts are all assembled into a vehicle, additional tests need to be performed to show that the parts interact the way they intended—that the car or truck can perform the basic operations required by that kind of vehicle.

Verifying the ability of units to interact with each other is called integration testing. There are some significant difference with integration tests when compared with unit tests:

Integration tests are not self-contained.: By their very nature unit tests depend on other units. Integration testing cannot be done on units in isolation.
Integration tests often access external resources.: File systems, databases, web sites, and RESTful servers are but several of the resources integration tests may need to access.
Integration tests are often slow.: It makes sense that testing multiple components may take longer than testing a single component. Additionally accessing remote resources such as databases and REST services add significantly to the execution time.
Integration tests may require special access.: Access to a test database may require special credentials. Interacting with a remote server may require a dedicated test login. Some of these resources may only be available over a VPN. Not every user will have access to the VPN, or be in possession of the database username and password needed for running tests.

An important concept in testing is the system under test (SUT). Integration tests can be done at many levels of granularity. Testing the safety airbag of a car by itself, for example, could be considered a unit test. Performing a simulated crash in which the airbag is deployed inside the car could be considered an integration test, as it ensures that the airbag functions appropriately for the seating arrangement of the car. But this integration test only examines the airbag and seats working together; the SUT is the crash safety system. At a higher level, a SUT of the entire vehicle would test whether the car can drive down the road.

As the SUT gets broader, the more the factors above come into play. Testing become slower and more difficult. Testing edge cases becomes problematic because the number of permutations skyrockets with every added component. It is important to have high test coverage for unit tests, covering as many edge cases as possible. But the broader the SUT, the fewer tests there will be, and the more they will cover normal situations or “happy path” rather than edge cases.

Maven Failsafe

Maven provides a plugin called Failsafe which runs integration tests in a manner similar to how you have been running unit tests during the build. In fact you can continue to use test frameworks such as JUnit and Hamcrest in your integration tests. The only differences are that they will be handled by a separate plugin and usually ran in a separate phase of the build process. And to separate integration tests from unit tests, the tests classes will follow a different naming convention.

You must decide in which face of the build to perform unit tests. Integration tests should only be performed after unit tests have been completed successfully, for obvious reasons, so it is best to perform integration tests after the test phase. Moreover it may be necessary to test the packaged version of your artifact with the packaged version of other artifacts, so waiting until after the package phase might a good idea as well.

Enabling Failsafe

Maven in fact provides two phases for integration tests: integration-test, in which the tests are actually performed; and verify, in which it is confirmed that the unit tests passed.

validate
initialize
compile
test
package
integration-test
verify
install
…

To enable Failsafe in your build, you will need to add maven-failsafe-plugin to your POM.

Adding the Failsafe plugin to the Maven POM.

…
  <build>
    <plugins>
    …
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>2.20.1</version>
        <executions>
          <execution>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    …
    </plugins>
  </build>
…

Unlike many plugins, which by default associate goals with certain phases, the Failsafe plugin must manually be set to execute in appropriate Maven phases. The main Failsafe goals integration-test and verify have the same names as the Maven phases with which they are normally bound. But because of the complexities of unit tests outlined above, some configurations may wish to perform integration tests without actual failing the build—that is, to perform the integration-test goal without ever calling the verify goal. The configuration above binds the two goals to the phases of the same name, which is the most typical configuration.

To run the integration tests, invoke the verify phase. As you already know, this will cause Maven to execute all the previous phases, including the integration-test phase.

mvn verify

Maven also comes with setup and teardown phases for the integration tests: pre-integration-test and post-integration-test. These are performed before and after the integration-test phase, respectively. You may wish to use these phases to start a server before integration tests and shut it down afterwards, for example.

If you instead only invoke the integration-test phase, the tests will never be verified and the build will still indicate success even if an integration failed. Moreover invoking the integration-test phase directly will not run the post-integration-test phase, which may have been configured to tear down the integration tests to release resources.

If for some reason you wish to skip integration tests you can use the -D parameter to pass the flag skipITs when invoking Maven; for example when installing your artifacts in your local repository:

mvn -DskipITs install

It is not a good idea to skip integration tests when performing a deployment.

Writing Failsafe Tests

Integration tests for Failsafe can be written with the same frameworks such as JUnit, Hamcrest, Mockito, and Restito as you use in your integration tests. By Maven convention they likewise will be placed in the src/test/ directory hierarchy. But for Failsafe to distinguish unit tests from integration tests, the Failsafe expects integration tests to use a different naming convention. While unit test classes usually end with …Test, the classes for integration tests usually end with …IT, which stands for “integration test”.

`src/test/java/`	Test source code root directory. e.g. `src/test/java/com/example/FarmIT.java`
`src/test/resources/`	Resources used by the test code.

src/test/java/

Test source code root directory.

e.g. src/test/java/com/example/FarmIT.java

src/test/resources/ Resources used by the test code.

REST Assured

Now that you have Maven set up to run integration tests, you need some integration tests to run. You already have many of the tools for doing this—the same ones you used to write your application. You could use the JAX-RS Client library to call an external server set up specifically for testing, as one example.

It's common to statically import all the REST Assured methods:

import static org.junit.Assert.*
import static org.hamcrest.Matchers.*
import static io.restassured.RestAssured.*
import static io.restassured.matcher.RestAssuredMatchers.*

There is an open-source library named REST Assured that makes it even easier to make RESTful calls and verify their responses. REST Assured could be thought of as a lightweight alternative to JAX-RS Client, with a fluent interface and integration with Hamcrest matching. The main class for initiating the fluent interface is io.restassured.RestAssured and its many static methods. In conjunction the io.restassured.matcher.RestAssuredMatchers class provides additional Hamcrest matchers for testing JSON and XML responses.

Requests

Specifying the requests for a test is done using RestAssured.when(), which returns a io.restassured.specification.RequestSender. Importantly this interface extends io.restassured.specification.RequestSpecification, which contains most of the fluent methods you'll use for setting up the HTTP request. Here are the useful ones. Note that many of these methods have similar variations with different parameters, so be sure and check the API documentation.

accept(String mediaTypes): Specifies the content type to send in the Accept header.
basePath(String basePath): Sets the base path to use, relative to the base URI, in making requests. Otherwise REST Assured will use the value in RestAssured.basePath, which is by default "".
baseUri(String baseUri): Indicates the base URI to use in making requests. Otherwise REST Assured will use the value in RestAssured.baseURI, which is by default "http://localhost".
body(byte[] body): Provides body content to send with the request. Other similar methods allow passing an InputStream or a String. Be careful using body(String body), because it isn't clear which charset is being used for the conversion to bytes. See Rest Assured Issue #926.
contentType(String contentType): Indicates the content type to use for the request.
header(String headerName, Object headerValue, Object... additionalHeaderValues): Provides a header to send in the request.
queryParam(String parameterName, Object... parameterValues): Provides a query parameter to include in the URL.

RequestSpecification also extends io.restassured.specification.RequestSenderOptions<R extends ResponseOptions<R>>, which allows you to indicate the actual HTTP method to use. There are methods for the most common HTTP methods, and a RequestSenderOptions.request(String method) for making an arbitrary HTTP request. The methods related to GET provide typical examples:

get(String path, Map<String,?> pathParams): Sets up a GET request, with a map of replacement values for path patterns such as pens/{penId}.
get(String path, Object... pathParams): Sets up a GET request, with a sequence of replacement values for path patterns such as pens/{penId}.
get(URI uri): Sets up a GET request to a specific URI.
get(URL url): Sets up a GET request to a specific URL.

Responses

Specify the expected response by calling RequestSpecification.then(), which returns a io.restassured.specification.ResponseSpecification. Like RequestSpecification, the ResponseSpecification interface provides many methods for asserting that the response matches expectations. Here are just a few of them:

body(Matcher<?> matcher, Matcher<?>... additionalMatchers): Asserts that the body conforms to one or more Hamcrest matchers.
body(String path, Matcher<?> matcher, Object... additionalKeyMatcherPairs): Asserts that certain JSON or XML contents conform to one or more Hamcrest matchers.
contentType(Matcher<? super String> contentType): Asserts that the response has the given content type.
header(String headerName, Matcher<?> expectedValueMatcher): Asserts that a named header was returned, matching the Hamcrest matcher
statusCode(org.hamcrest.Matcher<? super Integer> expectedStatusCode): Checks the HTTP status code against a Hamcrest matcher.

Representation of https://example.com/farm/pens/pigpen.

{
  "id": "pigpen",
  "name": "Pig Pen",
  "length": 40,
  "width": 30,
  "capacity": 100
}

One of the most powerful aspects of REST Asured is its ResponseSpecification.body(String path, Matcher<?> matcher, Object... additionalKeyMatcherPairs) method. REST Assured will automatically parse the JSON or XML response content. You can pass in a “path” to some location in the parsed tree and use Hamcrest matchers to ensure that the valures are as expected. If the response is JSON {"foo": "bar"}, for example, a call to body("foo", is("bar")) will verify the value of the "foo" value in the JSON object.

Suppose that a web application implementing the Farm RESTful API you have seen in these lessons is deployed at https://example.com/farm/. Sending an HTTP GET request to /farm/pens/pigpen should return the resource representation shown in the figure on the side. You can use REST Assured to test this, as shown in the integration test in the figure below.

Testing the returned representation of a pigpen.

public class PensResourceServiceIT {

  …

  @Test
  public void testGetPigpen() {

    when()
        .baseUri("https://localhost")
        .basePath("/farm/")
        .accept(MediaType.JSON_UTF_8.string())
        .get("pens/{penId}", "pigpen")
        .then()
        .statusCode(is(HttpUrlConnection.HTTP_OK))
        .body("length", is(40), "width", is(30)
            "capacity", greaterThan(90));
    …

  }

REST Assured maps JSON non-integer numbers to the primitive float, not double! If you are comparing a non-integer number such as 1.23, you must use for example is(1.23f) and not is(1.23), because the latter will be comparing to a double literal value. using is() or equalTo() will pass a Float instance, which will be compared using object equality. But remember that exact comparisons with floating-point numbers is not a good idea. You should be comparing floating point values using Matchers.closeTo(double operand, double error). See Confusion over REST Assured floating-point comparisons for the latest discussion on the situation.

Embedded Tomcat

TODO

Review

Gotchas

To run Surefire integration tests under a typical configuration, invoke the Maven verify phase, not the integration-test phase. Otherwise certain post-integration-test activities may not take place, and the integration tests will not be verified.
If you are comparing floating point values in REST Assured, you must use float and not double values.

In the Real World

TODO

Think About It

Is there an absolute measure of whether a test is a unit test or an integration test? Why or why not?
Would you want to perform integration tests against your production server? Why or why not?

Self Evaluation

Why are integration tests needed?
What is a “system under test”, and how does it relate to the definition of an integration test?
Would you normally have more or fewer tests with a larger SUT?
When using the Failsafe plugin, which Maven phase should you invoke on the command line?

Task

Integrate the Failsafe plugin into your Booker project. Use REST Assured to create integration tests of all the RESTful endpoints, covering the most common API calls. Normally it would be best to deploy a separate server for testing, but in this case you can assume that your deployed Booker server is the test server and it has not yet been deployed into production for public use.