Running Spock

Ignore Specifications Based On Conditions

We can use the @Ignore and @IgnoreRest annotation in our Spock specifications to not run the annotated specifications or features. With the @IgnoreIf annotation we can specify a condition that needs to evaluate to true to not run the feature or specification. The argument of the annotation is a closure. Inside the closure we can access three extra variables: properties (Java system properties), env (environment variables) and javaVersion.

In the following Spock specification we have a couple of features. Each feature has the @IgnoreIf annotation with different checks. We can use the extra variables, but we can also invoke our own methods in the closure argument for the annotation:

package com.mrhaki.spock

import spock.lang.*

class SampleRequiresSpec extends Specification {

    private static boolean isOsWindows() {
        System.properties['os.name'] == 'windows'
    }

    @IgnoreIf({ Boolean.valueOf(properties['spock.ignore.longRunning']) })
    def "run spec if Java system property 'spock.ignore.longRunning' is not set or false"() {
        expect:
        true
    }

    @IgnoreIf({ Boolean.valueOf(env['SPOCK_IGNORE_LONG_RUNNING']) })
    def "run spec if environment variable 'SPOCK_IGNORE_LONG_RUNNING' is not set or false"() {
        expect:
        true
    }

    @IgnoreIf({ javaVersion < 1.7 })
    def "run spec if run in Java 1.7 or higher"() {
        expect:
        true
    }

    @IgnoreIf({ javaVersion != 1.7 })
    def "run spec if run in Java 1.7"() {
        expect:
        true
    }

    @IgnoreIf({ isOsWindows() })
    def "run only if run on non-windows operating system"() {
        expect:
        true
    }

}

When we run our specification with Java 1.8 and do not set the Java system property spock.ignore.longRunning or we set the value to false and we do not set the environment variable SPOCK_IGNORE_LONG_RUNNING or give it the value false we can see that some features are ignored:

Now we run on Java 1.7, Windows operating system and set the Java system property spock.ignore.longRunning with the value true and the environment variable SPOCK_IGNORE_LONG_RUNNING with the value true. The resulting report shows the specifications that are ignored and those that are executed:

Code written with Spock 0.7-groovy-2.

Original post written on June 20, 2014

Ignoring Other Feature Methods Using @IgnoreRest

To ignore feature methods in our Spock specification we can use the annotation @Ignore. Any feature method or specification with this annotation is not invoked when we run a specification. With the annotation @IgnoreRest we indicate that feature methods that do not have this annotation must be ignored. So any method with the annotation is invoked, but the ones without aren't. This annotation can only be applied to methods and not to a specification class.

In the next example we have a specification with two feature methods that will be executed and one that is ignored:

@Grab('org.spockframework:spock-core:1.0-groovy-2.4')
import spock.lang.Specification
import spock.lang.IgnoreRest
import spock.lang.Subject

class SampleSpec extends Specification {

    @Subject
    private final underTest = new Sample()

    @IgnoreRest
    void 'run this spec'() {
        expect:
        underTest.message('Asciidoctor') == 'Asciidoctor is awesome.'
    }
    
    @IgnoreRest
    void 'run this spec also'() {
        expect:
        underTest.message('Groovy') == 'Groovy is awesome.'
    }

    void 'ignore this spec'() {
        expect:
        underTest.message('Word') == 'Word is awesome'
    }
    
}

class Sample {
    String message(String tool) {
        println "Getting message for $tool"
        "$tool is awesome."
    }
}

We can run this specification directly from the command line:

$ groovy SampleSpec.groovy
Getting message for Asciidoctor
Getting message for Groovy
JUnit 4 Runner, Tests: 2, Failures: 0, Time: 54
$

Written with Spock 1.0 and Groovy 2.4.10.

Original post written on April 10, 2017

Only Run Specs Based On Conditions

In a previous blog post we have seen the IgnoreIf extension. There is also a counterpart: the Requires extension. If we apply this extension to a feature method or specification class than the method or whole class is executed when the condition for the @Requires annotation is true. If the condition is false the method or specification is not executed. As a value for the @Requires annotation we must specify a closure. In the closure Spock adds some properties we can use for our conditions:

• jvm can be used to check a Java version or compatibility.
• sys returns the Java system properties.
• env used to access environment variables.
• os can be used to check for operating system names.
• javaVersion has the Java version as BigDecimal, eg. 1.8.

In the following example we use the @Requires annotation with different conditions:

package com.mrhaki.spock

import spock.lang.Requires
import spock.lang.Specification

class RequiresSampleSpec extends Specification {

    @Requires({ Boolean.valueOf(sys['spock.longRunning']) })
    def "run spec if Java system property 'spock.longRunning' is true"() {
        expect:
        true
    }

    @Requires({ Boolean.valueOf(env['SPOCK_LONG_RUNNING']) })
    def "run spec if environment variable 'SPOCK_LONG_RUNNING' is true"() {
        expect:
        true
    }

    @Requires({ javaVersion >= 1.7 })
    def "run spec if run in Java 1.7 or higher"() {
        expect:
        true
    }

    @Requires({ jvm.isJava8() })
    def "run spec if run in Java 1.8"() {
        expect:
        true
    }

    @Requires({ os.isWindows() })
    def "run only if run on windows operating system"() {
        expect:
        true
    }

}

If we have the same condition to be applied for all feature methods in a specification we can use the @Requires annotation at the class level:

package com.mrhaki.spock

import spock.lang.Requires
import spock.lang.Specification

@Requires({ jvm.isJava7Compatible() })
class RequiresSpec extends Specification {

    def "all feature methods run only if JVM is Java 7 compatible"() {
        expect:
        true
    }

}

Written with Spock 1.0-groovy-2.4.

Original post written on September 4, 2015

Including or Excluding Specifications Based On Annotations

One of the lesser known and documented features of Spock if the external Spock configuration file. In this file we can for example specify which specifications to include or exclude from a test run. We can specify a class name (for example a base specification class, like DatabaseSpec) or an annotation. In this post we see how to use annotations to have some specifications run and others not.

The external Spock configuration file is actually a Groovy script file. We must specify a runner method with a closure argument where we configure basically the test runner. To include specification classes or methods with a certain annotation applied to them we configure the include property of the test runner. To exclude a class or method we use the exclude property. Because the configuration file is a Groovy script we can use everything Groovy has to offer, like conditional statements, println statements and more.

Spock looks for a file named SpockConfig.groovy in the classpath of the test execution and in in the USER_HOME/.spock directory. We can also use the Java system property spock.configuration with a file name for the configuration file.

In the following example we first define a simple annotation Remote. This annotation can be applied to a class or method:

package com.mrhaki.spock

import java.lang.annotation.ElementType
import java.lang.annotation.Retention
import java.lang.annotation.RetentionPolicy
import java.lang.annotation.Target

@Target([ElementType.TYPE, ElementType.METHOD])
@Retention(RetentionPolicy.RUNTIME)
@interface Remote {
}

We write a simple Spock specification where we apply the Remote annotation to one of the methods:

package com.mrhaki.spock

import spock.lang.Specification

class WordRepositorySpec extends Specification {

    @Remote  // Apply our Remote annotation.
    def "test remote access"() {
        given:
        final RemoteAccess access = new RemoteAccess()

        expect:
        access.findWords('S') == ['Spock']
    }

    def "test local access"() {
        given:
        final LocalAccess access = new LocalAccess()

        expect:
        access.findWords('S') == ['Spock']
    }

}

Next we create a Spock configuration file:

import com.mrhaki.spock.Remote

runner {
    // This is Groovy script and we 
    // add arbitrary code.
    println "Using RemoteSpockConfig"

    // Include only test classes or test
    // methods with the @Remote annotation
    include Remote

    // Alternative syntax
    // to only look for annotations.
    // include {
    //     annotation Remote
    // }

    
    // We can also add a condition in
    // the configuration file.
    // In this case we check for a Java
    // system property and if set the
    // specs with @Remote are not run.
    if (System.properties['spock.ignore.Remote']) {
        exclude Remote
    }
}

When we run the WordRepositorySpec and our configuration file is on the classpath only the specifications with the @Remote annotation are executed. Let's apply this in a simple Gradle build file. In this case we save the configuration file as src/test/resources/RemoteSpockConfig.groovy, we create a new test task remoteTest and set the Java system property spock.configuration:

apply plugin: 'groovy'

repositories {
    jcenter()
}

dependencies {
    compile 'org.codehaus.groovy:groovy-all:2.4.4'
    testCompile 'org.spockframework:spock-core:1.0-groovy-2.4'
}

// New test task with specific 
// Spock configuration file.
task remoteTest(type: Test) {
    // This task belongs to Verification task group.
    group = 'Verification'

    // Set Spock configuration file when running
    // this test task.
    systemProperty 'spock.configuration', 'RemoteSpockConfig.groovy'
}

Now when we execute the Gradle test task all specifications are executed:

And when we run remoteTest only the specification with the @Remote annotation are executed:

Written with Gradle 2.6 and Spock 1.0-groovy-2.4.

The code is available on Github

Original post written on August 27, 2015

Optimize Run Order Test Methods

Spock is able to change the execution order of test methods in a specification. We can tell Spock to re-run failing methods before successful methods. And if we have multiple failing or successful tests, than first run the fastest methods, followed by the slower methods. This way when we re-run the specification we immediately see the failing methods and could stop the execution and fix the errors. We must set the property optimizeRunOrder in the runner configuration of the Spock configuration file. A Spock configuration file with the name SpockConfig.groovy can be placed in the classpath of our test execution or in our USER_HOME/.spock directory. We can also use the Java system property spock.configuration and assign the filename of our Spock configuration file.

In the following example we have a specification with different methods that can be successful or fail and have different durations when executed:

package com.mrhaki.spock

import spock.lang.Specification
import spock.lang.Subject

class SampleSpec extends Specification {

    @Subject
    private Sample sample = new Sample()

    def "spec1 - slowly should return name property value"() {
        given:
        sample.name = testValue

        expect:
        sample.slowly() == testValue

        where:
        testValue = 'Spock rules'
    }

    def "spec2 - check name property"() {
        given:
        sample.name = testValue

        expect:
        sample.name == testValue

        where:
        testValue = 'Spock is gr8'
    }

    def "spec3 - purposely fail test at random"() {
        given:
        sample.name = testValues[randomIndex]

        expect:
        sample.name == testValues[0]

        where:
        testValues = ['Spock rules', 'Spock is gr8']
        randomIndex = new Random().nextInt(testValues.size())
    }

    def "spec4 - purposely fail test slowly"() {
        given:
        sample.name = 'Spock is gr8'

        expect:
        sample.slowly() == 'Spock rules'
    }

    def "spec5 - purposely fail test"() {
        given:
        sample.name = 'Spock rules'

        expect:
        sample.name == 'Spock is gr8'
    }
}

class Sample {
    String name

    String slowly() {
        Thread.sleep(2000)
        name
    }
}

Let's run our test where there is no optimised run order. We see the methods are executed as defined in the specification:

Next we create a Spock configuration file with the following contents:

runner {
    println "Optimize run order"
    optimizeRunOrder true
}

If we re-run our specification and have this file in the classpath we already see the order of the methods has changed. The failing tests are at the top and the successful tests are at the bottom. The slowest test method is last:

Another re-run has optimised the order by running the slowest failing test after the other failing tests.

Spock keeps track of the failing and successful methods and their execution time in a file with the specification name in the USER_HOME/.spock/RunHistory directory. To reset the information we must delete the file from this directory.

Written with Spock 1.0-groovy-2.4.

Original post written on August 28, 2015