How to Run Cucumber Tests using Docker
It is crucial once you have an automated test suite to run it often , get feedback as quick as possible and make the most of your tests to increase your team productivity and product quality. In our previous article we covered how to run selenium Cucumber Tests in chromedriver headless but in order to have that done in a proper way we need to know how to run cucumber tests using docker in parallel .
Assuming we have docker setup already done in Jenkins and your local machine if not here is how your docker file should look like we are going to use jenkins pipeline scripts to kick off our tests in parallel.
Create your jenkins job
Go to Jenkins top page, select “New Job“, then choose “Build a free-style softwareproject“. This job type consists of the following elements: optional SCM, such as CVS or Subversion where your source code resides. optional triggers to control when Jenkins will perform builds.
Go to configure tab job : https://jenkins.yourjenkinshostname.com/view/{ViewName}/job/{JobName}/configure
Paste your pipeline script
stage 'Build'
node('QA-Slave') {
try {
currentBuild.description="Building Image"
git credentialsId: 'gitkey', url: 'kit project url'
docker.build "docker.yourhost.com/cucumber-autotest:${env.BUILD_NUMBER}"
}
catch (e) {
currentBuild.result="FAILED"
throw(e)
} finally {
notifier.notifyBuild(env.BUILD_NUMBER, currentBuild.description, currentBuild.result)
}
}
stage 'Test'
def runjob(String testexecutor,String profile){
currentBuild.description="Testing Image"
dockerContainerName="${testexecutor}-${env.BUILD_NUMBER}"
sh """
./bin/run_tests.sh ${dockerContainerName} ${"docker.yourhost.com/cucumber-autotest:${env.BUILD_NUMBER}"} -p ${profile} THREAD=${profile} -f pretty -f ReportPortal::Cucumber::Formatter -o 'output/test_report.txt';
"""
}
node('QA-Slave') {
timeout(time: 3, unit: 'HOURS'){
try{
parallel 'test1': {runjob('test1','thread1')},
'test2': {runjob('test2','thread2')},
'test3': {runjob('test3','thread3')},
'test4': {runjob('test4','thread4')},
'test5': {runjob('test5','thread5')},
'test6': {runjob('test6','thread6')},
'test7': {runjob('test7','thread7')},
'test8': {runjob('test8', 'thread8')},
'test9': {runjob('test9', 'thread9')},
'test10':{runjob('test10', 'thread10')}
}
catch (e) {
} finally {
step([$class: 'ArtifactArchiver', artifacts: 'output/**/*', excludes: null])
step([$class: 'JUnitResultArchiver', allowEmptyResults: true, keepLongStdio: true, testResults: 'output/*.xml'])
dockerCleanup();
if(notifier.getResults() >= 98){
currentBuild.result="SUCCESS"
sh "exit 0"
} else{
currentBuild.result="FAILED"
sh "exit 1"
throw new Exception("Pass rate lower than 98%")
}
deleteDir()
}
}
}
@NonCPS
def dockerCleanup() {
// Inspired by
// https://gitlab.zoopla.co.uk/devops/jenkins-libraries/blob/master/src/org/zpg/BasePipeline.groovy
// ... but 'mvn-smoke-tests' has made us cautious about over-cleansing
// So for now we'll do a simple `docker rmi`
// remove the *image* we created this build
// the container should auto-exit
sh """
docker rmi ${"docker.yourhost.com/cucumber-autotest:${env.BUILD_NUMBER}"};
"""
}
Lets try to understand some bits from this script:
-f pretty -f ReportPortal::Cucumber::Formatter -o 'output/test_report.txt'
This line is to push your test results to an external report system as in this case is reportportal .
Threads : thread1 , thread2 …threadn is a cucumber profile that sores one or more cucumber tags and should be added in your cucumber yaml file.
notifier.notifyBuild(env.BUILD_NUMBER, currentBuild.description, currentBuild.result)
You need it only if you use a notification system for your build test results , Im going to cover it in a separate article how you can send slack notifications using jenkins pipeline. I will do it separated as is a big topic.
In test stage you can see I used a function getresults()
if(notifier.getResults() >= 95){
currentBuild.result="SUCCESS"
sh "exit 0"
} else{
currentBuild.result="FAILED"
sh "exit 1"
throw new Exception("Pass rate lower than 95%")
}
Basically is our way to set a threshold in jenkins when to fail the build based in the test results. We automatically pass the build if fail rate is less than 2 %. I will cover how to do that in a further article.
Your run tests sh script
Inside runjob() function you’ve seem ./run_tests.sh , bellow you can see the code from it.
#!/bin/sh
# run_test.sh
(
CONTAINER_NAME=$1
IMAGE_ID=$2
echo "cont: " $CONTAINER_NAME;
echo "image: " $IMAGE_ID;
echo "args: " ${@:3};
docker run --cap-add sys_admin -v /dev/shm:/dev/shm -i -e BROWSER -e TEST_ENV --name $CONTAINER_NAME $IMAGE_ID -f junit -o output -f pretty ${@:3};
exit_code=$(docker inspect -f "{{ .State.ExitCode }}" $CONTAINER_NAME)
docker cp $CONTAINER_NAME:/code/output .;
docker rm $CONTAINER_NAME
echo "exit:" $exit_code
exit $exit_code
)
Using this jenkins pipeline script you can customise your cucumber tests to run on whatever environment you like , browser , brand ( if you have a big framework that incorporates multiple internal projects.)
Now there might be voices to say that we are defining the number of threads and we don’t leave jenkins to split the objects dynamically . You can run jobs in parallel using the pipeline script provided by Jenkins but the problem with this will run all your tests sequentially first and only on the second run will be running in parallel.
I found it a bit weird because as you are going to increase the number of tests this job will run again and again your tests sequentially in order to figure out how to split the time equally.
// in this array we'll place the jobs that we wish to run
def branches = [:]
//running the job 4 times concurrently
//the dummy parameter is for preventing mutation of the parameter before the execution of the closure.
//we have to assign it outside the closure or it will run the job multiple times with the same parameter "4"
//and jenkins will unite them into a single run of the job
for (int i = 0; i
It’s up to you what solution you wanna choose , the top one works well for me.
If you have any queries please do not hesitate to add it in the comment bellow.
Happy testing!
The post How to Run Cucumber Tests using Docker in parallel appeared first on TestingRepository.
