Running TestProject.io tests in an Azure DevOps release pipeline

TestProject.io is a useful tool used for test automation, we use it to test our front end product called Portal. So far it has been the responsibility of our QA lead to write and run the test suite manually on her local machine. She has built up a very good suite of reliable and extensive tests which we can now add to our release pipelines to trigger automatically.  

Test Agent 

TestProject.io doesn’t run tests on their own servers. Instead they have agents. An agent is the testing software that runs the tests and needs to have access to a browser like Chrome. It’s your responsibility to run an agent either locally or on hosted infrastructure. 

Our QA lead has been running the agent locally on her machine, but this obviously doesn’t work for release pipelines which can happen at any time. The first option is to start a new virtual machine for the agent inside our Azure subscription. Because the vm doesn’t need to be very powerful the monetary cost of this is small but it does mean we have to manage extra infrastructure.  

The second option is to use Docker to start the testing agent inside the Azure dev ops pipeline. We like this option because it gives us flexibility of having a testing agent wherever and whenever we need one without the extra infrastructure to manage. 

We accomplished this with just 4 tasks inside a task group that we can add to any of our release pipelines: 

Install Docker 

Just to make sure the Azure dev ops hosted agent has docker installed.

Create Docker compose file 

New-Item -Path . -Name "testproject.yaml" -ItemType "file" -Value "
version: ""3.1""
services:
  testproject-agent:
    image: testproject/agent:latest
    container_name: testproject-agent
    depends_on:
      - chrome
      - firefox
    volumes:
      - /path/to/host/folder:/home/vsts/work/_temp/
    environment:
      TP_API_KEY: ""$(ApiKey)""
      TP_JOB_ID: ""$(JobId)""
      TP_JOB_PARAMS: '""jobParameters"" : { ""browsers"": [ ""chrome"", ""firefox"" ] }'
      TP_AGENT_ALIAS: ""$(Release.DefinitionName)-$(Release.ReleaseId)-$(Release.DeployPhaseID)""
      CHROME: ""chrome:4444""
      FIREFOX: ""firefox:4444""
  chrome:
    image: selenium/standalone-chrome
    volumes:
      - /dev/shm:/dev/shm
  firefox:
    image: selenium/standalone-firefox
    volumes:
      - /dev/shm:/dev/shm"

This task is abit cheeky. Docker compose needs a definition yaml file but instead of having to supply one to every release we’ve used PowerShell to create one on demand. 

We’re going to run the testproject agent container with Chrome and Firefox containers inside a Docker compose application. The definition follows the example provided by TestProject.io in their documentation but a lot of the variables have been updated for Azure Dev Ops. 

The volumes section of the definition has been set to the ADO temp folder: 

volumes:
    - /path/to/host/folder:/home/vsts/work/_temp/

The ApiKey and JobId parameters have been set to variables so they can be set by the release pipelines using the task group: 

TP_API_KEY: ""$(ApiKey)""
TP_JOB_ID: ""$(JobId)""

The agent alias is set to the different ids of the release to prevent any conflicts with other releases and tracking purposes: 

TP_AGENT_ALIAS: ""$(Release.DefinitionName)-$(Release.ReleaseId)-$(Release.DeployPhaseID)""

Docker Compose

In this step we start the agent and run the tests by using the Docker compose command. By specifying the TP_JOB_ID variable in the yaml definition the application will perform the job and destroy itself afterwards.

docker-compose -f testproject.yaml up testproject-agent

We also specify the TestProject.io container at the end of the command to follow the output of that container. If we didn’t do this the command would end straight away.

Test Pass / Fail

In this final task we get the result of the test from TestProject and use it to pass or fail the release.

$uri = 'https://api.testproject.io/v2/projects/$(ProjectId)/jobs/$(JobId)/reports' 
$headers = @{
    'authorization' = '$(ApiKey)'
}

$response = Invoke-RestMethod -Uri $uri -Method Get -Headers $headers

$testRun = 'false'

Foreach($i in $response){
	if($i.agent -eq '$(Release.DefinitionName)-$(Release.ReleaseId)-$(Release.DeployPhaseID)'){
		$testRun = 'true'
		if($i.resultType -eq 'Failed'){             
			'Tests failed - '
			$i.reportUrl
			Write-Host "##vso[task.complete result=Failed;] FAILED"
		}
	}
}

if($testRun -eq 'false'){
           'Tests did not run!'
           Write-Host "##vso[task.complete result=Failed;] FAILED"
}

Using the TestProject api we get the all of the test reports for the job we have completed. Looping through them we use the release id to find the report of the test we just performed. We then check the result and fail the release if the test run has failed.

Adding our TestProject regressions into our release pipelines has been very valuable for us. We run a smaller smoke test of core functionality each time we deploy and a full regression every morning.