Sikuli Selenium Robot Framework Tutorial


In this tutorial I will demonstrate how to integrate Sikuli (open source image based test automation tool) and Selenium (object based web automation framework) into Robot Framework (keyword and data driven test automation framework). This is not a Robot Framework Tutorial nor a Sikuli Tutorial. For more information about these technologies please visit their respective websites as indicated below:

  1. Sikuli website
  2. Selenium (web driver) Robot Framework library:
  3. Robot Framework:

The following sections outline the detailed steps needed to perform the integration on Windows machine. Installation on different operating systems should be similar though it might not be exactly the same.

Python Installation

Robot Framework runs on Python so we need to install it. Please follow the steps below:

  1. Go to
  2. Click the download link on the left side of the page
  3. Select windows installer "Python 2.7.2 Windows Installer" or  you can directly access the link: You can always select the latest production version.
  4. Follow the instructions and use default settings
  5. To make sure Python was properly installed open a console window then type: "python" You should get the Python command prompt
  6. Type: "exit()" to quit
  7. You might consider setting PYTHONPATH environment variable if needed.

Java Run Time Installation

Sikuli is Java based so if you do not have Java Run Time (JRE) already installed then download the 32 bit package from oracle website and install it. You can obtain the package from: Select the version that matches your OS. In this tutorial I selected JRE6. Please note that Sikuli runs only using the JRE6 32 bit version.

Jython Installation

Jython is a Java implementation of the Python programming language. The reason why we need to install Jython is because we will be calling Sikuli Java API using Python scripts. You can also access Sikuli Java API from a native Java application if you wish. Please follow the steps below:

  1. Go to
  2. Click the download link on the left side of the page
  3. Download the most stable version. Clicking the download link takes you to
  4. Double click the jar file to start the installer. Select default settings
  5. To make sure Jython is properly installed, open a console window then type: C:\\path\\to\\jython\\bin\\jython.bat. You should get the jython command prompt
  6. Type "exit()" to quit
  7. You might consider setting JYTHONPATH environment variable if needed
  8. If you encounter virtual memory related problems you might need to edit [C:\\path\\to\\jython\\bin\\jython.bat] and increase the value in [set _JAVA_MEM=-Xmx512m]
  9. If you encounter encoding issues for example using Sikuli to type international characters then you might need to add [-Dfile.encoding=utf-8] in [set _FULL_CMD]

Installing Easy Install

As the name indicates it is used for easy installations. Please follow the steps below:

  1. Go to:
  2. Download the file "setuptools-0.6c11.win32-py2.7.exe"
  3. Start the installer and use default settings

Note that there is an alternative installation tool called pip if you want to give it a try. Here is how to install pip on Windows:

  1. Download to C:\\Python\\Scripts
  2. cd C:\\Python\\Scripts
  3. Type: easy_install pip
  4. Type: python

Setting Environment Variables

In order to call programs from the command prompt directly we need to add some directories to the system PATH environment variable. Please follow the steps below:

  1. On Windows go to: Start > Settings > Control Panel > System > Advanced > Environment Variables
  2. Add the following directories to PATH:
  • Path to python installation for example: C:\\python27
  • Path to python scripts directory for example: C:\\Python27\\Scripts
  • Path to jython bin directory for example: C:\\jython2.5.2\\bin
  • Path to Java for example C:\\Program Files (x86)\\Java\\jre6\\bin

Install Robot Framework

Robot Framework provides high level automation services such as test execution and reporting. You can visit Robot Framework download page on Google Code or you may follow the steps below to install it:

  1. Open a new command window
  2. Type: easy_install robotframework
  3. To verify installation type: pybot --version
  4. If that does not work go to: C:\\python27\\scripts then type:

Installing wxPython

This is a GUI toolkit for Python programming language. It is used by Robot Framework RIDE (will be installed in the next section)

  1. From the command line type: easy_install wxPython
  2. If this does not work go to:
  3. Download an installer for your OS
  4. Start the installer and follow the instructions using default settings

Note: For some reason the 64 bit version did not work with me. I tried the 32 bit version and it worked fine.

Robot Framework RIDE Installation

Robot Framework Ride is an awesome Integrated Development Environment (IDE) for Robot Framework test case development. Follow the steps below:

  1. Open a new command window
  2. Type: easy_install robotframework-ride
  3. To make sure Robot Framework RIDE is working fine type: python
  4. In Windows you can associate python files with the python executable. In that case you just need to only type from the command line

Sikuli Installation

Sikuli is used for visual testing. This installation is needed to get a copy of sikuli-script.jar which provides image based test API. The installation also includes an IDE that can help in debugging visual tests. Follow the steps below:

  1. Go to :
  2. Depending on your OS download the appropriate version. Use the Self-extracting installer for easy installation "Sikuli-X-1.0rc3 (r905)-win32.exe" You can always select the latest stable version
  3. Start the installer and follow the instructions using default settings

Selenium2Library Installation

This is a Robot Framework Selenium test library which provides Selenium 2 (web driver) test keywords. Follow the steps below:

  1. Open a command line console
  2. Type: easy_install robotframework-selenium2library

Sikuli Python and Selenium Robot Framework in Action

You can find below all the steps needed to develop and execute a Robot Framework test case using both Sikuli and Selenium at the same time. Note that Selenium library runs on Python (does not support Jython) however Sikuli needs Jython so using one Python interpreter is not going to work for both at the same time. The trick is to use Python (works fine with Selenium library) and run Jython on a remote machine (localhost simulating remote machine). This way we execute Selenium keywords using Python on the local machine and call Sikuli keywords on the remote machine using RPC-xml sever.

    1. Create the following directories (you can do it the way you prefer but this is just to make it easy to follow the steps) [C:\\robot] [C:\\robot\\data] [C:\\robot\\libs] [C:\\robot\\suites]
    2. Download the Python Remote Server and save it into [C:\\robot\\libs] You can find it here:
    3. Copy the Sikuli Remote Library code (you can find it at the bottom of the page) and save it as: C:\\robot\\libs\\
    4. Copy C:\\Program Files\\Sikuli X\\sikuli-script.jar into C:\\robot\\libs. Note that this step is only to put things in one place but you can skip this step and only point to Sikuli jar file
    5. Add C:\\robot\\libs\\sikuli-script.jar to your java class path. You need to set the CLASSPATH environment variable. Make sure there is no extra spaces otherwise problems may arise.
    6. Run the remote server. Go to:  C:\\robot\\libs and type: jython.bat
    7. Launch RIDE. From the command line type:
    8. Create a new project: File -> New project then set the parent directory to C:\\robot\\suites and give the suite a name for example "test"
    9. Switch to text edit view and paste the following text:
*** Settings ***
Documentation     Integrating Selenium, Sikuli into Robot Framework
Test Setup        Login To Yahoo Mail    ${user_name}    ${password}
Test Teardown     Tear Test Down
# Selenium library
Library           Selenium2Library    15
# Sikuli
Library           Remote    http://localhost:${port}
# Taking screenshots when a test fails
Library           Screenshot

*** Variables ***
# Yahoo mail URL
# Browser
${browser}        ff
# User name
${user_name}      user_name
# Password
${password}       password
# Default port number for the remote server
${port}           8270
# Sikuli images
${data_path}      c:\\robot\\data
# Used in Sikuli image comparison
${similarity}     0.90
# Time to wait for objects
${timeout}        10

*** Testcases ***
login To Yahoo Mail Test Case
Wait Until Page Contains Element    yuhead-sform-searchfield    ${timeout}
Input Text    yuhead-sform-searchfield    ENGLISH
Object Exists    ${data_path}\\search_box_left.png    ${similarity}

*** Keywords ***
Login To Yahoo Mail
[Arguments]    ${user}    ${pass}
[Documentation]    This keyword logs user into Yahoo mail
Open Browser    ${url}    ${browser}
Wait Until Page Contains Element    username
Input Text    username    ${user}
Input Password    passwd    ${pass}
Click Button    .save
Wait Until Page Contains Element    toolbar
Click Object    ${data_path}\\maximize.png
Sleep    1

Tear Test Down
Run Keyword If Test Failed    Take Screenshot
Close All Browsers

Here is the python code for Sikuli Remote Library:

import sys
from robotremoteserver import RobotRemoteServer
from org.sikuli.script import *

class SikuliRemoteLibrary:

	def __init__(self):
		self.SS = Screen()
		self.PT = Pattern()

	def _wait(self, imgFile, timeOut, similarity):
		self.PT = Pattern(imgFile)
		self.PT = self.PT.similar(float(similarity))
		self.SS.wait(self.PT, float(timeOut))
	except FindFailed, err:
		print "ERR: _wait"
		raise AssertionError(err)

	def click_object(self, imgFile, timeOut, similarity):
		self._wait(imgFile, timeOut, similarity)
	except FindFailed, err:
		raise AssertionError("Cannot click [" + imgFile + "]")

	def object_exists(self, imgFile, similarity, timeOut):
		self._wait(imgFile, timeOut, similarity)
	except FindFailed, err:
		raise AssertionError("Could not find [" + imgFile + "]")

	def type_at_object(self, imgFile, txt, timeOut, similarity):
		self._wait(imgFile, timeOut, similarity)
		self.SS.type(imgFile, txt)
	except FindFailed, err:
		raise AssertionError("Cannot type at [" + imgFile + "]")

	def paste_at_object(self, imgFile, txt, timeOut, similarity):
		self._wait(imgFile, timeOut, similarity)
		self.SS.paste(imgFile, txt)
	except FindFailed, err:
		raise AssertionError("Cannot paste at [" + imgFile + "]")

if __name__ == '__main__':
	SRL = SikuliRemoteLibrary()
	RobotRemoteServer(SRL, *sys.argv[1:])

Continuous Integration

We explained how to execute the test suite using RIDE but what if we want to execute it automatically in a CI (Continuous Integration) environment. In order to do that we should be able to invoke the test suite from the command line for example Windows command prompt or UNIX terminal. Let us do that:

    1. Create a directory called [C:\\robot\\arg]. This is where Robot argument files are stored. Recall that argument files are used to pass parameters to the framework at runtime.
    2. Create an argument text file for example C:\\robot\\arg\arg.txt
    3. Paste the following into the argument file:
--variable user_name:user_name
--variable password:pass_word
--outputdir ..\\out

As you can see we are passing two parameters (user name and password) and specifying the output directory relative to the directory where the test suite is located.

    1. Create a directory called C:\\robot\\run
    2. Create a batch file called run.bat
    3. Paste the following into the batch file:
call pybot.bat --argumentfile C:\\robot\arg\\arg.txt C:\\robot\\suites\\test.txt

This command executes the test suite using an argument file. Note that variables passed in the argument file override whatever variables defined in the test suite if they have the same name. The reason why we pass parameters in an argument file is because we might need to keep the variables in the test suite for development and debugging purposes when we use RIDE. It is up to test developer to articulate the test cases in the way he or she prefers.

So far we are ready to invoke robot from the command line using a windows bat file. We will not be doing that manually but instead we will use Jenkins. We are still working on the Windows OS.

Install Jenkins

  1. Go to the download page at:
  2. You can download the windows installer then run Jenkins as a windows service. The windows service in this case will not be able to interact with the GUI which causes selenium to fail opening the browser. The solution is to run Jenkins from the war file. Now go ahead and download the war file jenkins.war
  3. Run Jenkins as follows: java -jar jenkins.war --httpPort=80

Install Jenkins Plugin

Robot Framework has a Jenkins plugin for aggregating test execution results and linking to reports and logs. Follow the steps below to install the plugin:

  1. Go to:
  2. Download the file: robotframework.hpi
  3. Make sure Jenkins is up otherwise launch using the command: java -jar jenkins.war --httpPort=80
  4. Open a browser window and point to http://localhost:8080
  5. Click the link "Manage Jenkins"
  6. Click the link "Manage Plugins"
  7. Click the "Advanced" tab
  8. Go to "Upload Plugin" and browse to the robotframework.hpi file then upload it

Note: You might need to restart Jenkins in order to see robot framework plugin listed in the post build action drop down.

Create Jenkins Job

Let us now create and configure a Jenkins job to automatically execute robot framework test suite. Follow the steps below:

    1. Click the link "New Job"
    2. Give it a name and select "Build a free-style software project"
    3. Click "Add build step" drop down
    4. Click "Execute windows batch command"
    5. You can paste something like:
cd C:\\robot\\run
call run.bat
  1. Click the drop down "Add post-build action"
  2. Click "Publish Robot Framework test results"
  3. Set "Directory of Robot output" as "..\\out"
  4. Set "Log/Report link" as "log.html"
  5. Set "Output xml name" as "output.xml"
  6. Set "Report html name" as "report.html"
  7. Set "Log html name" as "log.html"

That is it we are now ready to kick off the Jenkins job. Make sure Sikuli remote library is running and then click to build the job. As the job executes you can go to the console and watch Jenkins invoking Robot Framework in a similar way to invoking Robot from the command line. We can configure Jenkins to run the job on a slave machine but this is beyond the scope of this article.

Tips and Tricks

    1. Firefox Profiles

Go to robot framework installation directory for example:


You will find Firefox browser profiles which are used by selenium library. It might be very useful to update the provided profile with whatever Firefox profile you wish. This is a great advantage and saves allot of test setup time if you need the profile to be configured in a certain way instead of programmatically doing that. You could maintain several copies of different profiles there and then copy the one you need at run time just before invoking robot. Here is a sample run.bat file that does that:

@echo off
@rmdir %SEL2RES%\firefoxprofile /q /s
@mkdir %SEL2RES%\firefoxprofile
@xcopy %SEL2RES%\my_ff_profile  %SEL2RES%\firefoxprofile /e /f /y
@echo on
call pybot.bat --argumentfile c:\\robot\\arg\\arg.txt C:\\robot\\suites\\test.txt

Where %SEL2RES% is an environment variable that points to the resources directory in robot framework selenium library.

    1. Window Minimized

Sikuli remote server and Jenkins should start minimized in order not to interfere with Sikuli while image based testing is executing. To do that create a windows shortcut for both Sikuli remote server batch file and Jenkins batch file then right click the link and go to properties. There you can chose to run the command as minimized.

That is it for today, If you have questions about the code please contact me as the main focus of this article was on the integration side not on the actual test cases. Thanks for reading.

Search Terms...

Leave a Reply