Sikuli Selenium Robot Framework Tutorial

Introduction

In this tutorial I will demonstrate how to integrate Sikuli (open source image based test automation tool) and Selenium (object based web automation framework) into Robot Framework (keyword and data driven test automation framework). This is not a Robot Framework Tutorial nor a Sikuli Tutorial. For more information about these technologies please visit their respective websites as indicated below:

  1. Sikuli website http://sikuli.org
  2. Selenium (web driver) Robot Framework library: https://github.com/rtomac/robotframework-selenium2library
  3. Robot Framework: http://code.google.com/p/robotframework

The following sections outline the detailed steps needed to perform the integration on Windows machine. Installation on different operating systems should be similar though it might not be exactly the same.

Python Installation

Robot Framework runs on Python so we need to install it. Please follow the steps below:

  1. Go to http://python.org
  2. Click the download link on the left side of the page
  3. Select windows installer "Python 2.7.2 Windows Installer" or  you can directly access the link: http://python.org/ftp/python/2.7.2/python-2.7.2.msi You can always select the latest production version.
  4. Follow the instructions and use default settings
  5. To make sure Python was properly installed open a console window then type: "python" You should get the Python command prompt
  6. Type: "exit()" to quit
  7. You might consider setting PYTHONPATH environment variable if needed.

Java Run Time Installation

Sikuli is Java based so if you do not have Java Run Time (JRE) already installed then download the 32 bit package from oracle website and install it. You can obtain the package from: http://www.oracle.com/technetwork/java/javase/downloads/index.html. Select the version that matches your OS. In this tutorial I selected JRE6. Please note that Sikuli runs only using the JRE6 32 bit version.

Jython Installation

Jython is a Java implementation of the Python programming language. The reason why we need to install Jython is because we will be calling Sikuli Java API using Python scripts. You can also access Sikuli Java API from a native Java application if you wish. Please follow the steps below:

  1. Go to http://www.jython.org
  2. Click the download link on the left side of the page
  3. Download the most stable version. Clicking the download link takes you to http://sourceforge.net
  4. Double click the jar file to start the installer. Select default settings
  5. To make sure Jython is properly installed, open a console window then type: C:\\path\\to\\jython\\bin\\jython.bat. You should get the jython command prompt
  6. Type "exit()" to quit
  7. You might consider setting JYTHONPATH environment variable if needed
  8. If you encounter virtual memory related problems you might need to edit [C:\\path\\to\\jython\\bin\\jython.bat] and increase the value in [set _JAVA_MEM=-Xmx512m]
  9. If you encounter encoding issues for example using Sikuli to type international characters then you might need to add [-Dfile.encoding=utf-8] in [set _FULL_CMD]

Installing Easy Install

As the name indicates it is used for easy installations. Please follow the steps below:

  1. Go to: http://pypi.python.org/pypi/setuptools
  2. Download the file "setuptools-0.6c11.win32-py2.7.exe"
  3. Start the installer and use default settings

Note that there is an alternative installation tool called pip if you want to give it a try. Here is how to install pip on Windows:

  1. Download http://peak.telecommunity.com/dist/ez_setup.py to C:\\Python\\Scripts
  2. cd C:\\Python\\Scripts
  3. Type: easy_install pip
  4. Type: python ez_setup.py

Setting Environment Variables

In order to call programs from the command prompt directly we need to add some directories to the system PATH environment variable. Please follow the steps below:

  1. On Windows go to: Start > Settings > Control Panel > System > Advanced > Environment Variables
  2. Add the following directories to PATH:
  • Path to python installation for example: C:\\python27
  • Path to python scripts directory for example: C:\\Python27\\Scripts
  • Path to jython bin directory for example: C:\\jython2.5.2\\bin
  • Path to Java for example C:\\Program Files (x86)\\Java\\jre6\\bin

Install Robot Framework

Robot Framework provides high level automation services such as test execution and reporting. You can visit Robot Framework download page on Google Code or you may follow the steps below to install it:

  1. Open a new command window
  2. Type: easy_install robotframework
  3. To verify installation type: pybot --version
  4. If that does not work go to: C:\\python27\\scripts then type: robot_postinstall.py

Installing wxPython

This is a GUI toolkit for Python programming language. It is used by Robot Framework RIDE (will be installed in the next section)

  1. From the command line type: easy_install wxPython
  2. If this does not work go to: http://www.wxpython.org/download.php
  3. Download an installer for your OS
  4. Start the installer and follow the instructions using default settings

Note: For some reason the 64 bit version did not work with me. I tried the 32 bit version and it worked fine.

Robot Framework RIDE Installation

Robot Framework Ride is an awesome Integrated Development Environment (IDE) for Robot Framework test case development. Follow the steps below:

  1. Open a new command window
  2. Type: easy_install robotframework-ride
  3. To make sure Robot Framework RIDE is working fine type: python ride.py
  4. In Windows you can associate python files with the python executable. In that case you just need to only type ride.py from the command line

Sikuli Installation

Sikuli is used for visual testing. This installation is needed to get a copy of sikuli-script.jar which provides image based test API. The installation also includes an IDE that can help in debugging visual tests. Follow the steps below:

  1. Go to : http://sikuli.org/download.shtml
  2. Depending on your OS download the appropriate version. Use the Self-extracting installer for easy installation "Sikuli-X-1.0rc3 (r905)-win32.exe" You can always select the latest stable version
  3. Start the installer and follow the instructions using default settings

Selenium2Library Installation

This is a Robot Framework Selenium test library which provides Selenium 2 (web driver) test keywords. Follow the steps below:

  1. Open a command line console
  2. Type: easy_install robotframework-selenium2library

Sikuli Python and Selenium Robot Framework in Action

You can find below all the steps needed to develop and execute a Robot Framework test case using both Sikuli and Selenium at the same time. Note that Selenium library runs on Python (does not support Jython) however Sikuli needs Jython so using one Python interpreter is not going to work for both at the same time. The trick is to use Python (works fine with Selenium library) and run Jython on a remote machine (localhost simulating remote machine). This way we execute Selenium keywords using Python on the local machine and call Sikuli keywords on the remote machine using RPC-xml sever.

    1. Create the following directories (you can do it the way you prefer but this is just to make it easy to follow the steps) [C:\\robot] [C:\\robot\\data] [C:\\robot\\libs] [C:\\robot\\suites]
    2. Download the Python Remote Server and save it into [C:\\robot\\libs] You can find it here: http://robotframework.googlecode.com/hg/tools/remoteserver/robotremoteserver.py
    3. Copy the Sikuli Remote Library code (you can find it at the bottom of the page) and save it as: C:\\robot\\libs\\SikuliRemoteLibrary.py
    4. Copy C:\\Program Files\\Sikuli X\\sikuli-script.jar into C:\\robot\\libs. Note that this step is only to put things in one place but you can skip this step and only point to Sikuli jar file
    5. Add C:\\robot\\libs\\sikuli-script.jar to your java class path. You need to set the CLASSPATH environment variable. Make sure there is no extra spaces otherwise problems may arise.
    6. Run the remote server. Go to:  C:\\robot\\libs and type: jython.bat SikuliRemoteLibrary.py
    7. Launch RIDE. From the command line type: ride.py
    8. Create a new project: File -> New project then set the parent directory to C:\\robot\\suites and give the suite a name for example "test"
    9. Switch to text edit view and paste the following text:
*** Settings ***
Documentation     Integrating Selenium, Sikuli into Robot Framework
Test Setup        Login To Yahoo Mail    ${user_name}    ${password}
Test Teardown     Tear Test Down
# Selenium library
Library           Selenium2Library    15
# Sikuli
Library           Remote    http://localhost:${port}
# Taking screenshots when a test fails
Library           Screenshot

*** Variables ***
# Yahoo mail URL
${url}            http://mail.yahoo.com
# Browser
${browser}        ff
# User name
${user_name}      user_name
# Password
${password}       password
# Default port number for the remote server
${port}           8270
# Sikuli images
${data_path}      c:\\robot\\data
# Used in Sikuli image comparison
${similarity}     0.90
# Time to wait for objects
${timeout}        10

*** Testcases ***
login To Yahoo Mail Test Case
Wait Until Page Contains Element    yuhead-sform-searchfield    ${timeout}
Input Text    yuhead-sform-searchfield    ENGLISH
Object Exists    ${data_path}\\search_box_left.png    ${similarity}

*** Keywords ***
Login To Yahoo Mail
[Arguments]    ${user}    ${pass}
[Documentation]    This keyword logs user into Yahoo mail
Open Browser    ${url}    ${browser}
Wait Until Page Contains Element    username
Input Text    username    ${user}
Input Password    passwd    ${pass}
Click Button    .save
Wait Until Page Contains Element    toolbar
Click Object    ${data_path}\\maximize.png
Sleep    1

Tear Test Down
Run Keyword If Test Failed    Take Screenshot
Close All Browsers

Here is the python code for Sikuli Remote Library:

import sys
from robotremoteserver import RobotRemoteServer
from org.sikuli.script import *

class SikuliRemoteLibrary:

	def __init__(self):
		self.SS = Screen()
		self.PT = Pattern()

	def _wait(self, imgFile, timeOut, similarity):
	try:
		self.PT = Pattern(imgFile)
		self.PT = self.PT.similar(float(similarity))
		self.SS.wait(self.PT, float(timeOut))
	except FindFailed, err:
		print "ERR: _wait"
		raise AssertionError(err)

	def click_object(self, imgFile, timeOut, similarity):
	try:
		self._wait(imgFile, timeOut, similarity)
		self.SS.click(imgFile)
	except FindFailed, err:
		raise AssertionError("Cannot click [" + imgFile + "]")

	def object_exists(self, imgFile, similarity, timeOut):
	try:
		self._wait(imgFile, timeOut, similarity)
	except FindFailed, err:
		raise AssertionError("Could not find [" + imgFile + "]")

	def type_at_object(self, imgFile, txt, timeOut, similarity):
	try:
		self._wait(imgFile, timeOut, similarity)
		self.SS.type(imgFile, txt)
	except FindFailed, err:
		raise AssertionError("Cannot type at [" + imgFile + "]")

	def paste_at_object(self, imgFile, txt, timeOut, similarity):
	try:
		self._wait(imgFile, timeOut, similarity)
		self.SS.paste(imgFile, txt)
	except FindFailed, err:
		raise AssertionError("Cannot paste at [" + imgFile + "]")

if __name__ == '__main__':
	SRL = SikuliRemoteLibrary()
	RobotRemoteServer(SRL, *sys.argv[1:])

Continuous Integration

We explained how to execute the test suite using RIDE but what if we want to execute it automatically in a CI (Continuous Integration) environment. In order to do that we should be able to invoke the test suite from the command line for example Windows command prompt or UNIX terminal. Let us do that:

  1. Create a directory called [C:\\robot\\arg]. This is where Robot argument files are stored. Recall that argument files are used to pass parameters to the framework at runtime.
  2. Create an argument text file for example C:\\robot\\arg\arg.txt
  3. Paste the following into the argument file:
  4. --variable user_name:user_name
    --variable password:pass_word
    --outputdir ..\\out
    

    As you can see we are passing two parameters (user name and password) and specifying the output directory relative to the directory where the test suite is located.

  5. Create a directory called C:\\robot\\run
  6. Create a batch file called run.bat
  7. Paste the following into the batch file:
  8. call pybot.bat --argumentfile C:\\robot\arg\\arg.txt C:\\robot\\suites\\test.txt
    

    This command executes the test suite using an argument file. Note that variables passed in the argument file override whatever variables defined in the test suite if they have the same name. The reason why we pass parameters in an argument file is because we might need to keep the variables in the test suite for development and debugging purposes when we use RIDE. It is up to test developer to articulate the test cases in the way he or she prefers.

So far we are ready to invoke robot from the command line using a windows bat file. We will not be doing that manually but instead we will use Jenkins. We are still working on the Windows OS.

Install Jenkins

  1. Go to the download page at: http://jenkins-ci.org
  2. You can download the windows installer then run Jenkins as a windows service. The windows service in this case will not be able to interact with the GUI which causes selenium to fail opening the browser. The solution is to run Jenkins from the war file. Now go ahead and download the war file jenkins.war
  3. Run Jenkins as follows: java -jar jenkins.war --httpPort=80

Install Jenkins Plugin

Robot Framework has a Jenkins plugin for aggregating test execution results and linking to reports and logs. Follow the steps below to install the plugin:

  1. Go to: http://code.google.com/p/robotframework-javatools/downloads/list/
  2. Download the file: robotframework.hpi
  3. Make sure Jenkins is up otherwise launch using the command: java -jar jenkins.war --httpPort=80
  4. Open a browser window and point to http://localhost:8080
  5. Click the link "Manage Jenkins"
  6. Click the link "Manage Plugins"
  7. Click the "Advanced" tab
  8. Go to "Upload Plugin" and browse to the robotframework.hpi file then upload it

Note: You might need to restart Jenkins in order to see robot framework plugin listed in the post build action drop down.

Create Jenkins Job

Let us now create and configure a Jenkins job to automatically execute robot framework test suite. Follow the steps below:

  1. Click the link "New Job"
  2. Give it a name and select "Build a free-style software project"
  3. Click "Add build step" drop down
  4. Click "Execute windows batch command"
  5. You can paste something like:
  6. cd C:\\robot\\run
    call run.bat
    
  7. Click the drop down "Add post-build action"
  8. Click "Publish Robot Framework test results"
  9. Set "Directory of Robot output" as "..\\out"
  10. Set "Log/Report link" as "log.html"
  11. Set "Output xml name" as "output.xml"
  12. Set "Report html name" as "report.html"
  13. Set "Log html name" as "log.html"

That is it we are now ready to kick off the Jenkins job. Make sure Sikuli remote library is running and then click to build the job. As the job executes you can go to the console and watch Jenkins invoking Robot Framework in a similar way to invoking Robot from the command line. We can configure Jenkins to run the job on a slave machine but this is beyond the scope of this article.

Tips and Tricks

  1. Firefox Profiles
  2. Go to robot framework installation directory for example:

    C:\\Python27\\Lib\site-packages\\robotframework_selenium2library-1.0.1-py2.7.egg\\Selenium2Library\\resources
    

    You will find Firefox browser profiles which are used by selenium library. It might be very useful to update the provided profile with whatever Firefox profile you wish. This is a great advantage and saves allot of test setup time if you need the profile to be configured in a certain way instead of programmatically doing that. You could maintain several copies of different profiles there and then copy the one you need at run time just before invoking robot. Here is a sample run.bat file that does that:

    @echo off
    @rmdir %SEL2RES%\firefoxprofile /q /s
    @mkdir %SEL2RES%\firefoxprofile
    @xcopy %SEL2RES%\my_ff_profile  %SEL2RES%\firefoxprofile /e /f /y
    @echo on
    call pybot.bat --argumentfile c:\\robot\\arg\\arg.txt C:\\robot\\suites\\test.txt
    

    Where %SEL2RES% is an environment variable that points to the resources directory in robot framework selenium library.

  3. Window Minimized
  4. Sikuli remote server and Jenkins should start minimized in order not to interfere with Sikuli while image based testing is executing. To do that create a windows shortcut for both Sikuli remote server batch file and Jenkins batch file then right click the link and go to properties. There you can chose to run the command as minimized.

That is it for today, If you have questions about the code please contact me as the main focus of this article was on the integration side not on the actual test cases. Thanks for reading.

Search Terms...
Leave a comment ?

23 Comments.

  1. hi, this is a really interesting article. only one thing I could not find was the SikuliRemoteLibrary.py – can you help and provide a download link. Thank you.

  2. Mohammed Abualrob

    It is at the bottom of the article

  3. For the step:
    Run the remote server. Go to: C:\robot\libs and type: jython.bat SikuliRemoteLibrary.py

    I seem to be getting an error on line 8: self.SS = Screen() saying that Screen is a NoneType object and is not callable. All the other steps worked fine for me.

    Also may I suggest you to add correct indents to SikuliRemoteLibrary.py for the try, except blocks.

    Thanks!

  4. Hi,
    I got an error when running SikuliRemoteLibrary.py :
    ” File “SikuliRemoteLibrary.py”, line 3, in
    from org.sikuli.script import *
    ImportError: No module named sikuli ”

    Could you pls correct it ?

    Thanks a lot in advance!

  5. @Mohammed Abualrob Thanks in advance
    It works well 😛
    @everyone
    You can download file SikuliRemoteLibrary.py at :
    https://docs.google.com/file/d/0B_VTRRrTuA3PckdIZTBwS1dHVkU/edit?pli=1.

    The problem is that those data contained a lot of tabs,spaces…To fix it you need:
    — Download IDLE Python
    — Open your file with IDLE. Press key: Alt + 6 then input value 2. Then OK.

  6. At the moment I want to build TestNG Framework. I want to use pure JAVA language. TestNG Framework doesn’t support remote execution. Selenium Grid supports it. To can run remote execution with Sikuli I use XML-RPC technique.

  7. I had the same problem as Henry Lu.

    ” File “SikuliRemoteLibrary.py”, line 3, in
    from org.sikuli.script import *
    ImportError: No module named sikuli ”

    I eventually got it fixed, and it turned out I missed some extra environment variables.

    I put the following into jythonsikuli.bat and run that instead of jython.bat to make it work:

    @echo off
    set JYTHONPATH=c:\sikuli\sikuli-script.jar\Lib
    set CLASSPATH=c:\sikuli\sikuli-script.jar
    set SIKULI_HOME=c:\sikuli\
    set PATH=%PATH%c:\sikuli\libs;
    jython.bat %*

    I saved this bat-file to my jython2.5.2\bin directory.

    Hope this helps!

  8. Hello
    According your article, on xp-32bit system it works well.
    On Win7-64 bit system, when I run jython.bat robotremoteserver.py, it pops up importError:No module named inspect. I add inpect.py to the folder, there will pops up other error.

  9. Get an error for the last line of test.txt:-

    Click Object ${data_path}\maximize.png

    the error says, image not found on disk, null error,

    please advise

  10. Hi vinod,
    Try to use the DOUBLE backslash in your path, like C:\\datapath\\maximize.png

  11. Thanks a lot Hahadali !!! It works for me 🙂

  12. Thanks for the detail description, I like the way you explain all the implementation. I owe you bier, just call me If you visit Dusseldorf Germany!

  13. where is the SikuliRemoteLibrary.py? i can’t find it everywhere.

  14. I have a sikuli script for opening a firefox browser by doubleclick icon on my windows7 desktop and I am able to run this from command line. Its working as expected.
    I want to run the same script from robot framework without using selenium library.

    I have integrated both sikuli and RFW. But I am not getting the keywords part. What are the keywords we need to use so that both RFW and sikuli script will run simutaneously. Please help me on how to do this.

  15. This Example in one system or Two systems ?

  16. Is this in one System or Two ??

  17. command: pybot.bat –argumentfile c:\users\ee208473\appdata\local\temp\RIDEfsjogn.d\argfile.txt –listener C:\Python27\lib\site-packages\robotide\contrib\testrunner\TestRunnerAgent.py:52992:False C:\RoboTrailAnjali\suites\tester2
    unexpected error: [ ERROR ] Suite ‘Tester2’ contains no tests. I am getting this error just before doing continous integration what todo

  18. command: C:\jython2.5.4rc1\bin>jythonsikuli.bat c:\robot\libs\SikuliRemoteLibrary.py
    Usage: java [-options] class [args…]
    (to execute a class)
    or java [-options] -jar jarfile [args…]
    (to execute a jar file)

    where options include:
    -client to select the “client” VM
    -server to select the “server” VM
    -hotspot is a synonym for the “client” VM [deprecated]
    The default VM is client.

    -cp
    -classpath
    A ; separated list of directories, JAR archives,
    and ZIP archives to search for class files.
    -D=
    set a system property
    -verbose[:class|gc|jni]
    enable verbose output
    -version print product version and exit
    -version:
    require the specified version to run
    -showversion print product version and continue
    -jre-restrict-search | -jre-no-restrict-search
    include/exclude user private JREs in the version search
    -? -help print this help message
    -X print help on non-standard options
    -ea[:…|:]
    -enableassertions[:…|:]
    enable assertions
    -da[:…|:]
    -disableassertions[:…|:]
    disable assertions
    -esa | -enablesystemassertions
    enable system assertions
    -dsa | -disablesystemassertions
    disable system assertions
    -agentlib:[=]
    load native agent library , e.g. -agentlib:hprof
    see also, -agentlib:jdwp=help and -agentlib:hprof=help
    -agentpath:[=]
    load native agent library by full pathname
    -javaagent:[=]
    load Java programming language agent, see java.lang.instrument

    -splash:
    show splash screen with specified image
    HOW TO GET PORT AND SERVER IMFORMATION?

  19. Hi,

    I’m sharing this library (SikuliXRobotLibrary) to anyone interested. The libary uses SikuliXv 1.1.0 methods. The installation instructions and how to run the demo test are in the following github link as well:

    https://github.com/jaredfin/SikuliXRobotLibrary

  20. jythonsikuli.bat file content is below.

    @echo off
    set JYTHONPATH=c:\sikuli\sikuli-script.jar\Lib
    set CLASSPATH=c:\sikuli\sikuli-script.jar
    set SIKULI_HOME=c:\sikuli\
    set PATH=%PATH%c:\sikuli\libs;
    jython.bat %*

    I executed like below I got following error.

    C:\jython2.7.0\bin>jythonsikuli.bat C:\robot\libs\SikuliRemoteLibrary.py
    The input line is too long.

    C:\jython2.7.0\bin>

  21. I want to use SIKULI + Robot Framework for automating a Windows application, I do not want any Selenium Library.. I have followed your perfect tutorial and installed everything.

    ROBOT framework 3.0
    Python 2.7
    JYTHON
    RIDE

    Could you help me in integrating these without the issue of python and selenium not being able to use same interpreter.

    I actually tried a lot to follow this tutorial
    http://blog.mykhailo.com/2011/02/how-to-sikuli-and-robot-framework.html
    but its very complex and i dont understand.

    PS: I came from quora 🙂

  22. How use keywords from Sikuli in RIDE?

  23. how can I import selenium to sikuli ide ? If I dont want to further make the system complex?

Leave a Reply

%d bloggers like this: