Integrating Arrow with Sikuli

What is Arrow

Arrow is a test automation framework that promotes test driven development (TDD). The framework was developed by Yahoo! and recently released as open source project on Github. For more information about Arrow you can visit the project site at: https://github.com/yahoo/arrow

What is Sikuli

Sikuli is an academic and research project at the MIT. It is an open source image based technology to automate testing graphical user interfaces. For more information about Sikuli please visit their web site at: http://sikuli.org

Installation on Linux

This article assumes that you have access to a Linux machine (In our case we will be using Ubuntu Linux flavor). If you do not have a physical Linux machine you can use a virtual Linux machine. For more information on how to run Ubuntu Linux on Windows check out this article: http://www.8bitavenue.com/2012/08/how-to-run-linux-on-windows

Preparing the Environment

Which Unix shell to use is a personal choice but I prefer working with TCSH. Here is how to install TCSH:

    1. Get the binary:
sudo apt-get install tcsh
    1. Change the shell:
chsh then enter /bin/tcsh
    1. Set some environment variables at startup
vi ~/.tcshrc
set prompt="%B%/>"
setenv LD_LIBRARY_PATH /usr/local/lib
alias cls clear
alias ll ls -al

Installing Subversion and Git

Apply the following commands:

sudo apt-get install subversion
where svn
sudo apt-get install git
git --version

Installing OpenCV

Sikuli uses OpenCV computer vision library which is written in C++. Installation from source on Linux is probably a problematic task. If you are having difficulty installing the library you can refer to the following article which outlines all the steps to get this job done:

http://www.8bitavenue.com/2012/08/installing-opencv-on-ubuntu

Installing Node.js

Arrow framework uses Node.js (server side java script). Here is how to install node on Linux:

sudo apt-get install g++ curl libssl-dev apache2-utils
sudo apt-get install git-core
git clone https://github.com/joyent/node.git
cd node
git checkout v0.6.18
./configure
make
sudo make install
node -v

Installing Node Package Manager (NPM)

curl https://npmjs.org/install.sh | sudo sh
npm -v

Installing Arrow Framework

sudo npm install -g yahoo-arrow
where arrow
where arrow_server
where arrow_selenium

Installing Arrow Tutorial

git clone https://github.com/arrowfmk/arrow_tutorial

Installing Java Development Kit (JDK)

//Search for the right name
apt-cache search jdk
sudo apt-get install openjdk-6-jdk openjdk-6-jre

Install Selenium Stand Alone Server

Search for selenium stand alone server for example: http://selenium.googlecode.com/files/selenium-server-standalone-2.22.0.jar

Run as: java -jar path/to/selenium-server.jar

Installing Eclipse

Sikuli is distributed as a Java jar file. This means we can use Java directly to access the imaging API or use a scripting language such as Jython. If you prefer Java you can take a look at the following article which outlines how to install Eclipse on Ubuntu Linux:

http://www.8bitavenue.com/2012/08/installing-eclipse-on-ubuntu

Installing Sikuli on Linux

Follow the steps below:

//Prerequisites
sudo apt-get install wmctrl

//Get the setup file
wget launchpad.net/sikuli/sikuli-x/x1.0-rc3/+download/Sikuli-X-1.0rc3%20%28r905%29-linux-x86_64.zip

//Extract the file
unzip Sikuli-X-1.0rc3%20%28r905%29-linux-x86_64.zip

//Go to directory
cd Sikuli-X-1.0rc3 (r905)-linux-x86_64/Sikuli-IDE

//Launch it
./sikuli-ide.sh

Installing Jython on Linux

Follow the steps below:

sudo apt-get install jython

//Java class path should point to sikuli jar file
vi ~/.tcshrc
//Paste
setenv CLASSPATH ~/lib/sikuli-script.jar

Installing XML-RPC JavaScript Module

//Go to home directory
mkdir lib
cd lib
npm install xmlrpc

Installing Sikuli Server

We are going to expose Sikuli API using a remote XML-RPC server. Here is the jython script for the server:

# -*- coding: utf-8 -*-

import sys
import xmlrpclib
from SimpleXMLRPCServer import SimpleXMLRPCServer
from org.sikuli.script import *

class SikuliRemoteServer:

#---------------------------------------------------------
#                                       INITIALIZE LIBRARY
#---------------------------------------------------------

    def __init__(self):
        print "INFO: Sikuli Remote Server Initializing..."
        self.SS = Screen()
        self.PT = Pattern()
        
#------------------------------------------------
#                                            WAIT
#------------------------------------------------

    def _wait(self, imgFile, timeOut, similarity):

        self.PT = Pattern(imgFile)
        self.PT = self.PT.similar(float(similarity))
        self.SS.wait(self.PT, float(timeOut))            

#----------------------------------------------------
#                                          MOUSE MOVE
#----------------------------------------------------
        
    def mouse_move(self, x=0, y=0):

        print "INFO: x = " + str(x)
        print "INFO: Mouse Move: y = " + str(y)

        try:
            self.SS.mouseMove(Location(x, y))
        except FindFailed, err:
            raise AssertionError("Could not move mouse")

#------------------------------------------------------
#                                    RIGHT CLICK OBJECT
#------------------------------------------------------
        
    def right_click_object(self, img, timeOut=10, sim=0.85):

        print "INFO: Right Click Object: Image File = " + img
        print "INFO: Right Click Object: Time Out   = " + str(timeOut)
        print "INFO: Right Click Object: Similarity = " + str(sim)

        try:
            self._wait(imgFile, timeOut, sim)
            self.SS.rightClick(img)
        except FindFailed, err:
            raise AssertionError("Cannot right click [" + img + "]")

#---------------------------------------------------
#                                       CLICK OBJECT
#---------------------------------------------------
        
    def click_object(self, imgFile, timeOut=10, similarity=0.85):

        print "INFO: Click Object: Image File = " + imgFile
        print "INFO: Click Object: Time Out   = " + str(timeOut)
        print "INFO: Click Object: Similarity = " + str(similarity)

        try:
            self._wait(imgFile, timeOut, similarity)
            self.SS.click(imgFile)
        except FindFailed, err:
            raise AssertionError("Cannot  click [" + imgFile + "]")

#----------------------------------------------------
#                                       OBJECT EXISTS
#----------------------------------------------------

    def object_exists(self, imgFile, sim=0.85, timeOut=15, noa=False):

        print "INFO: Object Exists: Image File = " + imgFile
        print "INFO: Object Exists: Similarity = " + str(sim)
        print "INFO: Object Exists: Time Out   = " + str(timeOut)

        try:
            self._wait(imgFile, timeOut, sim)
            return True
        except FindFailed, err:
            if noa:
                return False
            raise AssertionError("Cannot find [" + imgFile + "]")

#-----------------------------------------------------
#                                       TYPE AT OBJECT
#-----------------------------------------------------

    def type_at_object(self, imgFile, txt, timeOut=10, sim=0.85):

        print "INFO: Type At Object: Image File = " + imgFile
        print "INFO: Type At Object: Text       = " + txt
        print "INFO: Type At Object: Time Out   = " + str(timeOut)
        print "INFO: Type At Object: Similarity = " + str(sim)

        try:
            self._wait(imgFile, timeOut, sim)
            self.SS.type(imgFile, txt)            
        except FindFailed, err:
            raise AssertionError("Cannot type at [" + imgFile + "]")

#------------------------------------------------------
#                                       PASTE AT OBJECT
#------------------------------------------------------

    def paste_at_object(self, imgFile, txt, timeOut=10, sim=0.85):

        print "INFO: Paste At Object: Image File = " + imgFile
        print "INFO: Paste At Object: Text       = " + txt
        print "INFO: Paste At Object: Time Out   = " + str(timeOut)
        print "INFO: Paste At Object: Similarity = " + str(sim)

        try:
            self._wait(imgFile, timeOut, sim)
            self.SS.paste(imgFile, txt)            
        except FindFailed, err:
            raise AssertionError("Cannot paste at [" + imgFile + "]")

#------------------------------------------------------
#                                       HOVER ON OBJECT
#------------------------------------------------------

    def hover_on_object(self, imgFile, timeOut=10, similarity=0.85):

        print "INFO: Hover On Object: Image File = " + imgFile
        print "INFO: Hover On Object: Time Out   = " + str(timeOut)
        print "INFO: Hover On Object: Similarity = " + str(similarity)

        try:
            self._wait(imgFile, timeOut, similarity)
            self.SS.hover(imgFile)
            return True
        except FindFailed, err:
            print "Cannot hover on [" + imgFile + "]" 
            return False
            
#----------------------------------------------------
#                                       LAUNCH SERVER
#----------------------------------------------------

if __name__ == '__main__':

    print "INFO: Sikuli Remote Server Listening on Port 1975..."

    srs = SikuliRemoteServer()

    # Create server
    server = SimpleXMLRPCServer(("localhost", 1975))

    # Register methods
    server.register_function(srs.mouse_move, "mouse_move")
    server.register_function
    (
        srs.right_click_object, 
        "right_click_object"
    )
    server.register_function(srs.click_object, "click_object")
    server.register_function(srs.object_exists, "object_exists")
    server.register_function(srs.type_at_object, "type_at_object")
    server.register_function(srs.paste_at_object, "paste_at_object")
    server.register_function(srs.hover_on_object, "hover_on_object")

    # Listen for connections
    server.serve_forever()

Sikuli XML-RPC JavaScript Client

As indicated earlier, Sikuli API is exposed using XML-RPC server implemented in jython. Arrow framework test cases written in JavaScript should be able to communicate with Sikuli server using the following JavaScript XML-RPC client:

//Use xmlrpc module
var xmlrpc = require('xmlrpc')

//Create client object
//Set host and port
var client = xmlrpc.createClient
({ 
	host: 'localhost', 
	port: 1975, 
	path: '/'
})

//Send request
//Specify method
//and parameters
client.methodCall
(
	'object_exists', 
	['1.png'], 
	//Get response
	function (error, value) 
	{
    		console.log('Error: ' + error + ' Value: ' + value)
  	}
)

Arrow Communicating with Sikuli

In order to make sure a java script based test case in Arrow can talk to Sikuli server do the following:

//Copy or save the files
Copy sikuli-client.js into ~/lib
Copy sikuli-server.py into ~/lib
Copy sikuli-script.jar into ~/lib

//Launch server then run client
Run server as: jython sikuli-server.py
Run client as: node sikuli-client.js

That is it, if you have any questions please contact me. Thanks for reading.

2 Comments

Leave a Reply