August 31, 2012
HomeArticles and TutorialsIntegrating Arrow with Sikuli

Integrating Arrow with Sikuli

By  Articles and Tutorials, Software Quality Assurance  2 Comments

What is Arrow

Arrow is a test automation framework that promotes test driven development (TDD). The framework was developed by Yahoo! and recently released as open source project on Github. For more information about Arrow you can visit the project site at: https://github.com/yahoo/arrow

What is Sikuli

Sikuli is an academic and research project at the MIT. It is an open source image based technology to automate testing graphical user interfaces. For more information about Sikuli please visit their web site at: http://sikuli.org

Installation on Linux

This article assumes that you have access to a Linux machine (In our case we will be using Ubuntu Linux flavor). If you do not have a physical Linux machine you can use a virtual Linux machine. For more information on how to run Ubuntu Linux on Windows check out this article: https://www.8bitavenue.com/2012/08/how-to-run-linux-on-windows

Preparing the Environment

Which Unix shell to use is a personal choice but I prefer working with TCSH. Here is how to install TCSH:

    1. Get the binary:

C++
1
sudo apt-get install tcsh

    1. Change the shell:

C++
1
chsh then enter /bin/tcsh

    1. Set some environment variables at startup

C++
1
2
3
4
5
vi ~/.tcshrc
set prompt="%B%/>"
setenv LD_LIBRARY_PATH /usr/local/lib
alias cls clear
alias ll ls -al

Installing Subversion and Git

Apply the following commands:

C++
1
2
3
4
sudo apt-get install subversion
where svn
sudo apt-get install git
git --version

Installing OpenCV

Sikuli uses OpenCV computer vision library which is written in C++. Installation from source on Linux is probably a problematic task. If you are having difficulty installing the library you can refer to the following article which outlines all the steps to get this job done:

https://www.8bitavenue.com/2012/08/installing-opencv-on-ubuntu

Installing Node.js

Arrow framework uses Node.js (server side java script). Here is how to install node on Linux:

C++
1
2
3
4
5
6
7
8
9
sudo apt-get install g++ curl libssl-dev apache2-utils
sudo apt-get install git-core
git clone https://github.com/joyent/node.git
cd node
git checkout v0.6.18
./configure
make
sudo make install
node -v

Installing Node Package Manager (NPM)

C++
1
2
curl https://npmjs.org/install.sh | sudo sh
npm -v

Installing Arrow Framework

C++
1
2
3
4
sudo npm install -g yahoo-arrow
where arrow
where arrow_server
where arrow_selenium

Installing Arrow Tutorial

C++
1
git clone https://github.com/arrowfmk/arrow_tutorial

Installing Java Development Kit (JDK)

C++
1
2
3
//Search for the right name
apt-cache search jdk
sudo apt-get install openjdk-6-jdk openjdk-6-jre

Install Selenium Stand Alone Server

Search for selenium stand alone server for example: http://selenium.googlecode.com/files/selenium-server-standalone-2.22.0.jar

C++
1
Run as: java -jar path/to/selenium-server.jar

Installing Eclipse

Sikuli is distributed as a Java jar file. This means we can use Java directly to access the imaging API or use a scripting language such as Jython. If you prefer Java you can take a look at the following article which outlines how to install Eclipse on Ubuntu Linux:

https://www.8bitavenue.com/2012/08/installing-eclipse-on-ubuntu

Installing Sikuli on Linux

Follow the steps below:

C++
1
2
3
4
5
6
7
8
9
10
11
12
13
14
//Prerequisites
sudo apt-get install wmctrl
 
//Get the setup file
wget launchpad.net/sikuli/sikuli-x/x1.0-rc3/+download/Sikuli-X-1.0rc3%20%28r905%29-linux-x86_64.zip
 
//Extract the file
unzip Sikuli-X-1.0rc3%20%28r905%29-linux-x86_64.zip
 
//Go to directory
cd Sikuli-X-1.0rc3 (r905)-linux-x86_64/Sikuli-IDE
 
//Launch it
./sikuli-ide.sh

Installing Jython on Linux

Follow the steps below:

C++
1
2
3
4
5
6
sudo apt-get install jython
 
//Java class path should point to sikuli jar file
vi ~/.tcshrc
//Paste
setenv CLASSPATH ~/lib/sikuli-script.jar

Installing XML-RPC JavaScript Module

C++
1
2
3
4
//Go to home directory
mkdir lib
cd lib
npm install xmlrpc

Installing Sikuli Server

We are going to expose Sikuli API using a remote XML-RPC server. Here is the jython script for the server:

Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
# -*- coding: utf-8 -*-
 
import sys
import xmlrpclib
from SimpleXMLRPCServer import SimpleXMLRPCServer
from org.sikuli.script import *
 
class SikuliRemoteServer:
 
#---------------------------------------------------------
#                                       INITIALIZE LIBRARY
#---------------------------------------------------------
 
    def __init__(self):
        print "INFO: Sikuli Remote Server Initializing..."
        self.SS = Screen()
        self.PT = Pattern()
        
#------------------------------------------------
#                                            WAIT
#------------------------------------------------
 
    def _wait(self, imgFile, timeOut, similarity):
 
        self.PT = Pattern(imgFile)
        self.PT = self.PT.similar(float(similarity))
        self.SS.wait(self.PT, float(timeOut))            
 
#----------------------------------------------------
#                                          MOUSE MOVE
#----------------------------------------------------
        
    def mouse_move(self, x=0, y=0):
 
        print "INFO: x = " + str(x)
        print "INFO: Mouse Move: y = " + str(y)
 
        try:
            self.SS.mouseMove(Location(x, y))
        except FindFailed, err:
            raise AssertionError("Could not move mouse")
 
#------------------------------------------------------
#                                    RIGHT CLICK OBJECT
#------------------------------------------------------
        
    def right_click_object(self, img, timeOut=10, sim=0.85):
 
        print "INFO: Right Click Object: Image File = " + img
        print "INFO: Right Click Object: Time Out   = " + str(timeOut)
        print "INFO: Right Click Object: Similarity = " + str(sim)
 
        try:
            self._wait(imgFile, timeOut, sim)
            self.SS.rightClick(img)
        except FindFailed, err:
            raise AssertionError("Cannot right click [" + img + "]")
 
#---------------------------------------------------
#                                       CLICK OBJECT
#---------------------------------------------------
        
    def click_object(self, imgFile, timeOut=10, similarity=0.85):
 
        print "INFO: Click Object: Image File = " + imgFile
        print "INFO: Click Object: Time Out   = " + str(timeOut)
        print "INFO: Click Object: Similarity = " + str(similarity)
 
        try:
            self._wait(imgFile, timeOut, similarity)
            self.SS.click(imgFile)
        except FindFailed, err:
            raise AssertionError("Cannot  click [" + imgFile + "]")
 
#----------------------------------------------------
#                                       OBJECT EXISTS
#----------------------------------------------------
 
    def object_exists(self, imgFile, sim=0.85, timeOut=15, noa=False):
 
        print "INFO: Object Exists: Image File = " + imgFile
        print "INFO: Object Exists: Similarity = " + str(sim)
        print "INFO: Object Exists: Time Out   = " + str(timeOut)
 
        try:
            self._wait(imgFile, timeOut, sim)
            return True
        except FindFailed, err:
            if noa:
                return False
            raise AssertionError("Cannot find [" + imgFile + "]")
 
#-----------------------------------------------------
#                                       TYPE AT OBJECT
#-----------------------------------------------------
 
    def type_at_object(self, imgFile, txt, timeOut=10, sim=0.85):
 
        print "INFO: Type At Object: Image File = " + imgFile
        print "INFO: Type At Object: Text       = " + txt
        print "INFO: Type At Object: Time Out   = " + str(timeOut)
        print "INFO: Type At Object: Similarity = " + str(sim)
 
        try:
            self._wait(imgFile, timeOut, sim)
            self.SS.type(imgFile, txt)            
        except FindFailed, err:
            raise AssertionError("Cannot type at [" + imgFile + "]")
 
#------------------------------------------------------
#                                       PASTE AT OBJECT
#------------------------------------------------------
 
    def paste_at_object(self, imgFile, txt, timeOut=10, sim=0.85):
 
        print "INFO: Paste At Object: Image File = " + imgFile
        print "INFO: Paste At Object: Text       = " + txt
        print "INFO: Paste At Object: Time Out   = " + str(timeOut)
        print "INFO: Paste At Object: Similarity = " + str(sim)
 
        try:
            self._wait(imgFile, timeOut, sim)
            self.SS.paste(imgFile, txt)            
        except FindFailed, err:
            raise AssertionError("Cannot paste at [" + imgFile + "]")
 
#------------------------------------------------------
#                                       HOVER ON OBJECT
#------------------------------------------------------
 
    def hover_on_object(self, imgFile, timeOut=10, similarity=0.85):
 
        print "INFO: Hover On Object: Image File = " + imgFile
        print "INFO: Hover On Object: Time Out   = " + str(timeOut)
        print "INFO: Hover On Object: Similarity = " + str(similarity)
 
        try:
            self._wait(imgFile, timeOut, similarity)
            self.SS.hover(imgFile)
            return True
        except FindFailed, err:
            print "Cannot hover on [" + imgFile + "]"
            return False
            
#----------------------------------------------------
#                                       LAUNCH SERVER
#----------------------------------------------------
 
if __name__ == '__main__':
 
    print "INFO: Sikuli Remote Server Listening on Port 1975..."
 
    srs = SikuliRemoteServer()
 
    # Create server
    server = SimpleXMLRPCServer(("localhost", 1975))
 
    # Register methods
    server.register_function(srs.mouse_move, "mouse_move")
    server.register_function
    (
        srs.right_click_object,
        "right_click_object"
    )
    server.register_function(srs.click_object, "click_object")
    server.register_function(srs.object_exists, "object_exists")
    server.register_function(srs.type_at_object, "type_at_object")
    server.register_function(srs.paste_at_object, "paste_at_object")
    server.register_function(srs.hover_on_object, "hover_on_object")
 
    # Listen for connections
    server.serve_forever()

Sikuli XML-RPC JavaScript Client

As indicated earlier, Sikuli API is exposed using XML-RPC server implemented in jython. Arrow framework test cases written in JavaScript should be able to communicate with Sikuli server using the following JavaScript XML-RPC client:

JavaScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
//Use xmlrpc module
var xmlrpc = require('xmlrpc')
 
//Create client object
//Set host and port
var client = xmlrpc.createClient
({
host: 'localhost',
port: 1975,
path: '/'
})
 
//Send request
//Specify method
//and parameters
client.methodCall
(
'object_exists',
['1.png'],
//Get response
function (error, value)
{
     console.log('Error: ' + error + ' Value: ' + value)
   }
)

Arrow Communicating with Sikuli

In order to make sure a java script based test case in Arrow can talk to Sikuli server do the following:

JavaScript
1
2
3
4
5
6
7
8
//Copy or save the files
Copy sikuli-client.js into ~/lib
Copy sikuli-server.py into ~/lib
Copy sikuli-script.jar into ~/lib
 
//Launch server then run client
Run server as: jython sikuli-server.py
Run client as: node sikuli-client.js

That is it, if you have any questions please contact me. Thanks for reading.

More from my site

Tags:,

About Author

Mohammed Abualrob

Software Engineer @ Cisco

2 Comments
  1. David

    Curious to know, how was the SikuliRemoteServer developed? Good library/server by the way. The Sikuli server library keyword method names seem similar to the Java version I wrote a while back:

    http://code.google.com/p/simplesikuli/source/browse/trunk/src/org/robotframework/remotelibrary/SikuliLibrary.java

    so I assume you or someone adapted against my keyword model, unless we all happened to use the same naming convention. 🙂

    Reply
  2. Mohammed Abualrob

    Yes I was inspired by you however I went the python way. In order to drive a Java based library using python one has to use Jython interpreter but since Robot is python based I had to run it as a remote XML RPC service. Regarding the server I just borrowed an open source python based XML RPC server.

    Reply

Add a Comment

Your email address will not be published. Required fields are marked *