Jun 172009
The Mega Powers (Hulk Hogan and Randy Savage) ...

Recently I’ve been working on a REST API for reporting workflow status information in Alfresco.  After getting some of the functionality nailed down, it really bothered me that I wasn’t able to use Test Driven Development (TDD) in the process.  So I went looking, and I found quite a few open source tools out there in the wild that made good prospects for acceptance testing these REST APIs that I was working on.  It was time for a SMACKDOWN! OOOOOOOH YEAAAAAAAH!

Contender #1 – Selenium

I heard of Selenium in the past and have wanted to tinker with it for a long time, so I tried this one out first.  Note that Selenium is really just the “brand name”; there are actually several inter-related offerings here.  The first is Selenium IDE, which comes as an add-on to Firefox, which is pretty awesome.  Using this tool, you can basically record your tests, and play them back.  This is very easy to try out – literally within minutes I had recorded my first tests.

That was very cool, but I needed to test result sets with dynamic data, so I had to take a look at Selenium-RC, which has APIs that enable the use of your favorite programming language: Java, C#, Perl, PHP, Python, or Ruby.  The good news with this tool is that it actually uses the real browser to do its testing.  That’s also the bad news.  For each and every test, a new Firefox was launched, which would certainly take a while as the test suite grew larger.  The main advantage that I saw with this tool is that it would be great for testing web applications with Javascript for cross-browser compatibility.  In fact, Selenium-RC is leveraged by Selenium Grid, which allows you to test cross-browser and on different operating systems.  I dig it, but I just have some simple REST APIs to test, so the whole Selenium suite is overkill for me.

Reporting of results using Selenium-RC ultimately would have to be via JUnit reporting mechanisms, which are pretty decent as I recall, but you will have an extra step to set that up in your Ant build file.

Oh, one other note.  You can use Selenium IDE to record your tests and export them as a Selenium-RC Java class.  Pretty cool, but the implementation felt like it was quick and dirty as I recall – something about the class name I gave it and the source that was generated was ‘off’.

Contender #2 – HTMLUnit

This is basically just a Java API that makes it easy to extract information from web pages programatically.  Because I knew that several other tools were built using HTMLUnit at the core, I didn’t spend any time investigating the possibility of using it directly, which was a smart decision.

Contender #3 – JWebTest

This one is cool in concept.  It’s basically an abstraction of Selenium and HTMLUnit.  If you want to test using HTMLUnit most of the time for speed but want to switch to Selenium for a cross browser sanity check from time to time, JWebTest could be your answer.  I spent maybe an hour setting it up in my environment and writing a test with it.  It didn’t handle HTTP basic authentication right out of the box (when using a URL like “http://admin:admin@localhost:8080/alfresco/my/rest/service.json”), whereas Selenium did handle such a URL properly.  I did look (just now) to see if it handles HTTP basic authentication, and it looks like it does via the API – see WebTestCase.getTestContext() and TestContext.setAuthorization().

Regardless, the bottom line with this is that it’s a Java based API that I would have to use to program all of my tests, and the ability to switch on “Selenium mode” isn’t very compelling to me for testing REST APIs.  Therefore, this one doesn’t really add much value over using HTMLUnit directly for this use case.

Contender #4 – Canoo WebTest

Initially, my first reaction regarding Canoo was “Oh man, I don’t like that it uses Ant so heavily”.  For reasons I won’t go into here, I’m using Ant (not Maven) as my build tool for this project, so ultimately, that’s not a deal breaker.  That said, getting up and running with Canoo was pretty awesome.  The instructions say to put the WebTest bin directory in your path, which generally bothers me, but I did it anyway.  Then there’s a way to generate a project skeleton (a very maven-esque thing to do), which I did, and by running their shell script within the generated test project’s directory, I was off and running.  I then created my own test files (in Ant based XML with custom Canoo tasks), plugged them into the main build script, and BOOYAH!  I was off to the races.

The execution of the tests is pretty fast, certainly faster than Selenium, and result reporting is tight:


Canoo WebTest also has the advantage that adding new tests is a declarative exercise – no programming and compilation required.  HTTP Basic authentication is handled nicely via simple attributes on the <invoke> step.  Here’s an example:

        <webtest name="Check end date capability for assigned tasks">
            <invoke url="http://localhost:8080/alfresco/service/api/workflow/status/user/admin.json?endDate=2009-05-23"
                    description="Admin with end date 2009-05-23"
            <verifyText text='{"description":"Review","priority":2,"due":null,"properties":null,"percent":0,"completed":null,"status":"Not Yet Started","duration":null,"created":"2009-05-23 23:59:59.0","name":"wf:reviewTask"}'/>
                <verifyText text='{"description":"Adhoc Task","priority":2,"due":null,"properties":null,"percent":0,"completed":null,"status":"Not Yet Started","duration":null,"created":"2009-05-24 00:00:00.0","name":"wf:adhocTask"}'/>

The Winner – Canoo WebTest

Just to spell it out clearly: Canoo is my tool of choice for REST API testing, due to ease of use, speed of execution, good reporting, and easy handling of HTTP basic authentication per test.  If/when I move to using a Maven build system, it looks like that’s alright by Canoo, since they have a maven plugin.

Other Alternatives

Other possibilities for folks out there are:

  • Celerity – Ruby based testing framework.  Not for me since I’m not a Ruby wonk.
  • JSFUnit – Specifically geared towards testing JSF applications, which is not the case here.
  • WebDriver – Similar to Selenium, and is in fact rolling into Selenium according to the FAQ.  As such I didn’t look at this for longer than 5 minutes.
Reblog this post [with Zemanta]

  20 Responses to “Open Source Testing Tool Smackdown for REST Web Services”

  1. On behalf of James Lorenzen, who pointed out that leaving a comment on my blog is problematic (which I intend to fix very soon):

    Great article. I did the same thing not too long ago for our team and we decided on using groovy to test our REST services. See my article here http://jlorenzen.blogspot.com/2009/01/testing-rest-services-with-groovy.html. I also ran across this post yesterday discussing the same issue and he is using the grails functional-test plugin and it looks super simple http://thediscoblog.com/2009/06/15/grails-hip-tip-testing-restful-services/. I don’t think I would ever knowingly write tests in XML, not matter how easy it was. I am so anti-XML today it’s not even funny. Because of that I would us gant (groovy ant) instead of ant if I had a choice. Again, I don’t want to stare at ant xml files all day.

  2. Great recommendation on canoo. This is exactly what I was looking for. Easy to setup, configure, tons of baked in functionality. Thanks.

  3. marketing…

    [...]BRob On Tech » Blog Archive » Open Source Testing Tool Smackdown for REST Web Services[...]…

  4. Surprised nobody has tried JMeter: http://jmeter.apache.org/

  5. Have you also tried WCFStorm.Rest? ? http://www.wcfstorm.com/wcf/learn-more-rest.aspx

  6. Good write-up, I¡¦m regular visitor of one¡¦s web site, maintain up the nice operate, and It is going to be a regular visitor for a lengthy time.

  7. noi that chung cu gia re

  8. This is because this device assures to either benefit or cure the autistic
    people in more ways than one. In addition, the observing surgeons could transmit
    their comments to the operating surgeon, who could read them on the Google Glass monitor.
    But it seems Memorial Day wasn’t important enough to Google.

  9. Bon ce midi par contre j’ai grave tap dedans avec une galette des rois, la part 15 pp ouie! il me reste encore 2 jours tenir pour g rer ma r serve propoints, j’esp re que le repas “basket nike 2013 homme http://www.mrshatfield.com/SMS/Binder/gm2/3486.asp” de demain sera l ger. Pour ce soir j’ai regarder le menu du resto j’h site entre des pates, ou une tartine v g tarienne avec de la mozza, je crois que ce serais peut tre le moins gras.Revenir en haut de pageSupertoinette “basket nike 2013 homme http://www.mrshatfield.com/SMS/Binder/gm2/3486.asp” D du ForumLes r gimes et leurs recettes Conseils et entraide se basant sur la m thode Weight Watchers Aller la page Pr 1, 2, 3, 4, 5, 6, 7 SuivantePage 5 sur 7 Sujets similaires: Combien de propoint journalier Calcul capital point avec propoint 2 Nouvelle : propoint Naturhouse 2013 D marrage du r gime dukan Propoint Propoint Protifast d but 1er f vrier 2013 Calcul propoint 2 (clic) Suite.”basket nike 2013 homme http://www.mrshatfield.com/SMS/Binder/gm2/3486.asp

  10. A fascinating discussion is definitely worth comment. There’s no doubt that that you ought to publish more about this issue, it might not be a taboo matter
    but typically folks don’t talk about such topics. To the next!
    Kind regards!!

  11. Thanks a ton! That was very informative, I just saved your site.

  12. If you are going the Android route then take a look at the
    4G enabled phones such as the HTC EVO, Nexus S (Sprint’s first ‘pure’ Google device), and the Samsung Epic.
    The Ouya (which proved this with its wildly successful Kickstarter campaign) was made available to consumers
    just a couple of months ago at a cost of $99. Today in this market
    of mobile there is too much competition due to this competition developing a mobile application is not easy goal.

  13. And another advantage of this platform is open source Android OS supports third party applications
    and developers of free access to all countries.
    One of the greatest video games to arrive in the gaming world lately is Minecraft.
    There are two versions, one written in Python and one written in Objective C targeting i – OS.

  14. El segundo proyecto de Anna & Eugeni Bach consiste en otra
    reforma de una vivienda en el Ensanche barcelonés del año 1910.

  15. En el caso de que necesite una compañía de reformas en Vic póngase en contacto
    con nosotros para el presupuesto de su obra.

  16. It is one of the most downloaded sexy Android apps in the
    Android Market. It has the capacity to provide information that the user
    require, for example weather report or sport outcomes, also it can give facts about the things that took place on nearby locations.
    Wikipad is now the thinnest and lightest Android
    equipped tablet with 1.

  17. Digital Video Expo West and DV Film Fest runs from December 4-6 in Los Angeles, and offers a line-up of
    educational forums, including Apple and Avid certification training, sessions covering digital
    video technology from A to Z, and specialized tracks featuring industry hot topics.
    There is a range of apps available on Google Play catering to all kinds
    of users. We know that Apple Company created such a great product,
    Apple TV automatically grabs the digital media on your
    computer and makes it available for you to enjoy
    on your TV.

  18. I think that what you published was very reasonable.
    However, what about this? suppose you were to write a killer post title?
    I ain’t suggesting your information isn’t good., however suppose
    yyou addeed something that grabbed people’s attention?
    I mean Open Source Testing Tool Smackdown for REST Weeb Services  BRob On Tech
    is a littlpe vanilla. You should look at Yahoo’s home page and see how they write article titles too get people
    interested. You might aadd a related video or a picture or two to grab
    readers interested about everything’ve got to say. In my opinion, it would make yur
    posts a little livelier. http://www.sweetvacationspot.com

 Leave a Reply



You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>