Jun 172009
The Mega Powers (Hulk Hogan and Randy Savage) ...

Recently I’ve been working on a REST API for reporting workflow status information in Alfresco.  After getting some of the functionality nailed down, it really bothered me that I wasn’t able to use Test Driven Development (TDD) in the process.  So I went looking, and I found quite a few open source tools out there in the wild that made good prospects for acceptance testing these REST APIs that I was working on.  It was time for a SMACKDOWN! OOOOOOOH YEAAAAAAAH!

Contender #1 – Selenium

I heard of Selenium in the past and have wanted to tinker with it for a long time, so I tried this one out first.  Note that Selenium is really just the “brand name”; there are actually several inter-related offerings here.  The first is Selenium IDE, which comes as an add-on to Firefox, which is pretty awesome.  Using this tool, you can basically record your tests, and play them back.  This is very easy to try out – literally within minutes I had recorded my first tests.

That was very cool, but I needed to test result sets with dynamic data, so I had to take a look at Selenium-RC, which has APIs that enable the use of your favorite programming language: Java, C#, Perl, PHP, Python, or Ruby.  The good news with this tool is that it actually uses the real browser to do its testing.  That’s also the bad news.  For each and every test, a new Firefox was launched, which would certainly take a while as the test suite grew larger.  The main advantage that I saw with this tool is that it would be great for testing web applications with Javascript for cross-browser compatibility.  In fact, Selenium-RC is leveraged by Selenium Grid, which allows you to test cross-browser and on different operating systems.  I dig it, but I just have some simple REST APIs to test, so the whole Selenium suite is overkill for me.

Reporting of results using Selenium-RC ultimately would have to be via JUnit reporting mechanisms, which are pretty decent as I recall, but you will have an extra step to set that up in your Ant build file.

Oh, one other note.  You can use Selenium IDE to record your tests and export them as a Selenium-RC Java class.  Pretty cool, but the implementation felt like it was quick and dirty as I recall – something about the class name I gave it and the source that was generated was ‘off’.

Contender #2 – HTMLUnit

This is basically just a Java API that makes it easy to extract information from web pages programatically.  Because I knew that several other tools were built using HTMLUnit at the core, I didn’t spend any time investigating the possibility of using it directly, which was a smart decision.

Contender #3 – JWebTest

This one is cool in concept.  It’s basically an abstraction of Selenium and HTMLUnit.  If you want to test using HTMLUnit most of the time for speed but want to switch to Selenium for a cross browser sanity check from time to time, JWebTest could be your answer.  I spent maybe an hour setting it up in my environment and writing a test with it.  It didn’t handle HTTP basic authentication right out of the box (when using a URL like “http://admin:admin@localhost:8080/alfresco/my/rest/service.json”), whereas Selenium did handle such a URL properly.  I did look (just now) to see if it handles HTTP basic authentication, and it looks like it does via the API – see WebTestCase.getTestContext() and TestContext.setAuthorization().

Regardless, the bottom line with this is that it’s a Java based API that I would have to use to program all of my tests, and the ability to switch on “Selenium mode” isn’t very compelling to me for testing REST APIs.  Therefore, this one doesn’t really add much value over using HTMLUnit directly for this use case.

Contender #4 – Canoo WebTest

Initially, my first reaction regarding Canoo was “Oh man, I don’t like that it uses Ant so heavily”.  For reasons I won’t go into here, I’m using Ant (not Maven) as my build tool for this project, so ultimately, that’s not a deal breaker.  That said, getting up and running with Canoo was pretty awesome.  The instructions say to put the WebTest bin directory in your path, which generally bothers me, but I did it anyway.  Then there’s a way to generate a project skeleton (a very maven-esque thing to do), which I did, and by running their shell script within the generated test project’s directory, I was off and running.  I then created my own test files (in Ant based XML with custom Canoo tasks), plugged them into the main build script, and BOOYAH!  I was off to the races.

The execution of the tests is pretty fast, certainly faster than Selenium, and result reporting is tight:


Canoo WebTest also has the advantage that adding new tests is a declarative exercise – no programming and compilation required.  HTTP Basic authentication is handled nicely via simple attributes on the <invoke> step.  Here’s an example:

        <webtest name="Check end date capability for assigned tasks">
            <invoke url="http://localhost:8080/alfresco/service/api/workflow/status/user/admin.json?endDate=2009-05-23"
                    description="Admin with end date 2009-05-23"
            <verifyText text='{"description":"Review","priority":2,"due":null,"properties":null,"percent":0,"completed":null,"status":"Not Yet Started","duration":null,"created":"2009-05-23 23:59:59.0","name":"wf:reviewTask"}'/>
                <verifyText text='{"description":"Adhoc Task","priority":2,"due":null,"properties":null,"percent":0,"completed":null,"status":"Not Yet Started","duration":null,"created":"2009-05-24 00:00:00.0","name":"wf:adhocTask"}'/>

The Winner – Canoo WebTest

Just to spell it out clearly: Canoo is my tool of choice for REST API testing, due to ease of use, speed of execution, good reporting, and easy handling of HTTP basic authentication per test.  If/when I move to using a Maven build system, it looks like that’s alright by Canoo, since they have a maven plugin.

Other Alternatives

Other possibilities for folks out there are:

  • Celerity – Ruby based testing framework.  Not for me since I’m not a Ruby wonk.
  • JSFUnit – Specifically geared towards testing JSF applications, which is not the case here.
  • WebDriver – Similar to Selenium, and is in fact rolling into Selenium according to the FAQ.  As such I didn’t look at this for longer than 5 minutes.
Reblog this post [with Zemanta]

  70 Responses to “Open Source Testing Tool Smackdown for REST Web Services”

  1. Bodily energy. Plumbers, pipefitters, and steamfitters should be sturdy enough to lift and move heavy

  2. I’ll right away grab your rss feed as I can’t to find your e-mail subscription hyperlink or
    newsletter service. Do you’ve any? Kindly let me recognize so that I could subscribe.

  3. What’s up, yup this piece of writing is genuinely nice and I have learned lot
    of things from it regarding blogging. thanks.

  4. certainly like your web-site but you have to take a look at the spelling on quite
    a few of your posts. Many of them are rife with spelling problems and I to find it very troublesome to inform the truth however I’ll certainly come back again.

  5. whoah this weblog is magnificent i love reading your posts.
    Keep up the great work! You already know, a lot of people are hunting
    around for this info, you could help them greatly.

  6. Les tournois de poker ont été légalisés en 2008 afin de pouvoir être organisés au sein des casinos et en dehors.

  7. I view something really interesting about your site so I saved to fav.

  8. A lire également les revues de a salle de poker pour se familiariser avec les meilleures offres
    de bonus sur le Internet.

  9. Nos specialists ont aimé la facilité des jeux, ce qui signifie
    des positive factors plus importants pour la plupart des joueurs.

  10. PokerStars est le seul endroit où vous pouvez observer et affronter
    le Group PokerStars Professional, notre groupe d’élite de joueurs professionnels composé de légendes du poker telles que
    Bertrand « ElkY » Grospellier, Daniel Negreanu, Jason Mercier et Vanessa Selbst, et
    de champions WSOP tels que Chris Moneymaker.

  11. Hey there, I think your site might be having browser compatibility
    issues. When I look at your blog in Opera, it looks fine but when opening in Internet Explorer, it has
    some overlapping. I just wanted to give you a quick heads up!
    Other then that, terrific blog!

  12. Magnificent site. Plenty of useful info here. I’m sending it to a few buddies ans additionally
    sharing in delicious. And obviously, thanks in your sweat!

  13. I like the valuable information you provide for your articles.
    I’ll bookmark your blog and check once more here regularly.
    I am moderately sure I’ll be told many new stuff right here!
    Best of luck for the next!

  14. Hey, I think your website might be having browser compatibility issues.
    When I look at your website in Firefox, it looks fine but when opening in Internet Explorer, it has some overlapping.
    I just wanted to give you a quick heads up! Other then that, terrific blog!

  15. Lorsque cette fonctionnalité est activée, une vidéo challenge des solutions est automatiquement lancée
    à la suite de la lecture en cours.

  16. great issues altogether, you just won a brand new reader.
    What would you suggest in regards to your submit that you made a few days ago?
    Any positive?

  17. I like the helpful information you supply on your articles.

    I’ll bookmark your blog and test once more right here regularly.

    I’m quite certain I will learn many new stuff proper here!
    Good luck for the next!

  18. Hi there colleagues, nice article and nice urging commented here, I am
    in fact enjoying by these.

  19. After looking into a number of the blog articles on your blog, I truly like
    your way of writing a blog. I saved as a favorite it to my bookmark webpage list and will be checking back soon. Take a look at my web site too and
    tell me what you think.

  20. I am really delighted to glance at this webpage posts which consists of
    plenty of helpful information, thanks for providing such statistics.

 Leave a Reply



You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>