Jun 172009
 
The Mega Powers (Hulk Hogan and Randy Savage) ...

Recently I’ve been working on a REST API for reporting workflow status information in Alfresco.  After getting some of the functionality nailed down, it really bothered me that I wasn’t able to use Test Driven Development (TDD) in the process.  So I went looking, and I found quite a few open source tools out there in the wild that made good prospects for acceptance testing these REST APIs that I was working on.  It was time for a SMACKDOWN! OOOOOOOH YEAAAAAAAH!

Contender #1 – Selenium

I heard of Selenium in the past and have wanted to tinker with it for a long time, so I tried this one out first.  Note that Selenium is really just the “brand name”; there are actually several inter-related offerings here.  The first is Selenium IDE, which comes as an add-on to Firefox, which is pretty awesome.  Using this tool, you can basically record your tests, and play them back.  This is very easy to try out – literally within minutes I had recorded my first tests.

That was very cool, but I needed to test result sets with dynamic data, so I had to take a look at Selenium-RC, which has APIs that enable the use of your favorite programming language: Java, C#, Perl, PHP, Python, or Ruby.  The good news with this tool is that it actually uses the real browser to do its testing.  That’s also the bad news.  For each and every test, a new Firefox was launched, which would certainly take a while as the test suite grew larger.  The main advantage that I saw with this tool is that it would be great for testing web applications with Javascript for cross-browser compatibility.  In fact, Selenium-RC is leveraged by Selenium Grid, which allows you to test cross-browser and on different operating systems.  I dig it, but I just have some simple REST APIs to test, so the whole Selenium suite is overkill for me.

Reporting of results using Selenium-RC ultimately would have to be via JUnit reporting mechanisms, which are pretty decent as I recall, but you will have an extra step to set that up in your Ant build file.

Oh, one other note.  You can use Selenium IDE to record your tests and export them as a Selenium-RC Java class.  Pretty cool, but the implementation felt like it was quick and dirty as I recall – something about the class name I gave it and the source that was generated was ‘off’.

Contender #2 – HTMLUnit

This is basically just a Java API that makes it easy to extract information from web pages programatically.  Because I knew that several other tools were built using HTMLUnit at the core, I didn’t spend any time investigating the possibility of using it directly, which was a smart decision.

Contender #3 – JWebTest

This one is cool in concept.  It’s basically an abstraction of Selenium and HTMLUnit.  If you want to test using HTMLUnit most of the time for speed but want to switch to Selenium for a cross browser sanity check from time to time, JWebTest could be your answer.  I spent maybe an hour setting it up in my environment and writing a test with it.  It didn’t handle HTTP basic authentication right out of the box (when using a URL like “http://admin:admin@localhost:8080/alfresco/my/rest/service.json”), whereas Selenium did handle such a URL properly.  I did look (just now) to see if it handles HTTP basic authentication, and it looks like it does via the API – see WebTestCase.getTestContext() and TestContext.setAuthorization().

Regardless, the bottom line with this is that it’s a Java based API that I would have to use to program all of my tests, and the ability to switch on “Selenium mode” isn’t very compelling to me for testing REST APIs.  Therefore, this one doesn’t really add much value over using HTMLUnit directly for this use case.

Contender #4 – Canoo WebTest

Initially, my first reaction regarding Canoo was “Oh man, I don’t like that it uses Ant so heavily”.  For reasons I won’t go into here, I’m using Ant (not Maven) as my build tool for this project, so ultimately, that’s not a deal breaker.  That said, getting up and running with Canoo was pretty awesome.  The instructions say to put the WebTest bin directory in your path, which generally bothers me, but I did it anyway.  Then there’s a way to generate a project skeleton (a very maven-esque thing to do), which I did, and by running their shell script within the generated test project’s directory, I was off and running.  I then created my own test files (in Ant based XML with custom Canoo tasks), plugged them into the main build script, and BOOYAH!  I was off to the races.

The execution of the tests is pretty fast, certainly faster than Selenium, and result reporting is tight:

canooreport

Canoo WebTest also has the advantage that adding new tests is a declarative exercise – no programming and compilation required.  HTTP Basic authentication is handled nicely via simple attributes on the <invoke> step.  Here’s an example:

        <webtest name="Check end date capability for assigned tasks">
            <invoke url="http://localhost:8080/alfresco/service/api/workflow/status/user/admin.json?endDate=2009-05-23"
                    description="Admin with end date 2009-05-23"
                    username="admin"
                    password="admin"/>
            <verifyText text='{"description":"Review","priority":2,"due":null,"properties":null,"percent":0,"completed":null,"status":"Not Yet Started","duration":null,"created":"2009-05-23 23:59:59.0","name":"wf:reviewTask"}'/>
            <not>
                <verifyText text='{"description":"Adhoc Task","priority":2,"due":null,"properties":null,"percent":0,"completed":null,"status":"Not Yet Started","duration":null,"created":"2009-05-24 00:00:00.0","name":"wf:adhocTask"}'/>
            </not>
        </webtest>

The Winner – Canoo WebTest

Just to spell it out clearly: Canoo is my tool of choice for REST API testing, due to ease of use, speed of execution, good reporting, and easy handling of HTTP basic authentication per test.  If/when I move to using a Maven build system, it looks like that’s alright by Canoo, since they have a maven plugin.

Other Alternatives

Other possibilities for folks out there are:

  • Celerity – Ruby based testing framework.  Not for me since I’m not a Ruby wonk.
  • JSFUnit – Specifically geared towards testing JSF applications, which is not the case here.
  • WebDriver – Similar to Selenium, and is in fact rolling into Selenium according to the FAQ.  As such I didn’t look at this for longer than 5 minutes.
Reblog this post [with Zemanta]

  10 Responses to “Open Source Testing Tool Smackdown for REST Web Services”

  1. On behalf of James Lorenzen, who pointed out that leaving a comment on my blog is problematic (which I intend to fix very soon):

    Great article. I did the same thing not too long ago for our team and we decided on using groovy to test our REST services. See my article here http://jlorenzen.blogspot.com/2009/01/testing-rest-services-with-groovy.html. I also ran across this post yesterday discussing the same issue and he is using the grails functional-test plugin and it looks super simple http://thediscoblog.com/2009/06/15/grails-hip-tip-testing-restful-services/. I don’t think I would ever knowingly write tests in XML, not matter how easy it was. I am so anti-XML today it’s not even funny. Because of that I would us gant (groovy ant) instead of ant if I had a choice. Again, I don’t want to stare at ant xml files all day.

  2. Great recommendation on canoo. This is exactly what I was looking for. Easy to setup, configure, tons of baked in functionality. Thanks.

  3. marketing…

    [...]BRob On Tech » Blog Archive » Open Source Testing Tool Smackdown for REST Web Services[...]…

  4. Surprised nobody has tried JMeter: http://jmeter.apache.org/

  5. Have you also tried WCFStorm.Rest? ? http://www.wcfstorm.com/wcf/learn-more-rest.aspx

  6. Good write-up, I¡¦m regular visitor of one¡¦s web site, maintain up the nice operate, and It is going to be a regular visitor for a lengthy time.

  7. noi that chung cu gia re

  8. This is because this device assures to either benefit or cure the autistic
    people in more ways than one. In addition, the observing surgeons could transmit
    their comments to the operating surgeon, who could read them on the Google Glass monitor.
    But it seems Memorial Day wasn’t important enough to Google.

  9. Bon ce midi par contre j’ai grave tap dedans avec une galette des rois, la part 15 pp ouie! il me reste encore 2 jours tenir pour g rer ma r serve propoints, j’esp re que le repas “basket nike 2013 homme http://www.mrshatfield.com/SMS/Binder/gm2/3486.asp” de demain sera l ger. Pour ce soir j’ai regarder le menu du resto j’h site entre des pates, ou une tartine v g tarienne avec de la mozza, je crois que ce serais peut tre le moins gras.Revenir en haut de pageSupertoinette “basket nike 2013 homme http://www.mrshatfield.com/SMS/Binder/gm2/3486.asp” D du ForumLes r gimes et leurs recettes Conseils et entraide se basant sur la m thode Weight Watchers Aller la page Pr 1, 2, 3, 4, 5, 6, 7 SuivantePage 5 sur 7 Sujets similaires: Combien de propoint journalier Calcul capital point avec propoint 2 Nouvelle : propoint Naturhouse 2013 D marrage du r gime dukan Propoint Propoint Protifast d but 1er f vrier 2013 Calcul propoint 2 (clic) Suite.”basket nike 2013 homme http://www.mrshatfield.com/SMS/Binder/gm2/3486.asp

 Leave a Reply

(required)

(required)

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>