Jun 172009
 
The Mega Powers (Hulk Hogan and Randy Savage) ...

Recently I’ve been working on a REST API for reporting workflow status information in Alfresco.  After getting some of the functionality nailed down, it really bothered me that I wasn’t able to use Test Driven Development (TDD) in the process.  So I went looking, and I found quite a few open source tools out there in the wild that made good prospects for acceptance testing these REST APIs that I was working on.  It was time for a SMACKDOWN! OOOOOOOH YEAAAAAAAH!

Contender #1 – Selenium

I heard of Selenium in the past and have wanted to tinker with it for a long time, so I tried this one out first.  Note that Selenium is really just the “brand name”; there are actually several inter-related offerings here.  The first is Selenium IDE, which comes as an add-on to Firefox, which is pretty awesome.  Using this tool, you can basically record your tests, and play them back.  This is very easy to try out – literally within minutes I had recorded my first tests.

That was very cool, but I needed to test result sets with dynamic data, so I had to take a look at Selenium-RC, which has APIs that enable the use of your favorite programming language: Java, C#, Perl, PHP, Python, or Ruby.  The good news with this tool is that it actually uses the real browser to do its testing.  That’s also the bad news.  For each and every test, a new Firefox was launched, which would certainly take a while as the test suite grew larger.  The main advantage that I saw with this tool is that it would be great for testing web applications with Javascript for cross-browser compatibility.  In fact, Selenium-RC is leveraged by Selenium Grid, which allows you to test cross-browser and on different operating systems.  I dig it, but I just have some simple REST APIs to test, so the whole Selenium suite is overkill for me.

Reporting of results using Selenium-RC ultimately would have to be via JUnit reporting mechanisms, which are pretty decent as I recall, but you will have an extra step to set that up in your Ant build file.

Oh, one other note.  You can use Selenium IDE to record your tests and export them as a Selenium-RC Java class.  Pretty cool, but the implementation felt like it was quick and dirty as I recall – something about the class name I gave it and the source that was generated was ‘off’.

Contender #2 – HTMLUnit

This is basically just a Java API that makes it easy to extract information from web pages programatically.  Because I knew that several other tools were built using HTMLUnit at the core, I didn’t spend any time investigating the possibility of using it directly, which was a smart decision.

Contender #3 – JWebTest

This one is cool in concept.  It’s basically an abstraction of Selenium and HTMLUnit.  If you want to test using HTMLUnit most of the time for speed but want to switch to Selenium for a cross browser sanity check from time to time, JWebTest could be your answer.  I spent maybe an hour setting it up in my environment and writing a test with it.  It didn’t handle HTTP basic authentication right out of the box (when using a URL like “http://admin:admin@localhost:8080/alfresco/my/rest/service.json”), whereas Selenium did handle such a URL properly.  I did look (just now) to see if it handles HTTP basic authentication, and it looks like it does via the API – see WebTestCase.getTestContext() and TestContext.setAuthorization().

Regardless, the bottom line with this is that it’s a Java based API that I would have to use to program all of my tests, and the ability to switch on “Selenium mode” isn’t very compelling to me for testing REST APIs.  Therefore, this one doesn’t really add much value over using HTMLUnit directly for this use case.

Contender #4 – Canoo WebTest

Initially, my first reaction regarding Canoo was “Oh man, I don’t like that it uses Ant so heavily”.  For reasons I won’t go into here, I’m using Ant (not Maven) as my build tool for this project, so ultimately, that’s not a deal breaker.  That said, getting up and running with Canoo was pretty awesome.  The instructions say to put the WebTest bin directory in your path, which generally bothers me, but I did it anyway.  Then there’s a way to generate a project skeleton (a very maven-esque thing to do), which I did, and by running their shell script within the generated test project’s directory, I was off and running.  I then created my own test files (in Ant based XML with custom Canoo tasks), plugged them into the main build script, and BOOYAH!  I was off to the races.

The execution of the tests is pretty fast, certainly faster than Selenium, and result reporting is tight:

canooreport

Canoo WebTest also has the advantage that adding new tests is a declarative exercise – no programming and compilation required.  HTTP Basic authentication is handled nicely via simple attributes on the <invoke> step.  Here’s an example:

        <webtest name="Check end date capability for assigned tasks">
            <invoke url="http://localhost:8080/alfresco/service/api/workflow/status/user/admin.json?endDate=2009-05-23"
                    description="Admin with end date 2009-05-23"
                    username="admin"
                    password="admin"/>
            <verifyText text='{"description":"Review","priority":2,"due":null,"properties":null,"percent":0,"completed":null,"status":"Not Yet Started","duration":null,"created":"2009-05-23 23:59:59.0","name":"wf:reviewTask"}'/>
            <not>
                <verifyText text='{"description":"Adhoc Task","priority":2,"due":null,"properties":null,"percent":0,"completed":null,"status":"Not Yet Started","duration":null,"created":"2009-05-24 00:00:00.0","name":"wf:adhocTask"}'/>
            </not>
        </webtest>

The Winner – Canoo WebTest

Just to spell it out clearly: Canoo is my tool of choice for REST API testing, due to ease of use, speed of execution, good reporting, and easy handling of HTTP basic authentication per test.  If/when I move to using a Maven build system, it looks like that’s alright by Canoo, since they have a maven plugin.

Other Alternatives

Other possibilities for folks out there are:

  • Celerity – Ruby based testing framework.  Not for me since I’m not a Ruby wonk.
  • JSFUnit – Specifically geared towards testing JSF applications, which is not the case here.
  • WebDriver – Similar to Selenium, and is in fact rolling into Selenium according to the FAQ.  As such I didn’t look at this for longer than 5 minutes.
Reblog this post [with Zemanta]

  39 Responses to “Open Source Testing Tool Smackdown for REST Web Services”

  1. On behalf of James Lorenzen, who pointed out that leaving a comment on my blog is problematic (which I intend to fix very soon):

    Great article. I did the same thing not too long ago for our team and we decided on using groovy to test our REST services. See my article here http://jlorenzen.blogspot.com/2009/01/testing-rest-services-with-groovy.html. I also ran across this post yesterday discussing the same issue and he is using the grails functional-test plugin and it looks super simple http://thediscoblog.com/2009/06/15/grails-hip-tip-testing-restful-services/. I don’t think I would ever knowingly write tests in XML, not matter how easy it was. I am so anti-XML today it’s not even funny. Because of that I would us gant (groovy ant) instead of ant if I had a choice. Again, I don’t want to stare at ant xml files all day.

  2. Great recommendation on canoo. This is exactly what I was looking for. Easy to setup, configure, tons of baked in functionality. Thanks.

  3. marketing…

    [...]BRob On Tech » Blog Archive » Open Source Testing Tool Smackdown for REST Web Services[...]…

  4. Surprised nobody has tried JMeter: http://jmeter.apache.org/

  5. Have you also tried WCFStorm.Rest? ? http://www.wcfstorm.com/wcf/learn-more-rest.aspx

  6. Good write-up, I¡¦m regular visitor of one¡¦s web site, maintain up the nice operate, and It is going to be a regular visitor for a lengthy time.

  7. noi that chung cu gia re

  8. This is because this device assures to either benefit or cure the autistic
    people in more ways than one. In addition, the observing surgeons could transmit
    their comments to the operating surgeon, who could read them on the Google Glass monitor.
    But it seems Memorial Day wasn’t important enough to Google.

  9. Bon ce midi par contre j’ai grave tap dedans avec une galette des rois, la part 15 pp ouie! il me reste encore 2 jours tenir pour g rer ma r serve propoints, j’esp re que le repas “basket nike 2013 homme http://www.mrshatfield.com/SMS/Binder/gm2/3486.asp” de demain sera l ger. Pour ce soir j’ai regarder le menu du resto j’h site entre des pates, ou une tartine v g tarienne avec de la mozza, je crois que ce serais peut tre le moins gras.Revenir en haut de pageSupertoinette “basket nike 2013 homme http://www.mrshatfield.com/SMS/Binder/gm2/3486.asp” D du ForumLes r gimes et leurs recettes Conseils et entraide se basant sur la m thode Weight Watchers Aller la page Pr 1, 2, 3, 4, 5, 6, 7 SuivantePage 5 sur 7 Sujets similaires: Combien de propoint journalier Calcul capital point avec propoint 2 Nouvelle : propoint Naturhouse 2013 D marrage du r gime dukan Propoint Propoint Protifast d but 1er f vrier 2013 Calcul propoint 2 (clic) Suite.”basket nike 2013 homme http://www.mrshatfield.com/SMS/Binder/gm2/3486.asp

  10. A fascinating discussion is definitely worth comment. There’s no doubt that that you ought to publish more about this issue, it might not be a taboo matter
    but typically folks don’t talk about such topics. To the next!
    Kind regards!!

  11. Thanks a ton! That was very informative, I just saved your site.

  12. If you are going the Android route then take a look at the
    4G enabled phones such as the HTC EVO, Nexus S (Sprint’s first ‘pure’ Google device), and the Samsung Epic.
    The Ouya (which proved this with its wildly successful Kickstarter campaign) was made available to consumers
    just a couple of months ago at a cost of $99. Today in this market
    of mobile there is too much competition due to this competition developing a mobile application is not easy goal.

  13. And another advantage of this platform is open source Android OS supports third party applications
    and developers of free access to all countries.
    One of the greatest video games to arrive in the gaming world lately is Minecraft.
    There are two versions, one written in Python and one written in Objective C targeting i – OS.

  14. El segundo proyecto de Anna & Eugeni Bach consiste en otra
    reforma de una vivienda en el Ensanche barcelonés del año 1910.

  15. En el caso de que necesite una compañía de reformas en Vic póngase en contacto
    con nosotros para el presupuesto de su obra.

  16. It is one of the most downloaded sexy Android apps in the
    Android Market. It has the capacity to provide information that the user
    require, for example weather report or sport outcomes, also it can give facts about the things that took place on nearby locations.
    Wikipad is now the thinnest and lightest Android
    equipped tablet with 1.

  17. Digital Video Expo West and DV Film Fest runs from December 4-6 in Los Angeles, and offers a line-up of
    educational forums, including Apple and Avid certification training, sessions covering digital
    video technology from A to Z, and specialized tracks featuring industry hot topics.
    There is a range of apps available on Google Play catering to all kinds
    of users. We know that Apple Company created such a great product,
    Apple TV automatically grabs the digital media on your
    computer and makes it available for you to enjoy
    on your TV.

  18. I think that what you published was very reasonable.
    However, what about this? suppose you were to write a killer post title?
    I ain’t suggesting your information isn’t good., however suppose
    yyou addeed something that grabbed people’s attention?
    I mean Open Source Testing Tool Smackdown for REST Weeb Services  BRob On Tech
    is a littlpe vanilla. You should look at Yahoo’s home page and see how they write article titles too get people
    interested. You might aadd a related video or a picture or two to grab
    readers interested about everything’ve got to say. In my opinion, it would make yur
    posts a little livelier. http://www.sweetvacationspot.com

  19. A red wine should be served when it is only slightly below room temperature.
    If in search of authentic, quality French wine , it would take considerable
    effort to find a company who surpass the standards set by Yapp Brothers.
    “sustainable development to a broad consensus of the international community, all countries, all industries, in promoting sustainable development.

  20. I first tried Pumpkin leaves and vines which were like brocollini with rocket; it was par boiled and mixed with Chicken flavoured two-minute noodles.
    Mommessin owns several vineyards in the Beaujolais and Bordeaux regions of south eastern, or south central
    France. “sustainable development to a broad consensus of the international community, all countries, all industries, in promoting sustainable development.

  21. My final meal was composed of an omelet garnished
    with fresh tomatoes. The second dessert (sometimes we indulge)
    was a homemade peach, pear, and strawberry cobbler garnished with chocolate chips.
    Richard a business entrepreneur who carries successful business of Buying Wine
    Online and many different kinds of Buy Online Wine.

  22. At the Internet platform, you can come across a number of online
    wine retailers. The relatively low volume Vin de Pays du Val
    de Loire appellation covers quite a large area. Also, chardonnay can be paired
    withspices and herbs such as tarragon, parsley and shallots, cheese and vegetables.

  23. They refurbished this 10-room hotel and a self-contained 3-bedroom house with great care and
    attention to detail, and judging from the feedback
    of their guests, they have done a great job. Should I ever get so lucky, you’ll be the first to know.
    You want to find the perfect wine from your wine cellar in Fort Worth TX to go with your
    dessert, but you do not know what the best pairing might be.

  24. Here you will find a selection of fine wines from all of the
    world. The second dessert (sometimes we indulge) was a homemade peach, pear, and strawberry cobbler
    garnished with chocolate chips. Richard a business entrepreneur who carries successful business
    of Buying Wine Online and many different kinds of Buy Online Wine.

  25. Each Australian wine has its own beauty, but let’s see which ones
    stand out more than others. Identifying a specific bottle of
    wine’s quality and characteristics without tasting it tends to be very difficult, but with
    wine ratings you can at least get an idea and compare ratings to bottles you
    do know to get a feel for what they mean. The wine’s acidity seemed
    untamed and definitely out of balance.

  26. Here in the wineries the technique of development is done and wine is organized.

    Interestingly enough the fresh tomato rendered the wine rounder.
    Also, chardonnay can be paired withspices and herbs such as tarragon, parsley and shallots, cheese and
    vegetables.

  27. I first tried Pumpkin leaves and vines which were
    like brocollini with rocket; it was par
    boiled and mixed with Chicken flavoured two-minute noodles.
    Most of the wine shops and retail outlets have an excellent collection of the best vintage
    wines, which date back to a good time. Also, chardonnay can be paired
    withspices and herbs such as tarragon, parsley and
    shallots, cheese and vegetables.

  28. At the Internet platform, you can come across
    a number of online wine retailers. Most of the wine shops and retail outlets have an excellent collection of the best
    vintage wines, which date back to a good time.
    Before you say that you’ll never spend so much money on a single bottle
    of wine, please note that $100 may get you four movie theatre
    tickets, popcorn, soft drinks, and maybe an order or two of
    nachos.

  29. More than thirty years ago Mondavi got people to drink the then unfashionable Sauvingnon Blanc wines
    in part by giving them a new, charming name, Fume Blanc.
    Mommessin owns several vineyards in the Beaujolais and Bordeaux regions
    of south eastern, or south central France. “sustainable development to a broad consensus of the international community, all countries, all industries, in promoting sustainable development.

  30. Here in the wineries the technique of development is done and wine is organized.
    Most of the wine shops and retail outlets have an excellent collection of the best vintage wines, which date back to a good
    time. Whichever you choose, you should remember to pick one that you.

  31. My final meal was composed of an omelet garnished with fresh tomatoes.
    The relatively low volume Vin de Pays du Val de Loire appellation covers quite a large area.
    Richard a business entrepreneur who carries successful business of
    Buying Wine Online and many different kinds of Buy Online Wine.

  32. Admiring the time and effort you put into
    your site and in depth information you provide. It’s nice to come across a blog every
    once in a while that isn’t the same out of date rehashed information. Excellent
    read! I’ve bookmarked your site and I’m adding your RSS feeds to my Google account.

  33. Excellent article! I can’t wait for the next one!

  34. If you are going the Android route then take a look
    at the 4G enabled phones such as the HTC EVO, Nexus S (Sprint’s first ‘pure’ Google device), and
    the Samsung Epic. It automatically triggers the purchase requests from within the store instead of the developer having to implement
    that on his own. The latest malware detection The second report presented by Kaspersky states that SMS Trojans comprise almost 25% of the total threats
    in the second quarter of 2012.

  35. Find out such sites that let you know comment pirater un compte
    Facebook gratuity. Things end with the boys in perfect bliss over burgers, artisan soap, and comparing baby soft skin as cameras pull back.

    Everyone has obstacles they want to overcome, and sometimes those mountains seem
    gargantuan and too tall to climb.

  36. You can also ask the executive to remove the
    automatic billing service from your account if you still are under the subscription period.
    As with all things, this writer holds that the use of science as an aspect of what
    might solve a problem is constant. ” The song will resurface at various points in the movie, but the point is clear, this bel ami, Georges Duroy, is not a good sort of man.

  37. Hurrah, that’s what I was seeking for, what a stuff!
    present here at this webpage, thanks admin of this web site.

 Leave a Reply

(required)

(required)

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>