• No one is good at estimating - I’ve never met anyone who can consistently, correctly identify how long a development task will take.  In the case that you do manage to complete a task within your estimate, it’s most likely you’ve added a 50% buffer (i.e. which in essence is saying you have no idea how long it’ll take)
• Estimates cause stress - The developer tells the project/account/product manager that a feature will be done in 3 days.  The manager tells the client it will be done in 6 days (they of course add another 50% padding).  6 days later the feature is buggy, doesn’t work in IE and is no way client ready.  Everyone is angry and pointing fingers.
• Estimating takes time - It might take half a day to figure out all the nuances of a particular feature and account for everything that may go wrong before an estimate can be given.  This time is better spent developing working software.
• Estimates reduce the quality of software - “I gotta get this feature out the door tomorrow, there’s no way I’m writing tests that will help me update and maintain the system down the road.  I’m hacking this together so I can watch football on Sunday.”

If you ask most managers, the reason they want estimates is to: make sure projects are on-time, measure return on their investment, see if they require additional/fewer developers, help prioritize new features, and to help plan with other teams.

I don’t really buy it though.  It seems like all the reasons above can be accomplished without estimates.  Instead, the focus should be on “are we consistently delivering value through short incremental releases at a sustainable pace?”

When we talked about this as a group (almost everyone was a developer), they said this all sounds great, but how do we convince our company to scrap estimates?  We didn’t really come to any consensus but some interesting suggestions included:

• Release code to production often - Instead of working on a feature for a month, then at the end of the release finding out the a client requirement changed, or that the implementation wasn’t as per their expectation, release code often and get feedback quickly.
• Deliver software in thin, vertical slices -  In most typical construction projects you work on the foundation, then put in steel beams, then put up some walls, insert windows, put in carpeting, etc…  Once the entire building is mostly ready, can people start moving in (I have no idea if this is actually how buildings are built, you get my point though).  In Bombay I’ve seen some constructions projects in which they will finish a section of the building completely, before they start on the next floor/wing/etc…  People can start living there (deriving value), as soon as that piece is done.
• Focus on customer happiness not features - If you get your clients/customers involved, are transparent about your process, get feedback early and often and demonstrate constant progress, everyone should be happy, right?

One of the joys of working at a products company is that we don’t have clients making ridiculous demands (like asking us when stuff will be done ).  Certainly in a service company it’s a lot harder to pull this off.  Instead of drafting a proposal for a certain set of features delivered at a fix price in a fixed time period, maybe moving to a monthly retainer is better for both parties (though this definitely requires a high level of trust and maturity).

If the post above doesn’t convince you, my buddy Naresh wrote a better post (with some very interesting comments) here.  What do you think, are estimates a necessary evil?

No related posts.

1. Introduction (How can testing make our lives easier)
2. What is Selenium?
   1. Selenium IDE
   2. Trivial Example
   3. A Less Trivial Test Case (and some useful commands)
3. Selenium RC
   1. Setup
   2. An example using Python
4. The Future
5. Additional Resources
   1. Reference
   2. Tutorials
   3. Testivus

Introduction (How can testing make our lives easier)

Testing is important, saves time, and makes our life easier.  Imagine a world where you can easily identify when changing code breaks existing functionality.  With Selenium we can write test suites for projects that can be automatically executed prior to code pushes, in this way we can have a high level of confidence that changes to a code base don’t have any unintentional consequences.

Functional testing of web applications isn’t easy, different browser/OS combinations all react differently, and now with the proliferation of Ajax, traditional testing methods no longer apply.  Selenium is great because it allows you to wait for and verify that certain Ajaxy conditions are met. Additionally you can write one test script and run it on all browser/OS combinations.

This document is not intended to be a complete reference guide or tutorial (since there are many already available on the web including: http://www.devchix.com/2007/02/19/an-introduction-to-selenium-ide/, http://selenium-rc.openqa.org/tutorial.html, http://www.infoq.com/articles/testing-ajax-selenium).  Instead, the goal is to introduce Selenium, discuss the importance of testing, and provide some information specific to our process to help us improve the quality of the sites we develop.

What is Selenium?

“Selenium is a test tool for web applications.  Selenium tests run directly in a browser, just as real users do. And they run in Internet Explorer, Mozilla and Firefox on Windows, Linux, and Macintosh. (http://selenium.openqa.org/)”  Selenium comes in a few different flavors: the Firefox IDE (integrated development environment), Selenium Core, Selenium RC, Selenium Grid and Selenium on Rails. Selenium IDE and RC are discussed below.

Selenium IDE

“Selenium IDE is an integrated development environment for Selenium tests. It is implemented as a Firefox extension, and allows you to record, edit, and debug tests. Selenium IDE … allow[s] you to easily and quickly record and play back tests in the actual environment that

they will run. (http://selenium-ide.openqa.org/)”  Selenium IDE is written entirely in Javascript and has hooks for exporting test cases in your favorite language (Java, Ruby, Python, Perl, and PHP).

Clicking the record button (top right corner) allows you to execute commands in your browser to create a Selenium test. When you have executed all the commands in your browser, you can use an assert command to verify that a certain condition has been met.

Although the IDE does not compare Visual Studio or Eclipse, you are still able to step through commands, set breakpoints, view debug output, and double click commands to see them execute in the browser.

A Trivial Example

• Download the Selenium IDE (http://selenium-ide.openqa.org/)
• Launch the IDE from the Firefox Toolbar:

• Click record:

• In Firefox, go through the series of steps you need to accomplish to verify that a condition has been met:

• Walking through each of the commands in the test above:
• Open: Opens the base URL above (www.google.com) with the extention /ig?hl=en (the default when I go to Google):

• Type: target=q, value=mets best in NL. In this step, the value of the search bar has an html id of “q” (you can verify this using firebug, see the image below). By selecting target=q, it selects the search bar, and types in the value “mets best in nl.”
• clickAndWait: target=btnG. Click on Google Search. Using firebug you can also verify that the Google Search button has the id=”btnG” Wait till the resulting page is loaded.
• clickAndWait: target(link=ABC News: Santana Deal Makes Mets…) So not only can you target elements by their ID, you can also reference links by their display text:

• assertTextPresent: Santana Makes Mets NL Favorite. This final step creates a true/false condition at the end of the test. If that link is clicked and the text is present on the following page, then the test case passes with a value of True, otherwise False.

A Less Trivial Test Case (and some useful commands)

The following test case goes to the Shop Dev environment, adds a few products to the cart, goes through the checkout process, and asserts that checkout has successfully completed. In order to run this test you must be connected to the Shop VPN. Please note that I have removed the aforementioned example because of security issues.  I plan to include another one soon.

Let’s review some of the more useful commands used in this test case:

• waitForElementPresent – Verifies that the specified element is somewhere on the page. If you try using the recorder to generate test cases, you will find that in most cases, it executes them too quickly. waitForElementPresent allows you to use an id, name, javascript expression, xpath or link as an input.  For more info see: http://release.openqa.org/selenium-core/0.8.0/reference.html.  This is especially useful when waiting for certain conditions to be met, like waiting for a layer to pop-up (shopping cart), or waiting for the content on screen to change (like checkout).
• assertVisable – Determines if the specified element is visible. An element can be rendered invisible by setting the CSS “visibility” property to “hidden”, or the “display” property to “none”, either for the element itself or one if its ancestors. This method will fail if the element is not present. In this particular case we end our test with asserting that id=successResponse is visable and failureResponse is not.

Selenium RC

While the IDE is great for writing test cases, you can only write test cases in Selenese (the HTML table format), tests can only be executed in Firefox, and you don’t have the ability to kick off a series of tests. Enter Selenium RC.

“Selenium Remote Control provides a Selenium Server, which can automatically start/stop/control any supported browser. It works by using Selenium Core a pure-HTML+JS library that performs automated tasks in JavaScript.

The Selenium Server communicates directly with the browser using AJAX (XmlHttpRequest). You can send commands directly to the Server using simple HTTP GET/POST requests; that means that you can use any programming language that can make HTTP requests to automate Selenium tests on the browser. To further ease this process, we provide wrapper objects for a number of mainstream programming languages (Java, .NET, Perl, Python, and Ruby).

Finally, the Selenium Server acts as a client-configured HTTP proxy, to stand in between the browser and your website. This allows a Selenium-enabled browser to run JavaScript on arbitrary websites. (http://selenium-rc.openqa.org/)

Setup

• Download Selenium RC (http://selenium-rc.openqa.org/download.jsp)
• We made a modification to the Selenium Server Jar file to allow us to verify text is present in XML (thanks Nicholas and Sudhakar!) Download the updated jar file here: http://extranet.ipartners.com/selenium/selenium-server.jar
• Start the Selenium Server:

java -jar selenium-server.jar -interactive

Test it out by trying to open Google in Firefox:

cmd=getNewBrowserSession&1=*firefox&2=http://www.google.com

An example using Python

• If you don’t have it already download Python http://www.python.org/)
• Download the following files into the selenium-python-client folder:
  selenium-python-client folder: Again because of security reasons I had to remove this example, I will post a new one soon.

The Future

Eventually we hope to integrate testing more deeply in our process. One day as part of any code push automated test scripts will run and code will only get pushed if these tests pass in the target environment. Once we have a core framework in place, we can automate the process of nightly builds and testing. See tools like http://cruisecontrol.sourceforge.net/http://www.jsunit.net/

In the near future we will need to centralize test machines. For now having one test machine with virtual instances with different browser combinations will work. In the long term Selenium Grid will work till we get into Cloud Computing

A few more long term goals:

• Continuous Integration:http://www.martinfowler.com/articles/continuousIntegration.html
• Test Drive Deployment:http://www.martinfowler.com/bliki/TestDrivenDevelopment.html
• XP:http://www.extremeprogramming.org/

Additional Resources

Reference

• http://release.openqa.org/selenium-core/0.8.0/reference.html
• http://agiletesting.blogspot.com/2006/01/useful-tools-for-writing-selenium.html
• http://ttwhy.org/home/blog/2007/05/12/selenium-ui-element-locator/

Tutorials

• http://www.devchix.com/2007/02/19/an-introduction-to-selenium-ide/http://selenium-rc.openqa.org/tutorial.html
• http://www.infoq.com/articles/testing-ajax-selenium

Testivus

• http://www.developertesting.com/archives/month200705/20070502-000424.html

Related posts:

1. Testivus

http://www.developertesting.com/archives/month200705/20070502-000424.html

To summarize:

• All code needs to be tested.
• Ugly tests are better then no tests.
• Write tests often
• Run tests often

A few weeks ago I gave a short presentation regarding testing with Selenium. Earlier today I sent out some introductory documentation as well. Hopefully in the coming weeks we will all begin to familiarize ourselves with some of the testing tools out there and begin, more seriously, incorporating testing into our build process.

Related posts:

1. Functional Web Application Testing with Selenium IDE and RC

In the high-flying fast-paced world of web development it seems like the current “hot” technologies change every day. Luckily there some things that never change… bugs. While we are trying to improve our method of logging, assigning, tracking and resolving bugs, I’d like to share an article that gives general pointers on good bug logging. Although this article was written in ancient times (2000) it’s tips are still useful:

Every good bug report needs exactly three things.

1. Steps to reproduce
2. What you expected to see, and
3. What you saw instead.

Additionally here are the most relevant of the top ten tips for bug tracking:

1. A good tester will always try to reduce the repro steps to the minimal steps to reproduce; this is extremely helpful for the programmer who has to find the bug.
5. You’ll want to keep careful track of versions. Every build of the software that you give to testers should have a build ID number so that the poor tester doesn’t have to retest the bug on a version of the software where it wasn’t even supposed to be fixed.
10. Avoid the temptation to add new fields to the bug database. Every month or so, somebody will come up with a great idea for a new field to put in the database. You get all kinds of clever ideas, for example, keeping track of the file where the bug was found; keeping track of what % of the time the bug is reproducible; keeping track of how many times the bug occurred; keeping track of which exact versions of which DLLs were installed on the machine where the bug happened. It’s very important not to give in to these ideas. If you do, your new bug entry screen will end up with a thousand fields that you need to supply, and nobody will want to input bug reports any more. For the bug database to work, everybody needs to use it, and if entering bugs “formally” is too much work, people will go around the bug database.

I’m also going to add my own 11th top ten tip for bug tracking:

Screenshots help!

For more info check out the article here

No related posts.