Tag Archives: Testing

The first version of your product is going to suck… launch anyway

Posted on by
2

For most people the idea of launching a half-baked product is horrifying. You’ve been talking up your idea for months, been working crazy hours and even taken money from friends and family. This is the first time people will be seeing what you’ve been dreaming about and you want everything to be perfect. You feel like your entire reputation is at stake.

Unfortunately, you didn’t work with StartupGiraffe and things aren’t there quite yet. The site doesn’t work in IE (which your mom uses), crashes your browser if you click too many things too quickly (which your annoying co-worker has a tendency of doing) and is missing that killer “make my pictures look really old and cool” feature. So what do you? Launch anyway (to a handful of selected people)…

The first version of your product isn’t meant to have mass appeal. It’s purpose is to find out if you are heading in the right direction and see if you are really solving a customer need. The feedback you will receive will far outweigh the effort spent trying to “make everything perfect.”

The earliest stages of a startup focus heavily on product development. Launching early opens up a whole new world of awesome things to do for founders (especially non-technical ones). You can start acquiring customers, doing biz dev deals, speaking with investors, and planning v2 of your site (which will be much more awesome). In short, start demonstrating demand and traction.

Anyone disagree?

Deciding Between Startup Ideas

Posted on by
8

No one said working at a startup was easy. Although we started StoryStack with the intensity of a thousand suns we’ve hit a few bumps along the way. Through three iterations of the product and countless meetings with friends and advisors we’ve realized the product in it’s current form is unlikely to be successful.  We’ve found ourselves stuck somewhere between Informed Pessimism and Crisis of Meaning.  Now what?  How do we turn our passion and drive into productive energy?

We have a smart, resilient team and are throwing around new ideas all the time. Should we succumb to the sweet Siren song of new ideas? Do we stick by our guns and pivot and push forward?  Here are the factors we used to dictate our direction:

Addressing Pain Points –  Is this a product that customers are begging for?  There are a bunch of ways to test this before you write your first line of code:

  • B2B – Find potential customers and cold email them a brief description of your product.  If you find it easy to get decision makers on the phone or in meetings you’re in good shape.  If after your pitch you can get them to commit to purchasing your product when it’s ready, you’re money.  If they love it until you ask them to pay… tread carefully.
  • B2C – Setup a landing page (like LaunchRock) briefly describing your product.  If you have a little cash buy some Facebook or Google ads leading to your landing page and see the percentage of people who sign up (see Zygna’s Ghetto Testing).  If you’re broke: email, tweet and FB message your way to a sufficient sample size.
  • Another good indicator are homebrewed/hacked together solutions for the problem your product tries to address.  Are people stretching the limits of existing products or using multiple products in weird ways to get the same result?  People are lazy, if you make it way easier to do something they are already doing, you’re probably on track.

Strengths of your team –  If you taught yourself to be a your own technical co-founder and don’t have experience building scalable web products, tackling a mass consumer B2C web product might not be the best idea.  If you’re a good sales gal and know how to manage customer acquisition through multiple channels then the company you choose to build should leverage those skills.

The right market –  I loved these two posts from Elad Gil:

As you iterate in a market, you will often find that the initial idea you chose is less important then the broader market you are in.  A great market will always have opportunities in it.  Even if the first idea is terrible you will get to know the market and its needs and build something great on your next try.  In contrast a great idea in a terrible market will often fail. ~ You don’t need a good idea to start a great company

But how do you know if you are in the right market?   Paraphrasing from How Can You Tell If Your Market Is A Good One?

  • What’s changed about the market? Are there cost changes, new distribution channels or technologies?
  • Is the industry and customer base growing rapidly?
  • Is there a lot of interest in the market?

Passion - Even if you have the product that customers would line up for, fits excellently within the strength of your team and is in a killer up and coming market the number passion is still the most important factor.  Startups are hard.  Are you going to be thinking about your product 24×7?  Are you gonna love coming to work even when it sucks and it seems like you aren’t making progress?  Do you HAVE to work on this?

..the best startup you can create is one where you will be constantly engaged in thinking about improving the product, maximizing the user experience, and planning for the future -where you have real passion for making it work. ~ The Passion Gap: Why Foursquare, Groupon, Facebook, And Apple Are Winning

So what are we doing?  The short answer is that we’ve decided to delay the decision.  We’ve temporarily put StoryStack on the back burner and  while we work on a (soon to be announced) social, music product.  We’ll do a small launch next week, quickly reach the area between informed pessimism and crisis of meeting and go through the whole thought process again…

Functional Web Application Testing with Selenium IDE and RC

Posted on by
1


Introduction (How can testing make our lives easier)

Testing is important, saves time, and makes our life easier.  Imagine a world where you can easily identify when changing code breaks existing functionality.  With Selenium we can write test suites for projects that can be automatically executed prior to code pushes, in this way we can have a high level of confidence that changes to a code base don’t have any unintentional consequences.

Functional testing of web applications isn’t easy, different browser/OS combinations all react differently, and now with the proliferation of Ajax, traditional testing methods no longer apply.  Selenium is great because it allows you to wait for and verify that certain Ajaxy conditions are met. Additionally you can write one test script and run it on all browser/OS combinations.

This document is not intended to be a complete reference guide or tutorial (since there are many already available on the web including: http://www.devchix.com/2007/02/19/an-introduction-to-selenium-ide/, http://selenium-rc.openqa.org/tutorial.html, http://www.infoq.com/articles/testing-ajax-selenium).  Instead, the goal is to introduce Selenium, discuss the importance of testing, and provide some information specific to our process to help us improve the quality of the sites we develop.

What is Selenium?

“Selenium is a test tool for web applications.  Selenium tests run directly in a browser, just as real users do. And they run in Internet Explorer, Mozilla and Firefox on Windows, Linux, and Macintosh. (http://selenium.openqa.org/)”  Selenium comes in a few different flavors: the Firefox IDE (integrated development environment), Selenium Core, Selenium RC, Selenium Grid and Selenium on Rails. Selenium IDE and RC are discussed below.

Selenium IDE

“Selenium IDE is an integrated development environment for Selenium tests. It is implemented as a Firefox extension, and allows you to record, edit, and debug tests. Selenium IDE … allow[s] you to easily and quickly record and play back tests in the actual environment that

they will run. (http://selenium-ide.openqa.org/)”  Selenium IDE is written entirely in Javascript and has hooks for exporting test cases in your favorite language (Java, Ruby, Python, Perl, and PHP).

Clicking the record button (top right corner) allows you to execute commands in your browser to create a Selenium test. When you have executed all the commands in your browser, you can use an assert command to verify that a certain condition has been met.

Although the IDE does not compare Visual Studio or Eclipse, you are still able to step through commands, set breakpoints, view debug output, and double click commands to see them execute in the browser.

A Trivial Example

  • Click record:

  • In Firefox, go through the series of steps you need to accomplish to verify that a condition has been met:

  • Walking through each of the commands in the test above:
  • Open: Opens the base URL above (www.google.com) with the extention /ig?hl=en (the default when I go to Google):

  • Type: target=q, value=mets best in NL. In this step, the value of the search bar has an html id of “q” (you can verify this using firebug, see the image below). By selecting target=q, it selects the search bar, and types in the value “mets best in nl.”
  • clickAndWait: target=btnG. Click on Google Search. Using firebug you can also verify that the Google Search button has the id=”btnG” Wait till the resulting page is loaded.
  • clickAndWait: target(link=ABC News: Santana Deal Makes Mets…) So not only can you target elements by their ID, you can also reference links by their display text:

  • assertTextPresent: Santana Makes Mets NL Favorite. This final step creates a true/false condition at the end of the test. If that link is clicked and the text is present on the following page, then the test case passes with a value of True, otherwise False.

A Less Trivial Test Case (and some useful commands)

The following test case goes to the Shop Dev environment, adds a few products to the cart, goes through the checkout process, and asserts that checkout has successfully completed. In order to run this test you must be connected to the Shop VPN. Please note that I have removed the aforementioned example because of security issues.  I plan to include another one soon.

Let’s review some of the more useful commands used in this test case:

  • waitForElementPresent – Verifies that the specified element is somewhere on the page. If you try using the recorder to generate test cases, you will find that in most cases, it executes them too quickly. waitForElementPresent allows you to use an id, name, javascript expression, xpath or link as an input.  For more info see: http://release.openqa.org/selenium-core/0.8.0/reference.html.  This is especially useful when waiting for certain conditions to be met, like waiting for a layer to pop-up (shopping cart), or waiting for the content on screen to change (like checkout).
  • assertVisable – Determines if the specified element is visible. An element can be rendered invisible by setting the CSS “visibility” property to “hidden”, or the “display” property to “none”, either for the element itself or one if its ancestors. This method will fail if the element is not present. In this particular case we end our test with asserting that id=successResponse is visable and failureResponse is not.

Selenium RC

While the IDE is great for writing test cases, you can only write test cases in Selenese (the HTML table format), tests can only be executed in Firefox, and you don’t have the ability to kick off a series of tests. Enter Selenium RC.

“Selenium Remote Control provides a Selenium Server, which can automatically start/stop/control any supported browser. It works by using Selenium Core a pure-HTML+JS library that performs automated tasks in JavaScript.

The Selenium Server communicates directly with the browser using AJAX (XmlHttpRequest). You can send commands directly to the Server using simple HTTP GET/POST requests; that means that you can use any programming language that can make HTTP requests to automate Selenium tests on the browser. To further ease this process, we provide wrapper objects for a number of mainstream programming languages (Java, .NET, Perl, Python, and Ruby).

Finally, the Selenium Server acts as a client-configured HTTP proxy, to stand in between the browser and your website. This allows a Selenium-enabled browser to run JavaScript on arbitrary websites. (http://selenium-rc.openqa.org/)

Setup

java -jar selenium-server.jar -interactive

Test it out by trying to open Google in Firefox:

cmd=getNewBrowserSession&1=*firefox&2=http://www.google.com

An example using Python

  • If you don’t have it already download Python http://www.python.org/)
  • Download the following files into the selenium-python-client folder:
    selenium-python-client folder:     Again because of security reasons I had to remove this example, I will post a new one soon.

The Future

Eventually we hope to integrate testing more deeply in our process. One day as part of any code push automated test scripts will run and code will only get pushed if these tests pass in the target environment. Once we have a core framework in place, we can automate the process of nightly builds and testing. See tools like http://cruisecontrol.sourceforge.net/http://www.jsunit.net/

In the near future we will need to centralize test machines. For now having one test machine with virtual instances with different browser combinations will work. In the long term Selenium Grid will work till we get into Cloud Computing

A few more long term goals:

Additional Resources

Reference

Tutorials

Testivus

Testivus

Posted on by
1

Although we are striving towards integrating testing more closely into our development process we still have a long way to go. I thought this article does a good job of explaining testing philosophy in a fun way.

http://www.developertesting.com/archives/month200705/20070502-000424.html

To summarize:

  • All code needs to be tested.
  • Ugly tests are better then no tests.
  • Write tests often
  • Run tests often

A few weeks ago I gave a short presentation regarding testing with Selenium. Earlier today I sent out some introductory documentation as well. Hopefully in the coming weeks we will all begin to familiarize ourselves with some of the testing tools out there and begin, more seriously, incorporating testing into our build process.

Good Bug Tracking (aka How to make developers not want to kill you for logging ambiguous bugs they cant reproduce)

Posted on by
1

In the high-flying fast-paced world of web development it seems like the current “hot” technologies change every day. Luckily there some things that never change… bugs. While we are trying to improve our method of logging, assigning, tracking and resolving bugs, I’d like to share an article that gives general pointers on good bug logging. Although this article was written in ancient times (2000) it’s tips are still useful:

Every good bug report needs exactly three things.

1. Steps to reproduce
2. What you expected to see, and
3. What you saw instead.

Additionally here are the most relevant of the top ten tips for bug tracking:

1. A good tester will always try to reduce the repro steps to the minimal steps to reproduce; this is extremely helpful for the programmer who has to find the bug.
5. You’ll want to keep careful track of versions. Every build of the software that you give to testers should have a build ID number so that the poor tester doesn’t have to retest the bug on a version of the software where it wasn’t even supposed to be fixed.
10. Avoid the temptation to add new fields to the bug database. Every month or so, somebody will come up with a great idea for a new field to put in the database. You get all kinds of clever ideas, for example, keeping track of the file where the bug was found; keeping track of what % of the time the bug is reproducible; keeping track of how many times the bug occurred; keeping track of which exact versions of which DLLs were installed on the machine where the bug happened. It’s very important not to give in to these ideas. If you do, your new bug entry screen will end up with a thousand fields that you need to supply, and nobody will want to input bug reports any more. For the bug database to work, everybody needs to use it, and if entering bugs “formally” is too much work, people will go around the bug database.

I’m also going to add my own 11th top ten tip for bug tracking:

Screenshots help!

For more info check out the article here