Does good design really matter?
Because it’s so hard to measure the business value of good design, it naturally gets less attention from managers. It’s easier to watch things like visitors to a website, sales per day, rent and other costs.
Recently we got a chance to see how much it matters when we rolled out a redesign for our Freedback service. Here are some of the changes we made:
- Reduced the number of steps required to create a form from four to three.
- Removed list of forms on the left to reduce visual clutter.
- Added breadcrumb navigation so it’s easier to see where you are.
- Users now create forms first, then decide what happens when they’re submitted.
between the new and old versions)
Digg users:
It’s possible this mouse-over won’t work for the cache you’re seeing.
Click here to load the original page.
Results
To be honest, when we rolled out the new software we hoped to get lots of emails thanking us for the new interface. Didn’t happen. We did get a few phone calls, mostly from people reporting bugs, and by the way did we change something?
After squashing a few bugs all was well with the world. We never did get much customer reaction, so we watched our graphs to see if we could see any change in how people used the software.
Number of forms created
Revenues
Thoughts and Conclusions
- When things are easy to use, people use them more.
- We didn’t see a dramatic increase in revenue from new users who had never seen the older version, but older accounts coming up for renewal were more likely to upgrade. This agrees with previous experience from other system upgrades; existing users like to see software improving.
- Hopefully this will translate into more links, reviews and word of mouth traffic, but that’s harder to correlate to this change because it takes time.
Update:
My wife knows much more than I do about graphing data (she’s an astronomer), and was nice enough to plot a graph using the raw data that better showed the results we were seeing.
This graph is showing daily Freedback revenue before and after the change (note the y-axis doesn’t start at zero).
- The green dots are days.
- The black line is a four-week moving average.
- The 13% represents the difference between one month before and after the change. It held true for six weeks before and after as well.
- Our weekly revenue cycle (lowest on weekends) results in a lot of scatter.
I hope this data is useful to other entrepreneurs wondering if they should invest in design, questions or comments welcome. I’ll keep posting graphs and results from tests we’re running - if you’re interested you can subscribe to our feed.
April 27th, 2007 at 2:51 pm
While the 13% rise is very cool, and congrats, one crucial data point is missing… daily unique visitors before and after. If your uniques stayed about the same in the two periods, then there’s a much greater weight to the proof of good design being the driver of the form creation and revenue increases.
But if uniques went up 15% in the same period, then that 13% increase would prove that the old design was more successful (though it’s possible some rise in uniques might be due to people recommending the new design more or the new design providing better SEO benefits).
Without knowing if there was any change in unique visitors, it’s hard to evaluate the significance of that 13% and how much the new design contributed to it.
How about it? Let’s see a graph of daily uniques overlaid on the other graphs.
April 27th, 2007 at 2:55 pm
Hey Greg,
Great point!
Even more relevant is actual signups (account creations) over that time period since we didn’t make any changes to the external site.
Signups actually fell slightly over this time period, so it’s not an increase due to that. I’ll add that graph later today.
- Aaron
April 27th, 2007 at 3:34 pm
I think the interesting thing is that you state significantly more of the revenue increase was driven by users who had seen both versions converting from the free option to a paid option than by newer users who never saw the old interface.
Because you attribute some of that to “existing users like to see software improving,” it would be interesting to see if the increases hold over a longer term. It seems you’re on an upward trend, but once all the old users have seen that it’s improving, will you see a drop off when all those older users who are likely to be influenced by the upgrade have been influenced? If the increases hold past that point, then it shows that the design and process improvements are having an effect on all users, and the specific improvements are impacting growth.
April 27th, 2007 at 3:41 pm
Definitely. Historically revenues after we make a change, then settle down at a higher point than they were, but not always as high as their peak.
In general it’s hard to test over long periods because we’re always improving things, so the control is always improving. We’ve thought about splitting out a set of users to be a long-term control, but concluded that maintaining two codebases is a recipe for serious pain!
April 27th, 2007 at 8:39 pm
Hey Aaron -
I thought the previous version was intuitive, and I love the service, so I would have tolerated a really offensive design (not that you ever had one!).
But I think the re-design indicates that it’s more accessible to newbies, and that’s a good thing.
April 27th, 2007 at 8:43 pm
Thanks Shawn, glad you like it.
If you have any suggestions for ways we could keep on improving it please let us know!
April 29th, 2007 at 11:14 am
Can you really determine if good design was effective simply by customer response? I think most of us working in the field of design realize that most feedback will be negative feedback — people don’t generally go out of their way to tell you that you did your job correctly or you made a product aesthetically pleasing and usable; that is the default expectation.
Good design isn’t appreciated, that is the bottom line. Good design is expected. Positive design feedback typically comes from designers and negative design feedback typically comes from users.
Your basis for evaluation is off — determine if you have a successful design is your number of user issues diminishes not if your number of compliments increases. Less users with less issues = success.
April 29th, 2007 at 11:42 am
Hey Martin,
Thanks for your comment. I agree, you don’t really know anything based on what customers say to you.
If you check out the article and graphs you’ll see that our evaluation of success was in how users’ behaviour changed, both in usage and upgrades (revenue) and not so much to do with what they said.
April 30th, 2007 at 10:23 am
Not to slam anyone but I found the statement “I would have tolerated a really offensive design” as a funny statement and here is why:
As a designer and a having high standards, if sommeone is Tolerating a design then the design failed just as If you have to give instructions verbally or written the design failed.
Sure there is some learning with any site to figure it out and utilize it’s capabilities, but here we can see stats to show how effective a GOOD inutitive design can be effective in the education process as well as usability for existing users allowing them to save time and stop “tollerating” the shortcomings of bad design.
Excellent case study.
April 30th, 2007 at 10:59 am
[...] reading excellent http://www.icecreamforeveryone.com/uncategorized/does-good-design-really-matter.html about recent redesign and results. 2007. 04. 30. 19:58 [...]
April 30th, 2007 at 12:28 pm
Collin,
I think part of where that “tolerate a bad design” sentiment comes from is because Freedback helps people accomplish a task. So as long as it does a good job at that, one might tolerate other faults.
It is a blurry line though - better design should make the task easier, and that’s what we were aiming to to here.
May 3rd, 2007 at 9:09 am
Only partially related, what software did she use to make that graph and does it export to EPS :) I”m tired of the bad interface in SigmaPlat and Matlab is a little obtuse.
May 3rd, 2007 at 10:31 am
Oooh, great question. I have no idea. I think she may have done it in Excel of all places. I’ll find out.
May 3rd, 2007 at 4:34 pm
User-friendly is the key. The first one looked kind of difficult to someone who wasn’t computer literate.
May 6th, 2007 at 4:43 am
Hi Jason,
I did indeed use Excel to make this plot. Unfortunately, excel does not directly export as an EPS file, but there are some work-arounds for that. One could print to PDF and then pdf2ps and then to eps. but often the resolution of the output isn’t great. If I want a really nice looking plot, I use XMGRACE which is a unix based (with GUI) which is free for the taking: http://plasma-gate.weizmann.ac.il/Grace/. it exports nicely to ps, eps, jpg, png, etc.
May 17th, 2007 at 11:23 pm
[...] Does good design really matter? (tags: design business webdesign) [...]
September 1st, 2007 at 10:35 pm
[...] read more | digg story [...]
November 3rd, 2007 at 8:03 am
[...] read more | digg story [...]
February 23rd, 2008 at 2:22 am
Hi Guys,
I just installed a freedback form on my site. I was over the moon to discover how easy it was to register etc.
Great job.