
A coworker of mine while at Google, Doug Bowman, recently
jumped to Twitter, and let fly an arrow at Google regarding the overuse of user data to make design decisions. Unfortunately it's misguided.
Data is good for design. Data type mismatch is bad for anything. The right level of abstraction equals the right tool for the right job:
Using analytics to make design decisions is like using raw sales numbers to justify green for your new product logo. And, it requires that you ship the product again to test if the color change worked, which can be costly.
Here are the flaws:
- You think you can control for all of the variables, but you can't: "we didn't change anything but the button color, so that must be why people started buying!"
- Customers may respond poorly to churning the UI: changing the UI to see if there is an effect, may cause users to leave your unpredictable UI.
- You must "test" again with the same metrics to know if it worked: requires you to ship the product again. Sometimes costly.
Oh, and it pisses your employees off apparently.
What Google needs to do is stop trying to make small design decisions with broad metrics, and instead isolate the variables, frame the question you are tying to answer, and use the right method.
Here's an oversimplification:
- Design = preference = survey
- Experience = usability = user research
- Features = useful = contextual / workflow