Why Real-Time Data is Worthless

by Travis Ketchum · 0 comments

Obsessively checking your stats

The trendy feature of every analytics tool right now is real-time data. You’ll get to watch visitors move through your site so you can pull insights in real-time and react immediately.

This sounds great in theory but it’s a waste of your time.

Why? Because you need two things in order to get any value from real-time analytics.

Thing #1: Massive Amounts of Data

Go into your analytics and look at the traffic for a specific page. Now look at the traffic for a single day of that single page. Most likely, you’re now dealing with a very small data set.

When you have small data sets, it’s incredibly difficult to pull insights from them. When the data set is this small, a single outlier (a visitor behaving completely differently than normal) can throw all the data off. If you’re not careful, you could optimize for the outlier instead of your real customers.

Real-time makes that data set even smaller because you’re only looking at current visitors. Instead of seeing how your customers behave, you’re looking at a VERY small percentage of your traffic.

And when a data set is this small, you can’t grab insights from it. Real-time data doesn’t even become an option until you have massive amounts of it.

Even when you have massive amounts of data, looking at metrics even on a daily basis (let alone real-time) isn’t that valuable. There’s simply too much volatility from one day to the next. The only way to pull insights is by looking at the data on a weekly or monthly basis. Then use daily reports to deconstruct specific events.

Here’s a good rule of thumb: if you don’t employ a data scientist, you don’t have enough data to analyze it in real-time.

Thing #2: The Ability to Act Immediately

For the sake of argument, let’s say you have more than enough traffic to get good data within a few hours.

So you roll out some changes on your home page. 4 hours later, we have enough data to see how people are using it. We discover that no one clicks on the fancy new slider. So we want to redesign it.

Honestly time: will you be able to redesign and relaunch the slider within a couple of hours? I HIGHLY doubt it. Your team is busy and they’re already working on other projects at this point. Revamping the slider is a high priority but it’s not an emergency. Assuming the schedule of your team is somewhat flexible, you won’t be able to get to it until tomorrow at the earliest.

And since you can’t act on it till tomorrow, there’s no reason to keep checking on the data every hour. You’ll just be wasting your time.

This rule applies to any report, not just real-time data. If you have a weekly KPI report, you better be able to act on that data within the week. If nothing will change until next month, make it a monthly report. Your time is extremely valuable, don’t waste it by looking at data you can’t act on.

Now, there are two circumstances where you’ll have both of these conditions (good data and the ability to act).

Split Tests

Many split test vendors give you data that’s nearly real-time. And they allow you to quickly make changes to your tests (dropping the poor performer and selecting a winner). With enough traffic, you can act on this data immediately.

Product launches are a perfect example. When you launch a new product, the first 24 hours is critical to your success. Especially if you have carefully planned marketing campaigns driving traffic. You want sales to be as high as possible to hit that tipping point where word-of-mouth starts to kick in and work for you.

But this is the first time you’ve published your product page. The copy, layout, and offer are all untested. Building multiple versions, running a split test, and optimizing for conversions during this 24 hour window is critical to figuring out what works the best. So set up your split tests (Google Website Optimizer, Visual Website Optimizer, and Optimizely are your best options) and make the winning version permanent as soon as you have statistically significant data.

Since you’re looking for very specific data (conversion rates) and you have an influx of traffic to give you good data, you can act on it immediately through your split testing tool. Real-time data helps immensely in situations like this.

Now, this may not apply to all your split tests. you’ll need hundreds of conversions in order for real-time data to be of any value.

Social Games

If you have released a social game on Facebook, the iPhone, or iPad, you are drowning in data. Especially if you’ve achieved any degree of success. In this case, real-time data can help you figure out how your users are evolving through your game.

But the second constraint still applies. Your engineering team has to be iterating super rapidly for the real-time data to matter for anyone. If they push out product updates on a weekly basis (which is still crazy fast), ignore your real-time reports and focus on your daily or weekly metrics. Only use real-time data if it will impact the changes you make to the product today. If nothing will happen till tomorrow, use daily reports.

Bottom Line

When looking at any metric or KPI report, ask yourself two questions:

  1. Do I have enough data for it to be statistically significant?
  2. Can I act on this data before the next report?

If the answer is no for either of these, stop looking at the data. Either pick different metrics altogether or extend the time between your reports.

And when it comes to real-time data, you’ll rarely have either of these. Instead, focus on daily, weekly, or monthly reports for your KPIs. Your time is simply too valuable to be looking at data that doesn’t tell you anything.

About the author: Lars Lofgren, the KISSmetrics Marketing Analyst and has a Google Analytics Individual Qualification (he’s certified). Learn how to grow your business at his marketing blog or follow him on Twitter @larslofgren.

About Travis Ketchum
A smart ass marketer who doesn't take no for an answer and always questions the status quo. Connect with me on Google+. Convinced yet? Get more tips and great content 100% free.

Follow me on Twitter · Visit my website →

Building a sustainable business is all about how well you can gather and maintain an audience. An email list is still one of the most viable ways to do just that.

We wanted to find a way to build an email incredibly fast, in a way that people actually find interesting, engaging and well - cool. It took a lot of testing to weed-whack through all the hype and find something that really worked.

The result? We ended up building our own solution, focused around the idea of contests and rewarding people for taking the actions that ultimately led to more leads on our email list. Everyone wins (and some literally do!), because as it turns out people love contests regardless of their market place.

Click to continue…

{ 0 comments… add one now }

Leave a Comment

Previous post:

Next post: