Why free-to-play benchmarks don’t matter

• Tweet
• Pin It

I get a lot of questions asking me what the “standard” for a particular metric is in free-to-play games, whether that be conversion rate, ARPDAU, retention or something else.

My broad answer is “It doesn’t matter,” and it is very important to remember that.

The old world

In the old world of AAA games, forecasting, while not exact, was an established art. You would take the genre of the game. You would find out the expected Metacritic – although this was often an informed guess. You could then look up sales numbers, either from your own databases or from data supplied by companies like NPD, of similar games, in a similar genre, with similar review scores. This would give you a decent idea of how many units you could sell, provided that the game was as good as you had hoped, you did your marketing job properly and the sales force got enough boxes into shops.

It was not easy. There was a lot of hard work involved. But you could have some sense of how many units you might sell.

The new world

I was on holiday when I received a phone call from a sales director from the old world.

“We’ve just picked up an Asian PC F2P MMO and are bringing it to the West. I’d like to sanity check some forecasts with you.”

“Sure,” I said, gin and tonic in hand, “but you know that I basically can’t do this without playing the game?” He wanted to go ahead anyway.

“First month MAUs are 1 million.”

“Fine,” I said, “but do you really want to buy that many customers before you have fine-tuned the game for a Western audience? If it were me, I would spend no money on marketing until I had tested the game. I’d reduce that by half or more.”

“First month conversion rate is 10%,” he continued.

I spat out my gin. “That’s very high,” I suggested.

“They’re getting that in Asia,” he responded.

“But they’ve had five years to perfect the formula for their market,” I said, “I’d be inclined to halve that, at least, and probably much more.”

“First month ARPPU is $25.”

More gin hit the deck. “Are you sure?” I spluttered.

“That’s what they achieve in Asia,” he said again.

“But they’ve had five years,” I said again. “Let’s half that, or even more.”

So I had just halved his MAUs, I’d just halved his conversion rate, I’d just halved his ARPPU. Multiply that up and I’ve reduced his forecasts by 87.5% or nearly tenfold. And I still thought I was being optimistic.

It’s not to say that we couldn’t get those numbers, but just imagining we can lift a game wholesale, with its key operating metrics intact, from one territory to another is risky.

I didn’t get to do further work with this company. For all I know, the original forecasts may have been right. But my point is that when benchmarks can be out by a factor of 10, are they useful? I have seen monthly conversion rates of 0.1% and I have seen monthly conversion rates of 10%. One is 100x higher than the other. How useful is a benchmark there?

I realise that investors need benchmarks. I realise that financial teams want to understand if they are going to make a good Return on their Investment. But putting a benchmark into a model does not mean that you will achieve that number. Putting more work into making the benchmark figure more “accurate” will not help.

The reality is that we need to use benchmarks as a sighting shot. And then we need to test, and experiment, and learn. We need to use all of our design instincts, backed by research and analytics and data, to try to achieve high retention and conversion and ARPU.

Benchmarks are just a sighting shot. They are a start, no more. Stop worrying about what everyone else is doing and instead focus your energy on making your game more fun and engaging for your players.

You will make more money that way.

Related Posts

• 

  Kim Kardashian’s mobile game made $51 million in revenue in 4 months
  Kim Kardashian’s mobile game made $51 million in revenue in 4 months
• 

  Benchmarks are bollocks?
  Benchmarks are bollocks?
• 

  Is conversion rate usually quoted daily, monthly or lifetime?
  Is conversion rate usually quoted daily, monthly or lifetime?
• 

  Zynga’s conversion rate
  Zynga’s conversion rate