Open beta: the double-edged sword

We’ve come to a time in the MMO genre’s lifespan where players expect and anticipate an “open beta” period. If an MMO developer doesn’t hold an open beta, we assume they’ve got something to hide. If an open beta period is too short we snark about the developer not really caring about our opinions because they haven’t left time to react to them.

The problem is that most of us players aren’t really beta testing; we’re sampling. We want to get into the open beta so we can play for free for a few weeks in order to determine if we want to buy the game when it launches (in some cases we’re playing for free with no intention whatsoever of buying the game). And the problem that follows is that we’re judging the game when it isn’t at its best.

This post is of course a direct result of the Star Trek Online open beta debacle going on right now. Cryptic has invited lots and lots of players into the open beta and the game is really suffering for it. On the one hand, I’m sure Cryptic is gathering a ton of data about where the net-code needs more polish and how much hardware they’re going to have to have on-hand to ensure a smooth launch. So that’s good.

But the bad news is that players are trying to play and having a horrific experience. I was actually laughing at how awful the lag and rubber-banding was when I created my first open beta character, but I’d played in closed beta so didn’t really need the tutorials, and I know this behavior wasn’t typical for the game. It ran much better in closed beta and I, being ever the optimist (HA!), am assuming it’ll run much better after launch, too.

But Joe Gamer, for whom this is the only STO experience he’ll know before deciding whether or not to part with his cash, probably isn’t laughing. From the onerous process of downloading the client to troubles logging in to not being able to move once you get logged in, the open beta experience is strongly urging him to cancel his pre-order and give STO a pass. And he’s telling his friends, who aren’t in the open beta, how bad the experience has been, so they’re canceling their pre-orders as well.

Now I’m not here to defend STO. I have pre-ordered but almost against my better judgement. But my issues aren’t with lag and performance so much as they are with the design of the game. I will say that you shouldn’t base your final judgement of the game on the horrible, broken performance of the open beta.

So what can game developers do? I don’t think they can completely fix the problem, but what would help is to offer trial accounts from Day 1. Let people sample, for free, the final, launched game. That would cut down on the number of beta testers who treat open beta like a preview program. This solution comes with the downside of a less robust stress test, of course. I suppose designers have to weigh the benefits of a good stress test against the lost sales of people getting fed up with the performance of the game in a crowded open beta.

You only get one chance to make a first impression though. I’ll speak heresy and suggest that a really bad open beta might do more harm to a game than a rocky few days at launch. Once a customer has handed over his $50 he’s apt to put up with a rough patch and keep checking back in until things get better (assuming that happens in days, not weeks). There’s nothing to keep a beta-tester around and once they walk away from your game, they’re probably never going to come back.