XboxGameZone’s Andie Gbinigie takes a look at public betas today and asks if they are still relevant to the industry.
The Beta. The stage in the development cycle that generally begins when the product in question is (usually) feature complete. At this point, the product will have bugs that need to be caught and eliminated prior to progression to the next phase of development. It’s a term many in the industry – ranging from consumers to professionals – have come to understand differently.
While absolutely essential internally to development studios, the role and importance of the “public beta” has regularly been called into question, especially when it comes to games with multiplayer portions. Microsoft (and 343 Industries) recently confirmed that the upcoming Halo 4 would not have a public beta. Ubisoft also announced similar, for Assassin’s Creed III. Activision studios working on the Call of Duty franchise traditionally don’t have public betas for their multiplayer games, with 2007’s Call of Duty 4 being the last title to have held one.
So, are public betas still relevant in this age? Are we seeing a trend back towards the traditional development cycle, with fully internal testing being the norm? To answer those questions, we need to understand how and why public betas are run, and the benefits and costs of them to both developers and publishers.
Public beta = Test from the couch
Public betas are by definition, beta tests that are open to members of the public. These usually take the form of either open or closed tests. Closed versions are released to a selected group of individuals, by invitation. Open versions usually encompass a larger group, and are generally available to anyone with an interest. In an ideal world, public beta testers report the bugs that they find and use the platforms provided to suggest additional features, fixes or tweaks that should be made by the developer in time for the final release of the product. There is an old saying that “two eyes are better than one.” By that reasoning, making a beta release available to more eyeballs should yield considerably better results, right? Not necessarily.
Back when I was a part of the Community Team for Infinity Ward, the developer took the unpopular decision to not hold a public beta for the then-upcoming Modern Warfare 2. Creative Strategist at the time – Robert Bowling – delivered an explanation of the decision:
“The internal beta will allow us all the benefits of a public beta test, without the impact it takes on the time required to prep, distribute, and manage a public beta during this crucial time in development.” he said. “That fact alone allows us to have a smoother and faster process of iteration when addressing issues and pushing fixes out to beta participants.”
It is no secret that it takes some considerable time to prepare, distribute and manage a public beta, a fact confirmed to me by the developers I was able to speak to about this subject. It also costs a fair bit of money. I spoke to a developer who was able to shed some more light on the process. For obvious reasons, they wished to remain anonymous. “I can’t give you a number. What I can tell you is that when a publisher wants to deploy a public beta on consoles, they have to pay a certain price per download. In the case of a closed beta, they pay a fee for each key issued.” My source went on; “Microsoft and Sony do things quite differently. For instance, there is an additional fee to host a beta on the PS3, even if nobody downloads the client.”
I tracked down another source who also agreed to speak to me on the condition of anonymity. They went into a bit more detail. “It [deploying a beta] isn’t cheap. The cost for first parties is around £200,000, or at least it was when we ran our public beta. Whether that’s changed or not, I don’t know.” They continued; “In a marketing budget, that can be a drop in the ocean though. And for a development with a big online focus that needs server issues ironing out – well, we all know how much testers get paid! A beta lasting a month can get a hell of a lot more exposure than hiring 100 testers, and without the overheads of office space, desks, paperwork, PCs, consoles and so on.”
Both sources did agree that public betas were worth the work and initial expense, and more often than not, easily paid for themselves. “So long as the beta client is loaded with enough ways to diagnose issues players encounter, and the servers log things, then it’s all good. It can pay for itself to tighten multiplayer, and fix server issues that would kill a big launch.”
Must.....find.....bugs!
With that in mind, why would any publisher or developer choose not to have a public beta? For starters, public betas are treated by most development teams as a separate part of the project, with its own milestones and deadlines. As such, in order to prepare one, a lot of teams have to split their workforce to create “beta teams”. These new teams then devote their time and focus to the beta project; handling submission, monitoring, and data collection. So, valuable manpower and resources are being diverted away from the main project.
The often private tussle between developers and publishers also plays a key role here. Again, one of my sources was able to educate me on the matter. “There are usually two different reasons for holding a public beta. The developer reason revolves around not knowing if the servers/clients will hold up in bigger environments (online) and as such, absolutely have to be tested with more than just internal QA/QC.” they said. “The publisher reason revolves around generating a bit of hype to get people onboard for a game that isn’t necessarily top of the public’s buying agenda, based on their research.”
Buy this, get access to that! - Marketing 101
More often than not, the publisher usually comes out on top when it comes to making such decisions. And if they do decide to go ahead with a public beta (from a marketing standpoint), then the timing of the beta in relation to the development cycle becomes doubly important. “Arguably the sooner after Alpha [you hold a beta], the better. By then, most functionality should be working, and the game is still loose enough to allow changes based on that feedback.” my source explained. “Having internal builds with play testers under NDAs is one thing; having a build in an Alpha state in people’s homes is another. There will be a great many bugs, and things might not look as pretty as finished. So, most publishers wait until a game hits beta. That means all the game is missing is final debugging, and polish. But at that point, it means the course is pretty much set, and changes become a lot harder to make too.”
There is no set time period for when a project enters beta, or how long it stays in that phase either. As such, public betas have been known to be made available anything from as little as 3 weeks, to as long as 5 months before release. In the case of more recent development cycles (and usually amongst third parties), public betas generally get made available a month before the game’s release. “As a rule of thumb, at the stage of 1 month before release – the build is done, approved, and is going to manufacturing.” said my source; “So a beta at that point in time is handy to start to dictate work for patches, or for server load testing. But don’t expect us to be making any changes to the code that will go on the disks. And of course, the free advertising that a beta brings then is another matter entirely.”
Having looked at the issue of public betas from the viewpoint of those selling the games, we should also acknowledge what public betas mean to those that purchase games. As mentioned earlier, the “no public beta” announcement for a popular game is usually met with fierce disappointment by fans. While many gamers now view alpha/beta access as a badge of honour, a growing number use public betas to make purchase decisions. It has been a slow but perceptible shift in the mindset of many gamers that has undoubtedly had an effect on the number of developers and publishers releasing public betas primarily for marketing purposes. Those who still choose to do so find that they are running an increasing risk of being accused of releasing “glorified demos”, with knowledgeable gamers already being aware that the beta version has been polished to be as bug free as possible, in order to prevent putting off undecided customers from the final game.
It will be interesting to see how things change regarding this issue over the next few years, especially with the transition to the next generation of console gaming. While the likes of Microsoft and Ubisoft appear to be moving away from public betas for their marquee franchises, others such as EA and Capcom continue to offer these tests for public consumption. It would appear that public betas are still relevant to the industry. The question now is, for how long?
Related News:
- No multiplayer beta for Assassin’s Creed III says Creative Director
- No Beta Planned for Halo 4
- Preorder Medal of Honor: Warfighter to access the Battlefield 4 beta
First of all I want to say what a well written and very informative article.
I’ve played/tested quite a few betas over the years. My first was the World of Warcraft beta, which I didn’t like and hence lost Blizzard a game sale. Another game was the Medal of Honor reboot from a couple of years ago. I downloaded the MP beta for the PS3, played it for a short while and I knew then that the game was steaming pile of BS and yet another game sale lost. So to recap, If I’d not played the betas on the 2 games I’ve mentioned then I’d have most likely purchased them. Sometimes a try before you buy is a bad thing.
As for internal or public betas then I’m not fussed nowadays. A few years back I would have been screaming from the rooftops to whoever would listen to me, and rant about CoD games not having a public beta. I’m like a fine wine and have mellowed out. Maybe too much :D
Well first of all
Feedback: you did a great job, I enjoyed reading this for a big reason, you asked multiple developers, to give real insight of how the industry works rather than just your own opinion, I liked knowing more about the people of the industry, the ones who make the game, and what they think about the beta.
Comment: I understand all there is to a beta now better than before, but personally I am against the late Beta, as already said the game is finished, they are just tweaking small stuff here and there, why not have Alpha as a test, we start with the developers, share our ideas, help improve the game, ……….etc, cause sometimes the gameplay is just messed up, and need alot of feedback, a late beta is more of a MP Demo, cause no matter the feedback it will always be a super facial stuff, nothing deep, nothing big will change.
An Alpha should be to people committed to a series, to give valuable feedback, I am not saying go all hardcore, no, have a mix, but for those who really care about the series and not just passing by.
A Beta/Demo although takes alot of money, it may bring you alot more, since some people are not sure if you are providing an exciting experience or not, they may even not know about the game till the beta or demo, so it is two sided coin.
Personally I do alot better on Beta’s than the game LOL, well I invest alot in Betas, I was part of GRFS beta, Uncharted 3 Beta, Dust 514 beta, Bad Company 2 Beta, Battlefield 3 Beta, ……….etc, so yeah I sped major time around them, they didn’t affect my opinion about the game since I follow the news closely so I know exactly what I am getting.
I don’t know how I feel about a Beta, you know again my major feedback won’t change how the game plays, it is more of a demo, and at the same time it is something good to keep me busying till the game is released, especially if highly anticipated, but I think Betas in their current form are useless.
Advice: well do more of these, giving insight of the industry, asking developers the hard questions, ………..etc, that would be totally cool.
Great comment Sameer, and i agree with you pretty much all the way.
I honestly think we’d see more public betas if more gamers treated them as actual tests (reported bugs, etc)
As you can see, it costs a hell of a lot to put them on, so devs and pubs need to see it is worth their while! Maybe then, they will be less marketing demos, and more a compliment to internal QA.
In my opinion, a public beta is essential, especially when it costs the company nothing extra (taking into account the beta paying itself off). An internal beta fails to test all playstyles that will appear in the final game, a con which outweighs the possibility of issues not being reported.
Good article, Esco. I feel like public betas are no longer as relevant, but they should be. Very informative article.
It’s funny because at first I would assume a “beta” would be the best course of action for ANY developer looking to refine it’s game, but after reading this article I’m not so sure anymore… not sure if it’s worth the money, worth the time or manpower required. What makes me really wonder though is how much of an impact does (public) alpha/beta really make nowadays, and are the intentions of a developer hosting the testing the same intentions that we’ve come to understand over the last decade or so? Yes, working out the “kinks” sounds about right on paper, but even as you briefly stated, maybe the focus of these modern testing sessions is centered around the advertising advantages, not necessarily interpreting the consumer experience to improve gameplay mechanics. Even WORSE, (and this is my opinion of course), is it so far fetched to believe that developers may be looking for insight on how the “average”, most “successfully marketable” consumer plays and experiences the game? What the mass majority finds appealing (or not so appealing) about mechanics, features or functions? Because if you honestly think about it, this could account for the more recent “dumbing down” of popular titles and new IP’s, the loss of in-depth game mechanics and intricate story lines, the introduction of cheap and overpowered multiplayer aspects that basically have no appeal to any real “GAMER”…
It’s just extremely suspicious nowadays, alpha and beta testing in general, and you definitely made me think about this one… Excellent article Esco
Well, Beta is important to me as I don’t want to fall on the same mistake with ACR
Hopefully their internal test teams are on point with AC3
Thanks for the great comment Jimmy. And yes, from what i gathered, the publisher will almost always view the beta period as extra marketing time.
Its a shame, (as mentioned before) that more of the public who get these testing opportunities don’t use them the right way.
The “glorified demo” tag is justified when a beta is clearly too close to release to be considered as such. Could feedback help for patches down the line? Sure. But for all intents and purposes…it should be called a demo, and treated as such.
I’ll keep my comment short, in any beta I was a part of, I’ve never felt like anything more than a cog in a marketing machine. Sure, they fixed the hole in Op Metro. High five.
One question, now that cloud updates are implemented does that allow a dev a shorter time frame of beta to release to still be able to make significant changes?
Thanks for the comment Matt.
With cloud updates, there is a limit to the size of file that can be pushed to the servers. Speaking from an XBL perspective, this is dependant on whether the dev uses the basic MS server structure, or rents title servers via the Xbox Live Server Platform. I imagine smaller fixes can be pushed faster and more regularly, but would still be prohibited by cost.
After reading this I am kind of torn a bit. I understand the resources and expense that goes into a beta quite a bit better. I can understand why public betas are starting to go away, but the thing that sticks with me is that several sources agreed that the beta process is well worth it. Then there was mention of the late beta, in which the game is finished and the beta is being used more to collect notes for patching the game after release, which just sounds like a terrible Idea.
Wouldn’t it make sense for maybe the industry to change their release information? For instance, a developer and publisher set a date they want the game to release, but instead of making the release date public information, instead plan for a beta phase and when revealing games to fans, instead of giving the release date, they’d give the beta phase window like “Early summer we’ll be running a 6 week beta for ‘Super Game Pro X’. The game will be released X weeks after the conclusion of the beta”. That way, deadlines would still have to be made, but if larger problems emerge during testing, the beta can be extended as needed without harming the overall final release date. As a bonus, people who are undecided have more time to play the beta, and the more die-hard fans can feel like their voices are being heard when giving feedback.
Ok, I see. After reading this I have to admit that I care even less about betas. I never felt like I had a voice, but I always hoped that gathered info from playing and, for instance, popping a red blip on a heatmap at a certain ridiculous choke point would be used to implement things into the final game. Or server load testing would make connection issues that plague most games in first week a thing of the past. I also don’t like the idea of dev team members being pulled away from the actual project to help release something that is basically a giant billboard promoting the game for undecided buyers. If the full game is the album, the beta is the single. Which by definition, is a demo. I also feel that it is no coincidence that fans get enraged by CoD not releasing betas so their number one competitor, BF always does. That’s full on marketing, not testing. It really is a shame and a sham.
Hey Panther, thanks for the comment.
That would work, in an ideal world. Sadly, influences from shareholders, and marketing departments mean that release dates need to be nailed down well in advance. Outside factors like projected sales numbers and even preorders can mean shuffling of said dates around.
Those things can have marked effects on development schedules, and potential beta phases. Its unfortunate, but a heck of a LOT goes into shipping a game, and so many pieces of a puzzle need to come together. Can the process be better? Perhaps. But for now, its the way it is.
Quite sobering isn’t it Matt?
Granted not all devs and/or publishers are the same, but i agree with you to a point.
Ultimately, i’m old school. If a developer have a competent test team internally, then i can place my faith in them to do a stellar QA job. If they have a less than sparkling track record however, then a public beta is more than worth the money, even if part of it is marketing.
The main problem now a days with public beta’s is the publishers bean counters. Its cheaper to release a game with known issues and patch it after release. Especially when you consider a public beta on one of the 1st party platforms is going to cost the publisher a lot if they patch stuff via TU’s as the beta progresses without even including the money and man hours needed by the dev team who are working to a usually forced (and usually behind) deadline due to release dates set by the publisher.
Nowadays I am firmly of the opinion gamers see the first month or so of a games release as the “beta test” (or in the case of call of duty elite and mw3 for example) the first 9 months after release. A games success also plays a big part on any decision to make a public beta, the problem is to hold a viable and useful public beta you need the hardcore fans who love the game to participate as you know they are beta testing to test not to demo The problem with this is though if you have already alienated your player base. .
A good example of a working public beta system is World of Warcraft where for the most part you have the real die hard fans going over ever single thing as they care about “their” game and the levels of interaction and listening to feedback that Blizzard show to their community. You tried to do the same thing with say CoD and you’d have a beta full of people more interested in making videos about it and finding exploits and holding onto that knowledge till the game/dlc is live then abusing the exploit.
The reason for this is “imo” solely the publishers fault once you start looking at your games players as dollar signs and lose the community interaction and making them feel they are part of something , and above all respect your customer base or you lose a valuable commodity.
As I mentioned on twitter, I remember when beta testing was mindless drudgery and to actually beta test involved a long protracted vetting process, I actually can’t remember when I last saw a public beta test that actually had even a questionnaire to fill in before your could even apply. Nowadays it seems the only qualification you need is $60 or a pre order.
Whoever your “source” is I have a sneaky feeling they work for a company I’ve done alot of testing for or the mentality is industry wide nowadays. ;)