Scandal in Game Reviewing!

Behold the return of the moneyhat!

The esteemed and always awesome Mr. Wil Wheaton put up a post recently about the influence game publishers and PR reps appear to have on the games media. He cites a post from another site, which in turn cites yet another site, both seeming to indicate that videogame reviewers are pimping scores out to the highest bidder and totally misleading you about games because You Can’t Trust The Man.

I spent some time responding in Wil’s comment thread, but it’s way down at the bottom and it’s also quite long, so I figured I’d repost here. I’d love to hear what you think.

Wil, there’s an element to this dialogue that a lot of people don’t seem to be considering, based on the sources quoted and many of the comments here.

Let me preface this by pointing out that in exactly two months I will have been reviewing games professionally for 15 years. Ten and a half of those years were spent at Ziff Davis Media, home of EGM, and later, 1UP. Most of that time was spent at the Official U.S. PlayStation Magazine (yes! a magazine! made of paper!), and most of that time was spent in charge of the Reviews section. Since OPM went kaput in late ’06 I’ve been a freelance writer, primarily doing reviews for many of the big gaming publications: EGM, 1UP, GamePro, GameSpy, PlayStation: The Official Magazine, OXM…er, I’m sure I’m forgetting someone, but you get the idea. During this time I’ve reviewed well over five hundred games, all for publication in major outlets.

What I’m saying is that I know how reviewing games works.

Over the past decade and a half, I can think of exactly one — one — occasion in which a PR person attempted to directly influence the score of a game before the review was written. It was the type mentioned in this articles: “you can get review code early if you agree the game will be at least a certain score.” We of course turned them down.

On maybe, oh, ten or fifteen other occasions, a PR person called me (in my capacity as reviews editor) to debate one of my reviewer’s scores after publication. And in every one of those occasions save one (in which a memorably loony PR dude pretty much went off his meds) they went away satisfied that their game was given a fair chance. Disappointed it didn’t do better, sure, but satisfied that we were evaluating the game thoroughly and fairly.

And that is, of course, a rightful part of the PR person’s job: to ensure the game is being treated fairly. And in my experience, the vast majority of PR people, and the publishers they represent, are ethical, sensible people who are as appalled by sleazy back-room dealing as journalists and consumers are. Because they know what every publication should know:

If a reviewer isn’t honest about the bad games, no one trusts them about the good ones, either.

Trying to artificially inflate a score is an incredibly shortsighted maneuver; it may bump up the Metacritic rating of the current game, but it kills the credibility of both the publication and the game company. If consumers buy a game that’s been artificially praised, they don’t just resent the outlet that did the praising, they resent the game company, too! And they’ll be that much more hesitant to buy the next game.

This is what I would tell the vocal minority of PR people: If we’re not honest about your crappy game, no one’s going to believe us if we praise one of yours that’s legitimately good. And most folks recognize this. That’s why these kinds of sleazy deals are the exception, not the norm.

But here’s the thing that I find particularly amusing about all this. So many people involved in this discussion (including many commenters here) use this news as justification for not trusting the big enthusiast sites or magazines. You even mention in your post not being able to trust 1UP.

But it’s the big media outlets that are most immune to these kinds of deals! The big media outlets know that the game companies need them more than they need the game companies; they’re big enough that they get their clicks or their subscribers whether one particular game is reviewed early or late; they have the budget and manpower to generate tons of non-review content; and perhaps most importantly, they know that if one particular company is going to withhold review code, they have plenty of other companies willing to fill those spots.

Furthermore, the big media outlets have ad-sales teams completely separate from the editorial teams. I know at Ziff there was an impenetrable barrier between ad and edit; we referred to it as the separation of church and state, and it was inviolable. Oh, we might hear that publisher X was threatening to pull ads — I mean, stuff gets around, you know? — but there was never — ever — any pressure from that side, or from our managers, to change our editorial content in any way as a result.

Now, I do know that hasn’t always been the case everywhere. The Gerstmann/GameSpot debacle is the most offensive example of ad influencing edit, but I can think of a few other stories (or at least rumors) I’ve heard over the years.

And it’s been a bit over four years since I worked full-time at a gaming publication, so I suppose things may have changed a bit. But if they have, it sure hasn’t trickled down to me; none of the publications I mentioned above has ever attempted to influence the score of a review I’ve submitted. Not once. Not even a little bit.

And of course this makes sense when you think about the power these bigger publications hold. If we really need to be concerned about someone falling prey to publisher and/or PR pressure, I think it’s the smaller sites we need to beware of, the ones who have limited access to begin with, limited resources to devote to non-review content, and limited staff to serve as buffers between pushy PR and writers. To be clear, I strongly doubt many of those succumb to that pressure, either. But wouldn’t you agree that they have more incentive to?

One final note before I release my choke-hold on an entire page of your comments section: If we want to point fingers here, we should consider pointing them at aggregator sites like Metacritic. The section you quote mentions that “sites which use letter grades don’t get advanced copies” because of how Metacritic translates them. And if you think of this from a PR person’s perspective, it makes perfect sense: Metacritic calls a “C” a 50 out of 100. If that same reviewer reviewed the same game on another site, it would likely get a score around 75, because most game publications use a number-based rating system that roughly translates to percentage grades in school: e.g., 60 or lower tends to be “failing.”

To combat this, either all publications could adopt the same rating system (ah, no) — or Metacritic could get their heads out of their asses and use some sense when standardizing scores: If a C is 50, fine — but make sure that for sites that only rate 60-100, an 80 is also 50. It’s pretty simple math, you know? Calculate the mean (or is it median?) score for each source, and make that the middle of the scale.

In closing, I’ll say this: It’s fun to bash on The Man; it just doesn’t always make a whole lot of sense. Also, this sort of thing is news because it’s the exception, not the norm. Also, I’m rather hungry.

Your fan,
-joe rybicki

17 Replies to “Scandal in Game Reviewing!”

  1. This post is rather heartening.

    I think there’s a certain expectation that something must be rotten somewhere in reviewer circles. And I’m talking reviews for anything, be it games, movies, kitchen appliances, whatever…

    In the end, people read all reviews with a grain of salt. And, as a result, I suspect most people depend on other users’ opinions more than critics. If I see a Metacritic score from users that’s significantly lower, or higher, I tend to trust it over the reviews.

    Part of that is just statistical. A given game might get 50 critic reviews, but thousands of users will rate it too.

    Reviews are worthwhile, and often helpful, but I’ll always trust my fellow gamer.

  2. then how is it that so many games get above average scores and are in actuality steaming POS?

    as far as credibility

    reviews, devs, pubs et all

    have very little anymore

    you sir, may well be forthright.

    the extreme majority of crap titles that get good ratings tell me someone isn’t

  3. It just seems that long ago, the reviews industry started to be plagued with grade inflation, basically. I stopped trusting reviews of that nature a long time ago.

  4. @ABCDawg, I agree that user reviews are often a better indicator of game quality — but only once the reviews have hit a quantity to filter out the crazies who give, say, a Madden game a 0 because they don’t like the cover athlete.

    @HSMagnet & cephyn, What you’re seeing is what I reference in the piece: Most sites only use the top end of their scale, with most games falling somewhere between 70 and, oh, say, 85 or so. If you think of it in terms of a grade in school, it’s accurate. But if you think of it in terms where 50 is the average (or, more accurately, “mediocre”), it looks like everything’s being over-rated.

  5. As someone who reviews computers rather than games, I’m surprised at how often people assume that every laptop, tablet, or other product I’m sent to review is mine to keep. Most of the folks who suggest this are only doing so because they think I’ve got a room filled with computers and I can afford to just send them one. A few take the time to think about how it would affect my judgment if I got to keep every $500 product I reviewed.

    I’ve tried to make my language in each review more clear recently to state when a product was *loaned* to me by a PC maker or other company for the purposes of review — but I have to say, the more I do this for a living, the more respect I have for Consumer Reports which makes a point of paying for every item reviewed out of pocket to avoid even the appearance of undue influence.

    Short of that kind of (expensive) action, I figure there will always be people who detect not only personal bias (which is unavoidable to some degree), but outside influence.

  6. Agreed, Brad. I would so love to see a Consumer Reports for games.

    But note that game reviewers often don’t get to keep product either; review games are usually gold discs rather than retail, and what retail games do come in tend to go into a company closet. As for stuff that doesn’t — what we did at Ziff was put everything in a pile to use as contest fodder and such. Anything left over would be up for a periodic drawing. I feel like it did a pretty good job of eliminating any potential influence by any given company.

    That’s not to say the games media is not subject to other influencing factors, of course! Junkets happen all the time…but it seems like publishers are getting better at not letting game companies foot the bill.

  7. You are absolutely right about focusing ire on Metacritic and on the companies that manipulate those ratings just to get their pre-sales numbers up

    Homefront has probably been the most egregious example of this in recent history. They were around 85 on their reviews when pre-ordering, then dropped way down when the game came out, when, I assume, THQ allowed the rest of the weaker reviews to be posted.

    They even bragged about their pre-sales at the same time the game servers were still crippled. It’s frustrating and seems out of touch and even archaic.

    It probably wouldn’t bother me as much, but I keep seeing their VP Danny Bilson out talking, pretty arrogantly, about what a different company they are (usually patting himself on the back at the same time), but if you compare his hype and his spiel to what they are putting out, it sounds like the old shell game to me.

  8. Thanks for this; to be honest, the question about marketing vs. editorial integrity has always bugged me, and it’s not very often I finally get to see both sides of the argument, especially the one presented here. Although I could see where those with problems were coming from, but something always bugged me about the argument, that it seemed somewhat attacking-the-straw-man kind of deal. This puts in a bit more perspective.

    I’ll be honest, I’m still not sure if I entirely agree with everything you have here, although it does make me wonder if it’s not the review companies that may be to blame in some instances but individuals within it. Still,
    the larger-companies argument does have a valid point.

    I suppose the question that still remains is this, and you may be able to answer this first hand: Does the issue happen in reverse? Can, or would, reviewers inflate reviews of games in order to ingratiate themselves with a company in hopes of establishing good PR with the gaming company and ensure review code in the future? It seems like the temptation to do so would be enormous, even without pressure, especially from up-and-coming reviewers or those suffering financial problems.

  9. Brian, I suspect very new, very small outlets would feel that temptation you’re talking about. And I’m sure some give into that temptation. But in the age of the internet, I don’t think many who do will last long. I think the reader’s ability to gather scores from various places would cause sites like that to lose credibility fast.

    Of course, there’s a type of reader who will ONLY listen to the tiny outlets, as evidenced by the comments on Wil’s post. So maybe I’m being unrealistic.

    As for the bigger outlets, the unglamorous truth is that they don’t really need to do that, for the reasons mentioned above.

    It definitely sucks for smaller outlets in the current environment, I’ll say that for sure. (I ran into plenty of these problems myself doing Plastic Axe — the frustration of not being able to get access because you’re a small site…but not being able to grow because you don’t have access.)

  10. I’ll have to say, there is another part of the problem. People who once may have relied upon the reviews from the big boys, and gotten burned because games had flaws that were glossed over, or things that were fine for others but not for them, or false expectations, all of which can feed into a feeling of having been burnt.

    For some people, their inability to trust the published reviews is either extreme cynicism, having been burnt more than once on games that either weren’t as good as the review made them out to be, or weren’t really their bag of catnip (both of which I acknowledge is a factor of writing skill), or a combination of the two.

    Gamespot did significantly add to the ‘cynicism’ part of the coin for me, though. That, and hearing about shows being cancelled because an industry made noises about pulling ads from a channel.

  11. I really appreciated your article and take on this, Joe. Not only did I re-read a bunch of the GameSpot/Gerstmann stuff (ironic that GiantBomb/WhiskeyMedia is down today), it was heartening to hear your general defense of the integrity of the gaming journalism industry.
    I write for a fairly small & new site (horriblenight.com), and while we do make jokes about it internally since we pay for our own bandwidth/hosting, etc. (e.g., “Don’t be too hard on game X! We want game X’s developer to give us preview copies!”), even down here on the small end of the spectrum, our individual and collective journalistic integrity is more important to us than Company X’s praise/demos and…you hit the nail on the head. If your readers can’t trust you to provide objective reviews, you’re not going to have readers for very long.
    Cheers and well said.

  12. I agree with @ABCdawg’s assumption that people depend on other users opinions. I personally have never bought a game as a result of a game review. I’ve relied heavily on opinions of friends (people I trust), and the subject matter of the game that I may be interested in.

  13. Joe, with you having worked on both multi-format and single platform magazines, I’d be interested to know how much influence (if any) the platform holders have on the mag’s editiorial? I once called out the editor of OPM in the UK for an issue where some high quality games were relegated to half-page reviews, while the big Sony exclusive for the month was Tekken (2 or 3 I think), and it got around a 10 page review.

    So while scores might not be directly manipulated, I certainly think PR can ‘buy’ larger splashes in magazines, which is just as insidious – would you agree?

  14. Rich, that depends what you mean by “buy.” It’s fairly common for a publisher or PR firm to shop around an exclusive: “If we gave you the exclusive to HUGE GAME SEQUEL X, would you give it the cover and at least six pages? No? OK, we’ll find somebody else.”

    Is that insidious? I strongly believe it is not. Here’s why: If the game is big enough that you’d agree to those terms…well, then, you’d probably be fairly likely to give it that much coverage on your own, wouldn’t you?

    As for platform-exclusive mags, I can only speak to my experience: Sony had absolutely no say in what we put into OPM. (The disc was pretty much ALL them, to be clear — but the magazine itself was all us.) On very rare occasions, we would give them a heads-up about something we were doing so that it wouldn’t come as a shock (e.g., putting an interview with Microsoft’s J. Allard against an interview with Sony’s Jack Tretton), but whenever that happened the decisions were already made. They didn’t have any input in the matter.

    Eventually, I think this was one of the reasons OPM was closed; the demo disc was a big financial drain on everyone, and the fact that we weren’t OK with being a mouthpiece (as Nintendo Power used to be, way back in the day) made it harder for Sony to justify the expense.

    But there was never any pressure from above to give special attention to system exclusives.

    However, keep this in mind: For a platform-specific magazine, system exclusives tend to be legitimately bigger news. (Meaning they sell more copies.) So decisions would certainly be made with THAT in mind.

  15. > The big media outlets know that the game companies need them more than they need the game companies

    You must be high if you believe this. If game journalism didn’t exist, games would still be made. If there were no games, though, would game journalism exist? I think not.

  16. @Visitor, OK, granted. But in the real world where both exist, individual game companies need the big publications more than the big publications need any individual game company.

Comments are closed.