Metrics and The Vocal Minority

[Robby]

Metrics and The Vocal Minority
(News from Hero Engine)

by Daniel Cook from Spry Fox
Reprinted with Permission
I continue to be fascinated by the disconnect between vocal players / reviewers and player metrics.
The standard theory
Over on Play This Thing (Play This Thing! | Game Reviews | Free Games | Independent Games | Game Culture), a user made a comment about how vocal players are like vocal customers that complain about service at a restaurant. The silent majority may never complain, but are very likely to share the same issues as the complainer. The theory goes that even though only one or two people are pointing something out, the developer should pay extra attention to these words of wisdom since they represent large customer service issues that threaten your entire business if left unchecked.
This model is referenced by both vocal players and by self professed critics in order to boost their perceived importance and authority.
My experience
Yet repeatedly I keep running into situations where the model doesn’t hold true. A vocal sub-segment of the community claims that something is horribly broken. Yet when you run the metrics of player behavior or run surveys of the silent majority, the complaint is either not an important factor or is an attempt to manipulate the game in favor of a very specific minority voice.
Some examples (these have a lot more detail, but the gist will suffice)
Example 1
There was a virtual riot (complete with organized virtual protests) in Realm of the Mad God when we changed the range of projectiles. Suddenly players that could no longer kill enemies far off screen could because projectiles went half as far. We did this because a group of players had decided to primarily play using the mini-map and onscreen tactics were going extinct. The result was an outpouring of vitriolic complaints about how our changes had completely broken the game.
Yet fun scores and retention increased. Within a week or two of the change, people had moved on to talking about something else.
Example 2
In Triple Town, experienced developers that tried the game thought that the inclusion of an energy-like system in a casual game was a mistake. Traditional core players complained about the microtransaction system that was associated with the energy system. Kindle players disliked the fact that this isn’t a one time price for all you can eat. Choice phrases include one of my favorite quotes,“…genuinely good game, nasty monetization. Social games are evil, period.”
So one of the very first things we did was test how big an impact purchasing turns had on the player experience. The simple answer is that it doesn’t seem to alter short term retention or fun scores. We are still collecting long term retention data (the game just launched), but it is immensely obvious that the big huge overwhelming issues facing the game has nothing to do with the primary complaint of the most vocal players. It turns out that our biggest issue is the first 200-turn experience around the tutorial. This is something that not a single player or review has mentioned.
Example 3
In Realm of the Mad God, a small change was made to how a key for unlocking a special dungeon worked. This resulted in a massive fervor on the forums. A group of players was using these rare items as both a means of storing wealth and as a method of controlling who received rare drops. Since this group is the most dominant group on the forums, they created a large discussion about how the game was floundering.
Again, the metrics showed either no change or a positive change. On deeper investigation, the elite group essentially was using their economic power to dissuade the majority of the players from accessing top end content. There was indeed something very meaningful going on (players really want to self organize into smaller groups and control who is in the group and who is out of the group), but it was occurring in a dysfunctional fashion.
General themes
There are a few reason why the disconnect between public criticism and player behavior occurs
Communities are internally consistent games independent of your game
The first is that the community is playing a game that isn’t your game. Within a forum there are groups jockeying for power and influence. They’ll say whatever it takes to advance their position independent of the actual situation in the game. Sometimes this is an organized rhetorical attempt to gain an advantage.
More commonly, it is the result of limited perspective. To the players engaged in the game of running a mini-mafia inside of Realm, they were playing the game as their friends and the rules of the system dictated. The fact that ‘newbs’ suffered was unimportant since they weren’t meaningful players of the game at hand. The fact that ‘newbs’ represented 99% of the other players also wasn’t meaningful.
Loss of old skills and knowledge or the requirement that players learn new skills always provokes an angry response
Change that isn’t merely the addition of new content is almost always seen in a negative light. Learning has a real associated cost in time and effort. Anytime a game takes away a perceived skill, players feel that their time spent within the game is invalidated. They have immense difficulty imagining a better future as a result of the change and instead focus almost entirely on the intense sense of loss.
The game is seen as an opportunity to promote an ideology
Being anti “social games” has little to do with social games and almost everything to do with having an identity as a specific class of core gamer. How you buy and play games signals that you are part of an elite group and how you are so not your Mom.
In this light, any game that fits within a rough stereotype is seen as an opportunity to repeat a variety of message points. For core gamers, you get repetitive digs that X game is ‘a skinner box’, ‘immoral’, ‘not a real game’, ‘taking advantage of players’, etc. Whether these points fit the particular game at hand is rather unimportant.
Consider Hero Generations by Scott Brodie, a beautiful example of a painfully honest indie game trying to make the world a better place. Yet as soon as people see it is on Facebook, out come the various slurs that acts as a means of ‘dehumanizing the Other’ and help the post cement the poster’s position in the tribe of inward looking bigots. So you get hilarious comments like this one which one moment praises the game and then immediately starts repeating negative memes about Farmville:
“the game is nice, but just like the other crappy fb games, it looses its nostaliga in minutes. in reality, you do the same thing all the time, it sucks, tedious, and has bo ending, just like the milking machine called farmville.”
Needed: measurements of actual player behavior
The internet is a series of echo chambers where bias massively swamps any real signal about player behavior. The more specialized the community and the more that their interests are focused on activities other than the game at hand, the less meat there is in their mental models and proposed solutions. Arguably, this observation applies with equal measure to both vocal gamers, academic critics and journalists.
This situation puts game critics at a huge disadvantage. Their natural authority is inherently low and their comments are deeply and systemically disconnected from any sort of objective reality.
I continue to listen to passionate players, but I more and more preference measurable and observable player behavior over rhetorical opinion. There are indeed moments of truth that lead to improvements in the game to be skimmed from the morass of disingenuous deceitful, warped, navel gazing and self serving game commentary. For example, the complaints about destroying Realm of the Mad God’s accidental mafia helped crystalize thoughts about guilds. However, even in this case, the stronger signal was the behavior of the players, not the ranting itself.
Why are teams embracing design empiricism based off measurable player behavior? Because the alternative forms of critique offer so little that is meaningful or truthful.
take care,
Danc.
Links
- Good article on the forum game: http://www.eldergame.com/2008/06/taming-the-forum-tiger/
- Realm of the Mad God: http://realmofthemadgod.com
- Triple Town: https://plus.google.com/u/0/games/209089085730