I spent a substantial amount of time designing the first survey for the WoW Learning project. I was initially unsure what data would be useful, but I knew I wanted enough data to make statements about particular groups of people: men were more social, most women created tank characters initially, millennials were using WoW more for learning, etc. That resulted in the three-part design of the survey: in-game demographic data, the essay question about play motivations, and real-world demographic details. Privacy was important and encouraging people to complete was also important. It was reasoned that in-game demographics were details people would not be as sensitive about, so they were asked for first and real-world demographic details left until the end. It was also recognized that many people would not complete the essay section. Putting the in-game demographic details first meant that certain types of data could be collected that could also be used for other purposes, e.g. answering whether women initially choose healer characters. After use by a small test audience, the survey was modified to include sample answers or explanations of how answers should be calculated.

Where possible and sensible, permissible option lists were used to help reduce the need for data standardization after the survey. For example, I know there are only so many WoW character classes and roles. I know that players can only belong to one of several types of realms. In the case where I wanted time estimates from players, providing a list of ranges means everyone’s has the same degree of accuracy and is expressed in the same units. This was not deemed necessary for year of birth, country of residence, and nationality; they were left as free text. The birth year worked out fine, but some normalization had to be done on the countries and nationalities, e.g. English and Scottish changed to British and Belgium changed to Belgian, etc.

Read more to download the survey as a PDF and see options for specific questions.

Details on Specific Options:

The attached PDF shows the design of the survey as it appeared on SurveyMonkey, the third-party commercial survey hosting service I used. However, because it’s been saved as a PDF from a web browser, you can’t see the drop-down menu items. Where that happens, I’ve described the questions and their options in this section.

Part 1: Q1: How long have you been playing WoW (to the nearest half year)? If you played in the public or closed beta, you can count that too.

Options here were given to the nearest half year, up to 6.5 years:

• 0.5 years
• 1.0 year
• 1.5 years
• 2.0 years
• 2.5 years
• 3.0 years
• 3.5 years
• 4.0 years
• 4.5 years
• 5.0 years
• 5.5 years
• 6.0 years
• 6.5 years

At the time of the survey, it wasn’t possible to have played more than 6.5 years.

Part 1: Q2: Tell us about the first character you created. Tell us about the character on which you currently spend most of your time. Tell us about the character you enjoy playing the most.

This was broken down into race, class, role, and time played for each of the questions. For race:

• Blood Elf
• Draenei
• Dwarf
• Gnome
• Human
• Night Elf
• Orc
• Tauren
• Troll
• Undead

For class:

• Death Knight
• Druid
• Hunter
• Mage
• Paladin
• Priest
• Rogue
• Shaman
• Warlock
• Warrior

For role:

• Healer
• Tank
• Ranged DPS
• Melee DPS
• Pure Role-Playing

Pure role-playing was added for people who were using the game as a world in which to base their storytelling rather than engaging with the provided gaming content elements. With the population I was recruiting primarily from, I did not expect there to be many role-players and the option was missing from my original design, but I added it after testing.

For time played, options were given to the nearest 15 days until 300 days. After that, it jumped to the nearest 100 days. It’s important to note that the time asked for here is /played time. That is a measure of time spent actually in the game and not a measure of calendar time. This information is available for any WoW character you can still play. I ask for /played because I am trying to assess a person’s game experience. You could have two players who have both played the game for 6 calendar months, but one might have spent 20 hours a week versus the other only 1 hour. See the discussion in /Played Time as a Measure of WoW Experience. Possibilities here then were:

• Less than 15 days
• 15
• 30
• 45
• 60
• 75
• 90
• 105
• 120
• 135
• 150
• 165
• 180
• 195
• 210
• 225
• 240
• 255
• 270
• 285
• 300
• More than 300 days
• More than 400 days
• More than 500 days
• More than 600 days
• More than 700 days
• More than 800 days
• More than 900 days
• More than 1000 days
• No idea

Part 1: Q3: The next choices relate to the type of game realm where you play. If you play on more than one, pick the realm used by the character on which you spend the most time.

This was broken down into type of realm and location. For realm type:

• PvE (Player versus Environment)
• PvP (Player versus Player)
• RP (Role-Play)
• RP-PVP (Role-Play Player versus Player)

For realm location:

• China
• Korea
• Europe
• US

Part 1: Q5: How much time per week in hours do you estimate that you spend on average in game? If it’s more than 40, choose that from the menu and enter the number of hours into the short comment field provided.

This was broken down into 2-hour chunks:

• Less than 1 hour
• 1 to 2 hours
• 2 to 4 hours
• 4 to 6 hours
• 6 to 8 hours
• 8 to 10 hours
• 10 to 12 hours
• 12 to 14 hours
• 14 to 16 hours
• 16 to 18 hours
• 18 to 20 hours
• 20 to 22 hours
• 22 to 24 hours
• 24 to 26 hours
• 26 to 28 hours
• 28 to 30 hours
• 30 to 32 hours
• 32 to 34 hours
• 34 to 36 hours
• 36 to 38 hours
• 38 to 40 hours
• More than 40 hours

One survey design flaw I see often is, when providing a list of options, not covering edge cases where it is not applicable at all or someone does something you did not expect. Although I would expect that all participants have played the game and are regularly playing the game, there might be some who do not play at least an hour per week or who are spending more than a full-time job in the game.

Last Bit: Q1: If you’re happy to be contacted with any queries about your responses for this survey or to participate in future surveys, you can leave an e-mail address. This will be associated with your survey responses.

This was an optional question as I was purposefully trying to avoid collecting any identifiable information about participants. However, I could foresee circumstances where it might be useful to follow up on responses or issue invitations to future research surveys. They had simple yes/no answers, with the default being “No”.

Downloadable Resources:

• WoW Learning 2010 Survey (684 KB PDF)

No related posts.

Figure 1: Graph demonstrating that Aluminio’s reported character played times add up to more time possible in 4 years

In a previous post, I said that a player’s /played time in World of Warcraft (WoW) can be used a good indication of their experience in WoW.  In my 2010 April survey, I asked respondents to report their /played time for three types of characters: their first character ever created, the character on which they currently spend most of their time, and the character on which they enjoy playing the most. If the characters were the same, they were asked to repeat the information. When I did my calculation, I ignored any entries that were obvious duplicates. I also asked people to make an estimate to the nearest half year of how long they had been playing World of Warcraft.

While entering data in from my 2010 April survey, I noticed that case S1-025 contained /played numbers that did not add up.  In the raw survey data, the participant—whom I have called “Aluminio”—listed 3 characters:

1. Human priest ranged, played more than 700 days
2. Gnome mage ranged, played more than 900 days
3. Human paladin tank, played more than 900 days.

That adds up to more than 2500 days. Aluminio also reported playing World of Warcraft for a total of 4 years, which amounts to 1460 days. That’s far, far short of the more than 2500 days claimed for playing his three characters, as is clearly evident in Figure 1′s graph. It is impossible for someone on their own to have played all 1460 days 24 hours a day, seven days a week.

Credit: Michelle A. Hoyle under an Attribution ShareAlike 3.0 Unported License

Figure 2: Graph showing how much of a four-year period would have been spent on each of Aluminio’s characters.

My first thought was to wonder if they were using bought accounts.  In which case the /played time would include the time the original person spent on the account and, if these were all bought accounts, then the time would overlap potentially with survey participant’s personal playing time.  It need not just be someone who has bought someone else’s account either.  A player named “Sodiumo” in my guild took over the account of a friend who stopped playing.

My partner mentioned a second possibility: multiboxing. This is where a person has multiple account and multiple instances of the game running.  You set each of the other characters to follow the character you are actually playing as you take them through dungeons.  This would accumulate /played time on all the logged in characters, even though the person is really only actively playing one character.

A third possibility is the person had multiple accounts and someone else helped them play each character.   There are several similar cases in my own guild.  For example, a player we will call “Carbona” is the primary player on his account for questing, socializing, and raiding.  He lets his son, however, engage in player versus player encounters on his character.  Another player, whom we will call “Chlorinella”, ended up taking over the account of her brother who lost interest in the game; Now that Cataclysm, the latest World of Warcraft expansion, has been released, the brother has returned to playing.

Finally, a fourth possibility that occurred to me just now is that Aluminio may have made estimates of his /played time for each character instead of checking the /played time in game. I did provide instructions on how to obtain the /played time, but it is not outside the realms of possibility. Many people, myself included, are very poor at estimating time spent when it’s on and off again across multiple accounts, but we are far more accurate about how many total years (in calendar time) we have been playing because that can be easily tied to a fixed date.

When I designed the survey, I did not take into account that people might have been cheating, purchasing accounts, or account sharing.  Basing my estimate of their World of Warcraft experience on the /played time in such a circumstance is not very accurate.  The initial person who triggered this off is willing to be contacted if I have further questions and there is an e-mail address, so I think that would be a good thing to follow up on.

Related posts:

1. /played Time as a Measure of WoW Experience When working with people's experience in World of Warcraft, their /played time is a more accurate measure than calendar time....

Our research is based on participant- observation fieldwork. Each of us created at least two characters and joined at least one guild. We have jointly played for over 25 months and continue to play.

How much experience did they really have in World of Warcraft? Was the 25 months calendar time or in-game time? These are the questions that immediately went through my mind. I quickly concluded that it was not  25 months of in-game time as that would be more than 18,000 hours of play. Even among three people, that seemed unlikely even if they had been playing since the game was released. That led me to think about measuring game experience in immersive worlds, like World of Warcraft.

Typing “/played” in World of Warcraft will tell a player how many days, hours, and minutes they have spent online since creating that character.  This can be a more useful measure of a player’s experience with the game than elapsed calendar time.   For example, I have been playing since World of Warcraft’s public release date in February 2005.  My /played time is 268 days on my main character over a 69-month period.   Contrast that with someone else who, over that same period, only plays two hours a week.  Their /played time would be about 25 days (see Figure 2). I obviously have more experience in the game, even though our elapsed calendar time is identical.  There is an assumption there that I spent the time doing something in the game and not just chatting or idling, but it is going to be a more accurate measure of experience.

Credit: Michelle A. Hoyle under an Attribution ShareAlike 3.0 Unported License

Figure 1: Screenshot of questions in April survey

I asked respondents to report their /played time for three types of characters: their first character ever created, the character on which they currently spend most of their time, and the character on which they enjoy playing the most. If the characters were the same, they were asked to repeat the information. When I did my calculation, I ignored any entries that were obvious duplicates.

On the first page of my April 2010 survey, I had a set of questions asking people about the “/played” time for various types of characters they had played in the game (see Figure 1).  When I designed the survey, I wanted a feel for how much time people had spent on various characters.  I was not, unfortunately, thinking about their total WoW experience time.  Now that I am coding the data in NVivo, I am examining the data and combining the /played times they gave me (where it is obviously not for the same character) to get a rough idea of their minimum World of Warcraft experience.  In hindsight, I realize that I could have kept the original question, but I should have also asked people to calculate their /played time across all of their characters.   That would enable me to more accurately and easily compare what people have learned with the same amount of in-game play time rather than by using the far slipperier and less informative elapsed calendar time. Lessons learned and all that!

Related posts:

1. High /Played Times and Meaning Reasons why someone's /played time, used as a measure of experience in World of Warcraft, might be inaccurate....

Joy of Stats Book Cover

I have been working recently on importing data into SPSS from the first part of my April survey on World of Warcraft motivations. This has been a fairly straightforward process for the most part. The exception is the last question about the respondent’s guild type and purpose. The question was presented as the following:

In a short sentence (140 characters), describe the primary purpose of the guild in which you spend most of your time, or enter “no guild”.

Example: I’m in a social guild that believes in random acts of kindness. We love to dance but we also raid end-game content with other casual guilds.

I am not happy with how the coding for the type and purpose is going. When I created the SPSS codebook for that part of the survey, I initially broke it down into two parts: a type and a primary purpose. The type represented a breakdown primarily between “social” and “hardcore”. The purpose could be a key activity or a purpose. The divisions were chosen based on an an initial review of the received responses (see Table 1).

The “No guild” choice is self-evident. “Not codable” was intended for responses like “Forks” or “I am an herbalist in my guild”. This was usually where the respondent had not answered the question correctly, so no type or purpose could be determined. “No identifiable activity” was intended for responses like “Social”, where no obvious activity was ascertainable.

Andrew and I independently coded the 51 responses according to type and activity. We were allowed to use general knowledge about World of Warcraft but not specific knowledge about any guilds or people if identifiable somehow from the response. The response itself had to dictate the type and purpose.

We almost completely agreed on “type”, but we disagreed by about 28% on the “activity”. If an answer mentioned raiding and that seemed more key than anything else included (or nothing else was included), I coded it as “raiding”. Andrew often coded it as “fun” because he felt the raiding was deprecated or a lesser part even if it was the only thing mentioned. So, for example, if someone said something like “I’m in a social guild and we also do some raiding”, Andrew coded that as “fun”, because the “also raid” was a lesser thing, whereas I would have said “raiding”.

To fix that, he suggested that we add a category that indicated the amount of raiding. I would then have “raiding” and “some raiding”. That seemed sensible, but then left the question of what the “primary” purpose of the guild was if “some raiding” was ancillary. While the survey specifically asked for the primary purpose, but many of the responses, likely because of the poorly chosen example, included one or more activities. When coding, that meant the coder had to pick the one activity that seemed most prominent.

A colleague, in discussion about the first attempt at coding, suggested going for as wide a range of activities as possible before cutting back. I had, for example, initially omitted “performing random acts of kindness”, which occurred fairly often, as an activity. She thought that should be added as well as “social.” Social” as an activity was omitted because I was using the “social” designator appearing in responses to code for the guild type. However, in retrospect, it occurred to me it would be even more appropriate if I changed the guild type to be as interpreted by me based on my assessment of the respondent’s answer; the question did not ask the respondent to specify their guild type but to describe the purpose. I need to stay true to the question’s intent.

A modified codebook then would look something like Table 2.

That does mean that I need more formal definitions of “social”, “hardcore”, and “casual” then to use for the coding guild type. It also means there is a question about how many activities to code, given that very few responses only mentioned one purpose or activity. While the question did ask for a primary activity, I do not have a problem per se with coding for more activities. The questions are then: how many and should order be significant?

Open Issues for Comment and Resolution

• Is the second draft draft more sane in terms of how type and activity are determined in general?
• What are some existing formal definitions of guild types that can be used?
• How many purposes or activities should be coded?
• If multiple activities or purposes are coded, is the order significant? That is, should I assume that the first thing mentioned is the most important, the second, the next, and so on?

Any opinions or help are appreciated, as this type of analysis is new to me.

Related posts:

1. High /Played Times and Meaning Reasons why someone's /played time, used as a measure of experience in World of Warcraft, might be inaccurate....

As with all university-level research involving the participation of people, the WoW Learning project needed to secure approval from the relevant ethics committee in Informatics at the University of Sussex. In the interests of transparency and full disclosure, I’ve attached a copy of the approval to this posting, which you can download yourself.

The submitted proposal also had a project information sheet and an online survey consent form. I have not included those here.

If you have any comments or suggestions for improving the transparency or quality of interactions, I’d love to hear them. Virtual worlds ethnographic research and online data collection via forums is a relatively new area and the issue of what constitutes “informed consent” is still evolving. I’d also love to know if you found this a useful resource document for you own work.

Downloadable Resources

• WoW: Learning Ethics Approval (PDF – 312 kB)

No related posts.