Objective G/N/S and System Mechanic evaluation...

[deadpanbob]

All,

Ace posted an interesting question in this thread.

Marco made an interesting reply within that thread:

To put it another way: the theory that you might have a better experience with another system is valid--but there's no way for anyone but you to make that determination.

Valamir wrote the following:

It may tell me nothing.  When you run a linear regression you'll often have extreme outlying results far off of the line.  The only thing these results tell you is that such results are possible.  Your AC mod may simply be an outlyer.

Which in juxtaposition and context with each other, and combined with the morass of useless stuff in my head, lead me to a thought that will probably get me flamed off the board: we can measure G/N/S, it's realtionship to RPG system design, the subjective opions we all have thereof, and finally answer the question: Do people have good experiences with D&D (and its progeny) because of, in spite of, or unrelated to the system and its mechanics?

My background, and the job I currently hold, is in business and marketing analytics - which is a fancy way for saying I do a bunch of pretty hardcore statistical analysis of all of your (and I do me each of you individually) buying habits, attitudes, interests, surfing behavior, demographics, etc.  I then take this analysis, and turn it into strategies on how to successfully interrupt enough of you enough of the time in a profitible way for really big corporations to make even more money to pay their all ready bloated corporate management.

In any case, it is undeniably possible to more objectively measure the G/N/S theory, and whether or not system "x" helps or hinders the telling of a good story.

Yep.  Go back and read it again.  Now, before you flame me I'll point out that I said more objectively.  And, of course, there are some challenges.

The process is fairly well trod territory in both theory and practice, and it really does work.  I can provide several case studies from my own work experience in just the last few months alone.

Start with a hypothesis that you'd like to test - preferrably a qualitative/subjective/opinion based hypothesis.  Something like "Will adding these features (longer battery life, slimmer profile, more office software) to my companies' laptops make consumers more likely to buy them?"

Then, choose a target audience - continuing the example above "corporate purchasing managers for fortune 500 companies".  Then figure out a way to target the audience - in our example "find lists of business decision makers and their addresses from sources such as Dunn & Bradstreet"

While all that audience stuff is going on, a skilled and experienced marketing research professional (which I am not) - with a background in something like social psychology and organizational theory  - designs a survey.  The questions of this survey are designed to measure the purchasing manager's opinions of the new features relative to their overall willingness to buy and their overall rating of my companies' laptop computers.  Usually, but not always, the survey will include the same questions for my competitors laptop computers.

Once all the survey responses are in, it is possible through the application of quantitative satistical techniques to provide an effect measurement of and proof of the hypothesis.  For those of you out there keeping score, it would be more precise to say that we could effectively measure the variance of likelihood to buy based on the addition of the new features, rejecting or affirming the null hypothesis within the gold standard 95% confidence level.

This can be done by applying a specialized form of Principal Components Analysis and a Trans-Regression analysis to the data to create a Multi-Dimensional Preference Map (MDPREF).  The MDPREF will basically define a measurement space in which to apply the survey respondents' qualitative opinions in a very quantitative, measureable way.

Basically, we could provide a more objective measurement framework for the G/N/S theory and how it applies to RPGers actual perceptions and opinions about the games we play.  Once this framework was complete, the survey could be re-deployed about a different set of games to test the framework.  If that validation test works out well, then we'd end up with ultimately a realtively objective set of features that we could associate with games that tend to support any/all of the G/N/S modes.

I would be happy to volunteer to do the statistical analysis and the audience targeting.  I don't have enough experience in experimental marketing research test design to do the survey.  And I'd need some help turing the analytical results into a useable framework.

Any one interested in this?  Anybody out there have the requiste research background, or know someone who does who would donate their time to a worthy cause?

If this preceeding rant has bored everyone to tears, just ignore this post, and I'll never mention it again...

[Edited to add even more Jargon, to ensure that I can alienate as many people as possible]

Cheers,

Jason

[Andrew Martin]

That sounds awfully like a lot of busywork. Why not just simply feed back the results of play testing into your game system design, and apply good design? Your design will get better once you apply the feedback to it.

[ThreeGee]

I have to agree with Andrew. The only way you can get meaningful results is by doing the research very carefully. Just asking the people you know for their opinion is equally invalid whether or not you have pseudo-scientific questions. The data is only valid if the questions are both unbiased & unambiguous and if you ask a large number of randomly selected participants.

If you are wondering, I do social research, i.e. government studies, for a living. I have a great deal of experience in how not to do research.

/Grant

[deadpanbob]

I have to agree with Andrew. The only way you can get meaningful results is by doing the research very carefully. Just asking the people you know for their opinion is equally invalid whether or not you have pseudo-scientific questions.

ThreeGee,

Yes.  I guess I didn't make myself clear - I'm not talking about asking 'pseudo-scientific' questions.  Nothing about my post above was pseudo-scientific.  Its real world, practical research analysis - not made up.

Yes, careful question design is one of the hallmarks of good research, and in order to do this, the quesions would need to be carefully designed, and the hypotheses being tested would need to be well thought out, fully conceptualized, and as free from bias as they can be.

I wasn't talking about asking a few friends - I was talking about a randomly selected, hopefully statistically significant sample of RPGers.  I think I did make that point clear.

I'm not suggesting that this would be something I'd do to improve my own game designs per se.  The whole idea here is that in my short tenure at the Forge, a lot of posts have come up about the potential for applying the G/N/S theory in a non-subjective way.

My post is a challenge to all those who think it can't be done - because I think it can.

Sure, research like this doesn't prove causality - sure it's not 100% dyed-in-the-wool 'objective', but doing research like this would go a long way to prove out the concepts in the G/N/S theory.

It's not necessarily busy work either - I don't love what I do for a living (statistics and research analysis applied to business), but I am good at it.  I really love RPGs - playing and now designing them.  This would be a way for me to use my saleable skills to support my creative first love.

Certainly, its applicibility would be pretty narrow - but seriously, how broadly beyond the Forge do you think G/N/S is being used now?

How could extending the theory hurt?

Cheers,

Jason

[damion]

Jason,

Honestly, I love the idea.  Various comments though. My background would suggest that this is pretty legit though.

1)Your going through alot of work to basicly get an answer to a yes or no question.

1.1)What would you suggest as a good hypothosis to test?

2)The usual problem is that it's hard to come up with good questions. This depends on a hypothesis.


3)In this case, I think it would be hard to get people to answer the questions.
You might be able to get people on the forge to do it, but that's hardly a random sample. I think a lot of gamers in general would autmatically run from anything that mentioned GNS, or maliciously try to screw it up.

3.5)I think your best bet would be to try to get a RPG.net admin in on it and submit them as polls there. Or maybe just submit them as polls.

4)I think it would be useless to designers, as game designers don't really have the money or time to do this type of research. Heck many designers(not specifically directed at anyone) don't seem to have an understanding of basic probability, which I think should be required for designing a game that uses it alot.  They definatly wouldn't look at something like this.

Sorry for the negativity, I think it's a cool idea anyway though.

[deadpanbob]

1)Your going through alot of work to basicly get an answer to a yes or no question.

1.1)What would you suggest as a good hypothosis to test?

Well, in terms of hypothesis - I might want to start with the basics i.e. do the G/N/S distinctions in the theory actually hold any degree of truth based on the way people actually play the game.

G/N/S is a functional taxonomy that we can all use to make realtively subjective judgements about all things relating to gaming.  In that sense, the theory is a success, because it moves the discussion about RPGs into a less "You're wrong - now buzz off" type of exchange.

But I'm curious to see whether or not, for instance, (and to pick on Ron since its his essay) "Does Sorcerrer, as played, really tend to support Narrativist modes of play?"

That would be one very specific hypothesis that could be tested.

You are correct that no Indie game designer has the capital to pay for this research.  You are also correct that all mention of G/N/S would need to be scrubbed from the questions to make the results meaningful.

You are also correct that getting the help of an admin from a place like RPG.net would be critical in order to get enough of a less-skewed sort of random sample to base the analysis on.

Lots of challenges, but maybe worth it if the results of such analysis could be posted here at the Forge as a resource for game designers.

Probably just tilting at windmills/chasing paper dragons.  But interesting to think about and discuss.

Cheers,

Jason

[Marco]

"Does Sorcerrer, as played, really tend to support Narrativist modes of play?"

Jason

You'll need a heck of a control group--people who didn't buy Sorceror but are playing it anyway (because it's logical to assume that people who play Narrativistly would graviate towards Sorceror).

-Marco

[ThreeGee]

It would certainly be interesting to approach rpg theory in such a scientific manner, but we are talking about a lot of work. Designing a survey is no problem. Where I see problems is in getting enough people to participate.

A self-selective poll is no good. Even randomly picking people from rpg.net is skewed. Ideally, there would be an American Association of Roleplaying Gamers (AARG), or some such--an open organization that accepts even casual gamers of all persuasions. Then, you could just randomly select from the membership list. Unfortunately, by using rpg.net, you would first have to prove that their membership is representative of gamers everywhere; something I tend to doubt.

Once you have a sample pool, you have to convince the respondents to actually participate. Gamers are just as lazy as anyone else, so you either have to harrass them repeatedly to participate or call them up and convince them that now is the time to help. We are talking about a lot of man-hours for even a modest pool of sample.

Jason, do you know any other techniques for collecting data? I do not know what kind of work you actually do, besides crunching numbers. Since there are not millions of dollars riding on this project, there is no reason not to use cheaper research methods than those I am familiar with. For the sake of intellectual curiousity, I would love to see this actually happen.

/Grant

[deadpanbob]

A self-selective poll is no good. Even randomly picking people from rpg.net is skewed. Ideally, there would be an American Association of Roleplaying Gamers (AARG)

Or perhaps American Association of Ropleplaying Gamers & Hobbiests (ARRGH!) - sorry, I couldn't resist.

I also agree that picking from the RPG.net pool would limit the results of such analysis/research to "Among RPG.net gamers..." instead of "Among all RPGers everywhere..."

This would tend to cast doubt over the applicibility of any such research beyond some subset of RPGers as defined by computer ownership, access to the web, and knowledge of RPG.net.

How small a sub-set is that, and/or how could we reach a larger audience?

Well, to reach a larger audience on the cheap would be mighty tough.

I'm a consultant, BTW, and I know quite a bit about finding target audiences for marketing and/or research.  The methods I know how to use are probably cost prohibitive for a hobby based project: Working with list/data brokers (such as Acxiom, Dun & Bradstreet, Abbacus etal) to identify things such as magazine subscribers (Dragon Magazine) or people who've purchased RPGs (such as Borders or BN or Amazon house lists) and so on.

Looking for names like that would be costly.

Once the $$$ has been plunked down for the names, however, getting enough people to respond doesn't require repeated efforts.  Traditional survey response rates (whether conducted via phone or over the web via e-mail) are between 0.5% and 1.5% - but we'd only need to ensure approximately 500 responses, so we'd need a list of approximately 50K gamers to be on the safer side.  Its definitely doable, but very expensive.

So unless all the actual money making Indie designers that hang around here want to sink a chunk of change into this, we'd probably be best limiting it to RPG.net and potentially other RPG online communities.

The other (probably untenable) option would be for the Indie publishers here about to open their customer e-mail databases and let us solicit survey responses from those folks...

Cheers,

Jason

[contracycle]

Or perhaps American Association of Ropleplaying Gamers & Hobbiests (ARRGH!) - sorry, I couldn't resist.

Or daringly, just an Association of Roleplaying Gammers and Hobiests?

[damion]

Jason:
Good sample question. I guess the problem is that question is it gives you an
answer to "Does one narrativist game support narrativist play?" . Also you have to structure survey questions to figure out peoples modes of play without using words like 'narrativist, story-oriented, ect'. Kinda hard.
Not being familiar with the process so much, would it be possible to select multiple 'narrativist' games and see if they support narrativist play? This would point to a validation of the Design->Play linkage.  
If possibly that would allow more respondants(more games) and provide a more usefull answer.


Most surveying schemes have a few subltle selection criteria, this doesn't really make them anyless valid, if nothing else they all include the ' of people who respond to surveys'.

My point, I thnk online is really the only way to get a reasonably number of responses. One could also argue that questions of computer ownership and knowledge of RPG.net are orthogonal to alot of interesting questions.
Jason's sorcerer question is an example.