Toward a New Creative Agendum Evaluation

[M. J. Young]

Some years back, I devised a GNS/DFK quiz. There was a lot of debate about it, and although it is still available online (mainly, as with Ron's articles, for historic reasons), I recognize that it is mostly an effort to get at agendum through techniques, and it doesn't work terribly well. Not so long ago I wondered on one of these threads whether there is a way to get at agendum through a quiz or survey.

Tonight I was reading talk of surveys on some thread on Theory plus comments about data here on the GNS forum, and something started to coalesce in my mind which I think might be a basis for a new quiz format that really could reveal an individual's creative agendum.

The quiz would give descriptive text of accounts of game play--not mechanics, but the sort of stories people tell about their games after the game is over. Carefully crafted, some of these would be about how well someone did against the odds, some about the strange and wonderful things that happened, and some about the moral and personal crises of the characters. The trick would be to phrase them such that we can see what it was about play that really psyched the person telling the story.

The questions would relate to the stories. People would be asked questions like:

[*]How likely is it that the events recounted in that story would have occurred in your game?[*]How likely is it that a story such as this one would be told by you about one of your game sessions?[*]How likely is it that other players in your regular gaming group would tell such stories about your game sessions?[*]Is this the kind of story you would like to have come out of your game sessions?[*]Is this the kind of story others in your gaming group would like to have come out of those sessions?[/list:u]There would be a rating scale, probably seven points, such that one end would be things you see or want all the time, the middle would be indifference, and the far end would be something you never see or want not to have in your game.

My theory is that people who are gamist themselves relate better to stories about gamist play, and so too for narrativists and simulationists. The quiz ought to be able to tell what kind of play the group is actually tending toward, what the group tends to prefer, and what the individual taking the quiz tends to prefer.

Does this sound feasible? Claire "BeingFrank" Bickell--you indicated an interest in working on a survey of some sort. Does this sound like it could work? Anyone else have thoughts on this?

--M. J. Young

[Ron Edwards]

Hi M.J.,

This may sound odd from the guy who lost no opportunity to rail against your quiz in the past, but I say, there's nothing wrong with giving it a try.

Best,
Ron

[Marco]

I too think that it's worth a try--one thing that would be good would be collecting demographics about RPG's played, ages, years played, length of sessions, number of players too. Even if the agendum/technique data was iffy the hard facts could be very useful.

-Marco

[Jasper]

Maybe you should consider doing some confirming runs, whereby people from around here who are familiar with the Agendas and who can have a good handle on their own play can take the survey, thus establishing that the questions as you've crafted them do indeed test what you think they do.  You could start out with a whole heap of questions, and weed out the less revealing ones.  But maybe it's too early to be talking of gritty details...

[Wormwood]

M.J.,

I know I'm very interested in player retention of the game. I think that might work well as an indicator. Players with gamist leanings should retain game like things better than players with other leanings. This could probably be done by a question where a description of a game is given, and then made inaccesible for the answers, perhaps spaced between different questions. Of course the best test would be to do retention for actual games...

 -Mendel S.

[ffilz]

I'd certainly love to see an effective instrument. One thing to be careful with for confirming runs is the test subjects trying to divine the correct answer for what they think their CA is rather than being honest with the questions as presented (which could either be evaluated wrong, or the subject's self assesement of their CA might be wrong).

So what would be better for confirming runs would be for us to hand the instrument to folks we think we have a handle on their CA and let them take the test.

Frank

[beingfrank]

Sorry to take so long to get to this, life got exciting unexpectedly.

I think it's a brilliant idea, and I'd love to work on it with you and anyone else who's interested.

One thing I think we should consider, if we're looking at a survey that can be used by roleplayers with no experience of the concept of creative agenda, is some sort of forced ranking of what people want most.  I've done spot quizes with the people I play with about what they need in their gaming to have fun, and they tend to put down things that cover all three CA.  That makes sense, because I think many people like some of everything at sometime.  But then I get them to try and drop something off the list (I think we had 7 in the end) to get some sort of idea of what was most essential for them, and highest priority, and you'd think I'd asked them to cut off a limb.  I had one player insisting that they needed all 7 of the things, all the time, even though some of them were mutually exclusive in my view.

I think that roleplayers in general are likely to be very good at identifying things that they like, or think would be really neat in their game, or think that they should like, but not so good at analysing the relative importance of those things to them.  But maybe that's a different problem that should be addressed seperately in the survey?

I think doing a pilot is essential and we should do that with people who were not involved in the development.  And if possible, with people frequently exposed to the Forge, and those with little or none (which might avoid the problem of people picking answers to produce the CA they want, but will still be subject to people picking answers according to faulty self asessments, but I don't think the later can be avoided completely).

I also like Mendel's idea of player retention.  And it could perhaps work with some sort of measure of interpretation of player enjoyment.  Like the subject reads the gaming story, answers a bunch of questions on something else as a distraction, and then is asked what the person in the story enjoyed most.  Ideally, we'd like to ask them what they remember of the story, but that would be an open ended question and the answer would have to be individually coded.  That would be time and effort intensive, but we might be able to get a similar effect by presenting a range of possible sources of enjoyment for the player in the gaming story and getting the subject to say which they think was the case.

Ok, that's my first thoughts.

I think the next step, and probably the trickiest one is to start work on the gaming stories to be used as stimulus in the survey.  We can make these up ourselves (though I bags out of that part, as I think my gaming experience is too atypical for me to have any idea where to start).  Or we could call for actual gaming stories, try and find ones to use, or use them as a basis for our gaming stories, which would them be adaptions of real gaming stories, rather than complete fabrications.

Forums, mailing lists, and blog memes should make it fairly easy to get a large sample of actual gaming stories, but analysing them is going to be a bugger.

Also, as Marco says, the basic demographic data could, in itself, be very interesting.  What should go in there?  What basic questions do we want to ask people to get an idea of their experiences of roleplaying?

games played
age first started
years playing
age
gender
typical length of sessions in a campaign
method (ongoing ftf, pbem, convention oneshots, etc)
typical number of players
typical length of sessions[/list:u]

What else?

[clehrich]

In line with Claire, I think it should all be formulated positively.  Players should be asked, in essence, "Is this what makes you go 'boink'?"  But as Claire says, there has to be some sort of ranking system, or else everything is going to come up "Yeah, that's totally essential."

Anybody here know much about statistical sampling?  We don't want to push-poll here, nor do we want to generate a list of meaningless statistics about the kinds of things people would really like to believe they're playing.  Rather, we'd want to find out what people really are doing, and would actually like to do, and what moments in a game really zing them.  My sense is that M.J. would need advice from someone who's actually studied how you do this -- which ain't me.

[beingfrank]

Anybody here know much about statistical sampling?  We don't want to push-poll here, nor do we want to generate a list of meaningless statistics about the kinds of things people would really like to believe they're playing.  Rather, we'd want to find out what people really are doing, and would actually like to do, and what moments in a game really zing them.  My sense is that M.J. would need advice from someone who's actually studied how you do this -- which ain't me.

I have studied how you do this, but it was a while ago.  I want to dig out an old text book and refresh my memory about how to write surveys so that it's hard for people to cherry pick the answer they think they should get, and other things like that, but I think it's in a box at my folks' place in the country so I won't get my hands on it until next week.

In terms of sampling, accessing roleplayers who are not online is going to be impossible, but that's a limitation I think we can live with.

In terms of how we analyse this thing statistically, that is going to extremely important.  I don't think we can do anything too complicated at this stage because we just don't have a theoretical model that we can express in terms of what we can measure.  I think we'll end up doing a factor analysis of some sort, looking if there are 3 underlying factors that determine the patterns of people's answers, and if they correspond in any way to CAs.  Or possibly just Chi-Squared tests.

However, we should also be able to test other things.  Say, if anyone thinks that men are more likely to answer a certain way compared to women, or differences by age, or sorts of games played.  That stuff is reasonably straightforward, but we do want to think about it beforehand.

I don't want to get into the statistics at this stage, but basically, the more comparisons you do, the higher you have to set the bar above which you say 'yes, the difference between these groups is sufficient that they're unlikely to come from the same group, but are acutally different groups' in order to reduce the overall rate of false positives.  And the higher you set that bar, the more likely you are to have false negatives.  However, the tricky bit is that if you set the number of comparisions before you start analysis, you don't have to raise the bar as high as if you decide to do extra comparisons after looking at the data.  I can explain why if anyone's interested, but we probably don't need to get into that right now.  What it does mean is that if anyone has some pet private theory about how left-handed Russian Mage players are more likely to prefer Gamist play than the rest of the world, you should speak up now so we can plan ahead and decide if we do want to test for that.  If you do it after you've already looked at the data, it's both less elegant, and kind of shooting yourself in the foot because you have to set yourself a higher bar to reach.

[M. J. Young]

An odd demographic question that we might consider is a selection of RPG sites with which people might be involved. The list should include at least:

[*]RPGnet[*]Gaming Outpost[*]The Forge[*]RoleplayingTips.com[*]WotC[*]Pyramid Online[*]ENWorld[/list:u]There are probably a few others that should be included, but those are the ones I know in which gamers are actively involved, and it may go some distance to answer some of the myths about what kind of play different sites favor.

And Claire, I can't imagine doing much more than thinking about this before Christmas, so don't knock yourself out trying to get that book.

--M. J. Young

[jdagna]

It sounds to me like the biggest issue you may run into will be people remembering key moments from lots of different games and groups.  For example, I have cherished "hall of fame" moments that represent all three CAs.  The only common elements in all of these moments are that the group 1) became emotionally involved in the situation and 2) unified around the CA for the example.

You might still be able to filter out a preference through that muddle, but you'll have a lot of noise in there.

Perhaps you should require people to limit their answers to a single campaign and/or group and encourage them to take the survey for different ones?  It would probably provide more useful results about what mode they were using for that campaign/group, even if it would be less useful in determining an overall preference.

[ffilz]

If people were encouraged to evaluate each campaign separately, that could be a useful tool towards determining their CA preferences. If they find one campaign was Sim and another Nar, and they look back on the Nar campaign with greater fondness, then perhaps they should look for other Nar campaigns (of course they have to evaluate whether the CA differences led to their greater fondness or some other factor).

Frank