An Objective Standard of Quality for RPG's

[C. Edwards]

Hey Clinton,

One thing that I see missing - and one reason objective quality is hard to determine in RPGs - is a written statement about what an RPG is to be used for. While flavor text and background can help determine this, few RPGs come out and state the purpose of the RPG.

The way I see it, the purpose of an RPG is the behavior it promotes, which may end up being distinctly seperate from its stated purpose.  The problem, as I see it, with relating quality to how well a product matches its stated purpose is that honesty in advertising is almost non-existant.

Let's simplify the tool example by looking at the hammer.  If we could look at data that showed the behavior induced by the hammer on those totally unfamiliar with its use we may find that, yes, several people used the hammer to cut down trees.  I think the overwhelming majority of data would show that most used the hammer to bang, beat, and smash things.  Minus a nail or wooden peg, that is where the hammer excels in use.  The same idea applys to the saw.  While it may end up being used for a variety of purposes, like making music, the majority of data would show it being used to cut things.

If the data on a product didn't show an inclination to produce any one behavior in a consistent manner than it could be determined that while that product could be used to recreate a variety of behaviors  it doesn't specialize in inducing any particular behavior.  The data would, theoretically, also be able to show if say three seperate sets of behavior were promoted in a consistent manner.  Consistency would be a large determiner of relative quality.  If somebody was looking for a tool that was good at cutting things, the data might suggest a saw, a utility knife, and a hatchet.  Looking at what materials the tools were consistently applied to would then narrow down the options even more.

If we look at the man behind the curtain, all this comes down to is coherence.


Ron Edwards wrote:

System, system, system. Or more appropriately, design, design, design. The listed elements in Chapter One (character, situation, color, setting, system, initial premise) may be organized to facilitate greater coherence in Chapters Two (GNS, developed Premise) and Chapter Three (Stance), and thus to facilitate more enjoyable play. This principle is often summarized in the catch-phrase, "System does matter."

By "coherence," I mean the degree to which a group of people can hit upon and sustain a shared Premise (or topic for Exploration, in Simulationist play) - and by definition, continue to enjoy the social role-playing activity consistently. The people do not need to agree in every detail or event of play, and they certainly do not have to conform to a single, immutable Stance or GNS profile. However, to role-play together most successfully, their shared agreements do need to go beyond simply sharing the initial Premise. To whatever extent they do this, they are cohering.

The measure of quality I'm suggesting would relate to a games abilty to consistently promote a high degree of coherence based upon collected actual play data.  Then people could look solely at the games which "cohere" in the specific manner they desire, letting them apply their own subjective terms for quality.  This is already done in a word of mouth fashion.

If I were to pick up a very well made auger, for example, but had no clue what it was to be used for, I might determine that it's a terrible drill: it's much too large and unwieldy. Without the knowledge that it's to be used for making holes in the ground, I couldn't make an objective statement of quality about it.

Yes, but if you were looking for something that would make a hole in wood or metal and you had the data on different tools available, you would see that only 2 times out of 1000 has an auger been used to try and make a hole in wood or metal.  You could also see that a drill is used much more often for the specific need you are looking to fill.  It is of higher quality for your purposes.

What I see is people applying the component definition of quality which, if we are looking for the overall capability of an rpg to function in a certain manner is, I would say, the wrong definition to apply.

-Chris

[Marco]

If the data on a product didn't show an inclination to produce any one behavior in a consistent manner than it could be determined that while that product could be used to recreate a variety of behaviors it doesn't specialize in inducing any particular behavior. The data would, theoretically, also be able to show if say three seperate sets of behavior were promoted in a consistent manner. Consistency would be a large determiner of relative quality. If somebody was looking for a tool that was good at cutting things, the data might suggest a saw, a utility knife, and a hatchet. Looking at what materials the tools were consistently applied to would then narrow down the options even more.

I'm not sure how this applies to a quality rating. Is a game in which everyone plays gamist of higher quality than one where half the people display what you think of as gamist behavior and half display what you think of as Simulaitonist behavior?

What if I have a game that has a perfect spread (33% G, 33% S, 33% N) does that make it low quality or a "perfect" tool?

Finally, if almost everyone plays VtM in a Simulationist manner (I suspect this is the case), does that make it a high quality game? Ron has said that play of The Window will drift to either Sim or Nar (roughly 50%) does that make it low quality?

-Marco

[C. Edwards]

Quote:
If the data on a product didn't show an inclination to produce any one behavior in a consistent manner than it could be determined that while that product could be used to recreate a variety of behaviors it doesn't specialize in inducing any particular behavior. The data would, theoretically, also be able to show if say three seperate sets of behavior were promoted in a consistent manner. Consistency would be a large determiner of relative quality. If somebody was looking for a tool that was good at cutting things, the data might suggest a saw, a utility knife, and a hatchet. Looking at what materials the tools were consistently applied to would then narrow down the options even more.



I'm not sure how this applies to a quality rating. Is a game in which everyone plays gamist of higher quality than one where half the people display what you think of as gamist behavior and half display what you think of as Simulaitonist behavior?

What if I have a game that has a perfect spread (33% G, 33% S, 33% N) does that make it low quality or a "perfect" tool?

Finally, if almost everyone plays VtM in a Simulationist manner (I suspect this is the case), does that make it a high quality game? Ron has said that play of The Window will drift to either Sim or Nar (roughly 50%) does that make it low quality?

-Marco

This is, I think, the heart of the misunderstanding.

quality:  The totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs.

Quality of a product cannot be seperated from the needs of the user.

If I'm looking for a game that promotes sim play than V:tM, or Rolemaster, or whatever, would be of higher quality for my purposes.  The relative level of quality between them could then be taken into account.  Which game promotes sim play more consistently?  For this example, that game would be of the highest quality when it came to fulfilling the specific need.  If I wanted a game that could drift between sim and nar than perhaps the Window is of higher quality for my specific needs.

Context is everything.

-Chris

[deadpanbob]

So maybe we need to develop three measures of quality - at least - one each for G/N/S?

I envision something like movie ratings on the backs of RPGs everywhere - telling you the discerning game buyer which mode of G/N/S play this game supports most fully.

If only we could set the Forge up as the independent rating agency for RPGs, we'd be set.  ;-)

I still think, for what it's worth, that measuring quality requires first and foremost a determination of what's being measured.  A metric must be agreed upon by all parties in the rating process.

Sure, the sum of the parts is often more than the whole - but in my experience you can't rate the whole unless you understand and rate the parts.

Put another way, to say that a game tends to result mostly in instances of play that appear (upon observation of the behavior of those playing) to be Gamist modes of play, is a quality game may ignore the fact that this is true overall, but not on some specifics.  Say that this Gamist mode encouraging RPG has a very detailed skill system modeled after hard core developmental psychology, and the game itself is supposed to be say a real world Spy game.

However, the game has a resource mechanic and/or a currency mechanic that encourages Gamist play - to a degree that overwhelms the highly Simulationist supporting skill system.  Ovserving only the results in play, one could come to the conclusion that the game is a quality game because it seems to pretty coherently result in instances of Gamist mode play.  But without examining the fine details, we might miss the fact that there are parts of the game that don't coherently support that type of play at all.

Cheers,

Jason

[Valamir]

Perhaps the best place to start, is to replace notions of "quality" with all of the extraneous baggage that word entails with the notion of "utility".

After all, GNS was not designed to evaluate the "quality" of a game design.

Rather it is an effective tool for matching player decision type with a game that promotes that decision type.  In other words, evaluating the utility of a particular design for a particular player.

As far as what is better a 100% S game or a 33/33/33 game (not that such ratings are very useful except as a vague illustration)...that depends on the play group.

If I'm looking to play sim, and my 5 other players are looking to play sim, than 100% sim is clearly the better (read: more useful) game for us.

If instead my group of 6 consists of 2 of each style player, than there is really only 2 choices.

1) The group is dysfunctional and struggles to satisfy anyone.  Best bet is to either learn to adapt to each other, or find a new group.

2) The group works well together and we all learn to appreciate each others preferences and blah blah blah...in which case the 33/33/33 version MAY be a perfectly useable option.  GNS never says that this option is "bad" it merely says that this option is more likely to produce dysfunctional play.  Not that it MUST produce dysfunctional play.

Ergo:  we should be talking about objective measures of utility, not of quality.  I think we pretty much have been, but the "quality" label can make this get very shaky very quick.

[deadpanbob]

Ergo:  we should be talking about objective measures of utility, not of quality.  I think we pretty much have been, but the "quality" label can make this get very shaky very quick.

Yes!  That it for me.  I like this idea best of all.  Changing the word from quality to utility is a really good move.  When I changed this word internally, it did drop out all of my mostly work-related issues of quality.

So, Valamir, how would we start the process of building a utility rating?

Would the best place to start be a re-hash of those behaviors in play that tend to support a given mode of play for a given instance of play?

Cheers,

Jason.

[Valamir]

Hmmm, don't know Jason.  That would be a pretty ambitious undertaking.

I think it might be possible to come up with some conceptual measures.  The sort of thing like the formulas they teach you in Econ 101.  Formula which are so simplified as to be completely useless applied as actual foruma to the real world...but which serve as a tool teaching how certain factors relate to each other.

In other words a mathematical way of teach a certain way of thinking, so that even though the math is purely theoretical, the way of thinking it teaches has practical application.

But to actually create something that could be used to measure real utility in the real world with real gamers...You'd probably win some kind of nobel prize if you figure that out.  Utilities been used as a concept in Econ for decades, but I don't think its ever been successfully applied in practice.

I do remember reading some folks who actually came up with an actual formula for measuring personal utility (of questionable usefulness).  You could probably do some searches on the topic to find some previous work on the subject to start from.

[C. Edwards]

utility:  the quality or state of being useful.

By rating something by utility you are automatically implying how useful it is.  If that's not implying a subjective determination I don't know what is.

The only issue with the term "quality" is that people seem to insist on applying the component level definition when it doesn't apply.

quality: degree of excellence; of superior grade

vs.

quality: the totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs.

which is related to

quality: an essential and distinguishing attribute of something.

Utility would not enter the equation until a person actually looked at the quality rating and made their own value judgment as to its usefulness.

I hesitate to use GNS terms as an example, but here goes.  A quality rating might look like this:

Game A:  Sim 10

Game B:  Sim 5/Nar 5

Game C:  Gam 3/Nar 7

This would be based on observed behavior in play which was induced by the game and/or the exhibited behavior of players that have a tendancy to play the game. I think both are equally useful for this purpose.

I think some honest description of actual play based on observed behavior would need to accompany this, even if it is just actual excerpts of real play.  Descriptions aimed solely towards marketing like that on the back of GURPS would certainly not work.

I know all this repeating of definitions is somewhat obnoxious but I find changing to a less appropriate term because the appropriate definition of another term wasn't being utilized to be rather illogical and perplexing.

Damn, I sound like a Vulcan.

-Chris

[M. J. Young]

Which is the better quality tool, the Stanley Claw Hammer or the Craftsman Rip Saw?

Which of them is the more useful tool?

I have been working for several years, on and off, on an approach to doing game reviews. It struck me then, and still seems right now, that the starting point for any discussion of any game or game supplement is this: what is it attempting to accomplish? You must first ask and answer that about your game or product before you can ask any other question, because there are really only two other questions you can ask: how well did it succeed? And was this a worthwhile objective?

Thus if you want to know whether a  game is "good" you have to ask these questions. It is entirely inappropriate to judge Alyria based on how well it simulates reality, because it is making no effort at all to do that. It is far more appropriate to ask whether it creates strong character conflicts and leads to their resolution, because that's what it's trying to do. It might also be appropriate to ask whether designing a game that does this is a good thing (I think it is).

This is why objective standards are so difficult to achieve. They assume (as someone implied) that all role playing games are trying to do the same thing. This is as wrong-headed as the idea that all games are trying to do the same thing. A game is better if it makes clear what it is attempting to do; it is better if it does what it intends to do well; a game is better if it intends to do something worth doing, even if it is not entirely successful; it is best if it does something well that is worth doing.

This analysis doesn't even have to be limited to role playing games.

Now, tools such as GNS and consideration of stance and coherence may all be useful approaches to understanding why a game does or does not succeed in reaching its objectives; but the first step is still an evaluation of what those objectives appear to be. Whether this comes from color text, or setting, or author statements, or an analysis of the apparent function of the mechanics, it is the starting point for any other analysis.

--M. J. Young

[contracycle]

but the first step is still an evaluation of what those objectives appear to be.

???

Surely, objective 1: to develop a mini-market foir supplemental products
Objective 2: to provide entertainment to its audience, in pursuit of goal 1

If we consider a game as a product, thats all there is.
If we consider it as a work of art, then "quality" remains subjective and that is perfectly valid.  The only true statement we can make is "I liked it".

OTOH: "Which is the better quality tool, the Stanley Claw Hammer or the Craftsman Rip Saw? " is in fact answerable.  The one that is better made, shows higher attention to detail, uses more durable material in its construction, could reasonably be said to be of higher quality.  This does not make it any more useful than before, however, and so we may find ourselves abandoning the high-quality tool and reluctantly using the low quality, but appropriate, tool.  This does not alter the fact that there was a qualitative distinction between them.

[brainwipe]

From someone who is trying to put a 'quality' game together, this topic is of near-infinite usefulness and purpose. I am thoroughly enjoying reading the comments posed. I am yet to form an opnion, though.

[Jeremy Cole]

In other words a mathematical way of teach a certain way of thinking, so that even though the math is purely theoretical, the way of thinking it teaches has practical application.

While not math based, I think GNS is that.  Mind you, economics  is hardly maths based, 1st year basically, before you get into the quanititative crap.  

Anyhow, when I first joined this sight, I suggested a system should be looked for its ability to support each mode of play, as player's will often drift mid-campaign.  I think the extension here is for the review to encompass not what the game is trying to do, but look at what player's might do with the game, and how well they might do it.

If the mechanics support cheese fantasy, how well?
Grim fantasy, how well?
Empire building, how well?

More coming...

[Jeremy Cole]

In other words a mathematical way of teach a certain way of thinking, so that even though the math is purely theoretical, the way of thinking it teaches has practical application.

While not math based, I think GNS is that.  Mind you, economics  is hardly maths based, 1st year basically, before you get into the quanititative crap.  

Anyhow, when I first joined this sight, I suggested a system should be looked for its ability to support each mode of play, as player's will often drift mid-campaign.  I think the extension here is for the review to encompass not what the game is trying to do, but look at what player's might do with the game, and how well they might do it.

If the mechanics support cheese fantasy, how well?
Grim fantasy, how well?
Empire building, how well?

More coming...

[Jeremy Cole]

Vampire is a good example of this.  It could be reviewed on;
Powergaming
Tactical combat
Empire building
Story driven angst piece
etc...

The merits of the system could then be looked at with each in mind.  Perhaps they don't have to be distinctly put, otherwise everything might fall into the definitionalism GNS talks normally fall into, but the review should consider; 'what campaigns does this game cover, and to what effect?'

Jeremy

[Marco]

There's a lot here--but I want to respond to three things that caught my attention:

1. Telling a person how well a game supports a specific GNS mode is probably the least important factor in telling them how well they'll enjoy it (the exception--and maybe that's why it's getting play on The Forge is the hard-core Narrativist who has little use for traditional systems).

Telling me that something is good for Sim-play means nothing in and of itself.

2. Evaluating games as works of art (Contracycle) is valid--but subjective without points of reference (Critics and Cannon).  Determining the game's usefulness for "implied" objectives is problematic since it appears to be objective but really isn't (what's the implied objective of Deadlands? Why is that relatively crunchy combat in VtM? Without a riggorus statement from the game designer(s) it's not really possible to say).

3. What I'd like to see is how Contracyle's objective comparison (better materials, more durable, etc.) applies to an RPG experience (not the physical book or PDF or whatever).

What are the elements of the RPG experience that could be used to make a qualitative distinction between one and the other?

-Marco