An Objective Standard of Quality for RPG's

[deadpanbob]

What are the elements of the RPG experience that could be used to make a qualitative distinction between one and the other?

These elements will of course be subjective.  But some likely candidates, depending on who you talk to:

A mechanics system that gets out of the way of my immersive roleplaying experience

or

A mechanics system that has lots and lots of detail and options for conflict resolution (combat or not)

or

A mechanics system that is self-consistent and provides consistent results based on character concept

or

A game that encourages and rewards player character cooperation

or

A game that encourages and rewards player character competition

or

A game that encourages and rewards the creation of good stories

All of the above are subjective value statements - and you could probably get a group of people to agree/disagree along some scale with all of them.  Then, the problem becomes finding games that support those value statements - because the whole point for me is how do I as a game designer effectively design a game for my target audience (assuming that my target audience should be bigger than just me)?

Cheers,

Jason

[Valamir]

1. Telling a person how well a game supports a specific GNS mode is probably the least important factor in telling them how well they'll enjoy it (the exception--and maybe that's why it's getting play on The Forge is the hard-core Narrativist who has little use for traditional systems).

Don't get hung up on this "telling a person" stuff.  The objective of GNS is not to go out and proselytize to the masses.  GNS is not a dating service where you come to the Forge to be evaluated and hooked up with with the best match from our catalog.

Sans the "telling" part, I couldn't disagree more.  How well a game supports the mode that the player prefers to play in is one of the most important factors in how well they'll enjoy the game.  This is especially true if you factor out (i.e. control for the variable of) the social issues of enjoying the company of the people you're with...which should be factored out of an analysis because they're true of all human activity, whether its gaming, or going to the movies.  

Telling me that something is good for Sim-play means nothing in and of itself.

If you are looking for a game that will allow you to realistically modify ballistic combat, how does knowing whether a particular game does or doesn't do that well mean nothing to you.  Modeling reality well, being a sub-set of Simulationism.

How does knowing that Inspectres doesn't model "reality" in detail mean nothing to you if what you're looking for is an actual simulation of paranormal investigation.

I can't understand your point here.

[Marco]

1. Telling a person how well a game supports a specific GNS mode is probably the least important factor in telling them how well they'll enjoy it (the exception--and maybe that's why it's getting play on The Forge is the hard-core Narrativist who has little use for traditional systems).

Don't get hung up on this "telling a person" stuff.  The objective of GNS is not to go out and proselytize to the masses.  GNS is not a dating service where you come to the Forge to be evaluated and hooked up with with the best match from our catalog.

This is in the context of a review or a quality assessment. Not in the context of advice. If I read a review and it says "Sim 99%" that purportedly tells me that in studies the players made Sim-type decisions. You suggest this will inform (tell) me how well it'll suit my needs or how much I'll enjoy it.

Sans the "telling" part, I couldn't disagree more.  How well a game supports the mode that the player prefers to play in is one of the most important factors in how well they'll enjoy the game.  This is especially true if you factor out (i.e. control for the variable of) the social issues of enjoying the company of the people you're with...which should be factored out of an analysis because they're true of all human activity, whether its gaming, or going to the movies.  

This is, I think, clearly untrue. That is, it's a falacy: Just because the games I enjoy fall into the super-set of sim-play does not mean that I enjoy any game that falls into the super-set of sim play.

That's where I think the GNS stuff is being mis-applied. Call of Cthulhu, a Sim game, doesn't model ballistics particularly realistically--that has nothing to do with my enjoyment of it. It doesn't simulate "realisitc" paranormal investigation. It doesn't do anything like that--VtM is a sim game with what, emphasis on color or something ... I found the aggravated damage rules (part of the simulation) fatal to my enjoyment of it (rendered in GURPS, that is, but I found the char-gen--another part of the simulation--flawed in Story Teller).

Morrow Project has *great* ballistic rules, it's a really good simulation of reality (mutations: "leukemia")--but it wasn't even my top choice for post-app play once Hero got going. Both are sim systems. Both would score highly.

The difference is far more profound than GNS mode.

In short, no--how well a game supports my "mode" is probably one of the least important aspects as to how well I'd enjoy it.

I can't understand your point here.

Maybe that's because you're ascribing too much importance to GNS.

-Marco

[Jeremy Cole]

All of the above are subjective value statements - and you could probably get a group of people to agree/disagree along some scale with all of them.

Why try for an absolute statement?  Assess what each design is good for.  'This game effectively supports team oriented play, and its combat system is designed for heavy strategy play, and models in great detail a possible 34th century style of warfare' or whatever.  Then assess against those objectives.

Come to think of it?  Why define the elements, and why quantify things?  There will always be common use definitions, and game plays forms, but always leave it open to say - this attempts something done nowhere else.

Rather, form a procedure for reviewing.  How can this game be played?  How well does it facilitate such play?  How does it compare to games that attempt the same things (if there are any)?

In this way, a game is assessed in a qualitative fashion, and relayed as such to other persons, they gain a real understanding of the game.

Jeremy

[deadpanbob]

Rather, form a procedure for reviewing.  How can this game be played?  How well does it facilitate such play?  How does it compare to games that attempt the same things (if there are any)?

In this way, a game is assessed in a qualitative fashion, and relayed as such to other persons, they gain a real understanding of the game.

Jeremy,

This only helps me if I've played the other games that the one in question is compared to.

The reason to think about 'quality' or 'utility' etc. at all, for me, is because as a potential game designer, I'd like to be able to more realistically, and to borrow Ron's term, rigorously design a game that will satisfy my target audience.

Cheers,

Jason

[Valamir]

Maybe that's because you're ascribing too much importance to GNS.

-Marco

Actually, its quite the opposite.  I think you are ascribing too much importance to it.  In the sense that you are expecting something as simple as identifying whether or not a game is Simulationist to answer all of your questions about what the game is like and whether you will enjoy it.  

Since it can't do that, you're inclined to dismiss the theory as some nice talking points, but not real helpful.  But I think you're expecting far too much from what is essentially a very simple concept.


When I said it was one of the most important items that does not mean, and should not be taken to mean.  "Marco likes Sims.  Game X is a Sim.  Therefor Marco will like Game X"  Nope...

What it means is "Marco is looking for a Sim Game to play ABC with.  Game X is a Narrativist game about ABC.   Marco has been playing ABC with Game X and is pretty disatisfied with it.  Perhaps this means that Marco would enjoy himself more if he found a different game on ABC which better supports his Simulationist goals.  Quite likely Marco has already begun thinking of houserules and variations that will begin to move the game in that direction already."

Further what it means is "Ralph is looking to design a game about ABC, and hopes to design something that will appeal to Marco and gamers like Marco.  Ralph knows that what Marco et.al.  is really looking for in a game on ABC is something that emphasises various simulationist aspects of ABC like 1, 2, and 3.  Armed with this knowledge (or more generally, a generic target marget), Ralph can take the principles that have been built up on a foundation of GNS use them to help determine what sorts of mechanics are most and least appropriate for the type of game Ralph is seeking to design".

Or it means "Marco just played Game X.  Marco did not enjoy it at all.  It was one of the worst experiences of Marco's gaming career.  Marco was planning on writing a review for RPG.net on Game X, but unlike a large number of RPG.net reviewers, Marco is not satisfied with simply writing "Game X sucks and is a total piece of crap" and calling it a capsule review.  Marco knows that the reason he didn't enjoy the game has little to do with the quality of the game, and everything to do with it not being his cup of tee.  Armed with GNS theory, Marco is able to identify the sort of game, that Game X is and evaluate how well Game X fulfills its goals to be that sort of game.  The lexicon of GNS comes in handy as Marco prepares a review evaluating Game X on its own merits, even though he didn't particularly enjoy it.".

"Ralph recognizes that some games and some game players are like oil and water and just don't mix.  Ralph knows its tradition for one group to label the other group as munchkins, or bad roleplayers, or story-nazis and so forth.  But Ralph knows players from different groups and doesn't think this is the reason.  Ralph finds GNS.  GNS helps identify why different players can't stand to be in the same game together even though they both love RPGs.  GNS helps illustrate why these different preferences for play are all legitimate gaming practices and not examples of "bad roleplaying".  Armed with this knowledge, Ralph can even enjoy playing a game of D&D with a group he previously couldn't stand playing with, because he's learned to appreciate games for what they do and what they are.  Further, the rigorous thought that has gone into trying to understand GNS and its corollary theories has opened up a whole new world of roleplaying possibilities.  Not necessarily "thinking outside the box" but clearly illustrating that the box is a lot bigger than he thought it was.  And so Ralph's roleplaying and game experiences, and game design efforts have been improved and made better by discussing GNS...despite it not being perfect.  Even though there are still large chunks of it Ralph might disagree with, its served its purpose, and this is a "good thing".

Sorry, for coopting you into my stories, but these are the areas that GNS is for.  And areas that it performs well in.  Trying to apply the theory beyond these areas is certainly not forbidden...but it does void the terms of the warranty...if you follow me.

[Jeremy Cole]

Hi Jason,

This only helps me if I've played the other games that the one in question is compared to.

Because if you talk about what a design can be used to do, then you get the various elements of utility.  It doesn't matter if you have played something or not.  If you read a statement about a couple of fantasy games, 'Y has combat with more tactical options, X has more realistic combat', you don't have to have played each game to know a bit more about each, and you can assess the games lot better than an overall apples and oranges rating.

The reason to think about 'quality' or 'utility' etc. at all, for me, is because as a potential game designer, I'd like to be able to more realistically, and to borrow Ron's term, rigorously design a game that will satisfy my target audience.

The working definition of utility I was taught in Economics was the ability of a good to meet the multiple needs of an individual (yes, one individual wants more than one thing out of an item).

Attempt to find the multiple needs of your user group.  Then objectively state what you want, and compare it to other games with similar design elements (for comparison and theft).

Jeremy

[contracycle]

3. What I'd like to see is how Contracyle's objective comparison (better materials, more durable, etc.) applies to an RPG experience (not the physical book or PDF or whatever).

Well we are talking about RPG design, so we are necessarily discussing something that is an object to some degree.  The only practical reccomendation I could make in regards system would be a reference to another such object.

[M. J. Young]

but the first step is still an evaluation of what those objectives appear to be.

???

Surely, objective 1: to develop a mini-market foir supplemental products
Objective 2: to provide entertainment to its audience, in pursuit of goal 1

If we consider a game as a product, thats all there is.
If we consider it as a work of art, then "quality" remains subjective and that is perfectly valid.  The only true statement we can make is "I liked it".

If I read the first page of the Introduction chapter in Multiverser, I quickly learn that the game sits on these principles:
--all imaginable universes exist
--player character travel from universe to universe
--rules change to some degree to match the universes.
It doesn't take too much intelligence to conclude from this that the objective of the rules is to provide a game system that will support any imaginable type of adventure in any imaginable type of world, and move player characters from one such world to another in the course of play. This being the apparent objective of the game, the next question would be whether it accomplishes this.

That was too easy; I wrote that. Let me pick on Little Fears. I happen to have a copy right here in my room, and I happen never to have seen anything more of the interior than the character sheet. My second son was going to run the game, but my wife nixed it because she thought it too frightening for the younger members of the household gaming group. So I turn to the introduction, and I find a somewhat impressionistic bit of writing; yet I cannot avoid coming away from this with the idea that Jason Blair wants me through his game to remember what it was like to be a frightened child. Maybe there's more that I would discover, but this seems to be the objective. Not having played or read it, I can't tell you from my experience whether it accomplishes this--but that would be the next question. I might also ask whether such a game needed to be written; but that's a value judgment that tells more about me than about Jason's game in this case. It would be different, perhaps, if there were hundreds of games in which we played frightened children; in that case, we might argue that as good as this one was, it didn't add anything to the corpus. That is not the case.

I've also got a copy of Pendragon (4th ed.) here, which I started reading some time ago and set aside only to forget. But as I re-read the text on page 4, Welcome to Pendragon, I find the idea clearly presented that the game hopes to recreate adventures akin to the literature surrounding Camelot and King Arthur--not by any means the reality of that age, but the age as recreated centuries later. How well it does that is the question to ask. It would not be appropriate to ask, for an extreme example, whether the game engine could be successfully used to model space pirates in a world similar to that of Star Wars. The intent of the product is fairly clear.

Dungeons & Dragons third edition Players Handbook is a bit less clear in its introductory text as to what it is trying to achieve. The most I can conclude from the opening section is that it wants to make possible a variety of adventures in a somewhat non-specific fantasy world. Perhaps I could find the objective by doing a bit more reading, but not tonight. And perhaps this is an example of a game whose objectives are not very clear, either in the way they are stated or in the means by which they attempt to achieve them. That doesn't alter the reviewer's obligation first to attempt to understand what the writers wanted the book to do, and to evaluate it in that light. Perhaps in this light what they wished to do was create a new game in the tradition of an earlier one, and the question becomes whether they managed to improve the old game while maintaining its feeling and traditions. At this point, we might have very different opinions not because we disagree as to whether it meets its objectives, but because we disagree as to what those objectives are.

The same principle would apply to game materials that are not rule books. Multiverser: The Second Book of Worlds begins its introduction with "More of the multiverse is revealed within these pages"--and I don't think it an incredible leap to conclude that the purpose of this book is to provide settings and adventure opportunities for use with the Multiverser game system. The question then is whether it provides settings and adventures that work well within that system and provide for entertaining games.

I could look at other games, but I suspect the ones that would give me the best examples don't happen to be in my room. I've not read the introductory text for Sorcerer, but I would wager that it does a good job of expressing its purpose. Sometimes figuring out what a game or supplement is trying to do might take a bit more work.

And no matter how much better made the Craftsman Rip Saw might be, the Stanley Claw Hammer will always be better for driving nails. You need to evaluate their quality in the context of what they were made to do, not in the context of what you wanted to do with them.

--M. J. Young

[C. Edwards]

M.J. Young wrote:

And no matter how much better made the Craftsman Rip Saw might be, the Stanley Claw Hammer will always be better for driving nails. You need to evaluate their quality in the context of what they were made to do, not in the context of what you wanted to do with them.

I know this may seem like a minor distinction, or possibly no distinction at all, but I think the quality of an RPG has to be evaluated in the context of what it actually does, not what it was made to do.

Why?  Because I see game design as more akin to alchemy than chemistry.  What something was made to do isn't necessarily always what it ends up doing.  Just because an RPG doesn't end up doing exactly what it was made for doesn't mean that what it is doing doesn't work.  

-Chris

[Seth L. Blumberg]

And no matter how much better made the Craftsman Rip Saw might be, the Stanley Claw Hammer will always be better for driving nails. You need to evaluate their quality in the context of what they were made to do, not in the context of what you wanted to do with them.

Actually, I think that if I were looking for a tool to drive nails, I would have the sense to say "I need a hammer," and look up reviews specifically of hammers, ignoring any reviews of rip saws that might be present. In that context, what I want is a review which exposes the craftsmanship and construction of the hammer--not a review which discourses on its greater suitability for driving nails than for trimming 2x4s.

[Mike Holmes]

I'm not sure if you guys are getting MJ's point. He's saying that the comparison that one needs to make to determine quality is to compare the design intent with the actual play. That's his first and second questions, you'll note.

To do otherwise would be to use the saw to try to nail in a nail, to fail (in play as it were), and to conclude that the saw was a terrible hammer. So it is. But then it wasn't designed to be a hammer, so the comparison is a bit senseless.

In this way, Seth and Chris, you are agreeing with MJ, I think.

Mike

[C. Edwards]

I'm not sure if you guys are getting MJ's point. He's saying that the comparison that one needs to make to determine quality is to compare the design intent with the actual play. That's his first and second questions, you'll note.

Maybe from the designer's point of view, but certainly not from the player's perspective.

(why does this feel like the GNS thread about intent and behavior?)

Let's take a trip on down to the Home Depot.  We are standing in the tool section looking at hammers.  Hammers of different weights, hammers with smooth heads and waffled heads, hammers with straight claws and curved claws, sledge hammers, framing hammers, hatchet hammers, ball-ping hammers, a crap load of hammers that are similar and different in various ways.  Design intent and success are irrelevant to the consumer.  Just tell me what hammer works best for my specific need.

For RPGs the consumer needs to have an honest understanding of their play needs and any RPG product would need a categorization, a list of specs, that is gathered through actual play.  Fortunately design is often close to intent, but since people hell-bent on making a profit often refuse to be completely honest about their products, with themselves and consumers, the only way to provide any useful guildline of quality for the consumer is through rigorous third party "road-tests".  Something similar to a Consumer Reports Digest I suppose.

Note that while I'm not against making a profit, I am against doing so at the expense of honesty with the consumer.

I think all the analogies we've been using are starting to give me a facial tic.

-Chris

[Mike Holmes]

Fortunately design is often close to intent, but since people hell-bent on making a profit often refuse to be completely honest about their products, with themselves and consumers, the only way to provide any useful guildline of quality for the consumer is through rigorous third party "road-tests".  Something similar to a Consumer Reports Digest I suppose.

Well said. How often are RPGs billed as the "One True Game" that will solve all your RPG woes. To extend the analogy (and further exacerbate Chris' tic), it's as though they made a saw and said, this tool can do any home repair imaginable.

Occasionally they get something as "complete" as a saw/hammer hybrid tool, but it still fails to be a wrench.

Question is do reviewers take this view? Or do they assume that since they need a hammer, that everyone else does as well? How do we get that idea across to RPG reviewers?

Mike

[C. Edwards]

Question is do reviewers take this view? Or do they assume that since they need a hammer, that everyone else does as well? How do we get that idea across to RPG reviewers?

As I see it, the best option is to have an orginization, commitee, whatever, with a standardized set of review procedures and ratings to be utilized by a disciplined and well trained stable of reviewers that have a wide variety of play styles (acknowledged dubious term) and interests between them.

Each product could go through a gauntlet composed of actual play by different reviewers.  The reviewer would take their own data, along with feedback gathered from the other participants through observation and direct questioning and submit it to be averaged in with data from other playtests.  This would all hinge on a standardized data gathering protocol, including a set of well honed questions to apply to the game participants.

The result would be a set of specs that would suggest the quality (ability to fill specific needs) of a game based upon the rigorous testing and disciplined review it recieved.  I think that would be the optimal situation.  
As it is, the only "reviews" I put any stock in are the word of mouth recommendations here on The Forge and those in the Reviews section.

Yay! Not one analogy.

-Chris