An Objective Standard of Quality for RPG's

[Marco]

In another post the idea of finding a way to grade RPG's was brought it (by Valimir)

Since we know that all things produced are NOT of equal quality, if we are to uncover ways of evaluating quality we must therefor find measures that don't rely on such arguements.

It is my contention that there are only two measures that can be brought to bear (although I *am* prepared to change my mind about this).

1. Conformance to specifications: to do this you have to know what was intended to be built to a very specific degree. Then you see how well it complied. The up-side is that it's pretty objective. The down-side is that a game designed to be un-playable is high quality if it is unplayable (that and I've heard people talk about the design of VtM, for example, but without a req-document or the authors to explain what they were thinking that type of analysis is poor--maybe it was supposed to be Angsty with Crunchy Combat ... and built with flavor text that suggested the people playing it were doing a superior type of gaming to dungeon-crawling as a brilliant marketing move.)

2. Critics and Cannon: using the literature model, find voices of merrit (critics) and works of merrit (cannon) and use those as the yard-sticks. This is workable (you compare Hackmaster to Sorceror (cannon) and use Jorad Sorenson's (the critic) opinion of what both games address and how they address it and you can build an argument that one is "higher quality" than the other). The upside is that this works for something as subjectives as literature. The downside is that this necessitates an academic elite.

I don't think GNS-coherence is a sign of "quality."

I don't think that game-focus is a sign of "quality."

I think that when you boil it down, "quality" isn't that all that useful a term outside of the discussion of personal preference.

What do you all think?

-Marco
[Note: before someone suggests like number of grammar errors per page or something like that: we're cool--but that's not the discussion I'm askin' about. ]

[C. Edwards]

Perhaps quality could be derived from how consistently, and to the degree, that specific behaviour results from use of the game.  It would be a measure of clarity, compulsion, and focused design that does not rely on the designer's intent but rather how the product actually functions.

Charles S. Peirce, founder of the philosophy known as pragmatism, believed that the "meaning" of a message is the behavior it induces.  I would propose that the quality of a message (game system) can be determined by how consistently it results in a particular behavior, regardless of whether that behavior conforms to the designer's vision.

I think that the game designed to be unplayable would result in a wide set of player behavior, rendering it low on the quality scale.

My $1.50, hope it makes sense.

-Chris

[Andrew Martin]

When a physical object is low quality, it tends to break easily. For example, low quality wood breaks with lower levels of strain than higher quality wood. Similarly for RPGs, I've found that low quality RPGs "break" far more easily than high quality RPGs. Some tests I've found useful for testing quality are:

[*]Does the game system simplistically add attribute to skill number? (This is a simple shortcut that saves a lot of effort.)
[*]Does the game system start characters out at zero then require the character to be brought up to competent? (shortcut)
[*]Reasonably low handling time? The longer the time, usually (not always!) the lower quality the game is.
[*]Does the game have a point system for disadvantages? (shortcut)
[*]Is it logical for characters to describe their own descriptors by using the notation on the character sheet? Would the characters be locked up if they did so? :)
[*]Does the character's description match the character's game system values? Can the reverse be done from game system values to character description?
[*]Using the game system as the only rules (don't use the group's social contract), do the character actions match the flavour text? For example in a Supers RPG, if a superpowered character is described as flying around the world in a few seconds by the descriptive text, it should be possible for a PC superpowered character to do the same. The number of mismatches here is an objective measure of quality, where what's advertised isn't matched by what's delivered.
[*]Are the probabilities right? Is a competent character really competent when using the game system versus no or average opposition? This is a bit harder to measure for strange settings, but fairly easy to do for modern settings (Drive skill).
[/list:u]

That's just a few I can think of at the moment.

[C. Edwards]

Hey Andrew,

I don't think that it's very usefull to look at the individual components of a game system in an attempt to determine any objective overall quality when the quality of those components is a very subjective determination.

Just my opinion.

-Chris

[deadpanbob]

I don't think that it's very usefull to look at the individual components of a game system in an attempt to determine any objective overall quality when the quality of those components is a very subjective determination.

Chris,

I may have to disagree with you there.

First, however, to address Marco's second measure of quality - one thing to keep in mind is that the best works of literature are usually only understood with historical perspective.  Its very difficult to say what fiction will survive from today to 100 years hence - and be considered quality literature in that time by that time's cannon and critics.

We may not have enough history behind us in terms of judging RPGs by this measure.

In terms of measuring the individual components of the game - of course this is subjective.  Almost all ratings of quality are.  They way these things are done for, say, automobiles is that a group of experts (so-called) comes up with a list of individual elements of automobiles, and then they (or the general public in the case of say JD Power & Associates) rate a set of automobiles on those elements.  The cars with the highest overall scores are considered to be 'quality' automobiles.

Admittedly, with a hardline product like Automobiles, there are some relatively more objective measures that can be looked at (number of times on average in the shop for repairs over X time frame, MPG, etc).  But still, often time quality in these arenas comes down to opinion as well.

I think that if a whole group of disparte people thing a product is good, it probably is - especially if they can all cite both similar reasons for why its good and wildly unique reasons why its good.  Conversely, if a lot of people think a product is broken or of low quality, I tend to think that the sheer weight of the distaste means the product is probably crap.

Obviously, as in every human endevor, there will be exceptions.  My training and profession are my bias here - I'm a statistician by trade - so I tend to think in measures of centrality and the weight of the masses.

I will point out one thing - I've seen people here say that judging the quality of an RPG isn't objective like say math.  In my experience, math is very subjective.  There are often times more ways than one to solve a math problem - particularly those math problems that are more complex.  Sometimes it can be tough to judge the quality of a given mathematical problem just by whether or not is computed the right solution.

What I'm saying is, be careful throwing around words like subjective and objective - because even these are subjective terms ;-)

Cheers,

Jason

[C. Edwards]

Hey Jason,

I'm not saying that the quality of an RPG can be determined to the level of mathematical certainty.  I am saying that using value judgments like "this system has low handling time and character advancement proceeds at a perfect pace and that is why I consider this a quality product" are useless as a system for attempting to determine a relative degree of overall quality not based on personal opinion.  

The idea is to come to an agreement on whether there is a system of measurement that would yield a result that could be used as a determiner of relative (to other game systems) quality without utilizing value judgments.  


quality

The totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs.

The "totality" part of that definition is, I think, the key issue.  Let's say we have a group of players looking for a game.  They want a game that promotes a certain style of play (yes, I know, a dubious term).  After a look at the data on, say, V:TM they see that it doesn't consistently promote the style of play, the player behavior, for which they are looking.  After finding a game whose data suggests that it consistently promotes the style of play they want, let's say Sorcerer, they can then look at the individual qualities of that game (handling time, apparent character competence, etc.) and see how they measure up to their needs.
It's much more efficient to look at the relative overall quality of a game and then narrow in on individual aspects of the games that looked promising than to look at all the individual aspects in an attempt to determine what kind of play experience they would induce.

A game system's quality would simply be the result  of how consistently it promoted certain player behavior.  No value judgment as to what behavior or individual qualities are better would be involved.

How would this apply to a game like Scattershot?  I'm really not sure.  In a sense this kind of determination is already used in the word of mouth sense.  Most of us are aware on some level that certain games, like AD&D, consistently promote certain player behavior.  If we had hard data we might be able to say that, while AD&D certainly promotes a certain type of player behavior consistently, it is of lower relative quality level than The Questing Beast, which promotes a different style of play more consistently based on available data.

Of course with nothing more than word of mouth at hand and gathering hard data of the type that would be needed being a pain in the ass this doesn't seem to be an incredibly practical method.  I do think that in theory it would be an excellent benchmark for quality.

-Chris

[Valamir]

Hey Chris, I really like your above idea of quality being measured by consistant output.  Its actually very close in sentiment to what I was envisioning by my comparison of designer intent to actual play, except I had inserted the assumption that a particular consistancy of output was part of the designer's intent to begin with.

No doubt a bias from my own history of game design experience and belief in a designer's ability to conciously make design decisions to influence that output.

I think your definition is much more broadly applicable than the direction I was going.

Although I am left to wonder if there is a certain point in "output consistancy" which is impossible to go beyond, without concious efforts by the designer to do so.

[Marco]

Good discussion so far. A few points:

I rather liked the idea of how well a system promotes a "style" of play--but I think it has some operational issues:

1. The first is the control group: if people who think they'll like Sorceror buy Sorceror, and play it consistently then you've proved more about them than Sorceror. If you give it to people who haven't bought it and have no known inclination to it's specific play style you get a good test--but in practice that's really almost impossible to do.

The second is: is any behavior good so long as it's consistent? That could make VtM quality--and would that be satisfactory?  The VtM example suggests that the group finds that the behavior driven isn't what they want--does that mean it's low quality? Or does that mean it's high quality but not what they like? (I think there's a hint of value judgement here that suggests Critics and Cannon for the final "quality" evaluation).

2. I didn't understand Andrew's ideas (point systems for disadvantages are shortcut? Is that supposed to be good or bad? Handling time is bad so one would think shortcuts were good--but I don't know). The only one that I thought made sense was the question about how well the flavor text lives up to the mechanics (and how well the probabilities work out). The rest of it seemed very subjective to me.

This has the obvious issue that if a game doesn't have flavor text (or not much of it)--or none of it deals with mechanical issues ... what are you to do?

-Marco

[Valamir]

Good discussion so far. A few points:

I rather liked the idea of how well a system promotes a "style" of play--but I think it has some operational issues:

1. The first is the control group: if people who think they'll like Sorceror buy Sorceror, and play it consistently then you've proved more about them than Sorceror. If you give it to people who haven't bought it and have no known inclination to it's specific play style you get a good test--but in practice that's really almost impossible to do.

I'll have to ponder this.  I don't know that ther's away around it practically (barring some theoretical "RPG Crash Test Center".  But I don't know that its really an issue either.  RPG's can have a "target market" too.  So it seems to me that what bias is inherent in evaluating the play of people who liked the concept enough to buy it, is just evaluating the target market for the game rather than the mass market, and might actually be a MORE accurate standard.

The VtM example suggests that the group finds that the behavior driven isn't what they want--does that mean it's low quality? Or does that mean it's high quality but not what they like? (I think there's a hint of value judgement here that suggests Critics and Cannon for the final "quality" evaluation).

Well I think being able to understand that a game is high quality but not what they like is the biggest application for a model like GNS.  Theres a recent thread by ACE around here that serves as something of a testimonial for Forge-like discussions developing an appreciation for games even if they aren't our favorites.  A big motivation behind GNS is to get away from the "Suxors / Roxors" method of evaluation, and be able to judge a game on its own merits rather than simply personal taste.

Obviously, being human after all, that goal is often difficult to fully achieve.

The only one that I thought made sense was the question about how well the flavor text lives up to the mechanics <snip>This has the obvious issue that if a game doesn't have flavor text (or not much of it)... what are you to do?

This is actually (as you probably suspect) one of my biggest yardsticks, and is the primary criteria that V:tM violates IMO.

Alot of discussion about "appropriate mechanics" really boils down to how well the game's mechanics lives up to the "flavor text" (or more broadly, the setting and background color).  Even mechanics that desire to model reality as closely as possible can (in the techniques and mechanisms they choose) have a greater or lesser degree of reflecting that flavor built into them.

I don't know that it takes a huge volume of flavor text to measure against, especially if the setting is one where the genre conventions are pretty widely known.  For instance, a game like Children of the Sun, requires more flavor text to establish the setting firmly in our minds, than a game like Swashbuckler!, where the tropes of rapiers, wit, and derring-do are well understood by most.  So if you expend "Flavor Text" to include both "written Flavor Text" and "Flavor implied by the background" you'd have more to work with.

Further, and a matter for another thread, is the idea of a game that has absolutely no flavor to it (even implied) at all.  Is such a thing actually a game, or is it simply a system waiting to be attached to a flavorful setting before it becomes a complete game.  We've exchanged some emails on that one already.

-Marco[/quote]

[Clinton R. Nixon]

One thing that I see missing - and one reason objective quality is hard to determine in RPGs - is a written statement about what an RPG is to be used for. While flavor text and background can help determine this, few RPGs come out and state the purpose of the RPG.

Without an explicit purpose, quality is hard to determine on an objective level. It can still be determined on a subjective level by asking "Does this RPG do what I wish it to do?" That's not terribly helpful to others, though.

If I were to pick up a very well made auger, for example, but had no clue what it was to be used for, I might determine that it's a terrible drill: it's much too large and unwieldy. Without the knowledge that it's to be used for making holes in the ground, I couldn't make an objective statement of quality about it.

[Marco]

One thing that I see missing - and one reason objective quality is hard to determine in RPGs - is a written statement about what an RPG is to be used for. While flavor text and background can help determine this, few RPGs come out and state the purpose of the RPG.

Without an explicit purpose, quality is hard to determine on an objective level. It can still be determined on a subjective level by asking "Does this RPG do what I wish it to do?" That's not terribly helpful to others, though.

If I were to pick up a very well made auger, for example, but had no clue what it was to be used for, I might determine that it's a terrible drill: it's much too large and unwieldy. Without the knowledge that it's to be used for making holes in the ground, I couldn't make an objective statement of quality about it.

I agree with this completely. It's one of the things that makes a real, objective standard of quality very hard to create.

-Marco

[Mike Holmes]

I agree with this completely. It's one of the things that makes a real, objective standard of quality very hard to create.

There is one case in which I can know completely the intent of the designer. That is when I am he. See, this is how this is useful. When I make a game, I know what I want. Then I apply these measures as best I can (most often using the independent playtesting method as the best criteria of success).

This is the value to me of System Matters and GNS. Knowing what I want my design to do, I can go forth and more effectively create the game that I sought to make. What do I care whether or not Vampire does what it was designed for. All I care is that it doesn't do what I want. And that I can make a vampire game that meets my desires should I need to.

Mike

[Valamir]

From another thread:

I avoid the word "objective" strenuously. This is my scientific background speaking - the best term is "rigorous," meaning the claim is defensible through a combination of evidence and logic. Not true, not irrefutable, and in fact admittedly possibly wrong - but rigorous given what we know and how we've agreed to argue.

I'd agree with this, even though I tend to use the word objective a lot as a form of lazy shorthand.   IMO "objective" is a pretty useless term in and of itself.  For one, IMO, there is no and can be no such thing.  The idea of impartiality is a great theory but generally an illusion.  The only person who can be completely impartial is someone who completely doesn't care.  But if they don't care then they can hardly be relied upon to perform the effort needed to evaluate thoroughly before rendering a judgment.  If someone does care enough to make the effort to evaluate, than they are already bringing with them a set of biases and preconceptions that can at best be enumerated and mitigated, but never truly eliminated.

Since it is an impossible standard to achieve, we can discard it as a requirement altogether.

Just because a methodology has shortcomings and imperfections doesn't mean that using it can't provide compelling insights.  

The enemy of good enough is perfection.  Perfection is a paralyzing standard to apply.  GNS isn't perfect, and the various related theories aren't either.  But they're good enough to work for what we need them to do.  And when we encounter an area where they're not...we work on them.

[Marco]

There is one case in which I can know completely the intent of the designer. That is when I am he.

Mike

Sure. And knowing what you want to build is a good way to start--but that doesn't make it useful to anyone else (I can't look at The Window and say "that is high quality" or "that is low quality" and that's what an objective measure would ask for).

Don't get me wrong: how closely a game meets requirements is the only objective standard that I think will work. I think it's fraught with practical-application problems (will authors have to write high-level or low-level requirements docs prior to a Quality Review?) but I think it's the only one that's got a real fighting chance.

And it has the added benefit that one can't use it to declare VtM low-quality without it (you can't infer requirements from flavor text any more than a programmer can infer design from a general description).

-Marco

[Marco]

I agree with Ron. Riggorous is better than Objective. How might that differ in the application of analysis of an RPG?

-Marco