Topic: An Objective Standard of Quality for RPG's
Started by: Marco
Started on: 9/23/2002
Board: RPG Theory
On 9/23/2002 at 1:09am, Marco wrote:
An Objective Standard of Quality for RPG's
In another post the idea of finding a way to grade RPG's was brought it (by Valimir)
Since we know that all things produced are NOT of equal quality, if we are to uncover ways of evaluating quality we must therefor find measures that don't rely on such arguements.
It is my contention that there are only two measures that can be brought to bear (although I *am* prepared to change my mind about this).
1. Conformance to specifications: to do this you have to know what was intended to be built to a very specific degree. Then you see how well it complied. The up-side is that it's pretty objective. The down-side is that a game designed to be un-playable is high quality if it is unplayable (that and I've heard people talk about the design of VtM, for example, but without a req-document or the authors to explain what they were thinking that type of analysis is poor--maybe it was supposed to be Angsty with Crunchy Combat ... and built with flavor text that suggested the people playing it were doing a superior type of gaming to dungeon-crawling as a brilliant marketing move.)
2. Critics and Cannon: using the literature model, find voices of merrit (critics) and works of merrit (cannon) and use those as the yard-sticks. This is workable (you compare Hackmaster to Sorceror (cannon) and use Jorad Sorenson's (the critic) opinion of what both games address and how they address it and you can build an argument that one is "higher quality" than the other). The upside is that this works for something as subjectives as literature. The downside is that this necessitates an academic elite.
I don't think GNS-coherence is a sign of "quality."
I don't think that game-focus is a sign of "quality."
I think that when you boil it down, "quality" isn't that all that useful a term outside of the discussion of personal preference.
What do you all think?
-Marco
[Note: before someone suggests like number of grammar errors per page or something like that: we're cool--but that's not the discussion I'm askin' about. ]
On 9/23/2002 at 1:46am, C. Edwards wrote:
RE: An Objective Standard of Quality for RPG's
Perhaps quality could be derived from how consistently, and to the degree, that specific behaviour results from use of the game. It would be a measure of clarity, compulsion, and focused design that does not rely on the designer's intent but rather how the product actually functions.
Charles S. Peirce, founder of the philosophy known as pragmatism, believed that the "meaning" of a message is the behavior it induces. I would propose that the quality of a message (game system) can be determined by how consistently it results in a particular behavior, regardless of whether that behavior conforms to the designer's vision.
I think that the game designed to be unplayable would result in a wide set of player behavior, rendering it low on the quality scale.
My $1.50, hope it makes sense.
-Chris
On 9/23/2002 at 2:09am, Andrew Martin wrote:
RE: An Objective Standard of Quality for RPG's
When a physical object is low quality, it tends to break easily. For example, low quality wood breaks with lower levels of strain than higher quality wood. Similarly for RPGs, I've found that low quality RPGs "break" far more easily than high quality RPGs. Some tests I've found useful for testing quality are:
• Does the game system simplistically add attribute to skill number? (This is a simple shortcut that saves a lot of effort.)
• Does the game system start characters out at zero then require the character to be brought up to competent? (shortcut)
• Reasonably low handling time? The longer the time, usually (not always!) the lower quality the game is.
• Does the game have a point system for disadvantages? (shortcut)
• Is it logical for characters to describe their own descriptors by using the notation on the character sheet? Would the characters be locked up if they did so? :)
• Does the character's description match the character's game system values? Can the reverse be done from game system values to character description?
• Using the game system as the only rules (don't use the group's social contract), do the character actions match the flavour text? For example in a Supers RPG, if a superpowered character is described as flying around the world in a few seconds by the descriptive text, it should be possible for a PC superpowered character to do the same. The number of mismatches here is an objective measure of quality, where what's advertised isn't matched by what's delivered.
• Are the probabilities right? Is a competent character really competent when using the game system versus no or average opposition? This is a bit harder to measure for strange settings, but fairly easy to do for modern settings (Drive skill).
That's just a few I can think of at the moment.
On 9/23/2002 at 2:29am, C. Edwards wrote:
RE: An Objective Standard of Quality for RPG's
Hey Andrew,
I don't think that it's very usefull to look at the individual components of a game system in an attempt to determine any objective overall quality when the quality of those components is a very subjective determination.
Just my opinion.
-Chris
On 9/23/2002 at 3:09am, deadpanbob wrote:
RE: An Objective Standard of Quality for RPG's
C. Edwards wrote:
I don't think that it's very usefull to look at the individual components of a game system in an attempt to determine any objective overall quality when the quality of those components is a very subjective determination.
Chris,
I may have to disagree with you there.
First, however, to address Marco's second measure of quality - one thing to keep in mind is that the best works of literature are usually only understood with historical perspective. Its very difficult to say what fiction will survive from today to 100 years hence - and be considered quality literature in that time by that time's cannon and critics.
We may not have enough history behind us in terms of judging RPGs by this measure.
In terms of measuring the individual components of the game - of course this is subjective. Almost all ratings of quality are. They way these things are done for, say, automobiles is that a group of experts (so-called) comes up with a list of individual elements of automobiles, and then they (or the general public in the case of say JD Power & Associates) rate a set of automobiles on those elements. The cars with the highest overall scores are considered to be 'quality' automobiles.
Admittedly, with a hardline product like Automobiles, there are some relatively more objective measures that can be looked at (number of times on average in the shop for repairs over X time frame, MPG, etc). But still, often time quality in these arenas comes down to opinion as well.
I think that if a whole group of disparte people thing a product is good, it probably is - especially if they can all cite both similar reasons for why its good and wildly unique reasons why its good. Conversely, if a lot of people think a product is broken or of low quality, I tend to think that the sheer weight of the distaste means the product is probably crap.
Obviously, as in every human endevor, there will be exceptions. My training and profession are my bias here - I'm a statistician by trade - so I tend to think in measures of centrality and the weight of the masses.
I will point out one thing - I've seen people here say that judging the quality of an RPG isn't objective like say math. In my experience, math is very subjective. There are often times more ways than one to solve a math problem - particularly those math problems that are more complex. Sometimes it can be tough to judge the quality of a given mathematical problem just by whether or not is computed the right solution.
What I'm saying is, be careful throwing around words like subjective and objective - because even these are subjective terms ;-)
Cheers,
Jason
On 9/23/2002 at 6:43am, C. Edwards wrote:
RE: An Objective Standard of Quality for RPG's
Hey Jason,
I'm not saying that the quality of an RPG can be determined to the level of mathematical certainty. I am saying that using value judgments like "this system has low handling time and character advancement proceeds at a perfect pace and that is why I consider this a quality product" are useless as a system for attempting to determine a relative degree of overall quality not based on personal opinion.
The idea is to come to an agreement on whether there is a system of measurement that would yield a result that could be used as a determiner of relative (to other game systems) quality without utilizing value judgments.
quality
The totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs.
The "totality" part of that definition is, I think, the key issue. Let's say we have a group of players looking for a game. They want a game that promotes a certain style of play (yes, I know, a dubious term). After a look at the data on, say, V:TM they see that it doesn't consistently promote the style of play, the player behavior, for which they are looking. After finding a game whose data suggests that it consistently promotes the style of play they want, let's say Sorcerer, they can then look at the individual qualities of that game (handling time, apparent character competence, etc.) and see how they measure up to their needs.
It's much more efficient to look at the relative overall quality of a game and then narrow in on individual aspects of the games that looked promising than to look at all the individual aspects in an attempt to determine what kind of play experience they would induce.
A game system's quality would simply be the result of how consistently it promoted certain player behavior. No value judgment as to what behavior or individual qualities are better would be involved.
How would this apply to a game like Scattershot? I'm really not sure. In a sense this kind of determination is already used in the word of mouth sense. Most of us are aware on some level that certain games, like AD&D, consistently promote certain player behavior. If we had hard data we might be able to say that, while AD&D certainly promotes a certain type of player behavior consistently, it is of lower relative quality level than The Questing Beast, which promotes a different style of play more consistently based on available data.
Of course with nothing more than word of mouth at hand and gathering hard data of the type that would be needed being a pain in the ass this doesn't seem to be an incredibly practical method. I do think that in theory it would be an excellent benchmark for quality.
-Chris
On 9/23/2002 at 2:35pm, Valamir wrote:
RE: An Objective Standard of Quality for RPG's
Hey Chris, I really like your above idea of quality being measured by consistant output. Its actually very close in sentiment to what I was envisioning by my comparison of designer intent to actual play, except I had inserted the assumption that a particular consistancy of output was part of the designer's intent to begin with.
No doubt a bias from my own history of game design experience and belief in a designer's ability to conciously make design decisions to influence that output.
I think your definition is much more broadly applicable than the direction I was going.
Although I am left to wonder if there is a certain point in "output consistancy" which is impossible to go beyond, without concious efforts by the designer to do so.
On 9/23/2002 at 3:34pm, Marco wrote:
RE: An Objective Standard of Quality for RPG's
Good discussion so far. A few points:
I rather liked the idea of how well a system promotes a "style" of play--but I think it has some operational issues:
1. The first is the control group: if people who think they'll like Sorceror buy Sorceror, and play it consistently then you've proved more about them than Sorceror. If you give it to people who haven't bought it and have no known inclination to it's specific play style you get a good test--but in practice that's really almost impossible to do.
The second is: is any behavior good so long as it's consistent? That could make VtM quality--and would that be satisfactory? The VtM example suggests that the group finds that the behavior driven isn't what they want--does that mean it's low quality? Or does that mean it's high quality but not what they like? (I think there's a hint of value judgement here that suggests Critics and Cannon for the final "quality" evaluation).
2. I didn't understand Andrew's ideas (point systems for disadvantages are shortcut? Is that supposed to be good or bad? Handling time is bad so one would think shortcuts were good--but I don't know). The only one that I thought made sense was the question about how well the flavor text lives up to the mechanics (and how well the probabilities work out). The rest of it seemed very subjective to me.
This has the obvious issue that if a game doesn't have flavor text (or not much of it)--or none of it deals with mechanical issues ... what are you to do?
-Marco
On 9/23/2002 at 4:07pm, Valamir wrote:
RE: An Objective Standard of Quality for RPG's
Marco wrote: Good discussion so far. A few points:
I rather liked the idea of how well a system promotes a "style" of play--but I think it has some operational issues:
1. The first is the control group: if people who think they'll like Sorceror buy Sorceror, and play it consistently then you've proved more about them than Sorceror. If you give it to people who haven't bought it and have no known inclination to it's specific play style you get a good test--but in practice that's really almost impossible to do.
I'll have to ponder this. I don't know that ther's away around it practically (barring some theoretical "RPG Crash Test Center". But I don't know that its really an issue either. RPG's can have a "target market" too. So it seems to me that what bias is inherent in evaluating the play of people who liked the concept enough to buy it, is just evaluating the target market for the game rather than the mass market, and might actually be a MORE accurate standard.
The VtM example suggests that the group finds that the behavior driven isn't what they want--does that mean it's low quality? Or does that mean it's high quality but not what they like? (I think there's a hint of value judgement here that suggests Critics and Cannon for the final "quality" evaluation).
Well I think being able to understand that a game is high quality but not what they like is the biggest application for a model like GNS. Theres a recent thread by ACE around here that serves as something of a testimonial for Forge-like discussions developing an appreciation for games even if they aren't our favorites. A big motivation behind GNS is to get away from the "Suxors / Roxors" method of evaluation, and be able to judge a game on its own merits rather than simply personal taste.
Obviously, being human after all, that goal is often difficult to fully achieve.
The only one that I thought made sense was the question about how well the flavor text lives up to the mechanics <snip>This has the obvious issue that if a game doesn't have flavor text (or not much of it)... what are you to do?
This is actually (as you probably suspect) one of my biggest yardsticks, and is the primary criteria that V:tM violates IMO.
Alot of discussion about "appropriate mechanics" really boils down to how well the game's mechanics lives up to the "flavor text" (or more broadly, the setting and background color). Even mechanics that desire to model reality as closely as possible can (in the techniques and mechanisms they choose) have a greater or lesser degree of reflecting that flavor built into them.
I don't know that it takes a huge volume of flavor text to measure against, especially if the setting is one where the genre conventions are pretty widely known. For instance, a game like Children of the Sun, requires more flavor text to establish the setting firmly in our minds, than a game like Swashbuckler!, where the tropes of rapiers, wit, and derring-do are well understood by most. So if you expend "Flavor Text" to include both "written Flavor Text" and "Flavor implied by the background" you'd have more to work with.
Further, and a matter for another thread, is the idea of a game that has absolutely no flavor to it (even implied) at all. Is such a thing actually a game, or is it simply a system waiting to be attached to a flavorful setting before it becomes a complete game. We've exchanged some emails on that one already.
-Marco
On 9/23/2002 at 5:08pm, Clinton R. Nixon wrote:
RE: An Objective Standard of Quality for RPG's
One thing that I see missing - and one reason objective quality is hard to determine in RPGs - is a written statement about what an RPG is to be used for. While flavor text and background can help determine this, few RPGs come out and state the purpose of the RPG.
Without an explicit purpose, quality is hard to determine on an objective level. It can still be determined on a subjective level by asking "Does this RPG do what I wish it to do?" That's not terribly helpful to others, though.
If I were to pick up a very well made auger, for example, but had no clue what it was to be used for, I might determine that it's a terrible drill: it's much too large and unwieldy. Without the knowledge that it's to be used for making holes in the ground, I couldn't make an objective statement of quality about it.
On 9/23/2002 at 7:04pm, Marco wrote:
RE: An Objective Standard of Quality for RPG's
Clinton R. Nixon wrote: One thing that I see missing - and one reason objective quality is hard to determine in RPGs - is a written statement about what an RPG is to be used for. While flavor text and background can help determine this, few RPGs come out and state the purpose of the RPG.
Without an explicit purpose, quality is hard to determine on an objective level. It can still be determined on a subjective level by asking "Does this RPG do what I wish it to do?" That's not terribly helpful to others, though.
If I were to pick up a very well made auger, for example, but had no clue what it was to be used for, I might determine that it's a terrible drill: it's much too large and unwieldy. Without the knowledge that it's to be used for making holes in the ground, I couldn't make an objective statement of quality about it.
I agree with this completely. It's one of the things that makes a real, objective standard of quality very hard to create.
-Marco
On 9/23/2002 at 7:15pm, Mike Holmes wrote:
RE: An Objective Standard of Quality for RPG's
Marco wrote: I agree with this completely. It's one of the things that makes a real, objective standard of quality very hard to create.
There is one case in which I can know completely the intent of the designer. That is when I am he. See, this is how this is useful. When I make a game, I know what I want. Then I apply these measures as best I can (most often using the independent playtesting method as the best criteria of success).
This is the value to me of System Matters and GNS. Knowing what I want my design to do, I can go forth and more effectively create the game that I sought to make. What do I care whether or not Vampire does what it was designed for. All I care is that it doesn't do what I want. And that I can make a vampire game that meets my desires should I need to.
Mike
On 9/23/2002 at 7:24pm, Valamir wrote:
RE: An Objective Standard of Quality for RPG's
From another thread:
Ron Edwards wrote: I avoid the word "objective" strenuously. This is my scientific background speaking - the best term is "rigorous," meaning the claim is defensible through a combination of evidence and logic. Not true, not irrefutable, and in fact admittedly possibly wrong - but rigorous given what we know and how we've agreed to argue.
I'd agree with this, even though I tend to use the word objective a lot as a form of lazy shorthand. IMO "objective" is a pretty useless term in and of itself. For one, IMO, there is no and can be no such thing. The idea of impartiality is a great theory but generally an illusion. The only person who can be completely impartial is someone who completely doesn't care. But if they don't care then they can hardly be relied upon to perform the effort needed to evaluate thoroughly before rendering a judgment. If someone does care enough to make the effort to evaluate, than they are already bringing with them a set of biases and preconceptions that can at best be enumerated and mitigated, but never truly eliminated.
Since it is an impossible standard to achieve, we can discard it as a requirement altogether.
Just because a methodology has shortcomings and imperfections doesn't mean that using it can't provide compelling insights.
The enemy of good enough is perfection. Perfection is a paralyzing standard to apply. GNS isn't perfect, and the various related theories aren't either. But they're good enough to work for what we need them to do. And when we encounter an area where they're not...we work on them.
On 9/23/2002 at 7:28pm, Marco wrote:
RE: An Objective Standard of Quality for RPG's
Mike Holmes wrote: There is one case in which I can know completely the intent of the designer. That is when I am he.
Mike
Sure. And knowing what you want to build is a good way to start--but that doesn't make it useful to anyone else (I can't look at The Window and say "that is high quality" or "that is low quality" and that's what an objective measure would ask for).
Don't get me wrong: how closely a game meets requirements is the only objective standard that I think will work. I think it's fraught with practical-application problems (will authors have to write high-level or low-level requirements docs prior to a Quality Review?) but I think it's the only one that's got a real fighting chance.
And it has the added benefit that one can't use it to declare VtM low-quality without it (you can't infer requirements from flavor text any more than a programmer can infer design from a general description).
-Marco
On 9/23/2002 at 7:31pm, Marco wrote:
RE: An Objective Standard of Quality for RPG's
I agree with Ron. Riggorous is better than Objective. How might that differ in the application of analysis of an RPG?
-Marco
On 9/23/2002 at 7:32pm, C. Edwards wrote:
RE: An Objective Standard of Quality for RPG's
Hey Clinton,
One thing that I see missing - and one reason objective quality is hard to determine in RPGs - is a written statement about what an RPG is to be used for. While flavor text and background can help determine this, few RPGs come out and state the purpose of the RPG.
The way I see it, the purpose of an RPG is the behavior it promotes, which may end up being distinctly seperate from its stated purpose. The problem, as I see it, with relating quality to how well a product matches its stated purpose is that honesty in advertising is almost non-existant.
Let's simplify the tool example by looking at the hammer. If we could look at data that showed the behavior induced by the hammer on those totally unfamiliar with its use we may find that, yes, several people used the hammer to cut down trees. I think the overwhelming majority of data would show that most used the hammer to bang, beat, and smash things. Minus a nail or wooden peg, that is where the hammer excels in use. The same idea applys to the saw. While it may end up being used for a variety of purposes, like making music, the majority of data would show it being used to cut things.
If the data on a product didn't show an inclination to produce any one behavior in a consistent manner than it could be determined that while that product could be used to recreate a variety of behaviors it doesn't specialize in inducing any particular behavior. The data would, theoretically, also be able to show if say three seperate sets of behavior were promoted in a consistent manner. Consistency would be a large determiner of relative quality. If somebody was looking for a tool that was good at cutting things, the data might suggest a saw, a utility knife, and a hatchet. Looking at what materials the tools were consistently applied to would then narrow down the options even more.
If we look at the man behind the curtain, all this comes down to is coherence.
Ron Edwards wrote:
System, system, system. Or more appropriately, design, design, design. The listed elements in Chapter One (character, situation, color, setting, system, initial premise) may be organized to facilitate greater coherence in Chapters Two (GNS, developed Premise) and Chapter Three (Stance), and thus to facilitate more enjoyable play. This principle is often summarized in the catch-phrase, “System does matter.”
By "coherence," I mean the degree to which a group of people can hit upon and sustain a shared Premise (or topic for Exploration, in Simulationist play) - and by definition, continue to enjoy the social role-playing activity consistently. The people do not need to agree in every detail or event of play, and they certainly do not have to conform to a single, immutable Stance or GNS profile. However, to role-play together most successfully, their shared agreements do need to go beyond simply sharing the initial Premise. To whatever extent they do this, they are cohering.
The measure of quality I'm suggesting would relate to a games abilty to consistently promote a high degree of coherence based upon collected actual play data. Then people could look solely at the games which "cohere" in the specific manner they desire, letting them apply their own subjective terms for quality. This is already done in a word of mouth fashion.
If I were to pick up a very well made auger, for example, but had no clue what it was to be used for, I might determine that it's a terrible drill: it's much too large and unwieldy. Without the knowledge that it's to be used for making holes in the ground, I couldn't make an objective statement of quality about it.
Yes, but if you were looking for something that would make a hole in wood or metal and you had the data on different tools available, you would see that only 2 times out of 1000 has an auger been used to try and make a hole in wood or metal. You could also see that a drill is used much more often for the specific need you are looking to fill. It is of higher quality for your purposes.
What I see is people applying the component definition of quality which, if we are looking for the overall capability of an rpg to function in a certain manner is, I would say, the wrong definition to apply.
-Chris
On 9/23/2002 at 7:45pm, Marco wrote:
RE: An Objective Standard of Quality for RPG's
If the data on a product didn't show an inclination to produce any one behavior in a consistent manner than it could be determined that while that product could be used to recreate a variety of behaviors it doesn't specialize in inducing any particular behavior. The data would, theoretically, also be able to show if say three seperate sets of behavior were promoted in a consistent manner. Consistency would be a large determiner of relative quality. If somebody was looking for a tool that was good at cutting things, the data might suggest a saw, a utility knife, and a hatchet. Looking at what materials the tools were consistently applied to would then narrow down the options even more.
I'm not sure how this applies to a quality rating. Is a game in which everyone plays gamist of higher quality than one where half the people display what you think of as gamist behavior and half display what you think of as Simulaitonist behavior?
What if I have a game that has a perfect spread (33% G, 33% S, 33% N) does that make it low quality or a "perfect" tool?
Finally, if almost everyone plays VtM in a Simulationist manner (I suspect this is the case), does that make it a high quality game? Ron has said that play of The Window will drift to either Sim or Nar (roughly 50%) does that make it low quality?
-Marco
On 9/23/2002 at 8:05pm, C. Edwards wrote:
RE: An Objective Standard of Quality for RPG's
Quote:
If the data on a product didn't show an inclination to produce any one behavior in a consistent manner than it could be determined that while that product could be used to recreate a variety of behaviors it doesn't specialize in inducing any particular behavior. The data would, theoretically, also be able to show if say three seperate sets of behavior were promoted in a consistent manner. Consistency would be a large determiner of relative quality. If somebody was looking for a tool that was good at cutting things, the data might suggest a saw, a utility knife, and a hatchet. Looking at what materials the tools were consistently applied to would then narrow down the options even more.
I'm not sure how this applies to a quality rating. Is a game in which everyone plays gamist of higher quality than one where half the people display what you think of as gamist behavior and half display what you think of as Simulaitonist behavior?
What if I have a game that has a perfect spread (33% G, 33% S, 33% N) does that make it low quality or a "perfect" tool?
Finally, if almost everyone plays VtM in a Simulationist manner (I suspect this is the case), does that make it a high quality game? Ron has said that play of The Window will drift to either Sim or Nar (roughly 50%) does that make it low quality?
-Marco
This is, I think, the heart of the misunderstanding.
quality: The totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs.
Quality of a product cannot be seperated from the needs of the user.
If I'm looking for a game that promotes sim play than V:tM, or Rolemaster, or whatever, would be of higher quality for my purposes. The relative level of quality between them could then be taken into account. Which game promotes sim play more consistently? For this example, that game would be of the highest quality when it came to fulfilling the specific need. If I wanted a game that could drift between sim and nar than perhaps the Window is of higher quality for my specific needs.
Context is everything.
-Chris
On 9/23/2002 at 8:18pm, deadpanbob wrote:
RE: An Objective Standard of Quality for RPG's
So maybe we need to develop three measures of quality - at least - one each for G/N/S?
I envision something like movie ratings on the backs of RPGs everywhere - telling you the discerning game buyer which mode of G/N/S play this game supports most fully.
If only we could set the Forge up as the independent rating agency for RPGs, we'd be set. ;-)
I still think, for what it's worth, that measuring quality requires first and foremost a determination of what's being measured. A metric must be agreed upon by all parties in the rating process.
Sure, the sum of the parts is often more than the whole - but in my experience you can't rate the whole unless you understand and rate the parts.
Put another way, to say that a game tends to result mostly in instances of play that appear (upon observation of the behavior of those playing) to be Gamist modes of play, is a quality game may ignore the fact that this is true overall, but not on some specifics. Say that this Gamist mode encouraging RPG has a very detailed skill system modeled after hard core developmental psychology, and the game itself is supposed to be say a real world Spy game.
However, the game has a resource mechanic and/or a currency mechanic that encourages Gamist play - to a degree that overwhelms the highly Simulationist supporting skill system. Ovserving only the results in play, one could come to the conclusion that the game is a quality game because it seems to pretty coherently result in instances of Gamist mode play. But without examining the fine details, we might miss the fact that there are parts of the game that don't coherently support that type of play at all.
Cheers,
Jason
On 9/23/2002 at 8:35pm, Valamir wrote:
RE: An Objective Standard of Quality for RPG's
Perhaps the best place to start, is to replace notions of "quality" with all of the extraneous baggage that word entails with the notion of "utility".
After all, GNS was not designed to evaluate the "quality" of a game design.
Rather it is an effective tool for matching player decision type with a game that promotes that decision type. In other words, evaluating the utility of a particular design for a particular player.
As far as what is better a 100% S game or a 33/33/33 game (not that such ratings are very useful except as a vague illustration)...that depends on the play group.
If I'm looking to play sim, and my 5 other players are looking to play sim, than 100% sim is clearly the better (read: more useful) game for us.
If instead my group of 6 consists of 2 of each style player, than there is really only 2 choices.
1) The group is dysfunctional and struggles to satisfy anyone. Best bet is to either learn to adapt to each other, or find a new group.
2) The group works well together and we all learn to appreciate each others preferences and blah blah blah...in which case the 33/33/33 version MAY be a perfectly useable option. GNS never says that this option is "bad" it merely says that this option is more likely to produce dysfunctional play. Not that it MUST produce dysfunctional play.
Ergo: we should be talking about objective measures of utility, not of quality. I think we pretty much have been, but the "quality" label can make this get very shaky very quick.
On 9/23/2002 at 8:52pm, deadpanbob wrote:
RE: An Objective Standard of Quality for RPG's
Valamir wrote:
Ergo: we should be talking about objective measures of utility, not of quality. I think we pretty much have been, but the "quality" label can make this get very shaky very quick.
Yes! That it for me. I like this idea best of all. Changing the word from quality to utility is a really good move. When I changed this word internally, it did drop out all of my mostly work-related issues of quality.
So, Valamir, how would we start the process of building a utility rating?
Would the best place to start be a re-hash of those behaviors in play that tend to support a given mode of play for a given instance of play?
Cheers,
Jason.
On 9/23/2002 at 9:09pm, Valamir wrote:
RE: An Objective Standard of Quality for RPG's
Hmmm, don't know Jason. That would be a pretty ambitious undertaking.
I think it might be possible to come up with some conceptual measures. The sort of thing like the formulas they teach you in Econ 101. Formula which are so simplified as to be completely useless applied as actual foruma to the real world...but which serve as a tool teaching how certain factors relate to each other.
In other words a mathematical way of teach a certain way of thinking, so that even though the math is purely theoretical, the way of thinking it teaches has practical application.
But to actually create something that could be used to measure real utility in the real world with real gamers...You'd probably win some kind of nobel prize if you figure that out. Utilities been used as a concept in Econ for decades, but I don't think its ever been successfully applied in practice.
I do remember reading some folks who actually came up with an actual formula for measuring personal utility (of questionable usefulness). You could probably do some searches on the topic to find some previous work on the subject to start from.
On 9/23/2002 at 9:33pm, C. Edwards wrote:
RE: An Objective Standard of Quality for RPG's
utility: the quality or state of being useful.
By rating something by utility you are automatically implying how useful it is. If that's not implying a subjective determination I don't know what is.
The only issue with the term "quality" is that people seem to insist on applying the component level definition when it doesn't apply.
quality: degree of excellence; of superior grade
vs.
quality: the totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs.
which is related to
quality: an essential and distinguishing attribute of something.
Utility would not enter the equation until a person actually looked at the quality rating and made their own value judgment as to its usefulness.
I hesitate to use GNS terms as an example, but here goes. A quality rating might look like this:
Game A: Sim 10
Game B: Sim 5/Nar 5
Game C: Gam 3/Nar 7
This would be based on observed behavior in play which was induced by the game and/or the exhibited behavior of players that have a tendancy to play the game. I think both are equally useful for this purpose.
I think some honest description of actual play based on observed behavior would need to accompany this, even if it is just actual excerpts of real play. Descriptions aimed solely towards marketing like that on the back of GURPS would certainly not work.
I know all this repeating of definitions is somewhat obnoxious but I find changing to a less appropriate term because the appropriate definition of another term wasn't being utilized to be rather illogical and perplexing.
Damn, I sound like a Vulcan.
-Chris
On 9/24/2002 at 2:35am, M. J. Young wrote:
I don't know
Which is the better quality tool, the Stanley Claw Hammer or the Craftsman Rip Saw?
Which of them is the more useful tool?
I have been working for several years, on and off, on an approach to doing game reviews. It struck me then, and still seems right now, that the starting point for any discussion of any game or game supplement is this: what is it attempting to accomplish? You must first ask and answer that about your game or product before you can ask any other question, because there are really only two other questions you can ask: how well did it succeed? And was this a worthwhile objective?
Thus if you want to know whether a game is "good" you have to ask these questions. It is entirely inappropriate to judge Alyria based on how well it simulates reality, because it is making no effort at all to do that. It is far more appropriate to ask whether it creates strong character conflicts and leads to their resolution, because that's what it's trying to do. It might also be appropriate to ask whether designing a game that does this is a good thing (I think it is).
This is why objective standards are so difficult to achieve. They assume (as someone implied) that all role playing games are trying to do the same thing. This is as wrong-headed as the idea that all games are trying to do the same thing. A game is better if it makes clear what it is attempting to do; it is better if it does what it intends to do well; a game is better if it intends to do something worth doing, even if it is not entirely successful; it is best if it does something well that is worth doing.
This analysis doesn't even have to be limited to role playing games.
Now, tools such as GNS and consideration of stance and coherence may all be useful approaches to understanding why a game does or does not succeed in reaching its objectives; but the first step is still an evaluation of what those objectives appear to be. Whether this comes from color text, or setting, or author statements, or an analysis of the apparent function of the mechanics, it is the starting point for any other analysis.
--M. J. Young
On 9/24/2002 at 9:05am, contracycle wrote:
Re: I don't know
M. J. Young wrote: but the first step is still an evaluation of what those objectives appear to be.
???
Surely, objective 1: to develop a mini-market foir supplemental products
Objective 2: to provide entertainment to its audience, in pursuit of goal 1
If we consider a game as a product, thats all there is.
If we consider it as a work of art, then "quality" remains subjective and that is perfectly valid. The only true statement we can make is "I liked it".
OTOH: "Which is the better quality tool, the Stanley Claw Hammer or the Craftsman Rip Saw? " is in fact answerable. The one that is better made, shows higher attention to detail, uses more durable material in its construction, could reasonably be said to be of higher quality. This does not make it any more useful than before, however, and so we may find ourselves abandoning the high-quality tool and reluctantly using the low quality, but appropriate, tool. This does not alter the fact that there was a qualitative distinction between them.
On 9/24/2002 at 9:36am, brainwipe wrote:
RE: An Objective Standard of Quality for RPG's
From someone who is trying to put a 'quality' game together, this topic is of near-infinite usefulness and purpose. I am thoroughly enjoying reading the comments posed. I am yet to form an opnion, though.
On 9/24/2002 at 10:07am, Jeremy Cole wrote:
RE: An Objective Standard of Quality for RPG's
Valamir wrote:
In other words a mathematical way of teach a certain way of thinking, so that even though the math is purely theoretical, the way of thinking it teaches has practical application.
While not math based, I think GNS is that. Mind you, economics is hardly maths based, 1st year basically, before you get into the quanititative crap.
Anyhow, when I first joined this sight, I suggested a system should be looked for its ability to support each mode of play, as player's will often drift mid-campaign. I think the extension here is for the review to encompass not what the game is trying to do, but look at what player's might do with the game, and how well they might do it.
If the mechanics support cheese fantasy, how well?
Grim fantasy, how well?
Empire building, how well?
More coming...
On 9/24/2002 at 10:07am, Jeremy Cole wrote:
RE: An Objective Standard of Quality for RPG's
Valamir wrote:
In other words a mathematical way of teach a certain way of thinking, so that even though the math is purely theoretical, the way of thinking it teaches has practical application.
While not math based, I think GNS is that. Mind you, economics is hardly maths based, 1st year basically, before you get into the quanititative crap.
Anyhow, when I first joined this sight, I suggested a system should be looked for its ability to support each mode of play, as player's will often drift mid-campaign. I think the extension here is for the review to encompass not what the game is trying to do, but look at what player's might do with the game, and how well they might do it.
If the mechanics support cheese fantasy, how well?
Grim fantasy, how well?
Empire building, how well?
More coming...
On 9/24/2002 at 10:07am, Jeremy Cole wrote:
RE: An Objective Standard of Quality for RPG's
Vampire is a good example of this. It could be reviewed on;
Powergaming
Tactical combat
Empire building
Story driven angst piece
etc...
The merits of the system could then be looked at with each in mind. Perhaps they don't have to be distinctly put, otherwise everything might fall into the definitionalism GNS talks normally fall into, but the review should consider; 'what campaigns does this game cover, and to what effect?'
Jeremy
On 9/24/2002 at 11:50am, Marco wrote:
RE: An Objective Standard of Quality for RPG's
There's a lot here--but I want to respond to three things that caught my attention:
1. Telling a person how well a game supports a specific GNS mode is probably the least important factor in telling them how well they'll enjoy it (the exception--and maybe that's why it's getting play on The Forge is the hard-core Narrativist who has little use for traditional systems).
Telling me that something is good for Sim-play means nothing in and of itself.
2. Evaluating games as works of art (Contracycle) is valid--but subjective without points of reference (Critics and Cannon). Determining the game's usefulness for "implied" objectives is problematic since it appears to be objective but really isn't (what's the implied objective of Deadlands? Why is that relatively crunchy combat in VtM? Without a riggorus statement from the game designer(s) it's not really possible to say).
3. What I'd like to see is how Contracyle's objective comparison (better materials, more durable, etc.) applies to an RPG experience (not the physical book or PDF or whatever).
What are the elements of the RPG experience that could be used to make a qualitative distinction between one and the other?
-Marco
On 9/24/2002 at 1:06pm, deadpanbob wrote:
RE: An Objective Standard of Quality for RPG's
Marco wrote:
What are the elements of the RPG experience that could be used to make a qualitative distinction between one and the other?
These elements will of course be subjective. But some likely candidates, depending on who you talk to:
A mechanics system that gets out of the way of my immersive roleplaying experience
or
A mechanics system that has lots and lots of detail and options for conflict resolution (combat or not)
or
A mechanics system that is self-consistent and provides consistent results based on character concept
or
A game that encourages and rewards player character cooperation
or
A game that encourages and rewards player character competition
or
A game that encourages and rewards the creation of good stories
All of the above are subjective value statements - and you could probably get a group of people to agree/disagree along some scale with all of them. Then, the problem becomes finding games that support those value statements - because the whole point for me is how do I as a game designer effectively design a game for my target audience (assuming that my target audience should be bigger than just me)?
Cheers,
Jason
On 9/24/2002 at 1:26pm, Valamir wrote:
RE: An Objective Standard of Quality for RPG's
Marco wrote:
1. Telling a person how well a game supports a specific GNS mode is probably the least important factor in telling them how well they'll enjoy it (the exception--and maybe that's why it's getting play on The Forge is the hard-core Narrativist who has little use for traditional systems).
Don't get hung up on this "telling a person" stuff. The objective of GNS is not to go out and proselytize to the masses. GNS is not a dating service where you come to the Forge to be evaluated and hooked up with with the best match from our catalog.
Sans the "telling" part, I couldn't disagree more. How well a game supports the mode that the player prefers to play in is one of the most important factors in how well they'll enjoy the game. This is especially true if you factor out (i.e. control for the variable of) the social issues of enjoying the company of the people you're with...which should be factored out of an analysis because they're true of all human activity, whether its gaming, or going to the movies.
Telling me that something is good for Sim-play means nothing in and of itself.
If you are looking for a game that will allow you to realistically modify ballistic combat, how does knowing whether a particular game does or doesn't do that well mean nothing to you. Modeling reality well, being a sub-set of Simulationism.
How does knowing that Inspectres doesn't model "reality" in detail mean nothing to you if what you're looking for is an actual simulation of paranormal investigation.
I can't understand your point here.
On 9/24/2002 at 2:01pm, Marco wrote:
RE: An Objective Standard of Quality for RPG's
Valamir wrote:Marco wrote:
1. Telling a person how well a game supports a specific GNS mode is probably the least important factor in telling them how well they'll enjoy it (the exception--and maybe that's why it's getting play on The Forge is the hard-core Narrativist who has little use for traditional systems).
Don't get hung up on this "telling a person" stuff. The objective of GNS is not to go out and proselytize to the masses. GNS is not a dating service where you come to the Forge to be evaluated and hooked up with with the best match from our catalog.
This is in the context of a review or a quality assessment. Not in the context of advice. If I read a review and it says "Sim 99%" that purportedly tells me that in studies the players made Sim-type decisions. You suggest this will inform (tell) me how well it'll suit my needs or how much I'll enjoy it.
Sans the "telling" part, I couldn't disagree more. How well a game supports the mode that the player prefers to play in is one of the most important factors in how well they'll enjoy the game. This is especially true if you factor out (i.e. control for the variable of) the social issues of enjoying the company of the people you're with...which should be factored out of an analysis because they're true of all human activity, whether its gaming, or going to the movies.
This is, I think, clearly untrue. That is, it's a falacy: Just because the games I enjoy fall into the super-set of sim-play does not mean that I enjoy any game that falls into the super-set of sim play.
That's where I think the GNS stuff is being mis-applied. Call of Cthulhu, a Sim game, doesn't model ballistics particularly realistically--that has nothing to do with my enjoyment of it. It doesn't simulate "realisitc" paranormal investigation. It doesn't do anything like that--VtM is a sim game with what, emphasis on color or something ... I found the aggravated damage rules (part of the simulation) fatal to my enjoyment of it (rendered in GURPS, that is, but I found the char-gen--another part of the simulation--flawed in Story Teller).
Morrow Project has *great* ballistic rules, it's a really good simulation of reality (mutations: "leukemia")--but it wasn't even my top choice for post-app play once Hero got going. Both are sim systems. Both would score highly.
The difference is far more profound than GNS mode.
In short, no--how well a game supports my "mode" is probably one of the least important aspects as to how well I'd enjoy it.
I can't understand your point here.
Maybe that's because you're ascribing too much importance to GNS.
-Marco
On 9/24/2002 at 2:09pm, Jeremy Cole wrote:
RE: An Objective Standard of Quality for RPG's
deadpanbob wrote:
All of the above are subjective value statements - and you could probably get a group of people to agree/disagree along some scale with all of them.
Why try for an absolute statement? Assess what each design is good for. 'This game effectively supports team oriented play, and its combat system is designed for heavy strategy play, and models in great detail a possible 34th century style of warfare' or whatever. Then assess against those objectives.
Come to think of it? Why define the elements, and why quantify things? There will always be common use definitions, and game plays forms, but always leave it open to say - this attempts something done nowhere else.
Rather, form a procedure for reviewing. How can this game be played? How well does it facilitate such play? How does it compare to games that attempt the same things (if there are any)?
In this way, a game is assessed in a qualitative fashion, and relayed as such to other persons, they gain a real understanding of the game.
Jeremy
On 9/24/2002 at 2:26pm, deadpanbob wrote:
RE: An Objective Standard of Quality for RPG's
nipfipgip...dip wrote:
Rather, form a procedure for reviewing. How can this game be played? How well does it facilitate such play? How does it compare to games that attempt the same things (if there are any)?
In this way, a game is assessed in a qualitative fashion, and relayed as such to other persons, they gain a real understanding of the game.
Jeremy,
This only helps me if I've played the other games that the one in question is compared to.
The reason to think about 'quality' or 'utility' etc. at all, for me, is because as a potential game designer, I'd like to be able to more realistically, and to borrow Ron's term, rigorously design a game that will satisfy my target audience.
Cheers,
Jason
On 9/24/2002 at 3:15pm, Valamir wrote:
RE: An Objective Standard of Quality for RPG's
Marco wrote: Maybe that's because you're ascribing too much importance to GNS.
-Marco
Actually, its quite the opposite. I think you are ascribing too much importance to it. In the sense that you are expecting something as simple as identifying whether or not a game is Simulationist to answer all of your questions about what the game is like and whether you will enjoy it.
Since it can't do that, you're inclined to dismiss the theory as some nice talking points, but not real helpful. But I think you're expecting far too much from what is essentially a very simple concept.
When I said it was one of the most important items that does not mean, and should not be taken to mean. "Marco likes Sims. Game X is a Sim. Therefor Marco will like Game X" Nope...
What it means is "Marco is looking for a Sim Game to play ABC with. Game X is a Narrativist game about ABC. Marco has been playing ABC with Game X and is pretty disatisfied with it. Perhaps this means that Marco would enjoy himself more if he found a different game on ABC which better supports his Simulationist goals. Quite likely Marco has already begun thinking of houserules and variations that will begin to move the game in that direction already."
Further what it means is "Ralph is looking to design a game about ABC, and hopes to design something that will appeal to Marco and gamers like Marco. Ralph knows that what Marco et.al. is really looking for in a game on ABC is something that emphasises various simulationist aspects of ABC like 1, 2, and 3. Armed with this knowledge (or more generally, a generic target marget), Ralph can take the principles that have been built up on a foundation of GNS use them to help determine what sorts of mechanics are most and least appropriate for the type of game Ralph is seeking to design".
Or it means "Marco just played Game X. Marco did not enjoy it at all. It was one of the worst experiences of Marco's gaming career. Marco was planning on writing a review for RPG.net on Game X, but unlike a large number of RPG.net reviewers, Marco is not satisfied with simply writing "Game X sucks and is a total piece of crap" and calling it a capsule review. Marco knows that the reason he didn't enjoy the game has little to do with the quality of the game, and everything to do with it not being his cup of tee. Armed with GNS theory, Marco is able to identify the sort of game, that Game X is and evaluate how well Game X fulfills its goals to be that sort of game. The lexicon of GNS comes in handy as Marco prepares a review evaluating Game X on its own merits, even though he didn't particularly enjoy it.".
"Ralph recognizes that some games and some game players are like oil and water and just don't mix. Ralph knows its tradition for one group to label the other group as munchkins, or bad roleplayers, or story-nazis and so forth. But Ralph knows players from different groups and doesn't think this is the reason. Ralph finds GNS. GNS helps identify why different players can't stand to be in the same game together even though they both love RPGs. GNS helps illustrate why these different preferences for play are all legitimate gaming practices and not examples of "bad roleplaying". Armed with this knowledge, Ralph can even enjoy playing a game of D&D with a group he previously couldn't stand playing with, because he's learned to appreciate games for what they do and what they are. Further, the rigorous thought that has gone into trying to understand GNS and its corollary theories has opened up a whole new world of roleplaying possibilities. Not necessarily "thinking outside the box" but clearly illustrating that the box is a lot bigger than he thought it was. And so Ralph's roleplaying and game experiences, and game design efforts have been improved and made better by discussing GNS...despite it not being perfect. Even though there are still large chunks of it Ralph might disagree with, its served its purpose, and this is a "good thing".
Sorry, for coopting you into my stories, but these are the areas that GNS is for. And areas that it performs well in. Trying to apply the theory beyond these areas is certainly not forbidden...but it does void the terms of the warranty...if you follow me.
On 9/24/2002 at 3:47pm, Jeremy Cole wrote:
RE: An Objective Standard of Quality for RPG's
Hi Jason,
This only helps me if I've played the other games that the one in question is compared to.
Because if you talk about what a design can be used to do, then you get the various elements of utility. It doesn't matter if you have played something or not. If you read a statement about a couple of fantasy games, 'Y has combat with more tactical options, X has more realistic combat', you don't have to have played each game to know a bit more about each, and you can assess the games lot better than an overall apples and oranges rating.
The reason to think about 'quality' or 'utility' etc. at all, for me, is because as a potential game designer, I'd like to be able to more realistically, and to borrow Ron's term, rigorously design a game that will satisfy my target audience.
The working definition of utility I was taught in Economics was the ability of a good to meet the multiple needs of an individual (yes, one individual wants more than one thing out of an item).
Attempt to find the multiple needs of your user group. Then objectively state what you want, and compare it to other games with similar design elements (for comparison and theft).
Jeremy
On 9/24/2002 at 4:18pm, contracycle wrote:
RE: An Objective Standard of Quality for RPG's
Marco wrote:
3. What I'd like to see is how Contracyle's objective comparison (better materials, more durable, etc.) applies to an RPG experience (not the physical book or PDF or whatever).
Well we are talking about RPG design, so we are necessarily discussing something that is an object to some degree. The only practical reccomendation I could make in regards system would be a reference to another such object.
On 9/25/2002 at 3:55am, M. J. Young wrote:
RE: Re: I don't know
contracycle wrote:M. J. Young wrote: but the first step is still an evaluation of what those objectives appear to be.
???
Surely, objective 1: to develop a mini-market foir supplemental products
Objective 2: to provide entertainment to its audience, in pursuit of goal 1
If we consider a game as a product, thats all there is.
If we consider it as a work of art, then "quality" remains subjective and that is perfectly valid. The only true statement we can make is "I liked it".
If I read the first page of the Introduction chapter in Multiverser, I quickly learn that the game sits on these principles:
--all imaginable universes exist
--player character travel from universe to universe
--rules change to some degree to match the universes.
It doesn't take too much intelligence to conclude from this that the objective of the rules is to provide a game system that will support any imaginable type of adventure in any imaginable type of world, and move player characters from one such world to another in the course of play. This being the apparent objective of the game, the next question would be whether it accomplishes this.
That was too easy; I wrote that. Let me pick on Little Fears. I happen to have a copy right here in my room, and I happen never to have seen anything more of the interior than the character sheet. My second son was going to run the game, but my wife nixed it because she thought it too frightening for the younger members of the household gaming group. So I turn to the introduction, and I find a somewhat impressionistic bit of writing; yet I cannot avoid coming away from this with the idea that Jason Blair wants me through his game to remember what it was like to be a frightened child. Maybe there's more that I would discover, but this seems to be the objective. Not having played or read it, I can't tell you from my experience whether it accomplishes this--but that would be the next question. I might also ask whether such a game needed to be written; but that's a value judgment that tells more about me than about Jason's game in this case. It would be different, perhaps, if there were hundreds of games in which we played frightened children; in that case, we might argue that as good as this one was, it didn't add anything to the corpus. That is not the case.
I've also got a copy of Pendragon (4th ed.) here, which I started reading some time ago and set aside only to forget. But as I re-read the text on page 4, Welcome to Pendragon, I find the idea clearly presented that the game hopes to recreate adventures akin to the literature surrounding Camelot and King Arthur--not by any means the reality of that age, but the age as recreated centuries later. How well it does that is the question to ask. It would not be appropriate to ask, for an extreme example, whether the game engine could be successfully used to model space pirates in a world similar to that of Star Wars. The intent of the product is fairly clear.
Dungeons & Dragons third edition Players Handbook is a bit less clear in its introductory text as to what it is trying to achieve. The most I can conclude from the opening section is that it wants to make possible a variety of adventures in a somewhat non-specific fantasy world. Perhaps I could find the objective by doing a bit more reading, but not tonight. And perhaps this is an example of a game whose objectives are not very clear, either in the way they are stated or in the means by which they attempt to achieve them. That doesn't alter the reviewer's obligation first to attempt to understand what the writers wanted the book to do, and to evaluate it in that light. Perhaps in this light what they wished to do was create a new game in the tradition of an earlier one, and the question becomes whether they managed to improve the old game while maintaining its feeling and traditions. At this point, we might have very different opinions not because we disagree as to whether it meets its objectives, but because we disagree as to what those objectives are.
The same principle would apply to game materials that are not rule books. Multiverser: The Second Book of Worlds begins its introduction with "More of the multiverse is revealed within these pages"--and I don't think it an incredible leap to conclude that the purpose of this book is to provide settings and adventure opportunities for use with the Multiverser game system. The question then is whether it provides settings and adventures that work well within that system and provide for entertaining games.
I could look at other games, but I suspect the ones that would give me the best examples don't happen to be in my room. I've not read the introductory text for Sorcerer, but I would wager that it does a good job of expressing its purpose. Sometimes figuring out what a game or supplement is trying to do might take a bit more work.
And no matter how much better made the Craftsman Rip Saw might be, the Stanley Claw Hammer will always be better for driving nails. You need to evaluate their quality in the context of what they were made to do, not in the context of what you wanted to do with them.
--M. J. Young
On 9/25/2002 at 5:42am, C. Edwards wrote:
RE: An Objective Standard of Quality for RPG's
M.J. Young wrote:
And no matter how much better made the Craftsman Rip Saw might be, the Stanley Claw Hammer will always be better for driving nails. You need to evaluate their quality in the context of what they were made to do, not in the context of what you wanted to do with them.
I know this may seem like a minor distinction, or possibly no distinction at all, but I think the quality of an RPG has to be evaluated in the context of what it actually does, not what it was made to do.
Why? Because I see game design as more akin to alchemy than chemistry. What something was made to do isn't necessarily always what it ends up doing. Just because an RPG doesn't end up doing exactly what it was made for doesn't mean that what it is doing doesn't work.
-Chris
On 9/25/2002 at 5:52pm, Seth L. Blumberg wrote:
RE: An Objective Standard of Quality for RPG's
M. J. Young wrote: And no matter how much better made the Craftsman Rip Saw might be, the Stanley Claw Hammer will always be better for driving nails. You need to evaluate their quality in the context of what they were made to do, not in the context of what you wanted to do with them.
Actually, I think that if I were looking for a tool to drive nails, I would have the sense to say "I need a hammer," and look up reviews specifically of hammers, ignoring any reviews of rip saws that might be present. In that context, what I want is a review which exposes the craftsmanship and construction of the hammer--not a review which discourses on its greater suitability for driving nails than for trimming 2x4s.
On 9/25/2002 at 6:51pm, Mike Holmes wrote:
RE: An Objective Standard of Quality for RPG's
I'm not sure if you guys are getting MJ's point. He's saying that the comparison that one needs to make to determine quality is to compare the design intent with the actual play. That's his first and second questions, you'll note.
To do otherwise would be to use the saw to try to nail in a nail, to fail (in play as it were), and to conclude that the saw was a terrible hammer. So it is. But then it wasn't designed to be a hammer, so the comparison is a bit senseless.
In this way, Seth and Chris, you are agreeing with MJ, I think.
Mike
On 9/25/2002 at 7:17pm, C. Edwards wrote:
RE: An Objective Standard of Quality for RPG's
I'm not sure if you guys are getting MJ's point. He's saying that the comparison that one needs to make to determine quality is to compare the design intent with the actual play. That's his first and second questions, you'll note.
Maybe from the designer's point of view, but certainly not from the player's perspective.
(why does this feel like the GNS thread about intent and behavior?)
Let's take a trip on down to the Home Depot. We are standing in the tool section looking at hammers. Hammers of different weights, hammers with smooth heads and waffled heads, hammers with straight claws and curved claws, sledge hammers, framing hammers, hatchet hammers, ball-ping hammers, a crap load of hammers that are similar and different in various ways. Design intent and success are irrelevant to the consumer. Just tell me what hammer works best for my specific need.
For RPGs the consumer needs to have an honest understanding of their play needs and any RPG product would need a categorization, a list of specs, that is gathered through actual play. Fortunately design is often close to intent, but since people hell-bent on making a profit often refuse to be completely honest about their products, with themselves and consumers, the only way to provide any useful guildline of quality for the consumer is through rigorous third party "road-tests". Something similar to a Consumer Reports Digest I suppose.
Note that while I'm not against making a profit, I am against doing so at the expense of honesty with the consumer.
I think all the analogies we've been using are starting to give me a facial tic.
-Chris
On 9/25/2002 at 8:39pm, Mike Holmes wrote:
RE: An Objective Standard of Quality for RPG's
C. Edwards wrote: Fortunately design is often close to intent, but since people hell-bent on making a profit often refuse to be completely honest about their products, with themselves and consumers, the only way to provide any useful guildline of quality for the consumer is through rigorous third party "road-tests". Something similar to a Consumer Reports Digest I suppose.Well said. How often are RPGs billed as the "One True Game" that will solve all your RPG woes. To extend the analogy (and further exacerbate Chris' tic), it's as though they made a saw and said, this tool can do any home repair imaginable.
Occasionally they get something as "complete" as a saw/hammer hybrid tool, but it still fails to be a wrench.
Question is do reviewers take this view? Or do they assume that since they need a hammer, that everyone else does as well? How do we get that idea across to RPG reviewers?
Mike
On 9/25/2002 at 10:42pm, C. Edwards wrote:
RE: An Objective Standard of Quality for RPG's
Question is do reviewers take this view? Or do they assume that since they need a hammer, that everyone else does as well? How do we get that idea across to RPG reviewers?
As I see it, the best option is to have an orginization, commitee, whatever, with a standardized set of review procedures and ratings to be utilized by a disciplined and well trained stable of reviewers that have a wide variety of play styles (acknowledged dubious term) and interests between them.
Each product could go through a gauntlet composed of actual play by different reviewers. The reviewer would take their own data, along with feedback gathered from the other participants through observation and direct questioning and submit it to be averaged in with data from other playtests. This would all hinge on a standardized data gathering protocol, including a set of well honed questions to apply to the game participants.
The result would be a set of specs that would suggest the quality (ability to fill specific needs) of a game based upon the rigorous testing and disciplined review it recieved. I think that would be the optimal situation.
As it is, the only "reviews" I put any stock in are the word of mouth recommendations here on The Forge and those in the Reviews section.
Yay! Not one analogy.
-Chris
On 9/26/2002 at 12:55am, Jeremy Cole wrote:
RE: An Objective Standard of Quality for RPG's
I know this may seem like a minor distinction, or possibly no distinction at all, but I think the quality of an RPG has to be evaluated in the context of what it actually does, not what it was made to do.
Yes, this is what I have been saying. AD&D is the classic example, while other threads have shown it can be used with a narrative focus, it really does functionally support gamism play best. To review the game and state that it was designed as an all three hybrid, and play attempts in this way collapse, is to ignore that people like this game. However, to say that it mostly supports gamism, and then review on these grounds alone, is also to ignore a lot about playing the game.
To say that people could concievably use the system for ... and on these grounds it works ... , would I think, give a better review system.
More coming...
On 9/26/2002 at 12:55am, Jeremy Cole wrote:
RE: An Objective Standard of Quality for RPG's
I think what is needed is a system of reviewing the varying elements of the mechanical system, from character creation to mechanic resolution, and special rules and whatever else. These mechanics could be assessed on how well they meet different methods to play the game. Such a review system could be very helpful for potential players and the designer alike. "Experience is based on kills totals, giving incentive for gamism priorities" or whatever. Here people could take in the statements about the game, and assess it to their own play style.
Jeremy
On 9/26/2002 at 3:04am, M. J. Young wrote:
Hammers
Chris Edwards wrote: We are standing in the tool section looking at hammers. Hammers of different weights, hammers with smooth heads and waffled heads, hammers with straight claws and curved claws, sledge hammers, framing hammers, hatchet hammers, ball-ping hammers, a crap load of hammers that are similar and different in various ways. Design intent and success are irrelevant to the consumer. Just tell me what hammer works best for my specific need
All right, so let's look at hammers. My father had a sledge hammer, a ball peen hammer, a claw hammer, a tack hammer, and a mallet.
You could, in theory, drive nails with any of these. But if you use the sledge hammer, you're probably going to hurt your fingers--because even though you could probably put the nail in the wood on a single stroke, you can't easily manage the hammer with one hand while holding the nail with the other, and you're going to hit your hand. The tack hammer would also do the job, but it's light, and the amount of force it will transfer to the job is going to make it take forever. The mallet might not even do the job; you're going to do a lot of damage to the mallet trying to drive the nail, although eventually you would probably succeed. Only the ball peen and the claw hammer are really designed for the job. Ah, but the claw hammer has that extra feature: if the nail doesn't go in right, this is the tool for pulling it back out again. It is designed for the job.
But if you want to drive railroad spikes into the ground, or force wedges into cut tree trunks to split for firewood, the claw hammer just doesn't have the heft. You want a sledge for that.
And your claw hammer will probably flatten tacks and brads so quickly there won't be anything left of them, not to mention that the heads on many of these are too small for the claw to grasp. A tack hammer is much better for that sort of application.
And if you're wanting to snug up the joints in furniture, that claw hammer will leave unsightly marks on the surface which you might well avoid by using a mallet.
And the ball peen is a much better design for working with sheet metal. It has the extra heft and the shaped head.
So which hammer is best for the job? It apparently depends on the job. But saying that any one of these hammer is no good because it's not a claw hammer is missing the point. And saying that Pendragon is no good because it doesn't convert easily to space opera, or Little Fears is no good because the combat system fails to account for the potential damage of LAWS rockets on closetland monsters, or Axis and Allies is a terrible game because it doesn't give you any insights into the personalities of the commanders, is equally missing the point. You have to judge the tool based on its intended use, not on your particular needs or preferences.
Now, as for the distinction between what the game designers intended to create and what they actually created, that is a much more difficult question. That sounds a bit like saying "these guys accidentally created a good game which was not the game they intended to create". I'm not certain what to make of this. DeadEarth is a horrible game, but the depth of medical information in its disease section is worth the price? It calls to mind images of Springtime for Hitler from The Producers, when the characters have attempted to produce a broadway flop and wind up with a runaway success comedy. I am mindful of all the "one hit wonders" of the '60's, musicians who had one single rise to the top of the charts only to vanish forever from the music scene thereafter. Praising a game because it was a fortunate mistake seems a silly notion.
On the other hand, I have discovered a number of good aspects of Multiverser that I had not realized were there when I was writing it. It may be quite common for good games to succeed because the writer did something to address one concern and wound up opening an entirely unexpected possibility in play that made it a better game. So I can't argue it would never be right to praise a game for being good in ways never intended. I just think it would be wrong to condemn it for not being good in ways never intended.
--M. J. Young
On 9/26/2002 at 4:02am, C. Edwards wrote:
RE: An Objective Standard of Quality for RPG's
So which hammer is best for the job? It apparently depends on the job.
Exactly.
M.J., you keep using words like "good", "praising", "condemn" and "judge". I'm not proposing a system to tell people how "good" a game is. I'm proposing a system that will supply data as to how a game functions in play so the consumer can better judge for themselves how good a game is before its purchased. This is really no different than the system used to rate automobiles, but since everybody pretty much wants a vehicle that hugs the road, gets them wherever they're going in style, and will keep them alive in a multi-car pile up, the distinction between function and quality has been nearly erased in that arena. When consumers exhibit a wider variety of needs that distinction becomes much more apparent.
-Chris
On 9/26/2002 at 4:12am, Jeremy Cole wrote:
RE: An Objective Standard of Quality for RPG's
Now, as for the distinction between what the game designers intended to create and what they actually created, that is a much more difficult question. That sounds a bit like saying "these guys accidentally created a good game which was not the game they intended to create".
But there remains the big point that published design intentions function more as advertising than they do as actual guides to gameplay. People at my old highschool, although I try not to talk to them, love crunchy vampire, beating down on humans and other lessers. The stated design intentions don't show this. Show me anywhere in AD&D 2ed that says
'munchkin fest', but people love playing this way.
More to come...
On 9/26/2002 at 4:13am, Jeremy Cole wrote:
RE: An Objective Standard of Quality for RPG's
Axis and Allies is a terrible game because it doesn't give you any insights into the personalities of the commanders
Axis and Allies is a great example of what I'm getting at. If you read the hype (or design goals) you'd think you were playing a realistic representation of WWII. The best way to win is for Japan to March infantry and tanks over Siberia. They couldn't even draw the world right, New Zealand is shown as one island. Its ridiculous.
But as a lighthearted game, this game is so simple we can do three other things at the same time and still follow everything that's going on, as a beer and pretzel game it holds up. If you understand that giddy thrill of rolling 74 D6s, its fun. When suggesting to friends they might want to play Axis and Allies, I mention that its fun on these levels, I don't tell them about any fictional design goals.
It wouldn't help my friend's understanding of the game, so why should we think any differently when preparing a General Purpose Review?
Jeremy
On 9/26/2002 at 6:25pm, Seth L. Blumberg wrote:
RE: An Objective Standard of Quality for RPG's
I believe that, apart from issues of how compatible a game is with any given play style, there are craftsmanship issues which are broadly applicable across all games, and go deeper than "Are there a bunch of typos? How many Page XX references are there? How good is the index (if there even is one)?"
I am not prepared to describe any of them specifically at this time, however. I'll get back to you on that.
Furthermore, I believe that it is entirely possible for a game to succeed in a different play style than its authors intended. Amber Diceless RPG is one example that comes to mind.