For those of you who don't know, I am a Eurogames nut.  These are the new wave of boardgames, mostly from Germany, that are breathing new life into the boardgame market.  The most prolific designer of these games is Dr. Reiner Knizia.  He is responsible for games like Tigris & Euphrates, Ra, Blue Moon, and the Lord of the Rings cooperative boardgame.  In my humble opinion, these are some of the best games currently out there.  The designs are tight, both mechanically and psychologically.  Knizia has demonstrated repeatedly that he understands the intersection of people and systems to produce a quality game experience.  Although he is not designing roleplaying games, I believe that we can all learn a lot from his methodologies and designs.

As such, I am listening to his Q&A session here with a great deal of interest and enthusiasm.  In particular, Knizia discusses playtesting, which he says is the most important part of the design process.  He states that it is impossible to understand how a game will work by just creating the rules and thinking about how you think they will work.  Game rules are about the creation of a certain experience, and you cannot understand the experience without actually playing the game.

This leads me to a rant.  Ready?

Our playtest skills suck.

Allow me to expand on this a bit.

As game designers, we are after enabling the creation of a particular experience of play.  That's the whole point of game design.  That's why System Matters.  We are all about the intersection of systems and people.  That's what drives many of our standard questions for newbies, right?  "Tell me about an experience of actual play that you liked."  "Tell me about how you want your game to feel."  What we mean is, "What experience are you trying to create?"

But, are we sure that our games are actually creating these experiences of play?

Let me ask a related question.  Are we making sustainable games?  By this, I don't mean "games that can have expansions".  Rather, how quickly will your game wear out?  Will it be fun for one or two sessions?  Or is it the kind of game that could be played again and again?

Again, the parallel to Eurogames.  Increasingly, I'm finding that companies are producing games that are fun for a while but quickly lose their luster.  Think about the paperback novel at the grocery store. Read it once, then throw it away.  Disposable entertainment.

But think about games like Chess or Go.  People have dedicated their entire lives to delving the depths of these games, because there are depths to delve.  These are more like those treasured books like The Lord of the Rings (if you're a Tolkien nut) or The Book of the New Sun (if you're a Lupine) that you take from the shelf once again, preparing to have your mind expanded by what lies within.

What about us?  Are our games just the "flavor of the year"?  Or are we actually about making games with depth and longevity?  If you are, you will not be able to achieve this goal without long-term, consistent, disciplined playtest.

So, for example, I'm working on my next game project, and I'm pretty sure that I have an alpha just about ready to go.  My goal is to playtest it twice a week for the next several months, continuing to hone it and shape it while I begin to form the manuscript.  Will I be ready for GenCon?  Maybe GenCon 2008, but that might be rushing it.  I want it to be perfect, and perfect can't be rushed.

If you think that's hardcore, I'll point out that Knizia has playtest groups running every single night.

So, when is the game done?  Knizia said that he knows a game is ready when the rules stop changing in between playtests.  For him, that's generally nine months of full-time work.  Very few of us are full-time designers.  Are you prepared to test your game until it's ready?

So, here's the gauntlet.  I'm calling all of us to a higher standard.  If we're serious about being game designers, that means that we need to throw down and do the work.  Having a kewl idea is a great place to start, but commit yourself to playtesting until it is perfect.

Seriously, perfection is a misguided goal.  I agree that you can map roleplaying game design to board game design pretty closely in spots, but there is not a one to one correspondence.  For one thing, there are creative dependencies in a roleplaying game that don't exist in a board game. 

I think if your game is playable and fully baked, if it has been responsibly and zealously playtested and actually works and brings the fun, then you ought to cut the cord.  Contrast this with publishing a game with broken rules, or publishing a game that you've only tested with your pals.  Contrast this with the game that is "not perfect" and stays on your hard drive  forever.

Perfection is a long, long wait away, and I fear that it can turn into a comfortable excuse for never finishing a project. 

Hey, Jason!  Thanks for the reply.  I was starting to become concerned that someone had activated my "Inability to Annoy" Trait.  I try to come up with a solid rant, and they just never work....

Anyways, to your comments.  I'm actually going to start at the end.

Perfection is a long, long wait away, and I fear that it can turn into a comfortable excuse for never finishing a project. 

Good points, Seth.  It'd be useful to set benchmarks somehow, even if they were pretty general, based on the direction and complexity of individual games.  We should talk about those benchmarks.  "No changes between playtests" seems like a logical indicator of a fully baked project.

The notion of methodically seeking out playtest groups that fit particular patterns seems like an insurmountable obstacle - honestly, I'm embarrassingly grateful for any blind testing I can arrange. 

Jason wrote:
Good points, Seth.  It'd be useful to set benchmarks somehow, even if they were pretty general, based on the direction and complexity of individual games.  We should talk about those benchmarks.  "No changes between playtests" seems like a logical indicator of a fully baked project.

The notion of methodically seeking out playtest groups that fit particular patterns seems like an insurmountable obstacle - honestly, I'm embarrassingly grateful for any blind testing I can arrange. 

I playtest games until I don't have to change the rules after a playtest session.

This takes me usually 5-6 playthroughs, often a few more.

Why so less than Knizia?  I imagine it's because, compared to a Knizia game, the mathematics of my games are quite simple and the mechanical processes just less complicated.  There is, quite simply, a lot less to playtest, and a fair amount of preliminary testing can be done just by analyzing the rules inside my own head.  (Finalized testing of course requires people but the point is that I'm often part-way there with just the maths.)

If I could playtest every night, I would, of course.  But I'm not Knizia, so that's a bit of a hardship.

yrs--
--Ben

OK, Seth, you're on.  Grey Ranks has been in on-and-off development since October 2005.  Mostly off until the last six months.  Here are my playtesting approaches, in descending order of quantity:

I have a little brain trust I bounce ideas off, and much "testing" goes on by proposing and challenging design goals and assumptions via email and face to face.

I've had a number of mechanical playtests, that looks like this:

1.  Me sitting at a desk running the game's numbers, start to finish or maybe just to test a particular portion over and over and over again. 
2.  Me and a couple of friends doing the same thing, until things utterly break.

I've run the game as a one-shot (using various points of attack) for friends and fellow travelers:

1.  In my living room or at an apartment con with pals.
2.  At conventions for interested strangers.

I've solicited semi-blind tests (I'm available to clarify and answer questions before the actual session):

1.  With widely distributed play groups.

I'm running the game absolutely straight, as intended, for my weekly game group.  We start tonight, as a matter of fact.  I'm strictly an observer for this - my four gamer buddies are playing and I'm watching and listening. 

So... the first six editions of D&D were playtest editions, right (as opposed to the simple gross commercialism of the last couple)?

Is there a sweet spot on playtesting a RPG (between too little and never publishing)? Yes, of course. Do we really miss it by much? I'm not so sure.

Maybe when Ralph and I were playtesting Universalis we got lucky. But the rule did stop changing after only a few tries. It got to be what it was going to be and screwing around with it... well it wasn't actually making the game better in any way we could discern. So we published it.

On the subject of longevity... aren't some of those flavor of the day games from Reiner? He didn't invent chess. I have to question your assumption in four ways:
1. From an economic POV, the incentive has to be to create flavor of the day games. Call me a crass capitalist, but what's the incentive to create the game-to-end-all-games? Assuming it could be done?
2. Why is playing the same game over and over automatically superior to playing different games?
3. I think that many games that folks around here have made turn out to have a lot of longevity. I come back to a title like InSpecters or My Life With Master over and over.
4. But even those I come back to are often written to be short-form. MLWM has an endgame. It actually does me the favor of not requiring it to be run for more than a few sessions at a time so I can get to other games. No, I don't run it over and over again, one full game following the next. But that's not the same as saying that I've gotten tired of it.  

I think that you may be mistaking enthusiasm to try a ton of novel games as having gotten bored with the old ones. There's simply not enough time for anyone to play everything put out there.

Now does that mean that there's no time for the "long-form" game any more? Well, for my part I have been playing HeroQuest weekly now since it came out (Hero Wars before that). In addition to sampling all sorts of other games. And it only gets more and more interesting the more I play. Like chess.

Now, Laws isn't who you're talking about here, but I don't think that HQ is all that superior to other designs around here (just happens to push my buttons). I could as easily be playing a long-term game of Burning Empires.

If you want to turn on the "Annoy" function of the rant, then step on up and make this a clear accusation: name a game that suffers from lack of playtesting. Which ones are they? Who is making the mistake?

If all you're saying is that one should try hard to get enough playtesting in... well who could argue? If you're saying that people actually are not doing it, and it's leading to poor games, well you'll have to give examples to convince me that it's some pervasive phenomenon here. Are there some few with this problem? Oh, probably. But I'm fortunate enough not to have run into them in play, I guess. I don't see it as pandemic.

Saying this just so that you can perhaps see a counterpoint to this discussion: what if I were to suggest that you were proposing this just to support your own historical speed of design which resulted in Alyria taking... how many years? Not only is it important to finish the game (which you have), but its important to do so in a timely fashion, and not let doubt of imperfection cause you to call for another round of playtesting.

I think that nine months of work is about what games around here tend to get between design and full publishing, actually. Are you suggesting we need years more than Reiner does?

Mike

Hey Mike!

I'll admit that pointing past GenCon 2008 as my own deadline might be an overstatement.  Personally, I'd like to have a playable beta or ashcan version of Dirty Secrets ready to go by GenCon this year.  However, I'm willing to let that go if the game isn't ready to go

I'm not calling for "the game to end all games".  I'm actually calling for games like Inspectres or MLWM, which do have a lot of potential depth and longevity.  This isn't a "long-form" vs. "short-form" debate.  (I actually prefer short-form games anymore, since I don't really have the time or desire to play long-form games.)  However, that depth and longevity doesn't just "happen"; rather, playtest will reveal if it is there or not.  Also, playtest will reveal if the features that you think are there are actually there.  This will affect long-form games more than short-form games, though, as a cycle of play tends to be longer in a long-form game.

Hey, I'll gig myself on this one.  My first game Junk was woefully undertested, especially in terms of costing for various bits of equipment.  The only reason that no one has accused it of being horribly broken is that there aren't enough people who know about it

Oh, and justifying the length of time on Alyria?  Heh.  Not even a bit.  Most of that delay was the result of Alyria sitting around doing nothing, not any sort of focused playtest effort over all that time.  It wasn't even a matter of "fiddling until it's perfect"; it was a matter of it's simply gathering dust.  Actually, part of this thread is borne out of my desire to be a bit more disciplined and snappier in my getting the next project done without giving up on quality.

GreatWolf wrote:
I also agree that roleplaying and boardgame design are not exactly the same, and we probably would point to similar reasons.  For example, there tends to be a lot more "openness" in an RPG that needs to be managed, whereas boardgames tend to be more strictly bounded.  As a result, there's a bigger impact of player personality on an RPG.  But, at the same time, I would argue that this calls for more playtest than a boardgame.  Are we playtesting our rules with different personality types?  Are we creating a positive creative space with explicit expectations for all participants?  The very openness that gives RPGs their strength can be their downfall.

Guy,

I have no argument with this.  Even at the design stage, it is usually obvious that certain types of people will be interested in your game, and others won't.  Trying to persuade people who won't be interested to playtest your game is usually an exercise in futility.

Just to build on the point, though, there can be border cases.  Sometimes you don't know what people will trip over, which is why it could be productive to expand your playtest pool.  That way, you can decide if you need to do a better job of explaining certain things (particularly on the Social Contract level), or if you are going to choose to exclude certain people from your core audience.

A good example of this - a playtest group for Grey Ranks that had serious issues with the game's opening, which includes an extended mission in which each player frames a scene for which success is assured.  They really don't like this, and have told me so.  I've been concerned with the gradually escalating difficulty (which makes success easy at the beginning and super hard at the end of the game), and sort of designed around this by removing pointless die rolling for the first mission. So these guys approached it from a different angle than I did, with different assumptions, and their result was different from that of other playtest groups, and their negative feedback has been really useful to me in assessment.  I didn't expect it and I didn't plan for it in organizing playtests, though. 

I've been fortunate enough to be heavily involved in playtesting both boardgames and some big production RPG products. They are very different beasts, and the playtesting approach surely bears that out. Boardgame playtests seem to be much more objective and after the question "is it broken" first, then "is it fun" second... RPG playtesting often reverses these two goals -- "is it fun" first and then "does it mechanically work" second.

I think this is often based on the age old adage about RPGs in general - they are a set of rules that are completely up to you to embrace or change. The GM is always right. If you don't like a rule, ignore it, etc.

Just my observations... As such, I think RPG design/development is more art than science. An artist can't paint and paint and paint forever - eventually you need to accept the creation for what it is and hope others appreciate your perspective, insights and contribution.

I can take this need for more play testing to heart. I never got to play test Panty Explosion as much as I wanted (although we did test it a fair bit). I'm currently having trouble fining people to test The Year We All Died, and I'm determined to test that as much as I can before I release it.

I almost knocked Knizia over in a restroom at Gamestorm last year. Totally slammed right into him coming out of a stall. He was there running play tests and stuff and Christian, Travis and I were a few booths down doing demos. As I was washing my hands I told him how much I liked Samurai and that I had recently played it with my 7 year old niece and my mother. He said something like, "that's why testing is so valuable, so everyone can enjoy the game". Then he went off to use the toilet.

True story.

Jake

I'm not sure if this is the right place to post this, but this is why I think hosting PBP (play-by-post) in the actual-play or playtesting sections of this forum is a good idea. I mean, not everyone has access to a group of dedicated playtesters right outside their door ;)
Sure, I could just look around for other forums to run a playtest session online (or participate in one), but wouldn't it be good to do it HERE as well, with such a knowledgeable community at hand to help interpret the results and give advice?

One of the obstacles to PBP play testing might be that people are willing to have others participate in their play tests, but aren't necessarily willing to participate in others.
Perhaps a general rule of thumb would be to invite others to your PBP with the notion of participating in their PBP sessions in exchange?

"that's why testing is so valuable, so everyone can enjoy the game"

The gauntlet was thrown down in 2007.
I'm curious whether your game was ready for Gen Con 2008.

After re-reading through this post (and reviewing the game design video from Will Wright), I might come to a controversial statement.
But I'll only say it if we hear about whether the gauntlet challenge is still there.

Hello,

I assume you are talking about Dirty Secrets. If so, then yes, the game was published for that GenCon and it has proved itself multiple times as a well-playtested design and well-written rules-explanation.

However, this thread is not an open discussion topic. I recommend deciding on which forum is best for the point you'd like to make and starting a new thread there, linking back to this one. Please send me a private message if you want help with that decision.

Best, Ron

Thanks Ron. I must have missed the game's name in the post. I'm reading its playtests now.

TANGENT: It's interesting how you bring Eurogaming into the equation.  The hobby has changed my tastes in games (even narrowing down my scope of indies that I'm into).  The games are so well put together and balanced.  Something about getting into them has caused me to really be most attracted to games that don't rely on somewhat arbitrarily set parameters by a GM.  The best example I've found is Polaris, where there's really not even a lick of setting up a properly and numerically formidable situation for the heroes to go up against.  Instead, you're setting up (for lack of a better word) psychologically formidable and/or stimulating situations for your partner. 

Back to topic, I have not bought certain books due to reviews stating that the rules were unclear, so, I suppose in that way some games failed (for me) in terms of playtesting.  To bring it down to an even more elemental level, I'd say it's not the game itself that needs playtesting, but the product (being the words on the page as read by those whose shoulders you are not lurking over).  In this sense, board-gaming companies could learn from this rule too, as, all-too-often rules are unclear.  It seems rare I don't learn a game from both the rule book and Board Game Geek. 

I wouldn't say that a measure of a good game is if you'd play it over and over again for ever.  Some games are awesome experiences yet not emotionally taxing, or otherwise something you might not want to play all the time forever.  Also, there's just the matter of wanting variety of experience.  For a time, I was looking for the holy grail of games.  A Tri-Stat dX, a GURPS, a Universalis.  Something I could just use and stick with forever.  But now, that almost feels as silly as trying to find one TV show to watch over and over again forever.  I'm enjoying the specific experiences created by each different game, and the variety in the space between them.

Hi there,

This isn't really a discussion thread, but an instructional one for using the forum. I think a lot of the points you're making would be very powerful if they were part of Actual Play threads. I recommend that you start some threads aimed at these points there. Send me a private message if you'd like to get some orientation about that.

Best, Ron