A playtesting lesson from Reiner Knizia

[GreatWolf]

For those of you who don't know, I am a Eurogames nut.  These are the new wave of boardgames, mostly from Germany, that are breathing new life into the boardgame market.  The most prolific designer of these games is Dr. Reiner Knizia.  He is responsible for games like Tigris & Euphrates, Ra, Blue Moon, and the Lord of the Rings cooperative boardgame.  In my humble opinion, these are some of the best games currently out there.  The designs are tight, both mechanically and psychologically.  Knizia has demonstrated repeatedly that he understands the intersection of people and systems to produce a quality game experience.  Although he is not designing roleplaying games, I believe that we can all learn a lot from his methodologies and designs.

As such, I am listening to his Q&A session here with a great deal of interest and enthusiasm.  In particular, Knizia discusses playtesting, which he says is the most important part of the design process.  He states that it is impossible to understand how a game will work by just creating the rules and thinking about how you think they will work.  Game rules are about the creation of a certain experience, and you cannot understand the experience without actually playing the game.

This leads me to a rant.  Ready?

Our playtest skills suck.

Allow me to expand on this a bit.

As game designers, we are after enabling the creation of a particular experience of play.  That's the whole point of game design.  That's why System Matters.  We are all about the intersection of systems and people.  That's what drives many of our standard questions for newbies, right?  "Tell me about an experience of actual play that you liked."  "Tell me about how you want your game to feel."  What we mean is, "What experience are you trying to create?"

But, are we sure that our games are actually creating these experiences of play?

Let me ask a related question.  Are we making sustainable games?  By this, I don't mean "games that can have expansions".  Rather, how quickly will your game wear out?  Will it be fun for one or two sessions?  Or is it the kind of game that could be played again and again?

Again, the parallel to Eurogames.  Increasingly, I'm finding that companies are producing games that are fun for a while but quickly lose their luster.  Think about the paperback novel at the grocery store. Read it once, then throw it away.  Disposable entertainment.

But think about games like Chess or Go.  People have dedicated their entire lives to delving the depths of these games, because there are depths to delve.  These are more like those treasured books like The Lord of the Rings (if you're a Tolkien nut) or The Book of the New Sun (if you're a Lupine) that you take from the shelf once again, preparing to have your mind expanded by what lies within.

What about us?  Are our games just the "flavor of the year"?  Or are we actually about making games with depth and longevity?  If you are, you will not be able to achieve this goal without long-term, consistent, disciplined playtest.

So, for example, I'm working on my next game project, and I'm pretty sure that I have an alpha just about ready to go.  My goal is to playtest it twice a week for the next several months, continuing to hone it and shape it while I begin to form the manuscript.  Will I be ready for GenCon?  Maybe GenCon 2008, but that might be rushing it.  I want it to be perfect, and perfect can't be rushed.

If you think that's hardcore, I'll point out that Knizia has playtest groups running every single night.

So, when is the game done?  Knizia said that he knows a game is ready when the rules stop changing in between playtests.  For him, that's generally nine months of full-time work.  Very few of us are full-time designers.  Are you prepared to test your game until it's ready?

So, here's the gauntlet.  I'm calling all of us to a higher standard.  If we're serious about being game designers, that means that we need to throw down and do the work.  Having a kewl idea is a great place to start, but commit yourself to playtesting until it is perfect.

[Jason Morningstar]

Seriously, perfection is a misguided goal.  I agree that you can map roleplaying game design to board game design pretty closely in spots, but there is not a one to one correspondence.  For one thing, there are creative dependencies in a roleplaying game that don't exist in a board game. 

I think if your game is playable and fully baked, if it has been responsibly and zealously playtested and actually works and brings the fun, then you ought to cut the cord.  Contrast this with publishing a game with broken rules, or publishing a game that you've only tested with your pals.  Contrast this with the game that is "not perfect" and stays on your hard drive  forever.

Perfection is a long, long wait away, and I fear that it can turn into a comfortable excuse for never finishing a project. 

[GreatWolf]

Hey, Jason!  Thanks for the reply.  I was starting to become concerned that someone had activated my "Inability to Annoy" Trait.  I try to come up with a solid rant, and they just never work....

Anyways, to your comments.  I'm actually going to start at the end.

Perfection is a long, long wait away, and I fear that it can turn into a comfortable excuse for never finishing a project. 

I totally agree that this is a potential issue.  "There's a ditch on both sides of the road", and this is the other ditch.  So, I will state to the world, "Hey, people!  At some point, you do need to finish!"  However, I don't think that this tends to be our problem as a group.  Rather, my general sense is that our tendency is a desire to rush to publication too quickly, which then impacts the quality of the games.

To be clear, when I'm talking about "perfect", I am meaning "actually works and brings the fun", to quote you.  But, how do we know that we are succeeding?  Having touchstones like "I didn't have to change any rules after this playtest session" are helpful places to start.

I also agree that roleplaying and boardgame design are not exactly the same, and we probably would point to similar reasons.  For example, there tends to be a lot more "openness" in an RPG that needs to be managed, whereas boardgames tend to be more strictly bounded.  As a result, there's a bigger impact of player personality on an RPG.  But, at the same time, I would argue that this calls for more playtest than a boardgame.  Are we playtesting our rules with different personality types?  Are we creating a positive creative space with explicit expectations for all participants?  The very openness that gives RPGs their strength can be their downfall.

This is particularly true with more complex games.  One of the classic examples of this, sadly, is Rune. I need to be clear:  I'm a Rune lover.  I really enjoyed the frolicking chaos and glorious kill-stealing encouraged by the game.  I enjoyed the bold affirmation of "this is a Viking game, not some namby-pamby story game".  I enjoyed the fundamental mechanics that forced both cooperation and competition, requiring a careful balance of both.  BUT!  It wasn't ready for prime-time.  Our group horribly abused certain rules (e.g. buying multiple levels of Implacably Braced), while others were unclear (just how did you win a particular game session again?)  Certain abilities were clearly inferior to others, but they were costed at the same amount.  And so on, and so on.  It lacked that final polish that another couple rounds of playtest could have given it.  Rune had the potential to be a great game, but as it stands, it is simply a flawed gem, a sad look at what might have been.

Now, I should say that I'm seeing some encouraging trends.  Grey Ranks seems to be getting some serious butt-kicking playtest, which is very cool.  Tony is taking the extra time to test out his rules-writing for Misery Bubblegum.  I'm going to try to put my money where my mouth is and do the same.  But I still think that, as a group, we can do better.

[Jason Morningstar]

Good points, Seth.  It'd be useful to set benchmarks somehow, even if they were pretty general, based on the direction and complexity of individual games.  We should talk about those benchmarks.  "No changes between playtests" seems like a logical indicator of a fully baked project.

The notion of methodically seeking out playtest groups that fit particular patterns seems like an insurmountable obstacle - honestly, I'm embarrassingly grateful for any blind testing I can arrange. 

[GreatWolf]

Good points, Seth.  It'd be useful to set benchmarks somehow, even if they were pretty general, based on the direction and complexity of individual games.  We should talk about those benchmarks.  "No changes between playtests" seems like a logical indicator of a fully baked project.

Of course, I'm not sure what those benchmarks are.  Well, how about this?  You're in the middle of playtest right now.  How have you been evaluating your playtest?  What have been your benchmarks?

In another post, I'll write up what I've done so far, and I'll scrawl notes as I go through my ongoing playtest as I learn about the process.  Maybe, as we compare our experiences, we can start extracting some principles.

The notion of methodically seeking out playtest groups that fit particular patterns seems like an insurmountable obstacle - honestly, I'm embarrassingly grateful for any blind testing I can arrange. 

Tell me about it.  I'm thinking about trying to convince my Friday night group to offer our playtesting services to folks, for just this reason.  Or something like that.  This is an area where that mutual assistance that  we talk about would be really helpful.  Again, I'm trying to figure out how to put my money where my mouth is in this area, too.

Maybe playtest exchanges?  I test your game and you test mine?  Or something like that.  I'm still working this angle in my head.

[Ben Lehman]

I playtest games until I don't have to change the rules after a playtest session.

This takes me usually 5-6 playthroughs, often a few more.

Why so less than Knizia?  I imagine it's because, compared to a Knizia game, the mathematics of my games are quite simple and the mechanical processes just less complicated.  There is, quite simply, a lot less to playtest, and a fair amount of preliminary testing can be done just by analyzing the rules inside my own head.  (Finalized testing of course requires people but the point is that I'm often part-way there with just the maths.)

If I could playtest every night, I would, of course.  But I'm not Knizia, so that's a bit of a hardship.

yrs--
--Ben

[Jason Morningstar]

OK, Seth, you're on.  Grey Ranks has been in on-and-off development since October 2005.  Mostly off until the last six months.  Here are my playtesting approaches, in descending order of quantity:

I have a little brain trust I bounce ideas off, and much "testing" goes on by proposing and challenging design goals and assumptions via email and face to face.

I've had a number of mechanical playtests, that looks like this:

1.  Me sitting at a desk running the game's numbers, start to finish or maybe just to test a particular portion over and over and over again. 
2.  Me and a couple of friends doing the same thing, until things utterly break.

I've run the game as a one-shot (using various points of attack) for friends and fellow travelers:

1.  In my living room or at an apartment con with pals.
2.  At conventions for interested strangers.

I've solicited semi-blind tests (I'm available to clarify and answer questions before the actual session):

1.  With widely distributed play groups.

I'm running the game absolutely straight, as intended, for my weekly game group.  We start tonight, as a matter of fact.  I'm strictly an observer for this - my four gamer buddies are playing and I'm watching and listening. 

[Mike Holmes]

So... the first six editions of D&D were playtest editions, right (as opposed to the simple gross commercialism of the last couple)?

Is there a sweet spot on playtesting a RPG (between too little and never publishing)? Yes, of course. Do we really miss it by much? I'm not so sure.

Maybe when Ralph and I were playtesting Universalis we got lucky. But the rule did stop changing after only a few tries. It got to be what it was going to be and screwing around with it... well it wasn't actually making the game better in any way we could discern. So we published it.

On the subject of longevity... aren't some of those flavor of the day games from Reiner? He didn't invent chess. I have to question your assumption in four ways:
1. From an economic POV, the incentive has to be to create flavor of the day games. Call me a crass capitalist, but what's the incentive to create the game-to-end-all-games? Assuming it could be done?
2. Why is playing the same game over and over automatically superior to playing different games?
3. I think that many games that folks around here have made turn out to have a lot of longevity. I come back to a title like InSpecters or My Life With Master over and over.
4. But even those I come back to are often written to be short-form. MLWM has an endgame. It actually does me the favor of not requiring it to be run for more than a few sessions at a time so I can get to other games. No, I don't run it over and over again, one full game following the next. But that's not the same as saying that I've gotten tired of it.  

I think that you may be mistaking enthusiasm to try a ton of novel games as having gotten bored with the old ones. There's simply not enough time for anyone to play everything put out there.

Now does that mean that there's no time for the "long-form" game any more? Well, for my part I have been playing HeroQuest weekly now since it came out (Hero Wars before that). In addition to sampling all sorts of other games. And it only gets more and more interesting the more I play. Like chess.

Now, Laws isn't who you're talking about here, but I don't think that HQ is all that superior to other designs around here (just happens to push my buttons). I could as easily be playing a long-term game of Burning Empires.


If you want to turn on the "Annoy" function of the rant, then step on up and make this a clear accusation: name a game that suffers from lack of playtesting. Which ones are they? Who is making the mistake?

If all you're saying is that one should try hard to get enough playtesting in... well who could argue? If you're saying that people actually are not doing it, and it's leading to poor games, well you'll have to give examples to convince me that it's some pervasive phenomenon here. Are there some few with this problem? Oh, probably. But I'm fortunate enough not to have run into them in play, I guess. I don't see it as pandemic.


Saying this just so that you can perhaps see a counterpoint to this discussion: what if I were to suggest that you were proposing this just to support your own historical speed of design which resulted in Alyria taking... how many years? Not only is it important to finish the game (which you have), but its important to do so in a timely fashion, and not let doubt of imperfection cause you to call for another round of playtesting.

I think that nine months of work is about what games around here tend to get between design and full publishing, actually. Are you suggesting we need years more than Reiner does?

Mike

[GreatWolf]

Hey Mike!

I'll admit that pointing past GenCon 2008 as my own deadline might be an overstatement.  Personally, I'd like to have a playable beta or ashcan version of Dirty Secrets ready to go by GenCon this year.  However, I'm willing to let that go if the game isn't ready to go

I'm not calling for "the game to end all games".  I'm actually calling for games like Inspectres or MLWM, which do have a lot of potential depth and longevity.  This isn't a "long-form" vs. "short-form" debate.  (I actually prefer short-form games anymore, since I don't really have the time or desire to play long-form games.)  However, that depth and longevity doesn't just "happen"; rather, playtest will reveal if it is there or not.  Also, playtest will reveal if the features that you think are there are actually there.  This will affect long-form games more than short-form games, though, as a cycle of play tends to be longer in a long-form game.

Hey, I'll gig myself on this one.  My first game Junk was woefully undertested, especially in terms of costing for various bits of equipment.  The only reason that no one has accused it of being horribly broken is that there aren't enough people who know about it

Oh, and justifying the length of time on Alyria?  Heh.  Not even a bit.  Most of that delay was the result of Alyria sitting around doing nothing, not any sort of focused playtest effort over all that time.  It wasn't even a matter of "fiddling until it's perfect"; it was a matter of it's simply gathering dust.  Actually, part of this thread is borne out of my desire to be a bit more disciplined and snappier in my getting the next project done without giving up on quality.

[Thunder_God]

I also agree that roleplaying and boardgame design are not exactly the same, and we probably would point to similar reasons.  For example, there tends to be a lot more "openness" in an RPG that needs to be managed, whereas boardgames tend to be more strictly bounded.  As a result, there's a bigger impact of player personality on an RPG.  But, at the same time, I would argue that this calls for more playtest than a boardgame.  Are we playtesting our rules with different personality types?  Are we creating a positive creative space with explicit expectations for all participants?  The very openness that gives RPGs their strength can be their downfall.

Emphasis mine.

You also need to know who your game is for. If your game is not aimed for a certain demographic of player-types, and you're not going to try to sell them on the game, then why are you to try and playtest with them?

Additionally, certain people do not like certain games (cooperative versus competitive for example), and while you may want to have your game playtested by the "other" demographic, they know what they like, and aren't likely to be interested in playtesting your game. What then?

Also, as you said, in the end this all rests down on the availability of playtesting.

[GreatWolf]

Guy,

I have no argument with this.  Even at the design stage, it is usually obvious that certain types of people will be interested in your game, and others won't.  Trying to persuade people who won't be interested to playtest your game is usually an exercise in futility.

Just to build on the point, though, there can be border cases.  Sometimes you don't know what people will trip over, which is why it could be productive to expand your playtest pool.  That way, you can decide if you need to do a better job of explaining certain things (particularly on the Social Contract level), or if you are going to choose to exclude certain people from your core audience.

[Jason Morningstar]

A good example of this - a playtest group for Grey Ranks that had serious issues with the game's opening, which includes an extended mission in which each player frames a scene for which success is assured.  They really don't like this, and have told me so.  I've been concerned with the gradually escalating difficulty (which makes success easy at the beginning and super hard at the end of the game), and sort of designed around this by removing pointless die rolling for the first mission. So these guys approached it from a different angle than I did, with different assumptions, and their result was different from that of other playtest groups, and their negative feedback has been really useful to me in assessment.  I didn't expect it and I didn't plan for it in organizing playtests, though. 

[ynnen]

I've been fortunate enough to be heavily involved in playtesting both boardgames and some big production RPG products. They are very different beasts, and the playtesting approach surely bears that out. Boardgame playtests seem to be much more objective and after the question "is it broken" first, then "is it fun" second... RPG playtesting often reverses these two goals -- "is it fun" first and then "does it mechanically work" second.

I think this is often based on the age old adage about RPGs in general - they are a set of rules that are completely up to you to embrace or change. The GM is always right. If you don't like a rule, ignore it, etc.

Just my observations... As such, I think RPG design/development is more art than science. An artist can't paint and paint and paint forever - eventually you need to accept the creation for what it is and hope others appreciate your perspective, insights and contribution.

[Jake Richmond]

I can take this need for more play testing to heart. I never got to play test Panty Explosion as much as I wanted (although we did test it a fair bit). I'm currently having trouble fining people to test The Year We All Died, and I'm determined to test that as much as I can before I release it.

I almost knocked Knizia over in a restroom at Gamestorm last year. Totally slammed right into him coming out of a stall. He was there running play tests and stuff and Christian, Travis and I were a few booths down doing demos. As I was washing my hands I told him how much I liked Samurai and that I had recently played it with my 7 year old niece and my mother. He said something like, "that's why testing is so valuable, so everyone can enjoy the game". Then he went off to use the toilet.

True story.

Jake

[Mickey]

I'm not sure if this is the right place to post this, but this is why I think hosting PBP (play-by-post) in the actual-play or playtesting sections of this forum is a good idea. I mean, not everyone has access to a group of dedicated playtesters right outside their door
Sure, I could just look around for other forums to run a playtest session online (or participate in one), but wouldn't it be good to do it HERE as well, with such a knowledgeable community at hand to help interpret the results and give advice?

One of the obstacles to PBP play testing might be that people are willing to have others participate in their play tests, but aren't necessarily willing to participate in others.
Perhaps a general rule of thumb would be to invite others to your PBP with the notion of participating in their PBP sessions in exchange?

"that's why testing is so valuable, so everyone can enjoy the game"