Playtest methodology

[jefgodesky]

I'm not entirely sure if this is the right place for this topic, so perhaps the mods will move it somewhere else, but I'm going to begin an initial playtest of my RPG idea, and I'm wondering if there's such a thing as an established playtest methodology, or perhaps some pointers about how to run a proper playtest (as opposed to any other kind of regular campaign)?

[Justin D. Jacobson]

The question is too open-ended to get a meaningful response. The first question you need to answer is what the function of the playtest is. Depending on what you're testing for, the playtest might go very differently. Are you playtesting chargen, resolution mechanics, easy of play, player interaction, etc.?

[joepub]

I'm pretty sure that you do have the right spot, Jef.
and welcome to The Forge.

As to what Justin said...
I think that he's right that narrowing the question will get more meaningful responses.
It always does.

But I think I can still offer some advice for that broad, general question.

As far as methodology goes, it really varies system to system.
There are a few suggestions I have, which I have learned from talking to others and playtesting my own game.

a.) Move slowly through the chargen and system mechanics, and allow players enough time to really explore them. You want to hear as many questions like "So what happens if X and Y do..." as possible, because they'll often point out loopholes you hadn't even taken into account.

b.) Playtest with different people. I found very different issues with my game, depending on who I was playtesting with. It is important to have different groups, with different play styles, pointing out different gaps.

c.) Try letting another playtester GM the game while you play. Try letting another GM while you are a silent observer. Try watching one playtester explain the game to a new one.
When I had another player GM Perfect, I saw all these things I didn't like. And that gave me a better understanding of what I actually wanted the GM's role to be in Perfect.

That's all I got for you... things I consider to be pretty generic requirements for playtesting.
Of course, it ALL depends on what your game is, and what your goals are.

[jefgodesky]

Hmmm ... at the moment, I really need to test *all* of those things.  I have the beginnings of a system, but it's never been tried out before.  Thanks for the suggestions, Joe.  I'm very much at the very beginning of this process, and I was wondering if there was any set of "best practices" or what have you, analogous to, say, software testing.

[Josh Roby]

Jef?  Playtest it at least once yourself before you release it to others.  That way you'll have a basic idea of what needs a hard look and maybe a swift kick.

[Julian]

This isn't something you need to worry about yet, but it's really important:

Cold playtest.

At some point, you're going to be fairly happy with your game. Playtest sessions will be running smoothly, the players are happy, you write a final draft of the rulebook.

This is the time to hand it off to some other groups, who have no preconceptions about how the game plays, what it's about, etc.

You will learn an incredible amount from what they do, what questions they ask, and so on. When they ask questions, if you can't give them an answer that's basically a page reference, the thing they're asking about isn't actually in your game, just in your head.

[Justin D. Jacobson]

Depending on your timing, my suggestion would not be to try and playtest everything at once until your at the end of the process. That's not to say you can't run it like a full-blown game. However, I would focus on one or two issues during each playtest session. If you try to keep tabs on everything, your focus will be too scattered.

I also second heartily Joe's suggestion (c). Designers often fall into a trap of always GMing their own creations because they are so familiar with the rules, which is precisely why letting someone else can GM is so valuable.

[dindenver]

Hi!
  I think that you need to ask both kinds of questions. Directed questions with simple yes/no/multiple choice answers. And open-ended questions that let the playtester express their opinion.

An example might be:
Directed question:
  Did you have fun playing this game?
Open-ended question:
  Which mechanic was the hardest to grasp and why?

  Make sure the questions match the priorities/design goals of your game. If your game is about horror, don't ask questions about control of the environment or something that does not match your setting/goal.
  Try and ask questions that focus on mechanics of the game and less on issues that are harder to seperate or quantify. For instance, ask about what house rules they considered and why.

  But, Justin does have a point, playtest should be conducted differently based on how finished your game is and who your playtesters are.
  For instance, if the game is still heavily in development and the early stages of refinement, you want to try and play it natuarlly and and see where the mechanics flow and where they get in the way of the fun. But if it is late in the development cycle, you want to focus on getting people that are not part of your regular game group and make sure they know what questions you are gong to ask them and why.

[jefgodesky]

Jef?  Playtest it at least once yourself before you release it to others.  That way you'll have a basic idea of what needs a hard look and maybe a swift kick.

That's what I'm starting now--a playtest with myself and a few close friends.  I'm wondering if there's something I should be doing differently than a regular session.  It's still very early in the process, and this is something of a maiden voyage to see if she floats.

[Luke]

Hi Jeff,

You should be playtesting with your friends regularly -- at least for a year. As you playtest, keep the following things in mind:

• Have a firm grasp in your head of what the goals of your game are: What is your game about? (And your game is not about exploring a setting or surviving. If those are your answers, try again.) What rules does it use to evoke what it is about? How does it reward the players for participating in what it is about? Always, always, always have these questions (and their answers) in the forefront of your mind as you playtest.

• Play the rules as written. Do not drift or house-rule as you play -- no matter how your players howl. If something didn't work in the game, discuss it after the session's over. There's two reasons for this. First, you need to become aware of how your written word is being parsed by your audience. If something's not coming across, it could be unclear writing or it could be that the rule's no good. You can get a better sense of this if you stick to the rules as written in play. Second, many playtest players are aware that the rules are mutable at this stage in the design. They will bully, wheadle and cajole to get you, their loving friend, to change the rule into something favorable to them. It sounds terribly cynical, I know, but it's true. I see it all the time. By sticking to the rules as written, you can be assured of getting more honest feedback during play.

• Listen to yourself as you explain the rules. It's likely that what you have written out for the rules doesn't explain your ideas as clearly as you'd like. Your players will ask you to clarify. Listen to yourself as you explain. When you find an explanation that your players understand, take note and use that in the text.

• As your players are talking about the game and using the rules, listen to them. Watch their facial expressions, observe where they get stuck in the rules. Take note of these things. Are they stumbling over stuff because they simply dislike it, because the rule doesn't make sense to them or because it's poorly explained? Take note of what they like about the rules, too!

• Take notes and mark up your rules text as you play. Nothing extensive, but be sure to note the major issues and changes that arise in play.

• This final point is by far the hardest and the most important: Be honest with yourself. Listen to your heart and your gut. What do the rules feel like? Do they feel like they're working? Do they feel shaky or cludgy? Do they make you tense and uneasy when they should make you excited? Pay attention to those gut feelings and examine them later. What's happening in the rules text/execution that's making you feel that way? Most importantly, if something does not feel right -- no matter how much you objectively adore it or think "but it's got to be in there!" change it or kill it. Seriously. Stay married to nothing.

Designing a fun, functional game is hard. It's exhilirating and rewarding, but make no mistake, it is an intense and difficult process. Stay focused and don't get discouraged. Games can take any where from a year to a decade to gestate. Do what's necessary for your game and never be afraid to make the hard decisions, whether than means throwing the game out or going to press.

Good luck,
-Luke

[Nathan P.]

A useful technique is to record your playtest sessions and listen to them a day or two later. Make notes on where you had a trouble explaining things, when you had to pause to look up things, and the things that just rolled smoothly. Sometimes its hard to pay attention while your actually playing, so a recording is a good way to catch up with how an outside observer would see your playtest.

Also, after you've run the game yourself for a bit, have someone else in your group take over. You'll learn a lot from how they apprach the rules, and your experience as a player. I think it's a good step to take before or while you're finding outside groups to playtest.

Hope some of that helps.

[Graham W]

Jeff,

I asked something similar a while ago, on another site. Have a look at Tell me how to playtest.

Luke's advice also looks very good.

Graham

[timfire]

Another issue is what's sometimes called alpha vs beta playtesting.

Alpha playtesting (or whatever you want to call it) is when you're playing the game just to make sure it works the way you want it too. Most of the playtesting during this time will be run by yourself as a GM or a player.

During beta playtesting, you know the game works, but you want to make sure other people understand how it works, and if there are any hidden flaws. During this phase, most of the playtesting should be done by other people who have had no coaching other than the game text.

Most people are fine with alpha testing, but horrible with beta testing (and there are many reasons for this). But Beta testing is extremely important. See, when you run playtests yourself, you often steer play in certain directions subconsciously, so hidden issues don't always surface. You often also explain things in better detail than your text does. So you really need to get a variety of people to play your game cold who have no connection to you.

[Jixxala]

I second what Tim said.  As a person who has done a great deal of beta testing for MMORPGs I cannot agree more.  The general motto for many beta testers is "push every button and see what breaks".  In an RPG more than once I have heard "but no one would reaasonably do that!"  If your game mechanics allow for it, some fool will. Nothing is foolproof, as fools are so darn ingenious.  A good beta test is the only way to ferret out problems that will appear in retail.  Some small thing you and your friends never imagined could cause a game breaking problem, beta is your chance to find it and eliminate it.

Good luck and keep us informed on how things are going.
Jixx

[Ron Edwards]

Hi folks,

This was an informative thread, but it has to stop here. Threads in this forum should focus on actual playtesting, and discussion of what happened and how it went.

Best, Ron