Advice on getting results from playtesters?

[eraofthefallen]

I've released my new gaming system "Era of the Fallen" in beta mode since the beginning of the year.  I've posted announcements here, at rpg.net, and pen and paper games.  So far (9 months in) I've gotten precious few responses and no one who seems really serious.  Is it typical for a new RPG to have this much difficulty getting playtesters?  I've just reposted the annoucement in the Connections forum, so if anyone has feedback on the wording of the annoucement I'd be glad to hear it.

I've spent close to $100 between printing and mailing costs for the 3 copies of the testers manual I sent out to seemingly enthusiastic groups of playtesters.  None of them have yet produced any meaningful data in 6+ months.  I don't think any of them has actually played through a campaign.  Most of them don't even bother to respond to my emails.  I tried to be very clear up front that I was looking for detailed and frequent feedback, which I'm not getting.  Does anyone who has been through this process have advice on how 1 to get playtesters and 2 how to get them to deliver quality feedback I can use to improve the game?  As a small timer I can't afford to keep sending out manuals and not getting any return.

[dindenver]

Era,
  Well, this is a common problem (getting playtesters).
  The best experience I have seen are people who do playtest exchange. In otherwords, two designers agree to playtest each others games.

  As far as getting meaningful feedback, well, you are at their mercy. I would say your only recourse is to blog about it and see if a little public exposure doesn't a) produce results, b) protect others from sending playtest copies to people who do not playtest.

  Good luck with your game!

[lumpley]

Don't spend any more money! My strong recommendation is to switch to PDFs for playtesters.

It's rough, but their unresponsiveness is itself valuable playtesting results. If they loved your game, they'd tell you about it.

You've playtested it yourself, I hope - how did your internal playtests go?

-Vincent

[eraofthefallen]

Era,
  Well, this is a common problem (getting playtesters).
  The best experience I have seen are people who do playtest exchange. In otherwords, two designers agree to playtest each others games.

  As far as getting meaningful feedback, well, you are at their mercy. I would say your only recourse is to blog about it and see if a little public exposure doesn't a) produce results, b) protect others from sending playtest copies to people who do not playtest.

  Good luck with your game!

I think the playtest exchange has a good chance of getting good results.  Obviously both designers want good feedback and know what good feedback is.  But the end result is you have 1 completed playtest session.  Would you consider the game "done" at that point?  I won't consider releasing my game unless I have 10 playtests at a minimum, and I'd rather get 30.  Trading for each of them would take a very, very long time.

I guess I'm confused why it is so hard to get playtests done.  There are lots of gamers who like to play play RPGs.  This is a product that is mostly complete and will one day (hopefully soon) be demanding a purchase price of $20 or more, so they are getting something really for free.  It should be a win-win.  Where is the disconnect?

[eraofthefallen]

Don't spend any more money! My strong recommendation is to switch to PDFs for playtesters.

It's rough, but their unresponsiveness is itself valuable playtesting results. If they loved your game, they'd tell you about it.

Actually I don't mind spending the money on paper copies to keep a pdf off limewire so long as I'm getting feedback.  If I distribute pdfs to unmotivated playtesters, sure I save money, but I'm still left having wasted a lot of time for no results.  I could read the lack of response as "the game sucks dude" but I don't think any of them actually played it.  Why that is I could speculate on all day.  Family problems, working overtime, found something else better, wasn't all that into it in the first place, manual was too confusing, could be anything really. 

I'd much rather separate the wheat from the chaff and put my time and energy into the playtesters who actually give a damn.  The question is how to do that?  One possibility is for game designers to pool what appears to be a limited resource, good playtesters.  So one designer who has already tested his game and put it to market and no longer needs playtesters yet still has a list of people who have actually done it, enjoy it, and give good feedback.  Those same people are probably interested in testing other games and designers like me would love to have access to them.  Maybe a "playtester registry" or something and if you flake out you get blackballed from the list.  I don't know, just thinking out loud really.  But it seems there should be a way for small designers to work together to help each other out.

You've playtested it yourself, I hope - how did your internal playtests go?

-Vincent

I playtested an alpha version myself, but the rules have changed enough that I don't consider those results to apply to the beta.  I'd planned to run a campaign at ICON but the gaming track was so poorly run I didn't have a table, there were no signs so players couldn't find me, and basically it didn't happen.  Hopefully I'll have better luck next time as I put a lot of work into the campaign and think it is pretty cool.  Maybe Origins 2010 is a possibility.

[lumpley]

Well, nobody will playtest a game because they like playtesting. 9 times out of 10, playtesting is plain misery. If somebody playtests a game, it's because they like the game, see its potential, and are excited about the game's realization. It's because they really want to buy the game when it finally comes out, it's because they want to show the game to their friends and say "I helped make this real!"

I think that the search for outside playtesters is a game's first solid marketing test. If a game can get people excited to test it, that's a good sign that it'll have legs in the marketplace when it's done. If a game can't get playtesters, though, it's demonstrated that it's already not selling - there's something about its position in the market, its presentation, or its design that's failing to connect with its audience. If you were to release it in a final version today, you could safely expect more of the same.

Like I say, it's rough, but it's super valuable to know this early, when you can still change the game's position, its presentation, and (most importantly) its design. You'll know you've hit it when you start to get excited playtesters.

-Vincent

[eraofthefallen]

Well, nobody will playtest a game because they like playtesting. 9 times out of 10, playtesting is plain misery. If somebody playtests a game, it's because they like the game, see its potential, and are excited about the game's realization. It's because they really want to buy the game when it finally comes out, it's because they want to show the game to their friends and say "I helped make this real!"

This seems awfully pessimistic.  The playtesting I've done has been fun and the players enjoyed it, and my friends who've playtested other games had a blast playtesting.  I can't imagine anyone putting time and effort into playtesting if it was "plain misery".  If a game is in such bad shape its no fun at all to play, one session should be enough to tell and send it back.  I'd hope most games have enough thought put into them beforehand that they are fun to play even if a little rough around the edges.  Maybe you've just gotten a bad lot?

[lumpley]

Maybe so. It's a fact that a lot of the playtesting I've done has been early, internal playtesting for games that haven't gone on to see daylight. Maybe half.

Any thoughts about the rest of my post, about the search for playtesters being a game's first marketing test?

-Vincent

[eraofthefallen]

Maybe so. It's a fact that a lot of the playtesting I've done has been early, internal playtesting for games that haven't gone on to see daylight. Maybe half.

Any thoughts about the rest of my post, about the search for playtesters being a game's first marketing test?

-Vincent

I'm not sure what to think about that.  I don't have enough experience with playtesting to know how excited people should be.  It might be true, but if so it doesn't tell a designer much about what's not working if there's no feedback.

[dindenver]

EOTF,
  I think the take away has to be that either your pitch to the playtester is weak or your text is not compelling. I mean, imagine someone who has never seen or heard of your game, how do you grab their attention, get them excited enough to play your game? That's what you need to get firing for playtests to happen.
  I am saying it plainly like this, because I have had people decline to playtest my materials, and in every case, one of those two were my issue.

  Also, Vincent, thanks for sharing, I had no idea that the number of rejected projects that you worked on was close to half. That tells me a lot about my design process and where I am succeeding/failing (mostly failing because I am not letting go of projects that aren't firing on all cylinders).

[eraofthefallen]

EOTF,
  I think the take away has to be that either your pitch to the playtester is weak or your text is not compelling. I mean, imagine someone who has never seen or heard of your game, how do you grab their attention, get them excited enough to play your game? That's what you need to get firing for playtests to happen.
  I am saying it plainly like this, because I have had people decline to playtest my materials, and in every case, one of those two were my issue.

Ok that could be it, but rather than tell me that my presentation *might* not be up to par, why don't you just look at it and then give me your opinion?  I have an open call out in the Connections forum with my promo blurb and a link to the website.

[lumpley]

Well, seriously. If my first glance at someone's playtest material has me going "nope, not interested," of course I don't give the designer feedback. Feedback requires time and attention, it requires some critical thought and the effort to articulate it. You get that stuff from people who look at your material and invest in it, not from people who look and aren't interested. And thank goodness! The feedback of people who aren't interested isn't helpful, it's like "why I didn't give your game a chance," with usually an insult or two thrown in. Who needs it.

If nobody who looks at your playtest material gets interested in it, you get no feedback. You really have to figure out, as a designer-publisher-marketer, how to learn from this silence, make changes to your game or your pitch or both, and try again.

Now, me - my entering into this thread in the first place means that I'm investing in your success. I wouldn't be here talking to you if I weren't interested. However, I'm investing in your success as an independent designer, not in the success of this particular game you're pitching. That's why I've been giving you advice about how to approach this situation, not feedback about your game. As far as your game goes, me personally, I fall into the first-glance-not-interested group.

Since you're asking explicitly for feedback, though, okay, here's mine, but it just amounts to "why I didn't give your game a chance," and who needs it. Will it help?

If a game expects me to do a handling-intensive procedure like rolling 5d20, summing them, and subtracting 20 or whatever, the game had better be truly spectacular in other ways. If a game tells me that I can GM it diceless instead, using my judgment, it had better also show me a clear and brilliant vision of the GM's social and creative role in the group. Your pitch and your playtest material don't give me any confidence, at all, that the game IS spectacular in other ways, or that it DOES contain a clear and brilliant vision of the GM's social and creative role in the group. Instead, it looks like just another rehash of, like, AD&D 2nd Edition. Maybe incrementally better (like every game since 1990), but not any more interesting.

But please, let me just remind you - I'm not here to crap on your game, I'm here because I'm invested in your success as an independent designer. This game isn't going to give you success as an independent designer, and the fact that you can't get enthusiastic playtesters already shows it.

-Vincent

[eraofthefallen]

Well, seriously. If my first glance at someone's playtest material has me going "nope, not interested," of course I don't give the designer feedback. Feedback requires time and attention, it requires some critical thought and the effort to articulate it. You get that stuff from people who look at your material and invest in it, not from people who look and aren't interested. And thank goodness! The feedback of people who aren't interested isn't helpful, it's like "why I didn't give your game a chance," with usually an insult or two thrown in. Who needs it.

Well yes, of course there is helpful feedback and totally worthless feedback.  If I solicited feedback from a bunch of sci-fi gamers or political strategy gamers, I'd probably get a lot of not very worthwhile feedback.  Not because they suck, but because they aren't interested in fantasy style games.  But I'm not interested in that kind of "random" feedback and I haven't been complaining about a lack of it. 

What I am interested in are the little "hang ups" that can ruin an otherwise great game.  The people who initially think it is a great game and are enthusiastic but lose that enthusiasm have some very useful feedback.  All my original playtesters went in very enthusiastic.  Since this conversation started I've since made contact with one MIA who apparently no longer has a working PC but says she's still interested and just about to begin playing.  Maybe I'm expecting too much too soon and I'll get some feedback if I wait a while longer.  6 months seems like plenty of time to have gotten things rolling, so I'm not sure.

Now, me - my entering into this thread in the first place means that I'm investing in your success. I wouldn't be here talking to you if I weren't interested. However, I'm investing in your success as an independent designer, not in the success of this particular game you're pitching. That's why I've been giving you advice about how to approach this situation, not feedback about your game.

I appreciate that.

Since you're asking explicitly for feedback, though, okay, here's mine, but it just amounts to "why I didn't give your game a chance," and who needs it. Will it help?

Actually I was soliciting feedback on the presentation, but I'll take rules feedback.

If a game expects me to do a handling-intensive procedure like rolling 5d20, summing them, and subtracting 20 or whatever, the game had better be truly spectacular in other ways. If a game tells me that I can GM it diceless instead, using my judgment, it had better also show me a clear and brilliant vision of the GM's social and creative role in the group. Your pitch and your playtest material don't give me any confidence, at all, that the game IS spectacular in other ways, or that it DOES contain a clear and brilliant vision of the GM's social and creative role in the group. Instead, it looks like just another rehash of, like, AD&D 2nd Edition. Maybe incrementally better (like every game since 1990), but not any more interesting.

5d20 is "intensive handling"?  That's the first time I've heard that one.   I guess simplicity vs detail is a matter of taste.  Just out of curiosity what amount of die are you willing to roll before it becomes too much of a hassle?  If you're feeling lazy you can always roll 1d20 and multiply by 5, though that changes the numbers slightly.

I'm sort of losing you on the "social and creative role" of the GM in diceless roleplaying.  I'll admit I haven't been paying attention to a lot of the new games, but I don't recall the role of the GM being described in explicit terms in other games.  There is sort of a general assumption that if you've done ANY roleplaying you probably already have a good idea of what the GMs social and creative role is.  Is that assumption good for dice based games but not diceless?

If you were to rephrase "just another rehash of AD&D" as "a more realistic version of AD&D focused on storytelling" I'd not be opposed to that.   All RPGs which use a "Tolkien inspired" fantasy setting could probably be accused of treading a lot of the same ground as AD&D.  There's not a lot of "game changers" in that arena, and I don't think there needs to be.  Familiarity can be an asset.  Anyways, that's something of a tangent.  But for "random feedback" I'd say yours was pretty useful.  Helps to see through others eyes.

Cheers,
Brian

[pells]

Just my two cents, worth what it is worth. We did do a lot of internal play-testing, no external yet, but we're going to do it and soon (then again, depends how you understand soon, months is soon for me). And here's what we're going to do. That might not fit your product, its maturity, nor your business model. And note that this is on the verge of publishing.

Unlike Vincent (who gives great advices), we don't all have the chance to have play-testers "knocking" at our door to offer themselves ; and this at the very start of a project.

So, we'll facing the same difficulties as you : how to get people interested, the right ones that is, and how to get feedback from them. And, as I understand you, your project is more or less "advanced" in the realization (come on, 44 pages of a quick start guide). So, what's our approach ?

We intend to see the beta release (and its related play-testing counterpart) as a product in itself. And sell it (in pdf) !!
Not only do we make sure somehow that people who buys it will give it a try, but we would also be in our own beta version of promotion (defining a strategy, setting objectives, see if we can position ourselves on the market). And this is important, because after beta this is what is coming. Also, it changes the approach : we are not "asking" for play-testers, we are positioning our product.

Now, of course, you need to keep track of those beta testers, because :
- you will want to give them a free copy of the final version of the game
- maybe give them a paper copy "at the cost"
- offer credit in the final release if they participate
- offer them access of a forum where they can exchange with the author and contribute (a forum, of course, limited to those who buy the beta)

Also, this will mean a source of income for you, to complete the final product (buy illustrations, layout, whatever you miss). And you won't need to deal with stuff like (taken from your website) "To become an official playtester and recieve a full copy of the game manual, write me at: ..." and chat with me about how serious I might be. I'll give you money instead, no need to talk, to try to protect your IP, or see if I will play (which might disappoint you).

Note that this is almost the bounty business model. But, to do that, you need to think of this project as a product and begin to work on this !!!

Well, would this work ? I don't know yet. But I do think it is worth a try in your situation. If you can sell copies of the beta, you will be able to sell copies of the final product.

[eraofthefallen]

Just my two cents, worth what it is worth. We did do a lot of internal play-testing, no external yet, but we're going to do it and soon (then again, depends how you understand soon, months is soon for me). And here's what we're going to do. That might not fit your product, its maturity, nor your business model. And note that this is on the verge of publishing.

I think that's a pretty reasonable idea.  I considered it myself when I first started this, but decided against it because I felt it would be creating yet another barrier to people becoming playtesters by asking for money.  For me to offer a printed manual at cost would still cost around $20 since I use POD, so it is about the same cost as the final product will be but with none of the cool art or layout.  Not much incentive there.   But I think it could be useful for those who want to playtest but don't have a group to play with.  I've had several of those and have had to turn them away because I couldn't see any way for them to participate, but this could be a good solution to that.  If they never do find a group, as long as costs are paid, then it doesn't matter to me and there is always the chance they will give some feedback.  I'm not sure if any will be interested, but I'll make the offer and see.