Review Criteria for RPGs

[Gold Rush Games]

As part of our new Eureka! Publishing & Design Program I am going to be providing a review/critique of submissions. Each topic/element reviewed will be given a rating from 1 to 5.

 The idea is that products will have to receive a minimum score (of, say, 3.5) in order to be accepted into the program. The general scores used would be:

 1 = Poor (extremely poor quality, technically inferior or amateur)
 2 = Below average (sub-standard compared to existing products)
 3 = Average (on par with existing products; acceptable quality)
 4 = Good (exceeds market expectations; quality stands out)
 5 = Excellent (greatly exceeds market expectations; raises the bar)

 I have some of my own ideas, obviously, about what elements should be evaluated when considering a product for publication. Among them:

 * Marketability (is the genre a fresh twist or cliche?)
 * Cover illustration
 * Interior art
 * Organization
 * Writing (incl. style, voice, etc.)
 * Interior design (readability, graphic design, attractiveness, etc.)
 * Editing
 * Playability (fun factor)

 While I was sitting at my desk trying to think of the various elements of a submission that should be reviewed (i.e., scored), it dawned on me that I could get some suggestions from indie designers.

 So here I am, hat in hand, asking you folks this question:

 What other elements do you feel should be evaluated when considering a product for publication?

[Sylus Thane]

Well that is a question.

By that do you mean purely as in taking submissions and if they are good they go out with your companies name on it?

Or, as a third party publisher?

If it's the former, put whatever criteria you want, it is your name and reputation on the line. Although I would put a listing of resources you can help provide should someone not have them available to them, such as artwork and the like.

If it's the latter, I don't know. Can you really quantify things like that to an independent person coming to you to get their work published.

I guess it would help to know how you differentiate the two?

Sylus

[Jack Spencer Jr]

I think you're doing yourself a bit of a diservice with this. I mean, you can have some kind of system in place to decide what games to risk you capitol on, but who's to say what will sell and what won't? The Beatles were turned down by nearly every recording company in the industry. One even told them that guitar groups were on their way out.

Success is about 5% skill, 10% hard work and 90% luck. Yes that does add up to 105%. Success is wonky.

[Ron Edwards]

Hi Mark,

My main question is what the games are being compared to. If it's to one another, then I think your scale will be too speculative to mean much - your initial impression of the games going in will be the same as the one coming out, just quantified a little.

It would be interesting to run a few role-playing games out there on the shelves through your metric, and use those scores as the framework for comparing new submissions. I strongly recommend, however, being very careful about scoring "marketability." You probably know that I think RPGs true market has never been assessed or perceived by the trade magazines, distributors, or retail community.

Best,
Ron

[Gold Rush Games]

Okay, maybe my question wasn't phrased well. I'm not really looking for feedback about the fact that we are going to evaluate submissions to the Eureka! program. That's a foregone conclusion.

 For information about the Eureka! program, please see this thread:
 [urlhttp://www.indie-rpgs.com/viewtopic.php?t=6210[/url]

 Apologies if anything I've posted has been confusing or ,isunderstood. Sometimes I stayup late working and don't maintain a sufficient supply of coffee. ;)

 To Sylus Thane:

 What I meant about accepting submissions was for our Eureka! program, which is a new imprint we've created for publishing creator owned indie games.

 We will be putting our new Eureka! logo on the books but also allowing the designer to have their name and logo on the book as well. So it's sort of a hybrid of the options you mention. Does that make sense?

 To Jack Spencer, Jr.:

 Who's to say what will sell and what won't? I don't know. What I need to determine is what my company is going to publish, promote, market and sell through distribution channels.

 Will I always make the right choice? I hope so, but I may not. I'm human. That's why I'm trying to come up with a somwhat impatial method of evaluating product submissions. Plus I believe the critique would be of value to the designer, as well.

 To Ron Edwards:

 The submissions would be compared to existing products in the market. So a submission that had art of worse quality than the average product out there (in the reviewer's opinion) would get a score lower than 3. If the art was better than the average product (again, in the reviewer's opinion) it would score higher than a 3. And so on.

 I agree that it would be interesting to run a few existing role-playing games through my rating system and use those scores as the framework for comparing new submissions.

 You suggested I be very careful about scoring "marketability." I agree. I don't even know that I will keep that category. I just threw out some ideas to spark discussion.

 So try this, all. Forget about what I posted in terms of categories. What categories would you evaluate a product on if the designer asked you to publish, market and distribute it?

[Jack Spencer Jr]

That's why I'm trying to come up with a somwhat impatial method of evaluating product submissions.

OK, in an attempt to give you some more on-topic feedback, although not as on-topic as I would like.

I, personally, do not like the idea of an impartial method for evaluating submissions. I would rather place my chips on something I personally believed in and felt strongly about and fail than to have a successful product I don't feel anything for be the biggest game ever but I published it based on an impartial system of evaluation. In the first instance, whether is succeeds or fails, it was my decision. In the second instance, it was not my decision, but the decision of this impartial system. So success or failure, it wasn't my doing.

That said, it's easy for me to say this when it isn't my bananas on the line, right? I see nothing wrong with such a thing being *part* of the decision-making process, but be willing to chuck it, right? I think you had already said as much.

[Gold Rush Games]

I, personally, do not like the idea of an impartial method for evaluating submissions.

 To each their own, I suppose. I think an impartial scoring method or one that is as objective as possible, would be of more benefit to prospective clients and a fairer way to evaluate submissions. After all, no one would want their submission rejected because the reviewer simply didn't like the genre, would they? Maybe someone would, but I can't see it.

 "What? A sci-fi book? Sci-fi sucks for RPGs. I'm gonna give it a low score because I don't want to see GRG publishing any of this drek."

 That would be a very difficult position to defend. Fortunately I'm not using that type of valuation method so I won't have to. ;)

I would rather place my chips on something I personally believed in and felt strongly about and fail than to have a successful product I don't feel anything for be the biggest game ever but I published it based on an impartial system of evaluation.

 But are you speaking from the perspective of a publisher or as a designer?

That said, it's easy for me to say this when it isn't my bananas on the line, right? I see nothing wrong with such a thing being *part* of the decision-making process, but be willing to chuck it, right?

 Yes and no. It will not be the absolute authority on what we decide to accept into the program. Some of our preferences would naturally be built into the scoring process. If we received a pornographic RPG, as an exaggerated example, as a submission we would reject it no matter how good the art and writing was.

[Kester Pelagius]

Greetings Gold Rush Games Rep,

As part of our new Eureka! Publishing & Design Program I am going to be providing a review/critique of submissions. Each topic/element reviewed will be given a rating from 1 to 5.

<...>

 What other elements do you feel should be evaluated when considering a product for publication?

Quite frankly, if anything, I think the sort of thing you need to review is what is presented in the material first and foremost and worry about that before you even consider issues of style, prose, grammar, or anything else.   Too, for starters, this thread needs a direct (working) link to the pertinent information* about the Eureka! imprint/publication deal so we can all get on the same page and know what, precisely, is being asked in context.  Otherwise, to play paranoid devil's advocate, I'd have to ask what is in it for the author?

If this is a "I am paying you to print my book" deal then I'd have to say... nothing.  Sure, you're doing the folks a favor by looking over their material, but if they have the impression they are paying you to print their material then all they'll likely want is for you to print what they have.  No comments.  No questions.  Just do it.  Nothing against you, just be aware some folk are like that.

If this is a "subsidiary joint publication" imprint in which both author and publisher will have vested material interest then, of course, the answer to your question would be way different.  (Though I'd still want to know what rights are involved in the deal.)  My point being you've asked a question assuming everyone here has knowledge of your other threads and your service.  Most of us have probably read your posts here and at RPGnet, but what about the newbie just signed up members who might not have?

Or, and I know this will probably come as a shock to you, what if no one has ever heard of your game company.  Ok, calm down.  Count to ten.  I know it's ridiculous, but it could happen.

Heart beating slower now?  Good.

Links to the information are needed.   (Golden opportunities and all that, wot?)  Not just for newbies but as a memory refresh for those of us who might not have read your threads in a while.  You could put a direct link in your sig, if you want, no pressure.  Just a suggestion DSTM.

That said I think a few questions posed to you, as a representative of a gaming company, might be a good start.

For instance what would turn YOU off to a submission?

If a author asked you to sign a non-disclosure form would you reject their MSS out of hand?

What if the MSS being submitted had already been "garage" published in a limited run or two?

If a MSS submitted already had copyright filed (and the author made a point to let you know) what would be your response?

You mentioned artwork, does this mean you will be checking all illustrations used to be sure that artists copyrighted and intellectual material is not being stolen?

If a author wanted to negotiate the boilerplate in your contract BEFORE submitting anything to you, would this impact how you viewed the authors submission?

For that matter will you be reviewing the MSS to ensure that no copyrighted material- such as mention or use of established worlds and derivative works of fiction that may infringe upon intellectual properties- are used?

If you find that submitted MSS do, in some minor or major way, use copyrighted material, however incidentally or accidentally, will you work with the submitters to help them fix the errors in the submitted MSS or reject the material?

There's probably a dozen other questions I could ask, but I think that suffices to illustrate the potential problems.  Was that a sigh of relief?  ;)

I am sure you will make certain your legal department is going to give the MSS the once over to cover your arse, but some authors may not be aware that they've crossed that line.  Something to think about when considering how far you are willing to go to work with authors who would like to print their works through your imprint.  Or let it be known that any such MSS will be rejected, end of story.

Sounds like a headache to me but, hey, God favors the compassionate!

Right?

Much luck.  I apologize if it sounded like I was trying to break your balls.




Kind Regards,

Kester Pelagius


* hope you don't mind?

[Gold Rush Games]

Greetings Gold Rush Games Rep,

 I'm getting a lot of that reference today.

 Please, call me Mark or Mr. Arsenault. :)

Quite frankly, if anything, I think the sort of thing you need to review is what is presented in the material first and foremost and worry about that before you even consider issues of style, prose, grammar, or anything else.

 Part of the requirement for a submission to be considered for the program is that it has to be "ready to print," though. So we'd be evaluating the whole package (writing, style, design, "look," etc.) at once.

Too, for starters, this thread needs a direct (working) link to the pertinent information* about the Eureka! imprint/publication deal...

  http://eureka.goldrushgames.com

 The page is still being created, but you can still download the draft agreement, a cost & royalties spreadsheet and other items to give you a clear idea of where we're going with the program.

Otherwise, to play paranoid devil's advocate, I'd have to ask what is in it for the author?

 That's not paranoid. It's good sense to ask such a thing.

 In summary, what's in it for the author is:

[*] POD publication of their book (with their imprint/logo)
[*] Client maintains copyrights (we just license publishing rights)
[*] ISBN assignment & barcode
[*] Listing in R.R. Bowker's Books in Print database
[*] Listing in the Ingram Book Co. database
[*] Listing on Amazon.com, BN.com and others.
[*] Review copies sent out to those on our review list
[*] Association with an established publisher (est. 1995)
[*] Distribution sales via our sales office (Tundra Sales Org.)
[*] A percentage of all sales of the book
[*] Representation (and potential sales) at cons we attend
[*] Listing in the GRG catalog, on the GRG web site, etc.
[*] Marketing (not exhaustive, but comparable to what give our own products)
[*] A much cheaper alternative to traditional printing
[/list:u]

 All told, what we're offering would cost a new start up company thousands of dollars. We're offering it for as low as a few hundred.

...if they have the impression they are paying you to print their material then all they'll likely want is for you to print what they have.

 To clarify, that's not what Eureka! is about. We're not a printer. We're a publisher. Our sister company, Golden Pillar Publishing, is closer to what you describe, but even GPP provides everything Eureka! will except the game/hobby distribution.

...I'd still want to know what rights are involved in the deal.

 Have a look at our draft agreement, which you can access from the Eureka! web page.

Most of us have probably read your posts here and at RPGnet, but what about the newbie just signed up members who might not have?

 You're absolutely right. An oversight on my part. My apologies. I hope I have provided sufficient links to the information. The program is still not "live" yet, but I expect it to be by next month.

You could put a direct link in your sig, if you want, no pressure.  Just a suggestion

 Um.. There is a link to the Eureka! mailing list main page in my sig. It has some info there, including a link back to the Eureka! web page... ;) But I see what you're getting at. Thanks for the prod.

For instance what would turn YOU off to a submission?

 Poor quality art, writing, editing, a cliche product for an overused genre, and so on. We're looking for quality, innovative game products from indie designers. The good stuff.

If a author asked you to sign a non-disclosure form would you reject their MSS out of hand?

 No. In fact, I hadn't considered posting a non-disclosure declaration of our own to potential clients, but that's an excellent idea. I will try to work that into the submissions process.

What if the MSS being submitted had already been "garage" published in a limited run or two?

 Doesn't matter. Heck, I'd even happily look at titles that were previously printed, sold out, and whose owner wanted to keep it in print via POD. The product doesn't have to be "new." It just has to be good.

If a MSS submitted already had copyright filed (and the author made a point to let you know) what would be your response?

 Registering a copyright offers no more protection to a work than its mere creation and recording in tangible form. All works (in the U.S.) are protected the moment they come into being.

 With that said, a creator having a copyright registered already tells me that they are serious, know at least something about copyrights, and otherwise has little bearing on our consideration of whether to publish their book.

You mentioned artwork, does this mean you will be checking all illustrations used to be sure that artists copyrighted and intellectual material is not being stolen?

 All art will be reviewed, but not to ensure protection of IP. The publishing agreement has language in which the Client guarantees that their product does not violate any third party's IP rights.

If a author wanted to negotiate the boilerplate in your contract BEFORE submitting anything to you, would this impact how you viewed the authors submission?

 Not how we viewed the submission, but it could have an affect on our decision whether to accept it into the program or not. It would depend on what the specific item was.

For that matter will you be reviewing the MSS to ensure that no copyrighted material- such as mention or use of established worlds and derivative works of fiction that may infringe upon intellectual properties- are used?

 Client declarations, guarantees and warranties are written into the contract. It would be a bad idea for a Client to submit material that violated another party's IP rights.

 We do reserve the right to reject any submission, howeve, including for reasons involving legal liability.

If you find that submitted MSS do, in some minor or major way, use copyrighted material, however incidentally or accidentally, will you work with the submitters to help them fix the errors in the submitted MSS or reject the material?

 It would depend on the submission, frankly. If the submission was absolutely fantastic, but included a bit of material that we felt infringed upon another person or company's copyrights or trademarks, including violations of the d20 STL, then we would work with the client to correct it.

[quote/]There's probably a dozen other questions I could ask, but I think that suffices to illustrate the potential problems.[/quote]

 I see no problems thus far. Feel free to keep the questions coming. All I ask is that you review the "Draft Agreement" and the "POD Costs & Royalties" documents before posting more questions, in case they are answered in the documents.

I am sure you will make certain your legal department is going to give the MSS the once over to cover your arse...

 We've got that covered.

Sounds like a headache to me but, hey, God favors the compassionate!

 One man's headache is another man's livelihood. ;)

Much luck.  I apologize if it sounded like I was trying to break your balls.

 Not at all. Yours is one of the kinder messages I've dealt with lately.

[HinterWelt]

Mark,
This is just a pet peeve but you have defined subjective criteria and called it objective. I know you want to sound as positive and impartial as possible but unless you will be referring to an industry standard (I do not know of such a standard) then it will be what you perceive as an industry norm. In the end, you will publish what you believe to be winning submissions by your criteria. It is admirable of you to discuss what your internal criteria will be but they are your criteria.

Secondly, the most subjective of criteria, cliché genre will most likely be your largest downcheck. Again, this is your perception of cliché yet you represent it as an objective and impartial criteria. No problem using a subjective criteria because, again, it is your company.

Just as an aside, we hve done much of the benefits you have described and it has not cost us thousands and thousands of dollars. I believe POD is viable as a JIT modeled supply chain. If employed correctly it will yield a smaller per unit net but have a considerably lower break even.

Please do not take this post as a personal attack. It is merely pointing out what some posters may not understand (myself included 8-D). In the end, it sounds like you are offering a service to allow writers to POD Publish and that is a good thing. If you read this post with an accusatory or inflammatory tone in your mental voice read it again with a neutral or chatty tone and you will get my meaning.

Keep up the good discussion, it ALWAYS helps.
Bill

HinterWelt Enterprises
The Next Level in RPGs
William E. Corrie III
www.hinterwelt.com   
http://www.hinterwelt.com/chargen/

[Jack Spencer Jr]

After all, no one would want their submission rejected because the reviewer simply didn't like the genre, would they? Maybe someone would, but I can't see it.

Well, maybe. But then, I wouldn't feel comfortable going with a publisher who said "I really do not like your game. I simply do not like the genre. I don't get it. I don't see what other people find so interesting about it. But I think we can make some money with you're game, so I'll publish it."

But are you speaking from the perspective of a publisher or as a designer?

Neither. Ha! I'm speaking as an idealist. I am also speaking from the POV that is I were a publisher and something comes across my desk that sparks my interest and draws my attention, then it is likely to do the same for the game buying public.

[Gold Rush Games]

So you feel that it is a waste of my time to solicit feedback on review criteria from the people here in this forum? Gads. I've been criticized before for not caring what my customers think. Never before have I been criticized for caring what my potential customers think. ;)

[busling]

Mark,

Another point that you could review the submission on:
"Content matching the promise"

eg. "101 Spell books" not having any spell books would be a complete failure in this department.

Considering that you will have control over the marketing and such this point should not be too much of an issue.
A more delicate example of the content not matching the promise could be, a book that in the introduction talks heavily about characterisation and interaction, yet every conflict/situation is resolved throught violence.

People like to get what they pay for, I am sure that you are very aware of that. This criteria simple puts a point value on that aspect of the book. Good luck with your project.

[taalyn]

I've been following the thread, and there have been some great points brought up, but I wanted to say this:

I think it's a great idea, and I see no problem with the method of evaluation you're using. The point is to make money after all! Hopefully you'll be able to do that with games you (as a company) are excited about, but failing that, at least games you feel confident deserve to be successful, based on the care and work that has been put into them (as your "impartial" methodology determines).

You seem to be aware as well that "impartial" doesn't mean free from biases, and it isn't the only consideration, but a good baseline. Personally, I'm glad you thought enough of us to state what the considerations are out front (often, you get generic rejections with no idea what is "wrong" - at least with this list, we can try to look through point by point and try to determine how well or poorly any particular thing was done so we can fix it).

When I get the damn thing finished, I hope to submit my game to your program.

Aidan

[busling]

In a similiar line to my first reply.

Does the artwork match the content. It would be no good having H R Gieger(sp?) doing the artwork for a RPG about pixies. Or having a great pokemon cartoonist do work for a dark and gritty futuristic horror story.