As part of our new Eureka! Publishing & Design Program I am going to be providing a review/critique of submissions. Each topic/element reviewed will be given a rating from 1 to 5.

The idea is that products will have to receive a minimum score (of, say, 3.5) in order to be accepted into the program. The general scores used would be:

1 = Poor (extremely poor quality, technically inferior or amateur)
2 = Below average (sub-standard compared to existing products)
3 = Average (on par with existing products; acceptable quality)
4 = Good (exceeds market expectations; quality stands out)
5 = Excellent (greatly exceeds market expectations; raises the bar)

I have some of my own ideas, obviously, about what elements should be evaluated when considering a product for publication. Among them:

* Marketability (is the genre a fresh twist or cliche?)
* Cover illustration
* Interior art
* Organization
* Writing (incl. style, voice, etc.)
* Interior design (readability, graphic design, attractiveness, etc.)
* Editing
* Playability (fun factor)

While I was sitting at my desk trying to think of the various elements of a submission that should be reviewed (i.e., scored), it dawned on me that I could get some suggestions from indie designers.

So here I am, hat in hand, asking you folks this question:

What other elements do you feel should be evaluated when considering a product for publication?

Well that is a question.

By that do you mean purely as in taking submissions and if they are good they go out with your companies name on it?

Or, as a third party publisher?

If it's the former, put whatever criteria you want, it is your name and reputation on the line. Although I would put a listing of resources you can help provide should someone not have them available to them, such as artwork and the like.

If it's the latter, I don't know. Can you really quantify things like that to an independent person coming to you to get their work published.

I guess it would help to know how you differentiate the two?

Sylus

I think you're doing yourself a bit of a diservice with this. I mean, you can have some kind of system in place to decide what games to risk you capitol on, but who's to say what will sell and what won't? The Beatles were turned down by nearly every recording company in the industry. One even told them that guitar groups were on their way out.

Success is about 5% skill, 10% hard work and 90% luck. Yes that does add up to 105%. Success is wonky.

Hi Mark,

My main question is what the games are being compared to. If it's to one another, then I think your scale will be too speculative to mean much - your initial impression of the games going in will be the same as the one coming out, just quantified a little.

It would be interesting to run a few role-playing games out there on the shelves through your metric, and use those scores as the framework for comparing new submissions. I strongly recommend, however, being very careful about scoring "marketability." You probably know that I think RPGs true market has never been assessed or perceived by the trade magazines, distributors, or retail community.

Best,
Ron

Okay, maybe my question wasn't phrased well. I'm not really looking for feedback about the fact that we are going to evaluate submissions to the Eureka! program. That's a foregone conclusion.

For information about the Eureka! program, please see this thread:
[urlhttp://www.indie-rpgs.com/viewtopic.php?t=6210

Apologies if anything I've posted has been confusing or ,isunderstood. Sometimes I stayup late working and don't maintain a sufficient supply of coffee. ;)

To Sylus Thane:

What I meant about accepting submissions was for our Eureka! program, which is a new imprint we've created for publishing creator owned indie games.

We will be putting our new Eureka! logo on the books but also allowing the designer to have their name and logo on the book as well. So it's sort of a hybrid of the options you mention. Does that make sense?

To Jack Spencer, Jr.:

Who's to say what will sell and what won't? I don't know. What I need to determine is what my company is going to publish, promote, market and sell through distribution channels.

Will I always make the right choice? I hope so, but I may not. I'm human. That's why I'm trying to come up with a somwhat impatial method of evaluating product submissions. Plus I believe the critique would be of value to the designer, as well.

To Ron Edwards:

The submissions would be compared to existing products in the market. So a submission that had art of worse quality than the average product out there (in the reviewer's opinion) would get a score lower than 3. If the art was better than the average product (again, in the reviewer's opinion) it would score higher than a 3. And so on.

I agree that it would be interesting to run a few existing role-playing games through my rating system and use those scores as the framework for comparing new submissions.

You suggested I be very careful about scoring "marketability." I agree. I don't even know that I will keep that category. I just threw out some ideas to spark discussion.

So try this, all. Forget about what I posted in terms of categories. What categories would you evaluate a product on if the designer asked you to publish, market and distribute it?

Gold Rush Games wrote: That's why I'm trying to come up with a somwhat impatial method of evaluating product submissions.

Jack Spencer Jr wrote: I, personally, do not like the idea of an impartial method for evaluating submissions.

I would rather place my chips on something I personally believed in and felt strongly about and fail than to have a successful product I don't feel anything for be the biggest game ever but I published it based on an impartial system of evaluation.

That said, it's easy for me to say this when it isn't my bananas on the line, right? I see nothing wrong with such a thing being *part* of the decision-making process, but be willing to chuck it, right?

Greetings Gold Rush Games Rep,

Gold Rush Games wrote: As part of our new Eureka! Publishing & Design Program I am going to be providing a review/critique of submissions. Each topic/element reviewed will be given a rating from 1 to 5.

<...>

What other elements do you feel should be evaluated when considering a product for publication?

Kester Pelagius wrote: Greetings Gold Rush Games Rep,

Quite frankly, if anything, I think the sort of thing you need to review is what is presented in the material first and foremost and worry about that before you even consider issues of style, prose, grammar, or anything else.

Too, for starters, this thread needs a direct (working) link to the pertinent information* about the Eureka! imprint/publication deal...

Otherwise, to play paranoid devil's advocate, I'd have to ask what is in it for the author?

...if they have the impression they are paying you to print their material then all they'll likely want is for you to print what they have.

...I'd still want to know what rights are involved in the deal.

Most of us have probably read your posts here and at RPGnet, but what about the newbie just signed up members who might not have?

You could put a direct link in your sig, if you want, no pressure. Just a suggestion

For instance what would turn YOU off to a submission?

If a author asked you to sign a non-disclosure form would you reject their MSS out of hand?

What if the MSS being submitted had already been "garage" published in a limited run or two?

If a MSS submitted already had copyright filed (and the author made a point to let you know) what would be your response?

You mentioned artwork, does this mean you will be checking all illustrations used to be sure that artists copyrighted and intellectual material is not being stolen?

If a author wanted to negotiate the boilerplate in your contract BEFORE submitting anything to you, would this impact how you viewed the authors submission?

For that matter will you be reviewing the MSS to ensure that no copyrighted material- such as mention or use of established worlds and derivative works of fiction that may infringe upon intellectual properties- are used?

If you find that submitted MSS do, in some minor or major way, use copyrighted material, however incidentally or accidentally, will you work with the submitters to help them fix the errors in the submitted MSS or reject the material?

I am sure you will make certain your legal department is going to give the MSS the once over to cover your arse...

Sounds like a headache to me but, hey, God favors the compassionate!

Much luck. I apologize if it sounded like I was trying to break your balls.

Mark,
This is just a pet peeve but you have defined subjective criteria and called it objective. I know you want to sound as positive and impartial as possible but unless you will be referring to an industry standard (I do not know of such a standard) then it will be what you perceive as an industry norm. In the end, you will publish what you believe to be winning submissions by your criteria. It is admirable of you to discuss what your internal criteria will be but they are your criteria.

Secondly, the most subjective of criteria, cliché genre will most likely be your largest downcheck. Again, this is your perception of cliché yet you represent it as an objective and impartial criteria. No problem using a subjective criteria because, again, it is your company.

Just as an aside, we hve done much of the benefits you have described and it has not cost us thousands and thousands of dollars. I believe POD is viable as a JIT modeled supply chain. If employed correctly it will yield a smaller per unit net but have a considerably lower break even.

Please do not take this post as a personal attack. It is merely pointing out what some posters may not understand (myself included 8-D). In the end, it sounds like you are offering a service to allow writers to POD Publish and that is a good thing. If you read this post with an accusatory or inflammatory tone in your mental voice read it again with a neutral or chatty tone and you will get my meaning.

Keep up the good discussion, it ALWAYS helps.
Bill

HinterWelt Enterprises
The Next Level in RPGs
William E. Corrie III
www.hinterwelt.com
http://www.hinterwelt.com/chargen/

Gold Rush Games wrote: After all, no one would want their submission rejected because the reviewer simply didn't like the genre, would they? Maybe someone would, but I can't see it.

But are you speaking from the perspective of a publisher or as a designer?

So you feel that it is a waste of my time to solicit feedback on review criteria from the people here in this forum? Gads. I've been criticized before for not caring what my customers think. Never before have I been criticized for caring what my potential customers think. ;)

Mark,

Another point that you could review the submission on:
"Content matching the promise"

eg. "101 Spell books" not having any spell books would be a complete failure in this department.

Considering that you will have control over the marketing and such this point should not be too much of an issue.
A more delicate example of the content not matching the promise could be, a book that in the introduction talks heavily about characterisation and interaction, yet every conflict/situation is resolved throught violence.

People like to get what they pay for, I am sure that you are very aware of that. This criteria simple puts a point value on that aspect of the book. Good luck with your project.

I've been following the thread, and there have been some great points brought up, but I wanted to say this:

I think it's a great idea, and I see no problem with the method of evaluation you're using. The point is to make money after all! Hopefully you'll be able to do that with games you (as a company) are excited about, but failing that, at least games you feel confident deserve to be successful, based on the care and work that has been put into them (as your "impartial" methodology determines).

You seem to be aware as well that "impartial" doesn't mean free from biases, and it isn't the only consideration, but a good baseline. Personally, I'm glad you thought enough of us to state what the considerations are out front (often, you get generic rejections with no idea what is "wrong" - at least with this list, we can try to look through point by point and try to determine how well or poorly any particular thing was done so we can fix it).

When I get the damn thing finished, I hope to submit my game to your program.

Aidan

In a similiar line to my first reply.

Does the artwork match the content. It would be no good having H R Gieger(sp?) doing the artwork for a RPG about pixies. Or having a great pokemon cartoonist do work for a dark and gritty futuristic horror story.

Good suggestions.

Which do you folks think would be better? A 1-5 score or a binary (Pass/Fail) score?

I can see advantages and disadvantages to both, but I'm curious what you think, too.

Greetings Mr. Arsenault,

Gold Rush Games wrote:
Part of the requirement for a submission to be considered for the program is that it has to be "ready to print," though. So we'd be evaluating the whole package (writing, style, design, "look," etc.) at once.

Otherwise, to play paranoid devil's advocate, I'd have to ask what is in it for the author?

That's not paranoid. It's good sense to ask such a thing.

In summary, what's in it for the author is:



• POD publication of their book (with their imprint/logo)
• Client maintains copyrights (we just license publishing rights)
• ISBN assignment & barcode
• Listing in R.R. Bowker's Books in Print database
• Listing in the Ingram Book Co. database
• Listing on Amazon.com, BN.com and others.
• Review copies sent out to those on our review list
• Association with an established publisher (est. 1995)
• Distribution sales via our sales office (Tundra Sales Org.)
• A percentage of all sales of the book
• Representation (and potential sales) at cons we attend
• Listing in the GRG catalog, on the GRG web site, etc.
• Marketing (not exhaustive, but comparable to what give our own products)
• A much cheaper alternative to traditional printing

Gold Rush Games wrote:

For instance what would turn YOU off to a submission?

Poor quality art, writing, editing, a cliche product for an overused genre, and so on. We're looking for quality, innovative game products from indie designers. The good stuff.

Gold Rush Games wrote:

If a MSS submitted already had copyright filed (and the author made a point to let you know) what would be your response?

Registering a copyright offers no more protection to a work than its mere creation and recording in tangible form. All works (in the U.S.) are protected the moment they come into being.

With that said, a creator having a copyright registered already tells me that they are serious, know at least something about copyrights, and otherwise has little bearing on our consideration of whether to publish their book.

Gold Rush Games wrote:

You mentioned artwork, does this mean you will be checking all illustrations used to be sure that artists copyrighted and intellectual material is not being stolen?

All art will be reviewed, but not to ensure protection of IP. The publishing agreement has language in which the Client guarantees that their product does not violate any third party's IP rights.

Gold Rush Games wrote:

If a author wanted to negotiate the boilerplate in your contract BEFORE submitting anything to you, would this impact how you viewed the authors submission?

Not how we viewed the submission, but it could have an affect on our decision whether to accept it into the program or not. It would depend on what the specific item was.

Gold Rush Games wrote:

For that matter will you be reviewing the MSS to ensure that no copyrighted material- such as mention or use of established worlds and derivative works of fiction that may infringe upon intellectual properties- are used?

Client declarations, guarantees and warranties are written into the contract. It would be a bad idea for a Client to submit material that violated another party's IP rights.

We do reserve the right to reject any submission, howeve, including for reasons involving legal liability.

Gold Rush Games wrote:

There's probably a dozen other questions I could ask, but I think that suffices to illustrate the potential problems.

I see no problems thus far. Feel free to keep the questions coming. All I ask is that you review the "Draft Agreement" and the "POD Costs & Royalties" documents before posting more questions, in case they are answered in the documents.

Mark,

In my experience if you grade them with a specific number at the end it will help to make your final decision. Especially when there are several products that are similiar.

Hi Mark,

My first impression, based on your first post, was to recommend removing the numerical scale or collapsing it to 1-sucks, 2-OK, 3-good. But then I decided that 4 would be better - so that you really have to bite the bullet between 2 and 3. A mid-value on numerical scale tends to create an artificial cluster as the rater grows lazy. A poorish product with a great feature or two, and a pretty good product with a fatal flaw or two tend both to get rated "3" on a 1-5 scale.

Also, and this is a horribly unscientific anecdotal sort of thing, I recommend not going above a scale of 4. Once past 6 increments, issues such as the difference between "very average good" and "average average good" start cropping up.

Best,
Ron

Hi Mark,

The list of criteria look fine on the surface. But you're implying that you then intend to average the scores and compare the average to a minimum threshold.

The idea is that products will have to receive a minimum score (of, say, 3.5) in order to be accepted into the program...

* Marketability (is the genre a fresh twist or cliche?)
* Cover illustration
* Interior art
* Organization
* Writing (incl. style, voice, etc.)
* Interior design (readability, graphic design, attractiveness, etc.)
* Editing
* Playability (fun factor)

Gold Rush Games wrote: So you feel that it is a waste of my time to solicit feedback on review criteria from the people here in this forum? Gads. I've been criticized before for not caring what my customers think. Never before have I been criticized for caring what my potential customers think. ;)

Kester Pelagius wrote: So, though it is not explicitly stated, we can assume that this is a program strictly for books. No "boxed sets" or RPGs with "board game" or other gimmicky elements (like cards) need apply, yes?

Also, as an aside, what manner of binding are we talking about here?

Will authors have a number of choices to choose from ranging from, say, perfect bound to saddle stitched?

And what kind of paper will be available within the book? For the art? Covers?

Incidentally, will there be galleys or proofs made available for review by potential customers?

The main thing here will be a response of "Who is to say what is a cliche or overused genre?"

Every author will (hope and want to believe) their MSS has that extra *something* that sets their endeavor apart,

but if that work happens to fall into a genre category you aren't even going to accept submissions for that month. . . Loss of potential customer.

Or, from a writer's point of view, if you say your magazine is "open to all submissions" but then have a form rejection letter that simply states that you did not like the work, personally, as a matter of your own reading taste what you've just stated is that, as an editor, I, the writer, need to entertain you.

Say I have a game, X, that I have both registered copyright and trademark for. Would this make it easier for you or more difficult from the published end of things to work with?

Would you prefer that authors who approach you have their MSS ready to submit and print with all the bases covered, or at least as many as possible?

Ok, but what if a third party sees a book published by your company that has artwork in it that they feel is derivative (or a blatant rip off) of something of theirs and decide to file suit against your company.

Are you sure that such a disclaimer will protect you?

So, to be accepted into the program, you're saying that the author would have to first sign the contract??

But what about something done in parody or spoof?

...but what if someone plagerizes a obscure work that no one reading your slush that week is familiar with?

...what I want to know is how one mediates disputes between the author and the publishing house if a problem arises.

As for the rest of if. . . Kiddies, if you're reading this, and haven't read the PDF, or have and don't know what to make of it. Speak up now while the man is here. It's six pages and says a lot, but what does it really all mean? Do you know?

Curious about why it refers to NET and not GROSS?

Want to ask the man how this program will differ from going to the local Kinko's or vanity press?

Walt Freitag wrote: Hi Mark,

Averaging says that an excellent cover illustration can make up for poor playability, and that the cover plus good interior art and graphic design can make up for abysmally awful playability.

It also means that weakness in art or organization or writing style could keep a product with marvellous playability out (**unnecessary personal criticism deleted**)

So, truly excellent art or playability alone could get a product in, provided it was at least adequate in all the other areas.

Greetings Mr. Arsenault,

Hope all is well in your part of the world this fine day.

Per your orignal question. . . Mr. Edwards has a point. If you were offering a editing service a scale of 1-5 (or whatever) would be very helpful to the author, for sure. But if this is purely a classification for you, as publisher, to decided "yeah or nay" about a submission then maybe that is what you should go with. Keep it simple. Unless you would like to provide some feedback to authors so they can get a better idea of why their submission wasn't accepted, in which case such a scale would likely be very helpful, if taken in the spirit intended.

Thus the questions that need answering are probably 1) which would work best for you, and; 2) what do you intend to do with the information after you review a submission?

If it's going to stay "in house" then keep it simple. Just my opinion.


Now, on to. . .

Gold Rush Games wrote:

but if that work happens to fall into a genre category you aren't even going to accept submissions for that month. . . Loss of potential customer.

That would not happen. We would not ignore certain genres in certain months. That's silly. What I mean by a cliche or overused genre is, say... "A fantasy game. Just like D&D but better."

Gold Rush Games wrote:

Or, from a writer's point of view, if you say your magazine is "open to all submissions" but then have a form rejection letter that simply states that you did not like the work, personally, as a matter of your own reading taste what you've just stated is that, as an editor, I, the writer, need to entertain you.

No, as a publisher you need to sell you product to me and I, as the publisher, need to belileve that the product would appeal to our customers. There's a big difference between that and "entertaining me."

Eureka! would have no obligation to print every submission that comes its way and I have no intention of doing so. There has to be some sort of evaluation criteria to weed out those products that simply aren't up to par in terms of quality.

Gold Rush Games wrote:

Would you prefer that authors who approach you have their MSS ready to submit and print with all the bases covered, or at least as many as possible?

If you're referring to copyright and trademark registration, it doesn't matter, really. If you're referring to something else, perhaps you could restate your question more clearly.

Kester Pelagius wrote: Unless you would like to provide some feedback to authors so they can get a better idea of why their submission wasn't accepted, in which case such a scale would likely be very helpful, if taken in the spirit intended.

Thus the questions that need answering are probably 1) which would work best for you, and; 2) what do you intend to do with the information after you review a submission?

You're saying, and correct me if I'm wrong, that over used themes will have a lower chance of acceptance...

...if I have a MSS set and ready to go, with grahpics and title (and the sort of things that can have a SM or TM applied to them) would you prefer to see such MSS as, say, opposed to just a MSS that hasn't had artwork and titles worked up for it yet?

Thanks for taking the time to reponsed and clarify.

Hi Mark,

My apologies for any unnecessary personal criticism. If you had any role in the writing, art, production, or publication of the books I mentioned, I was unaware of it and I certainly did not select that example based on any perceived connection to you personally.

The point I was trying to get at is that your list emphasizes style over substance.

When you average a bunch of ratings with equal weighting, those overall aspects of quality that are represented by the most specific and numerous criteria are emphasized. To show what I mean, I could shift the emphasis the other way by grouping Cover Illustration, Interior Art, and Interior Design into a single list item (e.g. "Visual Presentation") and expand Playability into three list items, such as: Accessibility, Usability (game design quality if it's a game system, suitability for the system if it's supplemental material), and Depth (sustained play value over time).

But instead of going that far, how about just evening the playing field:

Content
* Game design functionality OR Suitability of material for stated purpose
* Writing (inc. style, voice, etc.)
* Accessibility (organization, clarity, completeness)
* Sustained play value (relative to price)
* Originality
Presentation
* Cover illustration
* Interior art
* Interior design
* Editing
* Integration of elements (art is appropriate to content, etc.)

As the criteria list currently stands, you can take away one item and you'd have suitable rating system for a coffee table book, a home-and-garden magazine, a clothing catalog, or a corporate annual report. One item out of eight reflects that we're talking about games and materials to be used in games.

It makes me sad to think that when I buy a cookbook, the publisher might have chosen it for publication based on criteria just like that, except with "How well the recipes actually work" substituted for "Playability." It makes me even sadder to think of textbook publishers who might be stuffing "Subject Matter Accuracy and Completeness" into that lonely eighth slot.

I suppose this just reflects the reality of the modern marketplace which you can't escape. But I still find it unfortunate. That's my opinion, which you did ask for. Nothing personal.

- Walt

Walt Freitag wrote: My apologies for any unnecessary personal criticism.

The point I was trying to get at is that your list emphasizes style over substance.

When you average a bunch of ratings with equal weighting, those overall aspects of quality that are represented by the most specific and numerous criteria are emphasized.

But instead of going that far, how about just evening the playing field:

Content
* Game design functionality OR Suitability of material for stated purpose
* Writing (inc. style, voice, etc.)
* Accessibility (organization, clarity, completeness)
* Sustained play value (relative to price)
* Originality
Presentation
* Cover illustration
* Interior art
* Interior design
* Editing
* Integration of elements (art is appropriate to content, etc.)

As the criteria list currently stands, you can take away one item and you'd have suitable rating system for a coffee table book, a home-and-garden magazine, a clothing catalog, or a corporate annual report.

I suppose this just reflects the reality of the modern marketplace which you can't escape. But I still find it unfortunate. That's my opinion, which you did ask for. Nothing personal.