In the "Conceptualizing[Cian]" thread, Ron wrote:

Ron Edwards wrote: ...
My point is that, if either were the case, it would serve you absolutely no purpose whatsoever to tell you. None. Even if you polled 9000 people for such responses, the information would be worthless. And as you expended all that energy to run the poll, and subsequently infinitely more energy attempting to apply its results, ... well, that's a whole bucketload of energy you're not spending on game development.
...

Hi,

I think the short answer is that, first, the Forge is about discussion and polls are not discussion -- they're just statements of opinion, and thus not very deep or constructive. (And, being that, would distract from what else goes on here.) Second, the sample of respondants you'd get here is going to be small and non-representative of the gaming community overall; certainly not even the online gaming community, much less all those people who have no interest in the net at all, yet game happily wherever they are.

Plus, overall, polling is a lame way to make a decision. The philosophy here is that you should make a game that you want to play, above all else. If you're not totally excited by something, don't do it. Thus, decision by committee isn't a valid/productive approach.

That's my understanding of the justification (and one I agree with). Hope that helps.

I've always understood the "poll-ban" to be about quality of discourse. Opinion-fishing is a typical part of the social dance on many internet forums, and may even take the lead role. For ex. the currently primary Finnish rpg forum at majatalo.org is a constant orgasm of questionnaire threads, where a participant throws out a question, tells his own opinion, and there we go. Properly these threads are meant to open a dialogue about the matter at hand (however banal the topics might be), but the form of the opening prescripts the course of the thread: ask for "opinions", and you get just that, with no analysis or thought. Most threads in such environment seem to be about "The best game you played", "Your favourite die size", "Opinions about Harn" and so on. It's rule by the commons, where the dialogue is constantly pushed to a level where everybody can have a voice.

Really, if you're Finnish, go to majatalo and see. It's a sight to behold. Not unlike the Forge birthday forum, but going on all year long.

The interesting thing is that I'm not seeing a blanket ban of questionnaires at the Forge. I imagine that if I wanted to have some empirical data, put up a simple questionnaire form in the web and asked people here to fill it, I wouldn't be going against the rules. The rule is against opinion threads, not polling in general. It's more about the Forge being the wrong place to do your questioning, not about the questioning per se.

I myself tend to hold the opinion that by far the most common reason for people to do polls is either
- social affirmation, like at majatalo, above
- misguided decisionmaking process
, which neither is a very good reason to start polling. So perhaps part of why we don't see polls here is that for a great majority of cases they're unnecessary. Protecting you from your own folly, perhaps?

Hiya,

Several things.

1. Disciplinary bias on my part: I consider that people trying to please the pollster or outright lying contribute far more to the spread of data in a poll than any "response" the poll is intended for.

In other words, you say, "How can 300 responses be useless?" and I say, "Because such responses are useless in any quantity." The large number of jobs and business decisions made on the basis of such "data" only means that a whole lotta jobs and decisions out there are a big scam.

And even if all those attempts to please and so on were eliminated ... so what? You are dealing with reactions, not about actually being interested, but about some vague-ass hypothetical "would be interested" reaction.

Case in point: you can ask a ton of people whether they would do X or Y when dealing with a person of A or B ethnicity. You can get tons of responses. And then if you carry out observational studies, you will find that the actual behaviors will differ radically, in both directions. (Yes, the verbally-racist person is not necessarily the behaviorally-racist one.)

2. Lack of discourse, which in and of itself means "waste of space" for this site's purposes. One might try to correct for this by saying, "Hey, give your reasons." But if a given respondent is going to explain his or her response, you might as well start with that as the discourse.

Furthermore, there is no explaining a reaction. For real discourse about the utility or point of a title (for instance), you'd need some kind of basis for what makes it interesting or not interesting, and some agreed-upon basis for how we tell it's interesting or not interesting.

Satisfying as it may be for people to react-and-type, and satisfying as it may be for the pollster to read all these posts (in terms of receiving attention), it's all a big anti-intellectual wad of slop.

Finally, I need to make it clear that my moderator/content positions might as well be considered totally arbitrary. I'm not interested in justifying them whenever called upon - it's sufficient that I'm putting in the effort and that you guys judge the effort by its results in the long term. Every time I answer a post like this, ten emails, posts, and PMs are getting shorted.

Best,
Ron

P.S. Cross-posted with Eero. He's 100% correct, and his points about meaningful questionnaires are well taken.

It is an article of faith here (the only article of faith here, come to think of it) that game design should be a labor of love. Game design should be about sharing the designer's personal vision with others.
And, when you start asking other people "Do you think this is cool? What do you think about a race of cat people in my game?" or whatever, the game moves, even just a little bit, from a product of personal vision to a product of marketing, designed to please others. And there is quite enough of that, things made to please sombodies idea of "the market" around already, in games, in books, and in movies.
We know already what games "the market" wants. Those games already exist, and there is no need for another. We don't know what game you wanted, and saw didn't exist, and had to make with your own two hands. Show us that game.

Very nice: Eero posted well, and Ron as the moderator confirmed his reasoning. I, as the other administrator, will confirm Kingston's reasoning.

The most important post you'll ever read at the Forge is right here, and it describes perfectly why I do not support polling as a useful method of design.

I understand and agree with some of the posts above, but I pose a question: Why is the information gleaned on a topic such as the effectiveness of a title useless? If all you're looking for is whether or not the title of a product projects interest in those unfamiliar with it (a "yes" or "no" answer), why would such a poll's results be considered useless without discussion?

Beyond that, your poll could be married with discussion, such as an explanation for a certain opinion. The poll simply allows for the "yes" or "no" to be easily referenced, and for certain "yes" or "no" answers (or those "none of the above" or "both" response), the author of the poll could simply scroll down to look up why a certain person answered the way they did. While not ideal for design ideas, questions of effectiveness (again, those "yes" or "no" questions) can be easily responded to in a poll.

Besides that, it's much easier for someone to answer a multiple choice poll than jump in on lengthy discussions of terminology and principle.

After all, as in the original topic posted, the question was the effectiveness of a title. Everyone has an opinion on whether or not they'd look into a game based on a title, even those that don't consciously think about it. Based on that title, they either look at the product or don't. No need for everyone to discuss what they don't consciously think about, just quick gut answers. Polling, it seems, would be perfect for this.

Besides the non-discussion problem, there is also the point that the Forge is not only a very small but extremely skewed sample.

It's just not a good place to take polls.

killacozzy wrote:
I understand and agree with some of the posts above, but I pose a question: Why is the information gleaned on a topic such as the effectiveness of a title useless? If all you're looking for is whether or not the title of a product projects interest in those unfamiliar with it (a "yes" or "no" answer), why would such a poll's results be considered useless without discussion?

After all, as in the original topic posted, the question was the effectiveness of a title. Everyone has an opinion on whether or not they'd look into a game based on a title, even those that don't consciously think about it. Based on that title, they either look at the product or don't. No need for everyone to discuss what they don't consciously think about, just quick gut answers. Polling, it seems, would be perfect for this.

As I understand it, this is basicly because Ron & Clinton want to keep the discussion going at the Forge, without diluting it in any way.

Think of it this way: an opinion thread or a poll has no significance for anybody but the originator of the thread; it only serves him, and is of limited interest to anybody else. Therefore it's hardly the kind of content one would like to have on his forums. Furthermore, a poll is movement of information (more properly, data) in one direction only, making the thread almost inevitably lesser in it's nature.

So it's not that a poll would necessarily offer useless information (although it's almost certain here, given the highly skewed nature of the Forge's participants),

In a poll you're just shooting pellets of gut instinct into the abyss, instead of reading and replying; the mode of communication is totally different.

Furthermore, consider the question you're asking: if the poll you'd like to have is truly such that it doesn't matter who the question is asked of, a genuine gut instinct question, then why ask it at the Forge? Why not put it up on your own server and direct folks there, for example? Why should the Forge, for example, host such a questionnaire? The connection seems tenuous at best.

You aren't understanding. The advice about this particular issue was that you shouldn't be needing a poll about a game name. That's not a good way to name a game. You should give it a name you like, instead. Good art is brought to being through caring, not by calculation.

I understand that this is an ideological thing to a degree. The board owners are not motivated to support game design/publishing methodologies they deem ineffective or wrongminded. They aren't interested in supporting work-for-hire because they don't like it's cultural implications, and likewise they don't support opinion polling as a game design tool. All about the kind of activity they want on their forum.

Somebody else talk to this guy, please. Here, PM, whatever.

Thanks,
Ron

LOL. I never intended to poll my question, and quite frankly, I don't even know how to create one! Someone presented a question that I was interested in hearing about, and I just disagree with you all on the validity and effectiveness of polling. I simply think it supports the design process. Sorry, I had no intention of offending you. My sincere apologies, Ron.

killacozzy wrote: Instead, as I've discovered, I've only been asked for all the details on my specific game before I can get an answer to the questions. How can such specific discussion help anyone else?.

Offending me is not possible in this venue. Others will be able to help you better than I can - so read what they have to say, carefully. I suggest waiting for a number of posts or PMs.

Best,
Ron

This is not a bearpit. Stop behaving like the bear.
If you don't understand what I mean, use the search feature.
If you don't believe you are behaving like the bear, take Ron's advice: carefully read the responses that were and have been written. Additionally, if you wonder why I accuse you of being the bear, this is the only example I should need to bring to bear:

killacozzy wrote: I just wonder why polling is thought of as useless or as counter-productive to discussion.

Even if people don't find the specific ideas helpful, you'll still see how a game is designed. And that's a very helpful thing, too.

From this, it is obvious to me that you are not reading, digesting, and examining the posts being written in response to you, or you would not have had to ask this question...again. The answer has been detailed in a number of ways above (hence, no doubt, Ron's desire not to deal with the question again).

I too think the hostility to polling is misplaced. I agree that on many boards the poll is just in effect a show of hands around the playground; but I'm equally surprised that this forum has never ventured to use the technique more constructively, rather than eliminating it outright.

No, I understand everything presented. I just disagree.

I just didn't understand the principle.... and still don't....

but if everyone's angry with the debate, let's just lock up the thread.

I'll endeavor to explain succinctly why polling is a useless tool for game design...and hense, as a site dedicated to game design, it is inappropriate for the Forge.

Polling is what you do when you lack conviction in what you are doing.

Polling is what you do when you lack the courage to take action when action is necessary.

Polling is what you do when you don't want to be held accountable for your own decisions.



Game design...good game design...requires you to have conviction in what you are doing, be accountable for the design decisions you make, and have the courage to take action when action is necessary (see numerous threads about shooting sacred cows and killing ones darlings).

Polling is an attempt to punt your responsibility as a game designer to others, to let them make the decision for you, and to be able to fall back to "poll results" to defend your choices.

For those reasons (and others) opinion polls are a ridiculous means of making decisions about anything...including game design.

Does that help you understand the principle? It isn't really necessary for you to agree with it.

All I have to say is: Well said, Ralph! That's exactly the kind of thing that keeps me going when the going gets rough. Conviction--Courage--Accountability. That's what it's all about. Bravo!

For a demonstration of those three as it pertains to game design, I direct the reader to any of a series of recent threads on Capes in which Tony amply demonstrates couragous design choices, conviction in those choices, and and ample willingness to be held accountable for them.

killacozzy wrote: I just didn't understand [your interpretation of] the principle.... and still don't....

greyorm wrote: boohoo, someone disagreed with me or criticized me, that means they're angry! -- give me a break

Valamir wrote: Polling is an attempt to punt your responsibility as a game designer to others, to let them make the decision for you, and to be able to fall back to "poll results" to defend your choices.

killacozzy wrote: Is it weak for me to ask fellow gamers which idea sounds most interesting even though I'm very excited about and, therefore, torn between them?

Look, K, your analogy is not similar enough to the current issue. It's not at all like drinking milk when not thirsty. It's more like you going to someone's house, and they don't allow shoes to be worn inside. You ask why, and they say "because I don't want people tracking dirt into my house." You respond that wearing shoes is easier than taking them off at the doorway and has lots of benefits and it really doesn't track dirt in, and you don't understand the logic of their statement. They look at you and say, "hey, this is my house, and I don't want people walking around wearing shoes because I believe that tracks dirt all over, end of story." And you keep saying, "but why?" And they keep saying, "because I don't want dirt in my house." No communication is happening, on either side. But at the end of the day, we're in Ron and Clinton's house here, so their rules go.

As other posters have pointed out, if you want to invite folks to your house where they are free to wear shoes (i.e. create your own site, where polling is allowed), go for it.

Hmm... I seem to remember that Matt Snyder polled us all just a while ago about which game project he should be finishing next. It's an interesting concept, certainly - can he keep up the convinction and interest in Dreamspire now that the masses have spoken? Or should he have listened to his inner demons for inspiration?

Anyway. It's OK to disagree about stuff. I don't think that anybody has any great passion about polls (those things phpBB allows you to put at the start of threads), but they just don't currently belong in the Forge repertoire. I'm sure that Ron and Clinton have evaluated this thread, and they could even change the rules. Or not, if they think it's better to not have them. What I'm pretty sure about is that nobody wants to have polling threads (which do not necessarily have polls at the start), which replace discussion with opinionating, with rare comments on other opinions and almost no analytic summarizing. That's the thing we're railing against, here.

As I explained previously, a big part of it is ideological. Many folks here, including Ralph and Ron, subscribe to the notion that polling in any form is an ineffective tool of design. It's an article of faith, really. About commitment and belief in yourself. About the legend of the artist versus the technocratic entertainment industry. What do you expect, this being an indie forum ;)

Anyway, I at least find that part of the discussion interesting - could one gain anything useful from polling, and if so, how'd you go about it? Would the poll need to comply with the rigor of statistical science, or is there design/publishing levels of methodology which would actually get by with less? What kind of useful information could one hope to collect in that way? How would the information be used?

But for the time being, one thing Forge does very rigorously is differentiating between discussion and polling. If you're really just interested in opinions and not discussion, at this point what you should do is put the poll up somewhere else and invite people to participate. You can then bring the methodology and results to the Forge for dissection, which has actually been done a couple of times. So discussion about polling is OK, discussion about results is OK, but actually polling people is not. I'd imagine that it'd be more OK if you were really obvious about it and asked Ron's permission beforehand and the poll pertained on the Forge's mission in some way, but I wouldn't hold my breath.

When you stop to think about it, we do do all kinds of polling and other statistical work. There's those "what game did you play last" things, and Ralph's quarterlies about the Forge's statistics, for example. And then there's the Indie Awards, which are a big voting thing held just outside Forge event horizon... the point is, anti-poll attitude is not dogma, it's something held for a reason. And the line is drawn again at need. So if there's good reason for allowing or encouraging certain kinds of polls, I would think that that will happen.

But yeah - in the end it's about Ron/Clinton's vision for the site. Convince Ron about the benefits of polls, and they happen.

[Andrew: a killer analogy there! Nothing more to add.]

Quick point about Forge policy: moderators of specialty forums, e.g. Chimera Creative, Adept Press, etc, have full power in those forums, and I have none, except in the HeroQuest and Adept Press ones. So if they want to run polls, or really do anything, like post animated GIFs of themselves dancing naked, it doesn't violate the Forge policies for the general discussion section.

Best,
Ron

K, I'm not accusing you of being a lazy designer.

Let me illustrate using your own question regarding the name Cian.

Cian is the name you chose for your game. If that's a name that means something to you...if it resonates in your gut...if it just plain fees right...then its right. If you poll 1000 people and a 1000 people tell you its wrong...then the poll is wrong. Thats what I mean by having conviction. If you are convinced that Cian is the right name for your game...than damn the torpedoes and damn the response of any poll.

There are only 2 reasons to want a poll on a question like that. Either you lack conviction that Cian is the right name for your game...in which case you already know the answer...if you're not convinced its right, then it isn't. Or you're under some misguided assumption that even though you think its right that other random people at large would know better than you. That's just a bad assumption and not at all true.

Pick a name that when you hear you know its right. If Cian is it keep it. If Cian isn't it, change it. No poll can ever answer that question for you.

This is even MORE crucial when you start talking about design elements more important than a name.

Brendan wrote:

killacozzy wrote: Is it weak for me to ask fellow gamers which idea sounds most interesting even though I'm very excited about and, therefore, torn between them?

Yes.

Andrew Morris wrote: Look, K, your analogy is not similar enough to the current issue.

Andrew Morris wrote: It's more like you going to someone's house, and they don't allow shoes to be worn inside. You ask why, and they say "because I don't want people tracking dirt into my house." You respond that wearing shoes is easier than taking them off at the doorway and has lots of benefits and it really doesn't track dirt in, and you don't understand the logic of their statement. They look at you and say, "hey, this is my house, and I don't want people walking around wearing shoes because I believe that tracks dirt all over, end of story." And you keep saying, "but why?" And they keep saying, "because I don't want dirt in my house." No communication is happening, on either side. But at the end of the day, we're in Ron and Clinton's house here, so their rules go.

As other posters have pointed out, if you want to invite folks to your house where they are free to wear shoes (i.e. create your own site, where polling is allowed), go for it.

Valamir wrote: Let me illustrate using your own question regarding the name Cian ... There are only 2 reasons to want a poll on a question like that. Either you lack conviction that Cian is the right name for your game...in which case you already know the answer...if you're not convinced its right, then it isn't. Or you're under some misguided assumption that even though you think its right that other random people at large would know better than you. That's just a bad assumption and not at all true.

Furthermore, it just forces the faithless, the hacks without conviction to go into discussion areas with questions like "my game's about dinosaurs—should I call it DinoMite?" That, in itself, seems to defeat the purpose.

K, I believe polls have value, in certain situations, and if certain conditions are met. Do I think there's value in discussing whether or not polls should be allowed here at the Forge? Not really. If I'm so all-fired up about the utility of polling, I'll start a site dedicated to just that purpose and send any interested parties to it. Likewise, do I feel there's a value in discussing why polls aren't allowed? Again, not really. If I feel it's that vital, my remedy is very simple -- go out and do it myself instead of questioning why someone else doesn't want to do it.

killacozzy wrote: I wasn't fighting to "wear shoes", I was simply discussing (or thought I was discussing) an alterate method.... like maybe a... "mat" to "wipe your shoes on"(?).... all for the sake of discussion. It's not about me not wanting to abide by law, it's just that the only way to make progress is to challenge the status quo. You make your "faith" stronger by questioning it.... and if your "faith" crumbles, then get another "faith".

Valamir wrote: Its conceivable (quite likely actually) that you could uncover a way of discussing the name of your game and the process involved in naming games in general in a way that is not shallow and not unsubstantial and that WOULD make for an entirely appropriate and appreciated topic for the Forge. The thread in question wasn't it...it was a poll. Simple as that.

killacozzy wrote: My first issue deals with a product's name. Do you think a game's title necessarily has to describe it? Examples that come to mind would be Dawnforge or Dungeons and Dragons, and if you'd never heard of either, you still may have an idea about the game inside. The converse would be games like Amber or Conan, which I suppose would require a bit of name recognition and familiarity with the source material.

Shreyas Sampat wrote: Now, regarding titles; are you asking for people to compare their experiences with name-recognition against easily interpreted titles? That might be a good topic for a separate thread.

Andrew Morris wrote: I'm not likening your position to insisting on wearing shoes in the no-shoes house. I'm likening your position to asking the house owner why they have that rule, after they've explained their reason...several times. Simply because you don't agree with their reason in no way lessens the fact that the question has been asked and answered.

Oh! It appears I wasn't polling, but discussing something.

Valamir wrote: You've been told by Ron Edwards content moderator of the site that he considers your post to be a poll and therefor not suitable for discussion here. Period.

OK, so the issue has been settled re polling in the Forge. But, don’t feel ostracized, Killa. The Forge takes getting used to, but has great value if used properly. The Forge is like a graduate level philosophy class versus a 100 level undergrad one (as in some other RPG forums). In the latter, discourse occurs, but very sloppily. In the former, discourse methods common in the latter are an annoyance. But, the choice of the vets is to either get frustrated with the newcomer and (consciously/unconsciously) drive him away, or to patiently lead him to grasp the new paradigm. You are new here, which means you will inevitably make more “undergrad” errors. On the other hand, the vets should keep their annoyance under their hats until you either adapt (or until it is clear you refuse to do so).
So, stick around, learn the feel of things around here, and you’ll benefit from it for your game.

I don't know if it will help at this stage, but I think it's worth revisiting Ron's statement that the poll of any size is worthless. That's the only thing anyone has said here that I don't think CAN be disagreed with and yet it doesn't seem to be carrying any weight.

Let's look at it this way: I want to do a poll on how pornography affects a marriage.

I decide that the people with the most experience on this are probably married couples who subscribe to Playboy, so I run an ad in the magazine to get responses. "Wow," I say, "it looks like 75% of people are helped by pornography and with a sample of 5000 responses, it's sure to be accurate!"

Just to confirm that, I take the poll over to a giant Baptist church. With 8000 people attending, I find that only 1% of the people said it helps.

Which poll provided valuable information?

Neither. Why?
1) Neither sample is representative of the population.
What population? I didn't even define what population I wanted to test my question on. As phrased, technically it's all married people around the world.
2) There was no control over the application of the survey.
Just think about this one... a guy comes up to his wife and says "Hey, honey, this guy in Playboy wants to know if we think porn has improved our marriage?" If he's that open about it, they probably think so. But what if he answered only based on his own perceptions? Do we know that the Baptist poll wasn't skewed by people being worried that others might see what they voted?
3) The poll doesn't test anything useful.
If we're just asking whether porn has improved a marriage, we've only got opinions. Someone might be saying "Yes" as a way to try to convince themselves it's working even as they cry themselves to sleep at night. Someone might be saying "No" without ever having tried it. In neither case do you really know how good their marriage was or what made it that way.

Now, let's look at the Forge.
1) The Forge is not a representative sample of anything.
Some of you know this from experience, because you ask the same question on RPG.net that you ask here, and you know that you get two totally different answers. The Forge isn't even representative of itself because a lot of people who read never post.
2) Again, there's no control over the application of the survery.
I've got a dirty secret. I like taking polls. I liked RPG.net's Poll forum because I could find all the polls and go answer all of them. All of them. Even the ones that are trying to only ask women or only D&D players. If they want me to read something and I don't have time, I'll just vote a snap impression. I just like voting, so I do. I don't know how many people are like me out there, but when many polls only get 20-30 votes, two of us will skew your results by 5-10%, enough to make multiple choice polls meaningless. (And, since people like me like polls, you can bet that we're a higher percent of poll respondents than we are of the general population). There are other issues, too, like whether people are voting multiple times, whether they read the question correctly and whether they actually clicked the right button.
3) The poll still doesn't test anything useful.
Don't believe me? I worked for a university that wanted to do a scientific survey of students regarding eating disorders and vegetarians. To make their results valid, they had to design it using software that cost them $3000 (it would have cost a company $12,000). Then they consulted with a stats guy, ran a sample survey, consulted again, ran another sample, consulted again and finally did the study, which the software ruled as "inconclusive". Heres an RPG-specific example. You ask people whether they like a name and 80% of them say "Yes!" You think you've got a slam dunk, right? Why would you think that? By thinking that you have a slam dunk, you're already applying the results of that survey to sales. But you didn't ask about sales, did you? You're just inferring information that isn't there because you think you know what polls do. (You're not alone... the news media do it every four years when they pretend that candidate approval polls predict how people will vote for president. They're wrong every time.)

So, regardless of Forge policy, I wouldn't use polls for anything other than enetertainment.

Guys, this ought to go to a new thread if people want to break it out into sub-issues or similar stuff. This thread reached a kind of natural end a month ago, as I see it.

Best,
Ron