Getting accurate estimations from players

[Hudson Shock]

There's a contest for the captain with the slowest boat.  Two captains enter.  How do you prove who actually has the slowest boat, since either captain can purposely go slower than the boat is capable of?

Answer: Have them race, but make the captains switch boats.


This is a riddle I read in an economics book a long time ago about tricks economists have for getting the truth out of people.  I'm wondering if I can get any similar insight from people here.

Say I have a system that allows people to pick any sort of skill or attribute for their character they like, from "Terrified of open spaces" to "Moderately fluent in French" to "Perfect Master of the Sun Source of All Martial Arts".  Let's rate their utility from -10 to 10:  -10 being something that is completely debilititating in almost every situation, and 10 is something that is incredibly useful almost all the time.

What I'd like is to get each players honest evaluation of how valuable each of the attributes they've made up.  But I don't want to limit it in any other way - if they want to load up on descriptors that would all rank at a 10, then good for them.  All I want is some way to get them to admit it - I don't want them to "downplay" their utility.

(I also recognize that this does not reflect an "objective" value.  A person may rate being a catburglar as very useful, but obviously if the adventure is all about court romance, it will be less so.)

Do any of you know any clever ways - along the lines of the slow boat race above - to pull out this sort of honest evaluation?  I've tried pricing strategies: making people bid for attributes.  The problem becomes that I don't want people to never be able to afford an attribute.  The idea is not to limit, but simply to measure.  I wonder if an answer might be found in some sort of auctioning system, but I run into the same problem, as well as the issue of players conspiring to underbid everything.

[Andrew Martin]

I've got friends who regularly change character disadvantages into advantages and turn characters advantages into disadvantages through skilled play. They can make a mockery of character advantages and disadvantages. Wouldn't any cost/buying system that works in the design stage of the character design be made useless by this kind of play?

What I'd like is to get each players honest evaluation of how valuable each of the attributes they've made up.

Unless the player is being illogical, or doesn't know what the group is playing, then logically pretty much all the attributes and skills of the character are going to be equally important to the player. After all, in a system where there's no guidelines: "a system that allows people to pick any sort of skill or attribute for their character they like", automatically implies that all skills and attributes that the player picks are equally valuable.

And if you are going to allow Design in Play (DiP), then as the player picks up and drops more descriptors for the character, logically those descriptors that are dropped or forgotten have become less important to the player (important to not important at all), and those descriptors that have been added have become more important to the player (not important by not being present to important).

So one doesn't really need to rate the character descriptor's importance on a -10 to +10 scale, as importance to the player is simply rated by presence or absence of the descriptor along with a plain English label of character competentence or incompetence as appropriate for colour. For example, "Terrified of open spaces" is worse than "frightened of open spaces" which is worse than "uncomfortable in open spaces" which is worse than "comfortable in open spaces" which is worse than "uncomfortable in closed spaces" and so on. Similary for "Moderately fluent in French", which is better than "knows a few words of French" which is better than "can't speak French".

[Marco]

Having thought a good deal about this, I think there are several potential answers--but no really good ones.

Firstly: A player's (IMO logical) assessment of how good an ability can be based on how good the character is in relation to the standard and how easily that ability will be brought into play (there are clearly other ways to estimate--I think this is a resonable one).

This is why combat abilities are often more expensive than non-combat ones: force can be used as a means of problem-solving (both in reality, unfortunately and in RPG's) whether or not the situation is inherently violent. If the GM doesn't bring combat "to the table" many (IMO common) RPG situations allow the player to initate it (people say fighting in Call of Cthulhu is useless--but even excepting cultists, a real bad-ass who finds himself run afoul of hostile townsfolk might further the groups aims by taking a few of them down handliy when tempers reach the breaking point).

How good you are in relation to the standard is important too: Having basic cat-burgular skills may mean that:
1. From a let-the-dice-fall-where-they-may standpoint (assuming the world is run "realistically") that the character can't expect to achieve really spectacular results (the crown jewels cannot be stolen by an average thief or it'd be gone already). Hope of spectacular results usually requires spectacular skill or hope for an unusual situation.

2. From a dramatic standpoint, while character's minor abilites sometimes sway the day, characters are more often (I'd say) successful for their primary traits. Han Solo is successful because he's the best pilot going, not because he's a mediocre space-chess player. There are many counter-examples of this, however, which brings the next point up.

Secondly: A person can judge the value of an attribute by measuring how committed the GM is to making the character's traits germane to the story. If the GM is committed to going down the character sheet and going "You speak French? Okay, the international spy is from France ..." then it's more important than if the GM runs a module and doesn't adjust it to the character.

Thirdly: Based on setting, genre expectations, and experience with the GM's representation of imaginary worlds, the players may have an image of a "standard case" of the game that is to be run. A game set on the surface of Dune will see little use for fishing and naval navigation skills. A game set on a space station will likely have no use for water breathing (and, again, I say this from a purely stereotype standpoint--a specific space station could involve great conduits of water for the dolphin-race cadre ... but that's an exception, not the rule).

So there's no way to know for sure.

What I (have done--and believe is a reasonable answer in some respects) is to say the following things:

1. Hard Sciences and Combat ability are a little more expensive than academic ability. This is a very questionable value judgment--it's based on observations of niche protection, expectation of reality (weakly, in some contexts), and some observations about how I've run games in the past.

2. Diminishing Returns for everything: this makes being really exceptional expensive but being, you know, decent at something pretty cheap. It's my observation that being moderately good at something is substantially less valuable than being really good at it and diminishing returns (each extra point cost more than the last) is good for that.

This is especially true of attacks in games where there is reasonably common armor that absorbs damage (it's mathematically true: damage below a certain number goes to zero against armored opponents).

3. I encourage players to give me their characters before the game situation is finalized which goes a long way to making sure there is mutual interest on the part of the players in the situation and that the characters are reasonably cast as protagonists (i.e. that their characters and thus abilities have some relevance to the situation--but not an absolute attention to each skill being useful in some projected likely scene).

-Marco
* Some people will say this is because games traditionally have big combat sections. I disagree with that--but whether it's true or not, the fact remains that even if the GM's complications are basically pacificistic in nature a player may choose to employ force or credible threat of force in many traditional RPG situations (even court intrigue).

[mindwanders]

You might want to have a look at the FATE system that requires the players to think about this as part of the character creation process. It has Aspects which give a bonus to the player if they are used by the player in positive way and a bonus later in the game if the GM uses them in a negative way.

There is no such thing as a positive or negative Aspect. There are only descriptive aspects which can be positive or negative depending on the circumstance.

You can find FATE here:

http://www.faterpg.com

[neelk]

Do any of you know any clever ways - along the lines of the slow boat race above - to pull out this sort of honest evaluation?  I've tried pricing strategies: making people bid for attributes.  The problem becomes that I don't want people to never be able to afford an attribute.  The idea is not to limit, but simply to measure.  I wonder if an answer might be found in some sort of auctioning system, but I run into the same problem, as well as the issue of players conspiring to underbid everything.

Using economic techniques is fun, though kind of overkill for a roleplaying game. For discovery of how much the players value something, one of the simplest ways of doing it is to use a sealed-bid second-price auction (aka a Vickrey auction).

Sealed-bid second price auctions work like this: when something comes up for auction, everyone writes their bid on a slip of paper without communicating with one another. The winner of the auction is the person who bid the highest, and he or she pays as much as the second-highest bid to acquire it. So if Albert bids 10, Betty bids 15, and Carly bids 20, then Carly will win the auction and pay 15. The effect of this is that it is strategically optimal to bid your true estimation of the good's value.

(Why? Suppose that you value the thing at value V, and the second-highest bid is P. Obviously, you should never bid more than V, because you don't want to pay more than you think the thing is worth. However, note that if you have the maximum bid, lowering your bid won't change the price P that you pay, but if you lower your bid too much you might move you into second place and lose the auction. So your optimal strategy is to simply bid your true valuation V.)

[Hudson Shock]

Thanks for the comments and your time.  But - and I don't want to seem ungrateful - comments that boil down to "there's no reason to do this" aren't really helpful.  Let's assume, just for the hell of it, that I've thought about this a lot and that the problem as I've outlined it is something that would be useful to solve, and not ignored.

EDIT - not to imply everyone did this, but there was some, and I don't want it to continue down this road.

Again, I know that these won't be objective "true" representations of how useful an attribute skill is.  One person might see "Strong Willed" as something that is helpful only during torture or mind control.  Another might see it as something that is useful almost all the time.  The two people would thus give "strong willed" on their character sheets different scores.

Regarding the players that can turn disadvantages into advantages - well, that's exactly what I'm trying to measure.  I don't want, as the rule writer or the GM, to say that "Fear of Spiders" is a -3 when a clever player actually has a dozen ways to turn it into an advantage.  I want that player to give an honest estimation of what he thinks of it - which might be very different than a less clever player would think.




If no one can't think of a way to pull this sort of truth out of people, that's okay.  Neither have I, so far.  I have less ideal, but workable, methods I can use instead.  But the people here are known for innovative solutions and I thought I'd throw it out and see if anybody salutes.


EDIT - I have F.A.T.E.  Great game.

[Hudson Shock]

Neelk, that is exactly the kind of thing I'm thinking of.  It uses logic and mechanics to pull truthful evaluations out of people without depending on honesty.

Thank you very much.  Unfortunately, off the top of my head, I don't think it's immediately applicable to my needs because any given attribute is something that will only go to the person who came up with it.

In other words, Albert makes up "Two years of Fencing training".  I don't see how Bob and Carla can bid on that, since only Albert can end up with it - it's for his character after all.

But, again, this is exactly the kind of auction I wouldn't have thought of on my own, and maybe it'll inspire me to a real solution.  Any other things like this - or a source of other similar ideas I can check out on my own - would be greatly appreciated.

[John Kim]

My first thought is that you imply that the usefulness of an ability is something that the player secretly knows but might not be honest about.  I don't think that's the case.  How useful an ability turns out to be depends a lot on everyone else in the group (i.e. what the GM does and what the other players do).  

--------

One concept from several games (notably Theatrix and The Babylon Project) is to evaluate advantages/disadvantages during play.  That is, when an advantage proves useful to you, you have to pay out points for that use.  Conversely, if a disadvantage proves to be a real hindrance to you, then you gain points at that time.  

Thus, people can take whatever abilities they want during character creation.  The PCs who are powerful as defined will be spending all their points just to use their existing abilities.  The PCs who are less effective will accumulate excess points.  They can spend these points to gain new abilities and thus the system will correct itself.  

That's the principle, at least.  There are a number of pitfalls.  (1) There is the possibility of players encountering "Well, I logically should be able to do that, but I'm out of points."  (2) Trying to evaluate usefulness in play adds more decisions and bookkeeping to action, thus slowing down play.  (3) This system in a sense penalizes creatively finding uses for your abilities -- or at least there is no advantage to doing so.

[neelk]

My first thought is that you imply that the usefulness of an ability is something that the player secretly knows but might not be honest about.  I don't think that's the case.  How useful an ability turns out to be depends a lot on everyone else in the group (i.e. what the GM does and what the other players do).  

I think he's right. Most people are a little uncomfortable revealing their true preferences, as a defensive measure arising from the fact that almost all people are political animals prone to taking advantage. But the habits and tactics appropriate for office politics and commercial negotiations range from unhelpful to destructive in a hobby pastime. Inventing mechanisms that make preference revelations "rational" can be a useful way of greasing the wheels and speeding the process of developing a shared understanding. So is eating pizza and chips with the others and bonding the old-fashioned way, but there's no harm in having multiple arrows in the quiver.  

However, it's worth bearing in mind that the idea of the value of a character in a roleplaying game is much weaker than (say) the value of a car at the auto dealership. If the players are generally happy with the social relationships at the table, I find they will overlook even very dramatic inequities in the in-game stuff. This happened to me in my Nobilis game -- the player I thought I had done the worst HGing job for, eventually told me that she had been having the most fun she had ever had in an rpg. This came as a total surprise to me, though a gratifying one!

One concept from several games (notably Theatrix and The Babylon Project) is to evaluate advantages/disadvantages during play.  That is, when an advantage proves useful to you, you have to pay out points for that use.  Conversely, if a disadvantage proves to be a real hindrance to you, then you gain points at that time.  That's the principle, at least.  There are a number of pitfalls.  (1) There is the possibility of players encountering "Well, I logically should be able to do that, but I'm out of points."  (2) Trying to evaluate usefulness in play adds more decisions and bookkeeping to action, thus slowing down play.  (3) This system in a sense penalizes creatively finding uses for your abilities -- or at least there is no advantage to doing so.

Nobilis has a system like this for disadvantages, but the pitfalls you suggest aren't the ones that actually arose in practice. (1) and (2) don't apply in a point-bidding based system like Nobilis -- the characters have a base level of competence they can spend points to increase, and the point-spending *is* the mechanic so there is no extra time lost.  And (3) doesn't happen because you can't solve a problem using an ability without a justification for it, so you still need to be creative in finding a way to make it applicable to the problem.

Instead, what most often happened is that the players played to their restrictions and then simply forgot to remind the HG to give them the points for it. I've seen this happen again and again as both a player and a game moderator.

This issue has come up on the Nobilis list before, and I want to link to this post by Tony Lower-Basch, which is really cool and has affected how I think about this ever since I read it.

[aplath]

One thing that I don't get is why they wouldn't be honest in the first place. I mean, the way you described they have no reason not to be honest.

And that's where I'm having a bit of trouble to find an answer. Without knowing the player's motivation to underestimate (or superestimate), it is hard to find a strategy to avoid it.

Andreas

[Hudson Shock]

One thing that I don't get is why they wouldn't be honest in the first place. I mean, the way you described they have no reason not to be honest.

And that's where I'm having a bit of trouble to find an answer. Without knowing the player's motivation to underestimate (or superestimate), it is hard to find a strategy to avoid it.

Well, the system he described isn't my system.  My system rewards characters with Fortune Points at the end of each session.  Less effective characters get Fortune Points - they will be less effective most of the time, but at a few moments of their choosing, they can really shine.

The problem is with the first session.  My system needs at least one session in order to work.  So, I can 1) have the GM pre-award Fortune Points based on his estimation.  I may end up doing this.  It's workable.  But I don't like the way it depends on the GM too much, both because I think the GM is usually overworked anyway, and it has the potential to become adversarial.

Or, I can 2) let the players make their best guess at how effective each attribute is.  The problem is that basic human phychology will cause them to downplay their character's effectiveness in order to get more Fortune Points.  I don't think really has anything to do with honesty.  Economics and psychology shows that people tend to act in their best interest, whether they are aware of it or not.  Systems like the auction Neelk mentioned are systems for using that behavior.

[Hudson Shock]

One concept from several games (notably Theatrix and The Babylon Project) is to evaluate advantages/disadvantages during play.  

Thank you, I will check these out.

That's the principle, at least.  There are a number of pitfalls.  (1) There is the possibility of players encountering "Well, I logically should be able to do that, but I'm out of points."  

I'm not using a "pay points to do something" system, so I've avoided that pitfall.

(2) Trying to evaluate usefulness in play adds more decisions and bookkeeping to action, thus slowing down play.  

I think I've minimized bookkeeping as far as it's possible to do so - a simple checkmark by a characteristic when it's used is all.

(3) This system in a sense penalizes creatively finding uses for your abilities -- or at least there is no advantage to doing so.

I believe my system elegantly balances the desires to gain rewards for having ineffective attributes, and to succeed at a task by using attributes imaginatively.

[Bankuei]

Hi Hudson,

An issue specific to roleplaying games is that any defining feature of a character can change in usefulness based on how the game is played.  Consider the classic issue between combat and social conflict.  A game heavy in one will find that skills in that area will become more important than the other.

Probably the best way to measure "how important" a given trait is- try measuring how often its called upon during play, and how important the results are of success/failure.   For example, if you find a skill is called on 20 times during a session, odds are that its pretty important.  Likewise if a skill is used to prevent the player from being barred from play(such as their character dying).

In this way, you don't have to worry about player perceptions of "how important" anything is, and you also take into account that some focus will shift based on the campaign or game.  

Chris

[Hudson Shock]

Well, this is precisely what my system does.  It does an evaluation after each session - if the campaign morphs over time into something favoring stealth over combat, stealth based charactesr will find themselves earning less "bonus points".

Again, the problem is in the first session, before any in-play data can be accumulated.

I'm going to step away from this thread for a couple of days, and let anybody who wants to hash something out.  It's too easy to fall into the trap of defining too much of my system to early, or become defensive about it, if I'm jumping into the discussion all the time.[/i]

[Callan S.]

Much like your example of the river boats (when I first heard it, it was about racing camels...funny change, eh), we have this for roleplay.

Say a player has a disadvantage. At some point the GM says it applies. The player starts arguing it wouldn't apply.

Why? Because honesty is punished. A person who just complies gets a punishment/the disadvantages penalty. Thus the system encourages the player to argue his way out of it, because doing so atleast sometimes comes out with a better result than just being compliant.

The system rewards trying to avoid it. But what if you reward it's application. How about everytime the disadvantage is applied the PC gets some experience or some other reward. Soon you'll have the player bugging the GM 'C'MON! My disadvantage WOULD apply, and heres why!' over and over.


Thanks for sticking with me. Now how does this apply to you? Well currently, your players are/will downplay something bought because frankly honest evaluation isn't rewarded.

What if you asked them to rate the effectiveness of each thing, and gave them some small reward which is scaled with the rating. The higher they rate things in effectiveness, the bigger the reward.

Most people want to be honest and giving them even a small reward for doing so will probably get you dead on estimates (They'll think "Ah, its a four...ah, but is that true? And if I make it the five I suspect it is, it'll give me a reward. Ah hell, its a five"). Some people might still underplay, because the reward isn't enough. But I'm inclined to thing you'll always get people who don't mesh well with a system.