I'm working on a game I've already posted about here and ran into a problem before getting to playtest mode.  It uses accumulated stress (Mental Trauma) which is rolled against - success you keep going, failure you suffer the effects of fear.  This is usually triggered by a frightening circumstance and ends in flight.  However, my critical mind couldn't leave that alone and I think it has a point, especially after searching the issue online and finding complaints regarding other system that used such fear mechanics.

Issues:
1) While the physiological symptoms of fear are often similar, people react to these symptoms differently: one person may flee while another may fall prone and go fetal, while yet another may start flailing away at anything that moves.  Thus, a mechanic that comes with a "canned" response does not seem sufficient for a game that centers on horror and, especially, fear.

2) I want a mechanic that is simple, while still treating fear dynamically.

3) Many players get frustrated with a mechanic that steals control of their character rather than allowing them the chance to improvise the effect.

Proposals:
1) A big portion of chargen is answering brief questions which build a character's psychological profile.  What if during this process the player defines a number of ways (say, 4 or 5) that the character responds to fear, depending on intensity?  This would allow options for improv.

2) Some sort of "reward" mechanic for players who play their frightened character appropriately.  This might prove important if rules do not exist to "force" the character to behave a certain way, since players will be tempted to merely fail to play fear out adequately.

The problem is I'm burning the wick at both ends (like everyone else, this isn't my day job and I'm really busy at my day job), so I'm a little stuck.  I'm not looking for someone to devise a mechanic for me.  Rather, I'd like to hear from people as to whether it bothers them if a mechanic steals control of their character (the proverbial failed saving throw against Fear) for a certain period of time and, moreover, if they would prefer to roleplay the effects of fear and take the lumps that go with it voluntarily.  If you do have suggestions regarding any of the information above, that's certainly welcome as well. 

In my experience you shouldn't be that worried of the people who don't want to play the game you want to design. Rather, worry about the people who are already interested in the sort of experience you want to create. Whether you force fear or reward play-acting it results in very different play experiences; choose the one that better reflects your own vision of fun play, and stick with that. I could, and have, played games with both approaches, and they work in different ways, but they both work.

The matter of realistic psychology is a red herring, unless it provides something valuable for actual play. Having the player list a large number of potential fear effects for his character prior to play is a horrible sort of preloading that is probably out of place for most design frameworks - you don't want to keep players stuck in preparing their characters for play when they could be playing already, and especially not for something so minor and, frankly, disinteresting as listing potential responses to a potential event in play. If you have to customize this stuff per-character and have to allow the player the responsibility and power of doing it, then at least have them do it on the spot, in play, rather than beforehand.

Rather, consider this: all fear responses can be categorized as either fight, flee or freeze; so any time a character gets gripped by fear, you can just have some mechanics for finding out which of these it is the character suffers of this time. The appropriateness of the response should then impact the fate of the character, probably. For example, perhaps you could have the player make some sort of stress test to get to choose the response from amongst those three. Perhaps they go in cycles, so that a character trying to fight flees if he fails in keeping a grip on himself, one trying to flee freezes and one trying to freeze (= keep calm) attacks. Perhaps the severity of the response then depends on the original stress levels that caused it in the first place, rather than the momentary situation at hand, which can then lead to wildly inappropriate responses, such as attacking a friend over a minor scare.

Also, check out Don't Rest Your Head, which deals with this exact topic with some explicit rules. I find it does a good job of getting the fear responses into play without unnecessary rules. Characters are constantly fleeing or fighting madly in that game when the horrors get too much for them.

Alternatively, if you want to go with rewards, you just need to have some sort of currency you can award to players whose characters get afraid. Perhaps some sort of addrenaline points that can then be used for improbable physical feats. Dead of Night deals with fear this way, as characters gain Survival points (the combined hitpoints, xp and hero points of the game) for doing things that might be stupid, but also in-genre for horror movies.

Well I hoped I'd hear from you, Eero, as you tend to have a very low-key and open approach in your responses (as opposed to those who will go down your throat over the definition of a word!).  I suppose I should clarify a little, first.  I personally do not want to just use a roll vs. flight response mechanic, which is basically where it is right now.  I wondered if I was being silly in thinking that it was too generic and did not allow for the many different responses possible (while categorized three ways, the manifestations are practically limitless).  What I found is that there are a fair number of people out there that think it is too restrictive.  So, in that knowledge I began to formulate a different way of handling it.  The problem is I've been plagued with work and have had little time to address the issue, which led me to post here tonight.  I'm just braindead.  Tomorrow I'll probably have a eureka moment.

My original experience with fear mechanics were D&D's saving throw vs. Fear.  My second experience was CoC's sanity system (a bit different and not precisely a "fear" mechanic).  There are two layers to how I want this to work: the "save" vs. the current situation (which I am calling the Disturbance Level), and then a gradual buildup of tension (Mental Trauma) from each disturbance, probably whether you succeeded the "save" or not.  I would like for this to have as few steps as possible.

My current thoughts on this, regarding how fear really works and how I'd like the game to simulate it, are as follows:

1) Just because you are able to keep your cool during an encounter does not mean that it hasn't affected you.  It is reasonable to assume that you would have an accumulation of "points" which represent the rising tension in spite of your success in coping in the heat of the moment.

2) For a certain amount of time, the accrued tension should be active and add to the Disturbance Level of the next encounter, making it all the more difficult to pass each successive test.

3) Over time, these points will bleed off naturally as you naturally calm.

4) When you do "freak out" in response to a failed disturbance check, the pent-up tension is dissipated in one burst.  Depending on how many points those ended up being, (including the current disturbance check's margin of failure), there might be a corresponding manifestation of fear or at least a measurable intensity of the reaction. 

Does all this make sense and sound less than boring?

That model seems pretty good, yes. However, do not underestimate the next important, even crucial step - you need to create a mechanical interface that allows the players to track these forces and latch them into the fiction at the correct moments, and that interface needs to be aesthetic and easy to use. It was very typical, especially during the '80s, to create rpg rules systems that started from the sort of reality-modeling that you're doing here, but failing to finish the system by thinking of usability. Those games are not being played too much anymore - things like the firefight hesitation mechanics in Twilight 2000 simply are too burdensome to routinely use in practical play, even if objectively the results are just fine.

Thinking from this viewpoint, your challenge is not primarily in figuring out what fear is and how it should be modeled; your challenge is in matching those thoughts into a rules system that is pleasurable for the eye and fun to use in practice. For example, say that you'll use some sort of point pools to track the tension levels of an individual character. How will you deal with recovery, which in your model happens slowly over time? The way I do the similar fatigue recovery thing in the primitive D&D campaign I'm currently running is to simply declare that "a couple of hours pass as you dig the collapsed tunnel" or whatever, and then take a bunch of fatigue tokens off, give a bunch to the players or whatever is appropriate in the context. I don't, as GM, really track the number of tokens I give or take, I just take "a couple" (2-3 usually) or "a bunch" (something like 6) - the only part where the tokens are tracked is when the players stack them in front of themselves and compare them to the character statistics, so that they can tell me when and if their character starts having trouble with fatigue/injury. This system works for us, but as you can see, the majority of the design concerns how the points are moved about, who needs to worry about them and what sort of procedures the players need to go through to apply the mechanics - the underlying model of reality is really simple, and it needs to be, as the point of the game is not in tracking fatigue and injury.

I'm focusing on the issue of rules usability because I myself improved my game design immeasurably by realizing that the game is not about the fiction, it's about the motivations, choices and procedures of the people playing the game. Those have to come first to create fun and usable games. This is why, increasingly, my first design question for myself is "What extent of mechanics can this part of the game support?" instead of "What sort of mechanics can I use to model this behaviour?" or some such.

A mechanic proposal.

"Bottling Fear." 
In the event a character gets afraid, they can improv how their character reacts.  Based on their reaction, they either gain or lose fear.  So, they could choose to "bottle up all their fear" and gain more Mental Trauma.  Or they could give in to their fears and not gain any Mental Trauma.  If they react in a way that is more then the situation deserves (Vomiting after a swarm of beetles pass over their feet, perhaps?) they can actually reduce their Mental Trauma.

In mechanical terms, you could rate them like this...

fear effect
1. creepy
2.
3. supernatural
4.
5. dangerous

reaction
1. fight
2. act sporadically
3. flight
4. freeze
5. vomit (I assume this is worse, since you are drawing attention to yourself and can't *snap out of it*)

A character would have to react in a way that matches the fear rating not to gain mental trauma.

Hopefully you find this helpful, but I understand if it may not be. :)

I like that, David - it gives the control to the player and fits well into the procedure of play. I wouldn't rate the reaction types by severity, though, as that sort of assumes that some reactions are better than others, when actually an aggressive response could be lethal against some horrors, while freezing might be just the right choice (remember that T-Rex from Jurassic Park?).

The system would be even simpler without any ratings at all: the GM could just give each horror situation a fear value, which would be the number of trauma tokens the character would gain if he kept his head and didn't surrender to fear. Let the player then choose if he can take another 5 or however many points at any given point. The sort of fear reaction the player chooses (or is forced to choose, if you'd use some sort of check to find out what it is) could then determine how many points of the bottled fear would be shed by giving in to it.

That's an interesting model, David.  I hadn't thought of categorizing for comparison that way.  I agree, though, that the various reactions differ in their severity based on the circumstances surrounding the occurrence.  Even so, that's an avenue I hadn't even considered.  My thoughts today while at work were still based on my suggestions last night:

Encounter has a Disturbance Level.  Character rolls against that level (level is difficulty).  If the roll is equal to or greater than the DL, the character copes but adds the DL to current Mental Trauma.  If the character fails, they react in fear appropriate to the margin of failure and no Mental Trauma is gained (because it was released in the outburst).  Back to the successful roll, though - later the character has another encounter with a DL.  Roll against that one as before.  If successful, the DL is added to existing Mental Trauma (number is getting bigger and bigger).  If the roll is a failure, the character suffers a fear reaction appropriate to the margin of failure PLUS current Mental Trauma, at which time Mental Trauma goes to 0. 

Unfortunately, that sounds like a lot more rigamaroll than is warranted.  You get where I'm trying to go though?  What I like about David's suggestion is that it's in the player's hands rather than randomized.  This could prove important, because according to my concept above, it is possible to be totally unshaken by a moving decapitated corpse but, due to the buildup of the Mental Trauma, a failed roll against something much smaller (like the bugs across the feet thing you mentioned, David) could suddenly cause a massive backlash of fear that is not only inappropriate for the circumstance but, worse, feels like an over-the-top reaction due to flawed mechanics (which it would be).

Based on what the two of you are suggesting together, along with my own observations concerning the flaws in my current concept, I'm sure the effect of fear has to be up to the player, not the system.  However, it also has to keep them honest in the case of those who wouldn't voluntarily have their characters suffer those effects.  I know it seems like a lot of stress over a "small" mechanic, but it's vital to this game and is more important (by far) than the damage system,etc.  Normal actions and combat are actually incredibly simplified, so I wanted this to be simple as well, but it also needs to really work and work well.  It also needs to be fun, as Eero said, so I'm stressing over the details.  It seems like I'm getting closer to the basic concept that I want and your suggestions have really helped.  If the information in this post gives either of you new insight or other suggestions, let me know.  I'll be thinking on this further tonight and tomorrow.

Thanks so much for your responses so far!

I'm not sure I have much to add, but I do have to say this. I don't know how your game works, but this doesn't seem like a small mechanic, I think this seems like a game defining mechanic. Spend as much time as you need on it.

Ok, actually, I have another mechanic concept for you, a form of gambling.

The player reacts to a disturbance.  Depending on how detrimental the action is to them compared to the disturbance level, the GM grades them.  (Screaming as the wall turns to blood is a grade of 0.  Doing nothing, acting casually is a grade of 5 or whatever. Acting a little afraid would be a grade 3, perhaps) 

Then, conflict resolution. (Mental Trauma + roll vs DL) or something.

If the player succeeds, good for them.  Depending on how your game is driven, they might gain a little mental trauma (Giving them Mental Trauma will generally drive them to an inevitable end, not giving them Mental Trauma gives them a chance to survive.) 

If the player fails, they gain the mental trauma of the grade the GM assigned.

One thing I might warn against is, if you have the mechanic resolve before the players react, the players are not going to put as much effort into the reaction.

I really like the gambling concept, that's a definite step in the right direction and a slap on my forehead for not thinking of it before.  It's actually not far removed from the way the dice worked, but it's entirely the player's choice.  What I mean is, if you think of the dice as the gamblers, when they indicate that you cope and therefor don't react badly, you gain points that are going to hurt you later.  The question becomes what the circumstances will be when this buildup finally gets you.  Of course, because the dice aren't intelligent, the "gambling" becomes senseless.  I'm going to bring my focus around to this suggestion of yours.  I have to chop it up and fit it precisely to what I'm trying to do here, but the suggestion alone is incredibly helpful.  I've been really frustrated at how blank I've been lately at a time when I need the ideas to come through concretely.

You are absolutely right about the fear mechanic not being a small one (and being game defining).  It absolutely is.  The game is called Illuminatus: the Dwindling Light, which was not my first choice.  My first choice is already in use by a generic fantasy system, and I didn't want to reuse it.  Too, it was perhaps too simple of a name, not enough flair.  Simple as it gets though, it was "Fear."  So, you can see where the mechanic was already the basis of the game, more or less.  It evolved from there.

Thank you for your responses and suggestions, you've really jarred the gears and got them turning again.