Hello,
me and a friend of mine have lately discussed how to support fear in a games rule-system and we would like some critique and helpful thoughts to our rough concept. It should be mentioned that this is system is done with the intention to work in a 60:ish/lovecraftian horror rpg, who's theme is paranoia and the unknown.

The idea is as follows: The gaming group is divided into three, the Sage (who takes on the role of rules advocate, who gives descriptions of the world, who makes a trail for the players to follow and generally is responsible for the story), the Role (who plays the antagonists, any other npc's in the story and who is responsible to provide the conflict for the players to resolve) and the Players (who plays the protagonists and who is responsible for driving the story forward).

Now, there are two rules at work here, namely that the Players, if inducing fear on each other (by asking for permission to describe a scene from the Sage for example, or by introducing a particularly scary plot/theory/idea) may invoke a mood-penalty on the Role's antagonists.
The Role may invoke the opposite on its own antagonists, that is a mood-advantage, if the Role creates a sufficiently scary atmosphere in any way. The rulings of this is at the discretion of the Sage.

The second rule is that the Role does not know anything about the story, he/she simply gets a group of things the npc have done/does know, motivations and in the case of the paranormal roles, the super-natural effects that it can invoke.  After this the Role may create the character in any way she/he chooses.

Our purpose where to create a spiral of increasing fear, where it is beneficial and encouraged for players to become more scared and where information on what is really going on unknown. My question with this post is are there something that we have overlooked?
Can you see a logical loop-hole with the theory?
And are there any immediat extensions that you would make?

Do you mean that the players are afraid or the characters are afraid? If the former, do you mean real fear, or empathy for the character's condition?

The point is to create an atmosphere of fear, thus to make the players afraid. All in the name of fun and games of course.

Now that I look on my reply I notice that it where very short and I'm sorry for that.

The point of the system is to foster fear in the players, and that is real fear not empathy, much the same as the kind of fear you would feel if you watched a really scary movie.

In some sense we have taken the concept of people making people afraid and tried to apply it in a rules context. Can you see any other way for this effect to be further enhanced Eero?

So if I understand you correctly, the idea is that the players will gain in-character bonuses for making each other afraid, while the antagonist player will likewise gain them for making the other players afraid? Seems fine to me as far as it goes. Some questions you might wish to consider are how you're gauging the level of fear and what the bonuses are actually usable for - it seems to me at first blush that I'd consider is somewhat jarring for mood that my getting immersive fear improves my character's ability at kicking butt. I would probably enjoy it more if it was a death spiral kind of thing where the nightmare actually gains strength from fear.

A game that has a system strongly resembling what you propose is Primetime Adventures. Therein the players award each other rewards for making up cool stuff - the rewards come from a communal pool, so there's a limit to how much there is to spread around. Practically a given group will come to a set standard in what they reward and how much, so it works well.

The part about objectively measuring fear is probably the hairiest here, all told.