Hey all, I've been working on a d20 based system and I've reached a point where I feel that it is very close to being playable. The only thing is that my nagging paranoia wont let me move on to working out everything else until I know my system is mathematically sound and doesn't have any glaring problems (i.e. harder monsters are nearly impossible or too easy to kill.) This is my first attempt at designing a pen and paper game and I'm naturally protective about my ideas so forgive me if I'm not open about the exact mechanics initially. I will expose various portions of it if it is needed to verify my mechanics but what I'm really looking for is general advice on the topic. Thank you for any help your able to give.

This is what playtesting is for.  Hedging about manipulating numbers isn't going to do you much good once you hit the play table.  If you haven't diverged too far from the d20 core, then you're likely to be in the ballpark. 

That said, coming from an artistic background I understand the hedginess you have about sharing your mechanics.  I've encountered the attitude numerous times, and I believe the same basic advice would apply to game design as it does with art: If you want useful feedback you have to share your work.  Really.  Nobody's going to steal it.  On the extreme, off the wall, chance that somebody does, so what?  It's highly unlikely you're doing something that hasn't been done already regardless and imitation is the highest form of flattery.  The more specific you are with what you need help with, the more likely it is that somebody will give you useful feedback.   

Hi froboi,

Are you going to run this for your own friends first...are you sort of wanting it to work on the first time it's played, for various reasons? Do you have a group that is willing to do some of the development work for this game, by playtesting it? Or do they just turn up to game night for assured entertainment?

Here's some thoughts, based on the (little) you've told us:

You've already realized that many peoples' dice-system inventions contain impractical limitations.  A dice system needs to be designed with the extremes in mind, and even then leaving room for doubt.  Most of the major RPG systems allow for the fact that, with enough modifiers on both sides, a successful combat attack becomes statistically impossible.  Therefore, they allow for the outside chance (e.g. a "natural 20" die roll) to hit ANY defense level, no matter how high.  Does such a luck system work for you?  Perhaps you truly want it to become impossible to hurt someone, if the differential is really that great.  D&D and GURPS, two of the biggest system, both allow for automatic hits in this way.  So obviously most players are okay with the idea.

I recommend computing the percentage chance of success/failure at your extremes.  In D&D, that chance is 5%, which is much too high in my opinion.  If the GM does not protect the players, the number of critical hits gets out of control at 5%.  The extreme roll for GURPS is 1 in 216, slightly less than half a percent, which is a much better extreme ratio (my opinion).  Many of my own design ideas have been based on 2d10, which leaves a 1% outside chance of success.  Or, if I decide I don't want automatic success, I decree that a natural 20 actually is handled as a 25.  This leaves a tiny chance of stretching the effective zone, statistically, to a fairly extreme range.  But the Aunt Gladys, with her kitchen knife, has no possibility whatsoever of killing the Archdemon Omega -- I leave that much as an absolute; his armor class is at least 25 points higher than her weapon class, so she is doomed, even if she rolls a natural 20 on 2d10.

Do the math.  Simulating your system based on "reasonable modifiers" is not adequate.  You have to imagine the most outrageous circumstances and test the game at its extreme.  Much depends on your concept, however.  Does the aging Aunt Gladys actually have a chance to slay the Archdemon Omega with a single stroke of her letter-opener?  Be careful when allowing for the possibility of immunity from attack via high stat's.  That said, it may be appropriate to allow for immunity at some point.

When imagining the relative usefulness of a skill or combat statistic, do the math and find out how many times, out of 500, the best and worst results will occur.  1% may sound like a small chance, but if characters fight a room full of 10 bad guys, and they do it five times over in a single adventure, you're talking about possibly hundreds, if not thousands, of accumulated chances to score critical results.  If a player might become totally immune to certain attacks (or failed results), then die rolling may become obsolete.

Just a thought.

There is the old standby of an add and reroll on some event/roll. So for instance on a d20, on a 19 or 20, add a d10. Or a d20-10 (min 0).

I do recommend that any add and re-roll range be smaller than the original range.