Like many others, I am in the process of creating THE best ever, world altering, comprehensive yet simple, rpg rules system. I have now reached the stage that I have previously reached with other older efforts but didn’t get past, solo playtesting. Also like many others I have limited access to playtesters and I therefore need to maximise the benefits from solo playtesting. Whilst this is mentioned there is little methodology, in comparison to the later stages of play testing, that I could find on any of the forums or articles I have seen on game design.
I am curious as to how others go through their solo testing. Do the more experienced designers have a standard battery of tests they subject their embryonic systems to? For example a skill check by a character experienced in that skill against a difficult objective or a skilled warrior with sword and shield fighting another skilled warrior with a spear.
Whilst testing do you have a fixed idea of what results you want? With fights a skilled warrior versus a proficient warrior would lead to the skilled warrior winning 75% of the time...
I’m interested in the methods used by others and any tips would be gratefully accepted. Whilst a lot of this will be very individual to the designer and the project I think this is a previously little covered subject that could help many others. 

Orig

Make up a bunch of characters. Make up a bunch of situations. Run the characters through the situations according to the rules as written (exactly). If you do something that isn't in the rules, ask yourself why it isn't in the rules.

Things to consider:
* generic conflict
* potentially lethal conflict
* social conflict
* all of the above, but between two player characters
* all of the above, but between three or more player characters
* "griefer" behaviors

Also, consider playtesting with friends online using virtual tabletop software, Instant Messenger, IRC, FoundryMUSH, or some other tool. It's easier to drum up people online to test things sometimes than finding people to play face-to-face.

It goes without saying, though, that if your game promises to be THE best ever, then you will have no shortage of players lining up at your door to playtest for you. ;)

I've done a fair bit of solo playtesting. I roll the dice and see where it takes me. I don't fudge numbers on many test runs. On others I try deliberately extreme methods of play. Maxing out one thing, taking worst case or best case values, and playing through many times.

Someone helped Rich Stokes by running Umlaut through a bespoke computer program to explore all the extreme ways of playing. The result was that it was indeed balanced and there wasn't an "optimal" build to start.

Thanks for the response it made me think a bit more on the subject. I've been trying to get a framework of some description that I can use. The way I'm thinking at the moment in broad brush strokes.

Use the mechanic in an opposed test with the same values (e.g. both have attack 20 defence 10) in opposition to see if the method is inherently stable/useful.

Repeat this at different levels.

Ascertain the value of 1 point of change. How often the greater score will win.

Ramp up the changes to see if/when it breaks.

Allocate numbers based on these results. I'd like to break this down some more...

The thought of number crunching via a computer sounds a great time save although probably not as applicable to my project. Any suggestions? Dice! Plus is the closest I found.

For dice modeling, I tend to use Troll, Excel, or custom Java programs (I wrote a dice package that handles pools, melds, probabilities, and permutations).

Also, I cannot recommend enough the power of just grabbing handfuls of actual dice and making up scenarios in your head and rolling and testing. Charts and graphs are useful, but they lack the immediacy of seeing a die result on the table.

Seems to be a focus on checking whether the games 'balanced' or 'broken'. I mean, either balanced or not broken don't seem...interesting qualities at all. Shouldn't you be checking if the system use produced interesting or fun results (atleast what are fun or interesting by your own standards)?

I have to be honest solo playtesting hasn't seemed fun with previous efforts and I abandoned other attempted systems. I have lacked direction during the process and it turned more into a mathmatical problem that seemed quite nebulous and tedious. So far this stage has always been the give up point. I was therefore hoping to get  framework/advice from the community on which block to start with and how to build on that. Particularly as this project will be my first diceless attempt and is based on player choice. Like everyone my time and attention span is limited so I wished to make the most of both.
The core concepts on the latest one seems theoretically fun but I was going to check the fun'ometer when the current bare bones concept has been expanded to make it playable and get feedback from actual live playtesters. I suspect I am too close to the attempt to be impartial...

Have you heard of "playstorming"? It's basically playtesting before the rules are all fleshed out. You playstorm with a bunch of people who are happy to be fiddly with the rules until you figure stuff out. It can be a lot of fun.

Have you continued with your solo playtesting? If so, how has it gone?

if your game has levels i would start your characters you generate at 1st and then artificially speed them through all the levels in your game. don't miss any out. this way you get an idea of the content in your game all the way from being a new character to one that can't get any more powerful. you could maybe do 1 level per playtesting session.

Yes I've done a bit of solo playtesting. Emphasis on a bit. I have a couple of different variations of the damage mechanic to choose from. My plan at the moment is to test both ten times. If one obviously appears better do further testing on that one at different levels with minor testing of other to verify the judgement.
The normal issues of weapon values & movement and how many times can someone be hit with a sword before they die are enough to cause me a headache. Any generic framework advice would still be appreciated but I am slowly getting my own. Partially in thanks to comments here :-)

No levels as such. Although toying with idea of all stats added together if fall within certain bands of values providing guide as experienced, veteran etc.