Hah, that's a pretty interesting conundrum. Bizarre, in fact.
Ultimately the answer is that it's worth whatever you say it is worth: there is no practicable market that could be addressed for price discovery. This is more like selling a heirloom than selling a car, so the negotiation has to proceed from motivations: how much you want to sell, and how much they want to buy. There is no guarantee that these two desires match each other; most of the time I would expect them not to, which is what makes this a weird situation.
One source for prices that might be even somewhat relevant is to consider work for hire. The difference is that when you hire somebody to work on a culture industry project, you get to decide what it'll be about; on the other hand, when you sell work for hire, you don't have emotional commitment to the work, and may thus be willing to let it go cheaper.
Another point of comparison are obviously book manuscripts and screenplays, which are regularly bought for culture industry purposes. A very common means of achieving economic sensibility in these types of deals is to profit-share in some way: this allows the business partners to make the difficult value-determination issue less pressing, as they no longer need to be able to predict the future value of the manuscript so very exactly.
In mainstream book publishing you might expect something like thousands of dollars in advance payment, and several percents of the net profits off the top. In a work for hire situation you might expect to be paid several cents per word, or more if you're being hired as an independent consultant (in which case you'd negotiate a fee for the project as a whole). Neither of these situations quite corresponds with yours, but you did ask for rough magnitudes.
Perhaps it'll help in figuring out the price if I say that I know some roleplaying game publishers personally (being one myself, albeit a small press one), and I can't think of a single one who'd be interested in buying a manuscript for more than a couple hundred dollars advance at most. (We're poor people passionate about our own projects, most don't have thousands upon thousands of dollars to throw at a risky venture like that; if we want to spend seriously, we'll spend on a project we control from the ground up.) On the other hand, a publisher is much more likely to make an agreement for royalties, which might in practice bring in something in the low thousands for you, if the product is successful. Depending on the exact nature of your project I thus would expect that if you were being paid without royalties, as a single-time payment, the numbers that wouldn't surprise me would be somewhere between $100 and $5000, with that upper end being reserved for some moderately serious projects. Any more than that, and I'd reread the press release to figure out why the publisher is overpaying for a rpg manuscript of all things :D
If you want any more exact discussion of the situation, we'll need three important data points: how expansive and ambitious is the game (word count of the manuscript might be a good indicator), what is the nature of the publishing plan (size of the planned print run might be a good indicator), and would you be remunerated primarily via advance or royalties. All of these influence the value calculations a lot.