Topic: Theory, or anecdotally supported opinion?
Started by: GB Steve
Started on: 12/7/2004
Board: RPG Theory
On 12/7/2004 at 12:00pm, GB Steve wrote:
Theory, or anecdotally supported opinion?
How much data is used to support roleplaying theory? And what level of proof is required?
There's possibly some discussion about this here which I may have missed but I wondered what effort has gone into getting supporting data beyond one's own or that from immediate surroundings.
Obviously a totally rigorous scientific approach is not possible, even though you might like to put 6 powergamers in a cage for 3 months*. But most sociology has recourse to data at some point and whilst this site does not necessarily make such claims to academic rigour, I think there is probably a need to for more verification, if only to back up what one says as being true.
GB Steve
*Not that I've got anything against this approach per se, it just doesn't really work well with the kind of games I try to play.
On 12/7/2004 at 1:26pm, Rich Forest wrote:
RE: Theory, or anecdotally supported opinion?
Hi Steve,
Nominally, science involves discovering something new about the universe, but this is not really necessary. What is really necessary is a grant.
Shulman, R. (1996). “How to write a Scientific Paper.” Annals of Improbable Research, 2.
Now obviously this quote is from a parody article. But I quote it to make a serious point – verification costs time, lots of work, and money. Now I know a lot of people do already put a tremendous amount of time and effort and serious thought into this site on an almost daily basis. Much more than I come anywhere near (I have trouble keeping up with reading the boards these days), and it never ceases to impress me when I step back and look at it. Now consider adding to this degree of committed discussion the additional weight of something even remotely approaching serious research, along with the procedural knowledge necessary to get serious research right. And then add to this consideration of the whole peer review process and grant application process and apprenticeship process and so on and on and on that’s built up around academic disciplines (not that you'd have to have the same system -- this is just for comparison), and then remember that professional scholars in academic disciplines doing real work and getting paid for it make mistakes all the time as a matter of just regular old ordinary human error.
I think you’d need funding. And you’d need training. Bare minimum.
I recognize, of course, that you haven’t said it needs to be rocket science, and it doesn’t need to be a full-fledged academic discipline with all the benefits and all the baggage that entails. But… I dunno, “sort of science” still ain't science, and it might even be worse than the current situation in some ways, if it made people think there’s a level of empirical verification that can't really be backed up. The board does have its successes in getting through careful discussions that are, at least, grounded in actual play, even if it is “anecdotal.” Meanwhile, I think the sheer amount of discussion itself, and especially the discussions with folks who are skeptical of aspects of the models, does have something going for it. I hope that, and I think that, things can and do move forward through this sort of discussion here, and it isn't just improved rhetorical savvy and cleverer versions of the same old claims.
So maybe there's some kind of progress there, perhaps. I think that in general, sloppy thinking is discouraged over the long term here by having to justify your claims in a discussion (as long as everyone succeeds in treating the discussion with respect and commitment). And remember that empirical observations (again, granted, anecdotal) from all members are not only welcome but encouraged, which should help protect the discussion from becoming free of any constraints, able to make any arguments. I’m not claiming this gives the Forge a "get out of evidence free" card, by the way. I’m just saying, given what the Forge is (a message board) and the rather modest claims it makes (as you noted, nobody as far as I can tell is saying it’s the one true, scientifically proven, way gaming works TM), given all that, I think the theory discussions are doing ok.
I could be wrong though. Hell, that’s the beauty of it.
Rich
On 12/7/2004 at 1:55pm, Matt Wilson wrote:
RE: Theory, or anecdotally supported opinion?
Hey Steve:
If I'm not mistaken, that's one of the purposes of the Actual Play forum here at the Forge. The "big model" of theory that's prominent here is a work in progress, continuously being tested against the reports posted in the Actual Play forum (and other sources of course).
So the answer to your first question:
How much data is used to support roleplaying theory?
is "more and more every day."
As for the level of proof required, I dunno. It's been a long time since I used a bunsen burner, but I think "theory" is something that can't be literally proven. It's more a case of strong evidence accumulating in favor of a different possibility. So people start cocking their heads or raising eyebrows, and they say something like, "hey, maybe narrativism is actually a subset of gamism, I mean look at this evidence," and then serious debate ensues. The Beeg Horseshoe Theory is an example of that.
On 12/7/2004 at 2:05pm, Ron Edwards wrote:
RE: Theory, or anecdotally supported opinion?
Whoops, I think I just presented my answer in the original thread, Steve.
Best,
Ron
On 12/7/2004 at 3:07pm, Eero Tuovinen wrote:
RE: Theory, or anecdotally supported opinion?
This will jump at the face of every American here, being that you've been schooled in logical empirism your whole lives. Heck, the word "science" is used in American to denote the empiristic sciences only. The great majority of modern positivist/empiristic philosophy originates in America, too. I'd dress my post up in empiristic language if it were doable, but in this particular issue it's simply not possible. I'll give it straight, accept or not freely.
Point 0: Forget for a second everything you've been taught about what's science and what not and so on. There are only multiple toolkits of thought, of which empirical science is one, analytical philosophy is another, and religion, for example, is a third one. Of the above examples the first two are frequently applied in different parts of academia, and consequently cause all kinds of hassle.
Point 1: "Roleplaying theory" is not a priori a sociological science. The word can mean, and frequently does, a branch of art theory. "Art theory?" you ask; it's one of the humanistic disciplines, and most of the discussions here, aware of it or not, fall under it's umbrella. As Steve himself notes, proof and reliability calculations rarely interfere.
Point 2: Art theory is a philosophical discipline, and has completely different requirements than empirical sociology. A "proof" in art theory is a deductive chain from commonly accepted assumptions. Because art theory for the most part deals in abstractions, the assumptions are rarely empirically derived. Instead, they are chosen for convenience. Most proofs in art theory concern themselves with deductive analysis of complex natural concepts, to reveal logical connections that become tools for further thought.
Point 3: What's art theory good for? It's used to construct tools of understanding for creating art, understanding communication in general and to reveal certain psychological constants of humanity (religion=art springs to mind). It is not used to prove anything about the real world.
See it already? You could strive for empiristic ideal in roleplaying theory, but that would have completely different goals than most of us, aware of it or not, use the theory for. We are for the most part not really interested in statistical truth. Consider some examples:
- Creative agendas: their existence is proved through concept analysis; because players have to have motivations for play, and some of those motivations are in disaccord with each other, something like a creative agenda exists. Naming the actual agendas is then done through rough empiricism, but is clearly secondary to the actual concept of a creative agenda: the list might change one day when we get more data, but that's rather unlikely, when dealing with general human motivations that creative agendas are.
- Chris Lehrich's ritual theory: another clear example. Chris decides to compare roleplaying to ritual, finds the most important points of analogue and there is the theory. The empirical part is already done, as he just needs to consider his own experiences for examples.
Note how the nature of art theory entails our motivations: we're looking for tools to understand our own play and to structure new kinds of play. Thus empirical proof itself is mostly singularly useless: so what if 90% of all American gamists thinks this? It just proves that they have a certain kind of background. My own play experience proves that, and it's the antithesis of this. As we're all human, it's not about some in-built limitations of one of us, but rather just about indoctrination and about the fact that what I've figuret out here is new. Thus my theory is proved, and we didn't need a thousand gamists for it; it sufficed for me to demonstrate that it - whatever it was - is possible.
If somebody really wants roleplaying theory to explore the statistical means of play, I find a Forge-esque online forum a particularly useless tool for that. Get out there and start polling, I say. That kind of data is mostly useless for game design and analysis, though; illustratively, the only party to do it I know of is WotC. And we all know that they do it because the poor things strive only to follow some nonexisting "market trend" instead of making good games.
On 12/7/2004 at 3:55pm, clehrich wrote:
RE: Theory, or anecdotally supported opinion?
Eero Tuovinen wrote: This will jump at the face of every American here, being that you've been schooled in logical empirism your whole lives.Not every American here, no. <points to self>
Basically Eero has it right, I think. I would really tend to avoid the word "science" in reference to RPG theory in general, not because it's inapplicable but because it seems to mean to most readers here a kind of pure empiricism founded upon statistical data. This is a particular kind of science, and by no means its totality. I would cite anthropology as a social science that does not, in the main, use this sort of data to any great extent.
In order to formulate a strong claim in such theory, one needs (1) concrete data, usually in anecdotal form; (2) a strong argument that such data can be generalized to at least some degree; (3) rigorous argumentation at the logical level.
That form of scientific analysis is not, in the main, predictive. In the context of RPG theory, that means that such analysis will not necessarily have practical value for the construction, design, or play of games. It may do so, but this is not an essential criterion of legitimacy or validity.
The disadvantages of such analysis appear obvious, from the various times we've argued about this here. So let me list off a couple of advantages.
First, no data is invalidated by being in some sense outside statistical norms. All groups, games, and so forth are of real value and are worthy of analysis. Within a purely statistical "number-crunching" approach, groups that fall well outside norms are often going to be dropped, unless they are studied precisely for their abnormalities.
Second, every form of data, at every level, is valid and interesting. It doesn't really matter whether we're talking about actual play (and that includes a huge amount that's rarely transcribed), social context, personal histories, what people say on the Forge about themselves or their games, what people say about Forge theory, or anything else. It's all data. A statistical approach has to begin by deciding what will be asked or evaluated; the rest may be illuminating in reaching explanations, but the method pre-determines the evidence collected.
Third, social interaction tends to be treated as extremely thick, dense, and complex. It is usually historically situated. These things are not generally true of statistical approaches.
I would argue that an anthropological and humanistic approach is appropriate to this medium, because the issues that in my opinion are most interesting are primarily those arising from social interaction and the formation of discourse. These things are most effectively and richly studied through humanities-style methods (including cultural anthropology loosely within that category).
I would, however, like to see some good statistical data as a form of background information, but it's going to take a professional, well-trained scholar to do this, and I don't see that happening any time soon.
On 12/7/2004 at 5:06pm, Eero Tuovinen wrote:
RE: Theory, or anecdotally supported opinion?
clehrich wrote:
This is a particular kind of science, and by no means its totality. I would cite anthropology as a social science that does not, in the main, use this sort of data to any great extent.
Indeed, that's what I tried to drive home above. Just compare and contrast what Chris tells us about his particular brand of humanistic research, emphasizing qualitative analysis over quantitive, with my above outline of philosophical art theory, which really isn't that interested in either type of empiricism. Just consider: with three posts Steve, me and Chris have outlined three completely different viewpoints, which all can be, and have been, called "science" in the European culture. I'd be very careful in just assuming that one of these is the a priori correct approach to our goals as roleplaying theorists.
Just so it's clear for the reader: although Chris and I are both incorrigible intellectualists of European flavor, we have different goals and methods in our approach to roleplaying theory. So don't go lumping us together when you start bashing, even if you hate everybody who uses big words ;)
That form of scientific analysis is not, in the main, predictive. In the context of RPG theory, that means that such analysis will not necessarily have practical value for the construction, design, or play of games. It may do so, but this is not an essential criterion of legitimacy or validity.
This is where my notions about the nature of roleplaying theory differ with Chris. While he draws from modern qualitative empirism for his methods and goals, I'd like to suggest that roleplaying theory is "about" pure aesthetic analysis in the analytical philosophical tradition. Consider: why are we all so interested in roleplaying theory? For myself I can say that it's because I like to understand how art (in this case rpgs) works. Simple. This understanding is precious because it's a straight, powerful tool of game design and play.
Now, the above notion of priorities leads us straight to postmodernism. I don't really care about whether GNS is "true" or "falsifiable" or anything like that. I care about whether it works, and on that I have no doubt at this stage. The same holds true for any other theory of art: if it inspires and guides me to do stuff I couldn't without it, it's obviously good.
An example: last winter I wrote a short analytic piece on S&S literature, giving a simplified framework for roleplaying that emulates the genre. The central claim of the "theory" (in the philosophical sense of the word) was that S&S is "about" power relationships between male protagonists, in a manner analogous to a wolf pack. Key content of a S&S story would be domination, submission, ownership (of women and riches) and stuff like that.
And do you know what? I'm not that interested in whether the above theory is "true". It's an interesting and fresh viewpoint, and it allows easy construction of an accompanying game system that structures play around the notions of the theory. Regardless I hold that the above snippet is "roleplaying theory" as a part of the overlapping category of literary theory. It's "true" in an intuitive way.
Contrast and compare with the priorities an empirist would bring to the issue.
In order to formulate a strong claim in such theory, one needs (1) concrete data, usually in anecdotal form; (2) a strong argument that such data can be generalized to at least some degree; (3) rigorous argumentation at the logical level.
A strong philosophical claim, on the other hand, is structured of (1) interpretation of an issue into abstract axioms; (2) deductive logic applied to the abstraction; (3) results applied back to reality to prove accordance with whatever.
Compare both with the standard empiristic method, with (1) gathering of data; (2) statistical analysis to retrieve axioms; (3) possible logical arguments to deduce further facts.
Quite different disciplines. In Finland (and in most of Europe) all three are counted "science", even philosophy.
I would argue that an anthropological and humanistic approach is appropriate to this medium, because the issues that in my opinion are most interesting are primarily those arising from social interaction and the formation of discourse. These things are most effectively and richly studied through humanities-style methods (including cultural anthropology loosely within that category).
My argument, as has been made clear above, holds that the real issue is understanding how and why certain acts work and others don't, as far as roleplaying is concerned. Or, to say it in other words, "What is beautiful, and why?", the aesthetic question. Swap in "entertaining" or whatever if you prefer.
In this matter introspective philosophical analysis is the strongest tool, and has actually been behind all important claims of roleplaying theory. An example: who thinks that Robin's player types are in the least interesting? I don't. The reason for this is that the player types in that theory are derived empiristically (through essentially the method Chris proposes) and do not shed light into anything we don't already know. They are arbitrary. Compare with GNS: that theory is structured through deductive reasoning to make clear what was previously hidden. It increases our understanding of what and why. And I'll eat my hat if Ron or his precedessors based their insight on either qualitative or quantitive research of anything. No, the method applied is clearly introspective.
On 12/7/2004 at 5:17pm, John Kim wrote:
RE: Theory, or anecdotally supported opinion?
clehrich wrote: I would argue that an anthropological and humanistic approach is appropriate to this medium, because the issues that in my opinion are most interesting are primarily those arising from social interaction and the formation of discourse. These things are most effectively and richly studied through humanities-style methods (including cultural anthropology loosely within that category).
I would, however, like to see some good statistical data as a form of background information, but it's going to take a professional, well-trained scholar to do this, and I don't see that happening any time soon.
Well, I agree partially -- in that poorly-handled statistical data can easily be worse than useless. However, both Eero and Chris here seem to dismiss the idea of collecting useful data. I think it would be quite possible and interesting to do. (No harder than doing "real" anthropological or humanities type studies, say.) For example, I have done statistics for the profiling threads, most recently Fall 2004, and many people found them interesting. I think with a little support and advice a more extensive survey could be very interesting. However, it's not just a matter of "go out and start polling". It is much better to organize and cooperate to do a broad survey, rather than having bunch of people individually collecting scattered different bits of polling data.
Chris has a point that cultural anthropology and certain other social sciences often don't use such statistical methods. A good example would be Gary Alan Fine's sociological study, "Shared Fantasy". On the other hand, I could argue that psychological studies of different activities frequently use statistical methods. Psychology seems at least as applicable to role-play as cultural anthropology. A good example is Nick Yee's study, Facets which analyzed motivations for playing MMORPGs (specifically EverQuest). The point being, I think both approaches have their place, and it would be nice to see a similar study applied to tabletop and/or LARPs.
Forge Reference Links:
Topic 13546
On 12/7/2004 at 5:33pm, Eero Tuovinen wrote:
RE: Theory, or anecdotally supported opinion?
Good overview, John. I agree on most counts, and wouldn't like to claim that there's only one true roleplaying theory methodology. However, a little correction: your Profiling statistics, interesting as they are, are not roleplaying theory. They're just data. For theory you'd have to do some analysis on the stuff, and what kind of analysis does one get from this kind of data? The kind that tells us about statistical popularity of certain games, most likely. Something interesting about short-term popularity versus long-term, perhaps. In any case, it's quite far from what's usually considered roleplaying theory here. I don't know if it's because statistical analysis is really untenable as a rpg theory method or because we just don't attract that kind of talent here, though.
On 12/7/2004 at 5:50pm, John Kim wrote:
RE: Theory, or anecdotally supported opinion?
Eero Tuovinen wrote: Good overview, John. I agree on most counts, and wouldn't like to claim that there's only one true roleplaying theory methodology. However, a little correction: your Profiling statistics, interesting as they are, are not roleplaying theory. They're just data. For theory you'd have to do some analysis on the stuff, and what kind of analysis does one get from this kind of data? The kind that tells us about statistical popularity of certain games, most likely. Something interesting about short-term popularity versus long-term, perhaps. In any case, it's quite far from what's usually considered roleplaying theory here. I don't know if it's because statistical analysis is really untenable as a rpg theory method or because we just don't attract that kind of talent here, though.
Sure. I thought I was clear that I was citing the profiling statistics only as data collection, not as theory. Nick Yee's work was the example of theory you can do with such data. You can correlate statements of general preference, or real-life data, with preference of game. For example, you could look at frequency of play versus which games preferred. Or look at whether you game with close friends or gaming-only acquaintences, and how that correlates with games preferred.
And yes, it's different than the sort of theory which historically has been discussed at the Forge -- which seems most strongly influenced by drama theory like Egri and some literature theory. I think cross-pollination with other approaches like anthropology and psychology is potentially very rewarding.
On 12/7/2004 at 7:13pm, Vaxalon wrote:
RE: Theory, or anecdotally supported opinion?
Here's a link that may serve to inform this discussion:
Therapy is Fantasy: Roleplaying, Healing and the Construction of Symbolic Order
On 12/8/2004 at 12:01am, beingfrank wrote:
RE: Theory, or anecdotally supported opinion?
I have been thinking again about the survey issue since John Kim posted the Profiling thread data. It would be neat to do a survey, get some data, analyse it, and be able to say something about the theory on the basis of that. But my problem is that I can't think of anything to test. The Big Model doesn't seem to have anything that's easily testable. Maybe that's a product of my limited understanding? The theory basically seems to say that self reports of actual play are inherently unreliable and that the only way to accurately work out what's happening in a game in terms of the model is to have an expert observe actual play and pick up on various (so far only partially documented) tells. And that a less reliable option is a detailed series of probing questions tailored to the individual situation. There's nothing there that can be whacked in a survey, stuck on the net and done by 400 people. The theory seems to suggest that asking people about their own experience and using what they tell you first as data is pointless. That makes it tricky.
I certainly don't think I understand the theory completely, and if anyone could come up with something that could be measured in a survey, I'd love to be involved in the design and analysis. Doesn't have to be Big Model. It could be any aspect of roleplaying theory. And probably the smaller, the better.