Discrimination, the Good Kind

I’ve been searching for a good reference article on testing methodologies and how to ensure a test is a good discriminator. I haven’t had much luck. I know that this discipline exists but I’m not sure what it’s called – “testing theory” perhaps?

Let me start by sucking the fun right out of pinball and describing each game as a test. It’s a really enjoyable test, most of the time, but it’s testing your skills and giving you a score, which you can then compare against others on the same machine under the same conditions. A machine is a good discriminator of skill if it rewards a variety of approaches to play, has balanced scoring, and is neither too hard nor too easy.

It’s fairly obvious that if a machine has one easy shot that is worth more than anything else, then play on that machine becomes an exercise in repeating that shot – more a test of endurance than skill. At PAPA 6, the Video Mode on Junkyard comes to mind, as does Riverboat Gambler at Pinball Expo. But what’s less obvious is the “too hard / too easy” requirements.

Imagine that the entrance exam to the world’s most prestigious college is insanely difficult, and for the purposes of discussion, imagine that it’s all multiple-choice questions. If the test is so difficult that a bright, qualified applicant might only know 5% of the answers, but there are 5 choices for each question, then simple guessing will tend to yield a score of 20%, making the 5% of actual knowledge much harder to discern in the results. In fact, you’ll be admitting a lot of lucky people who may actually be morons.

In pinball, the issue is luck versus skill. If the game is so hard that nobody can keep the ball in play, then the skill factor is replaced by luck – whoever gets the lucky bounce will get the most points. Conversely, if a game is too easy, you’re back to the endurance issue (and your tournament – oh, that’s right, we run a tournament and that’s why I’m talking here – won’t run very efficiently).

Obviously, this is some of the reasoning behind game selection and setup in each division at PAPA. Sometimes we might make a game a little too hard in A, but that doesn’t hurt too much – players can pick their machines, and decide if they want to take a chance (higher luck factor) on an unusually difficult machine.


  1. August 31, 2009 at 12:54 pm

    What’s most interesting to me about the gauntlet you’ve laid out in the example here, and at PAPA live, is how participation in the system is a pre-requisite for critique of the system. A voluntary action necessitates the equation… motivation fascinates me. Maybe someone can touch on why they travel to compete, to enter the arena?

    Another thing that occurs to me is how, in terms of pinball, the player best benefits from observing their own approach to the games in the arena, and is subsequently making adjustments within themselves in order to thrive amidst the gambit that is PAPA qualifying. Sculpting self-awareness around the game. That’s one hell of a disciplinary exercise. Makes me wonder what is sacrificed for focus.

    And that there’s so much more to success at PAPA than choosing the 5 games upon which you’ll make your qualifying attempt, but choosing your path through each game as it occurs… As though you have control over anything beyond hitting the start button.

  2. Bowen Kerins
    September 2, 2009 at 12:12 am

    Very good points and an interesting concept. One thing to add is that the format of PAPA qualifying encourages players to stay away from what I call “randomizers”, games that seem to bring out that lucky 20%. Because a consistent, all-good-but-no-great-games run is rewarded in PAPA qualifying, there is incentive to stick to the games that are less likely to give house balls or bizarre bounceouts.

    (Except for Classics, where that can’t be avoided at all.)

    If the format were a single-game qualifying like Pinball Expo, or a sum-your-games qualifying like Seattle / CA Extreme, then more players may prefer the randomizers! But at PAPA, I think choosing randomizing games (such as this year’s Monster Bash and Ripley’s) is asking for trouble.

    And during the games — there is definitely some control to be had! I say keep it simple, and stick to shots that are easy to figure out. Personally I like to shoot the easiest shots in the game until I’m used to them, then branch out as much as the game will allow. And of course there are some shots you just stay the hell away from, because they’re too risky to bother.

  3. hatThrowa
    September 2, 2009 at 3:28 pm

    Something else I’d like to the add to the equation is also window of opportunity. Some machines (the start of multiball in Who Dunnit?, for example) have make it or break it shots that easily can separate an awful game, from a top ranking game. I’ve found that it pays to stay away from these kinds of games when putting together an entry, because it puts more emphasis on making ONE thing happen at a time, rather than being able to recoup and make up for this in a more well-rounded game.

    extending off of sigma’s categorizations: game’s with make it or break it window’s of opportunity are more digital( scan-tron ), and less analog ( short answer ).

  4. Joe
    September 4, 2009 at 2:10 am

    I find Kevin’s comments about hard vs. easy, skill vs. luck to be interesting, in the sense of opening a big can of worms. There are different facets to “hard” and “easy”, and there are also different facets of “skill”. (There are arguably also different facets to “luck”, but I’m not going there… 🙂

    Some pinball games are inherently “hard”, even when configured more like typical location games as opposed to PAPA standards (“PAPA standards” –> steep, fast, tilts anywhere from moderate to sneeze, Extra Hard software settings). An example might be Shadow: many of the key shots are tight and/or potentially drainy, and there’s no single game feature you can point to as having safe “gimme” points.

    Other games have both “easy” and “hard” aspects but the game software is balanced well enough such that neither is an obvious choice. I think Whirlwind is a nice example here: I have a good chance of being able to just pound the right ramp a few dozen times and get a moderate score — somewhat easy, though endurance becomes a factor. Or I can try for multiball (somewhat hard), and IF I can start multiball and IF I can hit the side ramp a few times during multiball (quite hard) I’ll blow the aforementioned moderate score away — but if I fail at that whole setup, my score will probably suck. Great risk/reward balance.

    In examples like these, I think that the game doesn’t have to be set to 9 degree slope or sneeze tilt to be “hard” (or have hard aspects) — just selection of a well designed, well balanced game takes care of business.

    I figure sometimes the selection and configuration of games at PAPA is designed to increase the number of tickets sold — nothing wrong with that, of course, we’ve gotta fund the very impressive purse somehow. I think one possible approach would be the concept of the “buy-in extra game” — I think this has come up before. The idea would be that once per ticket, a player could pay the same price as a whole ‘nother ticket in their division to replay one game on their active ticket. (It would be an interesting discussion in itself whether the replay had to be on the same game, or if the player could choose any game in the bank — either way, any earlier score on a given title would be overwritten by the replay, no duplicates.) I think this would provide a very interesting new strategic element to game selection, while generating considerable additional revenue for the tournament, reducing the need to cut down on game times simply to increase the number of tickets that get pumped through.

    BTW, I told Kevin before, but it bears repeating: nice job on the tournament game selection and configuration this year. I thought it was a really nice spread, and very fair. Great conditions on the floor, too.

Comments are closed.