Secrets of the U.S. Puzzle Championship

Wordstar, Nurikabe, Double Minesweeper, and the rest of the puzzles in this year’s U.S. Puzzle Championship (USPC) were kept under tight security until the last possible instant. At precisely 1 p.m. EST on May 17, a password was released to open the protected file, and this year’s contestants had a frantic 150 minutes in front of them—printing out puzzles, penciling in solutions, and hoping to submit results to the server before time expired. The test, which determined who would compete for the American team at the World Puzzle Championships (WPC) in London, was challenging even for experts, but it was also eagerly anticipated by amateur enthusiasts. 2,180 hopefuls registered for the USPC and downloaded the puzzles, but only 273 submitted answers. Some people were surely in it for the competitive glory, but it seems most were in it for fun—a familiar dichotomy that’s hardly unique to puzzling.

I’ve always loved puzzles. They combine the joy of revelation with the satisfaction of effort rewarded. Nothing tops the epiphany of realizing that the 9 in the corner of the Sudoku means the middle square can’t be a 4. It feels like magic. And when that final square is filled in, the untold minutes spent in intense focus, locked out from the rest of the world, become justified. I once got an email from the director of a Sudoku tournament wishing me many “nice moments” with the puzzles. Nice moments—to me that’s what solving is about.

For the handful of people who had reason to be sanguine about finding a spot on the 2014 U.S. team, the USPC started in earnest 24 hours before the test itself. That’s when Nick Baxter, U.S. team captain and administrator of the diabolical USPC exam, released the “Instructions.” Solvers could then see examples of each puzzle type that would be included on the test, learn the point allocation for each puzzle, and even get the names of the puzzles’ designers. If you were experienced enough, knowing who created a puzzle provided a big clue for mastering its internal logic.

I was one of the 1,907 people who downloaded the puzzles but didn’t bother submitting results. I also didn’t bother to read the “Solving Tips” section of the USPC website, which recommended printing out the easier puzzles first. At the start of my own personal 150-minute crucible, I grabbed the first piece of paper off my printer and set to work. My poorly chosen adversary was puzzle number 23, Sukazu Gaiden. Valued at 35 points—ten more than any other puzzle on the test—Sukazu Gaiden was probably the toughest of the lot. Even I knew as much. But the puzzle seemed at to have Soduku-like elements, and hence, an internal logic I might be able to crack, so I decided to give it a shot. At the very least, I hoped for a few nice moments.

The Instructions (which would’ve been available to me the day before) read: “Place digits into the grid, one per square. For each row, column, and outlined region, the number of instances of each digit is equal to that digit. Orthogonally neighboring squares cannot contain the same digit.”

The example:

And finally, the puzzle itself:

I stared at it for a while. After seven minutes I decided that if I didn’t make any headway by the twenty-minute mark, I’d move on and try something else. At the twenty-minute mark, I had a sense of what was going on. I realized there had to be a lot of 4s. The sections containing seven, eight, or nine squares all needed exactly four 4s, none of which could touch each other. That strong constraint, coupled with the sheer abundance of 4s, allowed me to start filling in the grid with confidence. After 36 minutes I’d placed all the 4s, and I thought I was getting close to a full solution. Then in the 47th minute, disaster struck. I needed to place a 2, and had nowhere to put it. Any of the available spots I’d left myself would’ve violated the puzzle’s rules. I’d been treading cautiously so as not to do anything dumb, but I’d obviously made a mistake somewhere. My only hope was that the error had come in the last few steps and I could go back and correct it. In other words, I was screwed. At 50 minutes I wrote, “give up, crying” in my notes.

I described my humiliation to Baxter, and he assured me these kinds of mistakes happened even to the very best—and that even the very best wouldn’t have tried a puzzle like Sukazu Gaiden without studying a little first. Puzzles are vulnerable to tactics. Sudokus, for example, have become so well analyzed that the various solving heuristics—techniques that have been built up from logic—have their own names like Jelly-Fish, WXYZ Wing, Death Blossom, and Bowman’s Bingo (plus a whole lot more). The 24-hour pre-test cram session can help solvers create their own heuristics specifically for the puzzle variations in the USPC. That way, “when you get under the gun in the competition, you’re not learning this blind,” Baxter said.

It makes sense. In my career as a professional poker player, I’ve worked out countless strategic shortcuts away from the table. If I’d had to figure out on the spot how much to bet, or what kinds of hands my opponent might play from what position, or when my stack size had gotten small enough to dictate a change in plans, I probably would’ve collapsed from an overheated brain long ago. But in trying to solve a puzzle without first learning the heuristics, I was inviting my brain to overheat in exactly that way.

Baxter was gracious enough to provide me with data for this year’s USPC. Every time a participant submitted an answer or series of answers to the USPC website, Baxter logged the following information: 1) which puzzles had been attempted, 2) whether any answers had been changed from that participant’s previous submission, 3) whether any new puzzles had been attempted since a previous submission, and 4) the server time.

I wanted an idea of just how abysmal my performance on Sukazu Gaiden was, so I went into Baxter’s data and isolated those 34 submissions where, from the time of the person’s previous submission, only Sukazu Gaiden had been added. In other words, I was interested in the time it took to submit an answer for just that puzzle (I’ll refer to this as “average solo response time” going forward). Ignoring any responses of less than a minute (Baxter assured me no one could start and finish these puzzles so fast), I found that Sukazu Gaiden’s average solo response time was 27 minutes, 28 seconds. This turned out to be the second-highest of all the puzzles, behind only  #13—a 20-pointer called Star Search. Of course, there’s no way to tell if other puzzles were attempted unsuccessfully between one submission and the next, so we’ll never know if the contestant who answered Star Search nearly 100 minutes into the test had first tried one or more other puzzles without reaching a solution, or if he/she simply started late. But that is an irreconcilable limitation of the data, and it doesn’t get in the way of some telling analysis.

*ignoring puzzle 4, Holiday BBQ, because it allows the submission of partial answers

To help craft a potential strategy for solvers who wanted to improve their scores (and maybe even give themselves a shot at qualifying for the team), I dove deeper into the data. After calculating each puzzle’s Average Solo Response Time, I then computed a simple Points Per Minute metric. This is pretty much what it sounds like—the potential points earned for each minute spent working on the puzzle. A genius who finished all 23 puzzles in 150 minutes and had a shot at a perfect score of 380, would average 380/150 PPM, or 2.53.

In looking over the numbers, I discovered that there were clearly benefits to working on the hardest questions. Only 2.6% of participants even attempted puzzle #15, Geometric Distribution, but those who did finished in an average time of 11 minutes 38 seconds—second fastest, trailing only puzzle #19 (another tough one called Digital Clutter). As a result, an attempt at solving Geometric Distribution had the potential to earn a whopping 1.72 points per minute. The earlier, easier puzzles, meanwhile, had the puniest rate of return.

But did the easy puzzles have weaker returns because less talented folks like myself were trying them? To better understand what was happening, I decided to divide the participants into two tiers—call them elite solvers, and ordinary solvers. Elites (127 participants) submitted more than 100 points worth of puzzles. Ordinary solvers (146 participants) submitted 100 points or fewer.

**Only includes puzzles with at least five solo responses from each solver group 

As expected, when looking only within one solver group, the PPM data becomes more uniform. Elites solved the first three puzzles 7, 11, and 16 minutes faster, respectively, than ordinary solvers did, maintaining a relatively robust 0.83 PPM for those questions. The ordinary solvers, meanwhile, struggled to a PPM of just 0.40 over the same set.

Those first three puzzles, valued at 10, 5, and 10 points, all used styles well known to competitive solvers. Battleships, Cave, Masyu—veteran puzzlers have seen these a million times before, and have learned all their tricks. In fact, for the six puzzles that used a previously seen style (the three just mentioned, plus #18 Nurikabe, #20 Sudoku, and #21 Tapa), elites averaged 1.18 PPM, while ordinary solvers averaged only 0.62. For the other eleven puzzles that got enough responses, elites fell to 0.92 PPM, while ordinary solvers dropped off much less—to 0.51. For newer puzzle variations, when no one knew any tried-and-true solving heuristics, some of the experienced solver’s advantage got taken away. Conversely, on the familiar puzzles, the elites crushed us mortals. This was most evident in the most familiar puzzle of all—#20, a straight Sudoku. Elites smashed this puzzle for 2.06 PPM, easily their best result. Ordinary solvers, meanwhile, managed only 0.83 PPM, by far the biggest gap between the groups of any puzzle on board.

It’s perhaps counterintuitive, but the data shows that an ordinary solver—a person who hasn’t devoted the time and effort to be a real contender—would do well to attack the later puzzles, or at least the mid-level puzzles, first. The earlier puzzles are easier, but they’re easier largely because a lot of solvers already know how to do them. If you have to figure out a puzzle on your own, you might as well figure out one that will reward you with 20 or 25 or 35 points.

Do I aspire to be an expert solver—one who studies solving algorithms and prepares for new puzzles in advance? Or am I content to be a blank slate, coming at every puzzle with only pencil, paper, and my wits? There’s something appealing about inventing my own logic with every puzzle, of never relying on a piece of code hard-wired into memory. I’m still in the market for nice moments, and in my decade-plus of competing professionally at the poker tables, I’ve learned there are many more nice moments early on, when the game is still mysterious. But after enough years of practice, I’ve also learned something else: winning is fun, too.

Puzzle images and USPC data courtesy Nick Baxter and the U.S. Puzzle Championship.