Cracking Old Secret Codes

Stop what you're doing.

Now, read this superlative article by Noah Shachtman at Wired about how an old complicated code describing a secret society's rituals was finally cracked using sassy computer software created to help translate foreign languages:

Knight was part of an extremely small group of machine-translation researchers who treated foreign languages like ciphers—as if Russian, for example, were just a series of cryptological symbols representing English words. In code-breaking, he explained, the central job is to figure out the set of rules for turning the cipher’s text into plain words: which letters should be swapped, when to turn a phrase on its head, when to ignore a word altogether. Establishing that type of rule set, or “key,” is the main goal of machine translators too. Except that the key for translating Russian into English is far more complex. Words have multiple meanings, depending on context. Grammar varies widely from language to language. And there are billions of possible word combinations.

But there are ways to make all of this more manageable. We know the rules and statistics of English: which words go together, which sounds the language employs, and which pairs of letters appear most often. (Q is usually followed by a u, for example, and “quiet” is rarely followed by “bulldozer.”) There are only so many translation schemes that will work with these grammatical parameters. That narrows the number of possible keys from billions to merely millions.

The next step is to take a whole lot of educated guesses about what the key might be. Knight uses what’s called an expectation-maximization algorithm to do that. Instead of relying on a predefined dictionary, it runs through every possible English translation of those Russian words, no matter how ridiculous; it’ll interpret as “yes,” “horse,” “to break dance,” and “quiet!” Then, for each one of those possible interpretations, the algorithm invents a key for transforming an entire document into English—what would the text look like if meant “break dancing”?

The algorithm’s first few thousand attempts are always way, way off. But with every pass, it figures out a few words. And those isolated answers inch the algorithm closer and closer to the correct key. Eventually the computer finds the most statistically likely set of translation rules, the one that properly interprets as “yes” and as “quiet.”

The problem?

Hundreds of thousands of Europeans belonged to secret societies in the 18th century, Önnerfors explained to Megyesi; in Sweden alone, there were more than a hundred orders. Though they were clandestine, they were often remarkably inclusive. Many welcomed noblemen and merchants alike—a rare egalitarian practice in an era of strict social hierarchies. That made the orders dangerous to the state. They also frequently didn’t care about their adherents’ Christian denomination, making these orders—especially the biggest of them, Freemasonry—an implicit threat to the authority of the Catholic Church. In 1738 Pope Clement XII forbade all Catholics from joining a Masonic lodge. Others implied that the male-only groups might be hotbeds of sodomy. Not long after, rumors started that members of these orders actually worshipped the devil.

These societies were the incubators of democracy, modern science, and ecumenical religion. They elected their own leaders and drew up constitutions to govern their operations. It wasn’t an accident that Voltaire, George Washington, and Ben Franklin were all active members. And just like today’s networked radicals, much of their power was wrapped up in their ability to stay anonymous and keep their communications secret...

Read the whole magnificent, gripping thing.

Note that the code-crackers lost time as they had not taken on board the skills of Sherlock Holmes, who broke a rather less tricky code of Dancing Men in part by realising that one of the symbols represented a break between words...