Deconstructing Cory Doctorow’s “Experiment”

In an article in Publisher’s Weekly on 19 October—which presages a year’s worth of monthly articles—Cory Doctorow announced that he was going to engage in an “experiment.” He noted that “Free e-books work for me,” and that “People want proof that this works.” To accomplish this proof, Cory lays out an experiment in great detail, going to great lengths to present the reader with a reasoned and scientific explanation of how he will prove the claim that “free e-books work.”

Though it is a tedious undertaking, a careful examination of his methodology is critical to understanding what is really going on here. Cory Doctorow is using the language of science to legitimize a marketing scheme and making it appear as if it, or the data generated by it, has validity in the real world and significant impact on the future of publishing. As you will see below, nothing he offers has any bearing upon his thesis, methods of reportage are wholly unreliable, and his “experiment” is simply a promotional ploy worthy of P. T. Barnum and the legions of snake-oil salesmen who have gone before him.

The concession to “Science”

To establish the legitimacy of his experiment, he offers his book Overclocked (2007 Thunder’s Mouth) as his control—since all good scientific experiments have controls. He points out that the book had no publisher support, but was reprinted several times and “continues to do well.” He still gets royalty checks from it. He will use its numbers to establish the baseline against which the success of his experimental book, With a Little Help, attains—though he never actually gives us any of those numbers.

Cory’s methodology for the experiment

Cory says, just like doctors who vow to “do no harm,” he’s “taken an oath to lose no money” in this experiment. Cash in will be cash out, and he’ll keep his expenses as low as possible. That’s actually a great business plan, and the world economy would be much better off if everyone took his oath of financial responsibility.

He runs down the list of things he will be doing in his experiment, aptly pointing out that they will cost little or nothing to produce and that he will be carrying no significant inventory. His list includes:

E-books: multiple formats, distributed at the cost of bandwidth (unspecified), delivered free

Audiobooks: multiple formats, distributed at the cost of bandwidth (unspecified), created (read and engineered) by volunteers (cost unspecified), distributed free

Donations: accepted in recompense for the free work

Print-on-Demand trade paperbacks: Drop-shipped from Lulu.com, requiring no inventory. Costs: Document preparation (free, his mom will be doing the editing); production of covers by a variety of artists (compensation unspecified) and the option of custom covers for short runs, estimated set-up charge being $300.

Premium Hardcover edition: $250, limited run of 250, prepped in lots of 20 for roughly $65 US per book. Cost of each run is $1300, and if the order is not placed until there are 6 pre-orders in hand, the run is immediately profitable. The cost of the book includes $5 per for custom flashdrives, price estimate based on a net search for the unit he specifies in lots of 50.

Commissioned Story: A millionaire friend commissioned a brand new story for $10,000.

Advertisements: to be included in the POD and free e-books, price unspecified

Donation of Books: He maintains a list of places that would like a copy of the book (libraries, hospitals) and allows fans to purchase copies for those locations. His agent and Lulu are going to handle fulfillment, revenue neutral.

PW Income: PW paid $900 for the initial article, and is paying $400 for the monthly updates

Reporting of results

Cory says that each month he will compile data and let us know how his experiment is going. At the end of twelve months, we’ll know whether or not his experiment is a success.

All in all, a very ambitious program.

What’s wrong with this experiment?

In a word, his “experiment” is rubbish.

Unreliable data

The first and most important point that can be made about this experiment is the most fundamental. All of Cory Doctorow’s claims about his successes are self-generated and unaudited. We have Cory’s word alone that he’s doing well and has had wonderful success with giving away e-books. But he doesn’t supply numbers to back up these claims. As noted above, he didn’t even supply numbers on his “control” (see below) to lay out his baseline.

If Cory were as successful as he claims, his books would consistently appear on the bestseller lists. Industry news would be full of rumors of six and seven figure advances paid for his books. They aren’t. Cory himself, in laying out his experiment, notes that people want “proof” he’s not “deluded or a con artist,” but he provides no evidence to indicate that he’s not one or both.

In terms of the experiment, without clear and verified numbers, any results are meaningless.

Poorly defined thesis

Second, nothing in Cory’s program for With a Little Help supports, tests or proves his thesis: that giving away free e-books works. Giving away free e-books works at what exactly? Does it increase sales? Does it increase notoriety? Does it increase visibility or website traffic? What are we measuring, and how are we going to verify the numbers? Cory gives us no goal, no way to measure success or failure—beyond, apparently, not going broke.

The mythical “control”

Cory’s “control” is an illusion and is undercut in his very description of his control. In the article he says that all of his story collections have been released “for free” the day they were published. Since this is true of Overclocked, there is simply no way it can serve as a control to test the effects of free e-books. It had free e-book distribution. The only control to test free e-book distribution would be a book that did not have it.

Valid metrics

Third, and very importantly, if we are going to test the thesis that free distribution of e-books has an effect on sales of print books, we simply need three sets of numbers: e-books released, print books sold, and the ratio comparing the two. That’s it. We compare that to sales of similar books that didn’t have free e-book distribution and we can determine if and how much of an impact free content distribution has on print books.

If we wish to go further, we survey all those who downloaded the free e-book after three months and ask three questions: Did you read the free e-book? Did you purchase a physical copy of the book? Did receiving the free e-book prompt you to purchase the physical copy? Get those results and tally them against the aforementioned set of numbers and measure what effect free e-book distribution actually has on physical book sales.

What is Cory testing?

Fourth, the way Cory has set his experiment up is to keep score with dollars. He’s going to compare “earnings” between the previous book and this one. There is absolutely nothing wrong with being profitable. Being profitable equals success for any business venture.

Cory’s starting with $10,900 in the bank and has another $4,800 coming from Publisher’s Weekly. He’ll be out of pocket $1,300 for the first run of the limited hardback, and profitable after the sale of six books. Assuming that donations amount to enough to cover his bandwidth, that advertisements bring in any income, and that his POD books sell any copies at all, the wonder will not be if he profits, but if he actually loses any money.

But Cory is managing to keep his costs artificially low. He’s not reporting what he’ll be charged for artwork, bandwidth, website design and support. Those are legitimate costs for his experiment, and he can probably piggy-back the web expenses on Boing Boing, so they don’t appear to be an expense. For anyone else, however, they would be.

Cory’s enlisted a whole bunch of friends to do stuff for him, which is great, but is volunteer labor a sustainable business model? Voice talent of the caliber he’s using on his project can cost upwards of $2,000 a day, not including studio rental. Everyone else seems to be donating or discounting their services to him which, again, is wonderful, but results in unaccounted overhead which should be charged against the project.

What is Cory’s experiment?

Here’s what Cory’s real thesis is: Give any author $15,700 (Cory’s commission plus the PW fees) and a manuscript, and see what the return on that investment is after a year. Properly pumped in his business, utilizing the tools that Cory talks about (and plenty other new media tools he’s ignoring) my guess would be that the return on investment in a year should be at least 50%. In fact, doing just that sort of experiment is exactly the way science would test his claims. (And if someone wants to put the money up, I’ve got a manuscript.)

And, of course, in this experiment, the results would be independently accounted, audited and reported.

After that, we could then debate whether or not new media makes cottage-industry publishing a viable alternative to the modern publishing business model.

Trust, but verify

It might seem harsh to harp on the need for independent verification of numbers, but I do so for two reasons. First, in his “experiment” piece, Cory says that through podcast distribution he expects to reach “300,000 fans.” This number is unsourced and delusional.

I’ve been part of podcasting since before podcasting existed (having hosted audio files on my website in 2003 to promote my novel The Grand Crusade). I’ve won awards for my podcasting. I co-host Dragonpage Cover to Cover with Michael R. Mennenga, the founder and CEO of the Farpoint Media Network, and co-host of Slice of Scifi. Michael is one of the pioneers of podcasting and a leader in the science fiction end of podcasting. He considers 300,000 to be grossly overblown.

And even if Cory were talking about individual downloads, and not hits or page-views, that still doesn’t translate into individual listeners. Moreover, there’s no proof that a single audio file actually ever gets listened to. There is just no reporting mechanism—save for anecdotal reportage—to determine if anyone out there is listening.

In short, numbers Cory provides are fuzzy and unreliable, by a factor of at least two, and likely much more.

The second reason I’m a stickler for verification is this: in the last half-dozen years, as I’ve been working in podcasting and putting out digital media—being the first author to have books for sale in the Apple Appstore and having sold e-book material from my website since 2004—I have had people tell me that Cory Doctorow has been wildly successful. I’ve heard speakers at writers’ conferences tell beginners that Cory has used free e-book distribution to become a successful writer. And yet every single one of those individuals, when challenged to source their information, can only trace it back to what Cory has said about himself.

Cory is a fantastic self-promoter. I am in awe of his ability to talk Publisher’s Weekly into financing his “experiment” and anointing him as a guru of new media. But none of that means his claims are true. None of that means his claims are accurate. Repetition of those claims by legitimate sources means that countless people who cannot or will not use critical thinking skills to examine his claims will spend a lot of time and money on frustrating efforts that will get them nowhere. Worse, it will set them back financially and will rob them of time they could have used for improving their craft.

It is a shame that Publisher’s Weekly allowed itself to be hoodwinked. Sure, hire Cory because he’s a figure who promotes himself and, thereby, will promote you. That I get. But letting him frame a simple hustle as a defining and scientific experiment that will be used to judge the efficacy of new media as applied to the world of publication? Maybe in a land where Creationism is accepted as science, but not in the real world.

Cory’s experiment isn’t science.The only thing being tested here is our gullibility in believing this heaping serving of nonsense will prove anything at all.

TOMORROW: What Cory’s Doing Right!
MONDAY: What Cory isn’t doing, and should be.