Deconstructing Cory Doctorow’s “Experiment”

Deconstructing Cory Doctorow’s “Experiment”

In an article in Publisher’s Weekly on 19 October—which presages a year’s worth of monthly articles—Cory Doctorow announced that he was going to engage in an “experiment.” He noted that “Free e-books work for me,” and that “People want proof that this works.” To accomplish this proof, Cory lays out an experiment in great detail, going to great lengths to present the reader with a reasoned and scientific explanation of how he will prove the claim that “free e-books work.”

Though it is a tedious undertaking, a careful examination of his methodology is critical to understanding what is really going on here. Cory Doctorow is using the language of science to legitimize a marketing scheme and making it appear as if it, or the data generated by it, has validity in the real world and significant impact on the future of publishing. As you will see below, nothing he offers has any bearing upon his thesis, methods of reportage are wholly unreliable, and his “experiment” is simply a promotional ploy worthy of P. T. Barnum and the legions of snake-oil salesmen who have gone before him.

The concession to “Science”

To establish the legitimacy of his experiment, he offers his book Overclocked (2007 Thunder’s Mouth) as his control—since all good scientific experiments have controls. He points out that the book had no publisher support, but was reprinted several times and “continues to do well.” He still gets royalty checks from it. He will use its numbers to establish the baseline against which the success of his experimental book, With a Little Help, attains—though he never actually gives us any of those numbers.

Cory’s methodology for the experiment

Cory says, just like doctors who vow to “do no harm,” he’s “taken an oath to lose no money” in this experiment. Cash in will be cash out, and he’ll keep his expenses as low as possible. That’s actually a great business plan, and the world economy would be much better off if everyone took his oath of financial responsibility.

He runs down the list of things he will be doing in his experiment, aptly pointing out that they will cost little or nothing to produce and that he will be carrying no significant inventory. His list includes:

E-books: multiple formats, distributed at the cost of bandwidth (unspecified), delivered free

Audiobooks: multiple formats, distributed at the cost of bandwidth (unspecified), created (read and engineered) by volunteers (cost unspecified), distributed free

Donations: accepted in recompense for the free work

Print-on-Demand trade paperbacks: Drop-shipped from Lulu.com, requiring no inventory. Costs: Document preparation (free, his mom will be doing the editing); production of covers by a variety of artists (compensation unspecified) and the option of custom covers for short runs, estimated set-up charge being $300.

Premium Hardcover edition: $250, limited run of 250, prepped in lots of 20 for roughly $65 US per book. Cost of each run is $1300, and if the order is not placed until there are 6 pre-orders in hand, the run is immediately profitable. The cost of the book includes $5 per for custom flashdrives, price estimate based on a net search for the unit he specifies in lots of 50.

Commissioned Story: A millionaire friend commissioned a brand new story for $10,000.

Advertisements: to be included in the POD and free e-books, price unspecified

Donation of Books: He maintains a list of places that would like a copy of the book (libraries, hospitals) and allows fans to purchase copies for those locations. His agent and Lulu are going to handle fulfillment, revenue neutral.

PW Income: PW paid $900 for the initial article, and is paying $400 for the monthly updates

Reporting of results

Cory says that each month he will compile data and let us know how his experiment is going. At the end of twelve months, we’ll know whether or not his experiment is a success.

All in all, a very ambitious program.

What’s wrong with this experiment?

In a word, his “experiment” is rubbish.

Unreliable data

The first and most important point that can be made about this experiment is the most fundamental. All of Cory Doctorow’s claims about his successes are self-generated and unaudited. We have Cory’s word alone that he’s doing well and has had wonderful success with giving away e-books. But he doesn’t supply numbers to back up these claims. As noted above, he didn’t even supply numbers on his “control” (see below) to lay out his baseline.

If Cory were as successful as he claims, his books would consistently appear on the bestseller lists. Industry news would be full of rumors of six and seven figure advances paid for his books. They aren’t. Cory himself, in laying out his experiment, notes that people want “proof” he’s not “deluded or a con artist,” but he provides no evidence to indicate that he’s not one or both.

In terms of the experiment, without clear and verified numbers, any results are meaningless.

Poorly defined thesis

Second, nothing in Cory’s program for With a Little Help supports, tests or proves his thesis: that giving away free e-books works. Giving away free e-books works at what exactly? Does it increase sales? Does it increase notoriety? Does it increase visibility or website traffic? What are we measuring, and how are we going to verify the numbers? Cory gives us no goal, no way to measure success or failure—beyond, apparently, not going broke.

The mythical “control”

Cory’s “control” is an illusion and is undercut in his very description of his control. In the article he says that all of his story collections have been released “for free” the day they were published. Since this is true of Overclocked, there is simply no way it can serve as a control to test the effects of free e-books. It had free e-book distribution. The only control to test free e-book distribution would be a book that did not have it.

Valid metrics

Third, and very importantly, if we are going to test the thesis that free distribution of e-books has an effect on sales of print books, we simply need three sets of numbers: e-books released, print books sold, and the ratio comparing the two. That’s it. We compare that to sales of similar books that didn’t have free e-book distribution and we can determine if and how much of an impact free content distribution has on print books.

If we wish to go further, we survey all those who downloaded the free e-book after three months and ask three questions: Did you read the free e-book? Did you purchase a physical copy of the book? Did receiving the free e-book prompt you to purchase the physical copy? Get those results and tally them against the aforementioned set of numbers and measure what effect free e-book distribution actually has on physical book sales.

What is Cory testing?

Fourth, the way Cory has set his experiment up is to keep score with dollars. He’s going to compare “earnings” between the previous book and this one. There is absolutely nothing wrong with being profitable. Being profitable equals success for any business venture.

Cory’s starting with $10,900 in the bank and has another $4,800 coming from Publisher’s Weekly. He’ll be out of pocket $1,300 for the first run of the limited hardback, and profitable after the sale of six books. Assuming that donations amount to enough to cover his bandwidth, that advertisements bring in any income, and that his POD books sell any copies at all, the wonder will not be if he profits, but if he actually loses any money.

But Cory is managing to keep his costs artificially low. He’s not reporting what he’ll be charged for artwork, bandwidth, website design and support. Those are legitimate costs for his experiment, and he can probably piggy-back the web expenses on Boing Boing, so they don’t appear to be an expense. For anyone else, however, they would be.

Cory’s enlisted a whole bunch of friends to do stuff for him, which is great, but is volunteer labor a sustainable business model? Voice talent of the caliber he’s using on his project can cost upwards of $2,000 a day, not including studio rental. Everyone else seems to be donating or discounting their services to him which, again, is wonderful, but results in unaccounted overhead which should be charged against the project.

What is Cory’s experiment?

Here’s what Cory’s real thesis is: Give any author $15,700 (Cory’s commission plus the PW fees) and a manuscript, and see what the return on that investment is after a year. Properly pumped in his business, utilizing the tools that Cory talks about (and plenty other new media tools he’s ignoring) my guess would be that the return on investment in a year should be at least 50%. In fact, doing just that sort of experiment is exactly the way science would test his claims. (And if someone wants to put the money up, I’ve got a manuscript.)

And, of course, in this experiment, the results would be independently accounted, audited and reported.

After that, we could then debate whether or not new media makes cottage-industry publishing a viable alternative to the modern publishing business model.

Trust, but verify

It might seem harsh to harp on the need for independent verification of numbers, but I do so for two reasons. First, in his “experiment” piece, Cory says that through podcast distribution he expects to reach “300,000 fans.” This number is unsourced and delusional.

I’ve been part of podcasting since before podcasting existed (having hosted audio files on my website in 2003 to promote my novel The Grand Crusade). I’ve won awards for my podcasting. I co-host Dragonpage Cover to Cover with Michael R. Mennenga, the founder and CEO of the Farpoint Media Network, and co-host of Slice of Scifi. Michael is one of the pioneers of podcasting and a leader in the science fiction end of podcasting. He considers 300,000 to be grossly overblown.

And even if Cory were talking about individual downloads, and not hits or page-views, that still doesn’t translate into individual listeners. Moreover, there’s no proof that a single audio file actually ever gets listened to. There is just no reporting mechanism—save for anecdotal reportage—to determine if anyone out there is listening.

In short, numbers Cory provides are fuzzy and unreliable, by a factor of at least two, and likely much more.

The second reason I’m a stickler for verification is this: in the last half-dozen years, as I’ve been working in podcasting and putting out digital media—being the first author to have books for sale in the Apple Appstore and having sold e-book material from my website since 2004—I have had people tell me that Cory Doctorow has been wildly successful. I’ve heard speakers at writers’ conferences tell beginners that Cory has used free e-book distribution to become a successful writer. And yet every single one of those individuals, when challenged to source their information, can only trace it back to what Cory has said about himself.

Cory is a fantastic self-promoter. I am in awe of his ability to talk Publisher’s Weekly into financing his “experiment” and anointing him as a guru of new media. But none of that means his claims are true. None of that means his claims are accurate. Repetition of those claims by legitimate sources means that countless people who cannot or will not use critical thinking skills to examine his claims will spend a lot of time and money on frustrating efforts that will get them nowhere. Worse, it will set them back financially and will rob them of time they could have used for improving their craft.

It is a shame that Publisher’s Weekly allowed itself to be hoodwinked. Sure, hire Cory because he’s a figure who promotes himself and, thereby, will promote you. That I get. But letting him frame a simple hustle as a defining and scientific experiment that will be used to judge the efficacy of new media as applied to the world of publication? Maybe in a land where Creationism is accepted as science, but not in the real world.

Cory’s experiment isn’t science.The only thing being tested here is our gullibility in believing this heaping serving of nonsense will prove anything at all.

TOMORROW: What Cory’s Doing Right!
MONDAY: What Cory isn’t doing, and should be.

Twitter Digg Delicious Stumbleupon Technorati Facebook Email

17 Responses to “Deconstructing Cory Doctorow’s “Experiment””

  1. Great analysis. And kudo’s to a marvelous promotional scheme from Doctrorow. Free is a great strategy for some, disastrous for others, and simply unfair to those who participate because they are given no choice.

    Another element I would add in reviewing the usefulness of these results – Marketing concepts are continually evolving due to tech and taste. While Doctrorow may prove that Free is the right strategy this year, we all are faced with immediately looking to what will pay the bills next year.

  2. Steve Peterson 24. Oct, 2009 at 2:37 pm

    Mike,

    Excellent breakdown of the issues involved with Cory’s “experiment.” He’s certainly good at self-promotion, but his marketing efforts won’t show any of us the value of free e-book distribution. I’m definitely interested in knowing just how much free e-books do or do not affect actual book sales, and I look forward to someone actually providing data to this end.

    As e-book reading gets easier and more booklike (Kindles, iPhones, etc. and their future improvements) I’d expect giving away your e-book to be a less and less viable business model, even assuming that giving it away now results in more hardcopy sales. The technology is changing the market rapidly, and today’s answers probably won’t be the same next year.

    — Steve Peterson

  3. I agree with your analysis of the bias in this ‘experiment’. I do think that there is merit to the idea, at least for authors with a body of work and a fan base. I recall an article in Baen’s Universe a couple of years ago by Eric Flint where he cited sales of books for authors whose earlier works were posted in the Baen Free Library. They noticed a spike in sales of those books currently in print shortly after their backlist books were posted for free. Whether that would translate into sales of new books if an e-book version were offered free is the real question.

  4. You’ve left out what I consider to be the biggest factor here: he is Cory Doctorow.

    Now if this was Joe Schmoe, would PW have asked him to write a column? Would anyone ever even find his work somewhere in the online slush pile (as it were)?

    It amuses me when people who are prominent to begin with delude themselves into thinking they are starting with the clean slate of someone who is totally unknown.

  5. Your last point gets to the heart of it, I believe. Non-established writers simply cannot afford the loss of focus and potentially of money that this would entail. Not to mention the fact that Cory Doctorow has also scored a negative loss advertisement every month with Pub Weakly (Sorry, couldn’t resist the pun) Good luck with getting a sweet deal like that, Joe Unknown. As with everything nowadays in mainstream publishing, it’s connections, connections, connections, and sometimes luck.

  6. Bravo!
    Much of this would be cleared up if Doctorow simply released his sales figures, or if Publishers Weekly reported on his bookscan sales. Assuming Doctorow knows his own sales history, the only explanation for why he never releases it is that it complicates the claims he’s making.

  7. There is no doubt, as you’ve noted here, that Cory’s experiment isn’t scientific and any results he obtains won’t be objectively attained. The mere fact that he’s announcing the experiment and identifying the book(s) involved may bias the results, due to his fans learning about the experiment and buying copies to help his sales.

    I have to admit, though, that I’m looking forward to both authors’ (Doctorow’s and Stackpole’s) upcoming articles. As someone who has both the urge to write and a full-time job, I can’t imagine myself dedicating the time and energy to promoting any novel I might write via a traditional print publisher, so I am curious to see how much money Doctorow makes via the various “packages” for his book. But I’m also quite curious to see how Mike would have conducted the experiment himself.

  8. I enjoyed this analysis very much, as I did Cory’s article, and look forward to more from both. But I think you misspeak to characterize the experiment as being one to test the hypothesis that giving away free ebooks increases the sale of physical books. I believe Cory would maintain that his previous publishing record indicates this, and that it was listed here for the benefit of introducing himself to PW readers who may not be aware of his beliefs/methods. I took this experiment to be one testing whether or not an author of his stature is better served going with or without a publisher. In which case, criticisms that this wouldn’t work for new authors are invalid. It is a response to those he describes when he says, “The True Believers are the people who say that I’m a fool to give 90% of the cover price of my books to the publisher and bookseller.”

    However, I agree that Overclocked is a poor choice for comparison, since it was not given the publishing support that he received for other novels. Little Brother would be a better comparison, since it represents the kind of publishing support he could command NOW, or at least Makers, since it is his latest adult novel, and being given support in the form of weekly installments at Tor.com right now.

    Regardless, I’ll enjoy his further articles and your further responses.

  9. Lou, you’d have a point about mischaracterization of the experiment save for two things. First, the overwhelming emphasis is on his contention that giving away free e-books works. Second, while he says he’s gathering evidence concerning the proposition that he’s giving away too much to the publishers, he has, de facto, set up an experiment that proves this proposition a priori. Heck, all writers know we give away too much to the middlemen, and that we can make more if we do it ourselves; however, we can’t do it the same way they do, we can’t get distribution into bookstores, and we can’t get as many copies out there as traditional publishers do. So, if he were going to test the idea that publishers take too much, he’d have to do all that they do (mass distribution) as opposed to running a boutique publishing operation. Will he make money? Absolutely. But will he make as much as he would have though a traditional publishing deal? Not unless his reports of “success” are, in reality a lot more modest than most believe.

  10. Geoffrey, thanks very much for the cites mentioned in your blog post concerning my blog post. It’s interesting material to study. I look forward to reading it more closely when I’m not on deadline.

  11. It’s kind of refreshing to see someone of note poke a pin in yet another of Doctorow’s bags of gas.

    If he’s so successful selling books, why does he need earnings from Boing Boing, speaking fees, column writing and (amusing to me) the “occasional grant, teaching gig or residency”—you know, opportunities available to ALL authors.

    And I love how he bamboozled PW to PAY HIM to promote himself in the major industry magazine.

    And let’s not even get into his writing. I won’t say he’s the worst writer around, but all that critical acclaim comes from his networks of friends and colleagues–track the names back to see the connections.

    I stopped reading his columns and blog once I realized he has nothing original to say (mostly repackaging ideas by other people).

    He is, however, an excellent self-promoter. That’s a very different skill than being a great, or even good, writer.

  12. Janna Silverstein 28. Oct, 2009 at 9:48 am

    Excellent analysis, Mike. A skeptical approach here is key. I, too, look forward to seeing how all this turns out.

Trackbacks/Pingbacks

  1. Stackpole, Doctorow, and Intellectual Property — Geoffrey Allan Plauché - 26. Oct, 2009

    […] recently announced an experiment to prove that giving away free ebooks works. Michael Stackpole responded with a deconstruction of Cory’s experiment. He makes a number of good points about the experiment, though I think […]

  2. Jay Lake: [links] Link salad knows the way to San Jose - 28. Oct, 2009

    […] Michael Stackpole on Cory Doctorow — Deconstructing ebooks. […]

  3. lunch hour links for writers – 10/28/09 « helluo librorum - 28. Oct, 2009

    […] “free” sells (an oxymoron if I’ve ever heard one). Author Michael Stackpole spends some time Desconstructing Cory Doctorow’s “Experiment” and in all fairness also examines What is Cory Doing Right? Both posts are extremely interesting […]

  4. Randomness for 10/28 « Twenty Palaces - 28. Oct, 2009

    […] Michael A. Stackpole on Cory Doctorow’s book promo experiment. via Jay […]

  5. Stackpole, Doctorow, and Intellectual Property — Geoffrey Allan Plauché - 01. Sep, 2010

    […] recently announced an experiment to prove that giving away free ebooks works. Michael Stackpole responded with a deconstruction of Cory’s experiment. He makes a number of good points about the experiment, though I think […]