Most recent update: 9:05 PM, Monday June 30

If you were feeling glum in January 2012, it might not have been you. Facebook ran an experiment on 689,003 users to see if it could manipulate their emotions. One experimental group had stories with positive words like “love” and “nice” filtered out of their News Feeds; another experimental group had stories with negative words like “hurt” and “nasty” filtered out. And indeed, people who saw fewer positive posts created fewer of their own. Facebook made them sad for a psych experiment.

I first saw the story on Facebook, where a friend picked it up from the A.V. Club, which got it from Animal, which got it from the New Scientist, which reported directly on the paper. It’s exploding across the Internet today (e.g. MetaFilter), and seems to be generating two kinds of reactions: outrage and shrugs. I tend more towards anger; let me explain why.

Facebook users didn’t give informed consent: The study says:

[The study] was consistent with Facebook’s Data Use Policy, to which all users agree prior to creating an account on Facebook, constituting informed consent for this research.

The standard of consent for terms of service is low. But that “consent” is a legal fiction, designed to facilitate online interactions. (See Nancy Kim and Margaret Jane Radin’s books for more.) It’s very different from informed consent, the ethical and legal standard for human subjects research (HSR). The Federal Policy for the Protection of Human Subjects, a/k/a the Common Rule, requires that informed consent include:

(1) A statement that the study involves research, an explanation of the purposes of the research and the expected duration of the subject’s participation, a description of the procedures to be followed, and identification of any procedures which are experimental;

(2) A description of any reasonably foreseeable risks or discomforts to the subject; …

(7) An explanation of whom to contact for answers to pertinent questions about the research and research subjects’ rights, and whom to contact in the event of a research-related injury to the subject;

(8) A statement that participation is voluntary, refusal to participate will involve no penalty or loss of benefits to which the subject is otherwise entitled, and the subject may discontinue participation at any time without penalty or loss of benefits to which the subject is otherwise entitled.

Facebook’s actual Data Use Policy contains none of these, only general statements that “we may use the information we receive about you … for internal operations, including troubleshooting, data analysis, testing, research and service improvement.” and “We give your information to the people and companies that help us provide, understand and improve the services we offer. For example, we may use outside vendors to … conduct and publish research.” Neither of these comes close to a “description of the procedures to be followed” or a “description of any reasonably foreseeable risks or discomforts,” and the Data Use Policy doesn’t even attempt to offer a contact for questions or an opt-out.

Federal law requires informed consent: To be sure, the Common Rule generally only applies to federally funded research, and Facebook is a private company. But that’s not the end of the story. The paper has three co-authors: Facebook’s Adam Kramer, but also Jamie Guillory from UCSF and Jeffrey Hancock from Cornell. UCSF and Cornell are major research universities and receive large sums of federal funding. Both of them have institutional review boards (IRBs), as required by the Common Rule: an IRB examines proposed research protocols to make sure they protect participants, obtain informed consent, and otherwise comply with ethical and legal guidelines.

I don’t know whether the study authors presented it to an IRB (the paper doesn’t say), but it strikes me as the sort of research that requires IRB approval. It further strikes me that the protocol as described is problematic, for the reasons described above. I don’t know whether I’m more afraid that the authors never obtained IRB approval or that an IRB signed off on a project that was designed to (and did!) make unsuspecting victims sadder.

The study harmed participants: The paper also argues:

[The study software] was adapted to run on the Hadoop Map/Reduce system (11) and in the News Feed filtering system, such that no text was seen by the researchers.

This claim misses the point. For an observational study, automated data processing is a meaningful way of avoiding privacy harms to research subjects. (Can robot readers cause a privacy harm? Bruce Boyden would say no; Samir Chopa would say yes.) But that is because in an observational study, the principal risks to participants come from being observed by the wrong eyes.

This, however, was not an observational study. It was an experimental study—indeed, a randomized controlled trial—in which participants were treated differently. We wouldn’t tell patients in a drug trial that the study was harmless because only a computer would ever know whether they received the placebo. The unwitting participants in the Facebook study were told (seemingly by their friends) for a week either that the world was a dark and cheerless place or that it was a saccharine paradise. That’s psychological manipulation, even when it’s carried out automatically.

This is bad, even for Facebook: Of course, it’s well know that Facebook, like other services, extensively manipulates what it shows users. (For recent discussions, see Zeynep Tufekci, Jonathan Zittrain, and Christian Sandvig). Advertisers and politicians have been in the emotional manipulation game for a long time. Why, then, should this study—carried out for nobler, scientific purposes—trigger a harsher response?

One reason is simply that some walks of life are regulated, and Facebook shouldn’t receive a free pass when it trespasses into them simply because it does the same things elsewhere. Facebook Beacon, which told your Facebook friends what you were doing on other sites, was bad everywhere but truly ugly when it collided with the Video Privacy Protection Act. So here. Whatever you think of Facebook’s ordinary marketing-driven A/B testing is one thing: what you think of it when it hops the fence into Common Rule-regulated HSR is quite another. Facebook has chosen to go walking in a legal and ethical minefield; we should feel little sympathy when it occasionally blows up. (That said, insisting on this line would simply drive future research out of the academy and into industry, where our oversight over it will be even weaker. Thus …)

A stronger reason is that even when Facebook manipulates our News Feeds to sell us things, it is supposed—legally and ethically—to meet certain minimal standards. Anything on Facebook that is actually an ad is labelled as such (even if not always clearly.) This study failed even that test, and for a particularly unappealing research goal: We wanted to see if we could make you feel bad without you noticing. We succeeded. The combination of personalization and non-rational manipulation may demand heightened legal responses. (See, e.g., Ryan Calo, or my thoughts on search engines as advisors.)

The real scandal, then, is what’s considered “ethical.” The argument that Facebook already advertises, personalizes, and manipulates is at heart a claim that our moral expectations for Facebook are already so debased that they can sink no lower. I beg to differ. This study is a scandal because it brought Facebook’s troubling practices into a realm—academia—where we still have standards of treating people with dignity and serving the common good. The sunlight of academic practices throws into sharper relief Facebook’s utter unconcern for its users and for society. The study itself is not the problem; the problem is our astonishingly low standards for Facebook and other digital manipulators.

This is a big deal: In 2006, AOL released a collection of twenty million search queries to researchers. Like the Facebook study authors, AOL thought it was protecting its users: it anonymized the users’ names. But that wasn’t sufficient: queries like “‘homes sold in shadow lake subdivision gwinnett county georgia” led a reporter straight to user No. 4417749. Like Facebook, AOL had simply not thought through the legal and ethical issues involved in putting its business data to research purposes.

The AOL search-query release became known as the “Data Valdez” because it was a vivid and instantly recognizable symbol of the dangers of poor data security. It shocked the public (and the industry) into attention, and put search privacy on the map. I predict, or at least I hope, that the Facebook emotional manipulation study will do the same for invisible personalization. It shows, in one easy-to-grasp lesson, both the power Facebook and its fellow filters hold to shape our online lives, and the casual disdain for us with which they go about it.

UPDATE: The study was presented to an IRB, which approved it “on the grounds that Facebook filters user news feeds all the time, per the agreement.” See @ZLeeily, with hat tips to Kashmir Hill and @jon_penney.

UPDATE: Another @jon_penney pickup: it appears that the study itself was federally funded. Cornell amended the press release to say that the claim of federal funding was in error.

UPDATE: Kashmir Hill reports:

Professor Susan Fiske, the editor at the Proceedings of the National Academy of Sciences for the study’s publication, says the data analysis was approved by a Cornell Institutional Review Board but not the data collection. “Their revision letter said they had Cornell IRB approval as a ‘pre-existing dataset’ presumably from Facebook, who seems to have reviewed it as well in some unspecified way,” writes Fiske by email.

UPDATE: For much more on the IRB legal issues, see this detailed post by Michelle Meyer. She observes that the Common Rule allows for the “waiver or alteration” of informed consent for research that poses “minimal risk” to participants. The crucial issue there is whether the study “could not practicably be carried out without the waiver or alteration.” Meyer also has an extended discussion of whether the Common Rule apples to this research—following the Cornell restatement, it is much less clear that it does.

UPDATE: I’ve created a page of primary sources related to the study and will update it as more information comes in.