OkCupid’s Study Raise Questions About Testing Ethics | Koeppel Direct

OkCupid’s Study Raise Questions About Testing Ethics

Fresh on the heels of Facebook’s controversial disclosure of a social media experiment , OKCupid released results from three tests it conducted on users.

The popular online dating site selectively edited profiles, hid pictures and skewed compatibility scores to see how users responded. Despite the blowback that Facebook received, OKCupid is confident that users will find the results helpful rather than harmful.

“If you use the Internet, you’re subject to hundreds of experiments at any given time, on every site,” explained President Christian Rudder on the OKCupid company blog, “That’s how websites work.” 

Surprising findings. The three tests revealed some surprising findings about how people interact in the online dating world. For example, one of the tests adjusted user compatibility scores. If a user thought he was on the profile of someone who had a higher compatibility score, he was slightly more likely to reach out with a message. Users were also twice as likely to send at least four messages to people they thought they were compatible with.

Manipulating user outcomes. Based on OKCupid’s testing, it’s clear that it’s easy for websites to manipulate user outcomes without their knowing. The dating site adjusted the compatibility scores without notifying the small test group. The users received altered compatibility scores, some as staggering as 90% (up from 30%).

Once the testing was complete, users received emails revealing the true compatibility scores. OKCupid says it doesn’t know how many of its manipulated scores resulted in real-life dates – but there was a statistical uptick in communication between perceived good matches.

All to make the site work better? OKCupid’s user agreement specifies that personal data provided by users may be used in analysis and research. Although the users were technically given false information about their compatibility, Mr. Rudder explained that it was all done in the spirit of making the site work better.

“People come to us because they want the website to work, and we want the website to work,” he stated in his blog post. “We told users something that wasn’t true. I’m definitely not hiding from that fact.”

Users not happy. However, some users didn’t appreciate the testing.

“I understand that experimentation is part of the process,” said one 37-year-old user quoted in the New York Times, “But I do think that experiment is a lot more invasive. I would probably never see someone that the site said was a 30 percent match when we were actually 90 percent, so that is not cool, really.”

The two other tests included obscuring profile pictures to see how users communicated “sight unseen” and hiding profile text to see how it affected personality ratings. The photo experiment showed that users engaged in more meaningful conversations, shared more contact details and were quicker to respond to first messages. However, when pictures were reintroduced, some of those conversations stopped immediately.

Legal but ethically ambiguous. These randomized experiments conducted by Facebook and now OKCupid are “critical but always ethically hard” according to Mikolaj Jan Piskorski, a professor of social strategy at the International Institute for Management Development in Switzerland.

In a New York Times article, he compared the manipulation of profile information with medical experiments using a placebo. If some patients receive a placebo for a drug that might have improved their health, is the medical community responsible for these false hopes? With more social engineering experiments like this, social science is coming into the same dangerous waters. Piskorski recommends that instead of manipulating the data, these sites should use “natural experiments” that use observational data that occur naturally.

Only time will tell whether or not Facebook, OKCupid and other sites will take an observational approach. With so much consumer data at their fingertips, and a goal of making their sites as user-friendly as possible, these more invasive experiments may become the norm.


Your browser is out of date. It has security vulnerabilities and may not display all features on this site and other sites.

Please update your browser using one of modern browsers (Google Chrome, Opera, Firefox, IE 10).