“There’s no way to tell where the border is between measurement and manipulation”

Jaron Lanier discussing whether machine learning systems like Netflix recommendations, Facebook’s newsfeed, Google search etc are manipulative:

There’s no way to tell where the border is between measurement and manipulation in these systems. For instance, if the theory is that you’re getting big data by observing a lot of people who make choices, and then you’re doing correlations to make suggestions to yet more people, if the preponderance of those people have grown up in the system and are responding to whatever choices it gave them, there’s not enough new data coming into it for even the most ideal or intelligent recommendation engine to do anything meaningful.

In other words, the only way for such a system to be legitimate would be for it to have an observatory that could observe in peace, not being sullied by its own recommendations. Otherwise, it simply turns into a system that measures which manipulations work, as opposed to which ones don’t work, which is very different from a virginal and empirically careful system that’s trying to tell what recommendations would work had it not intervened. That’s a pretty clear thing. What’s not clear is where the boundary is.

If you ask: is a recommendation engine like Amazon more manipulative, or more of a legitimate measurement device? There’s no way to know. At this point there’s no way to know, because it’s too universal. The same thing can be said for any other big data system that recommends courses of action to people, whether it’s the Google ad business, or social networks like Facebook deciding what you see, or any of the myriad of dating apps. All of these things, there’s no baseline, so we don’t know to what degree they’re measurement versus manipulation.