“The Sky Is falling!” shouted Chicken Little. Perhaps you've heard the line before, but if not, a short recap of the children’s story: Chicken Little gets the whole forest into a tizzy, thinking that the sky is falling, though unbeknownst to Chicken Little, it was only an acorn that hit him. Everyone follows him, until they end up in the fox’s hole… only to never make it out again.
The moral of the story here is: don't jump to immediate conclusions and assume calamity has struck.
On October 3rd of this year, the community of Amazon sellers were thrown for a loop with a change in the Terms of Service that banned incentivized reviews.
Was this change just an inconsequential acorn that fell, or was the sky actually falling for Amazon sellers? Anecdotal evidence zoomed through the seller community with varying degrees of reliability.
At Review Kick, thousands of sellers who were presumably effected by this change in Amazon’s Terms. We recently gathered the data and ran rigorous analysis, to determine exactly how the change in Amazon policies effected sellers.
TL; DR: The change in Terms of Service ultimately leveled the playing field, to remove sellers who were “gaming the system”, by quickly generating lots of reviews. Sellers who had a sudden surge in reviews lost the most reviews.
The foundation of our study is based on a subset of the products that were promoted on Review Kick. In total, there were 7967 that we tracked.
We had data for 5995 of the products. This means that we had data for these 5995 products before and after the October 3rd TOS change, including the Category, Number of Reviews, and Rating.
The category that was most represented in our data set was Home & Kitchen.
The breakdown of the products in our dataset, by category, can be seen here:
Note that this graph only includes categories that had more than 10 products, so you won’t see some categories like Video Games. Furthermore, note that this is on a logarithmic scale, so the bars looked more compressed as they move to the right.
Of these 5995 products, 1548 lost reviews. Thus, 25.8% of our random sample lost reviews. This is a statistically significant result (ie not a random occurrence of luck).
We were curious to see which categories, if any, were most impacted by the Terms of Service.
Looking to see how many products lost reviews, by category, we get the following graph:
This graph shows that Kitchen & Dining and Electronics were outliers in how many products lost reviews.
This is very interesting data to see!
But I was curious, if they lost reviews, how many reviews did they actually lose, as a percentage and as a number?
For example, even if almost 65% of products in Kitchen & Dining lost reviews, it wouldn’t be so bad for sellers if it were only 1 or 2 reviews.
We plotted out this data, to see how the number of reviews correlated to the percentage of reviews. Did sellers that lost a bunch of reviews after the TOS lose a large percentage of their reviews? Or were they just well-established products with a ton of reviews, and losing 100 reviews was just a drop in the bucket for them?
Here is a scatter plot that shows the reviews lost, in percentage and number:
Let me explain what we’re looking at here.
In the graph above, each product that lost reviews is shown as a dot. On the x-axis are the number of reviews the product lost, and on the y-axis is the percentage of reviews that were lost.
We can see along the top of the graph that 9 products lost 100% of their reviews and on the right side of the graph we see that 5 products lost more than 2000 reviews.
The blue trend line highlights that products which lost more reviews tended to lose a higher percentage or their reviews. Additionally, if a product lost more than 200 reviews, that tended to represent about 2/3rds of their reviews on average.
Looking at this in another way, let’s look at the percentage of reviews that products lost overall. Here's the results in this chart:
In the graph above, we see the percentage of reviews that products lost. For example, 210 products lost a maximum of 10% or their reviews, and 25 products lost between 90% and 100%.
What this chart doesn’t show is that among products that lost reviews, the median was 28 reviews or 41% of their reviews.
Note that we used the median because some lost 2000, which is an extreme amount that would skew the data if we used an average. In many ways, the median (as described by the middle figure) negates any outliers, so is most representative.
“Among products that lost reviews, the median was 28 reviews or 41% of their reviews.”
I also wanted to see if we could identify any pattern in how Amazon enforced its Terms of Service change. For example, were there any types of products, categories, larger/smaller sellers or price points that lost an outsize number of reviews?
To answer this question, we compared the number of reviews before the TOS change, against percentage of reviews lost. The scatter plot looks something like this:
As the blue trend line shows in graph above, products that had a lot of reviews before the TOS change tended to lose a slightly higher percentage their reviews. However, the difference was not very large. No particular categories have been targeted for removal.
“We saw no categories targeted for removal of reviews after the TOS change.”
Among products that lost reviews and saw a reduced rating, their median loss in star rating was just 0.13, or about 3%.
This study confirmed many of the suspicions and hypotheses that I had going into this. As it turns out, a majority of sellers were not impacted—our data shows 26% of products lost reviews.
Moreover, it seems that there was a pattern that the products with more reviews had a greater likelihood of having reviews removed. Additionally, these products had a higher percentage of these reviews removed.
I interpret this to mean that Amazon targeted products had aggressive spikes in product reviews. This was fully legal and within Amazon’s Terms of Service, but is no longer a viable strategy after the TOS.
Ultimately, Amazon set up a seller environment that removes any unfair advantage of using reviews services. This will prevent stop any sellers that were “gaming the system” to boost their reviews, organic ranking, and of course, their sales.
It's not all doom and gloom though. In fact, prohibiting incentivized reviews is a great move by Amazon. Here’s why:
It became clear that some sellers were abusing the reviews system, and getting lots of reviews in a very short time. Moreover, the products that had many reviews naturally rose to the top of the organic rankings for main keywords, and made it intimidating and difficult for new products to enter the niche. Ultimately, this did a disservice to Amazon. New sellers did not enter the market, and Amazon customers were denied the opportunity of choice.
Related to the point above, there is opportunity again for new sellers to enter product niches where otherwise it would not have been a prudent decision before the TOS change. I think this will benefit Amazon customers, by having access to a greater variety (and presumably better quality products).
Finding success and making money on Amazon has shifted towards the long-game. This means sellers can no longer launch a product, collect a bunch of reviews, and shoot to the top of the organic rankings. I was certainly a beneficiary of this with Jungle Stix. However, times have changed and success will now rely on the following digital marketing fundamentals:
Despite being legitimate, honest reviews that were generated in exchange for discounts skewed the overall rating of the product to be higher than it would without incentivized reviews. We see this with the drop in rating of 0.13 stars for products that lost reviews after the TOS change.
I believe Amazon's main priority was to restore customer trust in its product reviews system. As we know, social proof and reviews are critical in creating a marketplace of trust. Therefore, the change will restore customer trust and increase the likelihood of people purchasing products on Amazon vs. in brick and mortar stores. This is especially important for private label sellers, who do not have a brand reputation to help convince a customer to purchase their product.
Customer trust and social proof is vital for private label sellers, who do not have the brand reputation to help convince a customer to purchase, compared to brick and mortar stores.
Here is my conclusion: it's time to get back to work!
Seriously though, I think Amazon just opened up the opportunity for new sellers to enter the market and differentiate their offering, whether it is by selling a different set of features, a higher quality product, better listing, etc.
If you are a real data geek, we have a summary of our data queries for you to DOWNLOAD HERE. I worked with our Data Scientist, Ellis, to pull these insights. By the way, we carried out this work in RStudio, a powerful open source and free program to do deep statistical and graphical analysis.
_______________
As with any data study, there are some limitations that are important to note. A few of which are: