Etsy's new Star Seller Program Reviewed


Etsy's new Star Seller Program Reviewed

July 30, 2021


Welcome to another day in Etsy land! This week Etsy announced a new seller evaluation tool that will launch from Sept 1st. And oh boy, do people have a lot to say!


Etsy Star Seller Program

So what's it all about? As per Etsy's announcement, there is a new, more public layer of seller evaluation being added that will include a "Star Seller" badge on qualifying Etsy shops in order to show that a shop is a trusted seller. 


Etsy already has the behind the scenes ODR score for evaluating shops, something they use to remove sellers who are regularly under-performing. And of customers evaluate sellers via the order review system.   


The ODR score program appears to have been cancelled, replaced by the new Star Seller program, although this hasn't been officially confirmed.

Here’s the breakdown of the new program:


Response rate: You respond to 95% or more of initial messages from buyers within 24 hours (just the first message, not ongoing conversations). 


Shipping: 95% or more of your orders ship within your stated processing time and with tracking* or you purchased a shipping label on Etsy. Digital orders will not require tracking information. 


5-star ratings: 95% or more of your reviews have 5-star ratings.


Orders, sales, and shop tenure: You’ll also need to have at least 10 orders and $300 in sales (before shipping and taxes) within the three month review period, and have been on Etsy’s platform for 90 days since your first sale.


On the surface you might think that this all sounds rather reasonable. And really it's not an entirely bad idea. Until you learn just how few really great sellers will actually qualify for this because of  just how bad the metrics are. 


To begin with, Etsy launched the Star Seller dashboard with all sorts of bugs, so it isn't even working very well from the get-go. I guess they've given themselves a month to sort that out, but Etsy is notorious for inaccurate stats so there's very little confidence that this will ever be an accurate measurement.


In my own two shops I don't immediately qualify, despite providing pretty damn good service (if I do say so myself). While instant download digital sellers don't have to worry about the shipping metric, I failed on the response rate in both my shops - not because I don't respond, or respond slowly - but simply because I get messages that don't require a response. This acts to skew the results, creating a false impression that I don't respond to all messages fast enough.


Just for reference, so that I don't sound completely like a bunch of sour grapes, I'm in the top 1% of Etsy sellers based on product selling volume, with over 10 years selling on the site and a consistent 5 star average rating. But I'm not a Star Seller... according to Etsy - but only because I haven't clicked 'respond' on every single message in my shop. yay for me.


For sellers who have to also take into account the shipping metric this is an even bigger problem, because not all products warrant tracking and not all customers want to pay for tracking. A perfectly awesome seller can see this metric completely tank their chances of ever achieving a Star Seller rating.

Because the metrics being used aren't very robust or accurate, the Star Seller rating ultimately becomes pointless and entirely meaningless. If you have enough selling volume going through your shop you can easily absorb the odd anomaly or error and still achieve the required 95%. If you're a small seller you ultimately have to achieve 100% because just one wrong action will drop you well below 95%.

And that's the rub that hurts the most. Etsy is introducing a rating system that rewards big sellers and punishes small sellers with metrics that are largely inaccurate.

So why is Etsy even doing this? Well, there are a couple theories floating around. It could be that Etsy just wants sellers to be providing the best possible service. But it's more likely they are just trying to get more buy-in for their own shipping label system. This puts more money in their pockets, so it makes sense they would find some way to encourage that behavior - and the psychology of handing out gold stars is a pretty little package to wrap that up in. But we don't know this for sure, it's just a guess.

To really understand exactly how this new program works and why it is just complete utter garbage though you need to get right down into the nitty gritty details of how analytics should actually work.


Thankfully Etsy seller David Bowman did just that for us and his Etsy forum post is very enlightening.  It's long, but it's well worth a read if you want to better understand exactly where/how Etsy has gone so very wrong with this new seller evaluation tool.

Here are a few excerpts if you want the short version:

"Let’s say you want to rate customer service. What would you analyze? Message response, shipping times, and buyer feedback are three good categories of information to look at.

This is where the Etsy star rating system runs into problems. It is not a valid process and does not determine whether or not the seller has good customer service."


On message response rate... "Etsy’s description of the measure specifically indicates they assess how sellers respond to buyers’ messages. If Etsy cannot determine which messages are from buyers, and further determine which messages merit a response, the findings on responses are not valid.


The analysis does not consider whether or not the seller responded to a buyer’s message, only that the seller did not respond to a message, whether or not it was from a buyer and whether or not it needed a response. Basically, the process used for analysis doesn’t measure what is intended, which is the definition of an invalid measure."


And on shipping... "not all delivery methods will generate a tracking number and not all products are worth tracking. So, while this is a good indicator that any particular product will be delivered, you cannot use the absence of a tracking number to indicate that the product will not be shipped or shipped on time. You can infer that a tracking number indicates likely delivery, but you cannot infer anything from a lack of a tracking number."


For the 5 star rating system he had this to say... "Buyers use the rating stars inconsistently because they don’t use the same definition for what a certain number of stars means. Basically, a four or five star review doesn’t mean the same thing to each buyer. One buyer might think that 5 stars is only for exceptional, better than expected product and service, but that 4 stars is for simply acceptable service and product. Another buyer might think 3 stars is for barely acceptable, 4 stars is amazing, and 5 stars is only for faster than expected delivery, super low cost, and freebies included in the package. 


In some cases, one person’s 4-star rating may express greater satisfaction than another person’s 5-star rating. Without a consistent definition for a number of stars, the process is not reliable."


Finally, there's the orders, sales and shop tenure... "This measure is also based on the faulty assumption that sellers who make more money are somehow better sellers--without taking into account the value of the products sold. A seller with 30 $10 sales isn’t necessarily more worthy of a badge than a seller with 149 $1.99 sales.


Overall, neither quantity nor aggregate value can work as measures for customer service. On one hand, they may prevent great sellers of low-cost items, and great sellers of high value items from ever obtaining the badge."



His entire analysis is just so spot on and comes from a position of a more scientific (and therefore more accurate) form of information analytics. It's pretty much exactly what myself and many other sellers are thinking/feeling, only worded in a much more intelligent and educated way.

If you're looking for another take on this new program Jess Van Den offers her own break-down and critique on her YouTube channel. And The Charmed Maker as this to say on hers.

I'd love to offer some contrasts, from happy people who think this is all just awesome and believe that Etsy have got this spot on. There are a few Etsy sellers who do think it's great (mainly because they qualify) but largely that's just not where anyone is at yet with this. The consensus across he board is one of dismay and outrage.

Even Kara Buntin, one of my favorite Etsy coaches, who is normally very pragmatic and even-keeled on all things Etsy has said that this roll-out is seriously flawed (and stupid). You can watch her initial reaction to the Star Seller program on her YouTube channel.

Stay tuned for updates from Etsy on this as I'm sure there will be more to follow!


Updated Aug 6:


Etsy posted their answers to seller questions today and so far what we are seeing is a firm stance on all points without any immediate amendments to the serious flaws offered by its seller community. They did concede that a few bugs were being addressed but that no changes would take place at this time otherwise.


They did however suggest that a few of the issues raised by sellers (the "pain points") would be looked at more fully by the team and in future improvements might be made to address those. This include the possibility of a 'no response needed' button in messaging and a solution for custom digital sellers using the physical listing option but delivering by email without tracking or shipping labels.

Largely things went exactly in the same style as any other launch Etsy has done. With a shockingly bad understanding of the selling process, the many flaws and issues sellers have with the various tools on the site, and without too much regard for how things they launch actually impact the seller.

The most concerning issue I had was the dismissive attitude toward the 4 star review. Etsy is insisting on treating anything less than 5 stars as equal, citing that they want to highlight only the very best customer experiences, from sellers that go "above and beyond".  But this isn't a true metric of consistently great customer service when buyers apply different ideas into what that actually constitutes... as discussed above.

I guess we'll see what happens when the program rolls out to buyers and what "improvements" Etsy makes along the way.


Post a Comment