Taming Post Claps
The Two Billion Claps Bug
TL;DR
A user was able to exploit a race condition in our backend system to manipulate clap counts on posts. Users are supposed to only be able to clap between 0 and 50 times for a given post, but this hack allowed them to go outside those bounds (both above and below). Our fix leverages DynamoDB condition expressions and strongly consistent reads to block updates on data that have been manipulated after a read, but before a write. Additionally, we implemented an eventually consistent clap rectification solution for those posts that were already affected by this bug.
Catching The Bug
We became aware of this problem thanks to a user who brought an extremely detailed writeup of a roundabout way to manipulate claps. We appreciate the help and are grateful that it was flagged to us!
Clarifying A Few Things
Some of the claims in the user’s report should be addressed to avoid confusion:
Essentially, Medium’s Partner Program payments directly depend on claps. The more claps you receive, the more money you will make.
This is actually untrue. The Partner Program (V4) rewards posts by the number of people that clapped for a post, not the number of claps itself. Ignoring all other factors, a post with 10 unique clappers with 10 claps will earn more than a post with 5 unique clappers with 50 claps. You can read more about the partner program here.
Furthermore, our recommendation algorithms also depend on the number of clappers, not number of claps.
Further, imagine not only zeroing the current counts but future counts, indefinitely. Any future likes of that user won’t show up.
It is true that this hack can make it seem that a post has -200 claps on our backend system. If 3 different users then come and each clap 50 times, the clap count will still be -50 (all negative clap count values show up as 0 in the UI). But, the 3 users that clapped for the post are not lost, so this post will still earn the same amount of money if the post displayed all 150 claps. This is what is meant by “the bug only affects the UI”.
How Severe Was This Bug?
The user claimed this bug was significant. We don’t quite agree.
There was some merit to the user’s point that “users might feel less inclined to vote/click, etc. on zero-clapped articles.” It would also be frustrating for the writer to see a discrepancy if the post displayed 0 claps but the stats page showed >1 clappers.
However, since this bug doesn’t effect post earnings or recommendations, we don’t feel it was as significant as the user claimed.
We also have to point out that a normal user cannot reproduce this in the UI; it was only accessible to the user via software scripting.
What Exactly Was Going On?
In our backend system, a few things occur when the clap endpoint is hit:
- We check to see if the user has clapped for the post before. If so, we add the existing clap count to the incoming claps (capped at 50). Otherwise, it’s a clean slate.
- We update the record in the database to be existing_clap_count + incoming_clap_count. This is where the problem lies. Let’s dive in.
Unconditional Writes
These types of writes will simply read the existing value in a database, do whatever updates/writes you command, and save the result. But, what happens if two people are trying to update an item at the same time?
See below, where Alice and Bob are both updating the price of an item. They both read the same price initially, but depending on several factors (of which can be out of our control such as network latency), Bob’s write wins only because by chance it occurred after Alice’s.
Now let’s shift gears to clap counts. Let’s say a user has already clapped for a post 30 times, and we get a burst of 10 requests, each with the user adding 20 more claps to the post. Since these arrive at almost the same time, some (could be all, but let’s say 5) requests will read the existing record with 30 claps. For these 5 requests, adding 20 more is completely valid. So, these 5 will each add 20 requests resulting in a user clapping 130 times for the post!
The other 5 requests might come after these writes and see 130 claps. But this is an invalid number (>50) so no operation will occur.
So, How Can We Fix This?
Is there a way to update the existing record if and only if the record contains a value that we expect? In other words, how can we ensure we don’t go over 50 claps if multiple requests are sent concurrently?
The answer is yes! Dynamo’s condition expressions for conditional writes achieves exactly what we need to handle concurrency. Going back to Alice and Bob, a conditional write will only occur if price = 10 , ridding us of the race condition seen above:
You can imagine this works fantastically for our clap count incrementing as well. From our example above, we only want to add 20 claps if the existing clap count is the initially read 30 claps. For each request that comes in, we:
- Read the existing clap count from the database. We use strongly consistent reads to ensure the record is not stale.
- Using a condition expression, only update the record if the clap count is equal to what was read in step 1. This ensures that no other process has manipulated the data in between this request’s read and write operations.
So, our 5 concurrent requests all read 30 claps from the database. The request that operates the fastest will succeed and set the value to 50. Subsequent requests that attempt to write will fail since the value has been updated to 50, not the expected 30.
Cleaning Up Borked Posts
“While none of us relish dwelling on past mistakes, sometimes revisiting them is the only way to ensure a smoother future.” — ChatGPT
Obviously, this came to our attention because there were a few people that already messed with posts. The conditional writes will prevent these from happening in the future, but what about the past?
It was necessary to run a backfill script to rectify borked posts. This script simply identified records in our database that contained clap counts outside of the [0, 50] bound and cleaned them up. However, because of the way our pipeline system operates, changes will only be reflected when a new event occurs on a post, such as a read, view, or clap. We estimate that this solution will affect the claps on about 14k users’ stories, either increasing or decreasing (😬) them.
Results
The results were 🔥 🔥.
Catching Conditional Errors
We are noticing an uptick in our logs of the error:
(ConditionalError): The conditional request failedwhich means our condition expressions are working!
Now, this doesn’t mean we are now discovering a bunch of “hackers” that have been flying under our nose…We run on a distributed platform rife with network failures and retries. So it’s highly likely that most of these are innocent errors caused by said network failures. Nonetheless, they are still good to catch because even innocent errors can lead to inaccurate data!
Historical Clap Cleanup
Here is an extraordinary case that our clap cleanup fixed.
The ol’ 6 views, 8 reads, and 2B+ claps trick…
This is a fun one that has floated around internally…and here is a snapshot of the metrics as of this writing:
Unfortunately (fortunately…?) we’ve uncovered the truth:
Looking Forward
As always, products evolve. This bug has caught our eye and motivated us to reconsider “What should claps be?”
Do not fear if this solution updates your post clap counts! Your partner program earnings will not be affected! But hopefully, you will appreciate the dedication to data quality as much as we do.
Happy writing!
