On Thursday, April 5, 2018, we were alerted by several users that they were seeing unusual repricing behavior on Amazon where offers that had the Buy Box were continually lowering their price until the Min price was reached.
Upon initial investigation it appeared that this was being caused by a failure to respect the strategy setting "Do not lower my price when I have the Buy Box" and a patch was deployed to prevent any price drops for offers in the Buy Box.
The patch did not resolve the issue and after further investigation it became clear that that the repricer was not properly recognizing that the user had already won the Buy Box when specific strategies were in use.
The root cause was identified as a minor performance enhancement made several days prior that was being slowly rolled out across the user base. The code change caused the Seller Id for our user to remain blank when identifying the owner of the Buy Box subsequently preventing the repricer from being able to determine when the Buy Box had been won. In turn, this caused the Get the Buy Box algorithm to continue dropping the price until the Min price was hit.
This issue affected approximately 200 users in total. Due to the slow rollout, most users were not affected for entire duration of the issue.
All timestamps are EDT.
Tuesday, April 3, 2018 A minor performance enhancement was deployed to fix intermittent database timeouts in the repricer.
Thursday, April 5, 2018 First report of incorrect prices is sent to our support team by a customer.
1:15 PM EDT Upon seeing strange repricing patterns in the users account and getting more user reports, the issue was escalated to our engineers.
3:37 PM EDT A patch is deployed to fix what was believed to be a strategy setting issue. This does not resolve the issue and the investigation resumes.
7:55 PM EDT A second patch is deployed to address the strategy setting. This patch appeared to work but did not actually resolve the underlying issue.
Monday, April 9, 2018 More user complaints come in that the issue had been occurring in their account over the weekend and the investigation resumes.
6:05 PM EDT The root cause is identified and a patch is deployed that fixes the issue. Impact analysis begins to find all affected listings.
Tuesday, April 10, 2018 8:02 PM EDT All prices finish adjusting to their correct state and the issue is closed.
Automated testing has already added for this exact situation, these tests prevent code from being released if they fail.
In addition a new sanity check is being added to the repricer to check if the offer data that was sent by the marketplace matches the listing data we have stored. This new check is being added to the same system that checks Min and Max prices, sanity checks are run after a new price is calculated as the very last operation before sending the price to the marketplace. This new check will cover a broad range of possible failures that could result in an incorrect price being computed. Price changes that fail a sanity check are discarded and the error is reported to the engineering team.
We are also reviewing our procedures for handling bugs that could possibly affect pricing. This includes giving support personnel the ability to call for an emergency troubleshooting session or have price changes paused during an investigation.