On Thursday, September 7, 2017, we were alerted by several users that they were seeing item prices in their selling account that were unusually high or low. Upon investigation we identified a strange pattern of shipping prices being presented by the Amazon MWS Subscriptions API. The initial hypotheses was that we were receiving bad data from Amazon and the repricer was shut down for affected users to avoid sending incorrect prices to the marketplace.
After further investigation it became clear that the strange shipping values were actually live on the Amazon website and the working hypotheses was changed to either Amazon or a third party tool having revised those shipping prices improperly. Once we became confident that any price changes sent by us would be properly accounting for the shipping that would eventually be charged to the consumer, the repricer was brought back online and normal operations resumed. The incorrect prices that users brought to our attention had been sent by our system to compensate for the altered shipping and keep the total sale price at the correct price point.
We are continuing to investigate the cause of this issue but it does not affect the proper functioning of our system.
All timestamps are EDT.
Thursday, September 7, 2017, 11:23 AM EDT First report of incorrect prices is sent to our support team by a customer.
2:01 PM EDT Upon seeing strange repricing patterns in the users account and getting more user reports, the issue was escalated to our engineers.
3:30 PM EDT After normal debugging and analysis procedures do not give a satisfactory answer, the issue was escalated once more and the entire engineering department gathered to analyze and brainstorm.
5:20 PM EDT Upon discovering widespread shifts in shipping prices coming from the Amazon API affecting far more users than initially suspected, the decision is made to halt repricing and post a public incident report. At this point, the database backups from the previous day were restored to identify every single shipping value that had shifted.
7:30 PM EDT After analyzing all of the shifted shipping prices and determining that they matched the values on Amazon and that the data in our system was not incorrect, the incident was closed and the repricer was brought back online.
We will be working to put in place systems to detect abnormal patterns of changes caused by external systems so we can better protect our users accounts with alerts. Due to the nature of this issue, preventative measures are not possible.