Incident involving incorrect shipping prices
Incident Report for Informed.co
Postmortem

Issue Summary

On Thursday, September 7, 2017, we were alerted by several users that they were seeing item prices in their selling account that were unusually high or low. Upon investigation we identified a strange pattern of shipping prices being presented by the Amazon MWS Subscriptions API. The initial hypotheses was that we were receiving bad data from Amazon and the repricer was shut down for affected users to avoid sending incorrect prices to the marketplace.

After further investigation it became clear that the strange shipping values were actually live on the Amazon website and the working hypotheses was changed to either Amazon or a third party tool having revised those shipping prices improperly. Once we became confident that any price changes sent by us would be properly accounting for the shipping that would eventually be charged to the consumer, the repricer was brought back online and normal operations resumed. The incorrect prices that users brought to our attention had been sent by our system to compensate for the altered shipping and keep the total sale price at the correct price point.

We are continuing to investigate the cause of this issue but it does not affect the proper functioning of our system.

Timeline

All timestamps are EDT.

  • Thursday, September 7, 2017, 11:23 AM EDT First report of incorrect prices is sent to our support team by a customer.

  • 2:01 PM EDT Upon seeing strange repricing patterns in the users account and getting more user reports, the issue was escalated to our engineers.

  • 3:30 PM EDT After normal debugging and analysis procedures do not give a satisfactory answer, the issue was escalated once more and the entire engineering department gathered to analyze and brainstorm.

  • 5:20 PM EDT Upon discovering widespread shifts in shipping prices coming from the Amazon API affecting far more users than initially suspected, the decision is made to halt repricing and post a public incident report. At this point, the database backups from the previous day were restored to identify every single shipping value that had shifted.

  • 7:30 PM EDT After analyzing all of the shifted shipping prices and determining that they matched the values on Amazon and that the data in our system was not incorrect, the incident was closed and the repricer was brought back online.

Corrective and Preventative Measures

We will be working to put in place systems to detect abnormal patterns of changes caused by external systems so we can better protect our users accounts with alerts. Due to the nature of this issue, preventative measures are not possible.

Posted Sep 08, 2017 - 10:01 EDT

Resolved
We have confirmed that all of the erroneous shipping values are visible on the Amazon website and the same values are being used everywhere including orders.

We are still trying to determine if the incorrect shipping prices originated from Amazon or from a third party however Appeagle does not change shipping values and we have verified that we did not change these values.

All repricing systems are now running, if you see incorrect shipping values in your account you should contact Amazon to find out where they originated.

Exercise caution when correcting shipping values as this may drop your price below your min price until the repricer is able to compensate.
Posted Sep 07, 2017 - 21:36 EDT
Investigating
Amazon is currently experiencing an issue where incorrect values are being added to shipping reports. As a result, we are calculating inaccurate prices for some non-FBA items. We are currently not sending price changes for effected items, but we highly encourage you to review your item's prices. As soon as we see activity return back to normal we will update here.
Posted Sep 07, 2017 - 17:20 EDT