On Wednesday 6/22 at 12:45p.m. EST, we began rolling out a routine update to Pardot’s AppExchange package. Beginning around 1p.m. EST, approximately 1% of Salesforce instances that received this update began receiving “Invalid Data” errors upon record save.
Our Support Team began receiving reports of these errors from users at approximately 2:20p.m. EST, at which time we immediately activated our incident response plan. We recognized the cause of this incident to be an existing Salesforce bug. The bug is unpredictable; it is randomly encountered when a package upgrade contains new Apex trigger code that references new custom objects or settings added in the same upgrade. Wednesday’s package update included a new custom setting as well as additions to our trigger code that referenced this setting. In some Salesforce instances, this caused compilation errors which appeared whenever that trigger was invoked, preventing records from being saved.
At 2:35p.m. EST a solution was identified by our Engineering team, and the update began deploying around 3:00p.m. EST. The updated release finished rolling out at approximately 6:30p.m. EST, after which no related error messages occurred.
To prevent a similar occurrence in the future, we’re first working with the team supporting packaging functionality to fix this bug involving custom setting and object usage. In the interim, we’re committed to process and documentation updates to increase visibility on this bug, and future releases will follow an alternate solution to avoid the possibility of triggering this bug again. We will also be investing in proactive monitoring that will help us detect conditions that are susceptible to conflict with this bug before release. Lastly, expanding the reach of our internal error message alerting will allow us to respond even sooner in the event of a similar occurrence.
We thank you for bearing with us while this incident occurred and for your continued trust in us.
Zach Bailey, Sr. Director of Software Engineering