AppExchange Package Errors 6/22
Incident Report for Pardot

On Wednesday 6/22 at 12:45p.m. EST, we began rolling out a routine update to Pardot’s AppExchange package. Beginning around 1p.m. EST, approximately 1% of Salesforce instances that received this update began receiving “Invalid Data” errors upon record save.

Our Support Team began receiving reports of these errors from users at approximately 2:20p.m. EST, at which time we immediately activated our incident response plan. We recognized the cause of this incident to be an existing Salesforce bug. The bug is unpredictable; it is randomly encountered when a package upgrade contains new Apex trigger code that references new custom objects or settings added in the same upgrade. Wednesday’s package update included a new custom setting as well as additions to our trigger code that referenced this setting. In some Salesforce instances, this caused compilation errors which appeared whenever that trigger was invoked, preventing records from being saved.

At 2:35p.m. EST a solution was identified by our Engineering team, and the update began deploying around 3:00p.m. EST. The updated release finished rolling out at approximately 6:30p.m. EST, after which no related error messages occurred.

To prevent a similar occurrence in the future, we’re first working with the team supporting packaging functionality to fix this bug involving custom setting and object usage. In the interim, we’re committed to process and documentation updates to increase visibility on this bug, and future releases will follow an alternate solution to avoid the possibility of triggering this bug again. We will also be investing in proactive monitoring that will help us detect conditions that are susceptible to conflict with this bug before release. Lastly, expanding the reach of our internal error message alerting will allow us to respond even sooner in the event of a similar occurrence.

We thank you for bearing with us while this incident occurred and for your continued trust in us.

Zach Bailey, Sr. Director of Software Engineering

Posted almost 2 years ago. Jun 24, 2016 - 16:21 EDT

Resolved
A fix has been fully deployed to affected orgs and errors should no longer occur on record save. Please contact Support at help.pardot.com should you experience further issues related to this matter. A full postmortem will follow.
Posted almost 2 years ago. Jun 22, 2016 - 18:37 EDT
Identified
The root cause has been identified and we're working to implement a fix. As an interim solution, the errors resulting from record save can be resolved by following these steps in Salesforce: Navigate to Setup > Develop > Apex Classes. From there, click Compile all Classes.
Posted almost 2 years ago. Jun 22, 2016 - 15:47 EDT
Investigating
We're currently investigating reports of the latest Pardot AppExchange release causing error messages in Salesforce upon record save. Our Engineers are working to resolve this as top priority. Thanks for your patience while we work.
Posted almost 2 years ago. Jun 22, 2016 - 15:03 EDT