The Pardot application and website experienced intermittent disruptions in service on Friday, October 21 from approximately 11:10am UTC to 13:20pm UTC and again from 15:50pm UTC to 20:40pm UTC. The first disruption impacted the East Coast of the United States, while the second disruption impacted a broader, global audience.
Websites rely on DNS (Domain Name System) to translate domain names to IP addresses and route web traffic to the appropriate location on the Internet. The root cause of the incident was an intermittent service disruption with Pardot’s DNS service provider, Dyn, as the result of a large-scale distributed denial of service attack (DDoS). During the incident, users may have experienced issues loading http://www.pardot.com/, https://pi.pardot.com/, and Pardot-hosted assets such as forms, files, and landing pages. All impacted Pardot services were fully restored once the underlying DNS disruption was resolved.
On Friday 10/21 at 11:15am UTC, the Pardot Site Reliability team noticed issues resolving https://pi.pardot.com and engaged Salesforce’s Global Site Reliability team to investigate the cause. During the investigation process, we discovered that our DNS vendor, Dyn, had been experiencing a service disruption affecting some of its public nameservers as of 11:10am UTC. At 13:20pm UTC, we learned that Dyn had mitigated the disruption and was continuing to monitor the situation. All impacted Pardot services were also restored at this time. At 15:50pm UTC, Dyn reported that the disruption had returned, this time impacting a broader audience. At 20:50pm UTC, Dyn resolved the underlying DNS service disruption, and we confirmed that Pardot functionality had recovered while continuing to monitor. At 22:30pm UTC, the “all clear” was given.
Remedy & Future Prevention
Our team is taking several steps to mitigate the impact should a similar DNS disruption occur in the future. First, we plan to improve the redundancy of our DNS infrastructure by leveraging a second provider as a failover in the event of an emergency. We will also migrate from Dyn public nameservers to Salesforce private label Dyn nameservers, which were not impacted during this disruption. Lastly, the Pardot team will continue to investigate and implement further improvements to DNS stability as Salesforce continues to improve our DNS practices.
We appreciate your patience during this interruption. Thank you for your continued trust in us.
Zach Bailey, VP of Pardot Engineering