Skip to main content

[INCIDENT] Elevated API Errors in MDA (Backed Services)

  • October 21, 2016
  • 9 replies
  • 68 views

mahesh_patil
Forum|alt.badge.img
SUMMARY

We're experiencing an elevated level of API errors. Initial analysis confirms DNS related issues with our Upstream service provider and they are investigating the same. 

FOR MORE INFORMATION

Please subscribe to this article for regular updates until the issue is resolved. If you have questions about this issue, please open a ticket with us by sending a note to support@gainsight.com.

UPDATE

Friday, October 21 9:20AM CST: Our service provider has resolved the DNS issue. During the outage MDA services including Rules, Outreaches and Data Processor were paused. We've restarted all services and are replaying any jobs that failed during the service incident. 

9 replies

mahesh_patil
Forum|alt.badge.img
  • Author
  • Gainsight Employee ⭐️⭐️
  • October 21, 2016
Our Upstream providers have identified the root cause of the issue causing DNS Resolution errors and are currently working to resolve.

emily_mcdaniel_629a86
Forum|alt.badge.img+2
Our service provider has resolved the DNS issue. During the outage MDA services including Rules, Outreaches and Data Processor were paused. We've restarted all services and are replaying any jobs that failed during the service incident. 

arnold_price
  • Contributor ⭐️⭐️
  • October 21, 2016
A customer rec'd a denial of service error message while attempting to complete a NPS survey yesterday at 2:20pm is this related to the API/MDA DNS issue?

emily_mcdaniel_629a86
Forum|alt.badge.img+2
Arnold - I am not sure if this is related. I know you're working with our support team to investigate and they'll be able to help. 

mandy_major
Forum|alt.badge.img
  • Gainsight Employee ⭐️⭐️
  • October 21, 2016
UPDATE

We are experiencing this issue once again, and seeing errors "Unexpected end of JSON error" in Data Management, Copilot, Connectors tabs.  Will update this thread with the latest information as it comes in.

darkknight
Forum|alt.badge.img+5
  • Expert ⭐️
  • October 21, 2016
Thanks Mandy...that explains why I was experiencing this:  https://community.gainsight.com/gainsight/topics/timeouts-and-delays-in-gainsight-application

VERY unnerving!

kristin_f3c469
Forum|alt.badge.img
  • Contributor ⭐️⭐️⭐️⭐️⭐️
  • October 21, 2016
Might have something to do with this
http://www.wsj.com/articles/denial-of-service-web-attack-affects-amazon-twitter-others-1477056080
Cyberattack Knocks Out Access to Websites

darkknight
Forum|alt.badge.img+5
  • Expert ⭐️
  • October 21, 2016
Thanks for the info Kristin!

mahesh_patil
Forum|alt.badge.img
  • Author
  • Gainsight Employee ⭐️⭐️
  • October 22, 2016
Resolved - All services and queues are processing normally. 

Please reach out to support@gainsight.com if you are still experiencing any issues.