Failure notifications on Real Time rules

How...is this not a thing? 

Hi @darkknight 

Real Time Rules runs every time there is an update by any one on Company/CTA. Imagine if you have 100 updates, this would result in 100 emails. For some customers, this number is in order of hundreds of thousands. This makes what we have for notifications in Rules or Connectors not work 1:1 with Real Time rules.

Some questions for you -

How frequently do you monitor for these failures / how soon you need to know about this? Assume you see a failure report once a day, is that a good frequency?

Email vs In-App Notifications - Is there a preference? With In-App, we might be able to group them better (One notification per rule). Or would you prefer to see all the failures in one place in Rules Engine? 

@rakesh 

For those 100 updates in your example, am I expecting every one of them to fail?  I would hope not.

Then it wouldn’t be 100 emails - it would only be the number of emails = number of failures.

I need to know when anything fails. Period end of sentence. If I don’t get a notification when something fails, then I have to manually check for failures. Because if I don’t, and something gets out of whack between systems, then I’ll have VPs breathing down my neck asking why Gainsight didn’t do what it was supposed to do, and depending on how long it takes for them to reach out to me, I may not have any log details to go check because it’s da 16 since it was supposed to have happened and the logs rolled off.

I need to know immediately when something fails, particular where data synchronization is involved (i.e. Gainsight couldn’t write to SFDC because of a record lock).  Subsequent processes may depend on the accuracy of the data.

I send my notifications to a Slack channel. It’s quicker/easier to reference and to collaborate on vs. in-app notifications or emails.

I think some type of failure notification is super important, and I do see the concern of possibly getting tons of alerts.

Is it possible for some type of middle ground, where you get an alert either per rule (or per rule action) that there are one or more failures, and either the alert itself contains a list of all said failures or points you exactly to where a list is and until you dismiss that particular alert you wouldn’t receive another unless you configure it otherwise? Your list would need to update in real time if more errors are happening so you can see more than one failure per alert.

Obviously that’s not a fully developed idea, but maybe opens a third door here.

What I think as an admin at least for me would be important, would be:

• The ability to receive some type of alert for any failure
• Some way to throttle the number of alerts for the same rule in a certain amount of time, while not losing visibility on the amount of errors in any one alert. (e.g, if there were 12 errors but I only received one alert, I would still know there were 12 errors on that rule)
• I think for the second point to be possible, there’d need to be some way to ‘acknowledge’ a failure since there’s no ‘resolve failure’ option for rules, it’s basically just fix and re-run it.
• Since the rules are real time, then the errors should be real time, at least that first one. As @darkknight mentioned the result of the action may have dependencies so being able to intervene in a timely manner could be critical. 

Hopefully that helps.

@bradley I don’t know Bradley, I think this may be the first time we somewhat disagree on something. 😂 That seems awfully complicated - moreso probably than just a simple notification process (something akin to Connectors posted below).  And the occurrence of multiple failures doesn’t necessarily mean its same issue that caused each failure. 

I do believe admins should have the autonomy to choose real-time notifications for any failure or maybe if they want a “daily digest” as well as the ability to identify the type of failures to notify, but I wouldn’t want the delivery of an MVP solution to hinge upon more granular throttling capabilties.

@bradley I don’t know Bradley, I think this may be the first time we somewhat disagree on something. 😂 That seems awfully complicated - moreso probably than just a simple notification process (something akin to Connectors posted below).  And the occurrence of multiple failures doesn’t necessarily mean its same issue that caused each failure. 

I do believe admins should have the autonomy to choose real-time notifications for any failure or maybe if they want a “daily digest” as well as the ability to identify the type of failures to notify, but I wouldn’t want the delivery of an MVP solution to hinge upon more granular throttling capabilties.

 

I don’t know that we disagree so much as I was trying to think of some alternative if the options were “NO alerts”, “one aggregated alert per day” or “alerts for every interaction” (the last of which seems like it wouldn’t happen as it was all but cited for the reason we don’t have alerts today). Never said they were good ideas either :)

I think that level of flexibility for alerts like in the screenshot would be rad for rules in general too!

@rakesh any update here? I’m having to check Real Time Rules multiple times daily to keep an eye out for failures (and yes, there are failures with indeterminate reasons, i.e “"Failed to query data from Query API").  

I shouldn’t have to manually check for failures. 

Hi are there any updates here? I recently had to create a separate bionic rule to “catch” the records that failed with the real-time rule (we have a real-time rule to push the CSM value from Gainsight to Salesforce). This seems to kind of defeat the purpose of the real-time rule…it is nice that most of the records are pushed over instantly, but we need to be alerted when these fail.

we have this as well - we have nightly rules that run to ‘clean-up’ any misses from the RTRs.