It would be convenient if we could create an email notification when a data file doesn't come in when it is expected. It would either be within the S3 job to say file is expected by X time on a specific day (or give a range of time/days) or maybe at the MDA object level so it could be done for API route as well.
Some customers have issues where the job fails on their side and no is notified. Also, some customers load files manually and this could serve as a reminder to them.
S3/MDA Notification
+2
+2To add to this, I think it could be done in copilot today, but would be really cool to have OOTB.
Great suggestion, Joe!!
We have multiple initiatives at ideation stage and will update you all how we can model this.
Ex: Data Traffic controller to tie together the Data acquisition and Processing. We also have another parallel / supporting initiative around Event driven framework. So we will consider this point as well.
BTW: Did you guys look into Web hook notification from S3? Certainly Webhook is not a replacement for the file arrival but it is for the notification post the file is loaded.
We have multiple initiatives at ideation stage and will update you all how we can model this.
Ex: Data Traffic controller to tie together the Data acquisition and Processing. We also have another parallel / supporting initiative around Event driven framework. So we will consider this point as well.
BTW: Did you guys look into Web hook notification from S3? Certainly Webhook is not a replacement for the file arrival but it is for the notification post the file is loaded.
+2Great to hear that these kinds of ideas are coming! Love the idea to tie everything together. In my specific example, the file isn't even making it to S3, the script just fails sometimes...but webhook is a good idea for other situations!
Agreed on the use case, Joe!! Certainly helps to alert the subscribers when the data is not available on time.
I've done something similar as a work around to another issue.
If you create a "Data validation Table". Then you are running a count on a single column in rules on the data load of the rows for that day. You then push that count to the new table, hard coding the table it is coming from. In this table it is table name, count, and whatever else.
In your case I would add an email address and hardcode an email address there as you are wanting an email notification.
At this point it is simply setting up an email rule based on that count and utilizing that email field as your 'to:' address.
The issue that this solution came in handy for me is a client didn't want to run rules if the data hadn't been loaded as they were doing wipe and reloads.
So, I did the above and had a lookup from the table they loaded to up to the "Validation Table", had to hardcode the table name in that Data table first.
This essentially had every row in the data table looking up to the same row in the Validation table. At this point we just used that count > 0 as a filter in whichever rules they wanted.
If you create a "Data validation Table". Then you are running a count on a single column in rules on the data load of the rows for that day. You then push that count to the new table, hard coding the table it is coming from. In this table it is table name, count, and whatever else.
In your case I would add an email address and hardcode an email address there as you are wanting an email notification.
At this point it is simply setting up an email rule based on that count and utilizing that email field as your 'to:' address.
The issue that this solution came in handy for me is a client didn't want to run rules if the data hadn't been loaded as they were doing wipe and reloads.
So, I did the above and had a lookup from the table they loaded to up to the "Validation Table", had to hardcode the table name in that Data table first.
This essentially had every row in the data table looking up to the same row in the Validation table. At this point we just used that count > 0 as a filter in whichever rules they wanted.
Joe / Wayne,
Here is a another thread on the same topic with alternative option recommended by Jeffrey Desilva.
https://community.gainsight.com/gainsight/topics/notification-when-data-file-doesnt-show-up-on-s3-as...
Wayne,
Interesting to know that you have come up with a creative solution on this. Can you help me understand this further and let me understand the feasibility to build from the product.
Here is a another thread on the same topic with alternative option recommended by Jeffrey Desilva.
https://community.gainsight.com/gainsight/topics/notification-when-data-file-doesnt-show-up-on-s3-as...
Wayne,
Interesting to know that you have come up with a creative solution on this. Can you help me understand this further and let me understand the feasibility to build from the product.
More then willing to, I think it is a first step to 'automated data validation.'
I'm planning to write a full article on it so others can utilize it I just haven't had time yet.
I'm planning to write a full article on it so others can utilize it I just haven't had time yet.
Sign up
If you ever had a profile with us, there's no need to create another one.
Don't worry if your email address has since changed, or you can't remember your login, just let us know at community@gainsight.com and we'll help you get started from where you left.
Else, please continue with the registration below.
Welcome to the Gainsight Community
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.