When I drop a file manually in S3, sometimes I forget a small step and the file fails. That is fine, but the email notification says successful. It should read better to say the error. I have to go look at the execution logs to see what was the issue. It is misleading and hard to filter out errors vs successes.
New Idea
Email notifications for S3 drops should read better
+1Hi Anthony,
Are you performing the task from Rules engine or Connectors or do you face this issue in both the places?
What kind of steps are you missing which caused these errors?
Are you performing the task from Rules engine or Connectors or do you face this issue in both the places?
What kind of steps are you missing which caused these errors?
+2Hi Sai,
This is only in the Connectors not Rules Engine. The setup was done by someone else, so sometimes I miss a step such as a date creeps into the file that isn't formatted correctly. Our raw data files need manipulation before dropping them in. I usually fix them using other code but its not always perfect.
This is only in the Connectors not Rules Engine. The setup was done by someone else, so sometimes I miss a step such as a date creeps into the file that isn't formatted correctly. Our raw data files need manipulation before dropping them in. I usually fix them using other code but its not always perfect.
Hi Anthony,
I would recommend using Rules engine to ingest data from S3 for a variety of advantages as listed below:
Advantages of using Rules Engine for S3 ingest
1. Linking data ingest with the business processes: S3 Dataset task helps in performing business actions only after successful data ingest. E.g. If the file does not arrive on time, the rule will fail.
2. Eliminating the need for storing raw data -
(a) If a customer has concerns over data-security and data-storage, they can just enable data for processing and create actions on top of it.
(b) Ingest selective data and aggregate during ingest process itself: For example, if the CSV exports (in customer's bucket) has data for all page views, a business rule can filter data from this file using advanced logic.
3. Advanced lookup during ingest: The functions and merges in Bionic rules help in performing complex lookups and resolutions. E.g. resolving `Email Address -> Domain -> Account ID/Name`
4. Increased limits for file size: S3 Dataset task will support file as large as 6 GB in size.
Regards,
Jitin
I would recommend using Rules engine to ingest data from S3 for a variety of advantages as listed below:
Advantages of using Rules Engine for S3 ingest
1. Linking data ingest with the business processes: S3 Dataset task helps in performing business actions only after successful data ingest. E.g. If the file does not arrive on time, the rule will fail.
2. Eliminating the need for storing raw data -
(a) If a customer has concerns over data-security and data-storage, they can just enable data for processing and create actions on top of it.
(b) Ingest selective data and aggregate during ingest process itself: For example, if the CSV exports (in customer's bucket) has data for all page views, a business rule can filter data from this file using advanced logic.
3. Advanced lookup during ingest: The functions and merges in Bionic rules help in performing complex lookups and resolutions. E.g. resolving `Email Address -> Domain -> Account ID/Name`
4. Increased limits for file size: S3 Dataset task will support file as large as 6 GB in size.
Regards,
Jitin
+2Thank you for the reply and helpful feedback. I was hoping more of someone actually making fixes than make sense instead of redoing our current process beacuse notification messages are not very good. :(
Reply
Sign up
If you ever had a profile with us, there's no need to create another one.
Don't worry if your email address has since changed, or you can't remember your login, just let us know at community@gainsight.com and we'll help you get started from where you left.
Else, please continue with the registration below.
Welcome to the Gainsight Community
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.