Skip to main content
anthonysabato
anthonysabato
Helper ⭐️
December 3, 2018

Email notifications for S3 drops should read better

  • December 3, 2018
  • 4 replies
  • 72 views
When I drop a file manually in S3, sometimes I forget a small step and the file fails. That is fine, but the email notification says successful. It should read better to say the error. I have to go look at the execution logs to see what was the issue. It is misleading and hard to filter out errors vs successes.

      4 replies

      sai_ram
      sai_ram
      Expert ⭐️⭐️
      January 11, 2019
      Hi Anthony, 

       

      Are you performing the task from Rules engine or Connectors or do you face this issue in both the places?

       

      What kind of steps are you missing which caused these errors?

      anthonysabato
      anthonysabatoAuthor
      Helper ⭐️
      January 11, 2019
      Hi Sai,

      This is only in the Connectors not Rules Engine. The setup was done by someone else, so sometimes I miss a step such as a date creeps into the file that isn't formatted correctly. Our raw data files need manipulation before dropping them in. I usually fix them using other code but its not always perfect.

      jitin_mehndiratta
      jitin_mehndiratta
      Helper ⭐️⭐️⭐️
      May 14, 2019
      Hi Anthony,

      I would recommend using Rules engine to ingest data from S3 for a variety of advantages as listed below:

      Advantages of using Rules Engine for S3 ingest

      1. Linking data ingest with the business processes: S3 Dataset task helps in performing business actions only after successful data ingest. E.g. If the file does not arrive on time, the rule will fail.

      2. Eliminating the need for storing raw data -

      (a) If a customer has concerns over data-security and data-storage, they can just enable data for processing and create actions on top of it.

      (b) Ingest selective data and aggregate during ingest process itself: For example, if the CSV exports (in customer's bucket) has data for all page views, a business rule can filter data from this file using advanced logic.

      3. Advanced lookup during ingest: The functions and merges in Bionic rules help in performing complex lookups and resolutions. E.g. resolving `Email Address -> Domain -> Account ID/Name`

      4. Increased limits for file size: S3 Dataset task will support file as large as 6 GB in size.

      Regards,

      Jitin

      anthonysabato
      anthonysabatoAuthor
      Helper ⭐️
      May 14, 2019
      Thank you for the reply and helpful feedback. I was hoping more of someone actually making fixes than make sense instead of redoing our current process beacuse notification messages are not very good. :(

      Terms & ConditionsAccessibility statement