Skip to main content
ashok_dugaputi
Helper ⭐️⭐️
June 28, 2017

Import a file from S3 to Bionic Rules directly

  • June 28, 2017
  • 18 replies
  • 159 views
I've been hearing multiple use cases / suggestions around being able to import the data file from S3 into Bionic Rules. 

1) Eliminate the need for setting up S3 jobs and then running the rule separately and potentially avoid 24 hours of delay before data is processed and available for consumption.

2) Loading data to SFDC directly without needing to store in MDA as a staging area

3) Without having to think about missing schedules. With Bionic Rules, reading the data from S3 and processing can be one rule.

4) Avoid storing unwanted data. Dimensionality reduction, filtering the bad quality records, applying the transformations.

5) Merging multiple files into one data set before loading. 

What are the other supporting use cases that you can think of?

    18 replies

    jitin_mehndiratta
    Helper ⭐️⭐️⭐️
    March 1, 2018
    Hi Jeff, 

    Can you provide a screenshot of the s3 path that you were trying to configure and also a screenshot of the gainsight managed bucket where the file is placed. Agreed the messaging is bad, but what that means is either the configured path is wrong or the file does not exist.

    Regards,
    Jitin
    darkknight
    Expert ⭐️
    March 1, 2018





    Jeff Kirkpatrick
    kevin_ly
    Helper ⭐️
    March 1, 2018
    Hey Jeff, 

    Try like what I have in my dev which works.



    I placed in the same input directory as you, except I have different file name. In case the picture looks a little small, may path is.

    MDA-Data-Ingest/input/Export from Bionic-2018-02-20.csv
    darkknight
    Expert ⭐️
    March 1, 2018
    Tried that too.  Gives me a little different error though.



    PS I don't have an Export directory - is that necessary?

    Your example in the S3 Dataset Task in Bionic Rules documentation doesn't show having to add all the folders in the path...just the filename (I took screenshots from the.gif on that doc):





    Jeff Kirkpatrick
    kevin_ly
    Helper ⭐️
    March 1, 2018
    Thanks for the update Jeff.

    I see that you have a support ticket opened therefore I shall look into that ticket. Doc has been flagged for update due to the same reason you have come upon, FYI.

    Best,

    Kevin Ly
    darkknight
    Expert ⭐️
    March 1, 2018
    Well dang it I was moving so fast trying to juggle multiple things I didn't realize I failed to select the Gainsight Bucket in the last attempt where I added the MDA-Data-Ingest/input path. 

    Thanks for the catch.  And thanks for flagging the support doc to have it updated.  I was originally going off that doc which is why it was failing to begin with.  An explicit reference in the verbiage to say "You must enter the full path" would be good.
    Jeff Kirkpatrick
    zach_davis
    Helper ⭐️
    March 16, 2018
    I noticed that this no longer supports the MM/DD/YYYY format. for instance my dates are stored as 03/13/2018 and it is giving me an error unless I trim off the leading zeros. Is this a known issue? THis is a standard date format used by most US customers.
    sai_ram
    Expert ⭐️⭐️
    October 16, 2018
    Hi All , Enhanced Support for Date and DateTime formats: Previously, only a few Date and DateTime formats were supported in an S3 Dataset task. Support has been extended to include many more Date and Datetime formats.

    Please refer the release notes for more information. 

    Thanks for posting!​