Skip to main content
(cross-posted to Rules Engine and Data Platform)



Having some trouble with outputting long text fields to S3 for automated pickup and ingest by Cisco.  Because these fields have line breaks in them, CSVs get jumbled - and tab delimited can get broken up by tabs in the formatted text being exported.



Any thoughts on making this easier? How to integration apps handle moving formatted text?  At the least, offering a pipe or other delimiter not usually found in formatted text?  Or adding a function to remove line breaks when exporting a delimited file?



Definitely interested in both solutions on our roadmap, and potential short-term workarounds.  Cisco will probably start to escalate this soon as it is blocking a clean, automated flow of info back to their data warehouse, so if we can avoid that with a good suggestion or coming enhancement, I'd be a fan!



This will be a common use case for Enterprise customers - I know there are multiple customers, including Cisco, who want a bi-directional API, but in the short term, making the data output more consumable would be a great start.



Thanks!
Scott,



As you suggested, adding "pipe"  as a delimiter in both "Export to S3" and "S3 task" in Bionic Rules should solve the problem, right? Do you see any other challenge w.r.t the export?



Regards,

Jitin
Hello Jitin - 



What's the standard for ETL output - and is there a separate End-of-Record delimeter? 



Pipe is better than comma, for sure, but I've also noticed people use pipes in their email sigs all the time now as dividers between social media info - so pasting in emails - a common practice - would blow up the output . . .



Scott.

Reply


Terms & ConditionsCookie settings