In our previous blog we saw how to load data into Amazon S3. Now in this blog, we will see How to read Amazon S3 Storage Files in SSIS (CSV, JSON, XML Format files). To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like perform File operations on Amazon S3 Files (e.g. Download, Upload, Create, Delete) then check these articles.
In nutshell, this post will focus on how to Read Amazon S3 Storage CSV, JSON and XML Files using respective SSIS Source tasks.
Components Mentioned in this article
|Amazon S3 CSV File Source|
|Amazon S3 CSV File Destination|
|Amazon S3 XML File Source|
|Amazon S3 JSON File Source|
- First, you will need to have SSIS installed
- Secondly, make sure to have SSDT
- You have obtained Amazon S3 account access key / secret key.
- Finally, do not forget to install ZappySys SSIS PowerPack
What is Amazon S3 Storage
Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 provides easy-to-use management features so you can organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements. Amazon S3 is designed for 99.999999999% (11 9’s) of durability, and stores data for millions of applications for companies all around the world.
In order to start, we will show several examples. ZappySys includes an SSIS Amazon S3 Source for CSV/JSON/XML File that will help you in reading CSV, JSON and XML Files from Amazon S3 to the Local machine, Upload files(s) to Amazon S3 Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to download files from Amazon S3 Storage.
You can connect to your Amazon S3 Account by entering your storage account credentials.
Read Amazon S3 Storage Files in SSIS (CSV, JSON, XML)
Let´s start with an example. In this SSIS Amazon S3 Source for CSV/JSON/XML File task example, we will read CSV/JSON/XML files from Amazon S3 Storage to SQL Server database.
- First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
- Drag and Drop relevant Amazon S3 Source for CSV/JSON/XML File Task from the SSIS Toolbox.
- Create a connection for Amazon S3 Storage Account.
- Select the relevant single file to read from Amazon S3 Storage in their relevant source of CSV/JSON/XML File Task.
- We can also read the multiple files stored in Amazon S3 Storage using wildcard pattern supported e.g. dbo.tblNames*.csv / dbo.tblNames*.json / dbo.tblNames*.xml in relevant source task
- We can also read the zip and gzip compressed files also without extracting it in the specific Amazon S3 Source for CSV/JSON/XML File Task.
- Finally, we are ready to load this file(s) data into the SQL Server.
Load Amazon S3 files data into SQL Server
- Inside Data Flow, Drag OLEDB Destination from SSIS Toolbox
- Connect our Source component to OLEDB Destination
- Double click OLEDB Destination to configure it
- Select Target Connection or click NEW to create new connection
- Select Target Table or click NEW to create new table based on source columns
- Click on Mappings Tab to Auto map columns by name. You can change mappings as you need
- Click OK to Save OLEDB Destination Settings
- That's it, You are now ready to run data flow. NOTE: If you wish to debug data flow and see records when you run, add data viewer by right click on blue arrow > Click Enable Data Viewer
- To execute data flow, Right click anywhere inside Data Flow Surface and click Execute Task
Above all, in this blog, we learned how to Read Amazon S3 Storage Files in SSIS. We used Amazon S3 Source for CSV file, Amazon S3 Source for JSON file and Amazon S3 Source for XML file to read the file(s) from Amazon S3 Storage and load data into SQL server. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.
Finally, you can use the following links for more information: