In this article, we will cover the points on how to load 10 million rows from SQL Server to Snowflake in just 3 minutes.

Snowflake is a data warehousing platform that resides in a cloud. Basically, it is a data warehouse software exposed as a service. It allows integrating many data sources via internal Snowflake Partner apps and loading them into the Snowflake storage engine. Another part of Snowflake is a computing engine that is responsible for serving your SQL queries. Both engines can work independently thus users that are querying a data warehouse are not affected by a data load that is happening at the same time. Snowflake is an elastic service, which means you pay only for the resources used. Specifically, you pay only for each second of the processing time you use.

To show how data loading works in Snowflake, we will take 10 million rows from SQL Server and load them into Snowflake using SSIS and ZappySys SSIS PowerPack. These SSIS PowerPack connectors will be used to achieve the task:

Export CSV File Task

Step-by-step – How to load 10 million rows from SQL Server to Snowflake in 3 minutes

Getting started

To achieve the goal, we will use a slightly modified Northwind database (example database from Microsoft) and SnowSQL – a command-line tool provided by Snowflake. So in the first steps, you will need to install the Northwind database and SnowSQL. After that, we will proceed to create a table in Snowflake – a table we will load the data into – and a file format which will be used to load data from the Snowflake staging area into the destination table. Once that’s complete, we will proceed in creating an SSIS package, adding and configuring the connectors, and finally running it to get the results. Let’s proceed!

Install a Northwind database

Download and run the creation script of a modified Northwind database. The only thing that was modified is a “CustomersForSnowflake” view added which returns 10M rows.

Install and configure SnowSQL command-line tool

Download and install SnowSQL; you will find the instructions on how to do that in https://docs.snowflake.net/manuals/user-guide/snowsql-install-config.html.

Once installed, you will need to configure the default Snowflake account name, user name, and password:

1. Go to Windows File Explorer and enter this path:
   %USERPROFILE%\.snowsql\
   Usually, when entered the path looks similar to this one:
   C:\Users\myUserName\.snowsql
2. Find a file named config and open it.
3. Then configure your credentials:

Create a table in Snowflake

Login to your Snowflake account, open a Worksheet and execute this query:

create or replace table Customers(
ID number,
CustomerID nchar(5),
CompanyName nvarchar(40),
ContactName nvarchar(30),
ContactTitle nvarchar(30),
Address nvarchar(60),
City nvarchar(15),
Region nvarchar(15),
PostalCode nvarchar(10),
Country nvarchar(15),
Phone nvarchar(24),
Fax nvarchar(24)
)

We will load 10M customers from the Northwind database and load them into this table.

Create a file format in Snowflake

Then in the same Worksheet create a file format for zipped CSV files by executing this query:

create or replace file format GzipCsvFormat
type = csv
field_delimiter = ','
null_if = ('NULL', 'null')
empty_field_as_null = true
compression = gzip
field_optionally_enclosed_by = '"'

We will use this file format when loading data from a Snowflake stage to the Snowflake destination table. This file format defines CSV format which is used by ZappySys Export CSV Task in SSIS.

Create an SSIS package

We are ready to create a new SSIS package and load some data into Snowflake. We are going to use two methods of how we load data into Snowflake. In one method we will load data into Snowflake’s local storage, while in the second one we will stage data in Amazon S3 bucket. Decide how you want to stage files, and then choose the appropriate workflow when creating a new package:

Loading 10 million rows from SQL Server to Snowflake

Add & configure ZappySys Export CSV Task

Firstly, drag and drop ZappySys Export CSV Task from SSIS toolbox and follow the instructions on how to configure it:

Configure source

1. Proceed with configuring the data source, from which you will be exporting data. We will use the Northwind database as an example:
   

   Export CSV Task: configuring OLE DB Connection to export data from SQL Server to Snowflake

2. Then move on and select it and enter the query or table name you want to export data from:
   

   Export CSV Task: configuring source SQL query for data loading from SQL Server to Snowflake

   We will be using this query:
   select top 10000000 * from CustomersForSnowflake

3. Then in the Split Options tab split the exported CSV into many files, e.g. into 50 MB chunks:
   

   Using data split options in the Export CSV Task

   NOTE: Snowflake recommends having files that are 10-100 MB in size when compressed. So it is perfectly safe to configure the value to 100 MB and above.

Configure target

Depending on which staging approach you chose to use, set the export target to a local path or S3 bucket:

Using local Snowflake storage

Just set the appropriate Save Mode and a file path:

Saving files locally to upload them to Snowflake local stage later

Using S3 storage

For storing staging files in S3, follow these steps:

1. Set Save Mode to Save to Connection and select <New ZS-AWS-STORAGE Connection>:
   

   Export CSV Task: configuring Amazon S3 endpoint as the target for data export

2. Then select S3 as Storage Service and fill in Access and Secret Key, select your region (optional):
   

   Configuring Amazon Connection Manager to store staging data in S3

3. Then in the Export CSV Task window, in Target tab configure it similarly:
   

   Export CSV Task: configuring target to store staging data in S3

   Another option you may consider is to use Amazon Storage Task or Azure Storage Task to upload files to S3 in a separate step after the Export CSV Step. For this perform the following steps.

   1. On the Target tab, set Save Mode to Save to Path (Local Disk) instead of Connection to S3 or Azure.
   2. Drag & Drop Amazon Storage Task or Azure Storage Task  and connect with the previous Export CSV Step
   3. Configure Storage Task to upload Local files to S3 or Azure Blob Storage.
   4. Continue to the next section for more instructions.

Add Execute Process Task to create a staging area

We are ready to add Execute Process Task to create a staging area in Snowflake. Again, depending on where you will store staging files, Snowflake’s local storage or Amazon S3, use one of the approaches below:

Create Snowflake local staging

Creating a local stage in Snowflake using SnowSQL command-line tool and SSIS

File path:

C:\Program Files\Snowflake SnowSQL\snowsql.exe

Arguments:

-q "CREATE OR REPLACE STAGE CustomersStaging" -d DEMO_DB -s Public

Create Amazon S3 staging

Creating an Amazon S3 stage in Snowflake using SnowSQL command-line tool and SSISFile path:

C:\Program Files\Snowflake SnowSQL\snowsql.exe

Arguments:

CREATE OR REPLACE STAGE CustomersStaging url='s3://your-bucket-name/destinationFolder/' credentials=(aws_key_id='AKIAXXXXXXXXXXXXXXXX' aws_secret_key='6p1ayaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaLXz88')

Add Execute Process Task to upload files to the staging area (local staging approach only)

We still need to upload CSV files that were saved locally for local Snowflake staging. So add another Execute Process Task and configure it similarly:

Uploading local CSV files to Snowflake stage using SnowSQL command-line tool and SSISFile path:

C:\Program Files\Snowflake SnowSQL\snowsql.exe

Arguments:

-q "PUT file://e:/temp/*.csv @DEMO_DB.PUBLIC.CustomersStaging"

Add Execute Process Task to copy files from the Snowflake/Amazon S3 stage to Snowflake table

Finally, add the final Execute Process Task to issue a command to load files from the stage into a real Snowflake table:

Copying data from a staging area to a Snowflake table using SSIS

File path:

C:\Program Files\Snowflake SnowSQL\snowsql.exe

Arguments:

-q "COPY INTO CUSTOMERS FROM @CustomersStaging file_format = (format_name = 'GZIPCSVFORMAT')" -d DEMO_DB -s Public

Execute the package

We are ready to execute the package, so just run it. In this example you see the workflow when using Amazon S3 as stage:

Executing an SSIS package to load data from Snowflake to SQL Server

The Results

Once the package executes, we can check the results:

Successful execution of SSIS package when loading data from Snowflake to SQL Server

If we go to Snowflake and execute the SELECT query, we see all 10M customers loaded:

Successful loading of 10 million rows from SQL Server to Snowflake using SSIS and ZappySys SSIS PowerPack

All done in less than 3 mins:

Conclusion

We achieved just what we aimed for – we loaded 10 million rows from SQL Server to Snowflake; all done in less than 3 minutes! Two approaches were taken into consideration: in one we staged CSV files in the local Snowflake stage and in the other we staged files in the Amazon S3 bucket. To accomplish this we used ZappySys Export CSV Task, which allowed us to export data from the SQL Server view to CSV format, split files into smaller ones, zip them and store them locally or upload to the Amazon S3 bucket. Finally, we used a standard SSIS Task “Execute Process Task” to issue Snowflake commands such as creating the stage area in Snowflake, uploading CSVs into the local stage and loading data into the Snowflake table.

Download a sample package

Load 10M rows from SQL Server into Snowflake.dtsx.zip

References

https://docs.snowflake.net/manuals/user-guide/data-load-bulk.html

https://docs.snowflake.net/manuals/user-guide/snowsql.html

https://docs.snowflake.net/manuals/user-guide/data-load-s3-create-stage.html

https://docs.snowflake.net/manuals/sql-reference/sql/create-file-format.html

https://docs.snowflake.net/manuals/sql-reference/sql/put.html

https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-table.html

The post Load 10M rows from SQL Server to Snowflake in 3 minutes appeared first on ZappySys Blog.

Export REST API to CSV is in some cases necessary to process the data because many tools can handle CSV files. In this new article, we will show different ways to export the data. The first example will do it using C#. The second example with use Python. Let’s take a look at these examples.

Requirements to export REST API to CSV

1. First of all, you will need ZappySys ODBC PowerPack installed.
2. Secondly, you will require Visual Studio installed for the C# example.
3. Finally, it is necessary Python installed for the Python example.

Export REST API to CSV using C#

C# is a pretty popular programing language. In the first example, we will show how to display the REST API information to a CSV file named. Let’s take a look at the code:

1. First of all, we will connect to REST API using a connection to the following Data Path:
   

   https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json

   using (OdbcConnection conn = 
         new OdbcConnection("Driver={ZappySys JSON Driver};DataPath='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json';"))

2. Secondly, we create a connection using a file stream. We will save the results to a file in the c:\sql\sample.csv:
   

   FileStream fs = new FileStream("C:\\sql\\sample.txt", FileMode.Create);
   StreamWriter writer = new StreamWriter(fs);
   StringBuilder output = new StringBuilder();

3. Also, we will add a query to REST API. Note that with the ZappySys ODBC PowerPack, you can do a simple SQL query to get REST API data. This is pretty simple and intuitive:
   

   OdbcCommand cmd = new OdbcCommand(
        @"SELECT  CustomerID,CompanyName FROM value", conn);

4. In addition, we will read the data and close the connections:
   

   conn.Close();  
   writer.Close(); 
   fs.Close();

5. The complete code will be the following:
   

   using System;
   using System.IO;
   using System.Data.Odbc;
   
   public class Program
   {
       public static void Main()
       {
           var outpath = @"C:\temp\sample.txt";
   
           using (var conn = new OdbcConnection("Driver={ZappySys JSON Driver};DataPath='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json';"))
           {
               conn.Open();
               var cmd = new OdbcCommand(@"SELECT CustomerID,CompanyName FROM $ WITH(Filter='$.value[*]')", conn);
               
               //Increases the timeout duration from the default 30 seconds, which may be insufficient in certain scenarios.
               cmd.CommandTimeout=600; // 600-seconds
               
               var rdr = cmd.ExecuteReader();
   
               using (var fs = new FileStream(outpath, FileMode.Create))
               {
                   using (var writer = new StreamWriter(fs))
                   {
                       //write file header
                       writer.WriteLine("CustomerID,CompanyName");
   
                       while (rdr.Read())
                       {
                           //write file row
                           writer.WriteLine("{0},{1}", rdr["CustomerID"], rdr["CompanyName"]);
                       }
                       conn.Close(); //close connection
                       writer.Close();
                       fs.Close();
                   }
               }
           }
   
           //Read from file and display the content
           Console.Write(File.ReadAllText(outpath));
   
           Console.WriteLine("\r\n===== Press any key to end the program =====\r\n");
           Console.Read();
       }
   }

    
6. Finally, you will be able to see the file created:
   

   REST API into CSV

Export REST API to CSV using Python

Python is another really popular programming language. The popularity is growing a lot. In this example, we will learn how to Export REST API to CSV using Python.

1. First of all, you will need to install Pip if not included in Python. Pip is Package Installer.
   For instructions about the installation, refer to this link.
2. Secondly, you will also need the pyodbc. The pyodbc allows connecting to ODBC using Python. To install it go to the scripts folder of Python where Python is installed and run this command:pip install pyodbc.
3. Once that pyodbc is installed, we will run the following code:Full Code
   

   import csv
   import pyodbc
   
   conn = pyodbc.connect(
       r'DRIVER={ZappySys JSON Driver};'
       )
   cursor = conn.cursor()	
   rows = cursor.execute("SELECT CustomerID,CompanyName FROM value WHERE COUNTRY='Germany' WITH (SRC='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json')") 
   with open(r'C:\sql\cus2.csv', 'w', newline='') as csvfile:
       writer = csv.writer(csvfile)
       writer.writerow([x[0] for x in cursor.description])  
       for row in rows:
           writer.writerow(row)

4. Now lets’s understand parts of above code. We have used the csv and pyodbc modules in the code:
   

   import csv
   import pyodbc

5. Also, we connect to the ZappySys ODBC Driver:
   

   conn = pyodbc.connect(
   r'DRIVER={ZappySys JSON Driver};'
   )

6. In addition, we have used a cursor to get the rows and send a SQL query to get data from the REST API:
   

   cursor = conn.cursor() 
   rows = cursor.execute("SELECT CustomerID,CompanyName FROM value 
   WHERE Country='Germany' WITH 
   (SRC='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json')")

7. Following code is to open the CSV file stream:
   

   with open(r'C:\sql\customer.csv', 'w', newline='') as csvfile:

8. Finally, we will write the data from REST API into the CSV file:
   

   writer = csv.writer(csvfile)
   writer.writerow([x[0] for x in cursor.description]) 
   for row in rows:
   writer.writerow(row)

9. To conclude, if everything is OK, you will be able to see the created CSV file:
   

   Python code REST API

Using ODBC DSN in Connection String

So far we have seen DSN less connection string approach for ODBC Driver but now lets look at another way to use ODBC Driver in your C# or Python code. You can define many settings on DSN Datasource rather than setting in the ConnectionString.

Configure DSN for REST API Connection

1. First of all, we will access the following URL:
   

   https://services.odata.org/V3/Northwind/Northwind.svc/Orders?$format=json

2. Secondly, in the windows start menu, Search for “ODBC” open the ODBC Data Sources.
3. Also, in the ODBC Administrator, press Add and select the ZappySys JSON
   Driver:
   

   Add ODBC JSON

4. Finally, specify the URL of step 1 and save the configuration:
   

   URL ODBC

Using ODBC DSN in C# Code

Now to use ODBC DSN you created simply change our previous C# Code as below (Just one line)

using (var conn = new OdbcConnection("DSN=Your-DSN-Name-Goes-Here"))
{
    conn.Open();
    ...........
    ...........
    ...........

Using ODBC DSN in C# code to call REST API

Using ODBC DSN in Python Code

Now to use ODBC DSN you created simply change our previous C# Code as below (Just one line)

conn = pyodbc.connect(
    r'DSN=Your-DSN-name-Goes-Here;'
    )
	
	...........
	...........
	...........

ZappySys JSON /REST API Driver Query Examples

Reading from XML files or API can be done using the same way as previous sections except you have to use ZappySys XML Driver. Read help file here to see json query examples.

ZappySys XML / SOAP Driver Query Examples

Reading from XML files or API can be done using the same way as previous sections except you have to use ZappySys XML Driver. Read help file here to see xml query examples.

Conclusion

To conclude, in this article, we show how to access REST API using C# and Python. We used the ZappySys ODBC PowerPack that allows accessing to REST API data and JSON files using SQL queries. It is possible to create simple SQL queries and access the data. It is also possible to access to XML files and Web API with this tool. If you liked this tool you can test the ZappySys ODBC PowerPack here.

References

Finally, if you want to read more about this topic, refer to these links:

• Download ZappySys ODBC PowerPack Installer
• How To Use the ODBC .NET Managed Provider in Visual C# .NET and Connection Strings
• Python ODBC Wiki

The post How to export REST API to CSV using c# or Python appeared first on ZappySys Blog.

Calling SSRS Reports in SSIS is straightforward using the ZappySys Reporting Services task. Also, we will show how to export files SSRS files using SSIS, how to send reports in emails and how to send parameters.

In addition, we will be using this ZappySys SSIS PowerPack component to make things work:

	SSIS Report Generator Task (Export SSRS Reports)
	Advanced SSIS File System Task

In order to understand the functionality, the following video tutorial can be useful to understand the functionality of this useful Reporting Services task:
[youtube https://www.youtube.com/watch?v=ajnSgbXls7I&w=560&h=315]

Requirements

1. First of all, we will need ZappySys SSIS PowerPack. This tool contains the Reporting Services Task.
2. Secondly, you will require SSDT for Business Intelligence to create SSIS projects.
3. Also, make sure to have an SSRS report deployed. For more information
   about creating reports and deploying, refer to these links:

• • Create a Basic Table Report (SSRS Tutorial)
  • Publishing Reports to a Report Server

Getting started

We will first show how to export an SSRS Report to Word format (docx). Also, the ZappySys SSIS PowerPack, allows converting your SSRS reports deployed into Word (docx and doc), Excel (xlsx and xls), PDF, TIFF, CSV and XML.

Calling SSRS Reports in SSIS

1. First of all drag and drop the ZS Reporting Task that is included with our ZappySys SSIS PowerPack:
   

   SSRS task in SSIS

2. Secondly, in general tab, make sure that the Export server report is enabled.
3. Additionally, specify the report service URL. If you are not sure what is your report service URL, check the step 3 in requirements.
4. In addition, in report path, select the report that you want to export to a file.
5. Also, check the output as a file option:
   

   Export report to file

6. Additionally, in the export settings tab, go to Save as and specify the path and extension of the file to export. You can export to Excel, Word, PDF, Tiff, CSV and XML:
   

   Export SSRS report using SSIS to Excel, Word, PDF

7. Finally, run the package and the file will be created:
   

   SSRS report exported to docx

Changing the names of the reports with the date format

In order to explain how to add the prefix or suffix of a file with the current date. For example, I want my SSRS reports with this format:
Filename20180323.xlsx where 20180323 means march 23, 2018. To understand how to do it, In this new example, we will show how to do it.

1. First of all, we will use the ZappySys Advanced File System Task:
2. Secondly, we will use the option Rename files and rename using the fun_getdate function like this: Filename<<yyyy-MM-dd,FUN_GETDATE>>.xlsx
   

   Rename the file with a date in SSIS

3. Also, you can watch all the available examples of functions by pressing the Expressions (x) button:
   

   Special SSIS functions

4. Finally, press insert variable and function examples and you will be able to see all the examples of special functions to simplify your life:
   

   Samples of functions

Moving your report files to a UNC path

Once that you have your files exported from reporting services to a file, you can move, copy them to another path. In this example, we will show how to move files to a Shared folder using a UNC path.

1.  First of all, specify your file name to move. You can use *.xml for example to move all the files.
2. Secondly, specify the UNC path in destination:
   

   SSIS move files to UNC path

3. Also, It is possible to sort files by name, size, modified date or other properties:
   

   Sort file in SSIS by name, size, modified date

4. Finally, press the Examples link, you can see some custom queries that you can create:
   

   Sample SSIS queries to files

Sending SSRS Report as an attachment in an email

The task also allows calling SSRS reports in SSIS and send the reports to an email. To do it, it is necessary to have an SMTP Server. Let’s take a look at the steps:

1. First of all, in the general tab, check the email report file and uncheck the Output as file option:
   

   Output SSRS report in SSIS as attachment

2. Also, in Email setting tab, press New in SMTP connections to create a new connection. You will need to provide an SMTP server name, an email and a password. It is also necessary to enter the port. You can find the list of some SMTP servers and ports here.
3. In addition, you have an option to specify the sender, receiver, subject, and body. It is possible to write the body in HTML format. You can also specify SSIS variables. The following HTML code shows how to write in bold,
   italic, with bullets and more:
   

   <h1>This is an example using headers</h1>
   <p><strong>This is an example to write using bold</strong> <strong><br />
   </strong><em>This is an example to write using italic <br /></em>
   This is an example using underline<br /><br />
   </p>
   <ol>
     <li>This is an example with lists</li>
     <li>Sedond of the list</li>
     <li>Third of the list</li>
   </ol>
   <ul style="list-style-type:circle">
     <li>This is an example with bullets</li>
     <li>Bullet sample </li>
     <li>Working with bullets</li>
   </ul>

    
   

   Dynamic HTML in SSIS

4. Additionally, in Visual Studio go to the menu and SSIS, Variables and create two variables that will be used in the report named Title and ReportName of type string and specify values for them:
   

   Using SSIS variables in emails

5. Finally, run the script and check your email received:
   

   Email received SSIS Html format

Calling SSRS Reports in SSIS with parameters

Another key point is the use of parameters. Sometimes your SSRS includes parameters. With our Reporting Service Task, it is possible to call an SSRS report and pass SSIS parameters. You will require a Report with a parameter to test it. If you do not know how to create a parameter in SSRS, click here.

1. First of all, you will need an SSRS Report with a parameter:
   

   Reporting Service Parameter

2. Secondly, in SSDT or Visual Studio, in the menu, go to SSIS and variables. Create an SSIS variable:
   

   Variables in SSIS

3.  Finally, in Report Services Task in Visual Studio, go to Report Parameters and map the SSRS parameter with the SSIS variable:
   

   Map SSIS variable with SSRS parameters

Calling SSRS Reports in SSIS using data-driven configuration

Data-driven configuration allows configuring different reports with different configurations. In addition, the option allows handling multiple reports using a table with all the configurations for each report. Also, this task creates a table in SQL Server with the configurations for each report. Let’s take a look to see how this works.

1. First of all, select the data-driving report export using config table.
2.  Also, you will need to specify the connection to the SQL Server
   database.
3. In addition, go to config table, press new to create a new table:
   

   SSRS Data-driven in SSIS

4. Additionally, ZappySys allows creating a sample table that can help you to understand how to configure local reports, Calling SSRS Reports in SSIS and save to files, specify paths, ports, email options and more:
   

   CREATE TABLE [{{SchemaName}}].[{{TableName}}](
   	Id int identity(1,1) NOT NULL  PRIMARY KEY, 
   	GroupName varchar(100) NULL,  /* Used to filter reports */
   	ReportType varchar(10) NOT NULL, /*Local or Server*/
   	ReportServiceUrl  varchar(500) null, 
   	ReportPath varchar(500) NOT NULL, 
   	IsEnabled bit NOT NULL default(1),
   	ExportAsFile bit NOT NULL default(1),
   	ExportAsEmail bit NOT NULL default(0),
   	Parameters varchar(max) NULL, /* ParameterName1;IsMultiValue;Value1|....ParameterNameN;IsMultiValue;ValueN */
   	ExportPath varchar(500) NULL,
   	AddFileSuffix bit NULL,
   	ExportFileSuffix varchar(500) NULL, /* e.g. _ddMMyyyy, _ddMMyyyyhhmmss, _ddMMyyyyhhmmss.fff (see this URL http://msdn.microsoft.com/en-us/library/az4se3k1(v=vs.110).aspx) */
   	Description varchar(500) NULL,
   	EmailFrom varchar(255) NULL,
   	EmailTo varchar(1000) NULL,
   	EmailCc varchar(1000) NULL,
   	EmailBcc varchar(1000) NULL,
   	EmailSubject varchar(500) NULL,	
   	EmailIsHtml bit NULL default(1),	
   	EmailBodyText varchar(max) NULL,
   	EmailAttachmentFile varchar(500) NULL, /*Unused: For future use only*/
   	LastExportStartDate datetime NULL,
   	LastExportEndDate datetime NULL,
   	LastStatus varchar(20) NULL,
   	LastMessage varchar(max) NULL,
   	Datasets varchar(max) NULL, /*if ReportType=local and report file is using dataset(s) then supply data in this column (multiple values must be pipe delimited). e.g. Dataset1|Dataset2)*/
   	DatasetConnectionStrings varchar(max) NULL, /*if ReportType=local and report file is using dataset(s) then supply data in this column (connection string must be same order as datasets (multiple values must be pipe delimited) enter just one connectionstring if all datasets using same connection. ConnectionString must be ADO.NET for SQLServer, ODBC or OLEDB connection string and correct driver must be installed on the system)*/
   	DatasetQueries varchar(max) NULL /*if ReportType=local and report file is using dataset(s) then supply data in this column (queries must be in the same order as dataset(s) listed in Datasets column. enter multiple queries in this format <xml><query>{query for 1st dataset}</query><query>{query for 2nd dataset}</query></xml>)*/
   );

    
   

   Sample Database in SSIS for SSRS

5. Finally, you can check the table with the configuration:
   

   Data-driven table created in SSIS

Calling SSRS Reports in SSIS to Export local report option

With local reports, you can work with rdl files. The rdl (Report Definition Language) files are part of the SSRS. They are physical files that contain the report format and other components. Using the export local report you can customize queries and see the results as reports. Here we will explain how to do it.

1. First of all, in the Reporting Services task in Visual Studio, make sure to select the option Export local report.
2. Secondly, in report path, you need to specify your rdl file path.
3. In addition, check the dataset. The dataset is usually detected by default. It is included in the SSRS.
4. Also, in Dataset Constr, you will need to specify the connection information. You need to specify the SQL Server name, Database name, Authentication type. For more information about connection string, click here.
5. Finally, in dataset queries, you need to specify your query using the XML and query tags:
   

   SSIS local SSRS report

Conclusion

In summary, calling SSRS Reports in SSIS is a straightforward task using the ZappySys Reporting Service task. In this article, we learned how to export a Reporting services Report to a file. Also, we learned how to send emails using SSIS with the report attached and we also learned how to rename, copy files and finally how to use Data-Driven configuration and create queries to local reports. If you liked the tool, you can download it here.

References

Finally, refer to these links for more information:

• Reporting Services Tutorial
  
• SSRS Task in SSIS

The post Calling SSRS Reports in SSIS (Export / Email) appeared first on ZappySys Blog.

In this article, we will learn how to import REST API in Power BI.

Power BI is a very popular Business Analytic tool used to get business information. It is very popular because it is easy to install, simple to learn and very intuitive. Also, REST API is very popular these days and we wanted to show you a way to integrate them in Power BI with the help of the ZappySys ODBC PowerPack. This software includes powerful drivers to query REST API URL, Local XML / JSON files and XML SOAP Web Service with simple SQL queries.

The following video will show how to import REST API in Power BI (JSON or XML Data):
[youtube https://www.youtube.com/watch?v=-3OPfhMY1aI&w=720&h=405]

Requirements

1. First, you will need to have Power BI Desktop (FREE) installed
2. In the second place, you will need to have internet access
3. On third place, our ZappySys ODBC Power Pack

Step-by-Step: Import REST API into Power BI

We’ll walk through the steps to import a REST API into Power BI. The steps outlined below focus on working with JSON APIs, but the same principles can be applied when importing SOAP XML Web Services or local JSON/XML files.

For demonstration purposes, we will utilize a REST API to retrieve data in JSON format. The example URL we’ll use is:

https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json

This URL leverages OData and presents information in JSON format. Our objective is to use the ZappySys ODBC Power Pack to establish a connection to this URL and retrieve the information seamlessly into Power BI.

Create ODBC DSN – JSON Driver

DRIVER={DRIVER-NAME-GOES-HERE}[;Prop1=xxxxxx][;Prop2=xxxxxx] .... [;PropN=xxxxxx]

DRIVER={ZappySys JSON Driver}
DRIVER={ZappySys JSON Driver};DataPath='c:\mydata.json'
DRIVER={ZappySys XML Driver};DataPath='http://myserver/api/xml/getOrders'
DRIVER={ZappySys XML Driver};DataPath='c:\mydata.xml'

DRIVER={DRIVER-NAME-GOES-HERE}[;Prop1=xxxxxx][;Prop2=xxxxxx] .... [;PropN=xxxxxx]

DSN=MyJsonAPI_DSN
DSN=MyJsonAPI_DSN;DataPath='http://myserver/api/json/getOrders'

Connect to REST API data source in Power BI (Connect JSON / XML data)

Publish Power BI dashboard

1. In order to publish the report, press Publish
2. Press Save
   

   Save changes in Power BI:

    

3. Select a workspace for the report
   

   Select Power BI workplace.

4. And that is all. You have now your report ready.

Import data using ODBC ConnectionString (DSN-less)

Let’s explore the process of importing data using the ODBC Driver without the need for configuring an ODBC DSN.

When importing data, you have the option to either use an ODBC DSN or a Direct ConnectionString. This section will guide you through the steps of utilizing the ODBC ConnectionString.

After opting for the ODBC option to retrieve data, you will encounter the “Select ODBC Data Source” screen. To proceed without a DSN, follow these steps:

1. Change the DSN dropdown to “(none).”
2. Enter the full ODBC ConnectionString, adhering to the syntax below. You can construct your own connection string using the ODBC Driver UI and utilize the “Copy ConnectionString” option.
   Syntax: 

   Driver={Your Driver Name}[;Property1=value][;Property2=value]......

3. Here is a screenshot with ConnectionString Setting
   

   Import Data from ODBC Driver – DSN less mode (Use Connection String)

4. Click “Next” to select the authentication method. Choose “Windows” and input the password value if necessary. If your connection string already includes the Password attribute, there’s no need to worry.
   

   Import Data from ODBC Driver – using Connection String – With Password Attribute

POST data to REST API URL

Up until now, we’ve explored the process of fetching data from URLs and importing it into Power BI. However, in certain scenarios, APIs may necessitate the use of the PUT or POST method. These methods involve submitting parameters in the body and receiving a response.

For a more in-depth understanding of how to seamlessly POST data to a URL in Power BI, watch the informative video below.

Calling XML SOAP Web Service in Power BI

So far, we have looked at examples to consume data using JSON driver. Now let’s look at an example, to call XML SOAP Web Service in Power BI.

To call SOAP API you need to know Request XML Body Structure. If you are not sure how to create SOAP Request body then no worries. Check this article to learn how to generate SOAP Request body using the Free tool SoapUI. Basically, you have to use SoapUI to generate Request XML and after that, you can replace parameters as needed in the generated body.

What is SOAP Web Service?

If you are new to SOAP Web Service sometimes referred as XML Web Service then please read some concept about SOAP Web service standard from this link There are two important aspects in SOAP Web service.

1. Getting WSDL file or URL
2. Knowing exact Web Service URL

What is WSDL

In very simple term WSDL (often pronounced as whiz-dull) is nothing but a document which describes Service metadata (e.g. Functions you can call, Request parameters, response structure etc). Some service simply give you WSDL as xml file you can download on local machine and then analyze or sometimes you may get direct URL (e.g. http://api.mycompany.com/hr-soap-service/?wsdl )

Example SQL Query for SOAP API call using ZappySys XML Driver

Here is an example SQL query you can write to call SOAP API. If you not sure about many details then check next few sections on how to use XML Driver User Interface to build desired SQL query to POST data to XML SOAP Web Service without any coding.

SELECT * FROM $
WITH(
	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
	,DataConnectionType='HTTP'
	,CredentialType='Basic' --OR SoapWss
	,SoapWssPasswordType='PasswordText'
	,UserName='myuser'
	,Password='pass$$w123'
	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
	,ElementsToTreatAsArray='HolidayCode'	
	,RequestMethod='POST'	
	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
	,RequestData='
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hol="http://www.holidaywebservice.com/HolidayService_v2/">
   <soapenv:Header/>
   <soapenv:Body>
      <hol:GetHolidaysAvailable>
         <!--type: Country - enumeration: [Canada,GreatBritain,IrelandNorthern,IrelandRepublicOf,Scotland,UnitedStates]-->
         <hol:countryCode>UnitedStates</hol:countryCode>
      </hol:GetHolidaysAvailable>
   </soapenv:Body>
</soapenv:Envelope>'
)

Now let's look at steps to create SQL query to call SOAP API. Later we will see how to generate code for your desired programming language (e.g. C# or SQL Server)

Video Tutorial - Introduction to SOAP Web Service and SoapUI tool

Before we dive into details about calling SOAP API using ZappySys XML Driver, lets first understand what is SOAP API and how to create SOAP requests using SoapUI tool. You will learn more about this process in the later section. The video contains some fragment about using SOAP API in SSIS but just ignore that part because we will be calling Soap API using ZappySys ODBC Driver rather than SSIS Components.  

Using SoapUI to test SOAP API call / Create Request Body XML

Assuming you have downloaded and installed SoapUI from here, now we are ready to use WSDL for your SOAP Web Service Calls. If you do not have WSDL file or URL handy then contact your API provider (sometimes you just have to add ?wsdl at the end of your Service URL to get WSDL so try that. Example: http://mycompany/myservice?wsdl ). If you don't know what is WSDL then in short, WSDL is Web service Description Language (i.e. XML file which describes your SOAP Service). WSDL helps to craft SOAP API request Body for ZappySys XML Driver. So Let's get started.

1. Open SoapUI and click SOAP button to create new SOAP Project
2. Enter WSDL URL or File Path of WSDLFor example WSDL for our sample service can be accessed via this URL

   http://www.dneonline.com/calculator.asmx?wsdl

   
   Create new SOAP API Project in SoapUI tool for SOAP API Testing
3. Once WSDL is loaded you will see possible operations you can call for your SOAP Web Service.
4. If your web service requires credentials then you have to configure it. There are two common credential types for public services (SOAP WSS or BASIC )
   1. To use SOAP WSS Credentials select request node and enter UserId, Password, and WSS-PasswordType (PasswordText or PasswordHash)
      
      Configure SOAP WSS Credentials for SoapUI (SOAP API Testing Tool)
   2. To use BASIC Auth Credentials select request node and double-click it. At the bottom click on Auth (Basic) and From Authorization dropdown click Add New and Select Basic.
      Configure Basic Authorization for SoapUI (SOAP API Testing Tool)
5. Now you can test your request first Double-click on the request node to open request editor.
6. Change necessary parameters, remove optional or unwanted parameters. If you want to regenerate request you can click on Recreate default request toolbar icon.
   Create SOAP Request XML (With Optional Parameters)
7. Once your SOAP Request XML is ready, Click the Play button in the toolbar to execute SOAP API Request and Response will appear in Right side panel. Test SOAP API using SoapUI Tool (Change Default XML Body / Parameters, Execute and See Response)

Create DSN using ZappySys XML Driver to call SOAP API

Once you have tested your SOAP API in SoapUI tool, we are ready to use ZappySys XML driver to call SOAP API in your preferred BI tool or Programming language.

1. First open ODBC Data Sources (search ODBC in your start menu or go under ZappySys > ODBC PowerPack > ODBC 64 bit)
2. Goto System DSN Tab (or User DSN which is not used by Service account)
3. Click Add and Select ZappySys XML Driver ZappySys ODBC Driver for XML / SOAP API
4. Configure API URL, Request Method and Request Body as below ZappySys XML Driver - Calling SOAP API - Configure URL, Method, Body
5. (This step is Optional) If your SOAP API requires credentials then Select Connection Type to HTTP and configure as below.
   ZappySys XML Driver - Configure SOAP WSS Credentials or Basic Authorization (Userid, Password)
6. Configure-Request Headers as below (You can get it from Request > Raw tab from SoapUI after you test the request by clicking the Play button) Configure SOAP API Request Headers - ZappySys XML Driver
7. Once credentials entered you can select Filter to extract data from the desired node. Make sure to select array node (see special icon) or select the node which contains all necessary columns if you don't have array node. Select Filter - Extract data from nested XML / SOAP API Response (Denormalize Hierarchy)
8. If prompted select yes to treat selected node as Array (This is helpful when you expect one or more record for selected node) Treat selected node as XML Array Option for SOAP API Response XML

Preview SOAP API Response / Generate SQL Code for SOAP API Call

Once you configure settings for XML Driver now you can preview data or generate example code for desired language (e.g. C#, Python, Java, SQL Server). Go to Preview tab and you will see default query generated based on settings you entered in previous sections. Attributes listed in WITH clause are optional. If you omit attribute in WITH clause it will use it from Properties tab.

Preview Data

Preview SOAP API Response in ZappySys XML Driver

Generate Code Option

Generate Example Code for ZappySys Driver

SOAP / REST API pagination in Power BI

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

SOAP / REST API Error Handling in Power BI

Sometimes errors occur... they just do and there is nothing you can do! Or can you? Actually, in ODBC PowerPack you can handle them in two ways.

METHOD 1 - Using Error Handling Options

When to use?

You may want to use them when your source is a resource located on the Internet; e.g. a file on a website, a file on an FTP server or just a plain API HTTP response. By default, when a remote server returns an error, data retrieval is stopped, an error is raised and no data is given back to you. This might not be always desirable.

Scenario 1

Imagine a scenario, that there is a web server which each day at 12 AM releases a new JSON file with that day's date as filename, e.g. http://www.some-server.com/data/2018-06-20.json. And, of course, you want to download it and use it daily in your Power BI report. But you have a problem: Power BI report data sources are refreshed each hour and you may get HTTP 404 status code (no file was found) when a file is not released yet. Which consequentially means other data sources won't be updated as well and you will see old and cached data on the report. That's where you could use Continue on any error or Continue when Url is invalid or missing (404 Errors) to avoid an error being raised and let other data sources to be updated.

Scenario 2

Another scenario is when you expect a web server to raise some kind of HTTP error when accessing a URL. You don't want ODBC Data Source to raise an error but instead, you want to get response data. That's where you can use Continue on any error or alike together with  Get response data on error to continue on an error and get the data:

METHOD 2 - Using Connection [Retry Settings]

Another scenario you may run into is a buggy web server. You ask it to give you some file or data and it, like a snotty kid, just doesn't give it to you! You have to ask twice or thrice before it does its job. If that's the case, you have to retry HTTP requests using Connection:

Security / Connection Types (Options for HTTP, OAuth, SOAP) in Power BI

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

• HTTP
• OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

• SOAP WSS (when accessing a SOAP WebService)
• Static Token / API Key (when need to pass an API key in HTTP header)
• Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
• JWT Token (As per RFC 7519)

OAuth

If you are trying to access REST API resource, it is a huge chance, you will need to use OAuth Connection. Read this article to understand how OAuth authentication and authorization works and how to use it (article originally was written for SSIS PowerPack, but the concepts and UI stay the same): 
https://zappysys.com/blog/rest-api-authentication-with-oauth-2-0-using-ssis/

Import data into Power BI from SQL Query

If you prefer to import data from an SQL query instead of selecting a table name, you can leverage advanced options during the import steps. Here’s a step-by-step guide:

1. After selecting the DSN, proceed to the import steps.
2. Click on the “Advanced Options” to access the SQL Query editor.

By utilizing these advanced options, you gain the flexibility to tailor your import process by specifying custom SQL queries to retrieve the exact data you need.

Import data into Power BI using SQL Query (ODBC Data source)

Edit Query / Using Parameters in Power BI (Dynamic Query)

let
    vKey=paraAPIKey,
    Source = Odbc.Query(
"dsn=ZS - OData Customers", 
"SELECT * FROM $ WITH (SRC='http://httpbin.org/post'," 
& "METHOD='POST'," 
& "HEADER='Content-Type:application/json'," 
& "BODY=@'{""CallerId"":1111, ""ApiKey"":""" & vKey & """}')")
in
    Source

Edit Query Settings after Import

There will be a time you need to change initial Query after dataset import in Power BI. Not to worry, just follow these steps to edit your SQL.

Edit Power BI Data Source SQL after REST / SOAP Web Service Import

Performance Tips for REST API / XML SOAP Calls

SELECT * FROM value
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json&$filter=Country eq ''USA'''
	,DataFormat='Odata'
)

-- Slow query - Client-side filtering
SELECT * FROM value
WHERE Country ='USA'
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json'
	,DataFormat='Odata'
)

SELECT * FROM $
WITH 
(  SRC='https://myhost.com/some-api'
  ,CachingMode='All'  --cache metadata and data rows both
  ,CacheStorage='File' --or Memory
  ,CacheFileLocation='c:\temp\myquery.cache'
  ,CacheEntryTtl=300 --cache for 300 seconds
)

SELECT * FROM $ 
LIMIT 10 --//add this just to test how fast you can get 10 rows
WITH(
  Filter='$.LargeArray[*]--FAST' --//Adding --FAST option turn on STREAM mode (large files)
 ,SRC='https://zappysys.com/downloads/files/test/large_file_100k_largearray_prop.json.gz'
 --,SRC='c:\data\large_file.json.gz'
 ,IncludeParentColumns='False'  --//This Must be OFF for STREAM mode (read very large files)
 ,FileCompressionType='GZip' --Zip or None (Zip format only available for Local files)
)

Conclusion

In this article, we guide you through the process of configuring the ZappySys Power Pack, focusing on the ZappySys driver for JSON. This driver is instrumental in extracting data from REST APIs or JSON files. Follow these steps to seamlessly add the extracted data to Power BI and generate insightful reports.

Additionally, we’ll walk you through the configuration of the Power BI gateway, ensuring that the data is refreshed on a specified schedule for up-to-date and accurate reporting. To try this configuration yourself, download ODBC PowerPack and embark on a journey of efficient data extraction and reporting.

References

• What is Power BI?
• Keep your dashboards and reports up-to-date with your on-premises data sources
• Power BI – Read REST API / JSON File / XML File / SOAP (Pagination, OAuth, OData)

The post How to import REST API in Power BI (Load JSON / SOAP XML) appeared first on ZappySys Blog.

In this article, you will learn how to convert JSON to CSV File using SSIS. Sometimes you need to convert JSON file to a CSV, which resides in your local disk or somewhere on the Internet, be it a simple download link or a more sophisticated API call. Using ZappySys SSIS PowerPack together with SSIS will make it easy to happen.

We will be using this ZappySys SSIS PowerPack component to make things work:

JSON Source (REST API or File)

Prerequisites

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed.

Step-by-Step – Export JSON to CSV File with SSIS

1. From SSIS Toolbox drag and drop Data Flow into the Control Flow.
2. Then drag and drop JSON Source (REST API or File) into the Data Flow.
3. Open JSON Source component for modifications.
4. Load JSON directly, from variable or URL:
   
5. Use JSONPath Expression in Filter section to select the data you need. We will use

   $.store.book[*]

    to extract all books from JSON array. Check Understanding JSONPath Expression with Examples article to see more examples how this filter works.
6. Close the window.
7. Drag and drop Flat File Destination into Data Flow and connect it to JSON Source:
   
8. Open Flat File Destination for configuration, configure it and map JSON Source columns with Flat File Destination columns.
9. You are done! JSON will be exported to a CSV (download a sample package).

Using API as JSON source

If your JSON happens to reside in a response of an API call which uses OAuth authentication, read this article how to configure JSON Source OAuth connection:

How to authenticate to an API with OAuth 2.0 using SSIS

Exporting multiple JSON arrays to a CSV

There may be a need to export two or more JSON arrays to several CSVs. In that case, you might be interested in this article:

Reading JSON Arrays from file / REST API

Transforming JSON array

Another use case you may encounter is when row values are placed in a JSON array. And then you desperately want to transform that array so that you could see those values in separate columns:

Convert JSON to CSV File: JSON array appears as one column

This would require to make a Simple Transformation on source JSON:

Convert JSON to CSV File : Transforming JSON array values into columns

We also support transforming other formats of JSON arrays. Check how to transform them all in the article Parse multi-dimensional JSON array in SSIS.

Common Errors

Truncation related error

The most common error you may face when you run an SSIS package is truncation error. During the design time only 300 rows are scanned from a source (a file or a REST API call response) to detect datatypes but at runtime, it is likely you will retrieve far more records. So it is possible that you will get longer strings than initially expected. For detailed instructions on how to fix common metadata related errors read an article "How to handle SSIS errors (truncation, metadata issues)".

Authentication related error

Another frequent error you may get is an authentication error, which happens when you deploy/copy a package to another machine and run it there. Check the paragraph below to see why it happens and how to solve this problem.

Deployment to Production

In SSIS package sensitive data such as tokens and passwords are by default encrypted by SSIS with your Windows account which you use to create a package. So SSIS will fail to decrypt tokens/passwords when you run it from another machine using another Windows account. To circumvent this when you are creating an SSIS package which uses authentication components (e.g. an OAuth Connection Manager or an HTTP Connection Manager with credentials, etc.), consider using parameters/variables to pass tokens/passwords. In this way, you won’t face authentication related errors when a package is deployed to a production server.

Check our article on how to configure packages with sensitive data on your production or development server.

Download a sample package

Be sure to download a sample SQL Server 2012 SSIS package in case you want to try it right away.

Conclusion

In this article we learned how to convert JSON to a CSV using SSIS PowerPack JSON Source in a very simple way. We also touched topics that delve into more complicated scenarios such as grabbing JSON from an API, exporting multiple JSON arrays and transforming them into columns. Check our Blog articles on JSON Source to find out what JSON Source is capable of more.

The post How to convert JSON to CSV File using SSIS appeared first on ZappySys Blog.

In this blog post you will see how easy it is to load large amount of data from SQL Server to Amazon S3 Storage. For demo purpose we will use SQL Server as relational source but you can use same steps for any database engine such as Oracle, MySQL, DB2. In this post we will use Export CSV Task and Amazon S3 Storage Task  to achieve desired integration with Amazon S3 with drag and drop approach. You can also export JSON or XML data to Amazon S3 using same techniques (Use Export JSON Task  or Export XML Task ).

Our goal is to achieve following things

• Extract large amount of data from SQL Server Table or Query and export to CSV files
• Generate CSV files in compressed format (*.gz) to speedup upload and save data transfer cost to S3
• Split CSV files by row count
• Upload data to Amazon S3 using highly parallel manner for maximum speed

There are three different ways you can achieve data export to Amazon S3 using SSIS.

1. Method-1 (Fastest): Use two step process (First export SQL Server data to local files using Export Task and then upload files to S3 using  Amazon S3 Storage Task  )
2. Method-2 (Slower): Use Export Task with Amazon S3 Connection as Target rather than save to Local files.
3. Method-3 (Slower): Use Data flow components like Amazon S3 Destination for CSV  (for JSON / XML  use Method1 or Method2)

Each method has its own advantage / disadvantage. If you prefer to upload / compress / split large amount of data then we recommend Method#1 (Two steps). If you have not very huge dataset then you can use Method#2 or Method#3. For Last method you can only use CSV export option (we don’t have JSON/ XML Destination for Amazon S3 yet – we may add in future)

Screenshot of SSIS Package

Extract SQL Server Data to CSV files in SSIS (Bulk export) and Split / GZip Compress / upload files to Amazon S3 (AWS Cloud)

Method-1 : Upload SQL data to Amazon S3 in Two steps

In this section we will see first method (recommended) to upload SQL data to Amazon S3. This is the fastest approach if you have lots of data to upload.  In this approach we first create CSV files from SQL Server data on local disk using SSIS Export CSV Task. After that in second step we upload all files to Amazon S3 using SSIS Amazon Storage Task.

Step-1: Configure Source Connection in Export CSV Task

To extract data from SQL Server you can use Export CSV Task. It has many options which makes it possible to split large amount of data into multiple files. You can specify single table or multiple tables as your data source.

For multiple table use vertical bar. e.g.  dbo.Customers|dbo.Products|dbo.Orders. When you export this it will create 3 files ( dbo.Customers.csv , dbo.Products.csv, dbo.Orders.csv )

Steps:

1. Drag ZS Export CSV Task from Toolbox
2. Double click task to configure
3. From connection drop down select New connection option (OLEDB or ADO.net)
4. Once connection is configured for Source database specify SQL Query to extract data as below
   

   Export SQL Server Table or Query as CSV file (Bulk export in SSIS)

5. Now go to target tab. Here you can specify full path for file. e.g. c:\ssis\temp\s3dump\cust.csv

Step-2: Compress CSV Files in SSIS ( GZIP format – *.gz )

Above steps will export file as CSV format without splitting or compression. But to compress file once exported you can go to Target tab of Export CSV Task and check [Compress file to *.gz format] option.

Compress exported SQL Server data files to GZip ( *.gz) in SSIS Export CSV Task

Step-3: Split CSV files by row count or data size in SSIS

Now lets look at how to split exported CSV files into multiple files so we can upload many files in parallel. Goto Split Options and check [Enable Split by Size/Rows]

Using SSIS Split Exported CSV files (Split by row count or size)

Step-4: Upload CSV files to Amazon S3 – Using multi threaded option

Now final thing is use Amzon S3 Task to upload files to S3.

Things to remember

Sometimes times due to high network activity you may get timeout errors during upload. In that case you can adjust few settings described here. Also try to reduce total parallel threads on S3 Connection see that helps.

Steps:

1. Drag ZS Amazon Storage Task from SSIS toolbox
2. Double click Amazon Storage Task to configure it
3. Specify Action = UploadFilesToAmazon
4. Specify Source file path (or pattern) e.g. c:\SSIS\temp\s3dump\*.*
5. Now in the Target connection dropdown click [New]
6. When Connection UI opens select Service Type = S3
7. Enter your Access Key, Secret Key and Region (Leave all other parameters default if you not sure)
8. Click Test and close connection UI
9. On the Target path on S3 Storage Task enter your bucket and folder path where you want to upload local files. For example your bucket name is bw-east-1 and folder is sqldata then enter as below
   bw-east-1/sqldata/
10. Click ok and Run package to test full package

Upload local files to Amazon S3 using SSIS AWS Storage Task

Method-2 : Upload SQL data to Amazon S3 without local stage (One step)

Now let’s change previous approach little bit to send SQL server data directly to Amazon S3 without any Landing area on local disk.  Export CSV Task , Export JSON Task  and Export XML Task all of them supports Amazon S3 / Azure Blob and Secure FTP (SFTP) connection as target (Only available in Pro Edition). We will use this feature in following section.

This approach helps to avoid any local disk need and it may be useful for security reason for some users. However drawback of this approach is, it wont use parallel threads to upload large amount of data like previous method.

Following change will be needed on Export task to upload SQL data directly to S3 / FTP or Azure storage.

Export SQL data to multiple files to Amazon S3, Azure, Secure FTP (SFTP) in Stream Mode using SSIS. Configure Compress GZip, Overwrite, Split Options

Method-3 : Using Amazon S3 destination – Generate Amazon S3 file from any source

Now let’s look at third approach to save data from any SSIS Source to Amazon S3 file. Advantage of this approach is you are not limited to few source options provided by Export CSV Task. If you have complex data transformation needed in Data Flow before sending data to S3 then use this approach.  We will use Amazon S3 Destination for CSV as below

1. Drag SSIS Data flow task from toolbox
2. Create necessary source connection (e.g. OLEDB connection)
3. Create Amazon S3 Connection (Right click in Connection Managers panel in bottom and click New connection and select ZS-AMAZON-STORAGE type )
4. Once connection managers are created Go to data flow designer and Drag OLEDB Source
5. Configure OLEDB Source to read desired data from source system (e.g. SQL Server / Oracle)
6. Once source is configured drag ZS Amazon S3 CSV File Destination from SSIS toolbox
7. Double click S3 Destination and configure as below
   1. On Connection Managers tab select S3 Connection (We created in earlier section).
   2. Properties tab configure like below screenshot
   3. On Input Columns tab select desired column you like to write in the target file. Your name from upstream will be taken as is for target file. So make sure to name upstream columns correctly.
   4. Click OK to save UI
8. Execute package and check your S3 Bucket to see files got created.

Loading SQL Server data into S3 Bucket Files (Split, Compress Gzip Options) – SSIS Amazon S3 CSV File Destination

Conclusion

In this post you have seen how easy it is to upload / archive your SQL Server data (or any other RDBMS data) to Amazon S3 Storage in few clicks. Try SSIS PowerPack for free and find out yourself how easy it is to integrate SQL Server and Amazon S3 using SSIS.

The post Loading data from SQL Server to Amazon S3 in SSIS (Split Files, GZip) appeared first on ZappySys Blog.

SSIS comes with out-of-the-box support for reading and writing to Excel. However, it’s very restrictive if you want to make things dynamic, as any metadata inside DataFlow cannot be changed at runtime. Here are a few problems with using the native Excel Source or Destination.

• Metadata cannot be changed at runtime.
• Datatype guessing nightmare – only the first few rows are scanned, and the wrong datatype is selected.
• Cannot export multiple tables to Excel.
• Cannot split data into multiple tabs based on specific grouping criteria
• Cannot specify the location where data has to be written

SSIS export to Excel dynamically using Excel Export Task

If you want to generate Excel dynamically without any of the above hassle, then you can check the SSIS Excel Export Task. It comes with many advantages not found in the native Microsoft SSIS Excel Destination

Here is the list of a few features for the SSIS Excel Export Task

• Export multiple tables/views to Excel files (e.g., Sales% or do SalesJan|SalesFeb).
• Support for all major office versions (e.g, Office 2003, 2007, 2013+).
• Export SQL query output to an Excel file.
• A completely dynamic approach without limitations on DataFlow Strict Metadata.
• Option to export data to a specific starting location in an Excel Sheet (e.g, A5 will start writing to the  5th row, 1st column).
• Support for a password-protected Excel file.
• Option to clear a range of cells before writing data to an existing Excel workbook (e.g., A1:D5000 will clear A to D columns from 1st to 5000 rows).
• Automatically Split exported Excel data into multiple files by Size or Number of records.
• Automatically Split exported Excel data into multiple files or tabs when Split By Column (e.g., SplitBy=Country will create a new file for each country). If the SplitOnTab option is specified, then data is written to a new Excel tab rather than a file for each new group.
• Support for Compression and Append.
• Fully managed means it works out of the box on both 32-bit and 64-bit systems without any changes (Note that Native SSIS Excel functionality is 32-bit only).
• Support for SQL Server 2005, 2008, 2012, 2014 (32-bit and 64-bit).

Step-By-Step

SSIS Export to Excel File – Generate Excel files for selected tables/views

SSIS Export to Excel File – Generate Excel files for selected tables/views

SSIS Export to Excel File – Split Excel Data into Multiple Files

SSIS Export to Excel File – Split Excel Data into Multiple sheets

SSIS Export to Excel File – Date Formatting, Encoding Options

SSIS Export to Excel File – Execution Log

Conclusion

In this post, we have seen how Excel generation can be a headache when you have dynamic metadata. You can use the SSIS Excel Export Task to solve many of these challenges.

The post SSIS export to excel dynamically (supports multiple tables) appeared first on ZappySys Blog.