In this blog post you will see how easy it is to load large amount of data from SQL Server to FTP/SFTP. SSIS Data Load – SQL Server to FTP/SFTP (Split Files, GZip).

For demo purpose we will use SQL Server as relational source but you can use same steps for any database engine such as Oracle, MySQL, DB2. In this post we will use Export CSV Task and ZS Secure FTP Task (SFTP, FTP, FTPS) to achieve desired integration with FTP/SFTP/FTPS with drag and drop approach. You can also export data to JSON or XML or Excel file to FTP/SFTP/FTPS using same techniques (Use Export JSON Task  or Export XML Task or Export Excel Task ).

Our goal is to achieve following things

• Extract large amount of data from SQL Server Table or Query and export to CSV files
• Generate CSV files in compressed format (*.gz) to speedup upload
• Split CSV files by row count
• Upload data to FTP/SFTP/FTPS using highly parallel manner for maximum speed

There are three different ways you can achieve data export to Secure FTP using SSIS.

1. Method-1 (Fastest): Use two step process (First export SQL Server data to local files using Export Task and then upload files to Secure FTP using  ZS Secure FTP Task (SFTP, FTP, FTPS) )
2. Method-2 (Slower): Use Export Task with Secure FTP Connection as Target rather than save to Local files.
3. Method-3 (Slower): Use Data flow components like Secure FTP CSV File Destination  (for JSON / XML / EXCEL   use Method1 or Method2)

Each method has its own advantage / disadvantage. If you prefer to upload / compress / split large amount of data then we recommend Method#1 (Two steps). If you have not very huge dataset then you can use Method#2 or Method#3. For Last method you can only use CSV export option (we don’t have JSON/ XML / EXCEL Destination for Secure FTP yet – we may add in future)

Screenshot of SSIS Package

Extract SQL Server Data to CSV files in SSIS (Bulk export) Split / GZip Compress / upload files to FTP/SFTP/FTPS

Method-1 : Upload SQL data to Secure FTP in Two steps

In this section we will see first method (recommended) to upload SQL data to Secure FTP. This is the fastest approach if you have lots of data to upload.  In this approach we first create CSV files from SQL Server data on local disk using SSIS Export CSV Task. After that in second step we upload all files to Secure FTP Storage using ZS Secure FTP Task (SFTP, FTP, FTPS).

Step-1: Configure Source Connection in Export CSV Task

To extract data from SQL Server you can use Export CSV Task. It has many options which makes it possible to split large amount of data into multiple files. You can specify single table or multiple tables as your data source.

For multiple table use vertical bar. e.g.  dbo.Customers|dbo.Products|dbo.Orders. When you export this it will create 3 files ( dbo.Customers.csv , dbo.Products.csv, dbo.Orders.csv )

Steps:

1. Drag ZS Export CSV Task from Toolbox
2. Double click task to configure
3. From connection drop down select New connection option (OLEDB or ADO.net)
4. Once connection is configured for Source database specify SQL Query to extract data as below
   

   Export SQL Server Table or Query as CSV file (Bulk export in SSIS)

5. Now go to target tab. Here you can specify full path for file. e.g. c:\ssis\temp\cust.csv

Step-2: Compress CSV Files in SSIS ( GZIP format – *.gz )

Above steps will export file as CSV format without splitting or compression. But to compress file once exported you can go to Target tab of Export CSV Task and check [Compress file to *.gz format] option.

Compress exported SQL Server data files to GZip ( *.gz) in SSIS Export CSV Task

Step-3: Split CSV files by row count or data size in SSIS

Now lets look at how to split exported CSV files into multiple files so we can upload many files in parallel. Goto Split Options and check [Enable Split by Size/Rows]

Using SSIS Split Exported CSV files (Split by row count or size)

Step-4: Upload CSV files to Secure FTP – Using multi threaded option

Now final thing is use ZS Secure FTP Task (SFTP, FTP, FTPS) to upload files to Secure FTP.

Steps:

1. Drag ZS Secure FTP Task (SFTP, FTP, FTPS) from SSIS toolbox
2. Double click ZS Secure FTP Task (SFTP, FTP, FTPS) to configure it
3. Specify Action = Upload Files To FTP Server
4. Specify Source file path (or pattern) e.g. c:\SSIS\temp\*.*
5. Now in the Target connection dropdown click [New]
6. When Connection UI opens Enter your Account, Secret Key (Leave all other parameters default if you not sure)
7. Click Test and close connection UI
8. On the Target path on ZS Secure FTP Task (SFTP, FTP, FTPS) enter your bucket and folder path where you want to upload local files. (For example: /datafolder/sqldata/)
9. Click ok and Run package to test full package

Method-2 : Upload SQL data to Secure FTP without local stage (One step)

Now let’s change previous approach little bit to send SQL server data directly to FTP/SFTP/FTPS without any Landing area on local disk.  Export CSV Task , Export JSON Task , Export Excel Task and Export XML Task all of them supports Azure Blob / Amazon S3 and Secure FTP (SFTP) connection as target (Only available in Pro Edition). We will use this feature in following section.

This approach helps to avoid any local disk need and it may be useful for security reason for some users. However drawback of this approach is, it wont use parallel threads to upload large amount of data like previous method.

Following change will be needed on Export task to upload SQL data directly to Azure / FTP or Amazon storage.

Export SQL data to multiple files to Amazon S3, Azure Blob, Secure FTP (SFTP) in Stream Mode. Compress GZip, Overwrite, Split Options

Method-3 : Using Secure FTP CSV File Destination – Generate Secure FTP file from any source

Now let’s look at third approach to save data from any SSIS Source to FTP/SFTP/FTPS file. Advantage of this approach is you are not limited to few source options provided by Export CSV Task. If you have complex data transformation needed in Data Flow before sending data to FTP/SFTP/FTPS then use this approach.  We will use Secure FTP CSV File Destination as below

1. Drag SSIS Data flow task from toolbox
2. Create necessary source connection (e.g. OLEDB connection)
3. Create Secure FTP Connection (Right click in Connection Managers panel in bottom and click New connection and select ZS-SFTP type )
4. Once connection managers are created Go to data flow designer and Drag OLEDB Source
5. Configure OLEDB Source to read desired data from source system (e.g. SQL Server / Oracle)
6. Once source is configured drag ZS Secure FTP CSV File Destination from SSIS toolbox
7. Double click Secure FTP CSV File Destination and configure as below
   1. On Connection Managers tab select Secure FTP Connection (We created in earlier section).
   2. Properties tab configure like below screenshot
   3. On Input Columns tab select desired column you like to write in the target file. Your name from upstream will be taken as is for target file. So make sure to name upstream columns correctly.
   4. Click OK to save UI
8. Execute package and check your Secure FTP to see files got created.
   

   Loading SQL Server data into Secure FTP Files (Split, Compress Gzip Options) – SSIS Secure Ftp CSV File Destination

Conclusion

In this post you have seen how easy it is to upload / archive your SQL Server data (or any other RDBMS data) to FTP/SFTP/FTPS in few clicks. Try SSIS PowerPack for free and find out yourself how easy it is to integrate SQL Server and Secure FTP using SSIS.

The post SSIS Data Load – SQL Server to FTP/SFTP (Split Files, GZip) appeared first on ZappySys Blog.

SQL Server 2012 and all previous versions lacking native JSON support, same theory is true for their ETL Platform SSIS. If you are one of them who wants to export JSON from SQL Server or any other RDBMS like MySQL, Oracle then you can easily output complex JSON using any of the following components.

Five ways of creating JSON inside SSIS

ZappySys toolset provides multiple ways creating JSON inside SSIS. You can choose method suitable for your scenario. Each method has pros and cons so choose it as per your need.

JSON File Destination

SSIS JSON Destination can be used to generate simple or complex JSON files out of relational data source such as SQL Server, Oracle, MySQL. In few clicks you can create desired JSON Shape from single or multiple tables (Parent/Child Nested Documents).

PROS: Multiple data flow components can be attached. Which means unlike Export JSON Task you are not limited to OLEDB or ODBC Sources only.

CONS: When you remove upstream component it may break Dataset mappings. Also server side Lookup Option is not available. File Splitting Options not available.

Here are some articles for use case of this component.

JSON File Destination – Generate JSON in SSIS from multiple inputs

JSON Generator Transform

SSIS JSON Generator Transform can be used to generate single or multiple JSON documents from any type of data sources (e.g. SQL Server, MySQL, Flat File, Excel) inside data flow task. It takes multiple inputs and outputs JSON String Output Column which can be used to feed JSON to other downstream system.

PROS: Multiple data flow components can be attached. Which means unlike Export JSON Task you are not limited to OLEDB or ODBC Sources only.

CONS: When you remove upstream component it may break Dataset mappings. Also server side Lookup Option is not available. File Splitting Options not available. Output is JSON String Column.

Here are some articles for use case of this transform.

Preview JSON – Generate for BigQuery Table Insert

Export JSON Task

SSIS Export JSON File Task can be used to generate simple or complex JSON files out of relational data source such as SQL Server, Oracle, MySQL. It also support exporting directly to cloud (e.g. Azure Blob Storage, Amazon AWS S3, Secure FTP). In few clicks you can create desired JSON Shape from single or multiple tables (Parent/Child Nested Documents)

PROS: Simple to use with flexible options to export to file, variable, AWS S3, Azure Blob, FTP / SFTP location. Allows to split records. File Splitting Options available (Split By Size  / Row Count).

CONS: You can only use limited data sources as Dataset input (e.g. OLEDB / ODBC or SQL Server).

Here are some articles for use case of this task.

Template Transform

Template Transform can be used to create dynamic strings inside data flow. You can use upstream columns as placeholders or use SSIS variables anywhere in the template to create desired text (e.g. XML / JSON document).

PROS: Very Simple to use.

CONS: Cannot use multiple inputs. It may not be used to create Complex JSON and options like do not output NULL attributes is not possible.

Here are some articles for use case of this transform.

Create JSON for API Request (POST) using SSIS Template Transform

Logging Task (Control Flow Approach)

 Just like Template Transform you can use ZS Logging Task to create JSON string from Template and Variable Placeholders.

Here is how to use Logging Task to save JSON into a variable.

1. Select Variable which you like to Set on Logging Task
2. Enter your Template JSON String which you like to produce (see example below). Notice the use of yyyy-MM-dd format specifier and also JSONENCODE function.
   

   {
    ActionId: {{User::vActionId}},
    CreatedOn: "{{User::vOrderDate,yyyy-MM-ddTHH:mm:ss}}",
    Notes: "{{User::vNotes,JSONENCODE}}"
   }

3. Select Message Type = None
4. Uncheck Append, and Prefix options like below
   

   Creating JSON / XML Document using SSIS Logging Task

Here are some articles for use case of this task.

Create JSON using SSIS Export JSON Task

Now let’s look at Export JSON Task form example to create JSON. Steps listed in below sections are very similar to JSON File Destination or JSON Generator Transform.

ZappySys offers innovative Layout Editor for JSON / XML creation from multiple sources. You can use Layout Editor to produce desired JSON Layout (Layout is stored in XML format behind the scene) and then in later step you will see how to make it dynamic so values are changed at runtime.

You can read layout as direct value, read from variable or read from file from disk.

Select source connection for data

First lets create new OLEDB or ADO.net connection from where we will read data. In our case we reading from Northwind Database.

SSIS JSON Export Task – Select Source Database Connection

Create Datasets

Now add datasets which will be used to produce JSON File. One dataset will be always Root Dataset (Example: If your Root information is about customer then select customer table or query as your Main dataset. All other sections will be child sections)

Add SQL Dataset – Export JSON from SQL Query Option

Add Output Columns for Main JSON Document

Now its time to add columns for our output. If * is added by default then remove. * means output all columns. When you add columns first make sure you select correct parent node under which you want to add columns. In our case “Mappings” is parent node.

Add Output Columns for JSON Document – Column Selection Screen

Add JSON Array / JOIN Conditions for related document

Now let’s look at how to add JSON Array. For example Customer may have one or more Orders so we need to add node called Orders which may Produce JSON fragment like below

Orders : [ {...} , {...} ] 

Once customer related columns added, its time to add Orders section.

1. This is Array of documents so first highlight “Mappings” node
2. Click “Add Document Array” option from toolbar.
3. On Add/Edit Elements UI specify name for JSON Attribute and specify correct relationship to parent dataset. In our case Orders related to Customers using CustomerID key.

Add Child Section – nested JSON document , Define Parent-Child Relationship

Handling JSON Array with dummy root dataset (Multiple Arrays)

So in previous section we saw how to add array and define JOIN condition with parent dataset. Now what if you really have no parent dataset to join with ? For example we need to produce following sample JSON where Order array is related to Customer by common Column CustomerID as JOIN column. But Customer is not bound with any parent column. So how do you define JOIN condition? In such case you have to use below technique called dummy root dataset. (See next section on how to handle only one dataset using Single Dataset Array Mode)

Sample JSON (Dummy root pattern)

{
    "Customers": [
      {
        "CustomerID": "ALFKI",
        "CompanyName": "Alfreds Futterkiste",
        "Orders": [
          {
            "OrderID": 1000,
            "OrderDate": "2015-12-01"
          },
          {
            "OrderID": 1001,
            "OrderDate": "2015-12-01"
          }
        ]
      },

      {
        "CustomerID": "ANATR",
        "CompanyName": "Ana Trujillo Emparedados y helados",
        "Orders": [
          {
            "OrderID": 1002,
            "OrderDate": "2015-12-01"
          },
          {
            "OrderID": 1003,
            "OrderDate": "2015-12-01"
          }
        ]
      }

    ]
  }
}

Since root level always needs dataset, we must provide dummy dataset (only one row) even though there is no need for Parent Dataset JOIN in Customer Array. Typically you can write query like below to product dummy row in SQL Server or other database using OLEDB Source or Use Dummy Data Source with Max Rows=1 setting to produce one fake record.

SELECT 1 as Col1

Using Dummy DataSource as Input

For generating JSON which needs dummy root. Perform the following steps

1. Drag Dummy Data Source from SSIS Toolbox. Change How many rows to 1. Click OK
2. Now drag two more OLEDB sources and Configure (e.g. Customers and Orders)
3. Now drag ZS JSON FIle Destination component or ZS JSON Generator
4. Now connect Dummy datasource then Customers and then Orders
5. Double click JSON component to configure layout. You may notice that Dummy Input is set as Root by default.
6. You can now select Mappings node and click on Add Array Documents icon. Configure like below. Notice we have not selected any JOIN criteria because its joined with Dummy Root.
   

   Add array to join with dummy dataset

7. Now select newly added node and Click Add column icon to add CustomerID, CompanyName column under Customers array
8. Now select Customers array node and click Array Documents icon to Add Orders array.
9. Once Orders Array ia added you can Select Orders node and Insert new element to add OrderID and OrderDate columns
10. Finally it will look like below
    

    JSON File Destination – Generate JSON in SSIS from multiple inputs

Single dataset used as array

Now lets look at one more pattern just like described in the previous section but now we have just one nested level (Only customers and not using Orders). If you have such case then choose Single dataset pattern option in the output mode. On Export Task you may see as check box rather than dropdown. To produced desired JSON (see blow) perform these steps.

1. Select Output mode as Single Dataset Array
2. In your Layout builder click on Add Document Array Icon and name it “Customers”.
3. After that highlight newly added node > Right-Click > Add Elements under that node
4. That’s it you will now see preview like below.

{
    "Customers": [
      {
        "CustomerID": "ALFKI",
        "CompanyName": "Alfreds Futterkiste"
      },

      {
        "CustomerID": "ANATR",
        "CompanyName": "Ana Trujillo Emparedados y helados"
      }
    ]
  }
}

How to export JSON in 2D Array format

If you have to export JSON as 2D array format then you can change following option.

Here are examples of output for each option.
Select say you have source data with 2 columns id and name.

Default

[{id:1,name:"AA"}, {id:2,name:"BB"}]

Multicontent

{id:1,name:"AA"}{id:2,name:"BB"}

ArrayDocs

[{id:1,name:"AA"}, {id:2,name:"BB"}]

Array2D

[[1,"AA"],[2,"BB"]]

Array2DWithHeader

[["id","name"],[1,"AA"],[2,"BB"]]

ArrayLines

[1,"AA"][2,"BB"]

ArrayLinesWithHeader

["id","name"][1,"AA"][2,"BB"]

Preview JSON output using live preview

You can enable / disable Live preview of JSON layout as below.

JSON Output preview in SSIS Export JSON File Task

Make things dynamic (Using SSIS Variable for SQL or Layout)

There will be a time when you have use dynamic values in your JSON generation. You may have to Alias something from variable or you have to supply Variable in your SQL used for Dataset. Not to worry. ZappySys Supports multiple ways to make it dynamic. You can use SSIS Variables following 3 ways to make Layout XML dynamic.

1. Use SSIS variable inside Dataset SQL (Only for ZS Export JSON Task)
2. Anywhere inside Layout XML (Click Edit Code option)
3. Load Layout From Variable (Indirect Method)

Using Variable inside Dataset SQL

Here is how you can use Variable inside dataset SQL for ZS Export JSON Task

SSIS Export JSON Task – Using Variable for Dynamic SQL

Using Variable inside Layout XML

Here is how you can insert SSIS Variable anywhere in Layout XML. In below Example we are choosing Alias for one column from some SSIS Variable.

SSIS JSON File Destination – Using SSIS Variable for Dynamic Layout

Load Layout From Variable

If you have some need to load Layout XML from File or SSIS Variable. You can use Load Layout from Variable Option.

For example in below screenshot we are creating layout first and ave to some variable. Or you can load from external file using ZS Advanced File System Task.  Connect Logging task to Export JSON Task and you ready to produce JSON.

Set SSIS Variable value (multi-line content using placeholder)

Here is how to load Layout From any SSIS Variable. You can also load Layout from XML file or Variable which contains layout File Path.

JSON Layout from SSIS Variable

Other JSON Output Options

Execute package and verify generated JSON it should look like below (Date format may be different in your case if you have set different on JSON options tab). For clarity we have formatted JSON below but in your output it may be compact version so reduce file size.

Indent JSON Output

You can check Indent option to make JSON look pretty like below.

{
  "CustomerID": "ALFKI",
  "CompanyName": "Alfreds Futterkiste",
  "ContactName": null,
  "Orders": [
    {
      "OrderID": 10643,
      "OrderDate": "1997-08-25T00:00:00"
    },
    {
      "OrderID": 10692,
      "OrderDate": "1997-10-03T00:00:00"
    },
    {
      "OrderID": 10702,
      "OrderDate": "1997-10-13T00:00:00"
    },
    {
      "OrderID": 10835,
      "OrderDate": "1998-01-15T00:00:00"
    },
    {
      "OrderID": 10952,
      "OrderDate": "1998-03-16T00:00:00"
    },
    {
      "OrderID": 11011,
      "OrderDate": "1998-04-09T00:00:00"
    }
  ]
}

Hide NULL Attribute Option

There will be a time when you want to Hide NULL attribute rather than showing MyAttr: null this is also possible by checking Hide NULL Attribute Option on Options Tab.

Export JSON to Amazon S3, Azure Blob, Secure FTP (SFTP / FTPS)

Export JSON Task support some really good options to save file to other than local file system. If you want to export JSON directly to Amazon / Azure Cloud or FTP Server without saving to local disk then you can use Azure / FTP / Amazon Connection Manager on Target Tab of Export JSON Task. This option is only available for PRO Edition Users.

Export JSON / CSV / XML Files to Azure Blob, Amazon S3, Secure FTP Storage

Conclusion

If you want to output JSON from SQL Server or any other relational DB then its not simple task in SSIS specially when you have nested JSON documents (e.g. Customer->Orders). In this post you have seen how to use SSIS JSON Export Task to generate nested JSON from multiple datasets.

The post 5 Ways to Export JSON from SQL Server using SSIS appeared first on ZappySys Blog.