In our last post (Regex Cheat Sheet) we explained use cases of SSIS Regular Expression Parser Task . Now lets look at some real world usecase. Basically in this article we are going to Extract data from audit log using SSIS. For demo purpose we will use log generated by SSIS Export CSV File Task output. When you export many tables dynamically using this task you want to know how many Row count exported for each Table. This information is logged in SSIS Output Log but no easy way to capture in SSIS variable. So what we will do we will use few tricks to capture that data using Regular Expression usecase.

So let’s get started.

Setup Export CSV Task (Output Multiple Tables to CSV Files Dynamically)

For example purpose we will use Export CSV File Task but you can use this technique for any other Tasks.

1. Drag and Drop ZS Export CSV Task from SSIS Toolbox and configure as below.
   
2.   Configure Export CSV Task to output multiple tables as explained here
3. Now create 2 SSIS User Variables
   1. FilePath (String Type)
   2. RowCount (String Type)

That’s it… now in the next section we will setup event handler to capture output from Log.

Setup EventHandle / Extract Audit Data using Regex

1. Now select task and click Event Handler tab
2. Select Executable Name from the dropdown, Select Event Name OnInformation from Handler Dropdown
3. Click Create Hyper Link to Create Event Handler like below
   

   Create Task Level Event Handler in SSIS

4. Now Drag below 3 tasks and connect like this
   1. Script Task (We will use as dummy start (No configuration needed)
   2. ZS Regular Expression Parser Task
   3. ZS Logging Task (Optional) but we will use to Output Extracted Values
      
5. Connect Dummy Script Task to ZS Regular Expression Parser Task and Right click on Green connected Arrow and Use Expression as below. Click OK to save
6. Configure ZS Regular Expression Parser Task as below
   1. Enter

      {{System::ErrorDescription}}

        as Direct Value
   2. Enter Below two mappings
      1. For RowCount set
         

         Total (\d+) records written to : (.*){{1,1}}

      2. For FilePath set
         

         Total (\d+) records written to : (.*){{1,2}}

7. Here is how it will look like
   

   Extract data from audit log using SSIS Regular Expression Parser Task

8. Thats it… Now connect 2nd step to last Logging Task…. in that Task you can set Text like below just to log extracted data. You can also use Execute SQL Task to log into Database Table
   

   {{User::FilePath}} ===> {{User::RowCount}}

9. Now run Package and you will see audit data we extracted for each file. Once File is exported we capture OnInformation event… and if Message contains substring like “records written to” then we extract data into variable for Row Count and Path then you can save to desired place for auditing purpose

The post Extract audit data from SSIS Execution Log (EventHandler and Regular Expression) appeared first on ZappySys Blog.

In this article, we will see how to read PayPal API data in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call PayPal APIs can be useful for any developer regardless of which programming language or tool set you use. We will also see How to send basic Authorization header along with body in OAuth2.

We will go through the steps to read data from PayPal Transactions and Load into MS SQL Server.

In nutshell, this post will focus on how to make PayPal API Call using SSIS.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of PayPal API. Click here to create API.

What is PayPal?

PayPal is an American company operating a worldwide online payments system that supports online money transfers and serves as an electronic alternative to traditional paper methods like checks and money orders. The company operates as a payment processor for online vendors, auction sites, and many other commercial users, for which it charges a fee in exchange for benefits such as one-click transactions and password memory. PayPal’s payment system, also called PayPal, is considered a type of payment rail.

Get Started with PayPal API

If you have need to automate PayPal operations (e.g. read / write / update / delete ) then you can use PayPal REST API. Before you can integrate a PayPal product or solution, you must set up your development environment to get OAuth 2.0 client ID and secret credentials for the sandbox and live environments. You exchange these credentials for an access token that authorizes your REST API calls. To test your web and mobile apps, you create sandbox accounts.

Get PayPal credentials

To generate OAuth 2.0 credentials for the sandbox and live environments:

1. Log into Dashboard and type your PayPal personal or business account email and password.
2. In the REST API apps section, click Create App. The purpose of this app is to generate your credentials.
3. Type a name for your app and click Create App. The page shows your sandbox app information, which includes your credentials.
   

   Note: To show your live app information, toggle to Live.

4. Copy and save the client ID and secret for your sandbox app.
   

   PayPal Client ID and Secret

5. Review your app details and save your app.

Read PayPal API Data using SSIS JSON Source

Let’s start with an example. We use SSIS JSON Source component to make the call to PayPal API, we will read PayPal data and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

In REST API calls, include the URL to the API service for the environment:

• Sandbox: https://api.sandbox.paypal.com
• Live: https://api.paypal.com

Create Basic OAuth Client Credentials Grant (Configure ZS-OAuth Connection)

To get API response data using SOAP / REST API call, you need to get an API Token from your token request URL. Follow the steps mentioned below to get API Token:

1. Now, Right Click on Connection Managers Window, and one Context menu appears for creating a new connection manager.
   

   Connection Manager Window: Create New Connection

2. Select ZS-OAuth Connection Manager from the list of Connection Managers and, double-click on it or click on Add button to add a new Connection Manager. By Default, The created connection manager is of Package-Level. You can set Project Level Connection manager too If you want.
   

   SSIS Connection Mangers List: Select O-AUTH Connection Manager

3. In the SSIS ZS OAuth Connection Manager, Select Provider as Custom, select version as OAuth2 and select Grant Type as  Client Credentials Grant. Enter the credentials and access token url and test the connection.
   

   https://api.sandbox.paypal.com/v1/oauth2/token

    
   

   ZS OAuth Connection – Client Credentials Grant

4. Click on OK button to save OAuth Connection configure settings.

That’s it we have successfully configured Connection for PayPal API in SSIS. In the next section, we will see how to use this connection and read various data from PayPal API.

Read PayPal API Data in SSIS

Once we have done creating HTTP Connection Manager we can move forward to read PayPal API data inside Data Flow. So lets Configure SSIS JSON / REST API Source.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.
   

   Drag and Drop JSON Source Component

3. Double click JSON Source and enter the following URL as below to get invoices and select that created OAuth Connection.
   

   https://api.sandbox.paypal.com/v1/invoicing/invoices?page=0&page_size=100&total_count_required=true

   

   SSIS JSON Source

4. Now Go to Pagination tab select mode as Url Parameter and add indicator as

   page

     and Add the increment value same as

   page_size

     value we set in url, there for in our case we set 100. For pagination call please click here for reference to this link.
   

   PayPal Pagination

5. Select the desire Array Filter and click on preview.
   

   PayPal Preview

6. That’s it, you are ready to load PayPal API Data into SQL Server.

Load PayPal API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

After all, we saw you how to extract information from REST API such as PayPal REST API using HTTP Connection and load into SQL Server. We also learned techniques like How to get Dynamic Token using HTTP Connection. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• Help File: JSON Source(REST API or File), ZS Logging Task
• PayPal API
• Dynamic token generation

The post How to read PayPal API data in SSIS appeared first on ZappySys Blog.

In this article, we will see how to read Walmart API data in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call Walmart APIs can be useful for any developer regardless of which programming language or toolset you use.

Are you a Walmart.com Supplier or Marketplace Seller? Then you may need to click here to visit the site that is focused on these profiles.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of Walmart API. Click here to create API.

Components Mentioned in this article

	ZS Logging Task
	JSON Source(REST API or File)

What is Walmart?

Walmart is the world’s largest retailer, and the Walmart Open API provides access to our extensive product catalog, thus enabling digital distribution partners to earn substantial affiliate revenues from customer referrals.

Read Walmart API Data using SSIS JSON Source

Let’s start with an example. We use SSIS JSON Source component to make the call to Walmart API, we will read Walmart data and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

Get Dynamic Token (Configure ZS-HTTP Connection)

To get API response data using SOAP / REST API call, you need to get an API Token from your login request URL. Follow the steps mentioned below to get API Token:

1. Now, Right Click on Connection Managers Window, and one Context menu appears for creating a new connection manager.
   

   Create a New HTTP Connection.

2. Select ZS HTTP Connection Manager from the list of Connection Managers and, double-click on it or click on Add button to add a new Connection Manager. By Default, The created connection manager is of Package-Level. You can set Project Level Connection manager too If you want.
   

   Select ZS-HTTP Connection from SSIS Connection Manager.

3. In the SSIS ZS HTTP Connection Manager, Go to General Tab and Configure HTTP Connection Manager. Enter API URL, Set Credentials Type to Dynamic Token – Two-Step Auth, and configure some basic settings for the same. We used Login URL as just example but you can enter any valid API URL endpoint here. Actual Login URL goes on Dynamic Token Tab (see next step).
   

   https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20

   WM_SEC.ACCESS_TOKEN

   

   ZS HTTP Connection – Configure General Tab

4. Now, In the Dynamic Token Tab do configure like below image. Please click here to refer about Walmart API Token.
   

   https://marketplace.walmartapis.com/v3/token

   grant_type=client_credentials

   Accept: application/json
   Cache-Control: no-cache
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>
   Authorization: Basic <<[$userid$]:[$password$],FUN_BASE64ENC>>

   

   ZS HTTP Connection – Dynamic Token Tab

5. On the Response Settings tab configure as below
   

   Walmart API – Extract Token – Response Settings

   1. Extract Mode = Extract from Body
   2. Expression Type = Json
   3. Response Content Filter =

      $.access_token

6. Click on OK button to save HTTP Connection configure settings.

That’s it we have successfully configured Connection for Walmart API in SSIS. In the next section, we will see how to use this connection and read various data from Walmart API.

Read Walmart API Data in SSIS

Once we have done creating HTTP Connection Manager we can move forward to read Walmart API data inside Data Flow. So lets Configure SSIS JSON / REST API Source.

1. Now, Drag and drop Data Flow Task from SSIS Toolbox in the currently open project and double click it to edit.
   
   

   Drag and Drop Data Flow Task.

2. Double click on the Data Flow task to see the Data Flow designer surface.
3. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.
   

   Drag and Drop JSON Source(REST API or File).

4. Double click JSON Source and enter the following URL as below.
   

   https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20

5. Check Use Credentials and select existing HTTP connection we have created already it.
6. Enter Following HTTP Headers. If you prefer Raw edit .. Just click on Raw Edit (#6 in screenshot) and paste below headers. Make sure to replace YourClientIdGoesHere and YourSecretGoesHere
   

   Content-Type: application/xml
   Accept: application/xml
   Authorization: Basic <<YourClientIdGoesHere:YourSecretGoesHere,FUN_BASE64ENC>>
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>

    
7. Select Array Filter (ItemResponse node) or type $.ItemResponse[*]  as below.
   

   ZS JSON Source – Configure

8. That’s it, you are ready to load Wallmart API Data into SQL Server and you can use more ZappySys Destination components.
9. For pagination call please click here for reference to this link.

Load Walmart API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Walmart API integration using ODBC Driver (Power BI, Excel, MS Access)

So far we have seen how to access Walmart API using SSIS. However there will be a time when you like to access API data inside other Apps using ODBC Driver. ZappySys Provides JSON/XML/CSV drivers for virtually any API including Walmart API. Here is how to access Walmart API using XML ODBC Driver.

1. Type “ODBC” in Start menu. Open ODBC Data sources (64 bit)
   
2. Click Add on User DSN tab (if you need to use DSN under different Account …i.e. Service Account then click on System DSN Tab)
3. Select ZappySys XML Driver
4. When New DSN UI shows up, Change DSN name to anything you like (e.g. WalmartAPI)
5. Click on Load ConnectionString button and enter below connection. Make sure to replace few things
   First ######YourClientId####### with your own clientid
   Second ######YourSecret####### with your own secret
   Third ######Base64OfClientIdAndSecret####### with base64 encoded string of

   clientId:Secret

       (both separated with colon). You can use any site like this one to generate base64 string of your clientid and secret (seperated by “:”)
   

   DRIVER={ZappySys XML Driver};
   ElementsToTreatAsArray='ns2:ItemResponse';
   DataPath='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20';
   DataConnectionType=HTTP;
   AuthScheme='{none}';
   Url='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20';
   TokenUrl='https://marketplace.walmartapis.com/v3/token';
   TokenRequestData='grant_type=client_credentials';
   TokenRequestMethod='POST';
   TokenResponseContentFilter='$.access_token';
   TokenRequestHeaders='Accept:application/json||Cache-Control:no-cache||WM_SVC.NAME:Walmart Marketplace||WM_QOS.CORRELATION_ID:TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>||Authorization:Basic <<[$userid$]:[$password$],FUN_BASE64ENC>>';
   TokenAuthHeader='WM_SEC.ACCESS_TOKEN';
   TokenResponseContentType=Json;
   UserName='######YourClientId#######';
   CredentialType=TokenDynamic;
   Password='######YourSecret#######';
   Filter='$.ns2:ItemResponses.ns2:ItemResponse[*]';
   RequestMethod='GET';
   RequestHeaders='Content-Type: application/xml || Accept: application/xml || Authorization: Basic ######Base64OfClientIdAndSecret####### || WM_SVC.NAME: Walmart Marketplace || WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>'

   

   ZappySys Driver – Load Connection String

    

6. Now go to preview tab and enter below query and click Run below SQL.

   select * from $

   –OR–
   NOTE: Change YourClientId and YourSecret in below sql before running

   SELECT * FROM $
   WITH(
   Src='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20'
   ,Header='Content-Type: application/xml
   Accept: application/xml
   Authorization: Basic <<YourClientid:YourSecret,FUN_BASE64ENC>>
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>'
   )

    

Walmart API Integration with Other BI Tools (Power BI, Excel, SSRS, MS Access…)

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

How to encrypt Authorization Token as Base64Encode

1. If you want to directly pass Authorization header Token as Base64Encode(clientId:clientSecret) encrypted in the get token call, you can encrypted it using ZS Logging task and used it. Please refer to this link for the same.
2. In visual studio just Drag and Drop ZS Logging Task in the design panel.
   

   Drag and Drop ZS Logging Task

3. Double click on ZS Logging Task to configure it.
4. In the ZS Logging Task configure like the following image.
   

   Logging Task : Base64Enocde

5. That’s all, Save Base64Encoded token for use in HTTP Connection.

Conclusion

After all, we saw you how to extract information from REST API such as Walmart REST API using HTTP Connection and load into SQL Server. We also learned techniques like How to get Dynamic Token using HTTP Connection. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• Help File: JSON Source(REST API or File), ZS Logging Task
• Walmart API: HTTP Connection

The post How to read Walmart API data in SSIS / ODBC appeared first on ZappySys Blog.

In this new article, we will show you how to perform PGP encryption using SSIS (encrypt / decrypt files using public / private key). Our previous article was about SFTP using our SFTP task for SSIS. Now we will show how to encrypt the information.

Requirements

1. First of all, you will need SSDT for Business Intelligence for SSIS projects.
2. Download and install FREE PGP Command line gnupg for Windows.
3. Optionally you can install SSIS PowerPack.

What is PGP / GPG (GnuPG) / OpenPGP ?

PGP stands for Pretty Good Privacy. It was created by Phil Zimmermann. PGP is the most popular email encryption in the world. It is an Encryption that requires a public key, a private key. You can encrypt, decrypt and sign documents, emails. PGP Tools now acquired by Symantec. In sort PGP Tools developed by Phil are not Open Source any more so use other on if you really need FREE one.

GPG (i.e. GnuPG)) stands for GNU Privacy Guard. This software is for multiple platforms like Linux and Windows. In this article, we will focus on Windows because we are using SSIS that is mainly installed in Windows (at least by the moment). This tool is free, so we will focus on this one instead of trying other licensed versions.

OpenPGP is standard around Public / Private Key Cryptography (RFC4880) . Check this page. Both GPG and PGP follow OpenPGP Standard however there will be a case when they are not 100% compatible with each other.

For more information read this link and this link.

Getting started

In this article we will use GPG only which is essentially same features as PGP but GPG is Open Source around OpenPGP Standard and PGP is not open source.

So let’s get started. First of all, we will start with a simple the command line.

GPG command line example

After you install gpg command line we can open it to get started. There are many more examples explained here and here.

1. First of all, check the requirements and open the command line.
2. Secondly, in the command line write gpg –help this will list all the commands available:
   

   gpg help command

3. Also, in order to generate your personal key pair, write the following command:
   

   gpg --gen-key

4. Finally, the command previously provided will ask for a name and email, write an O to press OK and a Passphrase. You will be able to create your key pair with that information.

PGP encryption using SSIS introduction

It is possible to perform PGP encryption using SSIS with the GPG command line which is free. We can use  native execute process task that comes with SSIS. In this article, we will show some commands of gpg.

1. Open a new SSIS Package.
2. Drag Execute Process Task from SSIS Toolbox and double click to configure (see next step).
3. Using SSIS Execute Process Task, we will invoke the gpg commands. There are different ways to invoke the gpg commands. One way is to execute command stored in a .bat file or .cmd file
4. To invoke command from a file create a file called gpg.cmd and type

   gpg --version

      , save file to path like c:\sql\gpg.cmd for this demo.
5. Configure Execute Process Task something like this if you invoke command from batch file:
   

   SSIS invoke gpg

6. Also, note that the windows style is hidden. That will hide the command line when running the package.
7. In addition, the StandardOutputVariable will create an object variable in the output.
8. To verify the output, we will use the ZS Logging task
   

   SSIS output results

9. Click OK to save Execute Process Task UI.
10. Now drag ZS Logging task from toolbox and configure like below. This will Log output captured by gpg command. This step is optional but will help you a lot if you want to know what happened in previous step (e.g. capture error message)
    

    SSIS Logging task

Execute GPG / PGP commands directly in SSIS Execute Process Task (Pass Arguments)

1. In the previous example, we used a cmd file with the commands. Sometimes we need to use SSIS variables and pass information dynamically. In this new example, we will invoke the gpg command line directly and send the arguments in the arguments property like this:

SSIS execute gpg encryption

Generate public key in PGP encryption using SSIS

Now lets look at common task with PGP cryptography which is generate Public / Private Key pair.

1. In order to generate a public key, you need to run the following arguments in SSIS:
   

   SSIS generation of public key

   --armor --output "c:\sql\ssiskey.txt" --export "Name"

2. In addition, the key name is the name of the pair key created and the
   output contains the path of the stored key.

How to encrypt a file in PGP encryption using SSIS

1. First of all, we will encrypt a file named ssiskey.txt:
2. The argument used is the following:
   

   --recipient "Name" --output "c:\sql\ssiskey.txt.gpg" --encrypt "c:\sql\ssiskey.txt"

   

   Encrypt file gpg in SSIS

3. We are creating an encrypted file named ssiskey.txt.pgp

How to decrypt a file in PGP encryption using SSIS

1. In order to start, we will decrypt the file created previously:
   

   Decrypt file using PGP

2. Also, we will use the following argument:
   

   --decrypt-files "c:\sql\ssiskey.txt.gpg"

    

How to import a public key in PGP encryption using SSIS

1. In order to send a message and decrypt it, you need a public key. If you publish a public key, the recipient needs to import the public key. To import a key use the following argument:
   

   gpg --import "c:\sql\publickey.txt".

   You can download a public key here:
   • Public key
2. Secondly, in the Execute Process Task, will look like this:
   

   How to import a public key in SSIS

How to encrypt multiple files in PGP encryption using SSIS

1. If you want to encrypt multiple files, you can use the ZS Advanced File System combined with the Foreach loop container to filter and select the files to encrypt.
2. In addition, the package will look like this:
   

   Send a list of files to the foreach loop

3. Also, in the Advanced File System, we will list the files as ADO.net DataTable:
   

   Store list of files in SSIS

4. In the next step, we will use the Foreach ADO Enumerator in the Foreach Loop container:
   

   Foreach loop from the Advanced Files system

5. Next, we will select the first column (column 0) to get the file path from the list.
   

   Send column 0 that contains the full path

6. Finally, we will use the following expression for the argument property to encrypt all the txt files of the ADO list:
   

   "--recipient \"Daniel Calbimonte\" --output \""+ @[User::filepath]+".gpg\""+" 
   --encrypt \""+@[User::filepath]+"\""

   

   SSIS expression to encrypt multiple files

Troubleshooting PGP encryption using SSIS

1. First of all, if your package works in SSDT or VS and fails in the agent, verify that you are using a proxy account with permissions to run the gpg command line. For more information about SQL Agent proxies, refer to this link.

References

For more information, refer to these links:

• Practical Introduction to GNU Privacy Guard in Windows
• The GNU Privacy Handbook

The post SSIS PGP Encryption / Decryption (Using FREE GPG Tool) appeared first on ZappySys Blog.

In this article, we will show how to send values from an SSIS Variable to an HTML file. We will use the SSIS Advanced File System Task to store the list of system files of a folder into a variable and then we will use the SSIS Logging Task to store the variable in an HTML file.

In the second part, we will read values from an HTML table and export the data into SQL Server. We will use the HTML file created in the first example. We will also use the ZS HTML Table Source to read the data from the HTML table and export to SQL Server using the OLEDB Destination.

We will be using this ZappySys SSIS PowerPack component to make things work:

	SSIS Logging Task (Free)
	SSIS HTML Table Source
	Advanced SSIS File System Task

Finally, we will show how to read data from a SQL Server query and store into an HTML file. We will use the Execute SQL Task combined with the ZS SSIS Logging Task.

Requirements

1. First, you will need SSDT installed.
2. Secondly, ZappySys PowerPack for SSIS.
3. Finally, SQL Server installed.

Getting started

How to save the list of system files into SQL an HTML table

In the first example, we will use our ZappySys Advanced File System to copy a list of the system files in a folder. This list of files will be stored in an object variable and then we will save to a file.

1. In the first place, we will use the Advanced File System Task and select the Get file list ADO.net DataTable action. We will list the files of the c:\sql folder and save the results into a variable:
   

   SSIS list files

2. In addition, we will create a variable of type object named listfiles that will store the files information:
   

   SSIS object variable

3. Also, we will use the ZS Logging Tasks and join with the ZS Advanced File. The ZS Logging Task will be used to store the variable into a file:
   

   SSIS save variable to HTML

4. In addition, it is possible to send the variable values into the Execution Log (SSIS output), to a file, to a MessageBox:
   

   SSIS output variable to log, message box and file variable to read and write data to HTML in SSIS

5. In this example, we will show the values into an HTML file named listfiles.html
   

   SSIS save variable iHTMLhtml

6. After executing the package, you will be able to see the HTML with a list of files, names, size, modification and creation information:
   

   ssis list files

How to copy data from an HTML table to SQL Server

The next example will save data from a file named listfiles.html created in the previous example with an HTML table. The example will export the table to SQL Server. However, with our HTML Table Source, you can export to any destination.

1. In order to start, we will Drag and drop a data flow task and double click it:
   

   SSIS data flow

2. Next, in the Data Flow task, use the HTML Table Source and join to the OLEDB Destination:
   

   SSIS HTML table SQL server

3. Also, in URLs enter the path of the HTML file created in the first example:
   

   SSIS HTML table read

4. In addition, go to the OLEDB Source, specify the SQL Server name, database, and log in to it.
   In the name of table or view press new to create a new table:
   

   CREATE TABLE [dbo.listfiles] (
   [_rowNumber] int,
   [_groupName] nvarchar(255),
   [FullPath] nvarchar(255),
   [Name] nvarchar(255),
   [Size] int,
   [AgeInDays] int,
   [LastEditInDays] int,
   [LastModifiedDate] datetime,
   [CreationDate] datetime,
   [LastModifiedDateUtc] datetime,
   [CreationDateUtc] datetime,
   [Extension] nvarchar(255),
   [IsReadOnly] nvarchar(255),
   [FolderPath] nvarchar(255)
   )

   

   SSIS OLEDB create table

5. Also, go to Mappings page to map all the columns.
   

   SSIS map columns

6. Finally, run the package and you will have all the data exported.

How to export SQL Server results to an HTML in SSIS

The following example will show how to export the results of an SQL query into an HTML file.

1. In order to start, we will use the Execute SQL Task. Specify the option Full result set and create an OLEDB connection specifying the Server Name, database name and the select statement to a table:
   

   SSIS SQL query results

2. Secondly, go to the Result Set page and in the Result Name, set 0 and in Variable name, create a new variable:
   

   ssis sql result to variable

3. Also, create a variable of type object:
   

   SSIS object data type

4. Finally, use the ZS Logging Task to save the results in a File. Specify the Log file path and insert the variable:
   

   SSIS save to HTML file

Conclusion about read and write data to HTML in SSIS

To conclude, in this article, we learned how to export a list of files system properties and names to SQL Server. How to read an HTML table to SQL Server and finally how to export SQL Server results to an HTML in SSIS. There are nice tools to read and write data to HTML in SSIS

If you liked the tools, you can download them HERE.

References

• SSIS PowerPack ZappySys
• Integration Services (SSIS) Variables
• Execute SQL Task

The post How to read and write data to HTML in SSIS appeared first on ZappySys Blog.

In this post you will learn how to use FREE SSIS Logging Task to perform ssis base64 encode decode and save to Variable or File  (i.e. convert Base64 Encoded value to plain text or encode plain text to base64) . You can also use many other format specifiers found here

How to invoke Placeholder Functions

There are 3 ways you can use placeholder functions in many tasks with Variable Icon next to the property.

1. User Variable Placeholder
2. Static String Placeholder
3. Mixed Placeholder (Static String with Variable)

Examples: 

{{User::var1,FUN_BASE64ENC}} 

<<some direct string,FUN_BASE64ENC>>

–OR–

<<{{User::var1}} and {{User::var2}} ,FUN_BASE64ENC>>

Encode Plain text to Base64 string (SSIS base64 encode decode)

Many times you have requirement to convert plain text into Base64 encoded string (This is very common when you have to transfer data over the web – set query string / http headers / body ). Typically to perform this you may have to write C# Script if you are SSIS developer but below is very simple way to do without writing any code. Here is the most common use case of Base64 encoded string

1. Download FREE Tasks/Components of SSIS PowerPack
2. Create new SSIS Package
3. Drag ZS Logging Task from Toolbox
4. Create SSIS variable which holds plain text value (Lets call it varPlainText. Set some initial value which you want to encode)
5. Configure SSIS Logging Task as below
   

   SSIS Base64 Encode, Decode using SSIS Logging Task (Without coding convert plain text to base 64 string)

6. Run it and you will see new file created with encoded value. You can also save encoded value to Variable if you change Logging Mode = Variable rather than File

Convert Base64 Encoded value to plain text (Decode)

To convert Base64 encoded string into plaintext you can perform similar steps described in the above section except you need to use BASE64DEC format specifier rather than BASE64ENC

Example:

<<QUJDREVGMTIzIyM=,FUN_BASE64DEC>>
--OR--
{{User::MySSISVar1,FUN_BASE64DEC}}

How to decode Base64 file in SSIS and save to new file

If you have a file coming as a base64 encoded string and you want to decode it and save it to a new file then it’s simple too. Basically, you will need SSIS Advanced File System Task  and SSIS Logging Task to achieve this (see package below).

SSIS decode base64 file – Read content convert to plain text and save to new file

Here are your steps.

1. Use SSIS Advanced File System Task and use read file content option to save file content into SSIS Variable
2. Configure Advanced File System as below.
   

   Read file content into SSIS variable using SSIS Advanced File System task

3. Next step connect SSIS Logging Task
4. Configure SSIS Logging Task as per below screenshot.
   

   Decode base64 file content in SSIS – Use Logging Task

Convert Base64 Encoded string of local file content (Binary or Text)

Using FUN_FILE_BASE64ENC we can encode the image or any other local file(Binary or Text) to the Base64 string.

FUN_FILE_BASE64ENC

Syntax:

<<file_path,FUN_FILE_BASE64ENC>>

Returns BASE64 encoded string of local file content (binary or text).

Parameters:

Examples:

<<c:\someimage.png,FUN_FILE_BASE64ENC>>
--OR--
{{User::MySSISVar1,FUN_FILE_BASE64ENC}}

Sample Output:

This returns base64 encoded string of binary file (png)

bXl1c2V..........Ac3NXb3Jk

The post SSIS Base64 Encode Decode without Coding appeared first on ZappySys Blog.

SQL Server 2012 and all previous versions lacking native JSON support, same theory is true for their ETL Platform SSIS. If you are one of them who wants to export JSON from SQL Server or any other RDBMS like MySQL, Oracle then you can easily output complex JSON using any of the following components.

Five ways of creating JSON inside SSIS

ZappySys toolset provides multiple ways creating JSON inside SSIS. You can choose method suitable for your scenario. Each method has pros and cons so choose it as per your need.

JSON File Destination

SSIS JSON Destination can be used to generate simple or complex JSON files out of relational data source such as SQL Server, Oracle, MySQL. In few clicks you can create desired JSON Shape from single or multiple tables (Parent/Child Nested Documents).

PROS: Multiple data flow components can be attached. Which means unlike Export JSON Task you are not limited to OLEDB or ODBC Sources only.

CONS: When you remove upstream component it may break Dataset mappings. Also server side Lookup Option is not available. File Splitting Options not available.

Here are some articles for use case of this component.

JSON File Destination – Generate JSON in SSIS from multiple inputs

JSON Generator Transform

SSIS JSON Generator Transform can be used to generate single or multiple JSON documents from any type of data sources (e.g. SQL Server, MySQL, Flat File, Excel) inside data flow task. It takes multiple inputs and outputs JSON String Output Column which can be used to feed JSON to other downstream system.

PROS: Multiple data flow components can be attached. Which means unlike Export JSON Task you are not limited to OLEDB or ODBC Sources only.

CONS: When you remove upstream component it may break Dataset mappings. Also server side Lookup Option is not available. File Splitting Options not available. Output is JSON String Column.

Here are some articles for use case of this transform.

Preview JSON – Generate for BigQuery Table Insert

Export JSON Task

SSIS Export JSON File Task can be used to generate simple or complex JSON files out of relational data source such as SQL Server, Oracle, MySQL. It also support exporting directly to cloud (e.g. Azure Blob Storage, Amazon AWS S3, Secure FTP). In few clicks you can create desired JSON Shape from single or multiple tables (Parent/Child Nested Documents)

PROS: Simple to use with flexible options to export to file, variable, AWS S3, Azure Blob, FTP / SFTP location. Allows to split records. File Splitting Options available (Split By Size  / Row Count).

CONS: You can only use limited data sources as Dataset input (e.g. OLEDB / ODBC or SQL Server).

Here are some articles for use case of this task.

Template Transform

Template Transform can be used to create dynamic strings inside data flow. You can use upstream columns as placeholders or use SSIS variables anywhere in the template to create desired text (e.g. XML / JSON document).

PROS: Very Simple to use.

CONS: Cannot use multiple inputs. It may not be used to create Complex JSON and options like do not output NULL attributes is not possible.

Here are some articles for use case of this transform.

Create JSON for API Request (POST) using SSIS Template Transform

Logging Task (Control Flow Approach)

 Just like Template Transform you can use ZS Logging Task to create JSON string from Template and Variable Placeholders.

Here is how to use Logging Task to save JSON into a variable.

1. Select Variable which you like to Set on Logging Task
2. Enter your Template JSON String which you like to produce (see example below). Notice the use of yyyy-MM-dd format specifier and also JSONENCODE function.
   

   {
    ActionId: {{User::vActionId}},
    CreatedOn: "{{User::vOrderDate,yyyy-MM-ddTHH:mm:ss}}",
    Notes: "{{User::vNotes,JSONENCODE}}"
   }

3. Select Message Type = None
4. Uncheck Append, and Prefix options like below
   

   Creating JSON / XML Document using SSIS Logging Task

Here are some articles for use case of this task.

Create JSON using SSIS Export JSON Task

Now let’s look at Export JSON Task form example to create JSON. Steps listed in below sections are very similar to JSON File Destination or JSON Generator Transform.

ZappySys offers innovative Layout Editor for JSON / XML creation from multiple sources. You can use Layout Editor to produce desired JSON Layout (Layout is stored in XML format behind the scene) and then in later step you will see how to make it dynamic so values are changed at runtime.

You can read layout as direct value, read from variable or read from file from disk.

Select source connection for data

First lets create new OLEDB or ADO.net connection from where we will read data. In our case we reading from Northwind Database.

SSIS JSON Export Task – Select Source Database Connection

Create Datasets

Now add datasets which will be used to produce JSON File. One dataset will be always Root Dataset (Example: If your Root information is about customer then select customer table or query as your Main dataset. All other sections will be child sections)

Add SQL Dataset – Export JSON from SQL Query Option

Add Output Columns for Main JSON Document

Now its time to add columns for our output. If * is added by default then remove. * means output all columns. When you add columns first make sure you select correct parent node under which you want to add columns. In our case “Mappings” is parent node.

Add Output Columns for JSON Document – Column Selection Screen

Add JSON Array / JOIN Conditions for related document

Now let’s look at how to add JSON Array. For example Customer may have one or more Orders so we need to add node called Orders which may Produce JSON fragment like below

Orders : [ {...} , {...} ] 

Once customer related columns added, its time to add Orders section.

1. This is Array of documents so first highlight “Mappings” node
2. Click “Add Document Array” option from toolbar.
3. On Add/Edit Elements UI specify name for JSON Attribute and specify correct relationship to parent dataset. In our case Orders related to Customers using CustomerID key.

Add Child Section – nested JSON document , Define Parent-Child Relationship

Handling JSON Array with dummy root dataset (Multiple Arrays)

So in previous section we saw how to add array and define JOIN condition with parent dataset. Now what if you really have no parent dataset to join with ? For example we need to produce following sample JSON where Order array is related to Customer by common Column CustomerID as JOIN column. But Customer is not bound with any parent column. So how do you define JOIN condition? In such case you have to use below technique called dummy root dataset. (See next section on how to handle only one dataset using Single Dataset Array Mode)

Sample JSON (Dummy root pattern)

{
    "Customers": [
      {
        "CustomerID": "ALFKI",
        "CompanyName": "Alfreds Futterkiste",
        "Orders": [
          {
            "OrderID": 1000,
            "OrderDate": "2015-12-01"
          },
          {
            "OrderID": 1001,
            "OrderDate": "2015-12-01"
          }
        ]
      },

      {
        "CustomerID": "ANATR",
        "CompanyName": "Ana Trujillo Emparedados y helados",
        "Orders": [
          {
            "OrderID": 1002,
            "OrderDate": "2015-12-01"
          },
          {
            "OrderID": 1003,
            "OrderDate": "2015-12-01"
          }
        ]
      }

    ]
  }
}

Since root level always needs dataset, we must provide dummy dataset (only one row) even though there is no need for Parent Dataset JOIN in Customer Array. Typically you can write query like below to product dummy row in SQL Server or other database using OLEDB Source or Use Dummy Data Source with Max Rows=1 setting to produce one fake record.

SELECT 1 as Col1

Using Dummy DataSource as Input

For generating JSON which needs dummy root. Perform the following steps

1. Drag Dummy Data Source from SSIS Toolbox. Change How many rows to 1. Click OK
2. Now drag two more OLEDB sources and Configure (e.g. Customers and Orders)
3. Now drag ZS JSON FIle Destination component or ZS JSON Generator
4. Now connect Dummy datasource then Customers and then Orders
5. Double click JSON component to configure layout. You may notice that Dummy Input is set as Root by default.
6. You can now select Mappings node and click on Add Array Documents icon. Configure like below. Notice we have not selected any JOIN criteria because its joined with Dummy Root.
   

   Add array to join with dummy dataset

7. Now select newly added node and Click Add column icon to add CustomerID, CompanyName column under Customers array
8. Now select Customers array node and click Array Documents icon to Add Orders array.
9. Once Orders Array ia added you can Select Orders node and Insert new element to add OrderID and OrderDate columns
10. Finally it will look like below
    

    JSON File Destination – Generate JSON in SSIS from multiple inputs

Single dataset used as array

Now lets look at one more pattern just like described in the previous section but now we have just one nested level (Only customers and not using Orders). If you have such case then choose Single dataset pattern option in the output mode. On Export Task you may see as check box rather than dropdown. To produced desired JSON (see blow) perform these steps.

1. Select Output mode as Single Dataset Array
2. In your Layout builder click on Add Document Array Icon and name it “Customers”.
3. After that highlight newly added node > Right-Click > Add Elements under that node
4. That’s it you will now see preview like below.

{
    "Customers": [
      {
        "CustomerID": "ALFKI",
        "CompanyName": "Alfreds Futterkiste"
      },

      {
        "CustomerID": "ANATR",
        "CompanyName": "Ana Trujillo Emparedados y helados"
      }
    ]
  }
}

How to export JSON in 2D Array format

If you have to export JSON as 2D array format then you can change following option.

Here are examples of output for each option.
Select say you have source data with 2 columns id and name.

Default

[{id:1,name:"AA"}, {id:2,name:"BB"}]

Multicontent

{id:1,name:"AA"}{id:2,name:"BB"}

ArrayDocs

[{id:1,name:"AA"}, {id:2,name:"BB"}]

Array2D

[[1,"AA"],[2,"BB"]]

Array2DWithHeader

[["id","name"],[1,"AA"],[2,"BB"]]

ArrayLines

[1,"AA"][2,"BB"]

ArrayLinesWithHeader

["id","name"][1,"AA"][2,"BB"]

Preview JSON output using live preview

You can enable / disable Live preview of JSON layout as below.

JSON Output preview in SSIS Export JSON File Task

Make things dynamic (Using SSIS Variable for SQL or Layout)

There will be a time when you have use dynamic values in your JSON generation. You may have to Alias something from variable or you have to supply Variable in your SQL used for Dataset. Not to worry. ZappySys Supports multiple ways to make it dynamic. You can use SSIS Variables following 3 ways to make Layout XML dynamic.

1. Use SSIS variable inside Dataset SQL (Only for ZS Export JSON Task)
2. Anywhere inside Layout XML (Click Edit Code option)
3. Load Layout From Variable (Indirect Method)

Using Variable inside Dataset SQL

Here is how you can use Variable inside dataset SQL for ZS Export JSON Task

SSIS Export JSON Task – Using Variable for Dynamic SQL

Using Variable inside Layout XML

Here is how you can insert SSIS Variable anywhere in Layout XML. In below Example we are choosing Alias for one column from some SSIS Variable.

SSIS JSON File Destination – Using SSIS Variable for Dynamic Layout

Load Layout From Variable

If you have some need to load Layout XML from File or SSIS Variable. You can use Load Layout from Variable Option.

For example in below screenshot we are creating layout first and ave to some variable. Or you can load from external file using ZS Advanced File System Task.  Connect Logging task to Export JSON Task and you ready to produce JSON.

Set SSIS Variable value (multi-line content using placeholder)

Here is how to load Layout From any SSIS Variable. You can also load Layout from XML file or Variable which contains layout File Path.

JSON Layout from SSIS Variable

Other JSON Output Options

Execute package and verify generated JSON it should look like below (Date format may be different in your case if you have set different on JSON options tab). For clarity we have formatted JSON below but in your output it may be compact version so reduce file size.

Indent JSON Output

You can check Indent option to make JSON look pretty like below.

{
  "CustomerID": "ALFKI",
  "CompanyName": "Alfreds Futterkiste",
  "ContactName": null,
  "Orders": [
    {
      "OrderID": 10643,
      "OrderDate": "1997-08-25T00:00:00"
    },
    {
      "OrderID": 10692,
      "OrderDate": "1997-10-03T00:00:00"
    },
    {
      "OrderID": 10702,
      "OrderDate": "1997-10-13T00:00:00"
    },
    {
      "OrderID": 10835,
      "OrderDate": "1998-01-15T00:00:00"
    },
    {
      "OrderID": 10952,
      "OrderDate": "1998-03-16T00:00:00"
    },
    {
      "OrderID": 11011,
      "OrderDate": "1998-04-09T00:00:00"
    }
  ]
}

Hide NULL Attribute Option

There will be a time when you want to Hide NULL attribute rather than showing MyAttr: null this is also possible by checking Hide NULL Attribute Option on Options Tab.

Export JSON to Amazon S3, Azure Blob, Secure FTP (SFTP / FTPS)

Export JSON Task support some really good options to save file to other than local file system. If you want to export JSON directly to Amazon / Azure Cloud or FTP Server without saving to local disk then you can use Azure / FTP / Amazon Connection Manager on Target Tab of Export JSON Task. This option is only available for PRO Edition Users.

Export JSON / CSV / XML Files to Azure Blob, Amazon S3, Secure FTP Storage

Conclusion

If you want to output JSON from SQL Server or any other relational DB then its not simple task in SSIS specially when you have nested JSON documents (e.g. Customer->Orders). In this post you have seen how to use SSIS JSON Export Task to generate nested JSON from multiple datasets.

The post 5 Ways to Export JSON from SQL Server using SSIS appeared first on ZappySys Blog.