In today’s data-driven world, the ability to seamlessly integrate and transfer files between different platforms is crucial for efficient business operations. If you’re a user of SQL Server Integration Services (SSIS) and looking to integrate SharePoint Online into your data workflow, you’re in luck. ZappySys SSIS PowerPack offers a powerful SharePoint Online Connector that makes downloading files from SharePoint Online a breeze. In this guide, we’ll walk you through the process of downloading files from SharePoint Online using SSIS and the ZappySys SSIS PowerPack SharePoint Online Connector.

Download files from SharePoint Online with SSIS:

In this section, we will learn how to configure and use SharePoint Online Connector in the API Source & API Destination to download files from SharePoint Online.

1. Begin with opening Visual Studio and Creating a New Project.
2. Select Integration Service Project and in the new project window set the appropriate name and location for the project. Click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on the left sidebar
   2. Solution Explorer and Property Window on the right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window at the bottom
   

   SSIS Toolbox – ZappySys Tasks

   NOTE: If you don’t see ZappySys SSIS PowerPack Task or Components in the SSIS Toolbox, please refer to this help link.
4. Now, Drag and Drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow Designer.
   
5. From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
6. Select New Connection to create a new connection:
   

Configuring the SharePoint Online Connector

1. Use a preinstalled SharePoint Online Connector from the Popular Connector List or press the Search Online radio button to download SharePoint Online Connector. Once downloaded simply use it in the configuration:
   

   SSIS ZS API Connection Manager – Select SharePoint Online Connector

2. Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed.
3. After configuring all the required properties in the API Connection Manager, please click on the “Test Connection” button to confirm that the connection is established successfully.
   

   Method1: Connect to SharePoint Online using User Credentials [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS ZS API Connection Manager – Generate Token – SharePoint Online with User Credentials

   Method2: Connect to SharePoint Online using Application Credentials [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – SharePoint Online with Application Credentials

How to download a single file from SharePoint Online

In the API Source Component, we’ve chosen the API Connection Manager. Next, we need to configure the remaining properties of the component to download the single file from SharePoint Online.

1. Choose the Download file endpoint from the “Select Table/Endpoint” dropdown menu.
2. Please choose DriveId from the drop-down menu.
3. Please set the FileId that indicates which file you wish to download. This is an essential parameter for the download process.
4. Please set the local folder’s physical path which indicates the desired file being downloaded by setting the TargetFilePath parameter.
5. If you intend to replace the local file with the one being downloaded, please select the ‘AlwaysOverwrite‘ option from the dropdown menu labeled ‘FileOverwriteMode‘. Alternatively, you can choose ‘FileIfExists’ or ‘SkipIfExists’ based on your specific needs.

Please refer to the sample screenshot below.

SSIS API Source – SharePoint Online – Download Single File

Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to SharePoint Online using the ZappySys SharePoint Online Connector for downloading a Single file from SharePoint Online.

How to download multiple files from SharePoint Online

In the API Destination Component, we have chosen the API Connection Manager. Now, we must proceed to configure the remaining properties of the component to facilitate the download of multiple files from SharePoint Online. Additionally, we should make use of an OLE DB Source and Derived Column component to map the files for the download process.

1. Drag and drop the OLE DB Source Component from the SSIS Toolbox. Double-click on the OLE D B Source Component to initiate its configuration.
   

   OLE DB Source – Drag and Drop

2. Create a new OLE DB Connection Manager and configure your SQL Server settings for the connection. Next, select the Data access mode, specify the name of the table or view, or write an SQL Command. Afterward, click on the ‘Preview’ button to view a preview and check the ‘Columns’ tab for the source columns.
   

   SSIS OLE DB Source – Configure SQL Connection and fetch the table data

3. Next, drag and drop the Derived Column Component from the SSIS Toolbox. Double-click on the Derived Column Component to begin its configuration.
   

   Drag and drop the Derived Column

4. Now, add a new Derived Column by setting the expression to concatenate FilePath and FileName. Click ‘OK’ to save the configuration.
   

   SSIS Derived Column – Add New Column and Set Expression

5. Now, drag and drop the API Destination Component from the SSIS Toolbox. Double-click on the API Destination Component to begin its configuration.
   

   Drag and Drop the SSIS API Destination Component

6. Double-click on the API Destination component to configure it for downloading multiple files from SharePoint Online.
   1. Choose the Download file endpoint from the “Select Table/Endpoint” dropdown menu.
   2. Please specify the ‘FileOverwriteMode’ option as ‘AlwaysOverwrite,’ or you can choose ‘FailIfExists’ or ‘SkipIfExists’ options based on your requirements.
      Please refer to the sample screenshot below.
      

      SSIS API Destination Component – Configure Settings Tab to download Files from SharePoint Online

   3. Now, go to the Mappings tab and specify the source columns with the corresponding target properties.
      Note: Here We’re mapping FileID with $$Id and FileNameWithPath with $$ResponseDataFile. You can create mappings based on your Source columns
      Please refer to the sample screenshot below
      

      SSIS API Destination Component – Configure the Mappings Tab to download multiple files from SharePoint Online

7. Once you have configured all the mentioned properties above, please click on the “Preview Data” button from the Settings tab to initiate the API call. With just a few clicks, we have successfully configured the call to SharePoint Online using the ZappySys SharePoint Online Connector for downloading multiple files from SharePoint Online.
   Please refer to the sample screenshot below.
   

   Download Multiple files from SharePoint Online

Conclusion

In this blog, we learned how to connect with SharePoint Online using SSIS API Connection Manager and download single or multiple files from SharePoint Online using a combination of SSIS components, including the SSIS API Source component, SSIS API Destination component, OLE DB Source component, and Derived Column component.

The post How to download files from SharePoint Online using SSIS appeared first on ZappySys Blog.

In today’s data-driven world, the ability to seamlessly integrate and transfer files between different platforms is crucial for efficient business operations. If you’re a user of SQL Server Integration Services (SSIS) and looking to integrate OneDrive into your data workflow, you’re in luck. ZappySys SSIS PowerPack offers a powerful OneDrive Connector that makes downloading files from OneDrive a breeze. In this guide, we’ll walk you through the process of downloading files from OneDrive using SSIS and the ZappySys SSIS PowerPack OneDrive Connector.

Download files from OneDrive with SSIS

In this section, we will learn how to configure and use OneDrive Connector in the API Source & API Destination to download files from OneDrive.

1. Begin with opening Visual Studio and Creating a New Project.
2. Select Integration Service Project and in the new project window set the appropriate name and location for the project. Click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on the left sidebar
   2. Solution Explorer and Property Window on the right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window at the bottom
   

   SSIS Toolbox – ZappySys Tasks

   NOTE: If you don’t see ZappySys SSIS PowerPack Task or Components in the SSIS Toolbox, please refer to this help link.
4. Now, Drag and Drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow Designer.
   
5. From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
6. Select New Connection to create a new connection:
   

Configuring the OneDrive Connector

1. Use a preinstalled OneDrive Connector from the Popular Connector List or press the Search Online radio button to download OneDrive Connector. Once downloaded simply use it in the configuration:
   
2. Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed.
3. After configuring all the required properties in the API Connection Manager, please click on the “Test Connection” button to confirm that the connection is established successfully.
   

   Method1: Connect to OneDrive using User Credentials [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – OneDrive with User Credentials

   Method2: Connect to OneDrive using Application Credentials [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – OneDrive with Application Credentials

How to download a single file from OneDrive

In the API Source Component, we’ve chosen the API Connection Manager. Next, we need to configure the remaining properties of the component to download the single file from OneDrive.

1. Choose the Download file endpoint from the “Select Table/Endpoint” dropdown menu.
2. Please choose DriveId from the drop-down menu.
3. Please set the FileId that indicates which file you wish to download. This is an essential parameter for the download process.
4. Please set the local folder’s physical path which indicates the desired file being downloaded by setting the TargetFilePath parameter.
5. If you intend to replace the local file with the one being downloaded, please select the ‘AlwaysOverwrite‘ option from the dropdown menu labeled ‘FileOverwriteMode‘. Alternatively, you can choose ‘FileIfExists’ or ‘SkipIfExists’ based on your specific needs.

Please refer to the sample screenshot below.

SSIS API Source – OneDrive – Download Single File

Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to OneDrive using the ZappySys OneDrive Connector for downloading a Single file from OneDrive.

How to download multiple files from OneDrive

In the API Destination Component, we have chosen the API Connection Manager. Now, we must proceed to configure the remaining properties of the component to facilitate the download of multiple files from OneDrive. Additionally, we should make use of an OLE DB Source and Derived Column component to map the files for the download process.

1. Drag and drop the OLE DB Source Component from the SSIS Toolbox. Double-click on the OLE DB Source Component to initiate its configuration.
   

   OLE DB Source – Drag and Drop

2. Create a new OLE DB Connection Manager and configure your SQL Server settings for the connection. Next, select the Data access mode, specify the name of the table or view, or write an SQL Command. Afterward, click on the ‘Preview’ button to view a preview and check the ‘Columns’ tab for the source columns.
   

   SSIS OLE DB Source – Configure SQL Connection and fetch the table data

3. Next, drag and drop the Derived Column Component from the SSIS Toolbox. Double-click on the Derived Column Component to begin its configuration.
   

   Drag and drop the Derived Column

4. Now, add a new Derived Column by setting the expression to concatenate FilePath and FileName. Click ‘OK’ to save the configuration.
   

   SSIS Derived Column – Add New Column and Set Expression

5. Now, drag and drop the API Destination Component from the SSIS Toolbox. Double-click on the API Destination Component to begin its configuration.
   

   Drag and Drop the SSIS API Destination Component

6. Double-click on the API Destination component to configure it for downloading multiple files from OneDrive.
   1. Choose the Download file endpoint from the “Select Table/Endpoint” dropdown menu.
   2. Please specify the ‘FileOverwriteMode’ option as ‘AlwaysOverwrite,’ or you can choose ‘FailIfExists’ or ‘SkipIfExists’ options based on your requirements.
      Please refer to the sample screenshot below.
      

      SSIS API Destination Component – Configure Settings Tab

   3. Now, go to the Mappings tab and specify the source columns with the corresponding target properties.
      Note: Here We’re mapping FileID with $$Id and FileNameWithPath with $$ResponseDataFile. You can create mappings based on your Source columns
      Please refer to the sample screenshot below
      

      SSIS API Destination Component – Configure Mappings Tab

7. Once you have configured all the mentioned properties above, please click on the “Preview Data” button from the Settings tab to initiate the API call. With just a few clicks, we have successfully configured the call to OneDrive using the ZappySys OneDrive Connector for downloading multiple files from OneDrive.
   Please refer to the sample screenshot below.
   

   Download Multiple files from OneDrive

Conclusion

In this blog, we learned how to connect with OneDrive using SSIS API Connection Manager and download single or multiple files from OneDrive using a combination of SSIS components, including the SSIS API Source component, SSIS API Destination component, OLE DB Source component, and Derived Column component.

The post How to download files from OneDrive using SSIS appeared first on ZappySys Blog.

In today’s data-driven world, the ability to seamlessly integrate and transfer files between different platforms is crucial for efficient business operations. If you’re a user of SQL Server Integration Services (SSIS) and looking to integrate OneDrive into your data workflow, you’re in luck. ZappySys SSIS PowerPack offers a powerful OneDrive Connector that makes uploading files to OneDrive a breeze. In this guide, we’ll walk you through the process of uploading files to OneDrive using SSIS and the ZappySys SSIS PowerPack OneDrive Connector.

Upload files to OneDrive with SSIS:

In this section, we will learn how to configure and use OneDrive Connector in the API Source to upload files on OneDrive.

1. Begin with opening Visual Studio and Creating a New Project.
2. Select Integration Service Project and in the new project window set the appropriate name and location for the project. Click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on the left sidebar
   2. Solution Explorer and Property Window on the right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window at the bottom
   

   SSIS Toolbox – ZappySys Tasks

   NOTE: If you don’t see ZappySys SSIS PowerPack Task or Components in the SSIS Toolbox, please refer to this help link.
4. Now, Drag and Drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow Designer.
   
5. From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
6. Select New Connection to create a new connection:
   

Configuring the OneDrive Connector

1. Use a preinstalled OneDrive Connector from the Popular Connector List or press the Search Online radio button to download OneDrive Connector. Once downloaded simply use it in the configuration:
   
2. Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed.
3. After configuring all the required properties in the API Connection Manager, please click on the “Test Connection” button to confirm that the connection is established successfully.
   

   Method1: Connect to OneDrive using User Credentials [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – OneDrive with User Credentials

   Method2: Connect to OneDrive using Application Credentials [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – OneDrive with Application Credentials

How to upload a local file to OneDrive

In the API Source Component, we’ve chosen the API Connection Manager. Next, we need to configure the remaining properties of the component to upload a local file to OneDrive.

1. Choose the Upload file endpoint from the “Select Table/Endpoint” dropdown menu.
2. Select the desired Drive ID from the DriveId drop-down.
3. Specify the desired source file path for the file being uploaded by setting the SourceFilePath parameter.
4. Specify the desired Target Path by setting the TargetFilePath parameter.

Please refer to the sample screenshot below.

SSIS API Source – OneDrive – Upload Single File

Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to OneDrive using the ZappySys OneDrive Connector for uploading a local file to OneDrive.

How to upload multiple files to OneDrive

In the API Source Component, we’ve selected the API Connection Manager. Next, we need to configure the remaining properties of the component to upload multiple files to OneDrive. Additionally, we should utilize a Foreach Loop Container to iterate through each file one by one in a loop.

1. Create 3 SSIS variables as described below.
   

   Variable Name	Datatype	Value
   varFileLists	Object	System.Object
   varFileName	String	fake.extension
   varFullFilePath	String	C:/fake/fake.extension

2. Go to Control Flow, From the SSIS toolbox drag and drop ZS Advanced File System Task on the control flow designer surface, and double click on it to edit it:
   
3. Select the “Get file list as ADO.NET Data Table” action from the Action drop-down menu. Set the file path in the Path textbox. Choose the variable (Object type) where you want to store the file list, and then click the OK button to save the settings.
   

   SSIS Advanced File System Task – Get file list as ADO.net Data Table

4. Now, drag and drop the “Foreach Loop Container” from the SSIS Toolbox and connect it with the “ZS Advanced File System Task.” Double-click on the “Foreach Loop Container” to edit its properties.
   

   SSIS Foreach Loop Container – Drag & Drop

5. In the Foreach Loop Editor’s “Collection” tab, choose the Enumerator as “Foreach ADO Enumerator.” In the “ADO object source variable” field, select your SSIS object-typed variable. Select the “Rows in the first table” radio option.
   

   SSIS Foreach Loop Editor – Collection – Select Variable & Enumeration mode

6. In the Foreach Loop Editor’s “Variable Mappings” tab, select the variables that you want to map to the values from the collection, and then click the “OK” button to save your selections.
   

   SSIS Foreach Loop Editor – Variable Mappings

7. Inside the Foreach Loop Container, drag and drop the Data Flow Task, and then rename it according to your preference.
   

   SSIS Data Flow Task – Drag & Drop inside Foreach Loop Container

8. Double-click on the Data Flow Task to see the Data Flow Designer, From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
9. Double-click on the API Source component to configure it for uploading multiple files to OneDrive.
   1. Choose the Upload file endpoint from the “Select Table/Endpoint” dropdown menu.
   2. Please specify the desired SSIS Variable [varFileName] containing the uploaded file name by setting the TargetFilePath parameter.
   3. To specify the desired local SSIS variable [varFullFilePath], which holds the file path, set the SourceFilePath parameter.
      When configuring the SSIS variable directly in the Raw Edit, JSON may become invalid due to the requirement of slashes within the local file path. To address this, we use a JSONENC Placeholder Function for proper escaping.
      

      <<{{User::varUserFullFilePath}},FUN_JSONENC>>

   4. If you wish to upload the local file to a specific folder in OneDrive, please set the folderName in the TargetFilePath parameter.

   Please refer to the sample screenshot below.

   

   SSIS API Source – OneDrive – Upload Multiple files configuration

   To configure dynamic filenames and file paths, you can set variables. Click on “Raw Edit,” and in the “Edit Parameters” popup, you can use SSIS variables, as shown in the above image.

   Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call.
   After that, you can execute the SSIS Package and check the execution to upload multiple files on OneDrive. Below is the sample screenshot attached.
   
   Multiple files uploaded on OneDrive using SSIS

   With just a few clicks, we have successfully configured the call to OneDrive using the ZappySys OneDrive Connector for uploading multiple files to OneDrive.

Conclusion

In this blog, we learned how to connect with OneDrive using SSIS API Connection Manager and upload single or multiple files to OneDrive using a combination of SSIS components, including the SSIS API Source component, SSIS ZS Advanced File System Task, Foreach Loop Container, and SSIS variables.

The post How to upload files on OneDrive using SSIS appeared first on ZappySys Blog.

In today’s data-driven world, the ability to seamlessly integrate and transfer files between different platforms is crucial for efficient business operations. If you’re a user of SQL Server Integration Services (SSIS) and looking to integrate Google Drive into your data workflow, you’re in luck. ZappySys SSIS PowerPack offers a powerful Google Drive Connector that makes downloading files from Google Drive a breeze. In this guide, we’ll walk you through the process of downloading files from Google Drive using SSIS and the ZappySys SSIS PowerPack Google Drive Connector.

Download files from Google Drive with SSIS:

In this section, we will learn how to configure and use Google Drive Connector in the API Source & API Destination to download files from Google Drive.

1. Begin with opening Visual Studio and Creating a New Project.
2. Select Integration Service Project and in the new project window set the appropriate name and location for the project. Click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on the left sidebar
   2. Solution Explorer and Property Window on the right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window at the bottom

   -`
   

   NOTE: If you don’t see ZappySys SSIS PowerPack Task or Components in the SSIS Toolbox, please refer to this help link.
4. Now, Drag and Drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow Designer.
   
5. From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
6. Select New Connection to create a new connection:
   

Configuring the Google Drive Connector

1. Use a preinstalled Google Drive Connector from the Popular Connector List or press the Search Online radio button to download Google Drive Connector. Once downloaded simply use it in the configuration:
   
2. Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed.
3. After configuring all the required properties in the API Connection Manager, please click on the “Test Connection” button to confirm that the connection is established successfully.
   

   Method1: Connect to Google Drive using User Account [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – Google Drive with User Account

   Method2: Connect to Google Drive using Service Account [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – Google Drive with Service Account

How to download a single file from Google Drive

In the API Source Component, we’ve chosen the API Connection Manager. Next, we need to configure the remaining properties of the component to download the single file from Google Drive.

1. Choose the Download a file endpoint from the “Select Table/Endpoint” dropdown menu.
2. To access detailed API documentation for downloading files, you can click on the following link: Help for: Download a File.
3. Please choose a File ID from the drop-down menu to indicate which file you wish to download. This is an essential parameter for the download process.
4. Please indicate the desired filename for the file being downloaded by setting the FileName parameter.
5. If you intend to replace the local file with the one being downloaded, please select the ‘AlwaysOverwrite‘ option from the dropdown menu labeled ‘FileOverwriteMode‘. Alternatively, you can choose ‘FileIfExists’ or ‘SkipIfExists’ based on your specific needs.

Please refer to the sample screenshot below.

SSIS API Source – Google Drive – Download Single File

Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to Google Drive using the ZappySys Google Drive Connector for downloading a Single file from Google Drive.

How to download multiple files from Google Drive

In the API Destination Component, we have chosen the API Connection Manager. Now, we must proceed to configure the remaining properties of the component to facilitate the download of multiple files from Google Drive. Additionally, we should make use of an OLE DB Source and Derived Column component to map the files for the download process.

1. Drag and drop the OLE DB Source Component from the SSIS Toolbox. Double-click on the OLE DB Source Component to initiate its configuration.
   

   OLE DB Source – Drag and Drop

2. Create a new OLE DB Connection Manager and configure your SQL Server settings for the connection. Next, select the Data access mode, specify the name of the table or view, or write an SQL Command. Afterward, click on the ‘Preview’ button to view a preview and check the ‘Columns’ tab for the source columns.
   

   SSIS OLE DB Source – Configure SQL Connection and fetch the table data

3. Next, drag and drop the Derived Column Component from the SSIS Toolbox. Double-click on the Derived Column Component to begin its configuration.
   

   Drag and drop the Derived Column

4. Now, add a new Derived Column by setting the expression to concatenate FilePath and FileName. Click ‘OK’ to save the configuration.
   

   SSIS Derived Column – Add New Column & Set Expression

5. Now, drag and drop the API Destination Component from the SSIS Toolbox. Double-click on the API Destination Component to begin its configuration.
   

   Drag and Drop the SSIS API Destination Component

6. Double-click on the API Destination component to configure it for uploading multiple files to Google Drive.
   1. Choose the Download a file endpoint from the “Select Table/Endpoint” dropdown menu.
   2. To access detailed API documentation for downloading files, you can click on the following link: Help for: Download File.
   3. Please specify the ‘FileOverwriteMode’ option as ‘AlwaysOverwrite,’ or you can choose ‘FailIfExists’ or ‘SkipIfExists’ options based on your requirements.
      Please refer to the sample screenshot below.
      

      SSIS API Destination Component – Configure Settings Tab

   4. Now, go to the Mappings tab and specify the source columns with the corresponding target properties.
      Note: Here We’re mapping FileID with $$Id and FileNameWithPath with $$ResponseDataFile. You can create mappings based on your Source columns
      Please refer to the sample screenshot below
      

      SSIS API Destination Component – Configure Mappings Tab

7. Once you have configured all the mentioned properties above, please click on the “Preview Data” button from the Settings tab to initiate the API call. With just a few clicks, we have successfully configured the call to Google Drive using the ZappySys Google Drive Connector for downloading multiple files from Google Drive.
   Please refer to the sample screenshot below.
   

   Download Multiple files from Google Drive

Conclusion

In this blog, we learned how to connect with Google Drive using SSIS API Connection Manager and download single or multiple files from Google Drive using a combination of SSIS components, including the SSIS API Source component, SSIS API Destination component, OLE DB Source component, and Derived Column component.

The post How to download files from Google Drive using SSIS appeared first on ZappySys Blog.

In today’s data-driven world, the ability to seamlessly integrate and transfer files between different platforms is crucial for efficient business operations. If you’re a user of SQL Server Integration Services (SSIS) and looking to integrate Google Drive into your data workflow, you’re in luck. ZappySys SSIS PowerPack offers a powerful Google Drive Connector that makes uploading files to Google Drive a breeze. In this guide, we’ll walk you through the process of uploading files to Google Drive using SSIS and the ZappySys SSIS PowerPack Google Drive Connector.

Upload files to Google Drive with SSIS:

In this section, we will learn how to configure and use Google Drive Connector in the API Source to upload files on Google Drive.

1. Begin with opening Visual Studio and Creating a New Project.
2. Select Integration Service Project and in the new project window set the appropriate name and location for the project. Click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on the left sidebar
   2. Solution Explorer and Property Window on the right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window at the bottom
   

   SSIS Toolbox – ZappySys Tasks

   NOTE: If you don’t see ZappySys SSIS PowerPack Task or Components in the SSIS Toolbox, please refer to this help link.
4. Now, Drag and Drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow Designer.
   
5. From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
6. Select New Connection to create a new connection:
   

Configuring the Google Drive Connector

1. Use a preinstalled Google Drive Connector from the Popular Connector List or press the Search Online radio button to download Google Drive Connector. Once downloaded simply use it in the configuration:
   
2. Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed.
3. After configuring all the required properties in the API Connection Manager, please click on the “Test Connection” button to confirm that the connection is established successfully.
   

   Method1: Connect to Google Drive using User Account [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – Google Drive with User Account

   Method2: Connect to Google Drive using Service Account [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – Google Drive with Service Account

How to upload a local file to Google Drive

In the API Source Component, we’ve chosen the API Connection Manager. Next, we need to configure the remaining properties of the component to upload a local file to Google Drive.

1. Choose the Upload a file endpoint from the “Select Table/Endpoint” dropdown menu.
2. To access detailed API documentation for uploading files, you can click on the following link: Help for: Upload File.
3. Please specify the desired filename for the uploaded file by setting the FileName parameter.
4. Specify the desired local file path by setting the DiskFilePath parameter.
5. If you wish to upload the local file to a specific folder in Google Drive, please choose the folder ID from the dropdown menu labeled ParentFolderId. Otherwise, you can skip this parameter as it’s optional.

Please refer to the sample screenshot below.

SSIS API Source – Google Drive – Upload Single File

Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to Google Drive using the ZappySys Google Drive Connector for uploading a local file to Google Drive.

How to upload multiple files to Google Drive

In the API Source Component, we’ve selected the API Connection Manager. Next, we need to configure the remaining properties of the component to upload multiple files to Google Drive. Additionally, we should utilize a Foreach Loop Container to iterate through each file one by one in a loop.

1. Create 3 SSIS variables as described below.
   

   Variable Name	Datatype	Value
   varFileLists	Object	System.Object
   varFileName	String	fake.extension
   varFullFilePath	String	C:/fake/fake.extension

2. Go to Control Flow, From the SSIS toolbox drag and drop ZS Advanced File System Task on the control flow designer surface, and double click on it to edit it:
   
3. Select the “Get file list as ADO.NET Data Table” action from the Action drop-down menu. Set the file path in the Path textbox. Choose the variable (Object type) where you want to store the file list, and then click the OK button to save the settings.
   

   SSIS Advanced File System Task – Get file list as ADO.net Data Table

4. Now, drag and drop the “Foreach Loop Container” from the SSIS Toolbox and connect it with the “ZS Advanced File System Task.” Double-click on the “Foreach Loop Container” to edit its properties.
   

   SSIS Foreach Loop Container – Drag & Drop

5. In the Foreach Loop Editor’s “Collection” tab, choose the Enumerator as “Foreach ADO Enumerator.” In the “ADO object source variable” field, select your SSIS object-typed variable. Select the “Rows in the first table” radio option.
   

   SSIS Foreach Loop Editor – Collection – Select Variable & Enumeration mode

6. In the Foreach Loop Editor’s “Variable Mappings” tab, select the variables that you want to map to the values from the collection, and then click the “OK” button to save your selections.
   

   SSIS Foreach Loop Editor – Variable Mappings

7. Inside the Foreach Loop Container, drag and drop the Data Flow Task, and then rename it according to your preference.
   

   SSIS Data Flow Task – Drag & Drop inside Foreach Loop Container

8. Double-click on the Data Flow Task to see the Data Flow Designer, From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double click on it to edit it:
   
9. Double-click on the API Source component to configure it for uploading multiple files to Google Drive.
   1. Choose the Upload a file endpoint from the “Select Table/Endpoint” dropdown menu.
   2. To access detailed API documentation for uploading files, you can click on the following link: Help for: Upload File.
   3. Please specify the desired SSIS Variable [varFileName] which contains file name for the uploaded file by setting the FileName parameter.
   4. To specify the desired local SSIS variable [varFullFilePath], which holds the file path, set the DiskFilePath parameter.
      When configuring the SSIS variable directly in the Raw Edit, JSON may become invalid due to the requirement of slashes within the local file path. To address this, we use a JSONENC Placeholder Function for proper escaping.
      

      <<{{User::varUserFullFilePath}},FUN_JSONENC>>

   5. If you wish to upload the local file to a specific folder in Google Drive, please choose the folder ID from the dropdown menu labeled ParentFolderId. Otherwise, you can skip this parameter as it’s optional.

   Please refer to the sample screenshot below.

   

   SSIS API Source – Google Drive – Upload Multiple files configuration

   To configure dynamic filenames and file paths, you can set variables. Click on “Raw Edit,” and in the “Edit Parameters” popup, you can use SSIS variables, as shown in the image below.

   Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to Google Drive using the ZappySys Google Drive Connector for uploading multiple files to Google Drive.

Conclusion

In this blog, we learned how to connect with Google Drive using SSIS API Connection Manager and upload single or multiple files to Google Drive using a combination of SSIS components, including the SSIS API Source component, SSIS ZS Advanced File System Task, Foreach Loop Container, and SSIS variables.

The post How to upload files on Google Drive using SSIS appeared first on ZappySys Blog.

In this post, you will learn how to Monitor and Run SSIS Package using Stored Procedure (T-SQL Code) – Packages stored in SSIS Catalog.  Calling SSIS Packages via T-SQL / Stored procedure code can be useful if you want to trigger SSIS package execution from your SQL code on certain conditions (i.e. Load data from SSIS Package before executing certain SQL code).

Techniques mentioned in this article applies to SSIS PowerPack or Native SSIS Packages.  Now let’s look at each piece step-by-step.

How to run SSIS package using Stored Procedure / T-SQL Code

Let’s look at simple example of Calling SSIS package deployed to SSIS Catalog. Basically, there are three steps (setting parameters is optional).

1. Required – Call catalog.create_execution System stored proc – This Creates new execution and returns you execution_id which can be saved to a variable
2. Optional – Call catalog.set_execution_parameter_value System stored proc – Call this multiple times if you want to set parameters
3. Required – Call catalog.start_execution to execute the package

Now let’s look at the example. Below example is the simplest way to call SSIS package using SQL code. Below code creates a new SSIS execution and returns immediately without waiting to finish package execution (asynchronous execution).

USE SSISDB -- <<< Change your SSIS Catalog DB name if its other than SSISDB
go
declare @my_execution_id bigint

declare @my_folder_name varchar(255)= 'MyFolder'
declare @my_project_name varchar(255) = 'MyProject'
declare @my_package_name varchar(255) = 'MyPackage.dtsx'

--////////////////////////////////////////////////////////
--First create Execution and get ExecutionID in a variable
--////////////////////////////////////////////////////////
exec catalog.create_execution 
	@folder_name = @my_folder_name
	,@project_name = @my_project_name
	,@package_name = @my_package_name
	,@reference_id = null
	,@execution_id = @my_execution_id output

--////////////////////////////////////////////////////////
--Start Execution
--////////////////////////////////////////////////////////
exec catalog.start_execution @my_execution_id

How to wait until SSIS package is finished (Set SYNCHRONIZED)

By default SSIS Catalog Package execution is in asynchronous mode. So what if you really want to wait until SSIS package execution is done (Synchronous mode) ? Just add one more sored procedure call before calling catalog.start_execution (see below)

USE SSISDB -- <<< Change your SSIS Catalog DB name if its other than SSISDB
go
declare @my_execution_id bigint

declare @my_folder_name varchar(255)= 'MyFolder'
declare @my_project_name varchar(255) = 'MyProject'
declare @my_package_name varchar(255) = 'MyPackage.dtsx'

--////////////////////////////////////////////////////////
--First create Execution and get ExecutionID in a variable
--////////////////////////////////////////////////////////
exec catalog.create_execution 
	@folder_name = @my_folder_name
	,@project_name = @my_project_name
	,@package_name = @my_package_name
	,@reference_id = null
	,@execution_id = @my_execution_id output

--////////////////////////////////////////////////////////
--Wait until package execution is done
--////////////////////////////////////////////////////////
EXEC [catalog].[set_execution_parameter_value] 
	@my_execution_id,  
	@object_type=50, 
	@parameter_name=N'SYNCHRONIZED', 
	@parameter_value=1

--////////////////////////////////////////////////////////
--Start Execution
--////////////////////////////////////////////////////////
exec catalog.start_execution @my_execution_id

Set SSIS Package Parameters via SQL Code for execution

Now lets look at common scenario to pass package parameters.

USE SSISDB -- <<< Change your SSIS Catalog DB name if its other than SSISDB
go
declare @my_execution_id bigint

declare @my_folder_name varchar(255)= 'ZappySys'
declare @my_project_name varchar(255) = 'DownloadFileFromCloud'
declare @my_package_name varchar(255) = 'DownloadFileFromCloud.dtsx'

--////////////////////////////////////////////////////////
--First create Execution and get ExecutionID in a variable
--////////////////////////////////////////////////////////
exec catalog.create_execution 
	@folder_name = @my_folder_name
	,@project_name = @my_project_name
	,@package_name = @my_package_name
	,@reference_id = null
	,@execution_id = @my_execution_id output

--////////////////////////////////////////////////////////
--Wait until package execution is done
--////////////////////////////////////////////////////////
EXEC [catalog].[set_execution_parameter_value] 
	@my_execution_id,  
	@object_type=50, 
	@parameter_name=N'SYNCHRONIZED', 
	@parameter_value=1

--////////////////////////////////////////////////////////
--Set package parameters
--////////////////////////////////////////////////////////
exec catalog.set_execution_parameter_value  @my_execution_id, 50, 'FilePath', 'C:\Temp\Dump.json'  
exec catalog.set_execution_parameter_value  @my_execution_id, 50, 'TargetServerName', 'Server001'  

--////////////////////////////////////////////////////////
--Start Execution
--////////////////////////////////////////////////////////
exec catalog.start_execution @my_execution_id

How to set SSIS environment for package execution using SQL Query

Many times you have need to select SSIS Environment rather than setting individual SSIS Parameters (e.g. Use DEV Environment parameters). See below code How to find correct reference_id for SSI environment  and pass to create_execution  stored procedure call.

Here is how you set SSIS environment using SSMS UI. In the next section see the code to select same via T-SQL code.

Set SSIS Environment for Package Execution – SSMS UI

Setting SSIS Environment for execution using SQL Code

USE SSISDB -- <<< Change your SSIS Catalog DB name if its other than SSISDB
go
declare @my_execution_id bigint

declare @my_folder_name varchar(255)= 'ZappySys'
declare @my_project_name varchar(255) = 'DownloadFileFromCloud'
declare @my_package_name varchar(255) = 'DownloadFileFromCloud.dtsx'
declare @my_environment_name varchar(255) = 'ENV_DEV'

--Find Environment reference_id from name
declare @my_env_refid int
select @my_env_refid=reference_id from [catalog].environment_references where environment_name=@my_environment_name

--////////////////////////////////////////////////////////
--First create Execution and get ExecutionID in a variable
--////////////////////////////////////////////////////////
exec catalog.create_execution 
	@folder_name = @my_folder_name
	,@project_name = @my_project_name
	,@package_name = @my_package_name
	,@reference_id = @my_env_refid --pass environment id here
	,@execution_id = @my_execution_id output

--////////////////////////////////////////////////////////
--Wait until package execution is done
--////////////////////////////////////////////////////////
EXEC [catalog].[set_execution_parameter_value] 
	@my_execution_id,  
	@object_type=50, 
	@parameter_name=N'SYNCHRONIZED', 
	@parameter_value=1

--////////////////////////////////////////////////////////
--Set package parameters
--////////////////////////////////////////////////////////
exec catalog.set_execution_parameter_value  @my_execution_id, 50, 'FilePath', 'C:\Temp\Dump.json'  
exec catalog.set_execution_parameter_value  @my_execution_id, 50, 'TargetServerName', 'Server001'  

--////////////////////////////////////////////////////////
--Start Execution
--////////////////////////////////////////////////////////
exec catalog.start_execution @my_execution_id

How to get last SSIS Package execution Status using SQL Query

Now let’s look at, how to get SSIS package execution Status for last execution using SQL Query.  In this example we are grabbing last package execution_id using Max(execution_id) function but you can hardcode execution_id too.

See different numeric status codes and description for SSIS Package Execution.

USE SSISDB -- <<< Change your SSIS Catalog DB name if its other than SSISDB
GO
declare @my_execution_id bigint
Select @my_execution_id=Max(execution_id) From SSISDB.catalog.executions 
--Where package_name like 'MyPackage%'

 select folder_name,project_name,package_name,status_text=case 
	when status=1 then 'Created' 
	when status=2 then 'Running' 
	when status=3 then 'Cancelled' 
	when status=4 then 'Failed' 
	when status=5 then 'Pending' 
	when status=6 then 'Ended Unexpectedly' 
	when status=7 then 'Succedded' 
	when status=8 then 'Stopping' 
	when status=9 then 'Completed' 
	else 'Not Sure'
	end
from catalog.executions  where execution_id=@my_execution_id

How to get last SSIS Package execution log using SQL Query

Now let’s look at full SQL Query which can give us execution status, errors and full execution log of last SSIS Package execution.

USE SSISDB -- <<< Change your SSIS Catalog DB name if its other than SSISDB
GO
declare @my_execution_id bigint
Select @my_execution_id=Max(execution_id) From catalog.executions 
--Where package_name like 'MyPackage%'

--////////////////////////////////////////////////
--See just SSIS package execution status
--////////////////////////////////////////////////
 select folder_name,project_name,package_name,status_text=case 
	when status=1 then 'Created' 
	when status=2 then 'Running' 
	when status=3 then 'Cancelled' 
	when status=4 then 'Failed' 
	when status=5 then 'Pending' 
	when status=6 then 'Ended Unexpectedly' 
	when status=7 then 'Succedded' 
	when status=8 then 'Stopping' 
	when status=9 then 'Completed' 
	else 'Not Sure'
	end
from catalog.executions  where execution_id=@my_execution_id

--////////////////////////////////////////////////
--See Only Errors
--////////////////////////////////////////////////
SELECT top 100 @my_execution_id ExecID,message, package_name, event_name, message_source_name,message_time as start_time
FROM   (SELECT        
		event_message_id, operation_id, message_time, message_type, message_source_type, message, extended_info_id, package_name,
        event_name, message_source_name, message_source_id, subcomponent_name, package_path, execution_path, threadID,
        message_code
        FROM            catalog.event_messages AS em
        WHERE   (operation_id =@my_execution_id) 
			AND (event_name NOT LIKE '%Validate%')
	) AS q
Where q.event_name='OnError'
ORDER BY message_time DESC
--////////////////////////////////////////////////
--See Full Log (Information, Warnings and Errors
--////////////////////////////////////////////////
SELECT top 1000 event_message_id as id
	, datalength(message) message_length
	, message
	, try_cast('<![CDATA[' + message + ']]>' as XML) as message_xml
	, message
	, package_name, event_name, message_source_name,@my_execution_id execid,message_time
FROM   (SELECT        
		event_message_id, operation_id, message_time, message_type, message_source_type, message, extended_info_id, package_name,
        event_name, message_source_name, message_source_id, subcomponent_name, package_path, execution_path, threadID,
        message_code
        FROM            catalog.event_messages AS em
        WHERE   (operation_id =@my_execution_id) 
			AND (event_name NOT LIKE '%Validate%') 
			--and event_message_id=20303352
	) AS q
ORDER BY event_message_id DESC

Here is the screenshot. Notice how we have message_xml column to see more than 8000 characters from log message in your SSMS output (Grid mode). SSMS can show maximum 8000 characters in one cell so using try_Catch function along with XML datatype will fix that issue.

See SSIS Package Execution log using SQL Query (T-SQL code) – View Execution Status, Error Messages, Information – Also add more than 8000 characters fix for SSMS grid

Full SQL Script (Execute SSIS Package / Monitor SSIS Package)

Here is the full script.

use [SSISDB]

GO

/*
 Author: ZappySys
 
 - This script demonstrates few aspects of SSIS package execution via code (call system stored procedures)
	>> How to execute SSIS package via stored procedure call
	>> How to set package parameters via code
	>> How to set envirnment via code
	>> How to monitor SSIS package execution via T-SQL code
	>> How to check SSIS package execution status / error and execution log
 - Feel free to modify and comment uncomment certain sections to meet your need.
 - We are outputting message column in xml format too so you get nice hyper link to see full message. This technique is needed to overcome string length limitation of SSMS

 */

declare @see_log bit =1 --set it to 0 if you want to execute package rather than see log
declare @my_execution_id bigint=null  -- enter execution id you like to see. Kepp it null to read last execution 

declare @my_folder_name varchar(255)= 'AdfExamples'
declare @my_project_name varchar(255) = 'SSISNativeExamples'
declare @my_package_name varchar(255) = 'HelloWorld.dtsx'
declare @my_environment_name varchar(255) = null  --e.g. This is SSIS Environment to attach with this execution (e.g. ENV_DEV , ENV_PROD)
--declare @my_environment_name varchar(255) = 'ENV_DEV'
	  
--declare @my_execution_id bigint=1200 --enter execid to see log

declare @starttime datetime
declare @statusLoop bit

if(@see_log = 1 and @my_execution_id is null)
	Select @statusLoop=0, @my_execution_id=Max(execution_id) 
	From catalog.executions 
	--WHERE package_name like 'Hello%'


if(@my_execution_id is null)
begin
	set @statusLoop=1 
	
	declare @my_env_refid int
	select @my_env_refid=reference_id from [catalog].environment_references where environment_name=@my_environment_name

	--First create Execution and get ExecutionID in a variable
	--https://docs.microsoft.com/en-us/sql/integration-services/system-stored-procedures/catalog-create-execution-ssisdb-database?view=sql-server-2017
	exec catalog.create_execution 
	  @folder_name = @my_folder_name
	 ,@project_name = @my_project_name
	 ,@package_name = @my_package_name
	 ,@reference_id = @my_env_refid
	 ,@execution_id = @my_execution_id output

	 --///////////////////////////
	--Wait until execution is done
	--///////////////////////////
	 -- IF You want to wait until execution is done. If you enable this then no need for status check loop
	 --https://docs.microsoft.com/en-us/sql/integration-services/system-stored-procedures/catalog-set-execution-parameter-value-ssisdb-database?view=sql-server-2017
	EXEC [catalog].[set_execution_parameter_value] 
		@my_execution_id,  
		@object_type=50, 
		@parameter_name=N'SYNCHRONIZED', 
		@parameter_value=1
	set @statusLoop=0 --if you enable this option no need for statucheck
	
	--///////////////////////////
	--Set Parameters
	--///////////////////////////
	--https://docs.microsoft.com/en-us/sql/integration-services/system-stored-procedures/catalog-set-execution-parameter-value-ssisdb-database?view=sql-server-2017
	--exec catalog.set_execution_parameter_value  @my_execution_id, 50, 'FilePath', 'C:\Temp\Dump.json'  
	--exec catalog.set_execution_parameter_value  @my_execution_id, 50, 'TargetServerName', 'Server001'  

	--///////////////////////////
	--Start Execution
	--///////////////////////////
	--https://docs.microsoft.com/en-us/sql/integration-services/system-stored-procedures/catalog-start-execution-ssisdb-database?view=sql-server-2017
	exec catalog.start_execution @my_execution_id
end

--///////////////////////////
--Status check loop for wait
--///////////////////////////
---comment these two lines if set SYNCHRONIZED = 1
if(@statusLoop = 1)
WHILE @my_execution_id IN (1,2,5,8)
	WAITFOR DELAY '00:03'
--

 select folder_name,project_name,package_name,status_text=case 
	when status=1 then 'Created' 
	when status=2 then 'Running' 
	when status=3 then 'Cancelled' 
	when status=4 then 'Failed' 
	when status=5 then 'Pending' 
	when status=6 then 'Ended Unexpectedly' 
	when status=7 then 'Succedded' 
	when status=8 then 'Stopping' 
	when status=9 then 'Completed' 
	else 'Not Sure'
	end
from catalog.executions  where execution_id=@my_execution_id


--///////////////////
----Show Just Errors
--///////////////////
SELECT top 10 @my_execution_id ExecID,message, package_name, event_name, message_source_name,message_time as start_time
FROM   (SELECT        
		event_message_id, operation_id, message_time, message_type, message_source_type, message, extended_info_id, package_name,
        event_name, message_source_name, message_source_id, subcomponent_name, package_path, execution_path, threadID,
        message_code
        FROM            catalog.event_messages AS em
        WHERE   (operation_id =@my_execution_id) 
			AND (event_name NOT LIKE '%Validate%')
	) AS q
Where q.event_name='OnError'
ORDER BY message_time DESC

--///////////////////
----Full Log
--///////////////////
SELECT top 1000 event_message_id as id
	, datalength(message) message_length
	, message
	, try_cast('<![CDATA[' + message + ']]>' as XML) as message_xml
	, package_name, event_name, message_source_name,@my_execution_id execid,message_time
FROM   (SELECT        
		event_message_id, operation_id, message_time, message_type, message_source_type, message, extended_info_id, package_name,
        event_name, message_source_name, message_source_id, subcomponent_name, package_path, execution_path, threadID,
        message_code
        FROM            catalog.event_messages AS em
        WHERE   (operation_id =@my_execution_id) 
			AND (event_name NOT LIKE '%Validate%') 
			--and event_message_id=20303352
	) AS q
ORDER BY event_message_id DESC

Conclusion

So finally to conclude, you saw that how easy it is to execute SSIS packages using SQL code, monitor SSIS package execution and extract valuable log via. SSIS PowerPack comes with additional logging features along with standard logging. Try SSIS PowerPack to get additional 70+ new SSIS tasks and components in your SSIS toolbox.

The post Monitor, Run SSIS Package using Stored procedure / T-SQL appeared first on ZappySys Blog.

If you are using SSIS for your ETL needs and looking to reduce your overall cost then, there is a good news. Microsoft recently announced support to run SSIS in Azure Data Factory (SSIS as Cloud Service). Yes – that’s exciting,  you can now run SSIS in Azure without any change in your packages (Lift and Shift).  SSIS Support in Azure is a new feature of Azure Data Factory V2 (i.e. ADF). This article also explains how to deploy SSIS Packages to Azure which use 3rd party components (e.g. SSIS PowerPack v2.6.13 or Higher). There are few additional steps involved if you wish to install Custom SSIS Components in Azure Data Factory (explained later in this article).

What is Microsoft Azure Data Factory (i.e. ADF)

Azure Data Factory (i.e. also referred as “ADF”) is a fully managed cloud service by Microsoft for your ETL needs. ADF comes with two completely different ETL approaches (referred as Integration Runtimes).

• Default Runtime (Copy Activity)
• SSIS Runtime

Both runtimes offer their own Authoring tools and they have completely different authoring, execution, and pricing models. Check later in this article for some tips to save Cost by automating Start / Stop of SSIS IR based on your limited execution need (Only Start SSIS-IR when you need to run package).

Choose ADF SSIS Runtime when:

• You are planning to leverage existing ETL (i.e. SSIS Packages) in Azure without significant changes
• You like to use  3rd party SSIS components in ADF (e.g. SSIS PowerPack) which are not found in Copy Activity
• You want more predictable pricing model (Fixed Monthly cost regardless number of executions )
• You find certain key features or performance of SSIS better than ADF Copy Activity
• You don’t want to invest in learning new technologies for now but want to shift your workloads in the cloud to reduce cost.
• You like SSIS Package designer (i.e. SSDT) better than ADF (Maybe due to ease of Use in Disconnected mode on the client side or Visual Debugger)

Choose ADF Default Runtime (Copy Activity) when: 

• Your data movement which is more Azure-cloud focused. For example, Move data from CosmosDB to Azure SQL Data Warehouse. Or Move Data from Azure Data Bricks to Blob Storage.
• You like certain features/connectivity options in Copy-Activity are better than SSIS
• You like to pay per execution rather than Fixed SSIS Runtime cost.
• You don’t have expertise in SSIS and you like to start using ETL (Pay as you go)

Advantages of running SSIS on Azure Data Factory

There few obvious advantages if you decide to run SSIS Package in Azure Data Factory (ADF) compared to running on your own dedicated server (on-premises or VM running in Cloud).

• Reduced Cost – This can be one big advantage why you may choose to run SSIS in Azure. Traditionally to run SSIS Packages in-house you needed SQL Server License (which can be very expensive). Microsoft does not sell SSIS License independently. But now with ADF you can get going with your ETL need without SQL Server license and fraction of the cost.
• Managed Service – ADF is managed service means you don’t have to worry about hardware, OS, Licensing or ongoing maintenance/patches. Azure infrastructure will take all these for you.
• DevOps – Time to setup new server is significantly less with Azure Cloud approach.  You can set up your SSIS ETL infrastructure in less than 30 mins with Azure.
• Isolation of Compute and Database Storage – In Azure Data Factory when you provision SSIS runtime your Compute Node (Where SSIS service runs) is totally isolated than your SSISDB (DB where SSIS Packages / Settings / Execution Log stored). Which means if your compute node is offline for some reason you can still deploy packages and view execution log.

Anyway’s enough said now let’s see how to provision and deploy SSIS packages in Azure Data Factory (ADF).

How to deploy and run SSIS Packages in Azure with 3rd party components

Before we deep dive to learn how to run packages in Azure Data Factory. Here are high-level steps to deploy SSIS Packages in Azure Data Factory. Here is another tutorial link from Microsoft (It doesn’t mention deploying third-party SSIS components but good to check anyway). Some steps are optional and only needed if you planning to run SSIS Packages with third-party components such as SSIS PowerPack

1. Create Azure Data Factory instance- Check this article
2. Create Azure SQL DB Instance (Used to store SSISDB)  – Check this article
3. (3rd party SSIS) Create Azure Blob Storage Container and Upload 3rd party SSIS component setup files
4. (3rd party SSIS) Create SAS URL for Azure Blob Container which has necessary 3rd party component setup files (SAS URL must have Read/Write/List/Create permission)
5. Create a new SSIS Runtime in Azure Data Factory
6. (3rd party SSIS) During setup enter SAS URL which points to a container with custom setup files.
7. Use SSDT BI for VS 2015 or Higher to Author/Test SSIS Packages locally (Change Project TargetServerVersion to SQL 2017 Format)
8. Use the latest version of SSMS (v17.x or Higher) to deploy SSIS Packages to Azure SQL DB (SSISDB)
9. Execute Package via SSMS and Monitor Package Executions

In the following sections, we will see each step in detail.

Important things to know about SSIS Runtime in ADF

There are a few important facts you should know about how Azure Data Factory (ADF) Hosts, Runs your SSIS Packages

• ADF uses Windows Server VM nodes to run your SSIS Packages in Azure.
• It uses SSIS 2017 Format so make sure any Packages using 3rd party components must be targeted to the correct SSIS version
• Each time you start and stop SSIS Runtime in ADF you will get a fresh VM node so nothing is persistent in ADF SSIS Node (Microsoft Azure will allocate any random node from VM Pool)
• If you wish to persist files/values then do not save on ADF VM (Use external storage such as Blob Storage or SQL DB)

Choosing Between Azure Virtual Machine (Azure VM) and Azure Data Factory (ADF) for Running SSIS Packages

Many customers ask whether they should run their SSIS packages on an Azure Virtual Machine (VM) or inside Azure Data Factory (ADF) using the Azure-SSIS Integration Runtime (IR). Both deployment options are supported by ZappySys, but the best choice depends on your workload characteristics, cost model, and operational requirements.

Running SSIS packages in ADF with Azure-SSIS IR is fundamentally different from running them on an Azure VM. ADF provides a fully managed environment with automatic scaling, built-in scheduling, and native integration with other Azure services, while VMs offer more control over the operating system and custom configurations.

For a deeper comparison of these two approaches, refer to our detailed guide here:
Choosing Between Azure Virtual Machine (Azure VM) and Azure Data Factory (ADF) for Running SSIS Packages

Getting Started – Setup SSIS Runtime in Azure

Now let’s look at how to create your first Azure Data Factory Instance and then configure to run SSIS Packages with Custom Components such as SSIS PowerPack.

Prerequisite

Before we move forward with our steps to Deploy and Run SSIS Package in Azure Data Factory let’s confirm that you meet the following prerequisite.

• You have a valid Azure Subscription. If you don’t have it then you can try for FREE here.
• Download and Install Azure Storage Explorer for FREE (Used to upload setup files and create SAS Url for 3rd party SSIS components setup)
• Download and Install SSMS v17.x or Higher for FREE. This is not same SSMS which comes with SQL Server Setup (This new version is independent download and support connecting to Azure SQL DB for SSISDB
• Download and Install SSDT BI for VS 2015 or higher (Used to design and test SSIS Packages with SQL Server 2017 support).
• You have downloaded setup files for SSIS PowerPack (To obtain setup file – For FREE Trial Click Here,  For Paid Customer Click Here)

Create a new Azure Data Factory (ADF)

Using your valid Azure Subscription create new Azure Data Factory Instance (Thi will host your SSIS Runtime). Click here to learn more how to create a new Azure Data Factory Instance.

Create a new Azure SQL Database (i.e. new SQL Instance)

Once you have a valid ADF Instance, next step is to make sure you have a valid Azure SQL Server Instance to host SSISDB (Catalog Database for SSIS). If possible choose the same region for both ADF and SQL Instance. ). Click here to learn more on how to create a new Azure SQL Server Instance.

Create and Setup Azure Blob Storage Container (Only for 3rd party SSIS setup)

Once you have provisioned ADF instance and Azure SQL Server instance, Now it’s time to prepare setup files for custom SSIS components. Behind the scene, ADF uses VM instances to run your SSIS Packages so you can install pretty much anything you want just like a regular VM. However, there is no Remote Desktop Option for SSIS node in ADF, in other words, it’s pretty locked down. The only way you can install custom components is through Custom Setup Option provided during your SSIS Runtime creation. This custom script is executed only once after SSIS VM is created in ADF. It also executed once you stop and start SSIS Runtime. You have no way to execute the custom script on demand while SSIS Runtime is running (Maybe Microsoft can change this in the future but no option for now).

Prepare custom setup files for ADF – SSIS Runtime

During SSIS Runtime setup you can instruct ADF to perform a custom setup (e.g. Run the custom installer, copy files, execute PowerShell scripts etc). Your custom setup files are downloaded in the SSIS node (i.e. VM) at some point during SSIS Runtime setup. This is done using SAS URL (i.e. URL with Shared Access Signature). Treat SAS URL like a Password if you have some sensitive data in your container. For an additional read on the custom component setup, you can refer this link.

Before we create SAS URL, you need to gather setup files on your local disk and then you can upload to Azure Blob Container. SSIS PowerPack needs the following 2 files to be uploaded in your blob container (root level).

1. main.cmd [See below Section] – This is the entry point of Custom Setup in ADF-SSIS
2. SSISPowerPackSetup_64bit_Trial.msi (Trial version, download it) OR SSISPowerPackSetup_64bit.msi (Paid Customer version, download it)

Each file listed above has download link next to it so download locally (except main.cmd which we will discuss next).

Create a new file and name it as “main.cmd“. Enter content like below.

Few things you should change as per your need

1. Change the Example License Key to your own License Key in Activate PowerPack license Step.
2. Change SSISPowerPackSetup_64bit.msi if your downloaded file has a different name
3. Comment Activate PowerPack license Step if you are using Trial (To comment just add

   ::

     in front of the line)

Other than the above three changes you can leave everything default. Save the file and upload to your setup blob container. Once ADF executes main.cmd it will generate new Log folder in your Blob container (You will see a new folder. Example 

main.cmd.log\tvm-325613345_1-20180503t144850z##2018-05-03T145334Z

main.cmd – For Trial Users

See below example script file which you have to create if you are a trial user (you don’t have a license key). The file must be named as main.cmd.

set DIR=%CUSTOM_SETUP_SCRIPT_LOG_DIR%

echo Calling Step 1 : %TIME% >> "%DIR%\steps_log.txt"
dir /s /b > "%DIR%\file_list.txt"

echo Calling Step 2 : %TIME% >> "%DIR%\steps_log.txt"

::Install SSIS PowerPack
msiexec /i  "SSISPowerPackSetup_64bit_Trial.msi" ADDLOCAL=ALL /q  /L*V "%DIR%\powerpack_trial_install_log.txt"

echo Calling Step 3 : %TIME% >> "%DIR%\steps_log.txt"

::Activate SSIS PowerPack Trial Extension Key (Optional) -- Uncomment below line to apply the trial extension key.
::"C:\Program Files (x86)\ZappySys\SSIS PowerPack (64 bit)\LicenseManager.exe" -p SSISPowerPack --register "lgGAAO0-----REPLACE-WITH-YOUR-TRIAL-EXTENSION-KEY-----czM=" --logfile "%DIR%\powerpack_register_log.txt"

dir "C:\Program Files\Microsoft SQL Server\*Zappy*.*"  /s /b >> "%DIR%\installed_files.txt"
dir "C:\Program Files (x86)\Microsoft SQL Server\*Zappy*.*"  /s /b >> "%DIR%\installed_files.txt"

echo DONE : %TIME% >> "%DIR%\steps_log.txt"

echo complete

main.cmd – For Paid Customers

See below example script file which you have to create if you are a paid customer and wish to apply license key. The file must be named as main.cmd. Make sure:

1. You uploaded the installer with the name SSISPowerPackSetup_64bit.msi.
2. Replaced the parameter’s –register argument with your real license key.

set DIR=%CUSTOM_SETUP_SCRIPT_LOG_DIR%

echo Calling Step 1 : %TIME% >> "%DIR%\steps_log.txt"
dir /s /b > "%DIR%\file_list.txt"

echo Calling Step 2 : %TIME% >> "%DIR%\steps_log.txt"

::Install SSIS PowerPack
msiexec /i  "SSISPowerPackSetup_64bit.msi" ADDLOCAL=ALL /q  /L*V "%DIR%\powerpack_install_log.txt"

echo Calling Step 3 : %TIME% >> "%DIR%\steps_log.txt"

::Activate PowerPack license (Optional)
"C:\Program Files (x86)\ZappySys\SSIS PowerPack (64 bit)\LicenseManager.exe" -p SSISPowerPack --register "lgGAAO0-----REPLACE-WITH-YOUR-LICENSE-KEY-----czM=" --logfile "%DIR%\powerpack_register_log.txt"

::De-Activate license (Optional) -- Uncomment below line to Unregister previously used license. Must register first and then unregister
::"C:\Program Files (x86)\ZappySys\SSIS PowerPack (64 bit)\LicenseManager.exe" -p SSISPowerPack --unregister --logfile "%DIR%\powerpack_un_register_log.txt"


::Show System Info
echo Calling Step 4 : %TIME% >> "%DIR%\steps_log.txt"
"C:\Program Files (x86)\ZappySys\SSIS PowerPack (64 bit)\LicenseManager.exe" -i -l "%DIR%\sysinfo_log.txt"

echo Calling Step 5 : %TIME% >> "%DIR%\steps_log.txt"
dir "C:\Program Files\Microsoft SQL Server\*Zappy*.*"  /s /b >> "%DIR%\installed_files.txt"
dir "C:\Program Files (x86)\Microsoft SQL Server\*Zappy*.*"  /s /b >> "%DIR%\installed_files.txt"

echo DONE : %TIME% >> "%DIR%\steps_log.txt"

echo complete

Create SAS URL of Azure Blob Container (Only for 3rd party SSIS setup)

Once you have prepared setup files now let’s upload them to your blob container and generate SAS URL using the below steps.

1. Install and Launch Azure Storage Explorer
2. Right-click on the Storage Accounts node > Click Connect to Azure storage …
   

   Connect to Azure Storage Account in Azure Storage Explorer

    

3. Once your account added > Expand to Blob Containers node > Right click > Create Blob Container…
   

   Create new Azure Blob Container in Azure Storage Explorer

4. Select newly created container and upload files we prepared in the previous section.
   

   Upload Custom Setup File for Azure Data Factory – SSIS Runtime

5. Right-click on the container node and click Get Shared Access Signature
   
   Make sure you adjust the expiration date to a long enough; otherwise, you may need to generate a new SAS URL often and restart your ADF SSIS Node to apply the new SAS URL. Also, select Write permission along with Read,  List, and Create. If you do not select Write, then ADF setup cannot write log files back to your Blob Container. We also recommend Delete permission in case Microsoft adds a new feature to delete older log files.
   .Generate SAS URL from Azure Storage Explorer (Desktop Version)
   

   Create SAS URL in Azure Storage Explorer for Azure Data Factory (ADF) – SSIS Runtime Custom Setup (3rd party component installation)

   Generate SAS URL from Azure Portal Online

   

   Create SAS URL in Azure Portal for Azure Data Factory (ADF) – SSIS Runtime Custom Setup (3rd party component installation)

    

6. Copy SAS URL to clipboard and save somewhere (we will use that user later in this article)
   

   Get SAS URL for Azure SSIS Runtime – Custom setup

Create a new SSIS – Integration Runtime for SSIS

Once you upload setup files for custom SSIS components and generated SAS URL for container we are ready to move on to our final step (Create SSIS Integration Runtime in Azure Data Factory)

Steps for Creating SSIS Runtime in Azure are clearly documented here except one change (see below).

Enter Custom Setup SAS URL during SSIS RUntime Creation in Azure Data Factory

Once your SSIS Runtime setup is done you can see your Node running in Dashboard (Status must be Green checkbox with  Running ).

Verify SSIS Runtime Node Status in Azure Data Factory Portal

Unregister / Move ZappySys License

There will be a time when you no longer use ADF instance or you like to use your license on a different ADF Instance. As of now there is no Customscript option for Stop action in ADF so perform the following work around to fire Unregister License command. Basically you have to restart your IR (the one which is holding your license) once to fire custom script which should contain license register action first and then unregister command right after it.

Here is full script to unregister your license. Make sure to use this script for old ADF instance which is being deleted or from where you like to unregister ZappySys license .

Use below code in main.cmd

set DIR=%CUSTOM_SETUP_SCRIPT_LOG_DIR%
 
echo Calling Step 1 : %TIME% >> "%DIR%\steps_log.txt"
dir /s /b > "%DIR%\file_list.txt"
 
echo Calling Step 2 : %TIME% >> "%DIR%\steps_log.txt"

::Install SSIS PowerPack
msiexec /i  "SSISPowerPackSetup_64bit.msi" ADDLOCAL=ALL /q  /L*V "%DIR%\powerpack_install_log.txt"
 
echo Calling Step 3 : %TIME% >> "%DIR%\steps_log.txt"
 
::Activate PowerPack license
"C:\Program Files (x86)\ZappySys\SSIS PowerPack (64 bit)\LicenseManager.exe" -p SSISPowerPack --register "lgGAAO0QmCxxxxxxxxxxxxxxxxWSxczM=" --logfile "%DIR%\powerpack_register_log.txt"

::De-Activate same license -- This is the work around to unregister license from ADF
"C:\Program Files (x86)\ZappySys\SSIS PowerPack (64 bit)\LicenseManager.exe" -p SSISPowerPack --unregister --logfile "%DIR%\powerpack_un_register_log.txt"
 
::Show System Info
echo Calling Step 4 : %TIME% >> "%DIR%\steps_log.txt"
"C:\Program Files (x86)\ZappySys\SSIS PowerPack (64 bit)\LicenseManager.exe" -i -l "%DIR%\sysinfo_log.txt"
 
echo Calling Step 5 : %TIME% >> "%DIR%\steps_log.txt"
dir "C:\Program Files\Microsoft SQL Server\*Zappy*.*"  /s /b >> "%DIR%\installed_files.txt"
dir "C:\Program Files (x86)\Microsoft SQL Server\*Zappy*.*"  /s /b >> "%DIR%\installed_files.txt"
 
echo DONE : %TIME% >> "%DIR%\steps_log.txt"
 
echo complete

Author SSIS Packages to Run in Azure Datafactory

Once your SSIS Runtime is set up and running your next step is to Author and Deploy SSIS Packages.

• To create SSIS Packages which can run in Azure Data Factory you must need SSDT BI for VS 2015 or higher

Change SSIS Project TargetServerVersion

Before you deploy package to ADF you must change to to the supported Runtime by ADF. As of now 2017 (i.e. SQL Server 2017) is supported runtime so choose that one.

• Make sure packages are set to SQL 2017 Compatibility mode (This is a must if you are using ZappySys SSIS PowerPack)

Change SSIS TargetServerVersion Property

Using Visual Studio to Deploy SSIS Packages to Azure (i.e. ADF SSIS-IR)

If you are using latest version of Visual Studio (VS2017 or Higher) then you can also deploy SSIS Project right from Visual Studio (See below).

Deploy SSIS Packages to Azure Data Factory from Visual Studio 2017 / 2019

Using SSMS to Deploy / Execute SSIS Packages to Azure (i.e. ADF SSIS-IR)

Now let’s talk how to deploy SSIS Packages from your local machine to SSISDB in Azure. The process is pretty same just like how you do when deploying SSIS to On-Premises SQL Server Instance except for few steps.

• One important thing to remember is to deploy SSIS Packages to Azure Data Factory you must use SSMS v17.x or Higher (Older version of SSMS doesn’t support ADF SSIS Deployment)
• Another important thing is when you connect to SQL Server Instance in Azure make sure you Click on Options on the Connect dialogbox and enter SSISDB in the Connect to database

Other than those two things all steps to deploy SSIS your packages to Azure Data Factory are listed here.

Here are the steps to connect to SSISDB Catalog for ADF SSIS-IR

1. Launch SSMS and click Connect -> Database Engine
2. Now on connection dialog box, Enter your Azure SQL Server FullyQualified Domain Name , UserID and  Password
3. Go to Connection Properties Tab like below and enter SSIS Catalog DB as SSISDB
4. Click OK and thats it. You will see Integration Services Catalog node in SSMS. If you didnt enter Catalog name as previous step then this special node is not visible in SSMS.

Connect to Database from SSMS

Connect to Azure Data Factory – SSISDB via SSMS 17.x Or Higher ( Deploy / Monitor SSIS Packages)

Deploy SSIS Package to Azure Data Factory using SSMS UI (Wizard mode)

Test / Execute and Monitor SSIS Package via SSMS UI

To execute SSIS Package via SSMS UI you can use below steps.

1. Connect to SSISDB server using the technique listed in the previous section
2. Navigate to Integration Services Catalog > Your Folder > Your Project > Your Package > Right Click and execute
   

   Execute SSIS Package using SSMS UI

3. To monitor or view past execution, Navigate to Integration Services Catalog > Your Folder > Your Project > Your Package > Right Click and Reports > Standard Reports > View All Executions
   

   Monitor SSIS Package Execution using SSMS UI

Schedule SSIS Package in Azure Data Factory

In Azure there is no SQL Agent like feature we have for on-premises SQL Server so how do you schedule packages when you deploy to ADF ? Well there is slightly different approach in ADF. You need to use feature called Trigger. Check this article to learn how to schedule SSIS Package in Azure (Use Trigger)

Here are few more approaches to to schedule SSIS Packages deployed in Azure Data Factory

If you want to schedule SSIS Package stored in Azure Data Factory on your On-Premises server then you can use Linked Server approach mentioned in the next section. You can create a new T-SQL Job Step and call a user-defined stored procedure which can trigger SSIS package execution as per next T-SQL code.

Execute and Monitor SSIS Package via T-SQL Code in Azure Data Factory

In a real-world scenario, you have to execute SSIS Packages in an automated way from code or trigger execution on certain schedule. Microsoft has documented this scenario here but below section will add few more details which you may find very useful. Basically, if you wish to call or schedule Azure Data Factory SSIS packages from your remote server job (i.e .Calling SSIS from On-Premises Agent Job via Job > Job Step > Integration Services Step ) then there are few caveats you should know. SQL Server Agent Job doesn’t allow Mixed Mode Authentication for SSISDB to call SSIS Packages (If you check Job Step UI in SQL Agent supports only Windows Authentication).

Well, not to worry solution is not too hard. There is an easy way you can trigger remote SSIS package execution by using Linked Server approach (i.e .Call SSIS Packages stored in Azure Data Factory SSISDB).

So let’s look at how to execute and monitor ADF SSIS Packages using Linked Server.

Create a Linked server for SSISDB (Azure SQL DB)

To execute and monitor SSIS Package execution from a remote machine first step is to create a Linked Server which points to a remote server which contains SSISDB catalog. This way you can call T-SQL Step in your Agent Job to trigger SSIS execution on the remote machine (e.g. Azure Data Factory SSISDB). When you create linked server make sure to enable rpc property as below.

USE [master]
GO
--DROP Linked Server
--EXEC master.dbo.sp_dropserver @server=N'SSISDB_ON_ADF', @droplogins='droplogins'

--CREATE Linked Server
EXEC master.dbo.sp_addlinkedserver 
	@server = N'SSISDB_ON_ADF', 
	@srvproduct=N'', 
--- For MSSQL 2012, 2014, 2016, 2017, and 2019 use below (SQL Server Native Client 11.0)---
    @provider=N'SQLNCLI11',
--- For MSSQL 2022 or higher use below (Microsoft OLE DB Driver for SQL Server)---
--- @provider=N'MSOLEDBSQL',
	@datasrc=N'my-azure-db-server001.database.windows.net,1433', 
	@catalog=N'SSISDB'

EXEC master.dbo.sp_addlinkedsrvlogin 
	@rmtsrvname=N'SSISDB_ON_ADF',
	@useself=N'False',
	@locallogin=NULL,
	@rmtuser=N'zsadmin',
	@rmtpassword='MyAdminUserPass@123'
GO
EXEC master.dbo.sp_serveroption @server=N'SSISDB_ON_ADF', @optname=N'rpc', @optvalue=N'true'
GO
EXEC master.dbo.sp_serveroption @server=N'SSISDB_ON_ADF', @optname=N'rpc out', @optvalue=N'true'
GO

Once you have linked server created for SSISDB you can call it from your Agent Job Step. Use T-SQL step to call user-defined stored procedure which can call SSIS package execution mentioned in next few sections.

Execute SSIS Package via T-SQL Code (Set Parameters, Environment)

Now connect to your server where linked server is created (i.e. your On-Premises server). There are mainly 3 steps to trigger SSIS package execution via code.

1. Create execution instance, Optionally you can supply SSIS Environment ID (grab execution id and store into some variable for later use)
2. Set System, User Parameters
3. Start package execution

You can set some SYSTEM variables for your execution or set Package level or Project level User defined SSIS Variables for your execution. You can also set SSIS environment ID (@reference_id when you create execution)

declare @my_execution_id bigint

exec [SSISDB_ON_ADF].SSISDB.catalog.create_execution 
	@folder_name = 'MyFolder'
	,@project_name = 'MySSISProject'
	,@package_name = 'MyPackage.dtsx'
	,@reference_id = null
	,@execution_id = @my_execution_id output

	
--///////////////////////////
--Set Parameters
--///////////////////////////
--https://docs.microsoft.com/en-us/sql/integration-services/system-stored-procedures/catalog-set-execution-parameter-value-ssisdb-database?view=sql-server-2017
exec [SSISDB_ON_ADF].SSISDB.catalog.set_execution_parameter_value  @my_execution_id, 50, 'FilePath', 'C:\Temp\Dump.json'  
exec [SSISDB_ON_ADF].SSISDB.catalog.set_execution_parameter_value  @my_execution_id, 50, 'TargetServerName', 'Server001'  

--///////////////////////////
--Start Execution
--///////////////////////////
--https://docs.microsoft.com/en-us/sql/integration-services/system-stored-procedures/catalog-start-execution-ssisdb-database?view=sql-server-2017
exec [SSISDB_ON_ADF].SSISDB.catalog.start_execution @my_execution_id

To wait until package execution is done, you can set SYNCHRONIZED=1 like below

declare @my_execution_id bigint

exec [SSISDB_ON_ADF].SSISDB.catalog.create_execution 
	@folder_name = 'MyFolder'
	,@project_name = 'MySSISProject'
	,@package_name = 'MyPackage.dtsx'
	,@reference_id = null
	,@execution_id = @my_execution_id output


--///////////////////////////
--Wait until execution is done
--///////////////////////////
-- IF You want to wait until execution is done. If you enable this then no need for status check loop
--https://docs.microsoft.com/en-us/sql/integration-services/system-stored-procedures/catalog-set-execution-parameter-value-ssisdb-database?view=sql-server-2017
EXEC [SSISDB_ON_ADF].[SSISDB].[catalog].[set_execution_parameter_value] 
	@my_execution_id,  
	@object_type=50, 
	@parameter_name=N'SYNCHRONIZED', 
	@parameter_value=1

--///////////////////////////
--Start Execution
--///////////////////////////
--https://docs.microsoft.com/en-us/sql/integration-services/system-stored-procedures/catalog-start-execution-ssisdb-database?view=sql-server-2017
exec [SSISDB_ON_ADF].SSISDB.catalog.start_execution @my_execution_id

If you are using SSIS Environment to specify parameter set at runtime then use below approach to first get correct environment reference id and set for your execution.

declare @my_execution_id bigint
declare @my_env_refid int

select @my_env_refid=reference_id from [SSISDB_ON_ADF].[SSISDB].[catalog].environment_references 
where environment_name='DEV_ENV'

exec [SSISDB_ON_ADF].SSISDB.catalog.create_execution 
	@folder_name = 'MyFolder'
	,@project_name = 'MySSISProject'
	,@package_name = 'MyPackage.dtsx'
	,@reference_id = @my_env_refid
	,@execution_id = @my_execution_id output

--///////////////////////////
--Start Execution
--///////////////////////////
--https://docs.microsoft.com/en-us/sql/integration-services/system-stored-procedures/catalog-start-execution-ssisdb-database?view=sql-server-2017
exec [SSISDB_ON_ADF].SSISDB.catalog.start_execution @my_execution_id

Monitor SSIS Package Execution via Code (view Log / Errors)

Here is the query to find out execution status of your SSIS package. Notice how each numeric code maps to the status description.

select folder_name,project_name,package_name,status_text=case 
	when status=1 then 'Created' 
	when status=2 then 'Running' 
	when status=3 then 'Cancelled' 
	when status=4 then 'Failed' 
	when status=5 then 'Pending' 
	when status=6 then 'Ended Unexpectedly' 
	when status=7 then 'Succedded' 
	when status=8 then 'Stopping' 
	when status=9 then 'Completed' 
	else 'Not Sure'
	end
from [SSISDB_ON_ADF].SSISDB.catalog.executions  where execution_id=@my_execution_id

To view SSIS package execution log you can use below query.  Notice how we have used XML datatype. This is a workaround to see the long string in SSMS result grid. If you have more than 8000 characters in your log record then SSMS cant display by default.

--///////////////////
--Full Log
--///////////////////
--SELECT cast('<![CDATA[' + message + ']]>' as XML) 
SELECT top 1000 event_message_id as id
	, datalength(message) message_length
	, message
	, try_cast('<![CDATA[' + message + ']]>' as XML) as message_xml
	, message
	, package_name, event_name, message_source_name,@my_execution_id execid,message_time
FROM   (SELECT        
		event_message_id, operation_id, message_time, message_type, message_source_type, message, extended_info_id, package_name,
        event_name, message_source_name, message_source_id, subcomponent_name, package_path, execution_path, threadID,
        message_code
        FROM            [SSISDB_ON_ADF].SSISDB.catalog.event_messages AS em
        WHERE   (operation_id =@my_execution_id) 
			AND (event_name NOT LIKE '%Validate%') 
			--and event_message_id=20303352
	) AS q
--Where q.event_name='OnError'
ORDER BY event_message_id DESC

To view only errors for SSIS Package execution use below query (replace @my_execution_id with your own value)

--///////////////////
----Show Just Errors
--///////////////////
SELECT top 100 @my_execution_id ExecID,message, package_name, event_name, message_source_name,message_time as start_time
FROM   (SELECT        
		event_message_id, operation_id, message_time, message_type, message_source_type, message, extended_info_id, package_name,
        event_name, message_source_name, message_source_id, subcomponent_name, package_path, execution_path, threadID,
        message_code
        FROM            [SSISDB_ON_ADF].SSISDB.catalog.event_messages AS em
        WHERE   (operation_id =@my_execution_id) 
			AND (event_name NOT LIKE '%Validate%')
	) AS q
Where q.event_name='OnError'
ORDER BY message_time DESC

Full T-SQL Code

Here is the full source code of all steps we discussed.

/*
 Author: ZappySys
 Description: Execute, Monitor, run SSIS on Azure Data Factory
 - This script demonstrates few aspects of SSIS package execution via code (call system stored procedures)
	>> How to execute SSIS package via sa tored procedure call
	>> How to set package parameters via code
	>> How to set environment via code
	>> How to monitor SSIS package execution via T-SQL code
	>> How to check SSIS package execution status / error and execution log
 - Feel free to modify and comment uncomment certain sections to meet your need.
 - We used Linked server approach with 4 parts naming in objects e.g. [SSISDB_ON_ADF].SSISDB.catalog.objectnamHere
 - If you are running this script directly on the server where SSISDB is stored then no need for 4 parts. (just use 3 parts SSISDB.catalog.objectnamHere) 
 we are outputting message column in xml format too so you get na ice hyper link to see the full message. This technique is needed to overcome string length limitation of SSMS

USE below script to create linked server -- Needed for Azure Data Factory - SSIS Runtime (Remote execution and Monitoring)

USE [master]
GO
--DROP Linked Server
--EXEC master.dbo.sp_dropserver @server=N'SSISDB_ON_ADF', @droplogins='droplogins'

--CREATE Linked Server
--For MSSQL 2012, 2014, 2016, 2017, and 2019 use below (SQL Server Native Client 11.0)---
EXEC master.dbo.sp_addlinkedserver @server = N'SSISDB_ON_ADF', @srvproduct=N'', @provider=N'SQLNCLI11', @datasrc=N'my-azure-db-server001.database.windows.net,1433', @catalog=N'SSISDB'
--For MSSQL 2022 or higher use below (Microsoft OLE DB Driver for SQL Server)---
--EXEC master.dbo.sp_addlinkedserver @server = N'SSISDB_ON_ADF', @srvproduct=N'', @provider=N'MSOLEDBSQL', @datasrc=N'my-azure-db-server001.database.windows.net,1433', @catalog=N'SSISDB'

EXEC master.dbo.sp_addlinkedsrvlogin @rmtsrvname=N'SSISDB_ON_ADF',@useself=N'False',@locallogin=NULL,@rmtuser=N'zsadmin',@rmtpassword='MyAdminUserPass@123'
GO
EXEC master.dbo.sp_serveroption @server=N'SSISDB_ON_ADF', @optname=N'rpc', @optvalue=N'true'
GO
EXEC master.dbo.sp_serveroption @server=N'SSISDB_ON_ADF', @optname=N'rpc out', @optvalue=N'true'
GO
--Below needed to support EXEC + INSERT (dynamic query)
EXEC master.dbo.sp_serveroption @server=N'SSISDB_ON_ADF, @optname=N'remote proc transaction promotion', @optvalue=N'false'
GO
 */

declare @my_folder_name varchar(255)= 'AdfExamples'
declare @my_project_name varchar(255) = 'SSISNativeExamples'
declare @my_package_name varchar(255) = 'HelloWorld.dtsx'
declare @my_environment_name varchar(255) = null  --e.g. This is SSIS Environment to attach with this execution (e.g. ENV_DEV , ENV_PROD)
--declare @my_environment_name varchar(255) = 'ENV_DEV'
	  
declare @my_execution_id bigint=null 
--declare @my_execution_id bigint=1200 --enter execid to see log

declare @starttime datetime
declare @statusLoop bit

--///////////////////////
--UN-COMMENT THIS if you just want to see last execution log rather than execute new one
--IF you just need to query status of last execution 
--///////////////////////
if(@my_execution_id is null)
	Select @statusLoop=0, @my_execution_id=Max(execution_id), @startTime = Max(start_time) From [SSISDB_ON_ADF].SSISDB.catalog.executions 


if(@my_execution_id is null)
begin
	set @statusLoop=1 
	
	declare @my_env_refid int
	select @my_env_refid=reference_id from [SSISDB_ON_ADF].[SSISDB].[catalog].environment_references where environment_name=@my_environment_name

	--First create Execution and get ExecutionID in a variable
	--https://docs.microsoft.com/en-us/sql/integration-services/system-stored-procedures/catalog-create-execution-ssisdb-database?view=sql-server-2017
	exec [SSISDB_ON_ADF].SSISDB.catalog.create_execution 
	  @folder_name = @my_folder_name
	 ,@project_name = @my_project_name
	 ,@package_name = @my_package_name
	 ,@reference_id = @my_env_refid
	 ,@execution_id = @my_execution_id output

	 --///////////////////////////
	--Wait until execution is done
	--///////////////////////////
	 -- IF You want to wait until execution is done. If you enable this then no need for status check loop
	 --https://docs.microsoft.com/en-us/sql/integration-services/system-stored-procedures/catalog-set-execution-parameter-value-ssisdb-database?view=sql-server-2017
	EXEC [SSISDB_ON_ADF].[SSISDB].[catalog].[set_execution_parameter_value] 
		@my_execution_id,  
		@object_type=50, 
		@parameter_name=N'SYNCHRONIZED', 
		@parameter_value=1
	set @statusLoop=0 --if you enable this option no need for statucheck
	
	--///////////////////////////
	--Set Parameters
	--///////////////////////////
	--https://docs.microsoft.com/en-us/sql/integration-services/system-stored-procedures/catalog-set-execution-parameter-value-ssisdb-database?view=sql-server-2017
	--exec [SSISDB_ON_ADF].SSISDB.catalog.set_execution_parameter_value  @my_execution_id, 50, 'FilePath', 'C:\Temp\Dump.json'  
	--exec [SSISDB_ON_ADF].SSISDB.catalog.set_execution_parameter_value  @my_execution_id, 50, 'TargetServerName', 'Server001'  

	--///////////////////////////
	--Start Execution
	--///////////////////////////
	--https://docs.microsoft.com/en-us/sql/integration-services/system-stored-procedures/catalog-start-execution-ssisdb-database?view=sql-server-2017
	exec [SSISDB_ON_ADF].SSISDB.catalog.start_execution @my_execution_id
end

--///////////////////////////
--Status check loop for wait
--///////////////////////////
---comment these two lines if set SYNCHRONIZED = 1
if(@statusLoop = 1)
WHILE @my_execution_id IN (1,2,5,8)
	WAITFOR DELAY '00:03'
--

 select folder_name,project_name,package_name,status_text=case 
	when status=1 then 'Created' 
	when status=2 then 'Running' 
	when status=3 then 'Cancelled' 
	when status=4 then 'Failed' 
	when status=5 then 'Pending' 
	when status=6 then 'Ended Unexpectedly' 
	when status=7 then 'Succedded' 
	when status=8 then 'Stopping' 
	when status=9 then 'Completed' 
	else 'Not Sure'
	end
from [SSISDB_ON_ADF].SSISDB.catalog.executions  where execution_id=@my_execution_id


--///////////////////
----Show Just Errors
--///////////////////
SELECT top 10 @my_execution_id ExecID,message, package_name, event_name, message_source_name,message_time as start_time
FROM   (SELECT        
		event_message_id, operation_id, message_time, message_type, message_source_type, message, extended_info_id, package_name,
        event_name, message_source_name, message_source_id, subcomponent_name, package_path, execution_path, threadID,
        message_code
        FROM            [SSISDB_ON_ADF].SSISDB.catalog.event_messages AS em
        WHERE   (operation_id =@my_execution_id) 
			AND (event_name NOT LIKE '%Validate%')
	) AS q
Where q.event_name='OnError'
ORDER BY message_time DESC

--///////////////////
----Full Log
--///////////////////
--SELECT cast('<![CDATA[' + message + ']]>' as XML) 
SELECT top 1000 event_message_id as id
	, datalength(message) message_length
	, message
	, try_cast('<![CDATA[' + message + ']]>' as XML) as message_xml
	, message
	, package_name, event_name, message_source_name,@my_execution_id execid,message_time
FROM   (SELECT        
		event_message_id, operation_id, message_time, message_type, message_source_type, message, extended_info_id, package_name,
        event_name, message_source_name, message_source_id, subcomponent_name, package_path, execution_path, threadID,
        message_code
        FROM            [SSISDB_ON_ADF].SSISDB.catalog.event_messages AS em
        WHERE   (operation_id =@my_execution_id) 
			AND (event_name NOT LIKE '%Validate%') 
			--and event_message_id=20303352
	) AS q
--Where q.event_name='OnError'
ORDER BY event_message_id DESC

Creating ADF SSIS – IR using PowerShell

You can also create Azure Data Factory – SSIS Integration Runtime using PowerShell Code. Check this article. Make sure you enter SAS URL for

$SetupScriptContainerSasUri

Save Cost – Run SSIS-IR on demand (Start / Stop Runtime from Pipeline Activity)

There will be a time when you like to execute package once a day or once a week. Which means you will pay Cost of your SSIS-IR for unused time too. If you like to save cost on your SSIS runtime unused time you can check below links for similar topic.

https://docs.microsoft.com/en-us/azure/data-factory/how-to-schedule-azure-ssis-integration-runtime

https://andyleonard.blog/2019/02/start-azure-ssis-execute-package-stop-azure-ssis/

http://microsoft-ssis.blogspot.com/2018/04/start-and-stop-integration-runtime-in.html

Basically idea is to Start SSIS Runtime just before you need to execute SSIS Package and Stop SSIS Runtime when you are done so you don’t incur any charges of unused SSIS-IR hours. This technique can save you huge amount of $$$ if you are executing SSIS Packages occasionally in ADF.

Access On-Premises Files in ADF-SSIS IR node

Utilize the serverless capabilities of Azure Data Factory to seamlessly interact with an Access file system. By configuring your ADF pipelines, you can perform operations on Access files while taking advantage of serverless computing resources. At some point, you may encounter the need to interact with a file system through SSIS Tasks, such as Excel Source or File System Task, while your jobs are executing in a cloud environment. Wondering how to address this challenge? Explore the article below to discover effective workarounds for implementing these solutions.

Open and save files on-premises and in Azure with SSIS packages deployed in Azure

Using Static IP for ADF Cluster (VNet) – For External System Firewall Whitelisting

There will be a time when your ADF SSIS-IR needs to access an external system (e.g., AWS RDS-SQL Server) where only whitelisted IPs are allowed to access resources. In such a case, you must assign a static IP to ADF so other systems can allow that IP address in the Firewall White list.

For this purpose you have to join your ADF into VNet (Virtual Network) and then assign Static IP as explained in this article.

Troubleshooting Common Errors

Cannot find custom setup script (main.cmd) in your Azure blob container

When you face the below error, make sure you regenerate the SAS URL with “Create” permission along with “Read,” “Write,” and “List.” In some cases, Create permission is needed, and Cluster start fails.

Conclusion

New features in Microsoft Azure Data Factory V2 gives you truly managed option at low cost to Deploy, Execute and Monitor SSIS Packages. Using ZappySys SSIS PowerPack you can add 70+ new tasks/connectors to your Azure Data Factory SSIS Runtime. Download and try SSIS PowerPack to find out many new API / Cloud focused features for a connected world.

The post How to run SSIS in Azure Data Factory (Deploy, Monitor SSIS in Cloud) appeared first on ZappySys Blog.

To consume Zendesk data using the REST API, you can use different methods. Now, let’s see how to create an OAuth Application so you don’t have to rely on a static Token or a UserID/Password to access data.

The OAuth method is more secure and recommended over other methods, so whenever possible, use it.

Create Zendesk OAuth Application

To create an OAuth App for Zendesk, perform the following steps.

1. Log in to your Zendesk Portal. And go tothe  following URL (This step assumes you have admin rights to createan  OAuth App)
   

   https://zappysys.zendesk.com/agent/admin/api/oauth_clients

   

   How to create a Zendesk OAuth Application for REST API Access

2. After clicking on the plus sign and entering app information (e.g., Display name, Unique ID, also known as ClientId or AppId )
   

   Configure Zendesk OAuth App

3. All the way at the bottom of that page, you will see Secret (i.e., Client Secret). Save that along with a unique identifier for later use.
4. That’s it. Now you can start using the Zendesk OAuth App to access the REST API

Understanding Scopes

In the next section, you will learn how to configure an OAuth connection. During that, you will need to specify scopes (Token Permissions you want to grant).

The syntax to define the scope is as follows. Scope can be read-only, write-only, or read-write. The resource is an optional part. If you omit the scope, it will be applied to all.

resource:scope

You can use the following resource scope for the following resources:

read
write
read write
users:read
users:read users:write
users:read users:write organizations:write

Examples of scopes for Zendesk Chat APIs

read
chat

Calling Zendesk REST API using SSIS PowerPack OAuth Connection

Here is how you can use Zerdesk OAuth to access the API using the SSIS PowerPack JSON / REST Source

1. First, drag the ZS JSON Source from the SSIS Toolbox into the Data Flow designer
2. Configure JSON Source as below. Make sure you update the URL to your domain.
   URL for Zendesk Tickets API
   

   https://{YOUR-DOMAIN}.zendesk.com/api/v2/tickets

   URL for Zendesk Chat API
   

   https://{YOUR-DOMAIN}.zopim.com/api/v2/chats

3. Configure the OAuth Connection As below for the Zendesk Ticket API. Scopes differ for the Ticket and Chat APIs; please enter the appropriate ones.
   

   Configure SSIS OAuth connection for Zendesk API Access

   

   Configure OAuth Redirect URL and Content Type

4. Configure the OAuth Connection As below for the Zendesk Chat API. Scopes differ for the Ticket and Chat APIs; please enter the appropriate ones.
   

   Zendesk Chat OAuth Settings – General Tab

   

   Zendesk Chat OAuth Settings – Advanced Tab

   

   Zendesk Chat OAuth Settings – OAuth2 Grant Options Tab

5. After you set the necessary parameters, click Generate Token (Login using your account with the required permission needed to access data)
   As of now, Zendesk doesn’t support Refresh Token. So once you click Generate Token, it will only create an Access Token (which doesn’t expire until you revoke it explicitly). The Refresh Token will be blank, as expected.
   OAuth 2.0 Flow (Google API Example)
   

   Connect to Google API using OAuth 2.0 (YouTube API Example)

6. Configure Pagination parameters as below
   

   SSIS JSON Source –
   Configure Zendesk REST API Pagination

7. Select Filter
   

   Select Filter – Extract Data from Zendesk API Response (Read Tickets)

8. Click Preview to test
   

   SSIS JSON Source – Data Preview – Zendesk REST API Call to extract tickets

Adding Retry Settings for HTTP or OAuth connection

Zendesk enforces an API rate limit, which means you cannot issue too many API calls in a given time frame. Check this document for exact information.

ZappySys provides API retry settings on both the OAuth Connection and the HTTP Connection. For Zendesk you can retry on a specific status code 429

1. Open connection manager UI
2. Go to Retry-Settings page
3. Select Retry web error on matching response status code.
4. Enter 429 for status code
5. Check multiply wait time
6. In the wait time enter 5000 (wait 5 seconds or multiple of 5 seconds)
7. Set Max retry count as 6

See below example (Its not for Zendesk but will give you an idea)

Retry Options

Configure Zendesk API Pagination and Filter for SSIS JSON Source

The post How to configure Zendesk OAuth Application and Connection for REST API appeared first on ZappySys Blog.

In our previous post we explored unlimited possibilities to call Amazon AWS API using SSIS. Now in this post we will learn how to import / export data from Amazon Athena using SSIS. Amazon Athena is very similar service like Google BigQuery which we documented already.  To read data from Amazon Athena we will use ZappySys JSON / REST API Source which supports excellent JSON parsing capability like this.

Prerequisites

Before we do hello world demo for calling Amazon AWS API, you will need to make sure following prerequisites are met.

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Access to valid AWS credentials (Access Key, Secret Key for your IAM User). Click here to learn more about IAM users and Access Key/Secret Key
4. Make sure SSIS PowerPack is installed. Click here to download.

Steps needed to read data from Amazon Athena

To read data from Amazon Athena you need to do atleast 3 steps at very high level. Later on this article we will see each step in depth.

1. Call StartQueryExecution API to send SQL Query (Its Job Style API so this call starts Job)
2. Keep checking Job Status by calling  GetQueryExecution API (In the response check State field for Status – We need to look for word SUCCESS)
3. Once Job is finished it produces CSV file in S3 Bucket (Output Location is supplied in Step-1)

Read Amazon Athena Query output in SSIS (S3 CSV files)

AWS Console – Athena Query Editor / Configure Output Settings

Few things you might have to setup before you can call Athena API in SSIS

1. Make sure your IAM user has correct permission (We used existing policies for ease of use but you can use custom policy for precise permission set)
2. Configure default output location for your Athena queries

Configure IAM user permission to call Amazon Athena API

Set Athena Query Output Location

Testing Athena Query inside Amazon AWS Console (Query Editor)

Creating Connection to call Athena API in SSIS

Any API calls made to Amazon Athena service using ZappySys JSON / REST API Source  or  ZS REST API Task needs OAuth connection.

For REST API Task configure as below (More details about using REST API Task is in the next section).

1. Drag ZS REST API Task from Control Flow Toolbox
2. Double click to edit.
3. Change Request URL Access Mode to [Url from Connection]
4. Now select <<New OAuth Connection>> from connection drop down
5. On OAuth connection select Amazon AWS API v4 as Provider
6. Enter your Access Key and Secret Key

Create SSIS Connection for Amazon AWS API Call (Athena REST API Example)

Creating Table in Amazon Athena using API call

For this demo we assume you have already created sample table in Amazon Athena. If you wish to automate creating amazon athena table using SSIS then you need to call CREATE TABLE DDL command using ZS REST API Task.

1. In the previous ZS REST API Task select OAuth connection (See previous section)
2. Enter Request Url as below
   
   

   https://athena.us-east-1.amazonaws.com/

3. In the HTTP Headers enter below 2 headers
   

   X-Amz-Target: AmazonAthena.StartQueryExecution
   Content-Type: application/x-amz-json-1.1

4. Change Request method to POST
5. In the Body enter below
   

   {
    "ResultConfiguration": {
    "OutputLocation": "s3://my-bucket/output-files/"
    },
    "QueryString": "create external table tbl01 (CustomerID STRING, CompanyName STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION 's3://my-bucket/input-files/';",
    "ClientRequestToken": "{{System::ExecutionInstanceGUID}}"
   }

   Notice: Rather than using hard coded  DDL SQL Statement we can use variable placeholder (e.g.

   "QueryString": "{{User::Query,FUN_JSONENCODE}}"

      ). Use 

   FUN_JSONENCODE

    to replace new line with \r\n else JSON becomes invalid.
6. Now click Test Request / Response. This will create a new Athena table which we will query in later section.

Notice that in above Body ClientRequestToken is a unique number each time you have to call StartQueryExecution API to make it simple we used SSIS system variable System::ExecutionInstanceGUID. Its unique for each execution. You can also use newid() function from SQL Server and save result to variable if you like.

Here is the screenshot of above Request

RESt API Task – Create table in Amazon Athena using SSIS (StartQueryExecution API Call)

Uploading data for Amazon Athena (Source files)

Once you have athena table created, we will need some data to query. Loading data to amazon athena table is nothing but upload files to S3. If you noticed we added below line in CREATE TABLE statement in the previous section. This means all source files are located at this S3 location.

LOCATION 's3://my-bucket/input-files/'

We have written blog post to explain this process (Click Here – How to load data from SQL server to S3 files).

Extract data from Amazon Athena using SSIS (Query Athena)

Now its time to read some data by writing SQL query for Athena data (i.e. S3 files). As we mentioned earlier, reading data from Athena can be done using following steps.

1. Call  StartQueryExecution API  using REST API Task to execute your Query (It returns you QueryExecutionID). Use same technique as Create Table example. Replace CREATE Table command in body with your SQL Query (New line must be replaced by \r\n in JSON).
2. Use REST API Task to call GetQueryExecution API and configure periodic status check loop (Task keeps waiting until Success or Failure Value not found in Response)
3. Once data is available read data using S3 CSV File Source
4. At last you can clean up query output files or keep it if some other process wants to read it (You can also clean up S3 files using bucket retention policy automatically at scheduled interval).

Now lets see each step in depth

Step1-Start Amazon Athena Query Execution

Now first thing is to execute Athena Query by calling StartQueryExecution API . Below steps are almost same steps as we saw in section Creating Table in Amazon Athena using API call

Only part is different here is SQL query is SELECT query rather than CREATE TABLE.

1. Drag and drop ZS REST API Task from SSIS toolbox
   
2. Enter Request Url as below
   
   

   https://athena.us-east-1.amazonaws.com/

3. In the HTTP Headers enter below 2 headers
   

   X-Amz-Target: AmazonAthena.StartQueryExecution
   Content-Type: application/x-amz-json-1.1

4. Change Request method to POST
5. In the Body enter below
   

   {
     "ResultConfiguration": {
       "OutputLocation": "s3://my-bucket/output-files/"
     },
     "QueryString": "select * from sampledb.elb_logs\r\nlimit 100000",
     "ClientRequestToken": "{{System::ExecutionInstanceGUID}}"
   }

   Notice: Rather than using hard coded  SQL Query we can use variable placeholder (e.g.

   "QueryString": "{{User::Query,FUN_JSONENCODE}}"

      ). Use 

   FUN_JSONENCODE

    to replace new line with \r\n else JSON becomes invalid.

   

6. Now go to Response Settings Tab and configure below way to extract QueryExecutionId (we will need it later).

   Set Extract Type as Json.
   Enter Expression as

   $.QueryExecutionId

   Check Save output to Variable and create new Variable (we can name as QueryExecutionId )
   

   Call Athena SQL, Save Amazon Athena QueryExecutionId in SSIS Variable

7. That’s it now we can move to next step

Step2 – Wait until Athena Query Execution is done

New version of ZS REST API Task includes Status check feature. This feature avoids complex looping logic. Here is how to configure to check Job Status (Wait until Query execution is finished). Steps are almost same as previous section except three things ( Headers, Body and Status Check Tab)

For Status check we will call GetQueryExecution API .

1. Drag and drop ZS REST API Task from SSIS toolbox
   
2. Enter Request Url as below
   
   

   https://athena.us-east-1.amazonaws.com/

3. In the HTTP Headers enter below 2 headers
   

   X-Amz-Target: AmazonAthena.GetQueryExecution
   Content-Type: application/x-amz-json-1.1

4. Change Request method to POST
5. In the Body enter below
   

   {"QueryExecutionId": "{{User::QueryExecutionId}}"}

   Notice: Here we have used Variable in the body. This variable we extracted in the previous step. It contains QueryExecution ID for the Job we submitted.
   

   SSIS Amazon Athena – Call GetQueryExecution API (Status Check)

6. Now go to Status Check Tab and configure below way to implement Status check loop. We want to stop check once status is SUCCESS or FAILED or CANCELLED. If we find FAILED or CANCELLED in State field then we must fail the step with error.
   1. Check Enable Status Check
   2. Enter Success Value as

      SUCCEEDED

   3. Check Fail Task option and enter this regex in the text box 

      FAILED|CANCELLED

   4. In the Interval enter 5 seconds delay after each iteration.
7. That’s it. Status check Tab will look like below. Now we can move to next step
   

   REST API Status check Loop – Keep checking until JOB is done

Step3-Read data from Athena Query output files (CSV / JSON stored in S3 bucket)

When you create Athena table you have to specify query output folder and data input location and file format (e.g. CSV, JSON, Avro, ORC, Parquet …) they can be GZip, Snappy Compressed. Once you execute query it generates CSV file. Zappysys can read CSV, TSV or JSON files using S3 CSV File Source or S3 JSON File Source connectors. In this section we will use CSV connector to read Athena output files.

Here is how to Athena output data

1. Drag and drop Data flow task from SSIS Toolbox
   
2. Double click data flow and drag and drop  ZS Amazon S3 CSV File Source
   
3. Double click to configure it.
4.  Click New Connection and configure S3 Connection
5. Once connection is created, browse to S3 bucket / folder location where Athena outputs files.
6. Replace file name with variable like below
   

   my-bucket/output-files/{{User::QueryExecutionId}}.csv

    

Reading data from Amazon Athena CSV files (Stored in S3)

Putting all together – Athena Data Extract Example

Here is final package with all pieces we talked.

Conclusion

You have seen that using ZappySys SSIS PowerPack how quickly you can integrate with Amazon Athena and other AWS Cloud Services.  Download SSIS PowerPack and try it out by yourself.

Keywords: Extract data from Amazon Athena | Read from Amazon Athena Query file | Import / Export data from Amazon Athena | Fetch data from Amazon Athena

The post Import / Export data from Amazon Athena using SSIS appeared first on ZappySys Blog.

In this blog post you will learn how to call Amazon AWS API using SSIS (virtually any API) without a single line of code (No more JAVA, C#, Ruby, Python). Yes you heard it right 🙂 . If you are a SSIS / ETL Developer or even coder everyone loves drag & drop interface. SSIS has many advantages over other approaches such as Programming SDKs, Command Lines. Main advantage is ease of use, security and long term maintenance without learning expensive coding approach.

In this post we will use following components to show various possibilities to implement Amazon AWS API integration scenarios inside your ETL workflows.

	JSON / REST API Source (REST API, JSON File or OData Service): Use this dataflow component when you have to fetch data from REST API webservice like a table. This component allows you to extract JSON data from webservice and de-normalize nested structure so you can save to Relational database such as SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local JSON files or direct JSON string (Wildcard pattern supported too e.g. c:\data\file*.json).
	Web API Destination (POST data to API URL) : Use this dataflow component when you have to call API inside Dataflow (see POST to URL). Possible Use case: Call Lambda function with various parameters for each input record found in database.
	REST API Task : Use this task when you don’t want to pull REST API data in tabular format but want to call rest API for POST data to server, DELETE data from server or things like download HTML page, extract Authentication tokens etc where you not necessarily dealing data in tabular format. This task also allows you many other options such as saving RAW response into variable or file.
	XML Source (SOAP, File, REST) : Use this dataflow component when you have to fetch data from XML or SOAP webservice and consume data like a table. This component allows you to extract data from webservice and save to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local XML files or direct XML string.

Prerequisites

Before we do hello world demo for calling Amazon AWS API, you will need to make sure following prerequisites are met.

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Access to valid AWS credentials (Access Key, Secret Key for your IAM User). Click here to learn more about IAM users and Access Key/Secret Key
4. Make sure SSIS PowerPack is installed. Click here to download.

Step-By-Step Example-1 (Call AWS API)

Now lets call some simple GET API call using SSIS REST API Task.

1. Install  SSIS PowerPack (Skip this step if you already installed SSIS PowerPack.
2. Open Visual Studio and create new Integration Services Project
3. Open SSIS Package and check your SSIS Toolbox you will see many tasks/components starting with ZS
4. From Control flow SSIS Toolbox Drag & drop ZS REST API Task.
5. Double click the REST API task to configure it.
6. On REST API Task change URL Access mode drop down to URL from Connection
7. Now in the connection dropdown click New ZS-OAUTH connection and configure connection as below
   1. Once you see OAuth connection dialog box change Provider Type from Custom to Amazon AWS API (v4)
   2. In the ClientId enter your AWS Access Key
   3. In the ClientSecret enter your AWS Secret Key
   4. Click OK to save connection
      

      Configure SSIS OAuth Connection – Use Amazon AWS API Provider, Enter Access Key, Secret Key

8. On REST API Task change few more settings as below
   1. Enter API URL you like to call (In our case we will use S3 API (Simple Storage Service). We assume you have ListBucket permission to make this call. If you dont have such permission try to get Full path of File (choose small file) . You have to tweak API url to adjust Service Type, Region,Bucket, Path
      

      https://s3.us-east-1.amazonaws.com
      -- OR -- Use below (list files) if you have single bucket permission---
      https://s3.us-east-1.amazonaws.com/YOUR-BUCKET

   2. Click Test Request. If you have valid Permission and setup looks ok then you will see Response window like below. Using this technique you can call any API to execute AWS operations (E.g. start EC2 VM, Create SQS Queue, Call Lambda Function, Drop or Update resource)
      

      Calling Amazon AWS API Using SSIS REST API Task Example. (Call any AWS API without SDK. such as EC2 API, SQS API, Lambda API, Redshift API, DynamoDB API)

Step-By-Step Example-2 (Loading data from AWS API to SQL Server)

Now lets do more interesting scenario. We will call AWS S3 API to get S3 File list from Bucket. After extract we will save that list to SQL Server Table. Since Amazon S3 API is XML based API we will use ZappySys SSIS XML Source. For JSON based API use JSON Source instead. XML Source / JSON Source both can parse API response into Rows and Columns so you can easily store it into SQL Server. Now lets see how to do this.

1. From Control flow SSIS Toolbox Drag & drop Data Flow Task
2. Double click Data Flow Task. From SSIS Toolbox Drag & drop ZS SSIS XML Source.
3. Double click the XML Source to configure it.
4. In the URL text box enter API URL like below to list S3 Files for specified bucket (Change YOUR-BUCKET to your own name)
   

   https://s3.us-east-1.amazonaws.com/YOUR-BUCKET

5. Now check Use Credentials and Select same Amazon API connection we created in previous example.
6. Click on Select Filter (This step allows us to flatten the XML hierarchy. Select the node which is Array icon like below. If prompted to treat selected node as array click Yes.
   

   Configure SSIS XML Source – Get Amazon S3 File List from specific bucket (AWS API)

7. Click Preview to see your data. Click OK to save.
   

   Preview Data in XML Source – Amazon S3 File List (Calling AWS API)

8. Now attach your XML source to target like OLEDB Destination to load data to SQL Server or other Target (e.g. Oracle, MySQL)
9. Execute SSIS Package to load data from Amazon AWS API to SQL Server.
   

   SSIS Package – Read Amazon S3 File List and load into SQL Server Table (Call AWS API Example)

File Upload Example – Low level API – Call PUT request

There will be a time when you want to take total control of your AWS API calls. One example is if you wish to Upload / Write data to S3 then components like ZappySys Amazon S3 CSV Destination or Amazon Storage Task might need additional permission such as HeadObject . If you have only write permission on bucket then this will fail to execute Task. In such case you can use REST API Task like below way.

Basically we exported data from Relation Database to CSV File using Export CSV File Task and then we uploaded file content using REST API Task.

Upload Text File to S3 (i.e. JSON, CSV, XML …)

Your URL Format can be like this. Region codes can be found here

https://YOUR-BUCKET.s3.YOUR-REGION.amazonaws.com/YOUR-FILE
https://YOUR-BUCKET.s3.YOUR-REGION.amazonaws.com/SOME-FOLDER/SOME-SUB-FOLDER/YOUR-FILE

Upload file to Amazon S3 – Call low level API (PUT request)

Upload Binary File to S3 (i.e. Zip, mp3, gzip, png, jpeg…)

Above method only works for Text Files. If you have Binary files (e.g. Zip file, mp3, png, jpeg) then you can use below workaround.

1. Check Is Multi Part / File Upload Option next to the Body editor (Read this Post for more info)
2. In the Body enter file path like below
   

   @c:\folder\some-file.xyz

3. That’s it, now your binary file can be uploaded to S3 same way as Text file we uploaded in earlier section.

NOTE: Binary file option only works in the latest Build Uploaded after 5/21/2020

Debugging AWS API Command Line Requests using Fiddler (Web Proxy)

Before we see more examples of calling AWS API lets first learn how to capture Request data using aws command line (CLI). We will use Fiddler to capture AWS API Requests.

1. Download and Install Fiddler (Free Tool)
2. Install  aws command line
3. Open command prompt and type aws configure command to set credentials.  For more info see configure aws credentials .. see below example.
   

   c:> aws configure
   AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
   AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
   Default region name [None]: us-west-2
   Default output format [None]: json

4. Once you set credentials launch Fiddler
5. On the Fiddler Tools menu > Click Fiddler Option > HTTPS > Check Decrypt HTTPS Traffic
6. Once you do that you may be asked to Trust Fiddler Certificate click OK
7. Close and Open Fiddler to apply setting
8. Now any command you type in aws command line will show up in fiddler. This command we can use in SSIS REST API Task or XML Source to call virtually Any API AWS supports.
9. For example to call some lambda function (supply input json data from file) use below command and watch Fiddler Trace. Notice we added –no-verify-ssl option so we can see requests in custom web proxy like fiddler (This option will trust Fiddler certificate).
   

   c:\>aws lambda invoke --function-name HelloWorld c:\temp\outputfile.txt --no-verify-ssl --payload file://c://test/customer.json

    
   

   Capture AWS Commandline (CLI) request using Fiddler

10. Once you have this information you can use it inside ZappySys Components which supports API calls (e.g. REST API Task, JSON Source, XML Source). For this example we will use REST API Task to call same Lambda Function. Things to change to call any API is URL, Method, ContentType, Body. If its GET call then you wont have Body.
    

    Calling Amazon AWS Lambda Function using SSIS REST API Task

     

11. That’s it.. You can now take this same concept and call virtually API AWS API right insight SSIS without any SDK or command line tools.

Call API Gateway Endpoint (Default URL)

If you wish to call API hosted on Amazon API Gateway Service then enter direct URL. You have to use OAuth Connection (AWS v4 Provider) as previous section.

Typical URL may look like as below if you calling API gateway Endpoint.

https://c5hhigf5mh.execute-api.us-east-1.amazonaws.com/prod/pets

Call AWS API with Region and Custom Service Name

There will be a time when you will have to call AWS API URL and it doesnt indicate Region / Service but you have to supply part of signature for Authentication.  In this case just enter AWS Service and Region along with ClientID, Secret Fields (Custom Service and  Region attributes were introduced in v3.0 or higher so if you are not seeing on OAuth Connection UI then you probably running older version).

Call Amazon Athena API

Check this article for detailed instructions

Call AWS EC2 API

Here is an example of calling EC2 API. This is example of listing EC2 instances.

Request:

POST https://ec2.us-east-1.amazonaws.com/
Content-Type: application/x-www-form-urlencoded; charset=utf-8

Action=DescribeInstances&Version=2016-11-15

Call AWS Lambda API

Here is some example of calling Lambda function

Request:

POST https://lambda.YOUR-REGION.amazonaws.com/2015-03-31/functions/YOUR-FUNCTION/invocations

{you-json-input-goes-in-body}

Call AWS ElasticSearch

If you are using AWS Hosted Managed ElasticSearch then also you can use OAuth AWS v4 Provider as below. Assuming you have configured correct Policy to allow your IAM User Account / IP address.

Call AWS Hosted ElasticSearch REST API in SSIS (V4 Request Signing)

Call AWS API using Native Task/Components

ZappySys provides many High quality Tasks / Components for AWS integration. See below list.

Amazon AWS Cloud Integration
Amazon S3 Task	Amazon Redshift Data Transfer Task	Amazon Redshift ExecuteSql Task
Amazon Redshift Cluster Management Task	Amazon DynamoDB Source	Amazon DynamoDB Destination
Amazon Redshift Source	Amazon SQS Queue Source	Amazon SQS Queue Destination

Conclusion

Amazon AWS Cloud integration from SSIS packages becoming more and more common scenario. ZappySys provides easy to use no coding connectors to achieve many time consuming scenarios. Having clean drag and drop approach is not only faster but more secure using inbuilt SSIS framework. Try SSIS PowerPack to explore many other scenarios not discussed in this article.

The post How to call Amazon AWS API using SSIS (EC2, Lambda, API Gateway, SQS) appeared first on ZappySys Blog.