In this tutorial, we will use the REST API Task to call some API. We will use the OAuth connection along the REST API Task to write / update data to Google Spreadsheet.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of Google API.

Component Mention in this article

	JSON Generator Transform
	Web API Destination
	REST API Task

Create a Google API Project

The first step to access any Google API is to create an API Project in Google Console. If you don’t want to go through this then Skip this Step-1 and in the next section select the Default OAuth App option on OAth Connection Manager (This is the easiest option for now unless you want to use your own OAuth App).

Check this article for step-by-step instructions. When you follow these instructions make sure you enable Google Sheet API.

Create OAuth Connection Manager in SSIS

Once you create a Google API project and obtained a Client ID and Client Secret your next step is to create OAuth Connection Manager in SSIS. ZappySys OAuth connection manager comes with many predefined OAuth Providers (e.g. Facebook, Twitter, Google, etc) but you can also define custom OAuth settings for any OAuth enabled API.

To create SSIS OAuth 2.0 Connection for Google API perform the following steps.

1. Download and Install SSIS PowerPack.
2. Create a new SSIS Package.
3. Right-click in Connection Manager Area and Click “New Connection”.
4. When prompted select ZS-OAUTH connection type, On the OAuth Connection Manager Select Provider=Google.
5. Click here to create credentials. Then select the “Use Custom OAuth App” option and enter your ClientID, ClientSecret.
6. In the Scopes enter below scopes (or click Select Scopes). This will allow read/write access to Drive Files or Sheet (E.g. Reading file content / Export file to CSV). You can use the Scope browser to see many other available permissions. For more information click here.
   

   https://www.googleapis.com/auth/drive
   https://www.googleapis.com/auth/drive.file
   https://www.googleapis.com/auth/drive.readonly
   https://www.googleapis.com/auth/spreadsheets
   https://www.googleapis.com/auth/spreadsheets.readonly

7. Click on the Generate Token button.
   

   SSIS OAuth Connection Manager – Access Google Drive API using OAuth 2.0

8. You will see UI as below… Click Allow (You may have to scroll to see that button sometimes).
9. Click on Test Connection to see the connection is working and click on the OK button to save configure setting UI.

Find Google Sheet ID for API calls

Before we can write or update Sheet data in SSIS we need to know Google Driver File ID. We will use this ID in the next few sections.

The easiest way to find File ID is to look at the URL like below. If you need to know ID at runtime dynamically then refer to the last section.

Get Google Drive File ID for API call (Sheet ID)

Create Variables to make dynamic URL

1. Here, we are going to make a Dynamic URL value.
2. In Visual Studio Project Right Click on Design Panel, Select Variables.
   

   Create Variables

3. Let’s Create Variables with correct DataType and Value. In variable “varRange” set only the starting point(Sheet1!A2) of data if you don’t know the ending point and leave the first-row for giving header row later or click here.
   

Write SQL Server data to Google Spreadsheet

Now let’s look at a real-world example. How to write data from SQL Server table or other source and send to Google Sheet using Web API Destination.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Dragging and dropping Data Flow Task into Control Flow

2. Double click on the Data Flow task to see the Data Flow designer surface.
3. Here, In Visual Studio, drag and drop the OLE DB Source, JSON Generator Transform and Web API Destination in the design surface and join the components with the blue arrow.
   

   Drag and Drop OLE DB Source, JSON Generator Transform, and Web API Destination

4. Here, you can use our Free ZS Dummy Data Source too. Click here for more information.
5. But, we are going through OLE DB Source so we need OLEDB Connection.

Create OLE DB Connection

1. Lets, create the OLE DB Connection.
2. Let’s, Right-click on Connection Managers Panel to Create OLEDB Connection, so you can use Source and Context Menu will appear, Select New OLEDB Connection from the Context Menu.
   

   Create OLE DB Connection

3. Now, click on New Button to create Connection.
   

   Create New Connection

4. Let’s Configure Connection Manager, just Follow steps one by one as we have created.
   

   Configure OLE DB Connection

5. Click on Test Connection to see correct configure it.
6. Click on the OK button to save connection configure setting UI.

Configure JSON Generator Transform

1. Double click on JSON Generator Transform to configure it.
2. Set Output Mode to Single Dataset Array. Right-click on Mappings, select Add Static Element then enter Name: “range” and Select variable we have created for value.
   

   Configure JSON Generator Transform – Add Static Value

3. Again, Right-click on Mappings, select Add Static Element then enter Name: “majorDimension” and Value: “ROWS“.
   

   Configure JSON Generator Transform – Add Static Value

4. Now, Right-click on Mappings, select Add Document Array, Check to Treat as 2D array option and enter Element Name as “values“.
   

   Configure JSON Generator Transform – Add Document Array

5. Then, right-click on Document Array, select Add elements > Select Add Multiple Columns and check on columns you like to add.
   

   Configure JSON Generator Transform – Add Element

6. Click on the OK button to save JSON Generator Transform configure setting UI.

Configure Web API Destination

1. Double click on Web API Destination to configure it.
2. Select the OAuth connection manager created in an earlier section (Assuming you had all scopes with write permission).
3. In the Setting tab,
   1. In URL, Generate Dynamic URL like below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4:append?valueInputOption=USER_ENTERED

   2. Change Request Method to POST.
   3. Select JSON Generator variable into Body (Request Data) or you can select Input column for Body (JSON Generator Output).
   4. Set application/JSON to Body Content-type.
      

      Configure Web API Destination

4. Click on the OK button to save Web API Destination configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet. Now update header row values for click here.
   

   Run or Execute Package and verify data into Google Spreadsheet

Append data to Google SpreadSheet using REST API Task

Now let’s look at how to write to Google SpreadSheet. To write multiple rows in a single request. Also, check this link to learn more.

For more information, you can see the example here. For parameter, detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve the Spreadsheet write scenario. For a simple example, let’s use the REST API Task.

1. Drag and Drop ZS REST API Task on the control flow designer.
   

   Drag and Drop REST API Task

2. Double click on ZS REST API Task to configure it.
3. In the Request Settings tab,
   1. set Request URL Access Mode to Url from a connection. Select OAuth Connection from the dropdown (Same connection we created in the previous section).
   2. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A1:append?valueInputOption=USER_ENTERED

   3. Change Request Method to POST.
   4. Enter Request Body as below
      
      Note: For example purpose, we included header line in data cells but in the real-world header may already exist in the sheet.

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["Item", "Cost", "Stocked", "Ship Date"],
          ["Wheel", "$20.50", "4", "3/1/2016"],
          ["Door", "$15", "2", "3/15/2016"],
          ["Engine", "$100", "1", "3/20/2016"],
          ["Totals", "=SUM(B2:B4)", "=SUM(C2:C4)", "=MAX(D2:D4)"]
        ]
      }

   5. Change Request Content type to application/JSON.
      

      Write data to Google SpreadSheet using REST API Task (OAuth 2.0 Connection)

4. Click on the OK button to save REST API Task configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.
   

   Execute REST API Task and verify Google Spreadsheet data

Update Google Sheet Cells

Now let’s look at the below example of updating multiple cells in google Spreadsheet. For more information, you can see the example here. For parameter, detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve the Spreadsheet Update scenario. For a simple example, let’s use the REST API Task.

Update Google Sheet header cells manually

1. Drag and drop ZS REST API Tasks on the control flow designer.
2. Double click on the First REST API Task to configure it.
3. In the Request Settings tab,
   1. Enter the URL below. Change Range designator(including Sheet name: Sheet1!A1) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A1?valueInputOption=USER_ENTERED

   2. Change Request Method to PUT.
   3. Enter Request Body as below
      

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["CustomerID", "CustomerName"]
        ]
      }

4. Click on the OK button to save REST API Task configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.
   

Single Range (Set Values for multiple cells)

1. Follow the previous section(write data to Google Spreadsheet) to update sheet data.
2. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4?valueInputOption=USER_ENTERED

   2. Change Request Method to PUT.
   3. Enter Request Body as below
      

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["Laptop", "$2000.50"],
          ["Mobile", "$1150"]
        ]
      }

3. That’s all, click on the Test Request/Response button and verify data into Google spreadsheet.
   

   Calling Google Sheet API to Update Multiple Cell Values

Clear data to Google SpreadSheet using REST API Task

Clear a sheet of all values while preserving formats

Lets clear a sheet of all values by REST API Task, Before processed we need sheet Id. So let’s get the SheetId by API Call. For more information about click here.

1. Drag and Drop two ZS REST API Tasks on the control flow designer and rename(Get Sheet ID and Clear Google Sheet of all values) it.
2. Double click on the First REST API(Get Sheet ID) Task to configure it.
3. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc

   2. Change Request Method to GET.
      REST API Task – Get Google Sheet ID
4. Now go to Response Settings Tab, Select Response content Type Json and Write Response Content Filter Expression. Check on Save Response Content. Select Save Mode to Variable and Create a New variable or select it.
   Example of Response Content Filter Expression for the first Sheet : 
   

   $.sheets.[0].properties.sheetId

   

   REST API Task – Save Google Sheet ID into Variable

5. Now, click on the Test Request/Response to check configures are ok.
6. Click on the OK button to save REST API Task configure setting UI.
7. Double click on the second REST API(Clear Google Sheet of all values) Task to configure it.
8. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}:batchUpdate

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc:batchUpdate

   2. Change Request Method to POST. Enter Request Body as below and change a variable parameter as per your need.
      

      {
        "requests": [
          {
            "updateCells": {
              "range": {
                "sheetId": {{User::varGoogleSheetID}}
              },
              "fields": "userEnteredValue"
            }
          }
        ]
      }

   3. Change Request Content type to application/JSON.
      
      

      REST API Task Configure – Clear sheet of all values

9. Click on the OK button to save REST API Task configure setting UI.
10. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.

Clears range cell values from a spreadsheet

1. Double click on REST API Task to configure it. Then, follow the way just like the previous section configures REST API Task.
2. Here you need to set the value of Variable “varSpreadsheetValues” to “clear” or you can change it directly in URL append to clear and value of variable “varRange” (Change cell range and sheet name) as per your need. For more information about it click here.
   1. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4:clear?valueInputOption=USER_ENTERED

3. Change Request Method to POST.
4. Enter Request Body as below
   

   {}

5. Change Request Content type to application/JSON.
6. That’s all, click on the Test Request/Response button and verify data into Google spreadsheet.

Conclusion

Google Drive API and Google Sheets API provide a great way to automate file-related functionality. However, to call Google API  you have to use the SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides a great way to integrate any Google API call via simple drag and drop approach without coding. Try out yourself see how long it takes to call virtually any REST API.

References

Finally, you can use the following links for more information:

• JSON Generator Transform
• Web API Destination
• REST API Task
• Help File:
  • JSON Generator Transform
  • Web API Destination
  • REST API Task

The post How to write/update Google Sheet in SSIS appeared first on ZappySys Blog.

ZappySys provides high-performance drag and drop connectors for Amazon AWS Integration. In this post, we will see How to Read / Write / Delete Amazon SQS Queue data in SSIS using ZS Amazon SQS components. ZappySys developed many AWS related components but in this article, we will look at 2 components for Amazon SQS Queue integration Scenarios (Read and Write). We will discuss how to use SSIS Amazon SQS Source and SSIS Amazon SQS Destination. Amazon SQS Source can be used to read messages from Amazon AWS SQS (Simple Queue Storage) and Amazon SQS Destination can be used to write/delete data from Amazon SQS (Simple Queue Storage) using SSIS.

Prerequisite

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

• First, you will need to have SSIS installed.
• Secondly, make sure to have SSDT.
• Thirdly, you have obtained Amazon SQS Credentials.
• Finally, do not forget to install ZappySys SSIS PowerPack.

Component Mention in this article

	Amazon SQS Source Connector
	Amazon SQS Destination Connector

What is Amazon SQS (Simple Queue Service)

Before we talk How to Read / Write / Delete Amazon SQS Queue data in SSIS lets talk What is Amazon SQS (Simple Queue Service) or sometimes referred to as AWS SQS. Amazon SQS offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. Amazon SQS offers common constructs such as dead-letter queues and costs allocation tags. It provides a generic web services API and it can be accessed by any programming language that the AWS SDK supports.

SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

Getting Started

In order to start, we will show several examples. ZappySys includes an Amazon SQS Source and Amazon SQS Destination, that will help you How to Read / Write / Delete Amazon SQS Queue data in SSIS. You can connect to your Amazon SQS account by entering credentials.

Amazon SQS Source can be used to read messages from Amazon AWS SQS (Simple Queue Storage) using SSIS. Read Amazon SQS Messages (AWS SQS) in bulk. Amazon SQS Destination can be used to write/delete data from Amazon SQS (Simple Queue Storage) using SSIS. Write messages to Amazon SQS (AWS SQS) in bulk. Support for Visibility Timeout (Hide messages from others for certain time after you read), data preview and max rows, reading message attributes in separate output columns and adjusting UTC DateTime in local time.

Write Data using Amazon SQS Destination in SSIS

Overview

In this tutorial, you will learn how to write data in Amazon SQS Queue storage. You can use ZS DummyData Source for practice it or select suitable ZS Source Adapters. In this case its from ZS DummyData Source. So let’s start with an example, In this section, you will learn how to write messages into Amazon SQS Queue. For this example, we are assuming that you have already SQS Queue Created.

Create Amazon SQS Connection

1. Firstly, In order to connect to Amazon SQS from SSIS, you will need Access Key and Secret Key (Login to AWS portal and Create Access/Secret Key with SQS service read access). If you are not sure about this ask your SysAdmin or responsible person to provide that to you. Your key will look something like this: (This is just an example key which may differ in your case).
   
   

   AccessKey: AKIAIOSFODNN7EXAMPLE
   SecretKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

   Click here to read more about how to get your AccessKey and Secret Key.
2. Secondly, Once you have Account Key (it’s like UserID) and Secret Key (its like password) you may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
3. Right-click in Connection Manager Panel and Select [New Connection] menu item.
   

   SSIS Create AWS Storage Connection

4. Now Select ZS-AWS-STORAGE connection type.
   

   Select ZS-AWS-STORAGE Connection

5. Let’s configure the Amazon SQS connection. Set Storage Service to SQS and Credential Type to Key-based access control. In the General tab enter your Amazon SQS Credentials and Set Default region as per you have.
   

   SSIS AWS Storage Connection – Configure

6. In the last, click on Test Connection and OK button to Save connection configure setting UI.

Write Data into Amazon SQS

1. Once you have created the Amazon SQS Connection successfully, let’s drag and Drop Data Flow Task from SSIS Toolbox on the control flow designer surface.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

2. Double click on the Data Flow Task to see Data Flow Designer surface.
3. Here, from the SSIS Toolbox drag and drop ZS DummyData Source on the data flow designer surface.
   

   SSIS ZS Dummy Data Source – Drag and Drop

4. Double click on Dummy Data Source to configure it.
   

   SSIS ZS Dummy Data Source – Configure

5. Click on OK button to save configure setting UI.
6. From the SSIS toolbox drag and drop Amazon SQS Destination on the Data flow designer surface and single click on the Dummy Data Source, Once you see the blue arrow from the source. connect it to Amazon SQS Destination.
   

   SSIS ZS Amazon SQS Destination

7. Now double click on Amazon SQS Destination to configure it.
8. In the [Connection Manager] tab select SQS connection manager we have created before it.
   

   SSIS Amazon SQS Destination – Configure Connection Manager

9. Click on [Component Properties] tab and Select Target Queue from the dropdown list.
   

   SSIS ZS Amazon SQS Destination – Configure Component Properties

10. Click on [Mappings] tab and verify mapping. Map document column from upstream to the MessageText target column.
    

    SSIS ZS Amazon SQS Destination – Column Mappings

11. In the last, click on the OK button to save configure setting UI.
12. After all, Execute the Package and verify target data by going to your AWS console under SQS service section.
    

    Run or Execute SSIS Package

    

    Verify Amazon SQS Destination

Read Data using Amazon SQS Source in SSIS

Overview

In this tutorial, you will learn how to read data from Amazon SQS storage in SSIS using ZS SSIS Amazon SQS Source Adapter. In this SSIS we will read data. So let’s begin it.

Read Data from Amazon SQS

1. Firstly, create a new SSIS package. In BIDS/SSDT create a new SSIS package.
2. From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
   

   SSIS Data Flow Task – Drag and Drop

3. Double click on the DataFlow task to see DataFlow designer surface.
4. From the SSIS toolbox drag and drop ZS Amazon SQS Source on the dataflow designer surface.
   
5. Double click Amazon SQS Source to configure it.
6. Select Amazon SQS Connection we have created before and select queue from the dropdown list.
   

   SSIS Amazon SQS Source – Configure

7.  Click on Preview button to see Data Preview and Click on  OK button to save settings.
8. From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface or you can use any of our suitable ZS Destination Adapter.
   

   SSIS ZS Trash Destination – Drag and Drop

9. Now single click on the Amazon SQS Source once you see the blue arrow from the source … connect it to Trash Destination.
10. Double click on ZS Trash Destination to Configure it.
    

    SSIS ZS Trash Destination – Configure

11. Click on OK button to save Trash Destination configure setting UI.
12. After all, right-click on the path and Add Data Viewer and Execute the package and verify source data in the data viewer.
    

    Run or Execute The SSIS Package

13. After messages read from SQS Queue, the same message cannot be read again until VisibilityTimeout. By default Queue, visibility timeout is used (-1 means use default) but you can override it on the source UI.

Read all messages from SQS FIFO Queue

If you are using Normal SQS Queue then by setting Max Rows=0 will read all messages from your queue. However reading all messages from FIFO Queue is not that simple. Amazon SQS wont return more messages until you delete previously read Messages from same Message Group (identified by MessageGroupId). Due to this constraint you have to try below technique of Loop to read 10 messages until all messages Read > Processed  > Deleted.

Here is note from Amazon Documentation about maximum 10 messages limitation on FIFO Queue.

When you receive a message with a message group ID, no more messages for the same message group ID are returned unless you delete the message or it becomes visible.

Community Reference:

https://stackoverflow.com/questions/51661834/aws-sqs-fifo-how-to-get-more-than-10-messages-at-a-time

If you want to add fault tolerance then you can set Max Retry for NULL Receive like below (e.g. Setting MaxRetry=3 will try max 3 times if no message is read …. It will try every 2 seconds for 3 times before giving up to read more rows).

Read all messages from Amazon SQS FIFO Queue

Read , Process and Delete Amazon SQS FIFO Queue Message in SSIS Data flow

Delete Message from Queue using Amazon SQS Destination in SSIS

Overview

In this tutorial, you will learn how to Delete Messages from the Queue using the ZS SSIS Amazon SQS Destination Adapter. So let’s begin it.

Delete Message from Queue

1. From the SSIS toolbox drag and drop Amazon SQS Destination on the Data flow designer surface and connect it to Amazon SQS Destination.
2. Now double click on Amazon SQS Destination to configure it.
3. In the [Connection Manager] tab select SQS connection manager we have created before it.
   

   Select Amazon SQS Connection

4. Click on [Component Properties] tab, Set Operation to Delete and Select Target Queue from the dropdown list.
   

   Select Delete Operation and Queue

5. Click on [Column Mappings] tab and verify mapping. Map document column from upstream to the MessageText target column.
   

   Do Columns Mapping

6. In the last, click on the OK button to save configure setting UI.
7. After all, Execute the Package and verify target data by going to your AWS console under SQS service section.

Purge All Messages from SQS Queue (Clear Queue)

There will be a time you like to clear entire queue (i.e. Delete All Messages from Queue). Amazon recently added PurgeQueue API feature so now its possible without doing Delete message technique mentioned in previous section. ZappySys support calling any AWS API using REST API Task along with AWS Provider in OAuth Connection. You can use same technique to call PurgeQueue API like below.

1. Open SSIS Package and Drag ZS REST API Task from Control Flow Toolbox
2. Click on URL mode dropdown and change to URL from Connection.
3. From COnnection Dropdown select NEW ZS-OAuth connection
4. Configure OAuth connection to use Provider=Amazon AWS v4
   1. Enter Access Key and Secret Key and click OK to save connection
5. Enter below URL on REST API Task (Replace Queue ID and name with your own values) – Refer to your AWS Console / SQS Queue Properties to get this URL
   

   https://sqs.ENTER-REGION-HERE.amazonaws.com/ENTER-QUEUE-ID-HERE/ENTER-QUEUE-NAME-HERE/?Action=PurgeQueue

   Here is an example of Sample Queue from ZappySys Demo
   

   https://sqs.us-east-1.amazonaws.com/718883169616/zappysystest.fifo/?Action=PurgeQueue

6. Click ok to Save REST API Task

How to Purge Amazon SQS Queue in SSIS (Clear Queue / Delete All messages using PurgeQueue REST API)

Conclusion

So, In this blog, we learn How to Read / Write / Delete Amazon SQS Queue data in SSIS. We used Amazon SQS Destination for write data and Amazon SQS Source to read data. Additionally, you can download SSIS PowerPack here to try many other scenarios that are not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• Amazon SQS Source
• Amazon SQS Destination
• Documents of Amazon SQS
• Help File:
  • Amazon SQS Source
  • Amazon SQS Destination

The post How to Read / Write / Delete Amazon SQS Queue data in SSIS appeared first on ZappySys Blog.

ZappySys provide high-performance drag and drop connectors for Amazon Redshift Integration. In our previous post we saw how to bulk load SQL data into Redshift using S3 staging technique (COPY command). Now in this post, you will see How to Read / Write Amazon Redshift Data in SSIS Data flow without need for S3 Staging. We will use  ZS Amazon Redshift components (Source / Destination). If you need to Upsert / delete data from Redshift Table then use Upsert Destination (Not discussed in this article).

ZappySys developed many AWS related components but in this article, we will look at 2 Components for Amazon Redshift Integration Scenarios (Read and Write). We will discuss how to use SSIS Amazon Redshift Source and SSIS Amazon Redshift Destination. Amazon Redshift Source connector supports familiar SQL Like query language so any DBA or SQL user can learn it in no time. ZappySys developed a highly sophisticated engine to turn your SQL Query into native Amazon Redshift query. Load query at runtime from the variables or Direct value.

Prerequisite

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Thirdly, You have obtained Amazon Redshift Credential.
4. Finally, do not forget to install ZappySys SSIS PowerPack.

Component Mention in this article

	Amazon Redshift Source
	Amazon Redshift Destination

What is Amazon Redshift

Before we talk How to Read / Write Amazon Redshift Data in SSIS lets talk what is Amazon Redshift (or sometimes referred to as AWS Redshift). Amazon Redshift is a Cloud-based Data warehouse service. This type of system also referred to as MPP (Massively Parallel Processing). Amazon Redshift uses a highly modified version of PostgreSQL Engine behind the scene. Amazon Redshift provides the advantage of Scale as you go, at a very low cost compared to onsite dedicated hardware/software approach. In this article, we will try to learn How to Read / Write Amazon Redshift Data in SSIS.

Getting Started

In order to start, we will show several examples. ZappySys includes an Amazon Redshift Source and Destination, that will help you How to Read / Write Amazon Redshift Data in SSIS. You can connect to your Amazon Redshift Account by entering credentials.

SSIS Amazon Redshift Source Connector can be used to read data from Amazon Redshift. You can use a simple Table mode or write custom SQL Query to extract desired data. You can use multiple modes (i.e. Table or Query) to read data from the Amazon Redshift Database. Support for data preview and max rows and Dynamic query (using SSIS Variable placeholder e.g. select * from mytable where id='{{User:: MyVar}}’).

SSIS Amazon Redshift Destination can be used to bulk insert a large number of records to Amazon Redshift storage from any data source using SSIS. Amazon Redshift is a cloud-based Data Warehouse service from Amazon AWS. Here, you can bulk Insert, Update and Delete Amazon Redshift data and supporting the batch size.

Click here to see Amazon Redshift related articles.

Write Data using Amazon Redshift Destination in SSIS

Overview

In this tutorial, you will learn how to write data in Amazon Redshift Database. You can use ZS DummyData Source for practice it or select suitable ZS Source Adapter (In this case it’s from SQL Server Source). So, Let’s start with an example. In this SSIS we will write data into Amazon Redshift Database using ZS Amazon Redshift Destination.

How to write data into Amazon Redshift Table

1. In order to connect to the Amazon Redshift Database from SSIS, you will need credentials. Ask your SysAdmin or responsible person to provide that you.
2. For this sample exercise, we will create a table in Redshift.
   

   CREATE TABLE public."tbl_pg_Customers"
   (
       "RecordID" character varying(50) NOT NULL,
       "CustomerID" character varying(50),
       "CustomerName" character varying(50),
       "Address1" character varying(50),
       "City" character varying(50),
       "State" character varying(50),
       "Zip" character varying(50),
       "Country" character varying(50),
       PRIMARY KEY ("RecordID")
   );

3. Once you have created a Table, you may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
4. In your SSIS toolbox, you should see many new Icons starting with “ZS” (i.e. ZappySys). If you don’t see them then check this.
5. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

6. Double click on the Data Flow Task to see Data flow designer surface.
7. Here, In Visual Studio, Drag and Drop the OLE DB Source and Amazon Redshift Destination in the design surface and join the components with the blue arrow.
   

   SSIS OLE DB Source and Amazon Redshift Destination – Drag and Drop

8. Now, we need OLE DB Connection and Amazon Redshift Connection. Click here for Create Amazon Redshift Connection.

How to Create an OLE DB Connection.

1. Let’s, Right-click on Connection Managers Panel to Create OLEDB Connection, so you can use Source and Context Menu will appear, Select New OLEDB Connection from the Context Menu.
   

   SSIS OLE DB Connection – Create Connection

2. Now click on the New button to create a connection.
   

   SSIS OLE DB Connection – Create New Connection

3. Let’s configure connection Manager, just follow the steps one as we have created in following the image.
   

   SSIS OLE DB Connection – Configure Connection

4. After all, Click on OK button to save configure setting UI.

Configure SSIS ZS Amazon Redshift Destination

1. Double click on OLE DB Source for configure it.
2. Let’s Configure in Connection Manager, just follow below image steps.
   

   SSIS OLE DB Source – Configure Source Editors

3. Click OK to save OLE DB Source Editor UI settings.
4. Now double click on Amazon Redshift Destination to configure it. In the [Connection Manager] tab Select Amazon Redshift Connection.
   

   SSIS Amazon Redshift Destination – Connection Manager

5. Click on [Component Properties] tab and Pick the Destination Table from Tables dropdown list.
   

   SSIS Amazon Redshift Destination – Component Properties

6. Click on [Column mappings] tab and verify mapping. If the target table is new Table and it’s empty then all input columns will be automatically mapped as shown below. If the table is an existing table with some data then you can manually pick a mapping column by dragging it from source list to target list.
   

   SSIS Amazon Redshift Destination – Column Mappings

7. Execute the package and verify that the data is saved.
   

   SSIS Amazon Redshift Destination – Run or Execute Package

Read Data using Amazon Redshift Source in SSIS

Overview

In this tutorial, you will learn how to read data from Amazon Redshift in SSIS using ZS Amazon Redshift Source adapter. In this SSIS we will read data with SQL Query and Table Mode. So Let’s begin it.

How to Read data from Amazon Redshift Table

1. You may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
2. From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
   

   SSIS Data Flow Task – Drag and Drop

3. Double click on the Data Flow Task to see the Data Flow design surface.
4. From the SSIS Toolbox, Drag and Drop Amazon Redshift Source in the data flow designer surface.
   

   SSIS Amazon Redshift Source – Drag and Drop

5. Now, We need Amazon Redshift Connection. Click here for creating Amazon Redshift Connection.

Configure SSIS ZS Amazon Redshift Source – Query Mode

1. Here, we are getting data using Query Mode with dynamic expression value.
2. Let’s Create Variables, In Visual Studio Right Click on Design Panel, Select Variables.
   

   SSIS Create Variables

3. Make sure correct DataType and Value, use the following image.
   

   SSIS Created Variables with Value

4. Double click on Amazon Redshift Source to configure it.
5. Select the connection we have created before, set AccessMode to Query and use the following SQL Query(You can replace table or column name) and insert Variables we have created before.
   

   SELECT  * FROM "public"."tbl_pg_Orders"
   WHERE "CustomerID" like '%R%' AND "OrderAmount" > '{{User::OrderAmount}}' AND
   "{{User::ColumnName}}" = '{{User::Value}}'

   

   SSIS Amazon Redshift Source – Configure with SQL Query Mode

6. Click on Preview button to see Data Preview and OK button to save configure setting UI.
7. Here, you can use any of our suitable ZS Destination Adapter.
8. From the SSIS toolbox drag and drop Trash Destination on the Data Flow designer surface.
   

   SSIS Trash Destination – Drag and Drop

9. Now single click on the Amazon Redshift Source, once you see the blue arrow from the source…connect it to Trash Destination.
10. Double click on Trash Destination to Configure it.
    

    SSIS Trash Destination – Configure

11. Click on OK button to save Trash Destination configure setting UI.
12. That’s all, just Run or Execute the package and verify data.
    

    SSIS Amazon Redshift Source – Run or Execute the Package

Configure SSIS ZS Amazon Redshift Source – Table Mode

1. Double click on Amazon Redshift Source to configure it.
2. Select the connection we have created before. Set Access Mode to Table and Select Table from table Drop down list.
   

   SSIS Amazon Redshift Source – Configure with Table Mode

3. Click on Preview button to see Data Preview and OK button to save Amazon Redshift Source configure setting UI.
4. From the SSIS toolbox, Drag and Drop Trash Destination on the Data flow designer surface.
   

   SSIS Trash Destination – Drag and Drop

5. Now single click on the Amazon Redshift Source, once you see the blue arrow from a source… connect it to Trash Destination.
6. Double click on ZS Trash Destination to configure it.
   

   SSIS Trash Destination – Configure

7. Click on OK button to save Trash Destination configure setting UI.
8. That’s all, just run or execute your package and see data.
   

   SSIS Amazon Redshift Source – Run or Execute Package

Conclusion

In this blog, we learn How to Read / Write Amazon Redshift Data in SSIS. We used Amazon Redshift Destination for write data and Amazon Redshift Source to read data. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• Amazon Redshift Source
• Amazon Redshift Destination
• Document of Amazon Redshift
• Help File:
  • Amazon Redshift Source
  • Amazon Redshift Destination

The post How to Read / Write Amazon Redshift Data in SSIS appeared first on ZappySys Blog.

In this article, we will see how to read How to write data into CSV file in SSIS from SQL Server. We will also explain how to compress (gzip) and split  CSV files into multiple files. To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. We will use the ZS CSV File Destination component for this article.

What is CSV File Destination?

CSV File Destination can be used to write data in CSV / TSV file format. You can also split large files by row count or size at runtime. It also supports writing files directly in a compressed format such as GZip (*.gz).

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).

Component Mentioned in this article

CSV File Destination

How to Extract Data from SQL Server Table to CSV File Destination.

In this tutorial, we are going to load data from SQL Server Table using OLE DB Source and ZS CSV File Destination. You can use ZS DummyData Source too for practice it.

1. Firstly, you need to Download and Install SSIS ZappySys PowerPack.
2. Once you finished the first step, Open Visual Studio and Create New SSIS Package Project.
3. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

4. Double click on the Data Flow task to see the Data Flow designer surface.
5. Here, In Visual Studio, drag and drop the OLE DB Source and CSV File Destination in the design surface and join the tasks with the blue arrow.
   

   SSIS OLE DB Source and CSV File Destination – Drag and Drop

6. Here, you can use our ZS Data Source too.
7. But, we are going through OLE DB Source so need OLE DB Connection and File Connection for CSV File Destination.

How to create OLE DB Connection.

1. Lets, create the first OLE DB Connection.
2. Let’s, Right-click on Connection Managers Panel to Create OLE DB Connection, so you can use Source and Context Menu will appear, Select New OLE DB Connection from the Context Menu.
   

   Create OLE DB Connection

3. Now, click on New Button to create Connection.
   

   Create New OLE DB Connection

4. Let’s Configure Connection Manager, just Follow steps one by one as we have created.
   

   Configure OLE DB Connection Manager

5. Click on Test Connection to see correct configure it.
6. Click on OK button to save connection configure setting UI.

How to Create a File Connection.

1. Let’s, Right-click on Connection Managers Panel to Create File Connection, so you can use Destination Path, and Context Menu will appear, Select New Connection from the Context Menu. In the File Connection Manager, select the Usage type and specify the path of the CSV file.
   CSV File: You can download good one CSV file from here for practice.
   

   https://zappysys.com/downloads/files/test/invoices.csv
   https://zappysys.com/downloads/files/test/invoices.csv.gz
   https://zappysys.com/downloads/files/test/invoices.csv.zip

   

   Create a New File Connection

2. Click on OK button to save connection configure setting UI.

Let’s see how to write data from SQL Server Table to CSV File.

1. Double click on OLE DB Source for configure it.
2. Let’s Configure in Connection Manager, just follow below image steps.
   

   Configure OLE DB Source Editor

3. Click OK to Save OLE DB Source Editor UI Settings.
4. Let’s configure CSV File Destination, Double click on it, In the File Connection Manager, select the file connection created before.
   

   SSIS ZS CSV File Destination – Configure Connection Manager Tab

5. In the Component Properties, set the Append property to False, enter the File Path and set the Overwrite file if exists to True.
   

   SSIS ZS CSV File Destination – Configure Component Properties Tab

6. In the Input Columns Tab, check the columns that you want as load data.
   

   SSIS ZS CSV File Destination – Configure Input column Tab

7. Click on OK button to save configure setting UI.
8. Finally, run the package and the data will be exported from source to destination immediately.
   

   SSIS ZS CSV FIle Destination – Execute the Package

How to do file split using CSV File Destination.

1. Double click on CSV File Destination to configure it.
2. In the Component Properties set Enable File Split to True and File Split Mode BySize.
   

   SSIS ZS CSV File Destination – File Split

3. Click OK button to save configure setting UI.
4. That’s all, Now you can run or execute the package and you will see numbers of CSV file in Destination Folder.

How to do file compression using CSV File Destination.

1. Double click on CSV File Destination to configure it.
2. In the Component Properties set Compression Type to GZip or Zip and File Split Mode BySize.
   

   SSIS ZS CSV File Destination – Compression Type

3. Click OK button to save configure setting UI. and You can Run or Execute.

How to do the maximum number of rows per file using CSV File Destination.

1. Double click on CSV File Destination to configure it.
2. In the Component Properties, you can set Max Rows Per File and Max Size Per File as per your need.
   

   SSIS ZS CSV File Destination – Max Rows Per File

3. Click OK button to save configure setting UI. and You can Run or Execute.

Conclusion

After all, we saw you how to write data into CSV File from SQL Server Table using OLE DB Connection, File Connection, and load into CSV File. We also learned, how to do various things like split CSV File, Compression File and the maximum number of rows per file of data. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following URL for more information.

• Help File: CSV File Destination

The post How to write data into CSV file in SSIS (GZip / Split) appeared first on ZappySys Blog.