ZappySys provide high-performance drag and drop connectors for Amazon Redshift Integration. In our previous post we saw how to bulk load SQL data into Redshift using S3 staging technique (COPY command). Now in this post, you will see How to Read / Write Amazon Redshift Data in SSIS Data flow without need for S3 Staging. We will use  ZS Amazon Redshift components (Source / Destination). If you need to Upsert / delete data from Redshift Table then use Upsert Destination (Not discussed in this article).

ZappySys developed many AWS related components but in this article, we will look at 2 Components for Amazon Redshift Integration Scenarios (Read and Write). We will discuss how to use SSIS Amazon Redshift Source and SSIS Amazon Redshift Destination. Amazon Redshift Source connector supports familiar SQL Like query language so any DBA or SQL user can learn it in no time. ZappySys developed a highly sophisticated engine to turn your SQL Query into native Amazon Redshift query. Load query at runtime from the variables or Direct value.

Prerequisite

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Thirdly, You have obtained Amazon Redshift Credential.
4. Finally, do not forget to install ZappySys SSIS PowerPack.

Component Mention in this article

	Amazon Redshift Source
	Amazon Redshift Destination

What is Amazon Redshift

Before we talk How to Read / Write Amazon Redshift Data in SSIS lets talk what is Amazon Redshift (or sometimes referred to as AWS Redshift). Amazon Redshift is a Cloud-based Data warehouse service. This type of system also referred to as MPP (Massively Parallel Processing). Amazon Redshift uses a highly modified version of PostgreSQL Engine behind the scene. Amazon Redshift provides the advantage of Scale as you go, at a very low cost compared to onsite dedicated hardware/software approach. In this article, we will try to learn How to Read / Write Amazon Redshift Data in SSIS.

Getting Started

In order to start, we will show several examples. ZappySys includes an Amazon Redshift Source and Destination, that will help you How to Read / Write Amazon Redshift Data in SSIS. You can connect to your Amazon Redshift Account by entering credentials.

SSIS Amazon Redshift Source Connector can be used to read data from Amazon Redshift. You can use a simple Table mode or write custom SQL Query to extract desired data. You can use multiple modes (i.e. Table or Query) to read data from the Amazon Redshift Database. Support for data preview and max rows and Dynamic query (using SSIS Variable placeholder e.g. select * from mytable where id='{{User:: MyVar}}’).

SSIS Amazon Redshift Destination can be used to bulk insert a large number of records to Amazon Redshift storage from any data source using SSIS. Amazon Redshift is a cloud-based Data Warehouse service from Amazon AWS. Here, you can bulk Insert, Update and Delete Amazon Redshift data and supporting the batch size.

Click here to see Amazon Redshift related articles.

Write Data using Amazon Redshift Destination in SSIS

Overview

In this tutorial, you will learn how to write data in Amazon Redshift Database. You can use ZS DummyData Source for practice it or select suitable ZS Source Adapter (In this case it’s from SQL Server Source). So, Let’s start with an example. In this SSIS we will write data into Amazon Redshift Database using ZS Amazon Redshift Destination.

How to write data into Amazon Redshift Table

1. In order to connect to the Amazon Redshift Database from SSIS, you will need credentials. Ask your SysAdmin or responsible person to provide that you.
2. For this sample exercise, we will create a table in Redshift.
   

   CREATE TABLE public."tbl_pg_Customers"
   (
       "RecordID" character varying(50) NOT NULL,
       "CustomerID" character varying(50),
       "CustomerName" character varying(50),
       "Address1" character varying(50),
       "City" character varying(50),
       "State" character varying(50),
       "Zip" character varying(50),
       "Country" character varying(50),
       PRIMARY KEY ("RecordID")
   );

3. Once you have created a Table, you may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
4. In your SSIS toolbox, you should see many new Icons starting with “ZS” (i.e. ZappySys). If you don’t see them then check this.
5. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

6. Double click on the Data Flow Task to see Data flow designer surface.
7. Here, In Visual Studio, Drag and Drop the OLE DB Source and Amazon Redshift Destination in the design surface and join the components with the blue arrow.
   

   SSIS OLE DB Source and Amazon Redshift Destination – Drag and Drop

8. Now, we need OLE DB Connection and Amazon Redshift Connection. Click here for Create Amazon Redshift Connection.

How to Create an OLE DB Connection.

1. Let’s, Right-click on Connection Managers Panel to Create OLEDB Connection, so you can use Source and Context Menu will appear, Select New OLEDB Connection from the Context Menu.
   

   SSIS OLE DB Connection – Create Connection

2. Now click on the New button to create a connection.
   

   SSIS OLE DB Connection – Create New Connection

3. Let’s configure connection Manager, just follow the steps one as we have created in following the image.
   

   SSIS OLE DB Connection – Configure Connection

4. After all, Click on OK button to save configure setting UI.

Configure SSIS ZS Amazon Redshift Destination

1. Double click on OLE DB Source for configure it.
2. Let’s Configure in Connection Manager, just follow below image steps.
   

   SSIS OLE DB Source – Configure Source Editors

3. Click OK to save OLE DB Source Editor UI settings.
4. Now double click on Amazon Redshift Destination to configure it. In the [Connection Manager] tab Select Amazon Redshift Connection.
   

   SSIS Amazon Redshift Destination – Connection Manager

5. Click on [Component Properties] tab and Pick the Destination Table from Tables dropdown list.
   

   SSIS Amazon Redshift Destination – Component Properties

6. Click on [Column mappings] tab and verify mapping. If the target table is new Table and it’s empty then all input columns will be automatically mapped as shown below. If the table is an existing table with some data then you can manually pick a mapping column by dragging it from source list to target list.
   

   SSIS Amazon Redshift Destination – Column Mappings

7. Execute the package and verify that the data is saved.
   

   SSIS Amazon Redshift Destination – Run or Execute Package

Read Data using Amazon Redshift Source in SSIS

Overview

In this tutorial, you will learn how to read data from Amazon Redshift in SSIS using ZS Amazon Redshift Source adapter. In this SSIS we will read data with SQL Query and Table Mode. So Let’s begin it.

How to Read data from Amazon Redshift Table

1. You may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
2. From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
   

   SSIS Data Flow Task – Drag and Drop

3. Double click on the Data Flow Task to see the Data Flow design surface.
4. From the SSIS Toolbox, Drag and Drop Amazon Redshift Source in the data flow designer surface.
   

   SSIS Amazon Redshift Source – Drag and Drop

5. Now, We need Amazon Redshift Connection. Click here for creating Amazon Redshift Connection.

Configure SSIS ZS Amazon Redshift Source – Query Mode

1. Here, we are getting data using Query Mode with dynamic expression value.
2. Let’s Create Variables, In Visual Studio Right Click on Design Panel, Select Variables.
   

   SSIS Create Variables

3. Make sure correct DataType and Value, use the following image.
   

   SSIS Created Variables with Value

4. Double click on Amazon Redshift Source to configure it.
5. Select the connection we have created before, set AccessMode to Query and use the following SQL Query(You can replace table or column name) and insert Variables we have created before.
   

   SELECT  * FROM "public"."tbl_pg_Orders"
   WHERE "CustomerID" like '%R%' AND "OrderAmount" > '{{User::OrderAmount}}' AND
   "{{User::ColumnName}}" = '{{User::Value}}'

   

   SSIS Amazon Redshift Source – Configure with SQL Query Mode

6. Click on Preview button to see Data Preview and OK button to save configure setting UI.
7. Here, you can use any of our suitable ZS Destination Adapter.
8. From the SSIS toolbox drag and drop Trash Destination on the Data Flow designer surface.
   

   SSIS Trash Destination – Drag and Drop

9. Now single click on the Amazon Redshift Source, once you see the blue arrow from the source…connect it to Trash Destination.
10. Double click on Trash Destination to Configure it.
    

    SSIS Trash Destination – Configure

11. Click on OK button to save Trash Destination configure setting UI.
12. That’s all, just Run or Execute the package and verify data.
    

    SSIS Amazon Redshift Source – Run or Execute the Package

Configure SSIS ZS Amazon Redshift Source – Table Mode

1. Double click on Amazon Redshift Source to configure it.
2. Select the connection we have created before. Set Access Mode to Table and Select Table from table Drop down list.
   

   SSIS Amazon Redshift Source – Configure with Table Mode

3. Click on Preview button to see Data Preview and OK button to save Amazon Redshift Source configure setting UI.
4. From the SSIS toolbox, Drag and Drop Trash Destination on the Data flow designer surface.
   

   SSIS Trash Destination – Drag and Drop

5. Now single click on the Amazon Redshift Source, once you see the blue arrow from a source… connect it to Trash Destination.
6. Double click on ZS Trash Destination to configure it.
   

   SSIS Trash Destination – Configure

7. Click on OK button to save Trash Destination configure setting UI.
8. That’s all, just run or execute your package and see data.
   

   SSIS Amazon Redshift Source – Run or Execute Package

Conclusion

In this blog, we learn How to Read / Write Amazon Redshift Data in SSIS. We used Amazon Redshift Destination for write data and Amazon Redshift Source to read data. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• Amazon Redshift Source
• Amazon Redshift Destination
• Document of Amazon Redshift
• Help File:
  • Amazon Redshift Source
  • Amazon Redshift Destination

The post How to Read / Write Amazon Redshift Data in SSIS appeared first on ZappySys Blog.

In this article, you will learn Using SSIS Trash Destination and Dummy Data Source (FREE). You can write data into the file using Trash Destination, You can use Trash Destination to write data into a raw file from any source. We will use FREE Dummy Data Source to generate random data for testing purpose. Trash Destination can be used to terminate your dataflow pipeline without doing any expensive processing on incoming data. This component is very useful during testing where you want to focus on the source and transforms but don’t really care about the destination.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).

Components mention in this article

	ZS Trash Destination
	ZS Dummy Data Source

Getting Started

SSIS ZS Trash Destination and Dummy Data Source are free. If you check save Data to file then by default all input columns are logged to file. If you wish to log the selected input column then use Advanced Editor. This component does not support an error output. Any bad random value ill cause error or Truncation (e.g. If your sample value is 5 characters long but output column length is set to 3 chars). If you have error output attached with RedirectRow option on error, Truncation then you can capture bad rows along with an exact reason of error.

Write data into the file using Trash Destination

Overview

In this tutorial, you will learn how to Using SSIS Trash Destination and Dummy Data Source.  Let’s use ZS DummyData Source to write data into ZS Trash Destination. Here you can select our, any of suitable Source Adapter.

Method 1 – Using ZS DummyData Source

1. Open Visual Studio and Create New SSIS Package Project.
2. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

3. Double click on the Data Flow task to see the Data Flow designer surface.
4. From the SSIS toolbox drag and drop ZS DummyData Source on the data flow designer surface.
   

   Drag and Drop ZS DummyData Source

5. Double click DummyData Source and select Customer profile from Template dropdown.
6. Once you change Template you will notice that a lot of columns automatically added. Notice how sample values are set for many columns along with their value delimiter. You may use many inbuilt placeholders in your sample values such as ({NULL} for a null value, {TAB} for the tab character, {CRLF} for new line).
7. Change [How many rows] setting to 5000.
   

   SSIS DummyData Source – Configure

8. Click OK to save settings.
9. We can use DummyData Source like this, click here.
10. Here, you can insert data into SQL Server Table, Check Here.
11. Now, Just Drag and Drop Our Free ZS Trash Destination on the data flow designer surface from SSIS Toolbox.
    

    SSIS Trash Destination – Drag and Drop

12. Now single click on ZS DummyData Source once you see the blue arrow from the source … connect it to ZS Trash Destination.
13. Double click on ZS Trash Destination to configure it.
    

    SSIS Trash Destination – Configure

14. Click OK to save configure setting UI.
15. Right-click on the path and Add Data Viewer.
16. Execute the package and verify source data in the data viewer.
    

    Execute Package and Verify Data Source

Method 2 – Using OLE DB Source

Let’s use ZS OLE DB Source to write data into ZS Trash Destination.

1. Open Visual Studio and Create New SSIS Package Project.
2. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

3. We are going through SQL Server Database, need OLE DB Connection for Data Source.

How to create OLE DB Connection.

1. Let’s, Right-click on Connection Managers Panel to Create OLEDB Connection, so you can use Source and Context Menu will appear, Select New OLEDB Connection from the Context Menu.
   

   Create OLE DB Connection

2. Now, click on New button for create Connection.
   

   Create New OLE DB Connection

3. In the Connection Manager, just follow the steps in the following image we have created.
   

   OLE DB Connection Configure

4. Click on OK button to save connection configure setting UI.

How to write into Trash Destination from OLE DB Source

1. Drag and Drop OLE DB Source from SSIS toolbox.
   

   OLE DB Source – Drag and Drop

2. Double click on OLE DB Source for configure it.
3. In the Connection Manager, just follow the steps in the following image we have created.
   

   OLE DB Source – Configure

4. Click on OK button to save OLE DB Source Editor setting UI.
5. Now, Just Drag and Drop Our Free ZS Trash Destination on the data flow designer surface from SSIS Toolbox.
   

   SSIS ZS Trash Destination – Drag and Drop

6. Now, single click on OLE DB Source once you see the blue arrow from the source … connect it to ZS Trash Destination.
7. Double click on ZS Trash Destination to configure it.
   

   SSIS ZS Trash Destination – Configure

8. Click OK to save configure setting UI.
9. Right-click on the path and Add data viewer.
10. Execute the package and verify source data in the data viewer.
    

    Execute Package and Verify Data

Conclusion

In this blog, we learned how to write data into a raw file and read data in SSIS. We used the Trash Destination for write data and Dummy Data Source to read data. You can download SSIS PowerPack from here to try many other scenarios not discussed in this blog. SSIS PowerPack is a collection of 70+ high performance, just drag and drop connector/task for SSIS(i.e. Microsoft SQL Server Integration Service). After you install SSIS PowerPack you will see many new Custom Tasks and Components in your Visual Studio SSIS Toolbox.

References

Finally, you can use the following links for more information:

• ZS Trash Destination
• ZS Dummy Data Source
• Help File:
  • ZS Trash Destination
  • ZS Dummy Data Source

The post Using SSIS Trash Destination and Dummy Data Source appeared first on ZappySys Blog.