ZappySys provides high-performance drag and drop connectors for PostgreSQL Integration. In this post you will see how to Execute SQL query against PostgreSQL in SSIS, in various SQL commands such as SELECT, INSERT, DROP, COPY, CREATE, etc using PostgreSQL Execute SQL Task can be used to execute DDL and DML statements for PostgreSQL without needing any extra driver. PostgreSQL is a powerful, open-source relational database with a strong reputation for reliability, feature robustness, and performance.

In nutshell, this post will focus on how to Execute SQL query against PostgreSQL in just a few clicks.

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Thirdly, You have obtained PostgreSQL Credential.
4. Finally, do not forget to install ZappySys SSIS PowerPack

Component Mentioned in this article

PostgreSQL Execute SQL Task

What is PostgreSQL?

PostgreSQL is a powerful, open source object-relational database system that uses and extends the SQL language combined with many features that safely store and scale the most complicated data workloads. The origins of PostgreSQL date back to 1986 as part of the POSTGRES project at the University of California at Berkeley and has more than 30 years of active development on the core platform.

PostgreSQL has earned a strong reputation for its proven architecture, reliability, data integrity, robust feature set, extensibility, and the dedication of the open source community behind the software to consistently deliver performant and innovative solutions. PostgreSQL runs on all major operating systems, has been ACID-compliant since 2001, and has powerful add-ons such as the popular PostGIS geospatial database extender. It is no surprise that PostgreSQL has become the open source relational database of choice for many people and organisations.

Getting started with using PostgreSQL has never been easier – pick a project you want to build, and let PostgreSQL safely and robustly store your data.

Download and Install PostgreSQL

Please go to below download link and download the Windows version Binary package and install it and follow the steps from the installer.

https://www.postgresql.org/download/

Download PostgreSQL

Create a PostgreSQL Database

1. To do this, go to Start Menu and search for pgAdmin and we will first open the pgAdmin Panel.
   

   Open pgAdmin from Start Menu

2. It will open the pgAdmin Panel in the system default browser. Select Server and Right-click on Databases to create a database.
   

   PostgreSQL Create Database

3. Now, Enter the Database name and click on Save to create it.
   

   Create a PostgreSQL Database

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS PostgreSQL ExecuteSQL Task that will help you how to Create and Drop table by Execute SQL query against PostgreSQL in SSIS. ZS PostgreSQL ExecuteSQL Task support execute queries against PostgreSQL without ODBC driver, load query at runtime from variable, file or Direct value, 100% Managed code means its 32bit and 64bit compatible, you can issue COPY, UNLOAD commands very easily, Support for SQL Server 2017, 2016, 2014, 2012, 2008 (32/64 bit) and Click here to see articles related to PostgreSQL.

You can connect to your PostgreSQL database by entering your PostgreSQL credentials.

Execute SQL query against PostgreSQL in SSIS

Let’s start with an example. In this tutorial, we will learn how to Create and Drop table using SQL query against PostgreSQL server using ZS PostgreSQL ExecuteSQL Task.

1. In order to connect to the PostgreSQL Database from SSIS, you will need Credentials. Ask your SysAdmin or responsible person to provide that to you.
2. Once you finished the above steps, Open Visual Studio and Create New SSIS Package Project.
3. In Visual Studio, drag and drop the ZS PostgreSQL ExecuteSql Task in the design surface.
   

   SSIS ZS PostgreSQL ExecuteSQL Task – Drag and Drop

4. We need a PostgreSQL Database connection. Click here to create a connection.
5. Now, double click on ZS PostgreSQL ExecuteSql Task to configure it.
6. In the PostgreSQL ExecuteSql Task, add the following SQL code to DROP a table if it exists and CREATE a table named pg_equipment in the SQL Statement section.
   You can replace the table name from the following code.
   

   DROP TABLE IF EXISTS playground_equip;
   CREATE TABLE pg_equipment (
   	equip_id serial PRIMARY KEY,
   	type varchar (50) NOT NULL,
   	color varchar (25) NOT NULL,
   	location varchar(25) check (location in ('north', 'south', 'west', 'east', 'northeast', 'southeast', 'southwest', 'northwest')),
   	install_date date
   );

   

   SSIS ZS PostgreSQL ExecuteSQL Task – Configure

7. In the above step, you can Select Variable as Sql Source Type.
8. Click OK to save ZS PostgreSQL ExecuteSql Task UI settings.
9. Finally, Right Click on Task and Select Execute Task to run it.

Conclusion

After all, In this blog, we learned Execute SQL query against PostgreSQL in SSIS using ZS PostgreSQL ExecuteSQL Task. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• About PostgreSQL
• PostgreSQL Documentation
• PostgreSQL Query
• Help File: PostgreSQL ExecuteSQL Task

The post Execute SQL query against PostgreSQL in SSIS appeared first on ZappySys Blog.

In our previous blog, we saw you how to read/write Azure Table Storage data. Now in this blog, we will see How to Read/Write Azure Queue Storage data in SSIS. To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like perform other operations on Azure Blob Storage Files (e.g. Download, Upload, Create, Delete) then check these articles.

In nutshell, this post will focus on How to Read/Write Azure Queue Storage data in SSIS using the following Source/Destination.

Component Mentioned in this article

	Azure Queue Storage Source
	Azure Queue Storage Destination

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Download and install Microsoft Azure Storage Emulator
4. Download and install Microsoft Azure Storage Explorer
5. Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Storage) then you can skip Step #3

What are Azure Queues?

Azure Queue storage is a service for storing large numbers of messages. You access messages from anywhere in the world via authenticated calls using HTTP or HTTPS. A queue message can be up to 64 KB in size. A queue may contain millions of messages, up to the total capacity limit of a storage account.

Common uses

Microsoft Azure Queue Concept

Common uses of Queue storage include:

• Creating a backlog of work to process asynchronously
• Passing messages from an Azure web role to an Azure worker role
• The Queue service contains the following components:

Getting Started

In order to start, we will show several examples. Zappysys includes an SSIS Azure Queue Storage Source that will help you in reading data of Queue and SSIS Azure Queue Storage Destination that will help you to write data in Queue. Here we are showing you is, how to Read/Write Azure Queue Storage data in SSIS.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of the local Azure Storage Emulator.

Setup Azure Storage client tools and Create Queue

1. Once you have downloaded and installed storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
   

   Azure Storage Emulator Physical Location

2. If you can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
   

   Command Prompt Screen after Microsoft Azure Storage Emulator Started

3. Now, you have to download and install Microsoft Azure Storage Explorer and then you can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Explorer Location.

4. If you don’t have Azure Storage account then it’s easy to get FREE Trial Azure Storage account or use your MSDN to get credit each month which will be more than enough for real testing. If you prefer to avoid all this hassle then Microsoft provides another great way to test Azure Storage functionality is totally offline mode on your local machine. You can just download Azure Storage Emulator and start testing.
5. Click here to learn more about how to configure Azure Storage Emulator for Testing and Development purposes. In order to connect to Azure Storage Service from SSIS, you will need Storage AccountName and an AccessKey. Ask your SysAdmin or responsible person to provide that information to you. Here are sample Account Name and Access Key(this is just example key which may differ in your case)
   

   Account Name: mystorageaccount 
   Access Key: Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==

6. For Creating a Queue, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Queues. Right-click on Queues and Select Create Queue and give a name you want.
   

   Microsoft Azure Storage Explorer – Create Queue

Write data using ZS Azure Queue Storage Destination

In this tutorial, you will learn how to write data into Azure Queue Storage. We will use ZS DummyData Source for practice it but you can use a suitable Source Adapter. So, Let’s start, In this SSIS we will write data into Azure Queue Storage using ZS Azure Queue Storage Destination.

1. Once you have created Azure Storage Queue and you got valid Account Name and AccessKey (for the local emulator you don’t need this) you may proceed to create new SSIS package. In BIDS/SSDT create a new SSIS package.
2. From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface and double click on it too.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

3. Click here to create Azure Storage Connection.
4. From the SSIS toolbox drag and drop Dummy Data Source on the dataflow designer surface.
   

   DummyData Source – Drag and Drop

5. Double click Dummy Data Source to configure it. From Template pick JsonDocuments and enter row count=100 to generate 100 JSON documents and click OK to save.
   

   DummyData Source – Configure

6. Let’s drag and drop Azure Queue Storage Destination on the dataflow designer surface from the SSIS toolbox.
7. Now single click on the Dummy Data Source, once you see the blue arrow from the source connect it to Azure Queue Destination.
8. Double click Azure Queue Destination to configure it. Select the connection we have created before.
   

   Configure Connection Manager

9. Click on [Component Properties] tab and select the Target Queue from the dropdown list.
   

   Configure Component Properties

10. Click on [Mappings] tab and verify mapping. Map Document column from upstream to the MessageText target column.
    

    Configure Column Mappings

11. Click OK to save settings.
12. Execute the package and verify target data by going to your Azure Queue. We use Azure Storage Explorer.
    

    SSIS Execute Package

    

    Microsoft Azure Explorer – Verify Queue Data

Read data using ZS Azure Queue Storage Source

1. Here you may proceed In BIDS/SSDT to create a new SSIS package.
2. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   SSIS Data Flow Task – Drag and Drop

3. Double click on the Data Flow task to see the Data Flow designer surface.
4. From the SSIS toolbox drag and drop ZS Azure Queue Storage Source on the dataflow designer surface.
   

   SSIS ZS Azure Queue Source – Drag and Drop

5. Double click Azure Queue Storage Source to configure it.
6. Select Azure Storage Connection we have created before and select queue from the dropdown and click on preview.
   

   Configure Azure Queue Source

7. Click on OK button to save configure.
8. Now, you can use a suitable Destination Adapter from here. In this case, we used Trash Destination.
9. From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface.
   

   SSIS ZS Trash Destination – Drag and Drop

10. Now single click on the Azure Queue Storage Source once you see the blue arrow from the source … connect it to Trash Destination.
11. Double click on ZS Trash Destination to Configure it.
    

    SSIS ZS Trash Destination – Configure

12. Click on OK button to save Trash Destination configure setting UI.
13. Execute the package and verify source data in the data viewer.
    

    SSIS ZS Azure Queue Source – Execute the Package

14. After messages read from Azure Queue, the same message cannot be read again until VisibilityTimeout. By default Queue, visibility timeout is used (-1 means use default) but you can override it on the source UI.

Conclusion

In this blog, we learn How to Read/Write Azure Queue Storage data in SSIS. We used the Azure Queue Destination for write data and Azure Queue Source to read data. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• Azure Queue Source
• Azure Queue Destination
• Microsoft Azure Queues
• Help File: Azure Queue Source, Azure Queue Destination, Trash Destination

The post How to Read/Write Azure Queue Storage data in SSIS appeared first on ZappySys Blog.

In this article, we will see how to read How to write data into CSV file in SSIS from SQL Server. We will also explain how to compress (gzip) and split  CSV files into multiple files. To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. We will use the ZS CSV File Destination component for this article.

What is CSV File Destination?

CSV File Destination can be used to write data in CSV / TSV file format. You can also split large files by row count or size at runtime. It also supports writing files directly in a compressed format such as GZip (*.gz).

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).

Component Mentioned in this article

CSV File Destination

How to Extract Data from SQL Server Table to CSV File Destination.

In this tutorial, we are going to load data from SQL Server Table using OLE DB Source and ZS CSV File Destination. You can use ZS DummyData Source too for practice it.

1. Firstly, you need to Download and Install SSIS ZappySys PowerPack.
2. Once you finished the first step, Open Visual Studio and Create New SSIS Package Project.
3. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

4. Double click on the Data Flow task to see the Data Flow designer surface.
5. Here, In Visual Studio, drag and drop the OLE DB Source and CSV File Destination in the design surface and join the tasks with the blue arrow.
   

   SSIS OLE DB Source and CSV File Destination – Drag and Drop

6. Here, you can use our ZS Data Source too.
7. But, we are going through OLE DB Source so need OLE DB Connection and File Connection for CSV File Destination.

How to create OLE DB Connection.

1. Lets, create the first OLE DB Connection.
2. Let’s, Right-click on Connection Managers Panel to Create OLE DB Connection, so you can use Source and Context Menu will appear, Select New OLE DB Connection from the Context Menu.
   

   Create OLE DB Connection

3. Now, click on New Button to create Connection.
   

   Create New OLE DB Connection

4. Let’s Configure Connection Manager, just Follow steps one by one as we have created.
   

   Configure OLE DB Connection Manager

5. Click on Test Connection to see correct configure it.
6. Click on OK button to save connection configure setting UI.

How to Create a File Connection.

1. Let’s, Right-click on Connection Managers Panel to Create File Connection, so you can use Destination Path, and Context Menu will appear, Select New Connection from the Context Menu. In the File Connection Manager, select the Usage type and specify the path of the CSV file.
   CSV File: You can download good one CSV file from here for practice.
   

   https://zappysys.com/downloads/files/test/invoices.csv
   https://zappysys.com/downloads/files/test/invoices.csv.gz
   https://zappysys.com/downloads/files/test/invoices.csv.zip

   

   Create a New File Connection

2. Click on OK button to save connection configure setting UI.

Let’s see how to write data from SQL Server Table to CSV File.

1. Double click on OLE DB Source for configure it.
2. Let’s Configure in Connection Manager, just follow below image steps.
   

   Configure OLE DB Source Editor

3. Click OK to Save OLE DB Source Editor UI Settings.
4. Let’s configure CSV File Destination, Double click on it, In the File Connection Manager, select the file connection created before.
   

   SSIS ZS CSV File Destination – Configure Connection Manager Tab

5. In the Component Properties, set the Append property to False, enter the File Path and set the Overwrite file if exists to True.
   

   SSIS ZS CSV File Destination – Configure Component Properties Tab

6. In the Input Columns Tab, check the columns that you want as load data.
   

   SSIS ZS CSV File Destination – Configure Input column Tab

7. Click on OK button to save configure setting UI.
8. Finally, run the package and the data will be exported from source to destination immediately.
   

   SSIS ZS CSV FIle Destination – Execute the Package

How to do file split using CSV File Destination.

1. Double click on CSV File Destination to configure it.
2. In the Component Properties set Enable File Split to True and File Split Mode BySize.
   

   SSIS ZS CSV File Destination – File Split

3. Click OK button to save configure setting UI.
4. That’s all, Now you can run or execute the package and you will see numbers of CSV file in Destination Folder.

How to do file compression using CSV File Destination.

1. Double click on CSV File Destination to configure it.
2. In the Component Properties set Compression Type to GZip or Zip and File Split Mode BySize.
   

   SSIS ZS CSV File Destination – Compression Type

3. Click OK button to save configure setting UI. and You can Run or Execute.

How to do the maximum number of rows per file using CSV File Destination.

1. Double click on CSV File Destination to configure it.
2. In the Component Properties, you can set Max Rows Per File and Max Size Per File as per your need.
   

   SSIS ZS CSV File Destination – Max Rows Per File

3. Click OK button to save configure setting UI. and You can Run or Execute.

Conclusion

After all, we saw you how to write data into CSV File from SQL Server Table using OLE DB Connection, File Connection, and load into CSV File. We also learned, how to do various things like split CSV File, Compression File and the maximum number of rows per file of data. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following URL for more information.

• Help File: CSV File Destination

The post How to write data into CSV file in SSIS (GZip / Split) appeared first on ZappySys Blog.

In this article, we will see how to read Walmart API data in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call Walmart APIs can be useful for any developer regardless of which programming language or toolset you use.

Are you a Walmart.com Supplier or Marketplace Seller? Then you may need to click here to visit the site that is focused on these profiles.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of Walmart API. Click here to create API.

Components Mentioned in this article

	ZS Logging Task
	JSON Source(REST API or File)

What is Walmart?

Walmart is the world’s largest retailer, and the Walmart Open API provides access to our extensive product catalog, thus enabling digital distribution partners to earn substantial affiliate revenues from customer referrals.

Read Walmart API Data using SSIS JSON Source

Let’s start with an example. We use SSIS JSON Source component to make the call to Walmart API, we will read Walmart data and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

Get Dynamic Token (Configure ZS-HTTP Connection)

To get API response data using SOAP / REST API call, you need to get an API Token from your login request URL. Follow the steps mentioned below to get API Token:

1. Now, Right Click on Connection Managers Window, and one Context menu appears for creating a new connection manager.
   

   Create a New HTTP Connection.

2. Select ZS HTTP Connection Manager from the list of Connection Managers and, double-click on it or click on Add button to add a new Connection Manager. By Default, The created connection manager is of Package-Level. You can set Project Level Connection manager too If you want.
   

   Select ZS-HTTP Connection from SSIS Connection Manager.

3. In the SSIS ZS HTTP Connection Manager, Go to General Tab and Configure HTTP Connection Manager. Enter API URL, Set Credentials Type to Dynamic Token – Two-Step Auth, and configure some basic settings for the same. We used Login URL as just example but you can enter any valid API URL endpoint here. Actual Login URL goes on Dynamic Token Tab (see next step).
   

   https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20

   WM_SEC.ACCESS_TOKEN

   

   ZS HTTP Connection – Configure General Tab

4. Now, In the Dynamic Token Tab do configure like below image. Please click here to refer about Walmart API Token.
   

   https://marketplace.walmartapis.com/v3/token

   grant_type=client_credentials

   Accept: application/json
   Cache-Control: no-cache
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>
   Authorization: Basic <<[$userid$]:[$password$],FUN_BASE64ENC>>

   

   ZS HTTP Connection – Dynamic Token Tab

5. On the Response Settings tab configure as below
   

   Walmart API – Extract Token – Response Settings

   1. Extract Mode = Extract from Body
   2. Expression Type = Json
   3. Response Content Filter =

      $.access_token

6. Click on OK button to save HTTP Connection configure settings.

That’s it we have successfully configured Connection for Walmart API in SSIS. In the next section, we will see how to use this connection and read various data from Walmart API.

Read Walmart API Data in SSIS

Once we have done creating HTTP Connection Manager we can move forward to read Walmart API data inside Data Flow. So lets Configure SSIS JSON / REST API Source.

1. Now, Drag and drop Data Flow Task from SSIS Toolbox in the currently open project and double click it to edit.
   
   

   Drag and Drop Data Flow Task.

2. Double click on the Data Flow task to see the Data Flow designer surface.
3. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.
   

   Drag and Drop JSON Source(REST API or File).

4. Double click JSON Source and enter the following URL as below.
   

   https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20

5. Check Use Credentials and select existing HTTP connection we have created already it.
6. Enter Following HTTP Headers. If you prefer Raw edit .. Just click on Raw Edit (#6 in screenshot) and paste below headers. Make sure to replace YourClientIdGoesHere and YourSecretGoesHere
   

   Content-Type: application/xml
   Accept: application/xml
   Authorization: Basic <<YourClientIdGoesHere:YourSecretGoesHere,FUN_BASE64ENC>>
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>

    
7. Select Array Filter (ItemResponse node) or type $.ItemResponse[*]  as below.
   

   ZS JSON Source – Configure

8. That’s it, you are ready to load Wallmart API Data into SQL Server and you can use more ZappySys Destination components.
9. For pagination call please click here for reference to this link.

Load Walmart API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Walmart API integration using ODBC Driver (Power BI, Excel, MS Access)

So far we have seen how to access Walmart API using SSIS. However there will be a time when you like to access API data inside other Apps using ODBC Driver. ZappySys Provides JSON/XML/CSV drivers for virtually any API including Walmart API. Here is how to access Walmart API using XML ODBC Driver.

1. Type “ODBC” in Start menu. Open ODBC Data sources (64 bit)
   
2. Click Add on User DSN tab (if you need to use DSN under different Account …i.e. Service Account then click on System DSN Tab)
3. Select ZappySys XML Driver
4. When New DSN UI shows up, Change DSN name to anything you like (e.g. WalmartAPI)
5. Click on Load ConnectionString button and enter below connection. Make sure to replace few things
   First ######YourClientId####### with your own clientid
   Second ######YourSecret####### with your own secret
   Third ######Base64OfClientIdAndSecret####### with base64 encoded string of

   clientId:Secret

       (both separated with colon). You can use any site like this one to generate base64 string of your clientid and secret (seperated by “:”)
   

   DRIVER={ZappySys XML Driver};
   ElementsToTreatAsArray='ns2:ItemResponse';
   DataPath='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20';
   DataConnectionType=HTTP;
   AuthScheme='{none}';
   Url='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20';
   TokenUrl='https://marketplace.walmartapis.com/v3/token';
   TokenRequestData='grant_type=client_credentials';
   TokenRequestMethod='POST';
   TokenResponseContentFilter='$.access_token';
   TokenRequestHeaders='Accept:application/json||Cache-Control:no-cache||WM_SVC.NAME:Walmart Marketplace||WM_QOS.CORRELATION_ID:TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>||Authorization:Basic <<[$userid$]:[$password$],FUN_BASE64ENC>>';
   TokenAuthHeader='WM_SEC.ACCESS_TOKEN';
   TokenResponseContentType=Json;
   UserName='######YourClientId#######';
   CredentialType=TokenDynamic;
   Password='######YourSecret#######';
   Filter='$.ns2:ItemResponses.ns2:ItemResponse[*]';
   RequestMethod='GET';
   RequestHeaders='Content-Type: application/xml || Accept: application/xml || Authorization: Basic ######Base64OfClientIdAndSecret####### || WM_SVC.NAME: Walmart Marketplace || WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>'

   

   ZappySys Driver – Load Connection String

    

6. Now go to preview tab and enter below query and click Run below SQL.

   select * from $

   –OR–
   NOTE: Change YourClientId and YourSecret in below sql before running

   SELECT * FROM $
   WITH(
   Src='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20'
   ,Header='Content-Type: application/xml
   Accept: application/xml
   Authorization: Basic <<YourClientid:YourSecret,FUN_BASE64ENC>>
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>'
   )

    

Walmart API Integration with Other BI Tools (Power BI, Excel, SSRS, MS Access…)

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

How to encrypt Authorization Token as Base64Encode

1. If you want to directly pass Authorization header Token as Base64Encode(clientId:clientSecret) encrypted in the get token call, you can encrypted it using ZS Logging task and used it. Please refer to this link for the same.
2. In visual studio just Drag and Drop ZS Logging Task in the design panel.
   

   Drag and Drop ZS Logging Task

3. Double click on ZS Logging Task to configure it.
4. In the ZS Logging Task configure like the following image.
   

   Logging Task : Base64Enocde

5. That’s all, Save Base64Encoded token for use in HTTP Connection.

Conclusion

After all, we saw you how to extract information from REST API such as Walmart REST API using HTTP Connection and load into SQL Server. We also learned techniques like How to get Dynamic Token using HTTP Connection. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• Help File: JSON Source(REST API or File), ZS Logging Task
• Walmart API: HTTP Connection

The post How to read Walmart API data in SSIS / ODBC appeared first on ZappySys Blog.

In this article, you will learn Using SSIS Trash Destination and Dummy Data Source (FREE). You can write data into the file using Trash Destination, You can use Trash Destination to write data into a raw file from any source. We will use FREE Dummy Data Source to generate random data for testing purpose. Trash Destination can be used to terminate your dataflow pipeline without doing any expensive processing on incoming data. This component is very useful during testing where you want to focus on the source and transforms but don’t really care about the destination.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).

Components mention in this article

	ZS Trash Destination
	ZS Dummy Data Source

Getting Started

SSIS ZS Trash Destination and Dummy Data Source are free. If you check save Data to file then by default all input columns are logged to file. If you wish to log the selected input column then use Advanced Editor. This component does not support an error output. Any bad random value ill cause error or Truncation (e.g. If your sample value is 5 characters long but output column length is set to 3 chars). If you have error output attached with RedirectRow option on error, Truncation then you can capture bad rows along with an exact reason of error.

Write data into the file using Trash Destination

Overview

In this tutorial, you will learn how to Using SSIS Trash Destination and Dummy Data Source.  Let’s use ZS DummyData Source to write data into ZS Trash Destination. Here you can select our, any of suitable Source Adapter.

Method 1 – Using ZS DummyData Source

1. Open Visual Studio and Create New SSIS Package Project.
2. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

3. Double click on the Data Flow task to see the Data Flow designer surface.
4. From the SSIS toolbox drag and drop ZS DummyData Source on the data flow designer surface.
   

   Drag and Drop ZS DummyData Source

5. Double click DummyData Source and select Customer profile from Template dropdown.
6. Once you change Template you will notice that a lot of columns automatically added. Notice how sample values are set for many columns along with their value delimiter. You may use many inbuilt placeholders in your sample values such as ({NULL} for a null value, {TAB} for the tab character, {CRLF} for new line).
7. Change [How many rows] setting to 5000.
   

   SSIS DummyData Source – Configure

8. Click OK to save settings.
9. We can use DummyData Source like this, click here.
10. Here, you can insert data into SQL Server Table, Check Here.
11. Now, Just Drag and Drop Our Free ZS Trash Destination on the data flow designer surface from SSIS Toolbox.
    

    SSIS Trash Destination – Drag and Drop

12. Now single click on ZS DummyData Source once you see the blue arrow from the source … connect it to ZS Trash Destination.
13. Double click on ZS Trash Destination to configure it.
    

    SSIS Trash Destination – Configure

14. Click OK to save configure setting UI.
15. Right-click on the path and Add Data Viewer.
16. Execute the package and verify source data in the data viewer.
    

    Execute Package and Verify Data Source

Method 2 – Using OLE DB Source

Let’s use ZS OLE DB Source to write data into ZS Trash Destination.

1. Open Visual Studio and Create New SSIS Package Project.
2. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

3. We are going through SQL Server Database, need OLE DB Connection for Data Source.

How to create OLE DB Connection.

1. Let’s, Right-click on Connection Managers Panel to Create OLEDB Connection, so you can use Source and Context Menu will appear, Select New OLEDB Connection from the Context Menu.
   

   Create OLE DB Connection

2. Now, click on New button for create Connection.
   

   Create New OLE DB Connection

3. In the Connection Manager, just follow the steps in the following image we have created.
   

   OLE DB Connection Configure

4. Click on OK button to save connection configure setting UI.

How to write into Trash Destination from OLE DB Source

1. Drag and Drop OLE DB Source from SSIS toolbox.
   

   OLE DB Source – Drag and Drop

2. Double click on OLE DB Source for configure it.
3. In the Connection Manager, just follow the steps in the following image we have created.
   

   OLE DB Source – Configure

4. Click on OK button to save OLE DB Source Editor setting UI.
5. Now, Just Drag and Drop Our Free ZS Trash Destination on the data flow designer surface from SSIS Toolbox.
   

   SSIS ZS Trash Destination – Drag and Drop

6. Now, single click on OLE DB Source once you see the blue arrow from the source … connect it to ZS Trash Destination.
7. Double click on ZS Trash Destination to configure it.
   

   SSIS ZS Trash Destination – Configure

8. Click OK to save configure setting UI.
9. Right-click on the path and Add data viewer.
10. Execute the package and verify source data in the data viewer.
    

    Execute Package and Verify Data

Conclusion

In this blog, we learned how to write data into a raw file and read data in SSIS. We used the Trash Destination for write data and Dummy Data Source to read data. You can download SSIS PowerPack from here to try many other scenarios not discussed in this blog. SSIS PowerPack is a collection of 70+ high performance, just drag and drop connector/task for SSIS(i.e. Microsoft SQL Server Integration Service). After you install SSIS PowerPack you will see many new Custom Tasks and Components in your Visual Studio SSIS Toolbox.

References

Finally, you can use the following links for more information:

• ZS Trash Destination
• ZS Dummy Data Source
• Help File:
  • ZS Trash Destination
  • ZS Dummy Data Source

The post Using SSIS Trash Destination and Dummy Data Source appeared first on ZappySys Blog.