Introduction
In this article, we will see how to read PayPal API data in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call PayPal APIs can be useful for any developer regardless of which programming language or tool set you use. We will also see How to send basic Authorization header along with body in OAuth2.
We will go through the steps to read data from PayPal Transactions and Load into MS SQL Server.
In nutshell, this post will focus on how to make PayPal API Call using SSIS.
Prerequisites
Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:
- SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
- Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
- Make sure ZappySys SSIS PowerPack is installed (download it).
- Credentials of PayPal API. Click here to create API.
What is PayPal?
PayPal is an American company operating a worldwide online payments system that supports online money transfers and serves as an electronic alternative to traditional paper methods like checks and money orders. The company operates as a payment processor for online vendors, auction sites, and many other commercial users, for which it charges a fee in exchange for benefits such as one-click transactions and password memory. PayPal’s payment system, also called PayPal, is considered a type of payment rail.
Get Started with PayPal API
If you have need to automate PayPal operations (e.g. read / write / update / delete ) then you can use PayPal REST API. Before you can integrate a PayPal product or solution, you must set up your development environment to get OAuth 2.0 client ID and secret credentials for the sandbox and live environments. You exchange these credentials for an access token that authorizes your REST API calls. To test your web and mobile apps, you create sandbox accounts.
Get PayPal credentials
To generate OAuth 2.0 credentials for the sandbox and live environments:
- Log into Dashboard and type your PayPal personal or business account email and password.
- In the REST API apps section, click Create App. The purpose of this app is to generate your credentials.
- Type a name for your app and click Create App. The page shows your sandbox app information, which includes your credentials.
Note: To show your live app information, toggle to Live.
- Copy and save the client ID and secret for your sandbox app.
PayPal Client ID and Secret
- Review your app details and save your app.
Read PayPal API Data using SSIS JSON Source
Let’s start with an example. We use SSIS JSON Source component to make the call to PayPal API, we will read PayPal data and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.
In REST API calls, include the URL to the API service for the environment:
- Sandbox:
https://api.sandbox.paypal.com - Live:
https://api.paypal.com
Create Basic OAuth Client Credentials Grant (Configure ZS-OAuth Connection)
To get API response data using SOAP / REST API call, you need to get an API Token from your token request URL. Follow the steps mentioned below to get API Token:
- Now, Right Click on Connection Managers Window, and one Context menu appears for creating a new connection manager.
Connection Manager Window: Create New Connection
- Select ZS-OAuth Connection Manager from the list of Connection Managers and, double-click on it or click on Add button to add a new Connection Manager. By Default, The created connection manager is of Package-Level. You can set Project Level Connection manager too If you want.
SSIS Connection Mangers List: Select O-AUTH Connection Manager
- In the SSIS ZS OAuth Connection Manager, Select Provider as Custom, select version as OAuth2 and select Grant Type as Client Credentials Grant. Enter the credentials and access token url and test the connection.
1https://api.sandbox.paypal.com/v1/oauth2/token
ZS OAuth Connection – Client Credentials Grant
- Click on OK button to save OAuth Connection configure settings.
That’s it we have successfully configured Connection for PayPal API in SSIS. In the next section, we will see how to use this connection and read various data from PayPal API.
Read PayPal API Data in SSIS
Once we have done creating HTTP Connection Manager we can move forward to read PayPal API data inside Data Flow. So lets Configure SSIS JSON / REST API Source.
- First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
Drag and Drop : SSIS Data Flow Task from SSIS Toolbox
- From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.
Drag and Drop JSON Source Component
- Double click JSON Source and enter the following URL as below to get invoices and select that created OAuth Connection.
1https://api.sandbox.paypal.com/v1/invoicing/invoices?page=0&page_size=100&total_count_required=true
SSIS JSON Source
- Now Go to Pagination tab select mode as Url Parameter and add indicator as
page and Add the increment value same as
page_size value we set in url, there for in our case we set 100. For pagination call please click here for reference to this link.
PayPal Pagination
- Select the desire Array Filter and click on preview.
PayPal Preview
- That’s it, you are ready to load PayPal API Data into SQL Server.
Load PayPal API data into SQL Server
ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.
Step 1: Add Upsert Destination to Data Flow
- Drag and drop the Upsert Destination component from the SSIS Toolbox.
- Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.
SSIS - Data Flow - Drang and Drop Upsert Destination Component
Step 2: Configure Target Connection
- Double-click the Upsert Destination component to open the configuration window.
- Under Connection, select an existing target connection or click NEW to create a new connection.
- Example: SQL Server, or PostgreSQL, or Amazon Redshift.
Step 3: Select or Create Target Table
- In the Target Table dropdown, select the table where you want to load data.
- Optionally, click NEW to create a new table based on the source columns.
Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS
Step 4: Map Columns
- Go to the Mappings tab.
- Click Auto Map to map source columns to target columns by name.
- Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
- You can manually adjust the mappings if necessary.
SSIS Upsert Destination - Columns Mappings
Step 5: Save Settings
- Click OK to save the Upsert Destination configuration.
Step 6: Optional: Add Logging or Analysis
- You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.
Step 7: Execute the Package
- Run your SSIS package and verify that the data is correctly inserted and updated in the target table.
SSIS Upsert Destination Execution
Conclusion
After all, we saw you how to extract information from REST API such as PayPal REST API using HTTP Connection and load into SQL Server. We also learned techniques like How to get Dynamic Token using HTTP Connection. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).
References
Finally, you can use the following links for more information:
