wordpress Archives | ZappySys Blog

Read WordPress data using SSIS REST API Call

ZappySys — Sat, 11 May 2019 06:22:31 +0000

Introduction

In this post you will learn how to Read WordPress data using SSIS REST API Call (Drag and drop approach without any coding). We will use SSIS JSON/ REST API Source to extract data from WordPress API. In this blog we will see how to get all WordPress Posts, Pages and Categories using JSON Source and load it into the SQL Server.

Now let’s look at step by step approach to call WordPress REST API.

Prerequisite

First, you will need to have SSIS installed
Secondly, make sure to have SSDT
Thirdly, You have obtained WordPress account credential with read and write permission.
Finally, do not forget to install ZappySys SSIS PowerPack

What is WordPress

WordPress is a free and open-source content management system based on PHP & MySQL. Features include a plugin architecture and a template system. It is most associated with blogging but supports other types of web content including more traditional mailing lists and forums, media galleries, and online stores.

Read data from WordPress using SSIS JSON Source

Let’s start with an example. We use SSIS JSON Source Task to make call to WordPress API, we will load WordPress Pages data to SQL Server.

Get all Posts, Pages and Categories using WordPress API

First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
Furthermore, Drag and Drop JSON Source(REST API and File) File Task from the SSIS Toolbox.
Drag and Drop JSON Source Component
Double click on that Component to Edit the Configurations. Select File path or web URL from the Access Mode drop-down.
Enter Web URL like this to Get all Pages. Please refer to this link of WordPress API for more information.
```
https://{{your-wordpress-domain}}/wp-json/wp/v2/pages?page=1&per_page=100
```
To Get all Posts
```
https://{{your-wordpress-domain}}/wp-json/wp/v2/posts?page=1&per_page=100
```
To Get all Categories
```
https://{{your-wordpress-domain}}/wp-json/wp/v2/categories?page=1&per_page=100
```
Here, your-wordpress-domain is your WordPress site domain name (here we are using our zappysys.com wordpress site).

* page is the index of the first item returned in the page of results.
* per_page the maximum number of items that can be returned per page. Each API endpoint may have a different limit for the number of items returned, and these limits may change without notice. (For more See this WordPress API Documentation for Pagination)

Set URL in JSON Source
Check Use Credentials Option and select a new ZS-HTTP Connection from the drop-down.
Configure new HTTP Connection for WordPress as below.
Basically, you need to enter 3 pieces on the General tab (i.e. WordPress Site URL, UserName and Password)

WordPress Connection: HTTP Connection Manager
Likewise, we are going to set Pagination related configurations for WordPress API request. For example, if you want to read 100 records in each response and keep doing it until last page found then set Page=1, Per_page=100 and Increment by 1.
JSON Source – WordPress request Pagination Configurations Settings
Select the desire Filter and click on Preview button to see the response data.
JSON Source: Preview WordPress Data
Finally, click OK to save JSON Source settings.
That’s it we are ready to load WordPress data to SQL Server.

Load WordPress data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

Drag and drop the Upsert Destination component from the SSIS Toolbox.
Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

Double-click the Upsert Destination component to open the configuration window.
Under Connection, select an existing target connection or click NEW to create a new connection.
- Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

In the Target Table dropdown, select the table where you want to load data.
Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

Go to the Mappings tab.
Click Auto Map to map source columns to target columns by name.
Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

In this post you have seen how easy it is to Read WordPress data using SSIS REST API Call in few clicks. We used JSON Source Component to call WordPress REST API and load data into SQL server. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information about the use of WordPress REST API with our tools:

JSON Source, you can also find Tutorial Video here.
Help File: Documentation of SSIS Json Source
WordPress API
WP API Authentication
WP API Pagination

The post Read WordPress data using SSIS REST API Call appeared first on ZappySys Blog.

Write/Read WordPress data using SSIS and REST API

ZappySys Team — Fri, 26 Apr 2019 23:03:36 +0000

Introduction

In this new article, we will show how to read WordPress data using REST API. REST API is a very popular software architectural style for Web Services.
SSIS is a very popular ETL software used to integrate almost any data (SQL Server, Oracle, Excel, DB2, CSV files, Big Data, Facebook, Gmail, etc).

Requirements

For this example, we will use a popular plug-in. The name is ZappySys PowerPack for SSIS which includes some nice generic connectors to REST API. You can download it here:

Getting started

We will be using this ZappySys SSIS PowerPack component to make things work:

	SSIS REST API Web Service Task
	SSIS JSON Source (File, REST API, OData Connector)
	SSIS JSON File Destination (Create JSON File)

Read WordPress data using SSIS with REST API

REST API is also the name of one of the ZappySys components. These tools can connect to thousands of REST API Servers. In this example, we will connect to WordPress.

First, drag and drop the REST API task to the design pane.
Secondly, write the following URL in the REST API task URL:
```
http://demo.wp-api.org/wp-json/wp/v2/pages
```
After that, press Test Request/Response.
That URL shows the page’s information in WordPress. This is a demo API. For more information about the WordPress links, check here.
Call WordPress in SSIS
Next, you can also check the Tags, Comments, Categories, Posts, Taxonomies, Media. Here you have some examples:
- To get the tags in WordPress, use this URL:
```
http://demo.wp-api.org/wp-json/wp/v2/tags
```
- In order to read comments in WordPress using SSIS and REST API, use this URL:
```
http://demo.wp-api.org/wp-json/wp/v2/comments
```
- For the categories, use this URL to read WordPress data using SSIS:
```
http://demo.wp-api.org/wp-json/wp/v2/categories
```
In addition, for your blog, you need to add /wp-json/wp/v2/ and pages, posts or tags in your WordPress URL.
Finally, if you want to save the results, in Response Settings, enable save response content and select the option save to file, and specify a file path:
Save response WordPress API

How to read WordPress data using SSIS and store it into a SQL Server database

In the next example, we will use the JSON Source component. This component is similar to the REST API, but this is used in the Data Flow to save the REST API results in any destination of your preference (SQL Server, PostgreSQL, MongoDB, etc). Also, this example will read WordPress data using SSIS and store it into a SQL Server database.

First of all, in the Control flow, drag and drop the Data Flow:
Data Flow Task
Drag and drop the JSON Source in the Data flow with this configuration:
JSON Source to WordPress API
Finally, you can save to an OLEDB Destination to save into a SQL Database:
Store WordPress into SQL Server

How to read WordPress data using SSIS and modify the JSON structure

Sometimes we need to modify the WordPress JSON output to a custom output in SSIS. We will now use the JSON Generator Transform this time, which is a very powerful tool for this purpose.

First of all, we will use the JSON Source connecting to the following
```
URL: http://demo.wp-api.org/wp-json/wp/v2/categories
```

Secondly, we need to convert the JSON output to the following format:

{
   "id" : 
     {
         "count" : 1474359596826126768,
         "description" : 
          {
            "link" : "xxx"
          }
    }
}

Next, we will add an element first:
SSIS WordPress API
Also, we will select the id attribute:
id attributes SSIS
In addition, we will add a static element to the id:
Add JSON element
Furthermore, we will add the count element.
Add element in JSON with SSIS
Finally, repeat the same process for the description and link. The JSON structure will be the following:

How to read WordPress data using SSIS and use pagination

If it is a lot of data in REST API, pagination allows showing the results in pages. Let’s take a look at it.

First of all, we will use page 2. To do it, we will use the following URL in the JSON Source task in the data flow:
```
http://demo.wp-api.org/wp-json/wp/v2/posts?per_page=2
```
SSIS URL pagination REST API
Secondly, there is a pagination tab that can be used to get the data using pagination. For more information about pagination, refer to our ZappySys Pagination article related.
SSIS pagination in SSIS

How to write WordPress data using SSIS

In order to write WordPress data using SSIS, we will use the REST API task with the following URL:
```
POST http://demo.wp-api.org/wp-json/wp/v2/posts/
```
Secondly, we have the post with id equal to 2 and we will use the body request as follows:

Conclusion

To conclude, in this article, we learned how to write and read WordPress data using SSIS. In addition, we learned also how to get pagination and how to change the format of the JSON results. If you want to try it yourself, do not hesitate to download ZappySys PowerPack!

References

Finally, for more information about how to read WordPress Data using SSIS, refer to these links:

The post Write/Read WordPress data using SSIS and REST API appeared first on ZappySys Blog.