The article shows how to download images from a web page using SSIS. Sometimes we need to download the images from a web page. In this post, we will show how to do this.

Prerequisites

Before we perform steps listed in this article, you will need to make sure following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Optional (If you want to Deploy and Schedule ) – Deploy and Schedule SSIS Packages

Steps-by-step process to download images from HTML using SSIS

Use REST API task to get the HTML body

1. Drag and drop the REST API Task from SSIS toolbox, select the html page you want and save it in a variable.

Select the page you want to get the images

2. Go to Response Settings. Check the option Save the response content. In Save Mode, select Save to File. In the option Enter File Path, write the path for the html file.

Save the page in a file

Parse the image with Regular Expression parser task

3. From the SSIS toolbox drag and drop Regular Expression Parser Task on the Control flow designer surface.

4. The next step is to save the source path from the images. You need to use Regex and here are two example you can use, in this page Regex101 you can check more details about the expressions we are using:

Expression 1:

<img.*?src="(.*?)"{{*}}

src="([a-z\-_0-9\/\:\.]*\.(png|jpg|jpeg|gif|png))"{{*}}

Regex expression to get the image code from the page

Read the image source with CSV source in order to download images from a Web page

5. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.

6. Double click on the DataFlow task to see DataFlow designer surface.

7. From the SSIS toolbox drag and drop CSV source and insert the variable you are using from the previous task

Use CSV source to read the variable

Get the image name, image full path and the destination folder to download images from a Web Page

8. From the SSIS toolbox drag and drop Derived Column transform to remove the HTML code and get the image name.

Expression for the Image name:

(DT_WSTR,1000)REVERSE(LEFT(REVERSE(REPLACE(REPLACE(Column1,"src=\"",""),"\"","")),(FINDSTRING(REVERSE(REPLACE(REPLACE(Column1,"src=\"",""),"\"","")),"/",1) - 1)))

Remove the HTML code:

REPLACE(REPLACE(Column1,"src=\"",""),"\"","")

Add a new column and remove HTML code

9. Now we will drag and drop another Derived Column, this one is for the file path for the images and validate if the image URL is valid. If the image URL has “http” in the stringm then it is valid otherwise you need to add the rest of the URL. Here is our example for the filePath and validate the URL image. If everything is OK, we can send the image URL.

File path: (DT_WSTR,2500)@[User::vDestinationFolder] + ImageName

Validate image URL: ((FINDSTRING(Column1,"http",1) > 0) ? Column1 : (DT_WSTR,2500)("https://www.urlexample/image/" + Column1))

Note: @[User::vDestinationFolder] is a variable we used for the local path

Add a new column and verify the HTML path

Send the request and save the image In the folder TO download images from a Web Page

10. Now we will drag and drop a web API destination and select the column in the input column to URL. If the request is right, we can save the image in the local file.

Send a request for all images you get

11. Now drag and drop an Export Column transformation and select the ResponseText from the request and the file path.

Select the image response and the local file path

12. Finally, we use a trash destination to close the flow.

The final result

Conclusion

If everything is OK, you will be able to download the images from your HTML page. To do that, we read the list of URLs from a list. Then we get the name of the images using expressions. Then we add a path for each image to store them. Finally, we save the images.

The post How to download images from a web page using SSIS appeared first on ZappySys Blog.

In our previous post, we saw how to read Salesforce data using SSIS Salesforce Source (Which uses SOAP API under the hood). This may be slow if you have millions of rows because Salesforce Source paginates every 2000 rows, so it’s not a good option to read millions of rows to iterate on a large Salesforce dataset (Table or Query). To increase the read speed, you can use BULK API. This article explains how to read salesforce data using BULK API and increase read speed.

NEW Version (v5.5+)

In the new version (v5.5), we have added an option to query in Bulk Mode. This means if you have millions of Salesforce object rows to read, no worries; it may be very fast in the new version. Previously, it used to take a long time because it read in batches of 2000 rows in each request. With the Bulk API, the batch size is really large.

Here is how to set Salesforce Bulk API Mode in SELECT query (Read operation).

1. Install new version of SSIS PowerPack. Make sure it is v5.5 or higher.
2. Open existing Salesforce Source UI
3. Make sure you check [Use Bulk API] option like below. if you do not see this option then most likely you are using older version.
   -OR-
   Just add #bulkmode prefix in the SQL query like below, it will use Bulk API
   

   Bulk API mode for SSIS Salesforce Source (Read Large amount of data using SOQL Query)

OLD Version

If you are using an older version of SSIS Power Pack (v5.4.0 or older), you can follow the steps mentioned in the following section.

We will see how to query using Salesforce Bulk API in SSIS. It involves three simple steps like the one below.

1. Call BULK API – Submit SOQL Query and receive a JobId back (Use SSIS Salesforce API Task)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query

2. Keep checking Job Status Until Status is JobComplete (Use SSIS Salesforce API Task)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}

3. Read Query Result  (Use SSIS CSV Source)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results

Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode / large dataset)

Now, let’s look at everything in detail.

Obtain Salesforce Credentials

The first step to call Salesforce API is to obtain credentials (security token or OAuth ClientId /Secret)

Method#1 – Check this article to use UserId/ Password / SecretToken
Method#2 – Check this article to use the OAuth Connection Option

Create Salesforce Connection

Once we have Salesforce Credentials, we can create a new Salesforce Connection (You can do this from Task UI or the below way)

1. Open Visual Studio ( Assuming you already have SSIS installed along with ZappySys PowerPack )
2. Create a New SSIS Package Project.
3. Right-click on the Connection Manager panel and click on New Connection.
   
4. Select ZS-SALESFORCE Connection Manager from the Connection Managers list and Click on the Add Button.
   
5. Now in Connection Manager UI, Enter your SalesForce Credentials (OAuth Client Id/. Secret –or— User Name, Password and Security Token), and leave all other properties as it is.
   

Call Create Job – Retrieve / Save JobId

Once the connection is created, we can create our first step. Basically, we will call this API to create Job (Bulk Read)

1. Here, in Visual Studio, drag and drop the ZS Salesforce API Task on the design surface.
2. Double click the Task and configure Request Settings Tab like below
   – Select Salesforce Connection (Created in the previous section)
   – Check Use direct URL
   – Enter URL like below (Replace {your-instance} with your own instance (i.e. na9) . For more information on how to find out my
   salesforce instance id check this article
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query

   – Request Method POST
   – Enter Body like below (You can use your own SOQL query). Keep in mind SQOL must be encoded for JSON format. This means the entire SQL must be in one line (replace new lines with \r\n , quotes with \” and tabs with \t)
   For example, to execute SELECT Id, Name FROM Accounts, you can write something like the one below in BODY.
   

   {
     "operation": "query",
     "query": "SELECT Id, Name\r\nFROM Account"
   }

   The above is for below (with a new line)
   

   SELECT Id, Name
   FROM Account

   

   Create Job – Call Bulk API to Read / Query using SOQL

3. Now Click Test Request (Copy the JobId which we will use in the Variable for Designtime Testing)
4. On the Response Tab, configure it like the below.
   – Select Response content Type = Json
   – Enter $.id as the expression
   – Check Save response option
   – Select Save to Variable and click New to create a variable. Name it JobId and enter a sample JobId value (copied from the previous step)
   

   Extract Salesforce JobId save to SSIS Variable

Do Status Check – Look for JobComplete

Once we create a job and store JobId in the variable we have to keep checking Job status every X seconds. Don’t worry you don’t have to use Loop or timer control for that. ZappySys made it so simple to check such thing without Loop /Timer.

1. Drag and drop the ZS Salesforce API Task in the design surface.
2. Double click and configure like below. URL will be different this time. Use variable as below and replace your instance ID
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}

   

   Check Salesforce Job Status Periodically (Without Loop + Timer Controls)

3. Now go to the Status Check Tab and configure it, as shown below. It will keep looking for the words below in the response to decide whether it is a Success or Failure. If those words are not found, they will be checked every 5 seconds (Maximum 1200 secs).
   – Check Enable Status Check option
   – For Success Value – Enter JobComplete
   -Check Fail task option
   – Enter Cancel|Failed|Abort in the text box
   

   Configure Job Status Check Loop using Salesforce API Task

Read data (Compressed CSV Format)

Last step is read the result from completed Salesforce Job. Its in CSV format so we will use Advanced SSIS CSV Source which supports API connection to Salesforce. For that, we will use the same Salesforce API Task used before.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
2. Double click on the DataFlow task to see DataFlow designer surface.
3. Here, In Visual Studio, drag and drop the ZS CSV Source (Web API or File) in the design panel
   
4. Double click on ZS CSV Source (Web API or File) Configure it like below.
   – Enter URL to read result like this (You can supply OPTIONAL maxRecords parameters to control how much data comes in a single response)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results
   --OR--
   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results?maxRecords=50000

   – Check Use Credentials and select Salesforce Connection
   – Click Preview (assuming JobId variable has a valid completed JobId)
   

   Read Salesforce Bulk API Job Result – Using SSIS CSV / API Source

5. Now, set up Pagination as below to read all records using an automated loop. You will need VERSION Released after Jun 20, 2022 to use Regular Expression feature in Header Value extractionBy default, Salesforce doesn’t send all records in response so we need to setup pagination. In below example we are reading Sforce-Locator attribute from Response and sending it to the next URL as locator=<continue-token> value
   – Choose Response Header Contains Continue Token 
   – Enter below expression in Page Num Indicator
   

   locator=Sforce-Locator(^((?!null\b).)*$)

   Basically, the above expression will keep looping until we find Sforce-Locator : null in the response header.
   

   Pagination for Salesforce Bulk Query API (Get Header Value from Sforce-Locator)

6. Click OK to save and now run entire package like below.

Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode / large dataset)

Conclusion

In this article we saw how to use various ZappySys Components / Task to achieve complex Salesforce API calls to read Query data using Bulk API. Download SSIS PowerPack and try yourself and also explore 70+ other tasks /components not discussed in this article.

The post How to query salesforce data using Bulk API in SSIS appeared first on ZappySys Blog.

If you have need to integrate REST API in SSIS (i.e. fetch JSON, XML, CSV via HTTP call) and wondering how to Read API data in SSIS and Load into SQL DB (i.e. SQL Server, MySQL, Oracle) then you are at the right place.

In this article we will cover simple step by step approach how to achieve API integration in few clicks. For detailed introduction please refer to our previous article.

In next few sections we will cover the following SSIS Components to read data from API like a Table (Parse in Rows / Column). Our instructions will be mostly for JSON Format but concepts are same for other formats (i.e. XML / CSV).

	JSON Source Connector (Read from REST API, JSON File or OData Service): Use this dataflow component when you have to fetch data from REST API webservice like a table. This component allows you to extract JSON data from webservice and de-normalize nested structure so you can save to Relational database such as SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local JSON files or direct JSON string (Wildcard pattern supported too e.g. c:\data\file*.json).
	XML Source Connector (SOAP, File, REST) : Use this dataflow component when you have to fetch data from XML or SOAP webservice and consume data like a table. This component allows you to extract data from webservice and save to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local XML files or direct XML string.
	CSV Source Connector (File, REST) : Use this dataflow component when you have to fetch data from File or REST API which returns data in CSV format . This component allows you to extract data from webservice and save to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local CSV files or direct CSV string.

There are few more API components not discussed in this articles. They are REST API Task (no Parser – Just raw API calls) and Web API Destination (Write / Send Data To API / Delete rows). They both have different use case to check them see it fits in your need.

Video Tutorial

Step-By-Step – Read API data in SSIS using JSON / XML / CSV Source

In this section you will learn how to use JSON Source Adapter to Read data from API URL.

Load API data into SQL Server Table / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

API Authentication

In our step by step example we did not cover Authentication to keep things simple but in real world you have to pass some sort of authentication details to make API calls (e.g. UserID/Password or Secure Token). There are many ways to authenticate with API Services. ZappySys offer mainly 2 Connection Managers to cover most Authentication scenarios.

1. HTTP Connection Manager
2. OAuth Connection Manager

Using HTTP Connection Manager

If you have use case of using UserID / Password or any other method supported by HTTP Connection Manager then use below option (Below example is for Basic Authentication)

SSIS JSON Source – Passing Basic Credentials (Supply UserID / Password) using HTTP Connection

Using OAuth Connection Manager

If your API support OAuth Standard for Authorization then use OAuth Connection Type as below.

Generate Token for Google API using OAuth 2.0

API Pagination

Pagination is another important concept to understand when you call API using ZappySys Connector. If you ready many records from your API then by default API might not return all rows so you have to paginate until all records are read (Like a Loop in Programming). ZappySys makes it easy to paginate. Check this article to configure many different Pagination Modes

Things have gone bad: Error handling & debugging

Incidentally, bad things can happen. A remote server may go offline or your server may go out of memory. In any case, you may want to know when that happens and take actions accordingly. For that purpose, you have to redirect bad rows to some other destination. For this example, we will take and use Web API Destination, but basically, you can use any SSIS component:

Handling errors

1. Add a Derived Column above Web API Destination with expression "(DT_WSTR,4000)ZS_JSON_OUT" and name it "JsonAsString". This will let you see what JSON you are actually passing.
2. Then add a database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don't forget to set Redirect row option for both, Error and Truncation columns:
   

   Redirected failed requests from Web API Destination to a desired destination when loading from SQL Server to REST API Service is failing. Derived Column JsonAsString added to be able to read JSON which was passed to Elasticsearch

3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled.

NOTE: You can read more about redirecting rows in SSIS Error Handling (Redirect bad rows) article.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Debugging Web API call using Fiddler in SSIS

Conclusion

REST API is becoming more and more popular each day. With that Data integration from RESTful API services going to be challenge. Luckily ZappySys SSIS PowerPack provides a great way to integrate any API in SSIS via simple drag and drop approach without coding. Try SSIS PowerPack for FREE  see how much time / money you can save and to integrate virtually any REST API.

The post How to Read API data in SSIS and Load into SQL Table appeared first on ZappySys Blog.

In our previous blog we saw How to write data into CSV file in SSIS (GZip / Split). Now in this blog, we will see How to Pivot CSV Data in SSIS using CSV Source. It also supports Pivot mode so you can convert single CSV string value into Rows. In this article we also see how make some CSV columns as parent and not consider it in pivot and load it as it is using JSON Generator and JSON Parser Transform Pivot Option.

In nutshell, this post will focus on how to Pivot CSV Data into multiple columns and rows.

How to Pivot CSV Data in SSIS using CSV File Source

Let´s start with an example. In this SSIS CSV Parser Transform example, we will Parse database column string into multiple columns and rows.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. Furthermore, drag and drop the CSV Source and Double click on it to configure it.
   

   Drag and Drop : CSV Source from SSIS Toolbox

3. Enter the CSV file path and select desire column delimiter most common in CSV is comma and check the First Row as headers and click on preview.
   

   CSV Source Preview

4. Now, go to Pivot Columns to Rows Tab and checked Enable Pivoting and click on Preview Data to view the data.
   

   CSV Source Pivoted Data Preview

5. That’s it we see how to read data from CSV File and also Pivot the CSV Data using CSV Source.
6. Now In the next section we will see how to how make some CSV columns as parent and not consider it in pivot and load it as it is.

How make some CSV columns as parent.

Sometimes you just need to make child columns in to rows and want to leave parent level columns as it is.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. Secondly, drag and drop the CSV Source and Double click on it to configure it.
   

   Drag and Drop : CSV Source from SSIS Toolbox

3. Furthermore, enter the file path and click on preview.
   

   CSV Source Preview

4. For example if we want to pivot Email and Phone only in pivoting and wants to exclude parent level columns FirstName and LastName as it is.
5. For that drag and drop JSON Generator Transform and Select Output Mode as Output One JSON Per Parent Input Row  and Add Unbound Nested Element and named it as Persons and click on OK.
   

   JSON Generator Add Unbound Nested Element

6. Now add that two parent level fields FirstName and LastName in the Persons.
   

   JSON Generator Add Multiple Elements

7. Now select Person Element and right click and Add Unbound Nested Element and named it as Contacts and click on OK.
   

   JSON Generator Add Unbound Nested Element

8. Now add that two child level fields Email and Phone in the Contacts.
   

   JSON Generator Add Multiple Elements

9. Therefore JSON Generator Transform looks like below screen. Now Copy that Sample Preview JSON String to use it in the JSON Parser Transom as sample string.
   

   JSON Generator Transform – Sample Data

10. Similarly, drag and drop JSON Parser Transform and select the Input JSON Column and Paste that copied Sample JSON String and select the Contacts filter to read the Child data and click on Include Parent.
    

    JSON Parser Transform – Select Filter

11. Now go to Pivot Columns to Rows Tab and checked the Enable Pivoting and click on the Preview.
    

    JSON Parser Transform – Enable Pivoting

12. Finally, we are ready to load this data.
    

    CSV Pivoted Data

13. Let’s load this data in the SQL Server.

Load Pivoted CSV File data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

After all, we saw you how to read CSV file data, pivot the data and Similarly, using SSIS JSON Generator and Parser Transform Pivot the CSV Data excluding some columns from pivoting.  To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following URL for more information.

• Help File: CSV Source

The post How to Pivot CSV Data in SSIS appeared first on ZappySys Blog.

In this post we will lean SSIS Magento data read / write operations. Magento is a very popular eCommerce platform and they offer JSON based REST API and XML based SOAP API. You can use either API based on your need to automate common integration needs.

We recommend using REST API (JSON API) if possible because they are simpler and faster. We will focus only on JSON API in this article but we do have blog post on how to call SOAP API.

About Calling Magento REST API

If you are new to Magento API we highly recommend to get familiar with Magento REST API here. Magento REST API  offers two ways to authenticate.

1. Token based Authentication
2. OAuth Based Authentication (OAuth 1.0a)

In this article we will look at only Token based approach. In token based approach we have to get new token using admin / customer account information.

We will cover later how to access admin or customer level API later in this article (See HTTP connection configuration).

Configure Connection / Call Magento REST API in SSIS

Now lets look at very simple example how to use Token based method to call Magento REST API in SSIS. We will first configure HTTP connection and then call REST API.

Let’s get started.

1. First, open SSIS Package after installing SSIS PowerPack
2. Drag ZS REST API Task from SSIS Toolbox on Control Flow Designer. Let’s rename task to Get Magento Categories
3. Enter REST API URL you like to call. For example we can use below URL. Replace **your-magento-host** with your own name.
   

   http://**your-magento-host**/index.php/rest/V1/categories

4. Select Url from Connection mode
5. From Connection Dropdown select new ZS-HTTP connection
   

   Create new connection for SSIS REST API Task

6. Configure General Tab of HTTP Connection like below
   URL can be anything for now because we will use override option in next step (On REST API Task).
7. Select Credentials Type as Dynamic Token.
8. Enter Userid / password
   

   Configure Magento REST API Connect for Token based Authentication

9. Now goto Dynamic Token Tab (Or click Configure Link) and configure like below.
10. Enter Token URL for API Login URL field.
    NOTE: If you are a customer and want to access your own account (e.g. view your orders) then replace /admin with /customer  in URL)
    Replace **my-host** with your host name.
    

    http://***my-host***/index.php/rest/V1/integration/admin/token

11. Change Method to POST.
12. Enter Body as below (Placeholders are automatically replaced when /token endpoint is called.
    

    {"username":"[$userid$]", "password":"[$password$]"}

13. Select Content Type as Application/Json
    

    Configure Magento Token Request

14. Now click Dynamic Token – Response Tab and configure like below.
    Select Regex and enter Expression as below. This will remove double quotes around token in response.
    

    "(.*)"{{0,1}}

    

    Extract Token from response – Remove double quotes around value using Regular Expression

15. Click OK to save connection
16. Now on REST API Task check Use Direct URL Option
17. Click Test Request to confirm its working
    

    Call Magento REST API using SSIS REST API Task

Read data from Magento and Load into SQL Server

Now lets look at how to read Magento data and load into SQL Server. Assume that you like to read all products.

Configure JSON Source (Read Magento REST API)

1. Drag Data flow and double click to edit:
   

   Dragging and dropping Data Flow Task into Control Flow

2. Drag ZS JSON Source on data flow and double click to edit
3. Configure JSON Source like below if you like to get product list for example
   1. Enter URL as below (change http to https if your host support secure channel).
      

      http://***yourhost***/index.php/rest/V1/products?searchCriteria[pageSize]=1000
      --- to extract only specific fields do below way --
      http://***yourhost***/index.php/rest/V1/products?searchCriteria[pageSize]=1000&fields=items[sku,name]

   2. Check Use Credentials. Use same HTTP connection we created in previous section
   3. In the Filter enter $.items[*] or click Select Filter to Browse and select items node.
   4. Click Preview to confirm
4. Here is how your setup will look like after configuration.
   

   SSIS JSON Source Configuration – Read data from Magento REST API

5. If you expecting more than 1000 rows then you have to set up pagination. Now go to pagination tab and configure as below (Below options are only available in v2.9.1 or higher). If you have older version then check next section for workaround.
   1. Pagination Mode =URL Parameter Mode
   2. Page Num Indicator = searchCriteria[currentPage]
   3. Max Rows Expression = $.total_count
   4. Page Data Expression = $.items[*]
6. Here is how your Magento REST API Pagination Setup will look like.
   

   REST API Pagination using Max Row count Mode

7. Click OK to save JSON Source. In next section we will see how to load data into Target like SQL Server.

Configure OLEDB Destination (Load into SQL Server)

1. Drag OLEDB Destination.
2. Select / Create Connection
3. Select or create NEW target Table (Click NEW button)
4. Click Mapping Tab to map source columns to target table
5. Click OK to save
6. Execute Package

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Configure Magento Pagination (For Old Version)

In previous section we configured pagination using newer version (v2.9.1 or higher). But what if you have older version and you dont see that feature. Use below steps for workaround.

Now let’s look at another common scenario when you real lots of data. Let’s say you have 5000 products but API call we saw earlier returns max 1000 rows. In that case we have to keep reading next page until all rows are returned. ZappySys provides many settings to configure various REST APIs out there, unfortunately there is no standard around pagination methods. Since Magento API has some bug in API pagination we have to do following hack. Magento keep sending same data of last page if you try to access page which doesn’t exists. Here is how to overcome that bug.

Configure REST API task to get total row count

First step to configure magento pagination is get total rows returned in our request which you like to paginate (in our case /products).

1. Go to Control Flow designer
2. Drag ZS REST API Task from SSIS Toolbox
3. Let’s rename task to Get Magento Product Count
4. Double click it and configure like below (Change HTTP to HTTPS if needed)
   

   http://***yourhost***/index.php/rest/V1/products?searchCriteria[pageSize]=10

   
5. Go to Response Tab and configure like below. Enter expression as $.total_count
6. Click OK to save

Define Expression for JSON Source – MaxRows Property

Now next step is to define expression on MaxRows property of JSON Rows so we stop once all rows consumed from Paginated response. If you don’t do this it will keep paginating forever 🙁

1. On Control Flow designer select Data Flow which contains JSON Source for Magento
2. Right click and go to Properties of that data flow
3. Find Expression and click Button to add new expression
4. On Expression dialogbox select [Your JSON Source].[MaxRows] and for expression enter variable name which holds total count (e.g. @[User::MaxRows] )
5. Click OK to Save

Configure JSON Source for Magento Pagination

Now last thing we have to do is go to JSON Source and configure few things for pagination.

1. Go to data flow and double click JSON Source
2. Click on Pagination tab and select Pagination by URL Parameter Mode
3. Enter Page name as below
   

   searchCriteria[currentPage]

4. Click OK to save UI
5. Now run entire page to test (NOTE: Do not execute just data flow because if you do that way, it will not extract total row count)
   

Write data to Magento using SSIS Web API destination

Now let’s look at example to load data into Magento. In below example we will create few sample products.

1. Create new data flow and go to data flow designer
2. Drag and drop ZS CSV Source from SSIS toolbox. You can use any Data source but for simple demo we will use it.
3. Double click CSV source and select Direct Value mode from dropdown. Enter following sample data (Lets create 12 sample products.)
   

   SKU,ProductName
   PROD-1,Product 1
   PROD-2,Product 2
   PROD-3,Product 3
   PROD-4,Product 4
   PROD-5,Product 5
   PROD-6,Product 6
   PROD-7,Product 7
   PROD-8,Product 8
   PROD-9,Product 9
   PROD-10,Product 10
   PROD-11,Product 11
   PROD-12,Product 12

4. Now click OK to save UI
5. Now drag ZS Template Transform from toolbox.
6. Connect CSV Source to Template Transform
7. Enter following sample text to build request for new product. Refer to Magento API help file / tutorial to learn more.
   

   {
     "product": {
       "sku": "<%SKU%>",
       "name": "<%ProductName%>",
       "attribute_set_id": 4,
       "price": 25,
       "status": 1,
       "visibility": 1
      }
   }

8. Notice that when you click Insert Variable > Columns > select desired upstream column placeholder. You can also encode special characters in your data (e.g. new lines) using function like <%MyColumn,JSONENCODE%>
9. Click OK and save UI
10. Now drag ZS Web API Destination from SSIS Toolbox. Configure like below.Enter URL same as before (used to read product), change Method to POST, change content type to Application/JSON.
11. Now run entire flow (You can watch your API requests in Fiddler for debugging)

Debug Magento Web Requests using Fiddler

Click on below article to learn how to debug web requests

How to use Fiddler to analyze HTTP Web Requests

Magento Integration in Other Apps (e.g. Power BI / Informatica / SQL Server)

You can use techniques listed on this page with ODBC Drivers too for integration in other tools like Power BI, SQL Server code, Informatica etc.

Conclusion

In this post we saw how easy it is to perform Magento Integration in SSIS without doing any programming. In few steps you can read or write data in Magento using SSIS PowerPack REST API components. Download FREE Trial and explore many other scenarios.

The post SSIS Magento data Read / Write using REST API Call appeared first on ZappySys Blog.

In this post you will learn how to call Amazon MWS API (Amazon Marketplace Web Service) or Amazon Product Advertising API using SSIS PowerPack.

Using drag and drop approach you can consume data from Amazon MWS XML Web service. In this post we will use ZappySys XML Source connector to read data from Amazon MWS API and load into SQL Server. We will also use REST API Task to call any API from Marketplace Web service and save output into Variable.

Prerequisites

This post assumes following things

1. Basic knowledge of SSIS and XML format.
2. Amazon MWS Account and and valid AWSAccessKeyId and Secret Key to call Amazon MWS API (How to obtain Key / Secret and SellerId ? ).
3. You have tested few API calls using Amazon MWS Scratchpad this will give you idea about various API you can call and parameters you need to pass.

Testing MWS Requests using ScratchPad

Very first step we recommend before you call MWS API in SSIS is to get familiar with Amazon MWS Scratchpad tool. Check this article for detailed steps.  You can also use Fiddler to debug some requests.

Calling Amazon MWS API in Scratchpad Testing Tool (Response Details Tab) – ListMatchingProducts Example

Call Amazon MWS API using SSIS

To consume data from any XML Source (File, SOAP Web Service or XML API) you can use  ZappySys XML Source
connector. or REST API Task. In below example you will see how to call Amazon MWS API and save output into Variable. With this approach you can call any API (GET / POST / DELETE / PUT)

1. Download and Install SSIS PowerPack
2. Create test package
3. From SSIS toolbox drag ZS REST API Task  and double click on the task to configure it
4. Select Request URL Access Mode = [Url from Connection]
5. Enter URL as below
   

   https://mws.amazonservices.com/Orders/2013-09-01

   In above URL
   ** mws.amazonservices.com => This is your endpoint. If you are not in USA then Click here to find correct endpoint
   ** /Orders => That’s your API you want to call
   ** /2013-09-01 => That’s your API version (You have to add this same version in your POST Body too (see next section).
6. In Select Url Connection dropdown select New OAUTH connection option.
7. When prompted on OAuth connection UI select Provider=Amazon MWS, Enter AWSAccessKey and Secret key and Click OK (If you have MWSAuthToken then enter in Access Token field else leave it empty). Do not test because it may not work yet.
   

   SSIS OAuth Connection – Call Amazon MWS API (Marketplace Web service API)

8. Another Setting you may turn on is Retry Handling. This allows to over come ThresholdLimitReached error when we call API too fast.
   

   Retry Options

9. Now select Method=POST, and in the body enter following
   

   Action=GetServiceStatus&SellerId=A77XXXXXXX&Version=2013-09-01

   If your API requires pagination (more than 100 records from response) then refer next section (Sample Package). In the case of Pagination your second request may look like below
   

   Action=YourActionNameWithNextToken&SellerId=A77XXXXXXX&Version=2013-09-01&NextToken={{User::vNextToken,FUN_URLENC}}

   NOTE: Enter all required parameters in the following format, as shown below. All values need to be URL-encoded. If you use SSIS variable, use it along with FUN_URLENC format specifier, e.g. {{User::myVariable,FUN_URLENC}}. If you use a hard-coded value, and not sure if it is URL-encoded, then encode it with the same FUN_URLENC format specifier/function, e.g. <<myValue,FUN_URLENC>>. Always check if SSIS configuration matches the one you see in MWS ScratchPad Request Details.
   Format:
   param1=value1&param2=<<value2,FUN_URLENC>>&param3={{User::myVar,FUN_URLENC}}
10. Now click Test and see content in the Preview window.
    

    SSIS REST API Task – Call Amazon MWS API – Get XML data save into SSIS variable

11. If you wish to save response data into SSIS variable then goto response Tab and check save option and select variable name.

Amazon MWS API Pagination

Many Amazon MWS API calls by default don’t return all records (Explained here). Rather than returning all rows you may get partial response (e.g. Max 50 or 100 rows). If you want to get all records then you have to use NextToken  found in your XML. For API which requires pagination, in that case your POST body may be different for 2nd or higher requests. Notice how we changed Action and appended NextToken=xxxxxxxxxxx in the second request. Also notice we used encoded token (URL encoded). This is very important because as per Amazon Specs this value has to be Url Encoded.

We also documented similar use case of Amazon MWS Pagination for Power BI (Check this one) its mostly same setup except 2-3 properties might be named different way.

Now lets look at sample request / response for paginated MWS requests (For clarity we have removed many common Parameters which are automatically added at runtime e.g. Signature, Timestamp). We use ListOrders (First Request) and ListOrderByNextToken (Second+ requests)

First Request

POST https://mws.amazonservices.com/Orders/2013-09-01

Action=ListOrders&SellerId=A10zzzzzzz&CreatedAfter=2018-09-01T04%3A00%3A00Z

First Response

<?xml version="1.0"?>
<ListOrdersResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

Second Request or higher (MWS API with Pagination using NextToken)

POST https://mws.amazonservices.com/Orders/2013-09-01

Action=ListOrdersByNextToken&SellerId=A10zzzzzzz&CreatedAfter=2018-09-01T04%3A00%3A00Z&NextToken=YOUR_ENCODED_TOKEN_FROM_PREV_REQUEST

Second or higher Response

<?xml version="1.0"?>
<ListOrdersByNextTokenResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersByNextTokenResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

Sample SSIS Package

If you want to see complete working package for pagination pattern then download from here. Change AccessKey, SecretKey, SellerID and Path as needed to make this package work.

Here is the screenshot of example SSIS package for Amazon MWS.

Amazon MWS APi Pagination (ListOrders , ListOrdersByNextToken Example)

Copy/Paste Properties

Here is copy/paste values for properties listed above.

DirectPath:  https://mws.amazonservices.com/Orders/2013-09-01
EnablePageTokenForBody: True
Filter:  $.ListOrders[$tag$]Response.ListOrders[$tag$]Result.Orders.Order[*]
HasDifferentNextPageInfo: True
HttpRequestData:  Action=ListOrders[$tag$]&SellerId=A1xxxxxxxx&CreatedAfter=2018-09-01T04%3A00%3A00Z&MarketplaceId.Id.1=ATVPDKIKX0DER
HttpRequestMethod:  POST
NextUrlAttribute: $.ListOrders[$tag$]Response.ListOrders[$tag$]Result.NextToken
NextUrlSuffix: &NextToken=<%nextlink_encoded%>
NetUrlWait: 10000 (ms) -- slow down to avoid throttling ... Max 6 requests per minute, increase if you get API Limit Error
PagePlaceholders:  body=|ByNextToken;filter=|ByNextToken

Understanding Pagination Properties

Here are various properties you have to set for Pagination.

POST /Orders/2013-09-01?AWSAccessKeyId=AKxxxxxxxxxx
  &Action=ListOrders
  &SellerId=A10zzzzzzz
  &SignatureVersion=2
  &Timestamp=2018-09-17T21%3A35%3A57Z
  &Version=2013-09-01
  &Signature=3GigipfRsQgzzzzzzzzzzzzzs%3D
  &SignatureMethod=HmacSHA256

Action=ListOrders[$tag$]&SellerId=A10zzzzzzz&CreatedAfter=2018-09-01T04%3A00%3A00Z

<?xml version="1.0"?>
<ListOrdersResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

<?xml version="1.0"?>
<ListOrdersByNextTokenResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersByNextTokenResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

NextUrlEndIndicator=false
StopIndicatorAttributeOrExpr= $.GetReportRequestList[$tag$]Response.GetReportRequestList[$tag$]Result.HasNext

Amazon MWS API Pagination (For Old Version)

So in previous section we saw how to achieve pagination in one step (for newer version of SSIS PowerPack). But there will be a case you cannot use latest SSIS PowerPack or you want to take complete control on each aspect of pagination. In such case use below method (Depreciated).  In newer version we added many new properties so you dont have to do Loop pattern like below.

Sample SSIS Package

If you want to see complete working package for pagination pattern then download from here. Change AccessKey, SecretKey, SellerID and Path as needed to make this package work.

Here is the screenshot of example SSIS package for Amazon MWS.

SSIS Example Package – Read data from Amazon MWS Web service (API Call). Paginate API calls using NextToken

Load Amazon MWS API data into SQL Server using SSIS

In this section we will see how to use ZappySys XML Source connector to read data from Amazon MWS Web Service and load into SQL Server

For making things simple we are calling

1. Download and Install SSIS PowerPack
2. Create test package
3. From SSIS toolbox drag Data Flow task and double click task to go to Data Flow designer
4. Drag ZS XML Source from SSIS Toolbox
5. Configure XML Source as below
   

   SSIS XML Source – Read data from Amazon MWS API (Amazon Marketplace Web service call)

6. Drag OLEDB destination from SSIS Toolbox.
7. Connect XML Source to OLEDB Destination and map input column to target table
8. Execute dataflow
   

   Get data from Amazon MWS API – Save to File or SQL Server using SSIS

Working with Amazon MWS Feed API (e.g. Upload file)

Amazon MWS Provides Feed APIs which has slight different requirements. Important requirement in Feed file upload is you have to supply Content MD5 Hash. If you use SSIS PowerPack then you don’t have to worry about that complexity because calculating MD5 Hash is automatically done.

Upload Amazon MWS Feed File in TSV / CSV format (Tab separated)

Here is the screenshot for how to upload Tab separated values in CSV file. You can specify body from Variable or Read from file. If you read from file then must check File Upload/Multi-Part option and start path with @ symbol as below.

Call Amazon MWS Feed API – Upload CSV file format (Tab separated values – TSV)

Upload Amazon MWS Feed File in XML format

To upload feed file in XML format use same settings as above except Content Type should be XML (text/xml) (Select in the dropdown)

Calling Amazon Product Advertising API

If you have need to call Amazon Product Advertising API then also you can use above techniques (Use MWS Provider on OAuth connection) to call Product Advertise API. Both API (MWS API and Product Advertise API) use same signature hashing algorithm so its possible to use MWS Provider for Hashing. You may have to change your API URL parameters as needed when you call Amazon Product Advertising API. Basically when you call any MWS or Product Advertising API using GET method then OAuth connection manager automatically appends Signature to URL at runtime.

Import data from MWS Custom Reports (Inventory Report Example)

Now lets look at how to extract data from Custom Report API . Reading data from Custom Reports not single step process because report generation is Job style API. Which means you send report request and wait  until its done. Once Report ready you have to read in CSV format or XML. Some Reports only available in CSV format. Check this post to understand how complex it can be to get data. We will make it simple for you to understand this in 3 steps. Basically calling reports requires minimum 3 API calls (see below)

1. Create a new report request – Cal RequestReport API . It returns ReportRequestId (You can use it in the next step)
2. Check Report Status see its done – Call GetReportRequestList API (Pass ReportRequestId got in the previous step to get data for only one Report request we care). Keep checking Report Status every few seconds until you get _DONE_ status in response.
3. Read Report Data – Call GetReport. This API call returns CSV or XML data. So use correct component for this step. If CSV data is returned then we must use CSV Source rather than XML Driver.

Now lets see how to read Amazon MWS custom report in SSIS. In this example, We will get Inventory by calling _GET_MERCHANT_LISTINGS_DATA_ report type.

Download Sample Package for MWS Report

To make things simple we have created a demo package – ready to go.
Download SSIS 2012 Demo – Amazon_MWS_API_GetReport_2012.

Prepare demo package for first run

Once you download and extract above sample package. Import to your existing SSIS Solution. Its SSIS 2012 format so it may upgrade to higher version if you using SSIS 2014, 2016 or 2017.

Once package is opened perform following steps in before you can execute the package.

1. Enter MarketplaceId, SellerId in Variables. Double click OAuth Connection and enter your MWS Account Key and Secret.
2. Create c:\temp directory (This is where we will save final report file)
3. Run this package
4. Go to Execution log and Find ReportId (This you can hardcode in Variables for Design time Preview / Metadata changes etc)

See below package screenshot

Read data from Amazon MWS Report in SSIS (Get Inventory Listing Example)

Steps Explained

Now lets see each steps in depth. We broken whole process in 3 groups.

Create Report Request (Step1)

This is the first step how you initiate report request.  Use REST API Task to call this step. Here is raw request. Many items in body added at runtime (e.g. Signature, Timestamp).

• On Request tab see how we used SSIS Variables to get many values at runtime dynamically.
• On Response Tab we have to extract RequestId and save into variable.

POST 
https://mws.amazonservices.com/Reports/2009-01-01

AWSAccessKeyId=Axxxxxxxxx&Action=RequestReport&MarketplaceId=ATVPDKIKX0DER&ReportType=_GET_MERCHANT_LISTINGS_DATA_&SellerId=Axxxxxxxxx&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2019-06-05T21%3A23%3A31.090Z&Signature=2AbGQQ4xxxxxxxxxxxxxxxxxxxxjJzox%2FU%3D

RequestReport Action – Generate Amazon MWS Custom Report

Extract value from XML response using XPATH and save to Variable (Get RequestId Example)

Here is how to configure above task

Wait until data is ready (Step2)

Once we send RepotRequest and we have RequestId we can call step2 using REST API Task. In this Task we can use Status Check Feature. This is pretty handy feature because we dont have to implement Loop in SSIS. Basically we have to keep checking report until we get _DONE_ status in the response (ReportProcessingStatus node in XML)

Here is how to configure the task.

1. Request Tab Configuration.
   URL:

   https://mws.amazonservices.com/Reports/2009-01-01

   Method: POST
   Body:

   Action=GetReportRequestList&ReportRequestIdList.Id.1={{User::ReportRequestId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}}

2. Response Setting Tab Configuration
   Response content type: Xml
   Filter Expression: 

   //*[local-name() = 'ReportProcessingStatus']  

3. Status Check Tab
   Check Enable Status
   Success Value: 

   _DONE_

   Check Every 5 Seconds

REST API Task – Status Check Feature (Wait until data is ready loop)

GetReportId (Step3)

Once above task finished we can call same request again but this time we have to configure slightly different way.

Here is how to configure the task.

1. Request Tab Configuration.
   URL:

   https://mws.amazonservices.com/Reports/2009-01-01

   Method: POST
   Body:

   Action=GetReportRequestList&ReportRequestIdList.Id.1={{User::ReportRequestId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}}

2. Response Setting Tab Configuration
   Response content type: Xml
   Filter Expression: 

   //*[local-name() = 'GeneratedReportId']  

   Check Save to Variable: Save to User::ReportId
3. Status Check Tab
   Make Enable Status is unchecked

Get Report Data  (Step4,Step5 or Step6)

Now we are ready to ready our report. There are two ways we can do in SSIS.

• First approach (step 4,5) is download the report and save to local disk and then in the next step we can use data flow (CSV Source) to parse the saved file.
• Second approach (step6) is rather than saving to disk we can directly call API using CSV File source and parse content that way. Lets talk each approach briefly.

Step4,5 – Load Report From File

If you wish to save Report on disk then you can use REST API Task and call request like below.

Here is how to configure the task to download the file.

1. Request Tab Configuration.
   URL:

   https://mws.amazonservices.com/Reports/2009-01-01

   Method: POST
   Body: Action=GetReport&ReportId={{User::ReportId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}}
2. Response Setting Tab Configuration
   Response content type: Xml
   Filter Expression: 

   //*[local-name() = 'GeneratedReportId']  

   Check Save to Variable: Save to User::ReportId
   
3. On Response Settings Tab : Set Response charset to Western European [Windows 1252]  (or directly set via Properties Grid to Windows-1252 under ResponseCharset Property)
   
4. Status Check Tab
   Make Enable Status is unchecked

To read download CSV file use settings described in below section except URL change it with local file path.

Step6 – Load report from URL

If you wish to load directly from URL then use same setting as above but in CSV Source inside data flow.

Here is how to configure CSV Source.

1. Request Tab Configuration.
   URL:

   https://mws.amazonservices.com/Reports/2009-01-01

   Method: POST
   Body:

   Action=GetReport&ReportId={{User::ReportId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}} 

2. General Tab
   Column Delimiter: {TAB}
3. On Encoding Tab : Set to Western European [Windows 1252]  (or directly set via Properties Grid to Windows-1252 under CharacterSet Property)

Conclusion

Amazon Marketplace Web service (MWS) provides great way to automate many functionality for Marketplace Seller. Using ZappySys SSIS PowerPack you con consume data from any REST API or Web service without replying on SDK / coding approach (e.g. C#, Java, Python, Ruby).

The post How to call Amazon MWS API using SSIS appeared first on ZappySys Blog.