In our previous blog post we saw how to import rest API in SQL Server. Using same concepts lets look at how to import Amazon MWS Data into SQL Server. We will explore many techniques to call Amazon MWS API and learn how to automate Amazon MWS data extraction without doing any ETL. We will call MWS XML or CSV API just using T-SQL code. We will also learn how to Download Reports from by calling MWS API.

You can also refer other articles here (Power BI) and here (SSIS)

About Amazon MWS  API

If you want to sell something on Amazon you can use their e-commerce platform with some monthly fee. You can setup your entire online store / inventory using admin interface.

Amazon Marketplace Web Service (Amazon MWS) is an integrated web service API that helps Amazon sellers to programmatically exchange data on listings, orders, payments, reports, and more. Using these API you can read or write data from your Seller account and integrate it inside your own Systems (e.g. Reporting / ETL / BI tools).

Obtain MWS API Access Key / Secret and Seller ID

Very first thing to call any MWS API is to obtain Access Key, Secret Key and Seller ID (i.e. Merchant ID). To obtain this you first have to register a developer.

Here is where to look for your Merchant ID and Key / Secret.  Here is direct link to User Permissions page.

How to get Amazon MWS developer keys and know your Merchant ID (i.e. Seller ID)

Getting Started

Now let’s look at step by step instructions on how to call Amazon MWS API and then import into Power BI. In below steps we will use  ZappySys ODBC PowerPack so make sure it’s installed first.

Using ScratchPad to Test Amazon MWS API

Each MWS API has set of required and optional parameters. You can refer each API page for details. For example ListMatchingProducts API which we will use as an example. To make MWS API testing / learning easy, Amazon provides online testing tool called MWS ScratchPad. We will use this tool to craft API requests and use that information in ODBC driver SQL queries and later import data in Power BI.

1. Open MWS ScratchPad  by vising https://mws.amazonservices.com/scratchpad/index.html
2. Select Products API category and pick ListMatchingProducts API
3. Enter your SellerID (i.e. Merchant ID), Developer AccessKey (i.e. AWSAccessKeyId) and Secret Key we obtained in the previous section. Enter MarketId (e.g. ATVPDKIKX0DER for USA Market) and other parameters as below and Click Submit to see response.
   

   Using Amazon MWS ScratchPad (API Testing Tool – Created by Amazon)

4. Click on the Response Details Tab to extract some important information we will use for ODBC configuration. Notice attributes in red rectangles. They are the ones which we will need in the POST request Body. ZappySys Drivers supply many common parameters automatically but any API endpoint specific parameter must be supplied in the Request Body. For example you dont have to supply SignatureVersion, Timestamp, Signature or SignatureMethod
   

   Calling Amazon MWS API in Scratchpad Testing Tool (Response Details Tab) – ListMatchingProducts Example

This API requires Market ID parameter so enter it as per your Market where you selling. For example USA marketplaceid = ATVPDKIKX0DER. You can lookup correct MarketplaceId by checking this help (endpoints / marketplaces).

Setup Amazon MWS API Connections

Create XML Driver Connection for Amazon MWS API (e.g. RequestReport )

Some APIs we need to call in this article is XML format and some API (e.g. GetReport) is CSV format – Tab delimited so we need to create 2 different connections using CSV driver and XML driver. So lets get started with Amazon MWS Connection for XML API.

1. Click on Add New Data source. Name as AMAZON-MWS-XML and select Native – ZappySys XML Driver option.
   

   Add Gateway Data Source

2. Now click Edit to Configure Settings. For now we will only care OAuth Connection settings. All other options are overwritten in SQL Query.
   

   Edit Gateway Data Source Settings

3. When UI opens for Driver Enter MWS API Service URL as below. Notice that we added /Products/2011-10-01 (This was extracted from previous section). You can check full list of Amazon MWS Endpoints here. Because we are connecting to USA region we will use below URL.
   

   https://mws.amazonservices.com/Products/2011-10-01

4. Select OAuth from the Connection dropdown and Click Configure Settings
5. On the Connection UI, Select OAuth Provider Amazon Market Web Service (MWS)
6. Enter your API Access Key and Secret Key (obtained from here)
7. Your connection will look like below.
   

   ODBC Connection Credentials for Amazon MWS API (Access Key, Secret Key, URL Endpoint)

8. Also configure Retry options to handle API limit reached error when you call API too often. For Amazon MWS we get Status code 503 when request is throttled so lets narrow down on which error we want to retry.
   

   Retry Options

9. Click OK to save OAuth Connection UI to go back to main UI
10. Now Go to Throttling Tab and enter some delay (e.g. enter 20000 for 20sec delay after each request). This will slow down API calls so we don’t face errors about API Limit reached.
11. Click OK to Save UI
12. Click Save button on Gateway UI to save and restart

Create CSV Driver Connection for Amazon MWS API (e.g. GetReport )

Now lets create a connection CSV API (e.g. GetReport ).

Follow almost same steps as previous section except use CSV Driver when you click on Add new Data source select Native – ZappySys CSV Driver and name data source as AMAZON-MWS-CSV . As we mentioned for now we mostly want to setup Connection and not worry about Request Body, Pagination etc.

Create Linked Server for SQL Server

Run following Script in SSMS to create Linked servers. Change ######## with Your gateway password. Change localhost with machine name or IP where ZappySys Gateway Is running. Use localhost if SQL Server and Gateway both on the same machine.

USE [master]
GO

EXEC master.dbo.sp_addlinkedserver @server = N'AMAZON-MWS-XML', @srvproduct=N'', @provider=N'SQLNCLI', @datasrc=N'localhost,5000', @catalog=N'AMAZON-MWS-XML'
EXEC master.dbo.sp_addlinkedsrvlogin @rmtsrvname=N'AMAZON-MWS-XML',@useself=N'False',@locallogin=NULL,@rmtuser=N'tdsuser',@rmtpassword='########'
GO

EXEC master.dbo.sp_addlinkedserver @server = N'AMAZON-MWS-CSV', @srvproduct=N'', @provider=N'SQLNCLI', @datasrc=N'localhost,5000', @catalog=N'AMAZON-MWS-CSV'
EXEC master.dbo.sp_addlinkedsrvlogin @rmtsrvname=N'AMAZON-MWS-CSV',@useself=N'False',@locallogin=NULL,@rmtuser=N'tdsuser',@rmtpassword='########'
GO

If you want to know how to configure Linked Server using UI then check this article.

Amazon MWS ListOrders Example

Now lets look at how to call ListOrders request with pagination in SQL Server. Create a following Stored Procedure to List MWS Orders. If you want to extract many orders then call Report rather than this proc (See next section for generating reports)

USE MyDatabase
GO
/*
Example: 
 exec usp_MWS_ListOrders @SellerId='Axxxxxxx', @MarketplaceId ='Bxxxxxxx',@CreatedAfter='2019-01-01'  
*/
Create proc [dbo].[usp_MWS_ListOrders]
	@SellerId varchar(100),
	@MarketplaceId varchar(100),
	@CreatedAfter varchar(100),
	@CreatedBefore varchar(100)=null,
	@MaxRows int=0
as
declare @sql varchar(max) 
declare @limitClase varchar(100)=''
if(@MaxRows>0)
	set @limitClase	=' LIMIT ' + cast(@MaxRows as varchar(20))

--//Confirm ISO Date format 2019-06-01T04%3A00%3A00Z  rather 2019-06-01T04:00:00Z
if(isnull(@CreatedAfter,'')<>'')
begin
	set @CreatedAfter='&CreatedAfter=' + @CreatedAfter
	set @CreatedAfter=replace(@CreatedAfter,':','%3A')
	if(@CreatedAfter NOT LIKE @CreatedAfter + 'T%')
		set @CreatedAfter=@CreatedAfter + 'T00%3A00%3A00Z'
	if(@CreatedAfter NOT LIKE @CreatedAfter + '%Z')
		set @CreatedAfter=@CreatedAfter + 'Z'
	--//T04%3A00%3A00Z
end
if(isnull(@CreatedBefore,'')<>'')
begin
	set @CreatedBefore='&CreatedBefore=' + @CreatedBefore
	set @CreatedBefore=replace(@CreatedBefore,':','%3A')
	if(@CreatedBefore NOT LIKE @CreatedBefore + 'T%')
		set @CreatedBefore=@CreatedBefore + 'T00%3A00%3A00Z'
	if(@CreatedBefore NOT LIKE @CreatedBefore + '%Z')
		set @CreatedBefore=@CreatedBefore + 'Z'
end

set @CreatedAfter = isnull(@CreatedAfter,'')
set @CreatedBefore = isnull(@CreatedBefore,'')


set @sql = 'SELECT * FROM OPENQUERY([AMAZON-MWS-XML]
, ''
SELECT ReportRequestId FROM $
' + @limitClase +' 
WITH(
     Src=''''https://mws.amazonservices.com/Orders/2013-09-01''''
	,Filter=''''$.RequestReportResponse.RequestReportResult.ReportRequestInfo''''
	,ElementsToTreatAsArray=''''Order''''
	,RequestData=''''Action=ListOrders[$tag$]' + @CreatedAfter + @CreatedBefore + '&MarketplaceId.Id.1='+ @MarketplaceId +'&SellerId='+ @SellerId +'''''
	,RequestMethod=''''POST''''
	,NextUrlAttributeOrExpr=''''$.ListOrders[$tag$]Response.ListOrders[$tag$]Result.NextToken''''
	,NextUrlSuffix=''''&NextToken=<%nextlink_encoded%>''''
	,WaitTimeMs=20000 -- slow down to avoid throttling. 6 requests per min
	,HasDifferentNextPageInfo=''''True''''
	,EnablePageTokenForBody=''''True''''
	,PagePlaceholders=''''body=|ByNextToken;filter=|ByNextToken''''
	--,DataConnectionType=''''OAuth''''
	--,ScopeSeparator=''''{space}''''
	--,ServiceProvider=''''AmazonMWS''''
	--,ClientId=''''AKIxxxxxx''''
	--,ClientSecret=''''AKIxxxxxx''''
	--,UseCustomApp=''''True''''
	,CacheFileLocation=''''c:\temp\mws-orders.cache''''
	,CacheEntryTtl=300 --send every 5 mins else use from cache
	,IncludeParentColumns=''''False''''
)
'')'
print @sql

EXECUTE( @sql)

Here is how to call this proc. Change SellerId, MarketplaceId and Date.

exec usp_MWS_ListOrders @SellerId='Axxxxxxx', @MarketplaceId ='Bxxxxxxx',@CreatedAfter='2019-01-01'

Download Amazon MWS Report data in SQL Server (CSV / XML Format)

Now lets look at how to extract data from Custom Report API (You must need ODBC PowerPack v1.1.1 or higher). However reading data from Reports not single step process because report generation is Job style API. Which means you send report request and wait  until its done. Once Report ready you have to read in CSV format or XML. Some Reports only available in CSV format. Check this post to understand how complex it can be to get data. We will make it simple for you to understand this in 3 steps. Basically calling reports requires minimum 3 API calls (see below)

1. Create a new report request – Cal RequestReport API . It returns ReportRequestId (You can use it in the next step)
2. Check Report Status see its done – Call GetReportRequestList API (Pass ReportRequestId got in the previous step to get data for only one Report request we care). Keep checking Report Status every few seconds until you get _DONE_  or _DONE_NO_DATA_ status in response.
3. Read Report Data – Call GetReport. This API call returns CSV or XML data. So use correct Driver for this step. If CSV data is returned then we must use CSV Driver rather than XML Driver.

Method 1 – Single query approach (New Driver)

In new driver ZappySys added few new features which makes it possible to call above 3 steps process in a single query. New query syntax offers StatusCheck and Mixing Content Formats using same driver. Because Most MWS Reports are in CSV Format we have to use CSV Driver but oddly first 2 steps listed in above sections are in XML Format so we have to use XML driver. Technically we need atleast 2 steps (i.e 2 queries) but we will show you how to achieve using single query as below.

1. Assuming you have followed steps to create a linked server AMAZON-MWS-CSV (see previous section)
2. Run below query. Replace necessary parameters as per your need. (i.e MarketplaceId , SellerId, CreatedAfter date)
   

   --- //////////////////////////////////////////////////////////////////////////
   -- Change _GET_MERCHANT_LISTINGS_DATA_ to something else if you need different report type (all 3 steps) -- https://docs.developer.amazonservices.com/en_US/reports/Reports_ReportType.html
   -- Replace MarketplaceId in all 3 steps (i.e for USA use ATVPDKIKX0DER) --https://docs.developer.amazonservices.com/en_US/dev_guide/DG_Endpoints.html
   -- Replace SellerId in all 3 steps
   -- Remove or Change CreatedAfter in first step (or use CreatedBefore parameter instread of CreatedAfter)
   --- //////////////////////////////////////////////////////////////////////////
   select * into tmpMWS_Report from OPENQUERY([AMAZON-MWS-CSV],
   '
   SELECT s3.* FROM $
   WITH(
   Src=''https://mws.amazonservices.com/Reports/2009-01-01'',RequestMethod=''POST''
   ,RequestData=''Action=RequestReport&ReportType=_GET_MERCHANT_LISTINGS_DATA_&MarketplaceId=ATVPDKIKX0DER&SellerId=Axxxxxxxxxx&CreatedAfter=2017-12-31T04%3A00%3A00Z''
   ,IncludeParentColumns=''False''
   ,EnableRawOutputModeSingleRow=''True''
   ,RawOutputFilterExpr=''//*[local-name()="ReportRequestId"]'' 
   ,RawOutputDataRowTemplate=''{ReportRequestId:"[$1]"}'' 
   ,RawOutputExtractMode=''Xml'' 
   ,WaitTimeMs=5000
   ,Meta=''[{"Name": "ReportRequestId","Type": "Int64"}]''
   ,Alias=''s1''
   
   ,Join1_alias=''s2''	
   ,Join1_Src=''https://mws.amazonservices.com/Reports/2009-01-01'',Join1_RequestMethod=''POST''
   ,Join1_RequestData=''Action=GetReportRequestList&ReportRequestIdList.Id.1=[$s1.ReportRequestId$]&MarketplaceId=ATVPDKIKX0DER&SellerId=Axxxxxxxxxxxxx''
   ,Join1_EnableRawOutputModeSingleRow=''True''
   ,Join1_RawOutputFilterExpr=''//*[local-name()="ReportProcessingStatus"]||//*[local-name()="GeneratedReportId"]'' 
   ,Join1_RawOutputDataRowTemplate=''{ReportProcessingStatus:"[$1]",GeneratedReportId:"[$2]"}'' 
   ,Join1_RawOutputExtractMode=''Xml'' 
   ,Join1_EnableStatusCheck=''True'', 
   ,Join1_StatucCheckMaxWaitSeconds=300,
   ,Join1_StatucCheckIterationWaitSeconds=25,
   ,Join1_StatusSuccessValue=''_DONE_''
   ,Join1_StatusFailedValue=''_DONE_NO_DATA_|_CANCELLED_''
   
   ,Join2_Alias=''s3''
   ,Join2_Src=''https://mws.amazonservices.com/Reports/2009-01-01'',Join2_RequestMethod=''POST''
   ,Join2_RequestData=''Action=GetReport&ReportId=[$s2.GeneratedReportId$]&MarketplaceId=ATVPDKIKX0DER&SellerId=Axxxxxxxxxxxxxx''	
   ,Join2_ColumnDelimiter=''{TAB}''
   ,Join2_ResponseCharset=''Windows-1252''	 
   ,Join2_EnableStatusCheck=''False'' 
   ,Join2_EnableRawOutputModeSingleRow=''False''
   ,Join2_WaitTimeMs=10000
   )') 
   go
   select * from tmpMWS_Report

    

Performance Tip

If you like to improve performance and dont want to use SELECT INTO approach to load into temp table as above then you can use EXEC (….) AT YourLinkedServerName  as below.

Advantage of this method is your Query speed will increase because system calls API only once when you call EXEC AT. In OPENROWSET it needs to call above query twice (Once to obtain metadata and once to get data).

INSERT INTO tmpMWS_Report
EXEC('select s3.* .................. ') AT [AMAZON-MWS-CSV]

Method 2 – Multiple query approach (For old driver)

Now lets see how to load Amazon MWS data in SQL Server using old driver (v1.1.2 or older) which didnt support single query approach like previous section. If you are using older driver or for some reason you like to split steps into multiple procedures then use below approach.

usp_MWS_Report  Stored Proc ( Generate ReportRequest, Wait until done, Read data )

USE MyDatabase
GO

/*
Examples: 
Get ReportType from here https://docs.developer.amazonservices.com/en_UK/reports/Reports_ReportType.html

exec usp_MWS_Report @MarketplaceId ='ATVPDKIKX0DER', @SellerId='AZxxxxxxxx', @ReportType = '_GET_MERCHANT_LISTINGS_DATA_'
exec usp_MWS_Report @MarketplaceId ='ATVPDKIKX0DER', @SellerId='AZxxxxxxxx', @ReportType = '_GET_FLAT_FILE_ORDER_REPORT_DATA_'

--Do not Throw Error if no data found (return empty resultset or run custom sql)
exec usp_MWS_Report @MarketplaceId ='ATVPDKIKX0DER', @SellerId='AZxxxxxxxx', @ReportType = '_GET_FLAT_FILE_ORDER_REPORT_DATA_',@ThrowErrIfNoData=0,@EmptyRowSql='select top 0 * from Orders'
--Supply date range
exec usp_MWS_Report @MarketplaceId ='ATVPDKIKX0DER', @SellerId='AZxxxxxxxx', @ReportType = '_GET_MERCHANT_LISTINGS_DATA_', @StartDate='2019-01-01T00:00:00Z', @EndDate='2019-01-31T00:00:00Z'

--loading into table 

--//////////////////////////////////
— Step-1 : Create Local Linked server to point to self
--//////////////////////////////////
USE [master]
GO
--change default catalog
--For MSSQL 2012, 2014, 2016, 2017, and 2019 use @provider=N'MSOLEDBSQL' below (SQL Server Native Client 11.0)---
EXEC master.dbo.sp_addlinkedserver @server = N'ME', @srvproduct=N'', @provider=N'SQLNCLI11', @datasrc=N'localhost', @catalog=N'Northwind'
--For MSSQL 2022 or higher use @provider=N'MSOLEDBSQL' below (Microsoft OLE DB Driver for SQL Server)---
--EXEC master.dbo.sp_addlinkedserver @server = N'ME', @srvproduct=N'', @provider=N'MSOLEDBSQL', @datasrc=N'localhost', @catalog=N'Northwind'

EXEC master.dbo.sp_addlinkedsrvlogin @rmtsrvname=N'ME',@useself=N'True',@locallogin=NULL,@rmtuser=NULL,@rmtpassword=NULL
GO
--//////////////////////////////////
— Step-2 : Enable RPC OUT and Disable MSDTC
--//////////////////////////////////
EXEC sp_serveroption 'ME', 'rpc out', true;  
GO
--Below needed to support EXEC + INSERT (dynamic query)
EXEC sp_serveroption 'ME', 'remote proc transaction promotion', false;
GO
--test simple query
select * from OPENQUERY(ME,'select 1')
GO

--//////////////////////////////////
— Step-3 : Load data into table from MWS Report
--//////////////////////////////////
SELECT * INTO #amazon_mws_report
FROM OPENQUERY(me,'
SET NOCOUNT ON;
EXEC usp_MWS_Report 
   @Columns=''[item-name], [item-description], [listing-id]'',
   @MarketplaceId =''ATVPDKIKX0DER'', 
   @SellerId=''Axxxxxxxxxxxxxxxxx'', 
   @ReportType = ''_GET_MERCHANT_LISTINGS_DATA_''
WITH RESULT SETS 
(
	(itmname varchar(1000),itmdesc varchar(1000),itmid varchar(1000))
)
')

select * from #amazon_mws_report


*/
CREATE proc [dbo].[usp_MWS_Report]
	@MarketplaceId varchar(100),
	@SellerId varchar(100),
	@ReportType varchar(100)='_GET_MERCHANT_LISTINGS_DATA_',
	@StartDate varchar(100)=null,
	@EndDate varchar(100)=null,
	@ReportId bigint=null,  ---i.e. 14783134522018025
	@ThrowErrIfNoData bit=1,
	@EmptyRowSql varchar(1000)=null, --SQL to run if no data found
	@Columns varchar(8000)='*'  --report columns which goes in final SELECT  e.g. [item-name], [item-description], [listing-id] ....
as


declare @retGeneratedReportId bigint
set @retGeneratedReportId=@ReportId
---////// STEP 1 /////////
if(@ReportId is null)
begin
	create table #step1(ReportRequestId bigint)
	insert into #step1
	exec usp_MWS_RequestReport @ReportType,@MarketplaceId,@SellerId,@StartDate,@EndDate

	--select * from #step1 --debug

	declare @retReportRequestId bigint
	select @retReportRequestId=ReportRequestId from #step1

	---////// STEP 2 /////////
	declare @status varchar(100)
	declare @cnt int
	create table #step2(ReportProcessingStatus varchar(100),GeneratedReportId bigint)
	
	--Keep looping until status is done or we waited too long
	while(isnull(@status,'') IN ('','_SUBMITTED_','_IN_PROGRESS_') )
	begin
		truncate table #step2
		insert into #step2
		exec usp_MWS_GetReportIdAndStatus @retReportRequestId,@MarketplaceId,@SellerId 
	
		select @status=ReportProcessingStatus, @retGeneratedReportId=GeneratedReportId from #step2
		if(isnull(@status,'') IN ('','_SUBMITTED_','_IN_PROGRESS_') )			
		begin
			--amazon says dont check often https://docs.developer.amazonservices.com/en_UK/reports/Reports_GetReportRequestList.html
			Print 'Status='+  isnull(@status,'') +'. Checking after 45 sec... #' + cast(@cnt as varchar(10))
			WAITFOR DELAY '00:00:45'			
		end
		else
			print @status

		set @cnt=@cnt+1

		--if(@cnt>50)
			--//throw err?
	end
	--select * from #step2 --debug
end

if(@retGeneratedReportId is null and @status='_DONE_NO_DATA_')
begin
	if(@ThrowErrIfNoData=1)
		RAISERROR('Cannot generate report - No data found for specified criteria',16,1); 
	else if (isnull(@EmptyRowSql,'')<>'')
		execute(@EmptyRowSql)
end
else if(@retGeneratedReportId is null)
	RAISERROR('Cannot generate report - Unknown error (either report got cancelled or other error occurred)',16,1); 

	
---////// STEP 3 /////////
--If xml format of report data then use XML driver else CSV (most are Tab delimited reports in CSV format)
--exec usp_MWS_GetReportDataXml @retGeneratedReportId,@MarketplaceId,@SellerId
exec usp_MWS_GetReportDataCsv @retGeneratedReportId,@MarketplaceId,@SellerId,@Columns

drop table #step1
drop table #step2

GO

usp_MWS_RequestReport  Stored Proc ( Step1 – Create Report Request )

USE [MyDatabase]
GO

CREATE proc [dbo].[usp_MWS_RequestReport]
	@ReportType varchar(100),
	@MarketplaceId varchar(100),
	@SellerId varchar(100),
	@StartDate varchar(100)=null,
	@EndDate varchar(100)=null
as

/*Learn more : https://docs.developer.amazonservices.com/en_UK/reports/Reports_RequestReport.html */

--//Confirm ISO Date format 2019-06-01T04%3A00%3A00Z  rather 2019-06-01T04:00:00Z
if(isnull(@StartDate,'')<>'')
begin
	set @StartDate='&StartDate=' + @StartDate
	set @StartDate=replace(@StartDate,':','%3A')
	if(@StartDate NOT LIKE @StartDate + 'T%')
		set @StartDate=@StartDate + 'T00%3A00%3A00Z'
	if(@StartDate NOT LIKE @StartDate + '%Z')
		set @StartDate=@StartDate + 'Z'
	--//T04%3A00%3A00Z
end
if(isnull(@EndDate,'')<>'')
begin
	set @EndDate='&EndDate=' + @EndDate
	set @EndDate=replace(@EndDate,':','%3A')
	if(@EndDate NOT LIKE @EndDate + 'T%')
		set @EndDate=@EndDate + 'T00%3A00%3A00Z'
	if(@EndDate NOT LIKE @EndDate + '%Z')
		set @EndDate=@EndDate + 'Z'
end

set @StartDate = isnull(@StartDate,'')
set @EndDate = isnull(@EndDate,'')

declare @sql varchar(max) 
set @sql = 'SELECT * FROM OPENQUERY([AMAZON-MWS-XML]
, ''
SELECT ReportRequestId FROM $
WITH(
	Src=''''https://mws.amazonservices.com/Reports/2009-01-01'''',RequestMethod=''''POST''''
	--,ElementsToTreatAsArray=''''Product''''
	,Filter=''''$.RequestReportResponse.RequestReportResult.ReportRequestInfo''''
	,RequestData=''''Action=RequestReport&ReportType=' + @ReportType  + '&MarketplaceId='+ @MarketplaceId +'&SellerId='+ @SellerId + @StartDate + @EndDate +'''''
	,CacheFileLocation=''''c:\temp\mws-cache-step-1.cache''''
	,CacheEntryTtl=300 --send every 5 mins else use from cache
	,WaitTimeMs=''''300''''
	,IncludeParentColumns=''''False''''
	,Meta=''''[{"Name": "ReportRequestId","Type": "Int64"}]''''
)'')'
--print @sql

EXECUTE( @sql)
--sleep 25 seconds after this - hope report is ready by that time else We have to loop
WAITFOR DELAY '00:00:25'  --lets wait 30 sec... incase we get luck and report is done?
GO

usp_MWS_GetReportIdAndStatus Stored Proc ( Step2 – Get Report Id and Report Status )

USE MyDatabase
GO


CREATE proc [dbo].[usp_MWS_GetReportIdAndStatus]
	@ReportRequestId bigint,
	@MarketplaceId varchar(100),
	@SellerId varchar(100)
as

declare @sql varchar(max) 
set @sql = 'SELECT * FROM OPENQUERY([AMAZON-MWS-XML]
, ''SELECT *  FROM $
WITH(
	 Src=''''https://mws.amazonservices.com/Reports/2009-01-01'''',RequestMethod=''''POST''''
	,ElementsToTreatAsArray=''''ReportRequestInfo''''
	,Filter=''''$.GetReportRequestListResponse.GetReportRequestListResult.ReportRequestInfo''''
	,RequestData=''''Action=GetReportRequestList&ReportRequestIdList.Id.1=' + cast(@ReportRequestId as varchar(20)) + '&MarketplaceId='+ @MarketplaceId +'&SellerId='+ @SellerId +'''''
	,WaitTimeMs=''''300''''
	,IncludeParentColumns=''''False''''
	--,CacheFileLocation=''''c:\temp\mws-cache-step-2.cache''''
	--,CacheEntryTtl=10  --do not use from cache too old.. but cache still prevents too many requests
	,Meta=''''[{"Name": "ReportProcessingStatus","Type": "String", Length: 100},{"Name": "GeneratedReportId","Type": "Int64"}]''''	
)'')'

--print @sql

EXECUTE( @sql)

GO

usp_MWS_GetReportDataCsv Stored Proc (Step3 – Get Report Data in CSV format )

Last step in report generation is download the report data in CSV format.

USE MyDatabase
GO

CREATE proc [dbo].[usp_MWS_GetReportDataCsv]
	@ReportId bigint,
	@MarketplaceId varchar(100),
	@SellerId varchar(100),
	@Columns varchar(8000)='*'

as

declare @sql varchar(max) 
set @sql = 'SELECT '+ @Columns +' FROM OPENQUERY([AMAZON-MWS-CSV], 
''SELECT * FROM $
WITH(
    Src=''''https://mws.amazonservices.com/Reports/2009-01-01'''',RequestMethod=''''POST''''
	,RequestData=''''Action=GetReport&ReportId=' + cast(@ReportId as varchar(20)) + '&MarketplaceId='+ @MarketplaceId +'&SellerId='+ @SellerId +'''''
	,WaitTimeMs=''''300''''
	,ColumnDelimiter=''''{TAB}''''
	,ResponseCharset=''''Windows-1252''''	
	,CacheFileLocation=''''c:\temp\mws-cache-step-3.cache''''
	,CacheEntryTtl=300 --once report is generated lets try to read from cache 
	)''
)'

print @sql

EXECUTE( @sql)
GO

Example – Import data from Amazon MWS into SQL Server Table

Finally when we have all stored procs created we can load into target table as below. In our previous article we saw how to use SELECT INTO along with OPENQUERY to load data into new temp table or insert into some existing table. However when you want to load Stored Proc output into table its not straight forward specially when we calling dynamic SQL.

Three tricks worth mentioning here which makes it possible to load data from stored proc into temp table dynamically.

1. Create a linked server to Self so we can use OPENQUERY against to call stored proc so its easy to use SELECT INTO.
2. Use WITH RESULT SETS to describe metadata of output. This is needed because we are using dynamic SQL inside proc and WITH RESULT SETS describes the metadata.
3. Use of SET NO COUNT ON to avoid error like below
   

   "SOMESERVERNAME" indicates that either the object has no columns 
   or the current user does not have permissions on that object

4. If you dont do above tricks then you may see errors like this (INSERT is not allowed when we use DYNAMIC SQL inside nested Stored Procs)

To load output into table you must know column names from the output.  If you dont know you can try to run stored proc without @Columns like below.

EXEC usp_MWS_Report @MarketplaceId =''ATVPDKIKX0DER'', @SellerId=''Axxxxxxxxxxxxx'', @ReportType = ''_GET_MERCHANT_LISTINGS_DATA_''

Once we know columns from report, you can run following script to insert into table.

Full Script – Load MWS Report Output into Table

--//////////////////////////////////
— Step-1 : Create Local Linked server to point to self
--//////////////////////////////////
USE [master]
GO
--change default catalog
--For MSSQL 2012, 2014, 2016, 2017, and 2019 use @provider=N'MSOLEDBSQL' below (SQL Server Native Client 11.0)---
EXEC master.dbo.sp_addlinkedserver @server = N'ME', @srvproduct=N'', @provider=N'SQLNCLI11', @datasrc=N'localhost', @catalog=N'Northwind'
--For MSSQL 2022 or higher use @provider=N'MSOLEDBSQL' below (Microsoft OLE DB Driver for SQL Server)---
--EXEC master.dbo.sp_addlinkedserver @server = N'ME', @srvproduct=N'', @provider=N'MSOLEDBSQL', @datasrc=N'localhost', @catalog=N'Northwind'

EXEC master.dbo.sp_addlinkedsrvlogin @rmtsrvname=N'ME',@useself=N'True',@locallogin=NULL,@rmtuser=NULL,@rmtpassword=NULL
GO
--//////////////////////////////////
— Step-2 : Enable RPC OUT and Disable MSDTC
--//////////////////////////////////
EXEC sp_serveroption 'ME', 'rpc out', true;  
GO
--Below needed to support EXEC + INSERT (dynamic query)
EXEC sp_serveroption 'ME', 'remote proc transaction promotion', false;
GO
--test simple query
select * from OPENQUERY(ME,'select 1')
GO

--//////////////////////////////////
— Step-3 : Load data into temp table from Amazon MWS Report
--//////////////////////////////////
SELECT * INTO #amazon_mws_report
FROM OPENQUERY(me,'
SET NOCOUNT ON;
EXEC usp_MWS_Report 
   @Columns=''[item-name], [item-description], [listing-id]'',
   @MarketplaceId =''ATVPDKIKX0DER'', 
   @SellerId=''Axxxxxxxxxxxxx'', 
   @ReportType = ''_GET_MERCHANT_LISTINGS_DATA_''
WITH RESULT SETS 
(
	(itmname varchar(1000),itmdesc varchar(1000),itmid varchar(1000))
)
')

select * from #amazon_mws_report

Import Amazon MWS Data into SQL Table (Load Report output into temp table)

Conclusion

In this article, we saw how to load Amazon MWS Data into SQL Server. We used simple SQL queries with ZappySys XML / CSV Driver to pull data from Amazon MWS API. ZappySys ODBC PowerPack makes it super easy to consume data from virtually any API (JSON / XML or CSV Web Services), no coding required. Download here to get started with ZappySys ODBC drivers.

The post How to import Amazon MWS data into SQL Server (T-SQL) appeared first on ZappySys Blog.

In our previous blog we saw how to call REST API in Power BI to import data from JSON based REST API services. Now let’s continue on that and learn how to import Amazon MWS data in Power BI using XML driver. To learn more about how to consume Amazon MWS data in ETL using SSIS check this article.

We will read data from Amazon MWS API using ZappySys XML Driver which is part of ODBC PowerPack.

Requirements

1. Active Amazon Seller Account needed (Make sure you have Professional Plan or higher (around $39/month) to call Amazon MWS API)
2. You are already registered as a Developer and you obtained AWS Access Key / Secret Key and Seller ID (i.e. Merchant ID) (see next section)
3. Download and Install ODBC PowerPack
4. You will also need Power BI Desktop (FREE) installed

About Amazon MWS / API

If you want to sell something on Amazon you can use their e-commerce platform with some monthly fee. You can setup your entire online store / inventory using admin interface.

Amazon Marketplace Web Service (Amazon MWS) is an integrated web service API that helps Amazon sellers to programmatically exchange data on listings, orders, payments, reports, and more. Using these API you can read or write data from your Seller account and integrate it inside your own Systems (e.g. Reporting / ETL / BI tools).

Obtain MWS API Access Key / Secret and Seller ID

Very first thing to call any MWS API is to obtain Access Key, Secret Key and Seller ID (i.e. Merchant ID). To obtain this you first have to register a developer.

Here is where to look for your Merchant ID and Key / Secret.  Here is direct link to User Permissions page.

How to get Amazon MWS developer keys and know your Merchant ID (i.e. Seller ID)

Getting Started

Now let’s look at step by step instructions on how to call Amazon MWS API and then import into Power BI. In below steps we will use  ZappySys ODBC PowerPack so make sure it’s installed first.

Using ScratchPad to Test Amazon MWS API

Each MWS API has set of required and optional parameters. You can refer each API page for details. For example ListMatchingProducts API which we will use as an example. To make MWS API testing / learning easy, Amazon provides online testing tool called MWS ScratchPad. We will use this tool to craft API requests and use that information in ODBC driver SQL queries and later import data in Power BI.

1. Open MWS ScratchPad  by vising https://mws.amazonservices.com/scratchpad/index.html
2. Select Products API category and pick ListMatchingProducts API
3. Enter your SellerID (i.e. Merchant ID), Developer AccessKey (i.e. AWSAccessKeyId) and Secret Key we obtained in the previous section. Enter MarketId (e.g. ATVPDKIKX0DER for USA Market) and other parameters as below and Click Submit to see response.
   

   Using Amazon MWS ScratchPad (API Testing Tool – Created by Amazon)

4. Click on the Response Details Tab to extract some important information we will use for ODBC configuration. Notice attributes in red rectangles. They are the ones which we will need in the POST request Body. ZappySys Drivers supply many common parameters automatically but any API endpoint specific parameter must be supplied in the Request Body. For example you dont have to supply SignatureVersion, Timestamp, Signature or SignatureMethod
   

   Calling Amazon MWS API in Scratchpad Testing Tool (Response Details Tab) – ListMatchingProducts Example

This API requires Market ID parameter so enter it as per your Market where you selling. For example USA marketplaceid = ATVPDKIKX0DER. You can lookup correct MarketplaceId by checking this help (endpoints / marketplaces).

Create ODBC DSN Connection for MWS API / Test Query

Once you have necessary details to call your Amazon MWS API, we can create ODBC DSN which will be used as a reusable connection for MWS API data access in apps like Power BI.

So let’s get started.

1. In your start menu search for ODBC and select Data Source (64 bit):  If you need to access DSN by Service or scheduled Job then make sure you choose 64-bit DSN if caller App is 64 bit App.
   

   Open odbc data source

2. Use the User DSN page and press Add. For DSN access by Service account or automated jobs which runs under different user, choose System DSN Tab, Make sure You create DSN for correct Bitness (i.e. for 64bit Apps create 64 Bit DSN)
   

   Add new data source

3. Select ZappySys XML Driver and click Finish. Amazon MWS API is XML format so we selected XML driver however you can choose JSON or CSV Driver instead of XML if you dealing with JSON format.
   

   Add new ZappySys XML Driver (For XML / SOAP API)

4. Once you see Driver Configuration UI window open, enter the following settings
5. ODBC DSN name (e.g. ZS – Amazon MWS API Connection)
6. Enter MWS API Service URL as below. Notice that we added /Products/2011-10-01 (This was extracted from previous section). You can check full list of Amazon MWS Endpoints here. Because we are connecting to USA region we will use below URL.
   

   https://mws.amazonservices.com/Products/2011-10-01

7. Select OAuth from the Connection dropdown and Click Configure Settings
8. On the Connection UI, Select OAuth Provider Amazon Market Web Service (MWS)
9. Enter your API Access Key and Secret Key (obtained from here)
10. Your connection will look like below.
    

    ODBC Connection Credentials for Amazon MWS API (Access Key, Secret Key, URL Endpoint)

11. Also configure Retry options to handle API limit reached error when you call API too often. For Amazon MWS we get Status code 503 when request is throttled so lets narrow down on which error we want to retry.
    

    Retry Options

12. Click OK to save OAuth Connection UI to go back to main UI
13. Now let’s use some information we gathered from the previous section (Using Scraptchpad Test).
14. Select POST for HTTP Request Method
15. Enter Body as below (Change last parameter with your own SellerID and if needed change your MarketPlaceId too if its not USA)
    

    Action=ListMatchingProducts&MarketplaceId=ATVPDKIKX0DER&Query=harry%20potter%20dvd&SellerId=ZZZZZ-YOUR-SELLER-ID-ZZZZZ

16. Click on Select Filter to select correct node (in our case it will be Product node as below.
    

    $.ListMatchingProductsResponse.ListMatchingProductsResult.Products.Product[*]

17. Now Go to Throttling Tab and enter some delay (e.g. enter 20000 for 20sec delay after each request). This will slow down API calls so we don’t face errors about API Limit reached.
18. You can also consider enabling Data Caching for MWS API as below to avoid API throttling / rate exceeded error. Change Cache Mode to Cache All (Use Cache for Metadata + Data Requests)
    
    NOTE: MWS may allow only certain number of requests per minute so you might want to use each call wisely. You can find Throttling information on API help page itself (like this one see throttling section. Also this one).
    

    Enable Caching Options for Metadata / Data for ZappySys ODBC Drivers

19. Now goto Preview Tab and click on Green Play button to test query
    

    Query Amazon MWS API using ZappySys XML Driver – Preview UI

20. That’s it. We have successfully tested first query to read data from Amazon MWS API

Caching Metadata and Using in SQL Query

We talked briefly about Data Caching but still there will be a time you will face such errors. So lets do another safeguard to reduce total API requests for MWS. Check this article to learn about Caching and META Clause in SQL.

Here are high level steps to Use predefined Schema File so Driver doesn’t call API request to guess metadata.

1. Go to ODBC Preview window and write desired query you like to use. Click Preview Button to see sample data
2. Now click Save Metadata of query which we can use later. Save to File in our case (name as mws-query-meta.txt ).
   

   Create Metadata for ODBC Driver Query – ZappySys API Drivers

3. Now change our query as below
   

   SELECT 
    ......
    ......
   FROM xxxxxxxxx
   WITH (META='c:\somefolder\mws-query-meta.txt')

4. That’s it. Use query like above when we import Amazon MWS data in Power BI in the next section.

Import Amazon MWS data in Power BI (List Orders Example)

Now it’s time to load our data in Power BI and create reports.

1. Open Power BI Desktop and select the option Get data
   

   Get data in power bi

2. Get Data will allow adding the ZappySys ODBC driver. Go to Other and select ODBC.
   

   select odbc in power bi

3. Now it’s time to import data. Basically, there are two modes to import data. Table Mode and Query Mode. Query mode is the most common way. Select your DSN and click Advanced Option to enter custom SQL Query to Import your MWS API data. You can use ODBC DSN Data sources Preview tool to generate SQL Query. For example you can enter query to import Orders for specified date range. We have added some pagination attributes which we will cover in the next section. Click OK to continue.
   

   SELECT * FROM $
   WITH(
   	 ElementsToTreatAsArray='Order'
   	,Src='https://mws.amazonservices.com/Orders/2013-09-01'
   	,Filter='$.ListOrders[$tag$]Response.ListOrders[$tag$]Result.Orders.Order[*]'
   	,RequestData='Action=ListOrders[$tag$]&SellerId=A1xxxxxxxxxxxxx&CreatedAfter=2018-09-01T04%3A00%3A00Z&MarketplaceId.Id.1=ATVPDKIKX0DER'
   	,RequestMethod='POST'
   	,NextUrlAttributeOrExpr='$.ListOrders[$tag$]Response.ListOrders[$tag$]Result.NextToken'
   	,NextUrlSuffix='&NextToken=<%nextlink_encoded%>'
   	,HasDifferentNextPageInfo='True'
   	,EnablePageTokenForBody='True'
   	,PagePlaceholders='body=|ByNextToken;filter=|ByNextToken'
   	,WaitTimeMs=300 -- API throttling for pagination
   	--Uncomment Caching Line to Speedup API calls and Avoid API Limit. Power BI issues many API calls behind the scene
   	--,CachingMode='All',CacheStorage='File',CacheEntryTtl=120,CacheFileLocation='c:\temp\mws-orders.cache'
   	--,DataConnectionType='OAuth'
   	--,ScopeSeparator='{space}'
   	--,ServiceProvider='AmazonMWS'
   	--,ClientId='AKIAIWxxxxxxxxxxxxx' --Skip this to use DSN credentials
   	--,ClientSecret='AKIAIWxxxxxxxxxxxxx' --Skip this to use DSN credentials
   	--,UseCustomApp='True'	
   )
   --Uncomment below if you are using Schema file for cached metadata
   --WITH (META='c:\somefolder\mws-query-meta.txt')
4. Use above query for example and paste in Query editor while you Import Amazon MWS data.
   

   Enter custom SQL to load SOAP Web Service Data in Power BI (ZappySys XML Driver)

5. On the next screen select Windows Authentication
   
   

   Select Authentication for Amazon MWS connection in Power BI data load

6. Once data is imported you can use dataset to build your dashboard
   

   Load Amazon MWS data in Power BI – Create Amazon Sales Dashboard

Using Pagination in Amazon MWS API calls

Now let’s look at how to query certain API which allows paginated response. Pagination is fully explained here. Certain APIs like ListOrders , ListOrderItems, ListInboundShipments, ListInventorySupply, GetFullfillmentOrder and few more supports Pagination.

When you call Paginated response next request changes Action with ByNextToken suffix. For example to get orders call ListOrders and next request will be ListOrdersByNextToken

MWS API has multiple pagination patterns so based on which action you calling you have to use correct SQL parameters.

Using NextToken for Pagination (ListOrders Example)

Many MWS API returns NextToken attribute in the response which you can pass in the body of next request until no more NextToken found in the response.

Check below example query for ListOrders API. We have enabled pagination by supplying few extra parameters in our original SQL Query.

SELECT * FROM $
WITH(
	 ElementsToTreatAsArray='Order'
	,Src='https://mws.amazonservices.com/Orders/2013-09-01'
	,Filter='$.ListOrders[$tag$]Response.ListOrders[$tag$]Result.Orders.Order[*]'
	,RequestData='Action=ListOrders[$tag$]&SellerId=A10JBWAX563WMC&CreatedAfter=2018-09-01T04%3A00%3A00Z&MarketplaceId.Id.1=ATVPDKIKX0DER'
	,RequestMethod='POST'
	,NextUrlAttributeOrExpr='$.ListOrders[$tag$]Response.ListOrders[$tag$]Result.NextToken'
	,NextUrlSuffix='&NextToken=<%nextlink_encoded%>'
	,WaitTimeMs=300 -- slow down to avoid throttling
	,HasDifferentNextPageInfo='True'
	,EnablePageTokenForBody='True'
	,PagePlaceholders='body=|ByNextToken;filter=|ByNextToken'
	--,DataConnectionType='OAuth'
	--,ScopeSeparator='{space}'
	--,ServiceProvider='AmazonMWS'
	--,ClientId='AKIxxxxxx'
	--,ClientSecret='AKIxxxxxx'
	--,UseCustomApp='True'	
)
--Uncomment below if you are using Schema file for cached metadata 
--WITH (META='c:\somefolder\mws-query-meta.txt')

Using NextToken with operations that return HasNext / MoreResultsAvailable

Some API has attribute which tells you to stop pagination (e.g. HasNext). Check below example query for GetReportRequestList. We have enabled pagination by supplying few extra parameters in SQL Query.

SELECT * FROM $
WITH(
	 ElementsToTreatAsArray='Product,ReportRequestInfo'
	,Src='https://mws.amazonservices.com'
	--,DataConnectionType='OAuth'
	--,ScopeSeparator='{space}'
	--,ServiceProvider='AmazonMWS'
	--,ClientId='Axxxxxxxxxxxxxx' --This will come from connection if you comment
	--,ClientSecret='Bxxxxxxxxxxxxxxxxx' --This will come from connection you comment
	--,UseCustomApp='True'
	,Filter='$.GetReportRequestList[$tag$]Response.GetReportRequestList[$tag$]Result.ReportRequestInfo[*]'
	,RequestData='Action=GetReportRequestList[$tag$]&Merchant=A10xxxxxxxxxxxxxxxx'
	,RequestMethod='POST'
	,NextUrlAttributeOrExpr='$.GetReportRequestList[$tag$]Response.GetReportRequestList[$tag$]Result.NextToken'
	,NextUrlEndIndicator='false'
	,StopIndicatorAttributeOrExpr='$.GetReportRequestList[$tag$]Response.GetReportRequestList[$tag$]Result.HasNext'
	,NextUrlSuffix='&NextToken=<%nextlink_encoded%>'
	,HasDifferentNextPageInfo='True'
	,EnablePageTokenForBody='True'
	,PagePlaceholders='body=|ByNextToken;filter=|ByNextToken'
)
--Uncomment below if you are using Schema file for cached metadata 
--WITH (META='c:\somefolder\mws-query-meta.txt')

Some API calls return slightly different response so rather than HasNext attribute you will see MoreResultsAvailable. In that case change StopIndicatorAttributeOrExpr property to use MoreResultsAvailable property.

Properties you need to change for pagination

Now let’s look at properties which need to be changed to support pagination.

POST /Orders/2013-09-01?AWSAccessKeyId=AKxxxxxxxxxx
  &Action=ListOrders
  &SellerId=A10zzzzzzz
  &SignatureVersion=2
  &Timestamp=2018-09-17T21%3A35%3A57Z
  &Version=2013-09-01
  &Signature=3GigipfRsQgzzzzzzzzzzzzzs%3D
  &SignatureMethod=HmacSHA256

Action=ListOrders[$tag$]&SellerId=A10zzzzzzz&CreatedAfter=2018-09-01T04%3A00%3A00Z

<?xml version="1.0"?>
<ListOrdersResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

<?xml version="1.0"?>
<ListOrdersByNextTokenResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersByNextTokenResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

Here is the preview of MWS API query output with Pagination Attributes.

Amazon MWS API Pagination Example – Preview SQL Query Result

Import data from MWS Custom Reports (Inventory Report Example)

Now lets look at how to extract data from Custom Report API (You must need ODBC PowerPack v1.1.1 or higher). However reading data from Reports not single step process because report generation is Job style API. Which means you send report request and wait  until its done. Once Report ready you have to read in CSV format or XML. Some Reports only available in CSV format. Check this post to understand how complex it can be to get data. We will make it simple for you to understand this in 3 steps. Basically calling reports requires minimum 3 API calls (see below)

1. Create a new report request – Cal RequestReport API . It returns ReportRequestId (You can use it in the next step)
2. Check Report Status see its done – Call GetReportRequestList API (Pass ReportRequestId got in the previous step to get data for only one Report request we care). Keep checking Report Status every few seconds until you get _DONE_ status in response.
3. Read Report Data – Call GetReport. This API call returns CSV or XML data. So use correct Driver for this step. If CSV data is returned then we must use CSV Driver rather than XML Driver.

Now lets see how to get this 3 steps done in Power BI. In this example, We will get Inventory by calling _GET_MERCHANT_LISTINGS_DATA_ report type.

1. Create a directory c:\temp
2. Create new ZappySys XML Driver DSN (User DSN) using same steps we described before. Lets name this DSN as ZAPPY-MWS-API-XML
3. Create new ZappySys CSV Driver DSN  (User DSN) using same steps we described before (Except the Driver Type should be ZappySys CSV Driver). Lets name this DSN as ZAPPY-MWS-API-CSV
4. Once you done with ODBC DSN Setup open or create new Power BI Project.
5. Click on Create Source > from Blank Query
   

   Power BI – Create New Data source from Query (M Script)

6. When Query Editor Shows up click on Advanced Editor and paste below code (Its Power BI M Language). If you are unsure how to write Power BI check this tutorial.
   Replace MarketPlaceId and Seller ID in the script (i.e. A1xxxxxx and A2xxxxxx ). See we are using ReportType=_GET_MERCHANT_LISTINGS_DATA_  but you can use any valid report ID from here. You can also pass other optional parameter available for each report Type (e.g. StartDate and EndDate)
   

   Edit Power BI Query Script – Use custom M Language Script to import data from ODBC or other Relational

   Power BI M Script (New Driver) – Import MWS Custom Report in Power BI
   

   //////////////////////////////////////////////////////////////////////////
   // Change _GET_MERCHANT_LISTINGS_DATA_ to something else if you need different report type (all 3 steps) -- https://docs.developer.amazonservices.com/en_US/reports/Reports_ReportType.html
   // Replace MarketplaceId in all 3 steps (i.e for USA use ATVPDKIKX0DER) --https://docs.developer.amazonservices.com/en_US/dev_guide/DG_Endpoints.html
   // Replace SellerId in all 3 steps
   // Remove or Change CreatedAfter in first step (or use CreatedBefore parameter instread of CreatedAfter)
   //////////////////////////////////////////////////////////////////////////
   
   let
       sql="SELECT s3.* FROM $
   WITH(
   Src='https://mws.amazonservices.com/Reports/2009-01-01',RequestMethod='POST'
   ,RequestData='Action=RequestReport&ReportType=_GET_MERCHANT_LISTINGS_DATA_&MarketplaceId=ATVPDKIKX0DER&SellerId=Axxxxxxxxxx&CreatedAfter=2017-12-31T04%3A00%3A00Z'
   ,IncludeParentColumns='False'
   ,EnableRawOutputModeSingleRow='True'
   ,RawOutputFilterExpr='//*[local-name()=""ReportRequestId""]' 
   ,RawOutputDataRowTemplate='{ReportRequestId:""[$1]""}' 
   ,RawOutputExtractMode='Xml' 
   ,WaitTimeMs=5000
   ,Meta='[{""Name"": ""ReportRequestId"",""Type"": ""Int64""}]'
   ,Alias='s1'
   
   ,Join1_alias='s2'	
   ,Join1_Src='https://mws.amazonservices.com/Reports/2009-01-01',Join1_RequestMethod='POST'
   ,Join1_RequestData='Action=GetReportRequestList&ReportRequestIdList.Id.1=[$s1.ReportRequestId$]&MarketplaceId=ATVPDKIKX0DER&SellerId=Axxxxxxxxxx'
   ,Join1_EnableRawOutputModeSingleRow='True'
   ,Join1_RawOutputFilterExpr='//*[local-name()=""ReportProcessingStatus""]||//*[local-name()=""GeneratedReportId""]' 
   ,Join1_RawOutputDataRowTemplate='{ReportProcessingStatus:""[$1]"",GeneratedReportId:""[$2]""}' 
   ,Join1_RawOutputExtractMode='Xml' 
   ,Join1_EnableStatusCheck='True', 
   ,Join1_StatucCheckMaxWaitSeconds=300,
   ,Join1_StatucCheckIterationWaitSeconds=25,
   ,Join1_StatusSuccessValue='_DONE_'
   ,Join1_StatusFailedValue='_DONE_NO_DATA_|_CANCELLED_'
   
   ,Join2_Alias='s3'
   ,Join2_Src='https://mws.amazonservices.com/Reports/2009-01-01',Join2_RequestMethod='POST'
   ,Join2_RequestData='Action=GetReport&ReportId=[$s2.GeneratedReportId$]&MarketplaceId=ATVPDKIKX0DER&SellerId=Axxxxxxxxxx'	
   ,Join2_ColumnDelimiter='{TAB}'
   ,Join2_ResponseCharset='Windows-1252'	 
   ,Join2_EnableStatusCheck='False' 
   ,Join2_EnableRawOutputModeSingleRow='False'
   )",
       Source = Odbc.Query("dsn=ZS-AMAZON-MWS-CSV", sql)
   in
       Source

    

   Power BI M Script (Old Driver) – v1.1.2 or older

   If you are using older version – v1.1.2 or older then use below script.

   /*
    All Copy Rights Reserved - ZappySys LLC. Feel free to share this snippet as long as you retain copyrights notice. https://zappysys.com
    Description:  This snippet will call Amazon MWS API to generate Inventory Report (Its multi step process - Job Style API call. First 2 calls gets XML data and Last call CSV data)
   */
   
   let
       //NOTE: In Power BI M Language... each declare is separated by comma... 
       //variable can be any type.. string, number, table, list , record, function reference. More info: https://docs.microsoft.com/en-us/powerquery-m/power-query-m-reference
       /* 
       let 
           var1=expression, var2=expression,  ... varN=expression
       in
           varN   //return some variable
       */
   
       //define some common variables
   
       MarketplaceId="A1xxxxxx",
       SellerId="A2xxxxxxxxxx",
   
       //////////////////////////////////////////////////
       /*Step-1 : Send new report request  - Must use XML Driver */
       //////////////////////////////////////////////////
   
       Step1Sql="SELECT ReportRequestId FROM $
   WITH(
       --Src='c:\temp\reportrequest.txt'
       Src='https://mws.amazonservices.com/Reports/2009-01-01'
   	--,ElementsToTreatAsArray='Product'
   	,Filter='$.RequestReportResponse.RequestReportResult.ReportRequestInfo'
   	,RequestData='Action=RequestReport&ReportType=_GET_MERCHANT_LISTINGS_DATA_&MarketplaceId=" & MarketplaceId & "&SellerId=" & SellerId & "'
   	,RequestMethod='POST'
   	,WaitTimeMs='300'
   	,IncludeParentColumns='False'
   	,CachingMode='All'
   	,CacheStorage='File'
   	,CacheFileLocation='c:\temp\mws-step1.cache'
   	,CacheEntryTtl=300
   	,Meta='[{""Name"": ""ReportRequestId"",""Type"": ""Int64""}]'
   )",
       Source1 = Odbc.Query("dsn=ZAPPY-MWS-API-XML", Step1Sql),
       ReportRequestId= Source1{0}[ReportRequestId],
       
       //NOTE: Technically we need to implement status check loop Wait for few seconds before 
       //Increase this if you get ERROR in 3rd step
       BlankTextForDelay = Function.InvokeAfter(()=>"", #duration(0,0,0,15)), //define delay - You must use it somewhere inorder to invoke delay
   
       //////////////////////////////////////////////////
       /*Step-2 : Get report id - once report is done  - Must use XML Driver */
       //////////////////////////////////////////////////
       Step2Sql=BlankTextForDelay & "SELECT *  FROM $
   WITH(
       --Src='c:\temp\report-status.xml'
       Src='https://mws.amazonservices.com/Reports/2009-01-01'
   	,ElementsToTreatAsArray='ReportRequestInfo'
   	,Filter='$.GetReportRequestListResponse.GetReportRequestListResult.ReportRequestInfo'
   	,RequestData='Action=GetReportRequestList&ReportRequestIdList.Id.1=" &  ReportRequestId & "&MarketplaceId=" & MarketplaceId & "&SellerId=" & SellerId & "'
   	,RequestMethod='POST'
   	,WaitTimeMs='300'
   	,IncludeParentColumns='False'
   	,CachingMode='All'
   	,CacheStorage='File'
   	,CacheFileLocation='c:\temp\mws-step2.cache'
   	,CacheEntryTtl=15
   	,Meta='[{""Name"": ""ReportProcessingStatus"",""Type"": ""String"", Length: 100},{""Name"": ""GeneratedReportId"",""Type"": ""Int64""}]'
   )",    
       Source2 = Odbc.Query("dsn=ZAPPY-MWS-API-XML", Step2Sql),
       ReportId= Source2{0}[GeneratedReportId],
   
   
       //////////////////////////////////////////////////
       /*Step-3 : Get report data in CSV format - Must use CSV Driver */
       //////////////////////////////////////////////////
       Step3Sql="SELECT * FROM $
   WITH(
          --Src='c:\temp\inventory.csv.txt'
   	 Src='https://mws.amazonservices.com/Reports/2009-01-01'
   	,RequestData='Action=GetReport&ReportId=" &  ReportId & "&MarketplaceId=" & MarketplaceId & "&SellerId=" & SellerId & "'
   	,RequestMethod='POST'
   	,WaitTimeMs='300'
   	,ColumnDelimiter='{TAB}'
   	,CachingMode='All'
   	,CacheStorage='File'
   	,CacheFileLocation='c:\temp\mws-step3.cache'    
   	,ResponseCharset='Windows-1252'	    
   	)",    
       Source3 = Odbc.Query("dsn=AMAZON-MWS-CSV", Step3Sql)
   
   in
       Source3  //return Source3 (i.e. Table type variable)

    

7. Once done click OK.
8. You may be prompted few times to trust each Query we defined in M Script above. Also you may be prompted to select DSN Credentials (Just Select Windows Credentials)
   

   Power BI – Run Native Database Query Prompt

9. If you get prompted to select DSN Credentials then select as below (Click Windows).
   

   Select Authentication for Amazon MWS connection in Power BI data load

10. After few seconds (Depending on Delay we define in Script – default 15 sec) you will see data loaded.
11. Once data is loaded click Close & Apply and it will close the query editor and we will see main designer where you can see Dataset and Fields we just created.
    

    Loading Amazon MWS Reports data in Power BI (List Inventory Example using M Script)

Conclusion

In this article, we saw how to build sales report by reading data from your Amazon Seller Account. We used simple SQL queries with ZappySys XML Driver to pull data from Amazon MWS API. ZappySys ODBC PowerPack makes it super easy to consume data from virtually any API (JSON / XML or CSV Web Services), no coding required. Download here to get started with ZappySys ODBC drivers.

The post How to Import Amazon MWS data in Power BI appeared first on ZappySys Blog.

In this post you will learn how to call Amazon MWS API (Amazon Marketplace Web Service) or Amazon Product Advertising API using SSIS PowerPack.

Using drag and drop approach you can consume data from Amazon MWS XML Web service. In this post we will use ZappySys XML Source connector to read data from Amazon MWS API and load into SQL Server. We will also use REST API Task to call any API from Marketplace Web service and save output into Variable.

Prerequisites

This post assumes following things

1. Basic knowledge of SSIS and XML format.
2. Amazon MWS Account and and valid AWSAccessKeyId and Secret Key to call Amazon MWS API (How to obtain Key / Secret and SellerId ? ).
3. You have tested few API calls using Amazon MWS Scratchpad this will give you idea about various API you can call and parameters you need to pass.

Testing MWS Requests using ScratchPad

Very first step we recommend before you call MWS API in SSIS is to get familiar with Amazon MWS Scratchpad tool. Check this article for detailed steps.  You can also use Fiddler to debug some requests.

Calling Amazon MWS API in Scratchpad Testing Tool (Response Details Tab) – ListMatchingProducts Example

Call Amazon MWS API using SSIS

To consume data from any XML Source (File, SOAP Web Service or XML API) you can use  ZappySys XML Source
connector. or REST API Task. In below example you will see how to call Amazon MWS API and save output into Variable. With this approach you can call any API (GET / POST / DELETE / PUT)

1. Download and Install SSIS PowerPack
2. Create test package
3. From SSIS toolbox drag ZS REST API Task  and double click on the task to configure it
4. Select Request URL Access Mode = [Url from Connection]
5. Enter URL as below
   

   https://mws.amazonservices.com/Orders/2013-09-01

   In above URL
   ** mws.amazonservices.com => This is your endpoint. If you are not in USA then Click here to find correct endpoint
   ** /Orders => That’s your API you want to call
   ** /2013-09-01 => That’s your API version (You have to add this same version in your POST Body too (see next section).
6. In Select Url Connection dropdown select New OAUTH connection option.
7. When prompted on OAuth connection UI select Provider=Amazon MWS, Enter AWSAccessKey and Secret key and Click OK (If you have MWSAuthToken then enter in Access Token field else leave it empty). Do not test because it may not work yet.
   

   SSIS OAuth Connection – Call Amazon MWS API (Marketplace Web service API)

8. Another Setting you may turn on is Retry Handling. This allows to over come ThresholdLimitReached error when we call API too fast.
   

   Retry Options

9. Now select Method=POST, and in the body enter following
   

   Action=GetServiceStatus&SellerId=A77XXXXXXX&Version=2013-09-01

   If your API requires pagination (more than 100 records from response) then refer next section (Sample Package). In the case of Pagination your second request may look like below
   

   Action=YourActionNameWithNextToken&SellerId=A77XXXXXXX&Version=2013-09-01&NextToken={{User::vNextToken,FUN_URLENC}}

   NOTE: Enter all required parameters in the following format, as shown below. All values need to be URL-encoded. If you use SSIS variable, use it along with FUN_URLENC format specifier, e.g. {{User::myVariable,FUN_URLENC}}. If you use a hard-coded value, and not sure if it is URL-encoded, then encode it with the same FUN_URLENC format specifier/function, e.g. <<myValue,FUN_URLENC>>. Always check if SSIS configuration matches the one you see in MWS ScratchPad Request Details.
   Format:
   param1=value1&param2=<<value2,FUN_URLENC>>&param3={{User::myVar,FUN_URLENC}}
10. Now click Test and see content in the Preview window.
    

    SSIS REST API Task – Call Amazon MWS API – Get XML data save into SSIS variable

11. If you wish to save response data into SSIS variable then goto response Tab and check save option and select variable name.

Amazon MWS API Pagination

Many Amazon MWS API calls by default don’t return all records (Explained here). Rather than returning all rows you may get partial response (e.g. Max 50 or 100 rows). If you want to get all records then you have to use NextToken  found in your XML. For API which requires pagination, in that case your POST body may be different for 2nd or higher requests. Notice how we changed Action and appended NextToken=xxxxxxxxxxx in the second request. Also notice we used encoded token (URL encoded). This is very important because as per Amazon Specs this value has to be Url Encoded.

We also documented similar use case of Amazon MWS Pagination for Power BI (Check this one) its mostly same setup except 2-3 properties might be named different way.

Now lets look at sample request / response for paginated MWS requests (For clarity we have removed many common Parameters which are automatically added at runtime e.g. Signature, Timestamp). We use ListOrders (First Request) and ListOrderByNextToken (Second+ requests)

First Request

POST https://mws.amazonservices.com/Orders/2013-09-01

Action=ListOrders&SellerId=A10zzzzzzz&CreatedAfter=2018-09-01T04%3A00%3A00Z

First Response

<?xml version="1.0"?>
<ListOrdersResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

Second Request or higher (MWS API with Pagination using NextToken)

POST https://mws.amazonservices.com/Orders/2013-09-01

Action=ListOrdersByNextToken&SellerId=A10zzzzzzz&CreatedAfter=2018-09-01T04%3A00%3A00Z&NextToken=YOUR_ENCODED_TOKEN_FROM_PREV_REQUEST

Second or higher Response

<?xml version="1.0"?>
<ListOrdersByNextTokenResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersByNextTokenResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

Sample SSIS Package

If you want to see complete working package for pagination pattern then download from here. Change AccessKey, SecretKey, SellerID and Path as needed to make this package work.

Here is the screenshot of example SSIS package for Amazon MWS.

Amazon MWS APi Pagination (ListOrders , ListOrdersByNextToken Example)

Copy/Paste Properties

Here is copy/paste values for properties listed above.

DirectPath:  https://mws.amazonservices.com/Orders/2013-09-01
EnablePageTokenForBody: True
Filter:  $.ListOrders[$tag$]Response.ListOrders[$tag$]Result.Orders.Order[*]
HasDifferentNextPageInfo: True
HttpRequestData:  Action=ListOrders[$tag$]&SellerId=A1xxxxxxxx&CreatedAfter=2018-09-01T04%3A00%3A00Z&MarketplaceId.Id.1=ATVPDKIKX0DER
HttpRequestMethod:  POST
NextUrlAttribute: $.ListOrders[$tag$]Response.ListOrders[$tag$]Result.NextToken
NextUrlSuffix: &NextToken=<%nextlink_encoded%>
NetUrlWait: 10000 (ms) -- slow down to avoid throttling ... Max 6 requests per minute, increase if you get API Limit Error
PagePlaceholders:  body=|ByNextToken;filter=|ByNextToken

Understanding Pagination Properties

Here are various properties you have to set for Pagination.

POST /Orders/2013-09-01?AWSAccessKeyId=AKxxxxxxxxxx
  &Action=ListOrders
  &SellerId=A10zzzzzzz
  &SignatureVersion=2
  &Timestamp=2018-09-17T21%3A35%3A57Z
  &Version=2013-09-01
  &Signature=3GigipfRsQgzzzzzzzzzzzzzs%3D
  &SignatureMethod=HmacSHA256

Action=ListOrders[$tag$]&SellerId=A10zzzzzzz&CreatedAfter=2018-09-01T04%3A00%3A00Z

<?xml version="1.0"?>
<ListOrdersResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

<?xml version="1.0"?>
<ListOrdersByNextTokenResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersByNextTokenResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

NextUrlEndIndicator=false
StopIndicatorAttributeOrExpr= $.GetReportRequestList[$tag$]Response.GetReportRequestList[$tag$]Result.HasNext

Amazon MWS API Pagination (For Old Version)

So in previous section we saw how to achieve pagination in one step (for newer version of SSIS PowerPack). But there will be a case you cannot use latest SSIS PowerPack or you want to take complete control on each aspect of pagination. In such case use below method (Depreciated).  In newer version we added many new properties so you dont have to do Loop pattern like below.

Sample SSIS Package

If you want to see complete working package for pagination pattern then download from here. Change AccessKey, SecretKey, SellerID and Path as needed to make this package work.

Here is the screenshot of example SSIS package for Amazon MWS.

SSIS Example Package – Read data from Amazon MWS Web service (API Call). Paginate API calls using NextToken

Load Amazon MWS API data into SQL Server using SSIS

In this section we will see how to use ZappySys XML Source connector to read data from Amazon MWS Web Service and load into SQL Server

For making things simple we are calling

1. Download and Install SSIS PowerPack
2. Create test package
3. From SSIS toolbox drag Data Flow task and double click task to go to Data Flow designer
4. Drag ZS XML Source from SSIS Toolbox
5. Configure XML Source as below
   

   SSIS XML Source – Read data from Amazon MWS API (Amazon Marketplace Web service call)

6. Drag OLEDB destination from SSIS Toolbox.
7. Connect XML Source to OLEDB Destination and map input column to target table
8. Execute dataflow
   

   Get data from Amazon MWS API – Save to File or SQL Server using SSIS

Working with Amazon MWS Feed API (e.g. Upload file)

Amazon MWS Provides Feed APIs which has slight different requirements. Important requirement in Feed file upload is you have to supply Content MD5 Hash. If you use SSIS PowerPack then you don’t have to worry about that complexity because calculating MD5 Hash is automatically done.

Upload Amazon MWS Feed File in TSV / CSV format (Tab separated)

Here is the screenshot for how to upload Tab separated values in CSV file. You can specify body from Variable or Read from file. If you read from file then must check File Upload/Multi-Part option and start path with @ symbol as below.

Call Amazon MWS Feed API – Upload CSV file format (Tab separated values – TSV)

Upload Amazon MWS Feed File in XML format

To upload feed file in XML format use same settings as above except Content Type should be XML (text/xml) (Select in the dropdown)

Calling Amazon Product Advertising API

If you have need to call Amazon Product Advertising API then also you can use above techniques (Use MWS Provider on OAuth connection) to call Product Advertise API. Both API (MWS API and Product Advertise API) use same signature hashing algorithm so its possible to use MWS Provider for Hashing. You may have to change your API URL parameters as needed when you call Amazon Product Advertising API. Basically when you call any MWS or Product Advertising API using GET method then OAuth connection manager automatically appends Signature to URL at runtime.

Import data from MWS Custom Reports (Inventory Report Example)

Now lets look at how to extract data from Custom Report API . Reading data from Custom Reports not single step process because report generation is Job style API. Which means you send report request and wait  until its done. Once Report ready you have to read in CSV format or XML. Some Reports only available in CSV format. Check this post to understand how complex it can be to get data. We will make it simple for you to understand this in 3 steps. Basically calling reports requires minimum 3 API calls (see below)

1. Create a new report request – Cal RequestReport API . It returns ReportRequestId (You can use it in the next step)
2. Check Report Status see its done – Call GetReportRequestList API (Pass ReportRequestId got in the previous step to get data for only one Report request we care). Keep checking Report Status every few seconds until you get _DONE_ status in response.
3. Read Report Data – Call GetReport. This API call returns CSV or XML data. So use correct component for this step. If CSV data is returned then we must use CSV Source rather than XML Driver.

Now lets see how to read Amazon MWS custom report in SSIS. In this example, We will get Inventory by calling _GET_MERCHANT_LISTINGS_DATA_ report type.

Download Sample Package for MWS Report

To make things simple we have created a demo package – ready to go.
Download SSIS 2012 Demo – Amazon_MWS_API_GetReport_2012.

Prepare demo package for first run

Once you download and extract above sample package. Import to your existing SSIS Solution. Its SSIS 2012 format so it may upgrade to higher version if you using SSIS 2014, 2016 or 2017.

Once package is opened perform following steps in before you can execute the package.

1. Enter MarketplaceId, SellerId in Variables. Double click OAuth Connection and enter your MWS Account Key and Secret.
2. Create c:\temp directory (This is where we will save final report file)
3. Run this package
4. Go to Execution log and Find ReportId (This you can hardcode in Variables for Design time Preview / Metadata changes etc)

See below package screenshot

Read data from Amazon MWS Report in SSIS (Get Inventory Listing Example)

Steps Explained

Now lets see each steps in depth. We broken whole process in 3 groups.

Create Report Request (Step1)

This is the first step how you initiate report request.  Use REST API Task to call this step. Here is raw request. Many items in body added at runtime (e.g. Signature, Timestamp).

• On Request tab see how we used SSIS Variables to get many values at runtime dynamically.
• On Response Tab we have to extract RequestId and save into variable.

POST 
https://mws.amazonservices.com/Reports/2009-01-01

AWSAccessKeyId=Axxxxxxxxx&Action=RequestReport&MarketplaceId=ATVPDKIKX0DER&ReportType=_GET_MERCHANT_LISTINGS_DATA_&SellerId=Axxxxxxxxx&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2019-06-05T21%3A23%3A31.090Z&Signature=2AbGQQ4xxxxxxxxxxxxxxxxxxxxjJzox%2FU%3D

RequestReport Action – Generate Amazon MWS Custom Report

Extract value from XML response using XPATH and save to Variable (Get RequestId Example)

Here is how to configure above task

Wait until data is ready (Step2)

Once we send RepotRequest and we have RequestId we can call step2 using REST API Task. In this Task we can use Status Check Feature. This is pretty handy feature because we dont have to implement Loop in SSIS. Basically we have to keep checking report until we get _DONE_ status in the response (ReportProcessingStatus node in XML)

Here is how to configure the task.

1. Request Tab Configuration.
   URL:

   https://mws.amazonservices.com/Reports/2009-01-01

   Method: POST
   Body:

   Action=GetReportRequestList&ReportRequestIdList.Id.1={{User::ReportRequestId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}}

2. Response Setting Tab Configuration
   Response content type: Xml
   Filter Expression: 

   //*[local-name() = 'ReportProcessingStatus']  

3. Status Check Tab
   Check Enable Status
   Success Value: 

   _DONE_

   Check Every 5 Seconds

REST API Task – Status Check Feature (Wait until data is ready loop)

GetReportId (Step3)

Once above task finished we can call same request again but this time we have to configure slightly different way.

Here is how to configure the task.

1. Request Tab Configuration.
   URL:

   https://mws.amazonservices.com/Reports/2009-01-01

   Method: POST
   Body:

   Action=GetReportRequestList&ReportRequestIdList.Id.1={{User::ReportRequestId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}}

2. Response Setting Tab Configuration
   Response content type: Xml
   Filter Expression: 

   //*[local-name() = 'GeneratedReportId']  

   Check Save to Variable: Save to User::ReportId
3. Status Check Tab
   Make Enable Status is unchecked

Get Report Data  (Step4,Step5 or Step6)

Now we are ready to ready our report. There are two ways we can do in SSIS.

• First approach (step 4,5) is download the report and save to local disk and then in the next step we can use data flow (CSV Source) to parse the saved file.
• Second approach (step6) is rather than saving to disk we can directly call API using CSV File source and parse content that way. Lets talk each approach briefly.

Step4,5 – Load Report From File

If you wish to save Report on disk then you can use REST API Task and call request like below.

Here is how to configure the task to download the file.

1. Request Tab Configuration.
   URL:

   https://mws.amazonservices.com/Reports/2009-01-01

   Method: POST
   Body: Action=GetReport&ReportId={{User::ReportId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}}
2. Response Setting Tab Configuration
   Response content type: Xml
   Filter Expression: 

   //*[local-name() = 'GeneratedReportId']  

   Check Save to Variable: Save to User::ReportId
   
3. On Response Settings Tab : Set Response charset to Western European [Windows 1252]  (or directly set via Properties Grid to Windows-1252 under ResponseCharset Property)
   
4. Status Check Tab
   Make Enable Status is unchecked

To read download CSV file use settings described in below section except URL change it with local file path.

Step6 – Load report from URL

If you wish to load directly from URL then use same setting as above but in CSV Source inside data flow.

Here is how to configure CSV Source.

1. Request Tab Configuration.
   URL:

   https://mws.amazonservices.com/Reports/2009-01-01

   Method: POST
   Body:

   Action=GetReport&ReportId={{User::ReportId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}} 

2. General Tab
   Column Delimiter: {TAB}
3. On Encoding Tab : Set to Western European [Windows 1252]  (or directly set via Properties Grid to Windows-1252 under CharacterSet Property)

Conclusion

Amazon Marketplace Web service (MWS) provides great way to automate many functionality for Marketplace Seller. Using ZappySys SSIS PowerPack you con consume data from any REST API or Web service without replying on SDK / coding approach (e.g. C#, Java, Python, Ruby).

The post How to call Amazon MWS API using SSIS appeared first on ZappySys Blog.