In this article, we will see how to read Walmart API data in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call Walmart APIs can be useful for any developer regardless of which programming language or toolset you use.

Are you a Walmart.com Supplier or Marketplace Seller? Then you may need to click here to visit the site that is focused on these profiles.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of Walmart API. Click here to create API.

Components Mentioned in this article

	ZS Logging Task
	JSON Source(REST API or File)

What is Walmart?

Walmart is the world’s largest retailer, and the Walmart Open API provides access to our extensive product catalog, thus enabling digital distribution partners to earn substantial affiliate revenues from customer referrals.

Read Walmart API Data using SSIS JSON Source

Let’s start with an example. We use SSIS JSON Source component to make the call to Walmart API, we will read Walmart data and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

Get Dynamic Token (Configure ZS-HTTP Connection)

To get API response data using SOAP / REST API call, you need to get an API Token from your login request URL. Follow the steps mentioned below to get API Token:

1. Now, Right Click on Connection Managers Window, and one Context menu appears for creating a new connection manager.
   

   Create a New HTTP Connection.

2. Select ZS HTTP Connection Manager from the list of Connection Managers and, double-click on it or click on Add button to add a new Connection Manager. By Default, The created connection manager is of Package-Level. You can set Project Level Connection manager too If you want.
   

   Select ZS-HTTP Connection from SSIS Connection Manager.

3. In the SSIS ZS HTTP Connection Manager, Go to General Tab and Configure HTTP Connection Manager. Enter API URL, Set Credentials Type to Dynamic Token – Two-Step Auth, and configure some basic settings for the same. We used Login URL as just example but you can enter any valid API URL endpoint here. Actual Login URL goes on Dynamic Token Tab (see next step).
   

   https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20

   WM_SEC.ACCESS_TOKEN

   

   ZS HTTP Connection – Configure General Tab

4. Now, In the Dynamic Token Tab do configure like below image. Please click here to refer about Walmart API Token.
   

   https://marketplace.walmartapis.com/v3/token

   grant_type=client_credentials

   Accept: application/json
   Cache-Control: no-cache
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>
   Authorization: Basic <<[$userid$]:[$password$],FUN_BASE64ENC>>

   

   ZS HTTP Connection – Dynamic Token Tab

5. On the Response Settings tab configure as below
   

   Walmart API – Extract Token – Response Settings

   1. Extract Mode = Extract from Body
   2. Expression Type = Json
   3. Response Content Filter =

      $.access_token

6. Click on OK button to save HTTP Connection configure settings.

That’s it we have successfully configured Connection for Walmart API in SSIS. In the next section, we will see how to use this connection and read various data from Walmart API.

Read Walmart API Data in SSIS

Once we have done creating HTTP Connection Manager we can move forward to read Walmart API data inside Data Flow. So lets Configure SSIS JSON / REST API Source.

1. Now, Drag and drop Data Flow Task from SSIS Toolbox in the currently open project and double click it to edit.
   
   

   Drag and Drop Data Flow Task.

2. Double click on the Data Flow task to see the Data Flow designer surface.
3. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.
   

   Drag and Drop JSON Source(REST API or File).

4. Double click JSON Source and enter the following URL as below.
   

   https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20

5. Check Use Credentials and select existing HTTP connection we have created already it.
6. Enter Following HTTP Headers. If you prefer Raw edit .. Just click on Raw Edit (#6 in screenshot) and paste below headers. Make sure to replace YourClientIdGoesHere and YourSecretGoesHere
   

   Content-Type: application/xml
   Accept: application/xml
   Authorization: Basic <<YourClientIdGoesHere:YourSecretGoesHere,FUN_BASE64ENC>>
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>

    
7. Select Array Filter (ItemResponse node) or type $.ItemResponse[*]  as below.
   

   ZS JSON Source – Configure

8. That’s it, you are ready to load Wallmart API Data into SQL Server and you can use more ZappySys Destination components.
9. For pagination call please click here for reference to this link.

Load Walmart API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Walmart API integration using ODBC Driver (Power BI, Excel, MS Access)

So far we have seen how to access Walmart API using SSIS. However there will be a time when you like to access API data inside other Apps using ODBC Driver. ZappySys Provides JSON/XML/CSV drivers for virtually any API including Walmart API. Here is how to access Walmart API using XML ODBC Driver.

1. Type “ODBC” in Start menu. Open ODBC Data sources (64 bit)
   
2. Click Add on User DSN tab (if you need to use DSN under different Account …i.e. Service Account then click on System DSN Tab)
3. Select ZappySys XML Driver
4. When New DSN UI shows up, Change DSN name to anything you like (e.g. WalmartAPI)
5. Click on Load ConnectionString button and enter below connection. Make sure to replace few things
   First ######YourClientId####### with your own clientid
   Second ######YourSecret####### with your own secret
   Third ######Base64OfClientIdAndSecret####### with base64 encoded string of

   clientId:Secret

       (both separated with colon). You can use any site like this one to generate base64 string of your clientid and secret (seperated by “:”)
   

   DRIVER={ZappySys XML Driver};
   ElementsToTreatAsArray='ns2:ItemResponse';
   DataPath='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20';
   DataConnectionType=HTTP;
   AuthScheme='{none}';
   Url='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20';
   TokenUrl='https://marketplace.walmartapis.com/v3/token';
   TokenRequestData='grant_type=client_credentials';
   TokenRequestMethod='POST';
   TokenResponseContentFilter='$.access_token';
   TokenRequestHeaders='Accept:application/json||Cache-Control:no-cache||WM_SVC.NAME:Walmart Marketplace||WM_QOS.CORRELATION_ID:TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>||Authorization:Basic <<[$userid$]:[$password$],FUN_BASE64ENC>>';
   TokenAuthHeader='WM_SEC.ACCESS_TOKEN';
   TokenResponseContentType=Json;
   UserName='######YourClientId#######';
   CredentialType=TokenDynamic;
   Password='######YourSecret#######';
   Filter='$.ns2:ItemResponses.ns2:ItemResponse[*]';
   RequestMethod='GET';
   RequestHeaders='Content-Type: application/xml || Accept: application/xml || Authorization: Basic ######Base64OfClientIdAndSecret####### || WM_SVC.NAME: Walmart Marketplace || WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>'

   

   ZappySys Driver – Load Connection String

    

6. Now go to preview tab and enter below query and click Run below SQL.

   select * from $

   –OR–
   NOTE: Change YourClientId and YourSecret in below sql before running

   SELECT * FROM $
   WITH(
   Src='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20'
   ,Header='Content-Type: application/xml
   Accept: application/xml
   Authorization: Basic <<YourClientid:YourSecret,FUN_BASE64ENC>>
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>'
   )

    

Walmart API Integration with Other BI Tools (Power BI, Excel, SSRS, MS Access…)

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

How to encrypt Authorization Token as Base64Encode

1. If you want to directly pass Authorization header Token as Base64Encode(clientId:clientSecret) encrypted in the get token call, you can encrypted it using ZS Logging task and used it. Please refer to this link for the same.
2. In visual studio just Drag and Drop ZS Logging Task in the design panel.
   

   Drag and Drop ZS Logging Task

3. Double click on ZS Logging Task to configure it.
4. In the ZS Logging Task configure like the following image.
   

   Logging Task : Base64Enocde

5. That’s all, Save Base64Encoded token for use in HTTP Connection.

Conclusion

After all, we saw you how to extract information from REST API such as Walmart REST API using HTTP Connection and load into SQL Server. We also learned techniques like How to get Dynamic Token using HTTP Connection. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• Help File: JSON Source(REST API or File), ZS Logging Task
• Walmart API: HTTP Connection

The post How to read Walmart API data in SSIS / ODBC appeared first on ZappySys Blog.

Any time you send HTTP Request according to HTTP Standard using tools like ZappySys REST API Drivers or SSIS Connectors you may receive web response along with HTTP Status Codes. Dealing with REST / SOAP API calls you may face errors and you may find it challenging to understand generic error message from server along with some Numeric value (Called HTTP Status Code).  In this article we will try to decode certain status codes and discuss various way to solve it.

Example of HTTP Error and Status Code

Before we dive deep on this topic of HTTP Errors Status Codes let’s check an example HTTP Error with Status Code 400 (BadRequest). As you notice we passed some bad data in Request Body to produce this error. As you may see in below screenshot there are few parts we need to configure to call REST API request.

1. URL  (and optionally parameters in URL)
2. Request Method  (i.e. GET / POST)
3. Request Body (Optional – Not needed for GET request but for POST /PUT API requests you may need to pass)
4. Request Content Type ( Optional – i.e. application/xml . This is not needed for GET Request)
5. Request Headers (Optional – Many tools generate default headers behind the scene. In our example we passed only two headers but see how many more added automatically in below Raw View obtained by tools like Fiddler )

Sample Request:

POST
https://accounts.google.com/o/oauth2/token

>>>> HEADERS <<<<<

User-Agent: ZappySysApp/1.0.2019.10310
Content-Type: application/xml
Accept: */*
Cache-Control: no-cache
Host: accounts.google.com
Content-Length: 13
Expect: 100-continue
Accept-Encoding: gzip, deflate
Connection: Keep-Alive

>>>> BODY <<<<<

some bad data

Sample Response:

HTTP/1.1 400 Bad Request

{
  "error": {
    "code": 400,
    "message": "Invalid JSON payload received. Unexpected token.\nsome bad data\n^",
    "status": "INVALID_ARGUMENT"
  }
}

Here is a screenshot of SSIS REST API Task which can be used for calling XML / JSON API to automate ETL workflows.

Example of Calling REST API which returns HTTP Error Status Code 400 (Bad Request) – Invalid POST Body / Syntax Error

Common HTTP Error Status Codes / Reasons / Possible Fix

Status codes can be anywhere between 100 to 505. Most of them documented here also here (RFC2616 – HTTP / 1.1). Out of these status codes we will discuss few most common ones in this article and how to fix them.

Microsoft has also provided great explanation for most common HTTP Status Code here.

Status Code 400 – Bad Request

Possible Reason(s):

• One or more URL Parameter is wrong
• Request body may be wrong (For example: Either data is invalid in Body or Syntax is invalid – such as JSON Property name misspelled or missing closing brackets)

Possible Fix:

• Make sure you passing URL Parameters correct way (Refer your API help file or some sample code)
• For POST / PUT request check your Parameter names, Data format, JSON / XML syntax (Refer your API help file or some sample code)
• If you want to validate JSON / XML Syntax use some FREE online validators for JSON  or XML

Status Code 401 – Unauthorized

Possible Reason(s):

• You did not pass userid / password or session token
• Your credentials are invalid or session token is expired
• Some tool may not save Sensitive Information unless you set Save Sensitive Information option. For example in SSIS if you set PackageProtectionLevel= DoNotSaveSensitive then password will not be sent along with your API request.

Possible Fix:

• Make sure you passing credentials either directly via HTTP Headers or URL Parameter
• For ZappySys Products use HTTP or OAuth connection manager to pass credentials securely without worrying about manual HTTP Headers
• If you configured PackageProtectionLevel= DoNotSaveSensitive in SSIS then Check this and this  article to learn how to pass sensitive information in SSIS at runtime / design time.

Status Code 404 – Page Not Found

Possible Reason(s):

• You entered invalid URL
• URL is intranet only and you trying to access from outside network

Possible Fix:

• Check your URL and correct it so its not throwing 404 error when you test in Browser (It must return data or throw some other error – E.g. 400 – Bad request or 401-Unauthorized)

Status Code 405 – Method Is Not Allowed

Possible Reason(s):

• You selected invalid Request Method (see #2 in above screenshot)

Possible Fix:

• Select correct Request Method. Refer to your API see it supports the method you selected. For example You selected POSt method to update some data but your API is expecting PUT method rather than POST.

Status Code 411 – Length Required

Possible Reason(s):

• You selected request method as POST or PUT but did not pass any body

Possible Fix:

• Make sure you supply some value in the Request Body field as per API specification. See #4 in the screenshot above.

Status Code 415 – Media Type Is Not Supported

Possible Reason(s):

• You did not pass correct Content-Type Header or your API was expecting different content-type setting than what you passed.

Possible Fix:

• Pass HTTP Header Content-Type with correct value (Check API documentation). Typically for JSON API its

  application/json

    and for XML SOAP API its

  text/xml

• If you are using Zappysys REST API Drivers or SSIS Connectors you will see Content Type Dropdown on UI see #4 in above screenshot. Change it to correct setting. If your content type not in the dropdown then you can set direct HTTP Header
  Example =>

  Content-Type: application/json

    see #5 in screenshot.

Debugging HTTP Web Requests

If you want to make sure what’s being passed to web server you can use this tool (Fiddler) and debug your web requests. It will show you all parts of your HTTP Request / Response (i.e.  Headers, Body, URL Parameters)

The post Understand HTTP Status Code and Fix Common Errors in REST API appeared first on ZappySys Blog.

Fiddler is one the most popular tool to inspect your Http Traffic. This tool helps you to test REST API / SOAP Web requests very easily. If you are using SSIS PowerPack or REST API ODBC Drivers you will find this post really useful to debug various REST API integration issues.

When you run Fiddler on your system, it acts as a tiny Web Proxy that sits between your client application and the webserver. Fiddler only starts capturing traffic when you running it. By default, it runs on the local machine (127.0.0.1) Port 8888. You can change the default port from Tools > Options.

How to download Fiddler (it’s FREE)

You can download Fiddler Classic using this link for FREE (If that link doesn’t work then try this one)

Make sure that you have Downloaded the Fiddler Classic and not a Fiddler Everywhere. Both products might have same features listed in this article but screenshots and steps listed in this article is about Fiddler Classic.

How to start/stop Fiddler two ways – Capture HTTP(s) Traffic

After you install Fiddler, Go to Start Menu  >Search for Fiddler

Launch Fiddler exe and it will start capturing HTTP traffic (For HTTPS URL see next section).

Run fiddler to start capturing web requests/responses made by various client applications on your system (e.g. Curl, Chrome, Internet Explorer). To start/stop capture go to File > Check/Uncheck [Capture Traffic] option.

By default when you run Fiddler it behaves like the default proxy server on your system. However, not all web requests appear in fiddler unless the client application using a system Default Proxy.

Start / Stop Fiddler Capture

How to capture HTTPS traffic in Fiddler

By default Fiddler doesn’t show the content of Web requests made to HTTPS Url (Secure site) because it’s encrypted. Perform the following steps if you want to see HTTPS Traffic.

1. Launch Fiddler
2. Go to Tools > Telerik Fiddler Options > HTTPS > Check [Decrypt Https Traffic Option]
   

   Fiddler Option – Decrypt HTTPS Traffic

3. Once prompted Accept certificate generated by Fiddler

How to filter traffic for specific Domain or URL (contains)

If you see lots of unwanted log entries after you enable Capturing then try following setting to Include / Clude certain log entries from the Capture making it easier to analyze.

1. Go to Filters Tab
2. Check Show only if URL contains. Type any words you like to check in URL. e.g. zappysys.com will include only those URL which has “zappysys.com” anywhere in the Full URL. You can add multiple included by adding space after words
3.  If you like to exclude some URLs then check Hide if URL contains option and enter string like previous step
   

How to view Web Request and Response in Fiddler

Once your web requests appear on the left side panel. Simply double click on the request entry to view. On your right side, you will see two panels.

1. Top  Panel – Request View
2. Bottom Panel – Response View

Here is an example of seeing API calls generated by aws command line requests/response in Fiddler.

Capture AWS Command line (CLI) request using Fiddler

How to Test API using Fiddler (Call REST API for without Postman)

Fiddler comes with another cool feature just like Postman so you can test your API call. Basically, there will be a time when you like to edit your request and test with different data (e.g. URL / Body or Headers). If that’s the case you can directly go to Composer Tab or Replay existing request right Right Click.

How to replay existing request / edit / send new request

To reply to the existing request

Test Web Requests in Fiddler Composer – Replay existing REST API requests or send new one (Edit Header, Body, URL)

How to view and format JSON / XML data in Fiddler

By default Fiddler will show you JSON  / XML Tab if request body or response data in that format (see JSON / XML tabs on top and bottom panels). But what if you like to see in Raw Text but in Syntax Highlighted  / formatted way?

Format Fiddler Request / Response (XML or JSON). View text with Syntax Highlighting

How to use fiddler Converters (e.g. Base64 / URL encode – decode)

There will be a time when you like to encode / decode strings appearing in web requests. Fiddler has another very handy feature call Send to Text Wizard. This option appears in most right click Menu in Raw / Syntax View / Headers view and other places.

Encode / Decode / Convert Text in Fiddler (e.g. Base64, URL Encode)

How to add Request Start Time, Overall Elapsed Time Columns in Fiddler

It’s really useful to see how much time each request is taking. For that, you can try the below steps in Fiddler Classic

1. Right-click on any Header Column
2. Select “Customize Columns”
3. Choose “Session Timers“
4.  Choose Overall_Elaspsed and  ClientConnectedTime from the dropdown

How to see request start time, overall elapsed time in Fiddler

How to edit and execute the processed requests in Fiddler

If you want to re-execute existing requests in Fiddler with different parameters then try the below steps

1. Select request from the left-hand side panel (or may be bottom side)
2. Press “E” and you will see a panel like below
3. Edit your request (i.e. URL, Header or Body)
4. Click on Run to Complete

Edit, Execute Processed Requests in Fiddler

How to enable AutoResponder – Fake Response (Rule based)

If you want to test some API calls for testing purposes, then you can use the AutoResponder feature of Fiddler. It allows you to match URL pattern and send response text from a local file. Here is how to enable Fake Response (Rule-based Auto Response)

1. Select URL from the log (Optional) – If you skip this step then in step#3 you will have to enter URL manually.
2.  Go to AutoResponder Tab
3. Click Add Rule (if you didn’t select Log entry then Rule editor will have blank URL)
4. Enter / Change URL Rule in the Rule editor if needed. By default it uses EXACT match rule. Click here to learn more about more rules
5. Enter a file path where you have saved Fake Response you like to serve if URL rule.

Examples:

Exact URL match:  

EXACT:HTTP://abc.com/api/v3/projects/123/milestones

Pattern URL match:

method:GET regex:(.*)/api/v3/projects/([\w\d-])+/milestones(.*)

Fiddler AutoResponder (Test Fake Response from local file – Rule match based on URL Pattern)

How to diff Fiddler requests

Sometimes you like to see difference between two requests. Fiddler comes with very handy feature. Just select two requests and click Diff Option in Right click menu. For diff to work make sure you install some diff utilities like KDiff3 or WinMerge.

Compare requests in Fiddler / Reissue same request with changes (Composer Tab)

How to show SSIS PowerPack requests in Fiddler

In most cases SSIS PowerPack requests will apperar in Fiddler without any extra configurations. However in some cases applications like SSIS PowerPack cannot detect it as System Default Proxy automatically. In that case try to enable Proxy settings in HTTP Connection or OAuth Connection.  If you are not using connection manager then some Components like JSON Source offers its own Proxy tab so just check Enable Proxy.

By default Fiddler register itself as Default Web Proxy on port 8888  (i.e. 

http://127.0.0.1:8888

Using Proxy Settings in SSIS PowerPack

View HTTP Request in Fiddler

To inspect HTTP Web Request using Fiddler perform following steps.

1. Click on the web request entry on left pane
2. Click on the Inspector Tab > Click Raw from top panel

View HTTP Response in Fiddler

To inspect HTTP Web Response using Fiddler perform following steps.

1. Click on the web request entry on left pane
2. Click on the Inspector Tab > Click Raw from bottom panel
3. You can also click on JSON or XML Tab if your want to see response coming in specific format

HTTTP Multipart POST – File Upload Example – Check Fiddler Request

View compressed (i.e. gzip, deflate) Response in Fiddler raw view

When response is compressed then by default you wont be able to see it in Raw tab. When response is compressed you may see header like below in response headers. You can also click on

Content-Encoding: gzip

To view compressed response in fiddler perform following steps.

1. Click on the web request entry on left pane
2. Click on the Inspector Tab > Click Transformer tab from bottom panel
3. Click on transformer tab and select No compression option and then click Raw tab

View uncompressed data in Fiddler (GZip, Deflate Encoding)

How to view Formatted JSON / XML

If you have JSON or XML response and you like to view formatted documents then you can go to Syntax View in the Response Panel and right-click and format

How to format Fiddler Request / Response (JSON/XML document)

Using Fiddler Text Wizard Tools (Encode / Decode – Hex, HTML, URL, Base64)

If you would like to perform some conversions like the one below, then you can use the TextWizard tool found in Fiddler. It’s very handy. Simply right click and click on Send to TextWizard option (found in most tabs / screens)

• Base64 Encode / Decode
• HTML Encode / Decode
• URL Encode / Decode
• Hex Encode / Decode
• and more…

Launching Fiddler TextWizard from Right click menu

Fiddler Send To TextWizard Menu

Launching Fiddler TextWizard from Toolbar

Fiddler TextWizard UI

How to compare requests / re-execute

Sometimes you want to compare two requests and find the difference. Here is how you can compare requests and re-execute same request or edit it.

Compare requests in Fiddler / Reissue same request with changes (Composer Tab)

How to show web request of Curl in Fiddler

By default some application such as Curl doesn’t use system default Proxy. That means if you running Fiddler and trying to analyze requests made from Curl command line then it wont show up. Its easy to tell curl that which proxy server to use. Just use -x command line option with proxy server IP and Port (e.g. Default Fiddler IP and port … 127.0.0.1:8888 )

curl.exe -v -x 127.0.0.1:8888 https://google.com

How to show SoapUI requests in Fiddler

If you are using an application like SoapUI then also it may not use Fiddler as default web proxy. to show your SoapUI web requests in Fiddler try following options

Configure Proxy Settings in SoapUI

How to show aws command line requests in Fiddler

If you are using aws command line application and want to show requests in Fiddler then add –no-verify-ssl like below. This way you can capture RAW API requests and use it like this in SSIS.

aws lambda invoke --function-name HelloWorld c:\temp\outputfile.txt --no-verify-ssl --payload file://c://test/customer.json

How to show Windows Service requests in Fiddler (Local System Account)

By default if Fiddler launched as Admin it will capture most of requests by various apps. But if your like to capture Web requests which are issued by windows service such as ZappySys Data Gateway then you might have to do little extra work (Assuming Service is not setup Run As Your account). By default many service Run under “Local System” account

Here is how to capture those requests which are issued by service account.

1. Go to Folder where Service Exe is located (If you are unsure simply right click on Service > Properties and check path)
2. Create new file in the same folder where Service Exe is located. File name must be <yourexename>.exe.config
3. Enter following XML text in your config file and save
   

   <configuration>
     <system.net>
       <defaultProxy enabled="true">
         <proxy proxyaddress="http://127.0.0.1:8888" bypassonlocal="False"/>
       </defaultProxy>
     </system.net>
   </configuration>

4. Restart Service and check Fiddler now see requests are captured

Capture Windows Service Web Requests in Fiddler (App Config method)

REST API integration using ODBC in BI Apps (e.g. Power BI, Excel, Informatica, Access, C#…)

If you want to consume API data in SQL Server (without ETL) or Live API consumption in Tools like Power BI, Excel, Tableau, SSRS, Talend, Informatica then use ODBC PowerPack from here. See below table for each use case.

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Remove Fiddler Root Certificate (Uninstall / Untrust)

The Fiddler certificate is unique per machine / user, so you do not have to worry about someone else reusing it to intercept traffic, but still, for some reason, if you would like to uninstall the Fiddler root certificate (installed when you checked the Decrypt option), then perform the following steps.

1. Goto Tools > Options > HTTPS
2. Uncheck the Decrypt option first if it’s checked
3. Click on Actions > Remove interception Certificates

How to remove Fiddler Root Certificate

Coding-Free REST API integration using ODBC Drivers (For many Reporting / ETL Tools)

If you ever need to read /write data from REST API without writing a single line of code, check these ODBC Drivers / API Connectors.

ODBC PowerPack is a collection of high-performance drag and drop ODBC / API Drivers for various Modern / Legacy data sources. Connect data in Reporting / ETL Tools / Programming Languages.

Coding-Free REST API integration in SSIS (Microsoft ETL Tool)

If you use SSIS as your primary etl tool and you have API integration usecase then check this product.

The post Tutorial – How to use Fiddler to analyze HTTP Web Requests (Tips) appeared first on ZappySys Blog.