Tutorial – How to use Fiddler to analyze HTTP Web Requests (Tips)

Introduction

Fiddler is one the most popular tool to inspect your Http Traffic. This tool helps you to test REST API / SOAP Web requests very easily. If you are using SSIS PowerPack or REST API ODBC Drivers you will find this post really useful to debug various REST API integration issues.

When you run Fiddler on your system, it acts as a tiny Web Proxy that sits between your client application and the webserver. Fiddler only starts capturing traffic when you running it. By default, it runs on the local machine (127.0.0.1) Port 8888. You can change the default port from Tools > Options.

How to download Fiddler (it’s FREE)

You can download Fiddler using this link for FREE (If that link doesn’t work then try this one)

Make sure that you have Downloaded the Fiddler Classic and not a Fiddler Everywhere.

Fiddler is a third party Telerik’s tool used to log, inspect, and alter HTTP and HTTPS traffic between a computer and a web server or servers.
It is not our product, this article is just to show you how you can analyze the API Request and Response call made from our Tasks/Components/Drivers.

How to start/stop Fiddler two ways – Capture HTTP(s) Traffic

After you install Fiddler, Go to Start Menu  >Search for Fiddler

Launch Fiddler exe and it will start capturing HTTP traffic (For HTTPS URL see next section).

Run fiddler to start capturing web requests/responses made by various client applications on your system (e.g. Curl, Chrome, Internet Explorer). To start/stop capture go to File > Check/Uncheck [Capture Traffic] option.

By default when you run Fiddler it behaves like the default proxy server on your system. However, not all web requests appear in fiddler unless the client application using a system Default Proxy.

Start / Stop Fiddler Capture

Start / Stop Fiddler Capture

How to capture HTTPS traffic in Fiddler

By default Fiddler doesn’t show the content of Web requests made to HTTPS Url (Secure site) because it’s encrypted. Perform the following steps if you want to see HTTPS Traffic.

  1. Launch Fiddler
  2. Go to Tools > Telerik Fiddler Options > HTTPS > Check [Decrypt Https Traffic Option]
    Fiddler Option - Decrypt HTTPS Traffic

    Fiddler Option – Decrypt HTTPS Traffic

  3. Once prompted Accept certificate generated by Fiddler

How to view Web Request and Response in Fiddler

Once your web requests appear on the left side panel. Simply double click on the request entry to view. On your right side, you will see two panels.

  1. Top  Panel – Request View
  2. Bottom Panel – Response View

Here is an example of seeing API calls generated by aws command line requests/response in Fiddler.

Capture AWS Commandline (CLI) request using Fiddler

Capture AWS Command line (CLI) request using Fiddler

How to Test API using Fiddler (Call REST API for without Postman)

Fiddler comes with another cool feature just like Postman so you can test your API call. Basically, there will be a time when you like to edit your request and test with different data (e.g. URL / Body or Headers). If that’s the case you can directly go to Composer Tab or Replay existing request right Right Click.

How to replay existing request / edit / send new request

To reply to the existing request

Test Web Requests in Fiddler Composer - Replay existing REST API requests or send new one (Edit Header, Body, URL)

Test Web Requests in Fiddler Composer – Replay existing REST API requests or send new one (Edit Header, Body, URL)

 

How to view and format JSON / XML data in Fiddler

By default Fiddler will show you JSON  / XML Tab if request body or response data in that format (see JSON / XML tabs on top and bottom panels). But what if you like to see in Raw Text but in Syntax Highlighted  / formatted way?

 

Format Fiddler Request / Response (XML or JSON). View text with Syntax Highlighting

Format Fiddler Request / Response (XML or JSON). View text with Syntax Highlighting

How to use fiddler Converters (e.g. Base64 / URL encode – decode)

There will be a time when you like to encode / decode strings appearing in web requests. Fiddler has another very handy feature call Send to Text Wizard. This option appears in most right click Menu in Raw / Syntax View / Headers view and other places.

Encode / Decode / Convert Text in Fiddler (e.g. Base64, URL Encode)

Encode / Decode / Convert Text in Fiddler (e.g. Base64, URL Encode)

How to add Request Start Time, Overall Elapsed Time Columns in Fiddler

It’s really useful to see how much time each request is taking. For that, you can try the below steps in Fiddler Classic

  1. Right-click on any Header Column
  2. Select “Customize Columns”
  3. Choose “Session Timers
  4.  Choose Overall_Elaspsed and  ClientConnectedTime from the dropdown
How to see request start time, overall elapsed time in Fiddler

How to see request start time, overall elapsed time in Fiddler

How to edit and execute the processed requests in Fiddler

If you want to re-execute existing requests in Fiddler with different parameters then try the below steps

  1. Select request from the left-hand side panel (or may be bottom side)
  2. Press “E” and you will see a panel like below
  3. Edit your request (i.e. URL, Header or Body)
  4. Click on Run to Complete

Edit, Execute Processed Requests in Fiddler

How to diff Fiddler requests

Sometimes you like to see difference between two requests. Fiddler comes with very handy feature. Just select two requests and click Diff Option in Right click menu. For diff to work make sure you install some diff utilities like KDiff3 or WinMerge.

Compare requests in Fiddler / Reissue same request with changes (Composer Tab)

Compare requests in Fiddler / Reissue same request with changes (Composer Tab)

How to show SSIS PowerPack requests in Fiddler

In most cases SSIS PowerPack requests will apperar in Fiddler without any extra configurations. However in some cases applications like SSIS PowerPack cannot detect it as System Default Proxy automatically. In that case try to enable Proxy settings in HTTP Connection or OAuth Connection.  If you are not using connection manager then some Components like JSON Source offers its own Proxy tab so just check Enable Proxy.

By default Fiddler register itself as Default Web Proxy on port 8888  (i.e.  http://127.0.0.1:8888  )

Using Proxy Settings in SSIS PowerPack

Using Proxy Settings in SSIS PowerPack

View HTTP Request in Fiddler

To inspect HTTP Web Request using Fiddler perform following steps.

  1. Click on the web request entry on left pane
  2. Click on the Inspector Tab > Click Raw from top panel

View HTTP Response in Fiddler

To inspect HTTP Web Response using Fiddler perform following steps.

  1. Click on the web request entry on left pane
  2. Click on the Inspector Tab > Click Raw from bottom panel
  3. You can also click on JSON or XML Tab if your want to see response coming in specific format

 

How to use Fiddler - Analyze Web Requests- File Upload Example - Check Fiddler Request

HTTTP Multipart POST – File Upload Example – Check Fiddler Request

View compressed (i.e. gzip, deflate) Response in Fiddler raw view

When response is compressed then by default you wont be able to see it in Raw tab. When response is compressed you may see header like below in response headers. You can also click on

To view compressed response in fiddler perform following steps.

  1. Click on the web request entry on left pane
  2. Click on the Inspector Tab > Click Transformer tab from bottom panel
  3. Click on transformer tab and select No compression option and then click Raw tab
View uncompressed data in Fiddler (GZip, Deflate Encoding)

View uncompressed data in Fiddler (GZip, Deflate Encoding)

How to compare requests / re-execute

Sometimes you want to compare two requests and find the difference. Here is how you can compare requests and re-execute same request or edit it.

Compare requests in Fiddler / Reissue same request with changes (Composer Tab)

Compare requests in Fiddler / Reissue same request with changes (Composer Tab)

How to show web request of Curl in Fiddler

By default some application such as Curl doesn’t use system default Proxy. That means if you running Fiddler and trying to analyze requests made from Curl command line then it wont show up. Its easy to tell curl that which proxy server to use. Just use -x command line option with proxy server IP and Port (e.g. Default Fiddler IP and port … 127.0.0.1:8888 )

How to show SoapUI requests in Fiddler

If you are using an application like SoapUI then also it may not use Fiddler as default web proxy. to show your SoapUI web requests in Fiddler try following options

Configure Proxy Settings in SoapUI

Configure Proxy Settings in SoapUI

 

How to show aws command line requests in Fiddler

If you are using aws command line application and want to show requests in Fiddler then add –no-verify-ssl like below. This way you can capture RAW API requests and use it like this in SSIS.

How to show Windows Service requests in Fiddler (Local System Account)

By default if Fiddler launched as Admin it will capture most of requests by various apps. But if your like to capture Web requests which are issued by windows service such as ZappySys Data Gateway then you might have to do little extra work (Assuming Service is not setup Run As Your account). By default many service Run under “Local System” account

Here is how to capture those requests which are issued by service account.

  1. Go to Folder where Service Exe is located (If you are unsure simply right click on Service > Properties and check path)
  2. Create new file in the same folder where Service Exe is located. File name must be <yourexename>.exe.config
  3. Enter following XML text in your config file and save
  4. Restart Service and check Fiddler now see requests are captured
Capture Windows Service Web Requests in Fiddler (App Config method)

Capture Windows Service Web Requests in Fiddler (App Config method)

REST API integration using ODBC in BI Apps (e.g. Power BI, Excel, Informatica, Access, C#…)

If you want to consume API data in SQL Server (without ETL) or Live API consumption in Tools like Power BI, Excel, Tableau, SSRS, Talend, Informatica then use ODBC PowerPack from here. See below table for each use case.

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.   ZappySys ODBC Drivers for REST API, JSON, XML - Integrate with Power BI, Tableau, QlikView, QlikSense, Informatica PowerCenter, Excel, SQL Server, SSIS, SSAS, SSRS, Visual Studio / WinForm / WCF, Python, C#, VB.net, PHP. PowerShell

BI / Reporting Tools Integration

ETL Tools Integration Programming Languages Integration
 

REST API integration in SSIS

If you use SSIS as your primary etl tool and you have API integration usecase then check this product.

Posted in Tools and tagged , , , , , , .