Using SOAP API with SharePoint in SSIS is a common requirement to Administer SharePoint. SharePoint on-premises is a nice Microsoft application used to share documents and collaborate with the company. You can have schedules, projects, documents, and more shared using a Web platform.

In addition, it is a very useful tool that your company may need. It is also possible to automate, administer tasks using SOAP API.
In this article, we will learn how to create SOAP API queries to SharePoint on-premises.

Also, we will show our ZappySys SSIS PowerPack that includes very powerful tools to export SharePoint data from REST API to any other source.

	SSIS REST API Web Service Task
	SSIS XML Source (File, REST, SOAP Web Service)

Requirements

1. Firstly, you will need SSDT installed.
2. Secondly, SSIS PowerPack installed.
3. SoapUI installed.

Getting started

We will start with SoapUI application. It will help us to get all the possible methods SharePoint SOAP API allows. Also, it will show the inner workings of how the requests should look, which we will later replicate on XML Source. SoapUI will be our polestar.

How to call SharePoint SOAP API (On-Premises)

To call SOAP API in SSIS you have to perform the following steps at a high level. Detailed steps are described in the next section

1. Import WSDL file or URL for your Service in SoapUI for testing.
   1. File > New SOAP Project.
   2. In the Initial WSDL field enter
      http://your.domain.com/_vti_bin/Lists.asmx?wsdl
      or
      http://your.domain.com/my-site/_vti_bin/Lists.asmx?wsdl
2. Add credentials for SOAP Web service (Basic Auth or WSS)
3. Create SOAP Request XML (Edit default parameters) and Click Play Button to test your request. Associative photo:
   

   Create SOAP Request Body from WSDL (Using SoapUI tool)

4. If your test works in SoapUI then you are ready to move to SSIS part.
5. Copy Service URL, Request Body and Headers to Notepad — we will use it in SSIS.
   For Header see Raw Tab on SoapUI > Copy two lines Content-Type and SOAPAction.
   

   Get ContentType and SOAPAction Headers from SoapUI Raw tab

In the next section, we will see how to call SharePoint SOAP Web Service in SSIS.

How to read data from a SharePoint List via SOAP Web Service

Now let’s move onto SSIS and get some data from SharePoint using XML Source.

1. Open SSIS, create a new Integration project and create a new SSIS package.
2. Drag and drop Data Flow onto the Control Flow from SSIS Toolbox.
3. Then drag and drop XML Source onto Data Flow (you may also use other similar connectors such as REST API Task or Web API Destination).
4. Then configure XML Source in this way:
   
   1. Enter URL:
      https://my.sharepoint.site.com/sites/mysite/_vti_bin/Lists.asmx
      or
      https://my.sharepoint.site.com/_vti_bin/Lists.asmx (depends if you have many sites or just one).
   2. Check Use Credentials checkbox and create ZS-HTTP connection. Use Basic or NTLM authorization scheme.
   3. Then copy configuration from SoapUI to XML Source. E.g. HTTP Request Method, Body, Body Content-Type, HTTP Headers (e.g. SOAPAction header).
   4. Use $.soap:Envelope.soap:Body.GetListItemsResponse.GetListItemsResult.listitems.rs:data.z:row[*] as filter. If it doesn’t work, use Select Filter button to select the path manually and select row node.
5. What’s left is to connect XML Source with OLEDB Destination or a similar destination component:
   

Conclusion

To conclude, we can say that using SSIS PowerPack component XML Source and SoapUI we could achieve our goal of getting data from SharePoint On-Premises. The next step could be writing data to SharePoint via the same SOAP API interface, but this time using Web API Destination component.

The post Read data from SharePoint List in SSIS (On-Premises SOAP API) appeared first on ZappySys Blog.

If you have need to integrate REST API in SSIS (i.e. fetch JSON, XML, CSV via HTTP call) and wondering how to Read API data in SSIS and Load into SQL DB (i.e. SQL Server, MySQL, Oracle) then you are at the right place.

In this article we will cover simple step by step approach how to achieve API integration in few clicks. For detailed introduction please refer to our previous article.

In next few sections we will cover the following SSIS Components to read data from API like a Table (Parse in Rows / Column). Our instructions will be mostly for JSON Format but concepts are same for other formats (i.e. XML / CSV).

	JSON Source Connector (Read from REST API, JSON File or OData Service): Use this dataflow component when you have to fetch data from REST API webservice like a table. This component allows you to extract JSON data from webservice and de-normalize nested structure so you can save to Relational database such as SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local JSON files or direct JSON string (Wildcard pattern supported too e.g. c:\data\file*.json).
	XML Source Connector (SOAP, File, REST) : Use this dataflow component when you have to fetch data from XML or SOAP webservice and consume data like a table. This component allows you to extract data from webservice and save to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local XML files or direct XML string.
	CSV Source Connector (File, REST) : Use this dataflow component when you have to fetch data from File or REST API which returns data in CSV format . This component allows you to extract data from webservice and save to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local CSV files or direct CSV string.

There are few more API components not discussed in this articles. They are REST API Task (no Parser – Just raw API calls) and Web API Destination (Write / Send Data To API / Delete rows). They both have different use case to check them see it fits in your need.

Video Tutorial

Step-By-Step – Read API data in SSIS using JSON / XML / CSV Source

In this section you will learn how to use JSON Source Adapter to Read data from API URL.

Load API data into SQL Server Table / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

API Authentication

In our step by step example we did not cover Authentication to keep things simple but in real world you have to pass some sort of authentication details to make API calls (e.g. UserID/Password or Secure Token). There are many ways to authenticate with API Services. ZappySys offer mainly 2 Connection Managers to cover most Authentication scenarios.

1. HTTP Connection Manager
2. OAuth Connection Manager

Using HTTP Connection Manager

If you have use case of using UserID / Password or any other method supported by HTTP Connection Manager then use below option (Below example is for Basic Authentication)

SSIS JSON Source – Passing Basic Credentials (Supply UserID / Password) using HTTP Connection

Using OAuth Connection Manager

If your API support OAuth Standard for Authorization then use OAuth Connection Type as below.

Generate Token for Google API using OAuth 2.0

API Pagination

Pagination is another important concept to understand when you call API using ZappySys Connector. If you ready many records from your API then by default API might not return all rows so you have to paginate until all records are read (Like a Loop in Programming). ZappySys makes it easy to paginate. Check this article to configure many different Pagination Modes

Things have gone bad: Error handling & debugging

Incidentally, bad things can happen. A remote server may go offline or your server may go out of memory. In any case, you may want to know when that happens and take actions accordingly. For that purpose, you have to redirect bad rows to some other destination. For this example, we will take and use Web API Destination, but basically, you can use any SSIS component:

Handling errors

1. Add a Derived Column above Web API Destination with expression "(DT_WSTR,4000)ZS_JSON_OUT" and name it "JsonAsString". This will let you see what JSON you are actually passing.
2. Then add a database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don't forget to set Redirect row option for both, Error and Truncation columns:
   

   Redirected failed requests from Web API Destination to a desired destination when loading from SQL Server to REST API Service is failing. Derived Column JsonAsString added to be able to read JSON which was passed to Elasticsearch

3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled.

NOTE: You can read more about redirecting rows in SSIS Error Handling (Redirect bad rows) article.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Debugging Web API call using Fiddler in SSIS

Conclusion

REST API is becoming more and more popular each day. With that Data integration from RESTful API services going to be challenge. Luckily ZappySys SSIS PowerPack provides a great way to integrate any API in SSIS via simple drag and drop approach without coding. Try SSIS PowerPack for FREE  see how much time / money you can save and to integrate virtually any REST API.

The post How to Read API data in SSIS and Load into SQL Table appeared first on ZappySys Blog.

In our previous post we saw various ways to transform JSON arrays. However there will be a time when your JSON / XML file wont have Array and you need to Pivot JSON Data.

Sample JSON data file

Here is a sample JSON file which we like to parse into rows and columns. Notice how it is using values inside in property name. This pattern needs Pivot option usage described later in this article.

{
	"version": 1.0,
	"products": {
		"P001": {
			"name": "Product 1",
			"price": 10
		},
		"P002": {
			"name": "Product 2",
			"price": 11
		},
		"P003": {
			"name": "Product 3",
			"price": 12
		}
	}
}

If you like to play with real data then use below URL. This JSON has 60MB worth of data for all Products offered on AWS.  And It has similar structure as above except many more attributes under products node.

https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/us-east-1/index.json

Using SSIS PowerPack to Pivot JSON data

ZappySys offers powerful JSON and XML Connectors. Below section will describe how to Parse and Pivot JSON data or Pivot XML data. For demo purpose we will use JSON Source but steps are almost same for XML too.

Step-By-Step JSON Pivot

1. First drag data flow from SSIS Toolbox and then double click data flow
   

   Drag SSIS Data Flow Task from Toolbox

2. Once Data flow designer is open, drag ZS JSON Source from SSIS Toolbox.
   

   Drag ZappySys JSON Source

3. Now for demo we will use Hard coded JSON using Direct Value option as below. You can also enter URL or File path if you select Direct Path option.
4. Go to Pivot Columns to Rows Tab and check Enable Pivoting Option
   

   Pivot JSON / XML Data option for ZappySys API Connectors / Drivers

5. Now come back to Filter Options Tab here you can either browse Filter or enter by hand. If file is too large then enter by hand. For example in our case we will enter

   $.products

     because we like to to parse structure below that node. If you have nested hierarchy (i.e. products node under orders node and so on) then use dot to separate them e.g. 

   $.customer.orders.products

6. For very Large file you need to Un-check Include Parent Columns option else you will get OutOfMemory Exception
7. Now click Preview to see our data.
   

   Pivot JSON Data using SSIS JSON Source

    

8. Now you can connect your JSON / XML source to destination. See below example.
   

   SSIS Package Execution –
   Loading Google BigQuery Data into SQL Server

Using ODBC PowerPack to Pivot JSON data

So far we talked how to use SSIS to read your JSON data and Pivot it but what if you want to consume JSON / XML / REST API data in some other App without doing ETL via SSIS?  Thats where ZappySys ODBC PowerPack comes in picture. Its a collection of many drivers including JSON Driver and XML Driver.  You can also use Data Gateway Approach, which allows T-SQL code to fetch JSON / XML data directly into SQL Server Table without any ETL / Programming.

Here is sample query you can run using JSON Driver or XML Driver

SELECT * FROM $
WITH(
	 Src='https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/us-east-1/index.json'
	,Filter='$.products'
	,IncludeParentColumns='False'
	,EnablePivot='True'
)

See below screenshot of JSON Driver Configuration to get idea.
Note: Use above query rather than what is displayed in the Screenshot.

Using ODBC DSN in C# code to call REST API

Preview / Generate Query in JSON Driver / XML Driver

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Advanced Pivot with Path / Search and Replace Option

Now let’s look at an advanced scenario where you like to extract Pivot_Path (know where Pivoted Property came from). For that, you can download the latest version which providers IncludePivotPath and EnablePivotPathSearchReplace options as below.

As you can see in the below example let’s say you like to extract table names and view names if defined under “views” node.

{
  "table-1": {
      "views": {
        "view-tbl1-1": {},
        "view-tbl1-2": {}
    }
  },
  "table-2": {
      "views": {}
  },
  "table-3": {
      "views": {
        "view-tbl3-1": {}
    }
  }
}

We set following

• On the Filter Options Tab
  Filter = 

  $..views

• on the Pivot Columns To Rows Tab
  Check Enable Pivoting
  Check Include_PivotPath
  Check Enable Pivot_Path search/replace
  Set Search for

  (.*).views.(.*)--regex

  Set Replace With

  $1

Advanced Pivot Option – Search and Replace Path

Conclusion

In this particle we saw how to use some advanced options for XML/ JSON Source and ODBC Drivers. You can download respective product depending your usecase. For SSIS usecase Download SSIS PowerPack and for other ETL / Reporting app integration scenarios (i.e. Power BI, Informatica, SSRS , Excel, MS Access, SQL Server, JAVA, C# ….) Download ODBC PowerPack.

The post Pivot JSON and XML data using SSIS or ODBC Drivers appeared first on ZappySys Blog.

In this post, we will learn how to Create Excel File in SSIS from source like JSON / XML.  We will use SSIS PowerPack to connect and query a JSON or XML file. This article also covers creating Excel from JSON File.

JSON stands for JavaScript Object Notation and it is an Open and Standard format to read an object with attributes and values.  JSON is replacing XML because it is faster to read data, it is easier to parse data, it does not require end tags, it is shorter and it can use arrays.

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Make sure that Microsoft Excel installed
4. Finally, do not forget to install ZappySys SSIS PowerPack

Components Mentioned in this article

	JSON File/REST API Source
	XML File Source
	Excel Destination

What is JSON File Source

JSON Source Connector can be used to extract and output JSON data stored in local JSON files, JSON data coming from REST API web service calls (Web URL) or direct JSON String (variables or DB columns). JSON component also supports JSONPath to filter data from nested array/sub-documents. This component is optimized to work with very large files.

What is XML File Source

SSIS XML Source can be used to consume multiple XML files or XML formatted response from SOAP Web service or REST API Web Service. It supports advanced filtering along with a flexible approach to configuring request parameters for web service.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS JSON Source for JSON or XML File as well as REST API that will help you in reading JSON or XML Files from JSON or XML File to the Local machine. Here we are showing you is, How to Load JSON or XML Data in Excel File using SSIS Pack.

Getting Started – Create Excel File in SSIS

Let´s start with an example. In this SSIS JSON or XML Source for JSON or XML File task example, we will read JSON or XML files from Microsoft Excel File.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

2. Drag and Drop relevant JSON or XML Source(REST API and File) for JSON or XML File Task from the SSIS Toolbox.
   

   SSIS Add JSON or XML Source

3. Double Click on JSON or XML Source (REST API or File) for edit.
   

   SSIS Json/XML Source File

4. JSON and XML Source Task is able to do more various call of Rest API using Http and Oauth2 Connection also Please refer below articles link for the same
   • Various Articles for JSON Source
   • Various Articles for XML Source
5. Now let’s create the Excel File Connection. In the connection Manager Window right click and select new connection.
   

   Create Excel File Connection

    

6. And Create The Zs-Excel Type Connection. And click Add.
   

   Add Excel Connection

    

7. Now select the excel file path in and test connection.
   

   Select Excel File For Connection

    

8.  Now add Our Excel File Destination Connector and connect upper JSON or XML source with it
   https://zappysys.com/products/ssis-powerpack/ssis-excel-file-destination/
9.  And now edit the Excel destination and select that created excel file connection to load the data in excel file.
   

   SSIS Excel Connection Advanced Edit

    

10. Give the Range Start cell in Component Properties. (A1 for starting of the file) and select the table name.
    

    Give RangeStartCell and TableName

     

11.  Select the input columns.
    

    Select Input Columns

12. That’s run the package and it will load all the JSON or XML File data into the Excel file.
    

    run-package-for-json-into-excel-file

    

    Stored Data into Excel File

     

Conclusion

Above all, in this blog, we learned how to Load JSON or XML Data in Excel File using SSIS Pack. We used JSON or Excel Source(REST API or File) for JSON or Excel file, Excel Destination for Excel file to read the JSON or XML file and load data into Excel File. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• Our Various Articles for JSON Source
• Our Various Articles for XML Source
• Excel Destination

The post Create Excel File in SSIS (Read from JSON / XML) appeared first on ZappySys Blog.

Read RSS feed in SSIS can be challenging. RSS named first RDF Site Summary and later named Rich Site Summary and Really Simple Syndication allows customer applications to be updated with the news of a site. For example, Microsoft RSS feeds, Apple RSS feeds, Samsung RSS feeds, etc.  With RSS the information comes to you directly and you do not need to waste time checking the web site. In this article, we will show how to read RSS feed in SSIS and later using ODBC.

Requirements

1. First of all, ZappySys SSIS Powerpack installed.
2. Secondly, SSDT for Business Intelligence (SSIS) installed.
3. Finally, ODBC Powerpack installed.

Getting Started

Understanding RSS data

1. In order to start, we will work with the WordPress RSS example:
   

   https://wordpress.org/news/feed/

2. When you go to the link, you see the feed like this:
   

   RSS information

3. Basically, it is an XML file with the Last Build date (lastBuildDate), the title, link, description. It shows the WordPress news per version and date.

Read RSS feed in SSIS

1. First of all, you need an SSIS Project and a Data Flow with the ZS XML Source (if you are not familiar with SSIS and the ZS XML Source, refer to this link.
   

   Drag and drop Data Flow

2. Secondly, In the ZS XML Source, add the following RSS WordPress URL:
   

   https://wordpress.org/news/feed/

   

   SSIS read RSS

3. After that, you will be able to see the following tags when you press the array filter:
4. This time, we will focus on the information under channel which is the most relevant, make sure that the array filter is this one:
   

   $.rss.channel[*].item[*]

   

   Using RSS filters in SSIS

5. In addition, press the preview data button and you will see the following information:
   

   SSIS preview RSS

Read RSS feed in SSIS and copy to SQL Server

1. In the previous section, we got data from RSS. Now we will copy to SQL Server.
2. First of all, drag and drop the OLEDB Destination and join the XML Source with the OLEDB Destination:
   

   SSIS RSS to OLEDB

3. Also, in the OLEDB Destination, press New button and select a SQL Server connection or create a new one:
   

   RSS to Oledb

4. In addition, in the table or view textbox press new to create a new destination table:
   

   Create a new SQL table to store RSS data

5. Next, modify the table name:
   

   Change SSIS destination table name

6. Also, go to the mappings page to generate the mappings:
   

   RSS Source to SQL Destination in SSIS

7. Finally, run the package and you will export the data from RSS to SQL Server.

How to read RSS feed in SSIS using pagination

If you have multiple rows, pagination helps a lot to handle multiple items. You can handle the information in pages. Instead of loading all the information, you can get the data from page 1, 2, 3 and more.

1. First of all, in the URL, you can include the following to specify the page:
   

   https://wordpress.org/news/feed/?paged=2

2. Also, note that it used the word paged and not page. Many URLs use page and not paged, but WordPress is different:
3. In addition, if you want to use variables, you can use a variable like this:
   
   RSS page as a variable
4. Also, we have a pagination tab with different pagination options. You can check the configuration here:
   

   RSS pagination configuration

5. Finally, for more information about pagination, refer to this link.

How to read RSS feed in ZappySys ODBC PowerPack

Zappysys ODBC PowerPack is a powerful tool to access to REST API, SOAP, XML, JSON, CSV, Odata. In this article, we will show how to access to RSS using ODBC PowerPack.

1. First of all, open your ODBC Data Source Administrator in Windows.
2. Secondly, in the user DSN, press Add.
3. Also, add the Zappysys XML Driver:
   

   odbc XML driver

4.  In addition, add the following URL:
   

   https://wordpress.org/news/feed/

   

   ODBC to RSS connection

5. Finally, you can use the filters and pagination as we did with the SSIS PowerPack in the previous sections.
   

   ODBC pagination RSS

Conclusion

To conclude, we can say that with SSIS PowerPack and ODBC PowerPack, it is easy to access, filter and paginate RSS information or any XML / SOAP API. We strongly recommend you to try the product by yourself and check the results.

References

For more information about RSS, refer to these links:

• RSS
• What Is RSS? RSS Explained

The post How to read RSS feed in SSIS and ODBC (with pagination) appeared first on ZappySys Blog.

In our previous post, we saw how to call REST API in SSIS. Now let’s learn how to read data from SAP S4 / HANA OData Service (i.e. S/4HANA). SAP HANA provides an OData REST API interface to access data in your application using HTTP Protocol. We will use the SSIS XML Source component to read SAP data and load it into the SQL Server Table in a few clicks.

About SAP HANA / OData REST API Service

You can expose your SAP Data using ODATA REST API Service. Here is a good article that shows how to expose data as an OData Service. For more information on the SAP OData Service feature check the SAP help page. If you are new to OData standard then read an article to learn more about it. If you need an XML-based OData Service then use the test URLs below:

https://services.odata.org/Northwind/Northwind.svc/
https://services.odata.org/Northwind/Northwind.svc/Customers
https://services.odata.org/Northwind/Northwind.svc/Orders
https://services.odata.org/Northwind/Northwind.svc/Invoices
https://services.odata.org/Northwind/Northwind.svc/Products

Now let’s look at how to read SAP Data in SSIS. At this point, we assume you have exposed your data as OData Endpoint.

Step-by-Step Instructions

Let’s look at the step-by-step approach to how to read SAP data using SSIS components. Then we will load SAP HANA data into SQL Server.

Read SAP HANA data using XML Source

First, let’s look at the steps on how to configure XML Source. We will read data from SAP S4 / HANA OData Service and then we will look at how to load data into SQL Server in the next section.

1. Open an SSIS package.
2. Drag Data Flow Task from SSIS Toolbox and double-click it to open it.
3. From the SSIS Toolbox drag ZS XML Source onto the background of Data Flow (in the designer). Double-click it to open the XML Source configuration window.
4. Change Data Format to OData. Selecting this option will take care of REST API pagination too.
5. Enter your OData Service URL. Its format looks similar to the one shown below. Replace 3 parts with your own value (i.e. replace {MY-INSTANCE}, {MY-PROJECT}, {MY-TABLE}):
   

   https://{MY-INSTANCE}/sap/opu/odata/sap/{MY-PROJECT}/{MY-TABLE}

   For example, if you are hosting SAP HANA in a cloud instance then your URL may look like this:
   

   https://myXXXXXX-api.s4hana.ondemand.com/sap/opu/odata/sap/MyTestProject/PurchaseOrders

6. Check the Use Credentials checkbox and select the New ZS-HTTP Connection option.
7. In HTTP Connection Manager select the Basic Authentication option and enter your SAP HANA UserID / Password.
   

   Configure SSIS HTTP Connection (SAP S4 / HANA OData Service – Using Basic Authentication)

8. Let’s get back to the XML Source connector we created in the previous stage. Press the Select Filter button or directly type

   $.feed.entry[*]

     into the field as shown below:
   

   Select Filter for OData Service (XML REST API)

9. Click Preview to make sure you can see the records.
10. Click OK to save the configuration and close the configuration window.

In the next section, we will configure the target to load SAP data into SQL Server using the ZappySys Upsert Destination Data Flow component.

Load SAP HANA data into SQL Server using Upsert Destination

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

In this post, we saw how easy it is to read SAP S4 HANA data using XML Source SSIS connector that is part of ZappySys SSIS PowerPack. Download it, try it out, and see it for yourself! It has many more connectors that were not mentioned in this post but could be also very helpful in your daily work.

The post Read SAP S4 / HANA data in SSIS (OData REST API) appeared first on ZappySys Blog.

In this post we will learn how to access data from Oracle UCM Web Service (Middle layer for WebLogic) and load into SQL Server or any other target. We will use SSIS XML Source to achieve this result.

About Oracle UCM Web Service

If you are not sure what is SOAP Web Service and how to call in SSIS, First read this article to understand general concepts of calling SOAP API in SSIS.

UCM Web Service offers calling various commands. You have to send XML Request and server will send you XML response with requested data. We will use Array Transformation Technique (Key / Value Pattern) to extract complex nested XML data.

Step-By-Step

Now let’s look at how to call Oracle UCM SOAP API in SSIS.

Read from Oracle UCM Web Service using XML Source

1. Drag Data Flow Task in SSIS Designer
2. Inside Data flow drag ZS SSIS XML Source from toolbox
3. First enter URL as below (Replace INSTANCE ID with your own instance)
   

   https://YOUR-INSTANCE-ID.oraclecloud.com/idcws/GenericSoapPort

4. Now Check Use Credentials and select new ZS-HTTP connection
5. Enter your User ID and Password to call Web Service using Basic Authentication. Click OK to save.
   

   Configure HTTP Connection – Call Oracle UCM Web Service (SOAP XML)

6. On XML Source UI Select Content Type = XML (text / xml, charset=UTF-8)
7. Select Request Method as POST
8. Click Enter Raw Edit for HTTP Headers and enter below
   

   SOAPAction: "urn:GenericSoap/GenericSoapOperation"

9. Click on Edit Body and enter your Request Body in XML format.For example to call GET_SEARCH_RESULTS service to retrieve the search results for the passed query text you can submit Request Body as below.
   NOTICE:  String in Search Text was surrounded by backtick (  `  ). Single tick / double quote didn’t work in our case.
   

   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ucm="http://www.oracle.com/UCM">
   <soapenv:Body>
     <ucm:GenericRequest webKey="cs">
       <ucm:Service IdcService="GET_SEARCH_RESULTS">
         <ucm:Document>
           <ucm:Field name="QueryText">dDocTitle &lt;starts&gt; `MY_sites_201`</ucm:Field>
         </ucm:Document>
       </ucm:Service>
     </ucm:GenericRequest>
   </soapenv:Body>
   </soapenv:Envelope>

   
10. Now Select Filter (Browse upto Fields under SearchResult node. Once you click OK try to edit your Expression so it looks like below. As you may notice we changed ResultSet[*] to ResultSet[?(@@name==’SearchResults’)]
    This will ensure that we extract records from only SearchResult Node and not other nodes such as UserAttribInfo or EnterpriseSearchResults
    

    $.env:Envelope.env:Body.ns2:GenericResponse.ns2:Service.ns2:Document.ns2:ResultSet[?(@@name=='SearchResults')].ns2:Row[*]

    

    Select Filter

11. Now go to 2D Array Tab. Select Key / Value Pattern Option.
12. Select Column Names / Value Filters as below. You can Use Browse Option to navigate to that field or just type below directly. Below expression will Transform Rows into Column. Foreach attribute name we will get one column.Column Name Filter =>  $.ns2:Field[*].@name
    Column Value Filter =>  $.ns2:Field[*].#text
13. Now Click Preview to confirm it works
    
14. That’s it, you can now click OK to close UI.

Loading Oracle UCM SOAP API data into SQL Server / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

In this post we saw how easy it is to consume XML SOAP Web Service in SSIS. We called SOAP Service for Oracle UCM Web Service and loaded data into target like SQL Server. You can download SSIS PowerPack and try many other scenarios like this using 70+ other SSIS Tasks /Components not mentioned in this article.

The post Call Oracle UCM Web Service in SSIS (Read XML SOAP API) appeared first on ZappySys Blog.

QuickBooks Online is a well-known Cloud-based Accounting Software. In this post, you will learn how to implement QuickBooks Online API Integration with SQL Server or any other RDBMS (e.g. Oracle, MySQL, Postgresql) using SSIS in few clicks. We will use SSIS XML Source to Read data from QuickBooks Online and Load into SQL Server / other targets (Using OAuth Connection). We will discuss on How to Create an Intuit Developer Account, How to Create QuickBooks Online App for OAuth, We will also discuss reverse scenario to Write data to QuickBooks Online (API POST for Insert or Update in QuickBooks Online) using SSIS REST API Task
In nutshell, this post will focus on how to call QuickBooks Online API using SSIS.
So let’s get started.

Prerequisites

Before we look into Step-By-Step section to extract and load data from QuickBooks Online to SQL Server let’s make sure you met the following requirements.

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download a FREE trial.
4. You have the basic familiarity with REST API concepts and QuickBooks Online API.

What is QuickBooks Online

QuickBooks is an accounting software package developed and marketed by Intuit. QuickBooks products are geared mainly toward small and medium-sized businesses and offer on-premises accounting applications as well as cloud-based versions that accept business payments, manage and pay bills, and payroll functions.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS XML Source that will help you to call QuickBooks Online API, read various data from QuickBooks Online with SSIS, create a new Contact on QuickBooks Online with SSIS and do many more actions using REST API. To learn more about QuickBooks Online API check this help file.

To access QuickBooks Online data using REST API call, you need to create an Intuit Developer Account and then create QuickBooks Online App.

Create an Intuit Developer Account

Follow the mentioned steps below to create an Intuit developer account:

1. you can visit this site developer.intuit.com, and click on Sign up from the menu bar.
2. A registration form will be opened in a Popup. you can fill up and then click on Create Account.
   

   Signup Form: Intuit Developer Account

3. Complete the validation check and click Continue.
4. On the confirmation dialog, provide a few details to fill out your profile and click All set.

The post How to read data from QuickBooks Online in SSIS appeared first on ZappySys Blog.

In this post we will learn how to read data from NetSuite in SSIS. We will use ZappySys XML Source for SOAP API access.

What is NetSuite CRM?

NetSuite is a CRM / ERP product. It gives you scalable cloud CRM / ERP solution targeted at high-growing, mid-sized businesses and large enterprises. It automates front- and back-office processes including: financial management, revenue management, fixed assets, order management, billing, and inventory management.

Preparing for NetSuite WebService Call in SSIS

For NetSuite API call we have to make sure certain settings and obtain Application ID and AccountID. So let’s get started.

How to enable NetSuite Web Service Feature

Before you call any NetSuite SOAP API (i.e. SuiteTask Web Service), First thing to get started with NetSuite SOAP API (i.e. ) is to enable Web Service Feature. Once you enable Web Service, you can programmatically access NetSuite data in your App (e.g. Power BI, Tableau, Informatica). Click this link to see steps to enable Web Service Feature

Enable NetSuite Web Service Feature (SuiteTalk API)

Obtain NetSuite Account ID

NetSuite Account ID is needed by Login API call. So perform following steps if you dont know account id.

1. Login to your NetSuite Portal
2. Click the Setup Tab
3. In the drop-down click Integration >  Web services preferences
4. On that page you will find your Account ID (example account ID = TSTDRV1234567 )

Obtain NetSuite Account ID (Needed for Web Service Login)

Obtain NetSuite Application ID

Once you have account ID now we need to get Application ID. For that follow these steps.

1. Login to your NetSuite Portal
2. Click the Setup Tab
3. In the drop-down click Integration >  Manage Integrations
4. On that page you will find your existing Applications along with Account ID / Status (example Application ID = AE2EACFD-1234-1222-1111-12345544555 )
5. If you don’t have any application created yet then just click on New.

Create new App / Get NetSuite Application ID

Understanding how NetSuite SOAP API Call works

In this section we will briefly understand how NetSuite SOAP API call works. When you use ZappySys Driver / Component it will take care many details for you but still you have to configure correct XML Body. There are two steps involved before you can call any SOAP API in NetSuite.

1. Call Login method and extract JSESSIONID from response Cookie (This token expires after some time)
2. Call any other API (Pass obtained JSESSIONID token in Cookie header)

NetSuite Login Call

Request

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml;charset=UTF-8
SOAPAction: "login"
Accept-Encoding: gzip, deflate

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Header>
    <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <applicationId>xxxxxxxxxxxxxx</applicationId>
    </applicationInfo>
  </soap:Header>
  <soap:Body>
    <login xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <passport>
        <email xmlns="urn:core_2017_1.platform.webservices.netsuite.com">xxxxxxxxxxxxx</email>
        <password xmlns="urn:core_2017_1.platform.webservices.netsuite.com">xxxxxxxxxxxx</password>
        <account xmlns="urn:core_2017_1.platform.webservices.netsuite.com">TSTxxxxxxxxxxxxxxx</account>
        <role internalId="3" xmlns="urn:core_2017_1.platform.webservices.netsuite.com" />
      </passport>
    </login>
  </soap:Body>
</soap:Envelope>

Response Notice JSESSIONID=6xI6xxxxxxxxx in the login call response below. This Session ID can be used to call next call. ZappySys HTTP Connection Manager configured with Dynamic Token option can extract Cookie and reuse it until its expired (1 hour by default).

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 16:50:41 GMT
Content-Length: 1676
Content-Type: text/xml; charset=utf-8
X-N-OperationId: 777274f6-49c6-4883-bcd1-1d70df90d78e
Strict-Transport-Security: max-age=31536000
NS_RTIMER_COMPOSITE: 13633xxxxxxxxx:70617xxxxxxxxxxxxx36F6D:80
Set-Cookie: JSESSIONID=6xI61rdFt26pxxxxxxxxxxxxxxxqvJIMlu!-17981123445; path=/; secure; HttpOnly
Set-Cookie: NS_VER=2018.1.0; domain=webservices.netsuite.com; path=/
Set-Cookie: NS_ROUTING_VERSION=LAGGING; path=/
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent
Keep-Alive: timeout=10, max=990
Connection: Keep-Alive

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_xxxxxxxxxxxxxxxxx_9fc719</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <loginResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <sessionResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
        <ns1:status isSuccess="true" xmlns:ns1="urn:core_2017_1.platform.webservices.netsuite.com"/>
        <ns2:wsRoleList xmlns:ns2="urn:core_2017_1.platform.webservices.netsuite.com">
          <ns2:wsRole>
            <ns2:role internalId="3">
              <ns2:name>Administrator</ns2:name>
            </ns2:role>
            <ns2:isDefault>false</ns2:isDefault>
            <ns2:isInactive>false</ns2:isInactive>
          </ns2:wsRole>
          <ns2:wsRole>
            <ns2:role internalId="15">
              <ns2:name>Employee Center</ns2:name>
            </ns2:role>
            <ns2:isDefault>false</ns2:isDefault>
            <ns2:isInactive>false</ns2:isInactive>
          </ns2:wsRole>
      </sessionResponse>
    </loginResponse>
  </soapenv:Body>
</soapenv:Envelope>

 

NetSuite API Call ( getAll method )

Now once you have JSESSIONID extracted you can pass it to call next API call as below. For example here is how to call getAll method to obtain currency list and exchange rates. If you are not sure how to craft correct Request XML then use tool like SoapUI. You can use NetSuite WSDL file from here and import it in SoapUI.   Request See how Cookie is passed for authentication purpose. Again you can pass this manually or use HTTP connection with Dynamic Token to pass it automatically along with each API call.

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml;charset=UTF-8
SOAPAction: "getAll"
Cookie: JSESSIONID=6xI61rxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxMlu!-1798141234
Accept-Encoding: gzip, deflate

<soapenv:Envelope 
   xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
   xmlns:urn="urn:messages_2018_1.platform.webservices.netsuite.com" 
   xmlns:urn1="urn:core_2018_1.platform.webservices.netsuite.com">
   <soapenv:Body>
      <urn:getAll>
         <record recordType="currency"/> 
      </urn:getAll>
   </soapenv:Body>
</soapenv:Envelope>

Response

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 16:50:42 GMT
Content-Length: 9648
Content-Type: text/xml; charset=utf-8
X-N-OperationId: 1b2989e4-b930-4a72-b471-00a8e131dfd4
NS_RTIMER_COMPOSITE: 21403xxxxxx:706172746E65xxxxxxxxxxxx6F6D:80
Strict-Transport-Security: max-age=31536000
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_TSTDRV1439151_100320189871507121000128715_6ae8e8</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <getAllResponse xmlns="urn:messages_2018_1.platform.webservices.netsuite.com">
      <platformCore:getAllResult xmlns:platformCore="urn:core_2017_1.platform.webservices.netsuite.com">
        <platformCore:status isSuccess="true" />
        <platformCore:totalRecords>11</platformCore:totalRecords>
        <platformCore:recordList>
          <platformCore:record internalId="1" xsi:type="listAcct:Currency" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:name>USD</listAcct:name>
            <listAcct:symbol>USD</listAcct:symbol>
            <listAcct:isBaseCurrency>true</listAcct:isBaseCurrency>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:overrideCurrencyFormat>false</listAcct:overrideCurrencyFormat>
            <listAcct:displaySymbol>$</listAcct:displaySymbol>
            <listAcct:symbolPlacement>_beforeNumber</listAcct:symbolPlacement>
            <listAcct:locale>_unitedStatesEnglish</listAcct:locale>
            <listAcct:formatSample>$1,234.56</listAcct:formatSample>
            <listAcct:exchangeRate>1.0</listAcct:exchangeRate>
            <listAcct:fxRateUpdateTimezone>_americaNewYork</listAcct:fxRateUpdateTimezone>
            <listAcct:currencyPrecision>_two</listAcct:currencyPrecision>
          </platformCore:record>
          <platformCore:record internalId="2" xsi:type="listAcct:Currency" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:name>GBP</listAcct:name>
            <listAcct:symbol>GBP</listAcct:symbol>
            <listAcct:isBaseCurrency>true</listAcct:isBaseCurrency>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:overrideCurrencyFormat>false</listAcct:overrideCurrencyFormat>
            <listAcct:displaySymbol>£</listAcct:displaySymbol>
            <listAcct:symbolPlacement>_beforeNumber</listAcct:symbolPlacement>
            <listAcct:locale>_unitedKingdomEnglish</listAcct:locale>
            <listAcct:formatSample>£1,234.56</listAcct:formatSample>
            <listAcct:exchangeRate>1.33165497</listAcct:exchangeRate>
            <listAcct:fxRateUpdateTimezone>_americaNewYork</listAcct:fxRateUpdateTimezone>
            <listAcct:currencyPrecision>_two</listAcct:currencyPrecision>
          </platformCore:record>
          <platformCore:record internalId="3" xsi:type="listAcct:Currency" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:name>CAD</listAcct:name>
            <listAcct:symbol>CAD</listAcct:symbol>
            <listAcct:isBaseCurrency>true</listAcct:isBaseCurrency>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:overrideCurrencyFormat>false</listAcct:overrideCurrencyFormat>
            <listAcct:displaySymbol>$</listAcct:displaySymbol>
            <listAcct:symbolPlacement>_beforeNumber</listAcct:symbolPlacement>
            <listAcct:locale>_canadaEnglish</listAcct:locale>
            <listAcct:formatSample>$1,234.56</listAcct:formatSample>
            <listAcct:exchangeRate>0.76457273</listAcct:exchangeRate>
            <listAcct:fxRateUpdateTimezone>_americaNewYork</listAcct:fxRateUpdateTimezone>
            <listAcct:currencyPrecision>_two</listAcct:currencyPrecision>
          </platformCore:record>
        </platformCore:recordList>
      </platformCore:getAllResult>
    </getAllResponse>
  </soapenv:Body>
</soapenv:Envelope>

NetSuite Search API (Pagination Example)

Now let's look at more complex API calls where Pagination is needed. For example when you call search API to pull data from specific table (e.g. Account) you need to loop through multiple responses until all records are consumed. Request (Page-1)

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml; charset=utf-8
SOAPAction: "search"
Cookie: JSESSIONID=BKQ7xxxxxxxxxxxx4Ug!-995xxxxxx
Accept-Encoding: gzip, deflate

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Header>
    <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <applicationId>xxxxxxxxxxxxxxx</applicationId>
    </applicationInfo>
    <searchPreferences xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <pageSize>50</pageSize>
    </searchPreferences>
  </soap:Header>
  <soap:Body>
    <search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <searchRecord xmlns:q1="urn:accounting_2017_1.lists.webservices.netsuite.com" xsi:type="q1:AccountSearch">
        <q1:basic>
          <description operator="doesNotContain" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
            <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">x</searchValue>
          </description>
          <name operator="contains" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
            <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">a</searchValue>
          </name>
        </q1:basic>
      </searchRecord>
    </search>
  </soap:Body>
</soap:Envelope>

  Response (Page-1) In below response notice 2 things. SearchId is like a cursor id which you have to pass for each page you request after first page. And you can stop looping after 4th page based on totalPages count. platformCore:searchId = WEBSERVICES_ABCD_12345-xxxxxxxxx platformCore:totalPages = 4

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 18:40:33 GMT
Content-Length: 49266
Content-Type: text/xml; charset=utf-8
X-N-OperationId: c1e79336-63b7-4eb7-989e-5c3d24c56891
NS_RTIMER_COMPOSITE: 118xxxxxxxx:70617xxxxxxxxxxxxxxxxxxxxxD:80
Strict-Transport-Security: max-age=31536000
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent

<?xml version="1.0" encoding="utf-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <searchResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformCore:searchResult xmlns:platformCore="urn:core_2017_1.platform.webservices.netsuite.com">
        <platformCore:status isSuccess="true" />
        <platformCore:totalRecords>193</platformCore:totalRecords>
        <platformCore:pageSize>50</platformCore:pageSize>
        <platformCore:totalPages>4</platformCore:totalPages>
        <platformCore:pageIndex>1</platformCore:pageIndex>
        <platformCore:searchId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformCore:searchId>
        <platformCore:recordList>
          <platformCore:record internalId="1" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>1111</listAcct:acctNumber>
            <listAcct:acctName>Checking</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>xxxxxxxxxxxxxxxxxxx</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Checking</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          <platformCore:record internalId="2" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>22222</listAcct:acctNumber>
            <listAcct:acctName>Savings</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>yyyyyyyyy</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Savings</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          .........
          .........
          .........
          .........		  
        </platformCore:recordList>
      </platformCore:searchResult>
    </searchResponse>
  </soapenv:Body>
</soapenv:Envelope>

  Request (Page-2 and onwards) Here is the second page request. Notice few things here. In heraders we are now sending SOAPAction: "searchMoreWithId" rather than "search".  Also our SOAP Body is also different compared to first page request. Now we only sending pageIndex we want to fetch and searchId. We have also change <search> tag with <searchMoreWithId> under Body.

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml; charset=utf-8
SOAPAction: "searchMoreWithId"
Cookie: JSESSIONID=BKQ7xxxxxxxxxxxx4Ug!-995xxxxxx
Accept-Encoding: gzip, deflate

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Header>
    <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <applicationId>xxxxxxxxxxxxxxxxxx</applicationId>
    </applicationInfo>
    <searchPreferences xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <pageSize>50</pageSize>
    </searchPreferences>
  </soap:Header>
  <soap:Body>
    <searchMoreWithId xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <searchId>WEBSERVICES_ABCD_12345-xxxxxxxxx</searchId>
      <pageIndex>2</pageIndex>
    </searchMoreWithId>
  </soap:Body>
</soap:Envelope>

Response (Page-2 and onwards)

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 18:40:33 GMT
Content-Length: 49266
Content-Type: text/xml; charset=utf-8
X-N-OperationId: c1e79336-63b7-4eb7-989e-5c3d24c56891
NS_RTIMER_COMPOSITE: 118xxxxxxxx:70617xxxxxxxxxxxxxxxxxxxxxD:80
Strict-Transport-Security: max-age=31536000
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent

<?xml version="1.0" encoding="utf-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <searchMoreWithIdResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformCore:searchResult xmlns:platformCore="urn:core_2017_1.platform.webservices.netsuite.com">
        <platformCore:status isSuccess="true" />
        <platformCore:totalRecords>193</platformCore:totalRecords>
        <platformCore:pageSize>50</platformCore:pageSize>
        <platformCore:totalPages>4</platformCore:totalPages>
        <platformCore:pageIndex>2</platformCore:pageIndex>
        <platformCore:searchId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformCore:searchId>
        <platformCore:recordList>
          <platformCore:record internalId="1" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>8000</listAcct:acctNumber>
            <listAcct:acctName>Checking</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>xxxxxxxxxxxxxxxxxxx</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Checking</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          <platformCore:record internalId="2" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>8001</listAcct:acctNumber>
            <listAcct:acctName>Savings</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>yyyyyyyyy</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Savings</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          .........
          .........
          .........
          .........		  
        </platformCore:recordList>
      </platformCore:searchResult>
    </searchMoreWithIdResponse>
  </soapenv:Body>
</soapenv:Envelope>

   

Getting Started

Now let’s check step by step instructions on how to load NetSuite data in SSIS. In this exercise we will focus on calling search API but you can call virtually any API using the same technique.

For detailed help on NetSuite SOAP API and other useful information check this documentation. To browse schema you can use this link. From schema browser we will use Search Type name later on in this article. For example later on in this article to search accounts we will use AccountSearch as below example. This name can be found from here(click search tab) You can change it to something else. For now dont worry too much until we get to that point.

Configure SSIS HTTP Connection for NetSuite

Now let’s look at how to configure SSIS HTTP Connection for NetSuite API login. We will use Dynamic Token Approach.

1. 1. Open SSIS Package
   2. Drag Data flow from SSIS Toolbox and double click it
   3. Inside data flow SSIS toolbox, find ZS XML Source and drag it on the surface.
   4. Double click it to configure. First enter NetSuite API URL. If you need to use latest WebService version e.g. NetSuitePort_2019_1 then you must use company specific endpoint.
      

      https://webservices.netsuite.com/services/NetSuitePort_2017_1

      –OR–

      To use

      NetSuitePort_2019_1

        or higher use company specific endpoint like below otherwise you will get error. NOTE: You must change Body in coming section to match Version if you use below way.

      https://111222333444.suitetalk.api.netsuite.com/services/NetSuitePort_2019_1

   5. Check Use Credentials option and Click New HTTP connection.
   6. Configure general properties of connections as below.
      1. Select Credentials Type as Dynamic Token
      2. Use any URL (It will be ignored)
      3. Enter your NetSuite UserID, Password
      4. Enter Header as Cookie
      5. Enter AUth Scheme as JSESSIONID=
   7. First Screen will look like as below. Click Configure.
      

      Configure SSIS NetSuite Connection

   8. On Dynamic Token > Request Tab we have to set Token URL, Method, Body, Header.
      1. In the URL enter same URL again
         

         https://webservices.netsuite.com/services/NetSuitePort_2017_1

      2. In the Body enter as below. Change APPLICATION-ID and ACCOUNT-ID with correct ID we obtained earlier. [$userid$] and [$password$] is special placeholders will be replaced automatically at runtime.
         NOTE: Try to match your URL version in the Body Request. If you used For example if you used 2019_1 in URL then change below body to match that version

         <?xml version="1.0" encoding="utf-8"?>
         <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
           <soap:Header>
             <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
               <applicationId>xxxxxxYOUR-APPLICATION-IDxxxxxx</applicationId>
             </applicationInfo>
           </soap:Header>
           <soap:Body>
             <login xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
               <passport>
                 <email xmlns="urn:core_2017_1.platform.webservices.netsuite.com">[$userid$]</email>
                 <password xmlns="urn:core_2017_1.platform.webservices.netsuite.com">[$password$]</password>
                 <account xmlns="urn:core_2017_1.platform.webservices.netsuite.com">xxxxxxYOUR-ACCOUNT-IDxxxxxx</account>
                 <role internalId="3" xmlns="urn:core_2017_1.platform.webservices.netsuite.com" />
               </passport>
             </login>
           </soap:Body>
         </soap:Envelope>

         NOTE: Remove below line from above Body if you get INVALID_ROLE error later in this tutorial. If if you get INVALID_LOGIN_CREDENTIALS error then refer here.

         <role internalId="3" xmlns="urn:core_2017_1.platform.webservices.netsuite.com" />

      3. Select Content Type as text/xml
      4. In the Headers click on Raw Edit Button and enter like below.
         

         SOAPAction: "login"

   9. Once all settings done your Request Tab will look like below
      

      SSIS NetSuite Connection – Configure Token Request

   10. On Response Tab configure like below. You have to enter Cookie name as JSESSIONID
       

       SSIS NetSuite Connection – Configure Token Response

   11. Click OK to Save Connection.
   12. We will configure XML Source in next section.

Configure SSIS XML Source to read NetSuite data

Now let’s continue configuration of XML Source. In this section we will show you how to get data without pagination. In next section we will see how to modify XML Source Settings to pagination request to read many records (e.g. 1000+ rows).

1. On your XML Source enter NetSuite Webservice URL like below.
   

   https://webservices.netsuite.com/services/NetSuitePort_2017_1

2. Now check Use Credentials and select previously created HTTP connection.
3. Select Method as  POST and content type as text/xml
4. Edit Request Body and enter like below (or any other XML body for API you like to call. Make sure SOAPAction in next step updated correctly too for other action)
   NOTE: Try to match your URL version in the Body Request. If you used For example if you used 2019_1 in URL then change below body to match that version

   <soapenv:Envelope 
      xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
      xmlns:urn="urn:messages_2018_1.platform.webservices.netsuite.com" 
      xmlns:urn1="urn:core_2018_1.platform.webservices.netsuite.com">
      <soapenv:Body>
         <urn:getAll>
            <record recordType="currency"/> 
         </urn:getAll>
      </soapenv:Body>
   </soapenv:Envelope>

5. NOTE: If you used NetSuitePort_2019_1 or higher version in URL then make sure you changed version  2017_1 to 2019_1 in below body
6. Now you click Raw Edit above headers grid and enter method name in SOAPAction (e.g. getAll in our case)
   

   SOAPAction: "getAll"

   NOTE: SOAPAction Header is very important so make sure you change it to match Action you like to call. You can Use Correct WebService WSDL File (Metadata)  in Tool like SOAPUI as explained here

   Version 2017_1 WSDL Link
   Version 2018_1 WSDL Link
   Version 2019_1 WSDL Link
   For Other Versions Just Change URL Version Part to different Number e.g. 2019_2_0

7. Now enter Filter like below or navigate by clicking Select Filter button.
   

   $.soapenv:Envelope.soapenv:Body.getAllResponse.platformCore:getAllResult.platformCore:recordList.platformCore:record[*]

   

   Configure XML Source – NetSuite SOAP API getAll (Body, Header, ContentType, Method, Filter)

8. Now click preview data.
   

   Preview NetSuite data in XML Source

9. Click OK to save XML Source UI
10. Now you can connect XML Source to target like OLEDB Destination and load NetSuite data into SQL Server.

Configure SSIS XML Source with Pagination

So in previous section we saw how to configure XML Source for basic API calls (without pagination). In this section we will see how to call more complex API (e.g. search ) which requires pagination. Some ideas about NetSuite Pagination discussed here too.

Foe pagination setup using XML Source we have to configure following properties via Properties Grid or Advanced Edit rather than Normal UI. May of these properties not available via UI as of now.

1. Create new data flow and drag new ZS XML Source.
2. Right click on XML Source and click Properties in Data flow designer. Find property by name in the Grid and set correct value.
   NOTE: In newer version v3.1.2 or higher you can use XML Source UI to enter some multi line properties for ease of use (e.g. FirstPageBodyPart and NextPageBodyPart can be entered on Advanced Pagination Tab, all other properties are easier to enter from Grid Mode if you sort alphabetically)

<search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <searchRecord xmlns:q1="urn:accounting_2017_1.lists.webservices.netsuite.com" xsi:type="q1:AccountSearch">
        <q1:basic>
          <description operator="doesNotContain" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
            <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">x</searchValue>
          </description>
          <name operator="contains" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
            <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">a</searchValue>
          </name>
        </q1:basic>
      </searchRecord>
    </search>

<searchMoreWithId xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <searchId>[$pagetoken$]</searchId>
      <pageIndex>[$pagenumber$]</pageIndex>
    </searchMoreWithId>

Once above items are set in property grid. Double click XML source to edit few more items from UI.

1. Enter URL as below
   

   https://webservices.netsuite.com/services/NetSuitePort_2017_1

2. Check Use Credentials and select HTTP connection we created in previous section.
3. Click Raw Edit button (above headers grid) and enter below
   

   SOAPAction: "search[$tag$]"

4. Click on Edit button next to Body. Enter following XML fragment for search request. Change ApplicationId and adjust PageSize if needed.
   

   <?xml version="1.0" encoding="utf-8"?>
   <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
     <soap:Header>
       <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <applicationId>ABCD-1234-XXXXXXXXXXXXXXX</applicationId>
       </applicationInfo>
       <searchPreferences xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <pageSize>100</pageSize>
       </searchPreferences>
     </soap:Header>
     <soap:Body>
   [$tag$]
     </soap:Body>
   </soap:Envelope>

5. Select POST method, change content type to XML (text / xml, charset=UTF-8)
6. That’s it. Click ‘Preview to confirm.
7. Click ok to save XML Source.
   

   Read NetSuite data in SSIS with Pagination ( search, searchMoreWithId API call)

Configure SQL Server Destination to load NetSuite data

Once you have XML Source configured you can connect XML Source to Target (OLEDB Destination) and load data into SQL Server.

Conclusion

In this post we explored many advanced options of HTTP Connection and XML Source to enable NetSuite API integration in SSIS. Download SSIS PowerPack today for FREE to uncover many other data integration scenarios not discussed in this article.

The post How to read data from NetSuite in SSIS (SimpleTalk SOAP API) appeared first on ZappySys Blog.

In this blog, we will learn how to call SOAP / REST API using Dynamic Token in SSIS (i.e. Two steps authentication approach – Fist Call Login API to get token and then call API).

In our previous blog post, we saw how to call Web API using some industry standards approaches, such as OAuth 2.0 Authorization or Basic Authorization. However, not all APIs follow the industry standard to authenticate API requests. After talking with many customers we noticed that very common non-standard technique to authenticate API is Dynamic Token Approach. In this approach first, you have to call Login request to get a temporary token (i.e. sessionid ). Once you get that temp token you can supply it along with your API requests. These temp token/session id usually expires in short time (i.e. expire in one hour or so).

Sounds complicated? Well not really if you use ZappySys SSIS PowerPack or ODBC Drivers for API. In this post, we will go through the steps, to authenticate using Dynamic Token approach. We will show you features of  ZappySys SSIS HTTP Connection Manager. For example purpose, we will call XML SOAP API using SSIS XML Source Connector but same concepts can be applied to read from JSON API using  SSIS JSON Source Connector or CSV API using CSV Source Connector.

So let’s get started.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS XML Source Connector and  SSIS JSON Source Connector that will help you to call SOAP / REST API, read various data from SOAP / REST API with SSIS.

To get token from login / authentication URL you have to gather a few pieces from your API documentation.

• How to call Login API for your API – (This is not the same URL as Vendor Site Login Page, but its usually API Login endpoint e.g. https://somecompany/api/v2/login)
• API Response Format (e.g. JSON, XML ,CSV or Random Text )

In our below example steps, we will use XML API which requires POST method first to pass Userid / password to authenticate. If Authentication is successful it may return a response in XML format which contains token.

In the next step, we will call other API (e.g. get-data) and pass token as Authorization Header. All these steps will be done automatically using  ZappySys SSIS HTTP Connection Manager.

Create HTTP Connection for Dynamic Token

To get API response data using SOAP / REST API call, you need to get an API Token from your login request URL. I am using ZappySys sample SOAP API to show you for the Demo purpose. Follow the steps mentioned below to get API Token:

Create in SSIS

1. Right Click on Connection Managers Window, and one Context menu appears for creating a new connection manager.
   

   Connection Manager Window: Create New Connection
2. Select ZS HTTP Connection Manager from the list of Connection Managers and, double-click on it or click on Add button to add a new Connection Manager. By Default, The created connection manager is of Package-Level. You can set Project Level Connection manager too If you want.
   

   List of SSIS Connection Managers – Select HTTP Connection Manager

Create in ODBC

To use HTTP Connection in ODBC Drivers you can change connection Type to HTTP like below and it will show similar UI discussed in this article.

Create an HTTP connection inside an ODBC driver

Configure HTTP Connection Credential Type

Once you add new SSIS ZS HTTP Connection Manager, Go to General Tab and Configure HTTP Connection Manager for Two Step Authorization Flow. Enter API URL, Login Credentials, and configure some basic settings for the same. We used Login URL as just example but you can enter any valid API URL endpoint here. Actual Login URL goes on Dynamic Token Tab (see next step)

SSIS ZS HTTP Connection Manager: General Tab Configurations Settings

Configure Request Settings

Once you enter Username / password details on General Tab now we need to setup some information like Login Request URL, Body, Headers. Lets look at few options.

Passing User ID /Password in Body

1. Go to Dynamic Token tab, Enter the following pieces.
   1. Token/API Login URL
      For our example enter API Login URL as:
      

      https://zappysys.com/downloads/files/test/soap-login.aspx

   2. Request Method as POST
   3. Request Body as below. Notice we used 

      [$userid$]

        and 

      [$password$]

        placeholders. These are replaced automatically at runtime so you do not have to hard code in Body and keep it secure. (See General Tab to enter UserID/Password)
      Pass User ID / Password in Body
      

      <soapenv:Envelope xmlns:soapenv=""http://schemas.xmlsoap.org/soap/envelope/"" xmlns:urn=""urn:partner.soap.sforce.com"">
         <soapenv:Body>
            <urn:login>
               <urn:username>[$userid$]</urn:username>
               <urn:password>[$password$]</urn:password>
            </urn:login>
         </soapenv:Body>
      </soapenv:Envelope>

   4. Select Request Body Content Type. For JSON data it may be application/json. For Key/Value Pair  key1=value1&key2=value2 leave it Default.Passing User ID /Password in Request BODY
      

      SSIS ZS HTTP Connection Manager: Dynamic Token Tab Configurations Settings of Request Tab

Passing User ID / Password in URL (GET Request)

Our Example API pass credentials in the POST body but some API may pass via URL as GET request.

In such case, you can enter like this

https://mysite.com/api/v2/login?userid=[$userid$]&password=[$password$]

[$userid$]

[$password$]

Passing User ID /Password in Header as Basic Authentication

Another scenario is API requires HTTP Basic authentication, (Pass User ID / Password in Header as Base64 encoded value) then you can pass credentials like this:

Authorization: Basic <<[$userid$]:[$password$],FUN_BASE64ENC>>


Here we have used FUN_BASE64ENC function but there are many other functions can be used anywhere in Body or Headers

Using Basic Authentication in Dynamic Token approach

Configure Response Settings

Now lets configure how to extract token (single value) from response. Which can be XML, JSON or sometimes Raw Text. We support 4 modes (default = None means full response text is treated as token). If you want to extract just single value and not full text then use below settings.

XML Expression Type (Use XPath)

1. Now, Go to Response Settings tab under a Dynamic Token tab, Enter Response Settings like Token Extract Mode, Expression Type, Content Filter Expression as below XPath Expression. The expression basically extracts content from sessionid XML node using below XPath Expression.

   //*[local-name()='sessionid']

   

   ODBC ZS HTTP Connection Manager: Dynamic Token Tab Configurations Settings of Response Tab

2. That’s it click on OK button to save your Connection Manager Configurations.

JSON Expression Type (Use JSONPath)

If you API returns JSON response (like Walmart API Example) then you can do something like below. Choose Expression Type=JSON and in Response Content Filter You can enter expression like $.some_property_name  (e.g. $.access_token). Check more about How to write JSON Path expression here.

SSIS ZS HTTP Connection Manager – Get a JSON response for the token

Regular Expression Type (Use Regex)

Another advanced mode to extract token from response is Regex Mode. You can write any valid regular expression like this one to extract just one value.

For example if your response returns raw string with token in double quotes but you want just token and no double quotes around it then try something like below.

Extract Token from response – Remove double quotes around value using Regular Expression

Call API in SSIS with Dynamic API Token

Now once you have setup HTTP connection manager to authenticate and get API token, we are ready for next step. Now we can use that token in SOAP / REST API Service call. So, let’s move to the SSIS Part.

1. Once SSIS Package is open, Goto Data Flow Section, For that Drag and Drop DataFlow Task from the SSIS Toolbox.
   

   Drag and drop Data flow task

2. Now, Drag and Drop SSIS ZS XML Source Component from the SSIS Toolbox (if your API is JSON based then drag ZS JSON Source).
   

   Drag and Drop ZS SSIS XML Source Component

3. Let’s double-click on XML Source Component to configure it’s Properties. Enter Request URL.
   

   https://zappysys.com/downloads/files/test/soap-getdata.aspx

   

   SSIS XML Source: Enter your SOAP Request URL

4. Check Use Credentials (Username/Password OR Token), Select HTTP Connection Manager from the drop-down list, we have created just before.
   

   ZS SSIS XML Source Component: Check Use Credentials and Select Connection Manager

5. Select the Request Method and Enter SOAP Request Body something like this. Notice that we used [$token$] as a placeholder. This is automatically replaced at runtime by ZS-HTTP Connection Manager.Passing Token in Body
   

   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:urn="urn:partner.soap.sforce.com">
      <soapenv:Body>
         <urn:sessionid>[$token$]</urn:sessionid>
      </soapenv:Body>
   </soapenv:Envelope>

   Passing Token in Header
   
   If your API expects token via HTTP Header (e.g. Authorization: xxxx-my-token-xxxxxxxx ) then make sure you configure Auth Scheme and Header name on HTTP Connection. By default it sends Token in Authorization header if you dont do anything.
   

   ZS SSIS XML Source Component: Enter SOAP Request Body

6. Select Content-Type as per your Request, here we are selecting

   text/XML;charset=UTF-8

     and adding 1 HTTP Header:

   SOAPAction: "https://zappysys.com/soap-getdata"

   

   ZS SSIS XML Source Component: Select Content-Type Add HTTP Request Header

7. select Filter Path: set as per your requirements. here for this example, we are setting:
   

   $.soapenv:Envelope.soapenv:Body.urn:Row[*]

   

   ZS SSIS XML Source Component: Select Filter

    

8. Check Include Parent Columns checkbox, for including parent Columns, and then click on Preview button to see the Preview data.
   

   SSIS XML Source Component: See the Preview data

9. That’s it. Click on OK button to Save your Configuration.
10. Finally drag any destination (i.e. OLEDB Destination) from your SSIS toolbox. If you want to load this data into SQL, then Drag OLE DB Destination Component, and configure it.

Conclusion

In this blog, we learned how to call any SOAP/REST API with Dynamic Token, retrieve Service response data with SSIS using XML Source Component OR JSON Source Component   in a very simple way. Furthermore, you can achieve many more functionalities with this tool. Check our blogs/articles on XML Source Component to find out what this tool is capable of more.

References

Finally, you can use the following links for more information about the call any SOAP / REST API with Dynamic Token feature with our tools:

• Landing Page for XML Source, you can also find Tutorial Video here.
• Help File: Documentation of XML Source.

The post How to call SOAP / REST API using Dynamic Token in SSIS / ODBC appeared first on ZappySys Blog.