In our previous article we saw how to POST data to REST API using few different ways. Now let’s go one step further and discuss another common scenario to batch REST API requests in SSIS. For efficient data transfer many API provides you to submit multiple records in a single request. In this article we will look at few ways to batch our requests using SSIS JSON Generator Transform and  SSIS Web API destination.

Method 1 – Batch records using SSIS JSON / XML Generator Transform

First let’s look at common way to group multiple records and generate single JSON or XML document for desired batch size. We will also see another technique where you can assign Unique BatchID to each document.

For example below screenshot shows how to product one JSON document for every 10 input rows. We have also use BatchID column on the root level. This will use use BatchID from first row of that batch. You can use XML Generator Transform if you like to produce XML Document rather than JSON.

To generate Unique BatchID you can use function like newid() in SQL Server.

SELECT newid() as BatchID,CustomerID,CompanyName FROM Customers

Using SSIS JSON Generator Transform with Batch Option (Multiple Records in a single document)

Here if full flow to batch your JSON Requests and Submit to REST API.

POST data to REST API in Bulk Mode (Batch multiple rows)

Method 2 – Using SSIS Web API destination Batch Mode with Header / Footer

Another method of batching is use Web API destination and turn on Batch mode.  You can use Fiddler to debug your web requests incase you are not sure how its generating and sending POST requests after this setting.

Configure SSIS Web API Destination for Bulk API call (Insert multiple rows)

Method 3 – Batch CSV records using CSV Generator Transform

So far we looked at how to batch JSON and XML format. Now let’s look at scenario where you have to POST CSV formatted data to REST API endpoint.

Assume that you have an API endpoint where you can submit data in multiple batches. Also you have few additional requirements

1. First row of First batch must have column names (any other batch must not contains CSV Header)
2. In each Batch we want to submit maximum 5 rows as per API call
3. Each batch must contain Unique Batch Number in URL

Here is high level flow.

Generate CSV using Batch Setting , Send data to REST API in multiple batches, Debug REST API in Fiddler

Include CSV Header in First Batch Only

Now let’s break down the problem and find out how to do. First problem is we need to include CSV Header only in First batch. To achieve this we use old school trick where we output column names as data row… and when we generate CSV in the next section we will turn off Column Headers option.

If you are consuming records from SQL Server or any other RDBMS you can write following style source query. If you don’t use relational source then use native UNION Transform in SSIS.

select 'CustomerID' as Column1, 'OrderID' as Column2
UNION ALL
select top 12 cast(CustomerID as varchar(1000)), cast(OrderID as varchar(1000)) from Invoices

Batch records and output CSV data

Now next step would be to generate CSV data but in batches… so in our example we have to output size of 5 rows per batch. For this perform the following steps

1. Drag ZS CSV Generator Transform (Included in v2.8 and Higher)
2. Connect source to CSV Generator Transform
3. Now before we edit in UI mode, right click and select Properties. Change following 3 properties  (You can also change QuoteAroundValue and Dimiliter if needed). In future version batch setting will be visible in UI mode too.
   1. ArrayBatchSize  = yourbatchsize (in our case it was just 5 rows per batch)
   2. OutputMode = SingleFileFormat
   3. FirstRowsHasColumnNames  =False
4. Now double click to configure ZS Generator Output
5. Right click on Mappings folder node and Add Elements > Select Multiple Columns  > Check columns you like to output
6. Click OK to Save UI

Generate CSV data in SSIS (Batch Output Setting)

Generate unique URL with Batch using Script Component

Now next task we have to achieve is generate Unique Batch Number. Put that Batch ID in URL like below…

http://httpbin.org/post?batch=0
http://httpbin.org/post?batch=1
http://httpbin.org/post?batch=2
.....

1. For this drag SSIS Script Component and select Transformation Mode
2. Now go to Inputs and Outputs tab > Select Output Columns > Click Add Column > Name it as URL > Configure DataType as DT_WSTR , Length = 500
3. Now got Script Tab and Click Edit. Remove old code and enter following code (Assuming You selected C#). This code will generate new URL with Unique batch ID each time. You can use this same technique to generate Unique row number too (in that case rather than DT_WSTR type you can use DT_I4 type)

   [Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
   public class ScriptMain : UserComponent
   {
       private static int _batch;
       public override void Input0_ProcessInputRow(Input0Buffer Row)
       {
           Row.Url = string.Format("http://httpbin.org/post?batch={0}",_batch);
           //If you have to send GUID rather than Integer for Batch ID then use below code
           //Row.Url = string.Format("http://httpbin.org/post?batch={0}",System.Guid.NewGuid());
   
           _batch = _batch+1;
       }
   }

4. Click OK to Save script.

Output Unique URL or Number using SSIS Script Transform

Configure Web API Destination to POST CSV Data

Now final step is to configure SSIS Web API Destination to POST CSV data to API.

1. Drag ZS Web API Destination and connect Source Transform to Web API destination.
2. Double click to configure
3. Select Connection Manager
4. Once connection setup, select Input Column for URL ( This will be same column from Script Transform)
5. Select column for Body (e.g. ZS_CSV_OUT)
6. Click OK to save

POST data to REST API in multiple batches using SSIS Web API destination

Debug Web API Requests using Fiddler

Now we ready to run entire flow. Once you execute data flow, observe Input rows before CSV Generator Transform and after. CSV Generator transform must produce less rows before we now grouping multiple rows in a single batch.

Another important aspect to play with API request is to debug it, We recommend you to to look at this article about using Fiddler to debug web requests.

Download Sample Package

Here is the Sample SSIS Package (2012 format) – Rest API_Batching

Conclusion

In this post, we saw how easy it is to handle complex REST API  / CSV /JSON / XML operations in SSIS. We configured REST API POST Operation using Batching Method. Hope this post helps you to achieve your API integration without coding.  Feel free to download SSIS PowerPack and try many other components not discussed in this article.

The post How to batch REST API requests in SSIS (Bulk Operation) appeared first on ZappySys Blog.

In our previous article, we saw step-by-step approach to read data from workday using SSIS. In this article, we will focus on how to load SQL Server data to Workday (e.g. POST, Create, Update). We will use SSIS Web API Destination and the combination of other Transforms such as SSIS Template Transform and SSIS XML Generator Transform .

The main thing in this article how to create SOAP Request correctly inside your request. This article explains how to use Free tools like SoapUI to create Workday Request. Once you do that you can call virtually any Workday API using the same technique. If you need more help contact our Support and we will be happy to help you step by step.

Step-By-Step : Import SQL Server data to Workday using SSIS

Lets build our SSIS Package to load data from SQL Server or any Source (e.g. Oracle, DB2) to Workday using SOAP API calls. Using below approach you can Create new records or Update existing records in Workday. You can also delete records with same concept.

Basic steps outlined below.

1. Fetch records from SQL Source and build XML Request for each Row (e.g. Create Account)
2. Build XML Request (SOAP Body) using Template Transform or  SSIS XML Generator Transform. If you have Array nodes (e.g. One to Many) then you have to use SSIS XML Generator Transform else use Template Transform for ease of use.
3. Pass input record (e.g. SOAP Body) to Web API destination to call Workday API call (CREATE, UPDATE, DELETE requests)
4. Parse XML Response (i.e. output) using SSIS XML Parser Transform or save raw XML to SQL Server database.
5. Redirect Bad rows or failed requests to log file for review

Now let’s look at each step in detail

Obtain Workday API URL

Once you have WSDL file, next step is craft correct URL for API service you like to call. The service name can be obtained from here (Check service column)

Syntax: https://<workday host name>.workday.com/ccx/service/<tenant name>/<service-name>
Example: https://MY-INSTANCE.workday.com/ccx/service/MY-TenantID/Human_Resources

Craft SOAP Body (XML API Request) using SoapUI

Now its time to craft some SOAP Request. Check steps outlined here (Use SoapUI tool) . Once you have Request Body XML you can change parameters as per your need.

Here is sample  SOAP XML Body for Get Employee call from Human_Resopurces service.

Creating SSIS Connection for Workday SOAP API call using WSS Security

Loading SQL Server data to Workday using SSIS

Let’s look at the real-world scenario. You have Accounts table stored in SQL Server and you like to create same accounts in Workday by calling appropriate API calls.

1. Drag Data flow task from SSIS Toolbox. Double click to edit.
2. Drag OLEDB Source configure to read SQL Table (e.g. Accounts)
3. Drag ZS Template Transform from the toolbox. Connect OLEDB Source to Template Transform. If you need flexible XML Generation then use XML Generator Transform but it may require some learning curve so for simplicity we are skipping that from this article.
4. Enter your SOAP request in the template text (like below) you like to call (This is obtained from the previous section – using tool like SoapUI)
   For Example: To create a new account you can use enter like below. Replace xxxxxxxxxx with Columns placeholder.
   To insert Column name as Placeholder click <<Insert Placeholder>> and then Select [Columns] node. Template Transform outputs column name TemplateOutput. You can use this as Body to feed next step (i.e. Call Workday API using Web API Destination )When you insert placeholder to make sure you use XML Encoded Columns if you expecting a Long text or special characters part of your data.
   Syntax for encoded value is   

   <%CustomerName,FUN_XMLENC%>

     . You don’t need FUN_XMLENC for numeric fields. For normal placeholder without encoding use just name with column placeholder indicators e.g.  

   <%Amount%>

   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:bsvc="urn:com.workday/bsvc">
      <soapenv:Header/>
      <soapenv:Body>
         <bsvc:Workday_Account_for_Worker_Add>
            <bsvc:Worker_Reference>
               <!--You have a CHOICE of the next 2 items at this level-->
               <bsvc:Employee_Reference>
                  <bsvc:Integration_ID_Reference>
                     <bsvc:ID>xxxxxxxxxx</bsvc:ID>
                  </bsvc:Integration_ID_Reference>
               </bsvc:Employee_Reference>
               <bsvc:Contingent_Worker_Reference>
                  <bsvc:Integration_ID_Reference>
                     <bsvc:ID>xxxxxxxxxxxx</bsvc:ID>
                  </bsvc:Integration_ID_Reference>
               </bsvc:Contingent_Worker_Reference>
            </bsvc:Worker_Reference>
            <bsvc:Workday_Account_for_Worker_Data>
               <!--type: string-->
               <bsvc:User_Name>xxxxxxxxxxxxxx</bsvc:User_Name>
            </bsvc:Workday_Account_for_Worker_Data>
         </bsvc:Workday_Account_for_Worker_Add>
      </soapenv:Body>
   </soapenv:Envelope>

   

   Template Transform –
   Create Workday SOAP Request – Create new records using SSIS

5. Drag ZS Web API Destination from SSIS Toolbox. Connect Template Transform to Web API Destination.
6. Configure Web API Destination as below (See we used same HTTP connection created in the previous section)
7. Select HTTP Connection
8. Select Input Column for Body (select TemplateOutput)
9. Select Body Content Type as XML (text/xml;charset=UTF-8)
10. Click Raw Edit above Headers grid. Enter following text (We just need one header)
    

    SOAPAction: ""

    

    SSIS Web API Destination –
    Loading SQL Server data to Workday (SOAP Call – Create, Insert, Update, Delete Records)

11. Now you can connect Web API Destination Output (Blue Arrow to Either XML Parser or some other Destination). Web API destination gives you a response in Raw XML Format. This article explains how to use JSON or XML Parser to parse API response coming from Web API destination.

Video Tutorial (See Part#4) – Create XML and POST data to SOAP Web Service or REST API

Now lets look at examples of creating XML from multiple data sources and POST XML request to SOAP Web Service URL or any other XML Based REST API URL. This video has 5 parts to cover full length tutorial but if you want to fast forward to see XML Generator Transform and Web API Destination then see part#4.

Sending complex SOAP Request (Nested Parent-Child Array)

If you have to call more complex SOAP request which requires Parent-child structure (Nested Array) then you cant use Template Transform. In that case, check this article for real-world example.

Create nested XML for SOAP Request – Multiple inputs (POST data to API using SSIS Web API Destination)

Extract single XML node from SOAP Response

There will be a time when you need to extract just one value out of your response and save into SSIS variable / use it later on. If that’s the case then refer to this article.

Conclusion

In this article we have learned how to load data from SQL Server to Workday using SSIS ( drag and drop approach without coding). Combination of few ZappySys Components (e.g. Web API Destination, XML Generator ) makes it super simple to call any Web API to load data from one system to API endpoint. Download SSIS PowerPack to try many other automation scenarios not discussed in this article.

The post Load SQL Server data to Workday using SSIS / SOAP API appeared first on ZappySys Blog.

In this post you will see how to generate XML which has namespace and prefix for elements. We will use SSIS XML Generator Transform for this purpose. Common pattern would be generate XML SOAP message using XML Generator Transform and send to Web service using SSIS Web API Destination. For full tutorial on how to export XML using SSIS check this link.

Understanding XML Namespace and Prefix

Just like how many programming languages offer concept called namespace to avoid similar name conflict for classes, XML also has namespace concept.

Sample XML with namespace and prefix

<?xml version="1.0"?>
<soapenv:Envelope xmlns:p1="http://someurl-1" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
	<soapenv:Body>
		<p1:CustomerID>CID1</p1:CustomerID>
		<p1:CompanyName>some company name</p1:CompanyName>
	</soapenv:Body>
</soapenv:Envelope>

If you notice in above XML there are two important things. Under <soapenv:Envelope> element you have xmlns:p1 and xmlns:soapenv attributes. Basically these are two namespace declarations and they both use different prefix (i.e. p1 and soapenv). Once you define namespace prefix you can use in the declaring element or any child element under that node.

Syntax for XML Namespace declaration:

xmlns:yourprefix="url or unique identifier"

Example – Using SSIS XML Generator Transform

Now lets look at how to generate XML using SSIS XML Generator Transform. This same technique can be used by SSIS Export XML Task.

1. Download and Install SSIS PowerPack from here (Skip this if you already did)
2. Open Visual Studio and create new SSIS Package
3. Drag Data Flow Task from Control Flow Toolbox
   

   Dragging and dropping Data Flow Task into Control Flow

4. Drag ZS Dummy Data Source (You can use any source you like. e.g. OLEDB Source)
5. Change Template to Customer to create fake customer data. Click OK to save.
6. Now drag and drop ZS XML Generator Transform
7. Connect Source to XML Generator
8. Double click XML Generator to configure.
9. Go to Options tab and change following setting
   1. under Namespaces define two namespaces as below (Change p1 to desired prefix)
      p1=http://someurl, soapenv=http://schemas.xmlsoap.org/soap/envelope/
   2. Change Root element name to Envelope
   3. Change Root Element Prefix to soapenv
      

      Configure SSIS XML Generator Transform – Define namespace and prefix , root element

10. Now go to layout tab and select dataset and click Edit
11. Change default Row element name to Body and prefix to soapenv as below. Click OK to save
    

    Configure SSIS XML Generator Transform – Change Row element name and prefix

12. Now right click on Mappings folder and Click “Add Elements” or click in the toolbar icon.
13. On the Add Elements dialog
    1. select Add multiple option.
    2. Change Prefix to p1 (or whatever you entered before on Options tab)
    3. Select columns you want to output in XML
       

       Configure SSIS XML Generator Transform – Add elements with prefix

14. Now you will see preview update for your XML layout as below
    

    SSIS XML Generator Transform – Preview of complex XML using namespace and prefix

15. Now attach your XML Generator Transform to some downstream component such as ZS Trash Destination and run the package

Conclusion

Using SSIS PowerPack you can generate complex XML in few clicks. Download SSIS PowerPack for FREE Trial and discover many more features.

The post How to generate XML with namespace / prefix in SSIS appeared first on ZappySys Blog.

In our previous blog post we saw how to import xml into SQL server using SSIS. In this post we will see how to generate xml in SSIS. If you wish to export JSON rather than XML then check this article

To produce complex nested XML you can use following two task/components.

	Export XML File Task
	XML Generator Transform

Video Tutorial – Create XML in SSIS

Below video tutorial will show you how to produce complex XML from single or multiple input datasets using SSIS.

Techniques illustrated in below video is almost identical for SSIS Export XML File Task and SSIS XML Generator Transform… but for demonstration we will use XML Generator Transform. If you wish to use Export XML File task then dataset will be Table or Query rather than Data flow Input.

Here is another video

SSIS XML Generator Transform

Use SSIS XML generator Transform when you want to produce XML inside data flow and feed to downstream system or save to XML file (You can use Flat File destination). If you have need for more flexible approach or save XML into SSIS Variable so you can POST XML data to Web API or SOAP webservice call then use Export XML File Task.

SSIS XML Generator Transform – Generate XML and save to file

SSIS XML generator transform – Create XML Output inside Data flow

SSIS Export XML File Task

When you have need to generate XML from multiple SQL query / tables and save into XML File or SSIS Variable then you can consider Export XML File Task. Few options available in Export File Task which are not found in XML Generator Transform such as Compression, Splitting options, Simple export without Layout.

SSIS Export XML File Task – Select Source Tables, Queries, Connection

SSIS Export XML File Task – Split file options (Split by Size, Row count and Group by column

SSIS Export XML File Task – Split file options (Split by Size, Row count and Group by column

Create Input Datasets

First step to product XML is define input datasets. If you have no parent child relation (e.g. Customer and Orders) then most likely you will need one dataset. If you are using XML Generator Transform then any attached inputs to your transform becomes datasets. First dataset is automatically set to Root dataset.

For Export XML File task you have to define datasets manually. You need to pick connection manager and then select Table or Query. To make query dynamic you can use Variable Placeholder

select * from MyTable Where Month='{{User::varMonthName}}'

SSIS Export XML File Task – Layout Editor (Define nested elements and Parent-Child relation)

Create XML Elements /Attributes

XML Element has start and end tags (end tag is optional if no children elements found). To add column as Element you can click on Add Element option.

Example of Elements

<rootelement>
  <rowelement><prop1element>AAA</prop1element></rowelement>
  <rowelement><prop1element /></rowelement>
  <rowelement><prop1element>CCC</prop1element></rowelement>
</rootelement>

SSIS Export XML File Task – Layout Editor Right Click Menu

Create nested XML elements (indentation)

When you need nested element (e.g. <LocationInfo><ZipCode>xxxxxx</ZipCode></LocationInfo> ) then Add element and check Unbound option on the UI (e.g. add LocationInfo as Unbound).

Setting Advanced Option

In this section you will learn how to set many advanced options for XML generation.

SSIS Export XML File Task – Select XML Format Options

Handling NULL in XML output

By default NULL attributes or elements not included in output because its hard to identify Blank string vs NULL string. If you wish to explicitly flag NULL items in output then you may uncheck Do not include NULL option. By doing so you will notice two changes. In your root node xmlns:xsi attribute is added to add standard namespace and when element is NULL you will see xsi:nil=”true” in the self closing element tag. This indicates that column data is NULL. If you see self closing tab without xsi:nil=”true” then its Blank value rather than NULL.

Define XML Namespace (Prefix)

By default no XML namespace is defined in the XML output. But if you wish to include any attribute or element with prefix then it requires namespace declaration. For more information on how to create XML with namespace check this article.

Consider the following example. It has some prefix called ns1 for element and ns2 for attribute. To achieve such format make sure you define two namespaces on XML Options tab (e.g. ns1=http://someurl1,ns2=http://someurl2 ). Once you do this ns1 and ns2 can be used as prefix for any element or attribute.

Example of Namespace/Prefix use

<data xmlns:ns1="http://someurl1" xmlns:ns2="http://someurl2" >
  <ns1:row ns2:id="111"><prop1>AAA</prop1></ns1:row>
</data>

Using XML Namespace Prefix for Element or Attributes

In order to use namespace you have to set Prefix property for any Element or Attribute when you Add them. For example to add element <row> with prefix ns1 (e.g. <ns1:row> ) you have to set prefix to ns1 on the Add/Edit Element UI. Before you can use Prefix make sure namespace is defined as per previous section.

Removing XML declaration

Some times you have requirement to omit XML declaration (e.g. first line  <?xml version=”1.0″ encoding=”utf-8″?> ). For that check “Do not output XML declaration” option on XML options tab.

Remove Invalid XML Characters

If you ever encounter invalid XML characters and want to remove them then check this article

Change / Remove Root Element from output

By default XML document has root element called “Root” (See XML Options tab to change this). You can change this to custom name or  remove it if you wish to hide start and end tag of root element from output.

Change / Removing XML Row Element from output

By default row of dataset will start with default Row element (e.g. <DatasetRow>… </DatasetRow>). if you wish to hide those start and end tag then make sure you remove default Row element name in Dataset Declaration (Edit dataset and remove Row Element name)

Define Parent Child Join in XML datasets (One to Many)

If you have parent child relationship in XML (e.g. each Customer Tag has multiple Orders) then you need to define at least two datasets (One for parent (i.e. Customer) and one for Child (i.e Orders)).

Define Parent to Child Relationship for XML Output (One to Many JOIN)

Making things Dynamic (Query Parameter / Layout Information)

To make things dynamic you can insert variable placeholders anywhere in the editor (Look for {{Insert variable}} menu item in the editor). If you want to make entire Layout dynamic then use option to Load Layout from File or Variable. If this option is not available then consider SSIS expression to make it dynamic
Example:

select * from MyTable Where Month='{{User::varMonthName}}'

Controlling XML Date Time formatting (How to use ISO date)

By default date/time values are exported as yyyy-MM-dd HH:mm:ss format but if you wish to change so its exported to ISO format or other format then goto XML Options tab and set Date format to yyyy-MM-ddTHH:mm:ss for ISO. If Datetime is UTC then change format to yyyy-MM-ddTHH:mm:ssZ (append “Z” at the end)

Conclusion

Creating nested XML may not be easy task but innovative drag and drop approach of SSIS PowerPack certainly makes it easy to produce complex XML with every possible option on hand. Try SSIS PowerPack for free (45+ tasks and components) and explore what else you can do with it.

The post How to export XML from SQL Server using SSIS appeared first on ZappySys Blog.