In this post we will learn how to read Freshdesk data in SSIS using ZappySys JSON / REST API Source. We will see how to authenticate, paginate / parse and load Freshdesk data into SQL Server Table in just few clicks. If you don’t use Freshdesk and considering using Zendesk instead then check our previous post on how to Read Zendesk data in SSIS.

Freshdesk is a cloud based / managed service like Zendesk to for Support / Customer service.

Getting Started

So let’s get started. We will see step by step instructions to configure Freshdesk API connection, setup JSON Source to read data from Freshdesk and finally load data into SQL Server table. We will use List Tickets API in this article for demo purpose but read other useful information on that page if you are new to REST API.

Setup Freshdesk REST API Connection

Very first step to call any Freshdesk API in SSIS is to configure HTTP connection (Basic Authentication – Using UserID/ Password).

1. Open Visual Studio (i.e. SSDT) to create new SSIS Package
2. From Control Flow SSIS Toolbox drag REST API task
3. Select URL From HTTP Connection Option from Drop down
4. Click New HTTP Connection
5. Configure HTTP connection like below.
6. Enter any Valid URL (This will be ignored if you Override URL on REST API Task)
7. Select Basic Authentication (Use User ID and Password)
   

   Configure Freshdesk REST API Connection in SSIS – Basic Authentication (User ID / Password)

8. Click OK to close HTTP Connection
9. On Main UI of REST API Task, check Use Direct URL and enter below URL (Change YourDomain Part)
   https://{your-domain}.freshdesk.com/api/v2/tickets
10. Click Test Request / Response. If you see OK status then you have successfully called Freshdesk API. That’s all.
    

    Calling Freshdesk REST API in SSIS ( Read Tickets example)

In the section we will see how to Parse REST API response and load data into SQL Table.

Read Freshdesk data in SSIS

Now let’s look at steps to configure SSIS JSON Source to read from Freshdesk REST API.

1. Drag Data Flow task from toolbox. Double click to go to Data Flow designer
2. Drag ZS JSON Source (For File / REST API)
3. Configure JSON Source like below.
4. Check Use Connection option and select HTTP connection created in the previous section.
5. Enter API URL from which you like to read data. In out example we want to read tickets created after 2019-01-01 so we will call API like below. You can skip ?updated_since=2019-01-01 if you like to read all tickets.
   

   https://{your-domain}.freshdesk.com/api/v2/tickets?updated_since=2019-01-01

6. You can also Use Variable Placeholder in URL to make things dynamic like below
   

   https://{your-domain}.freshdesk.com/api/v2/tickets?updated_since={{User::MyDateVar,yyyy-MM-dd}}

7. Click Preview to see sample data.
   

   Read Freshdesk data in SSIS – Fetch Tickets example (With REST API pagination)

8. Configure Pagination like below. Basically you have to select Pagination Mode = URL Parameter Mode. Enter Page num indicator as 

   page

   

   Configure REST API pagination (URL Parameter Mode)

9. Thats it. Now click OK to save UI.

Load Freshdesk data into SQL Server Table

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

In this article we saw how easy it is to read data from any REST API service like Freshdesk REST API and load into Relational DB like SQL Server. Try SSIS PowerPack for FREE and explore many other 70+ Tasks /Components not discussed in this article.

The post Read Freshdesk data in SSIS – REST API Call appeared first on ZappySys Blog.

In this post we will learn how to access data from Oracle UCM Web Service (Middle layer for WebLogic) and load into SQL Server or any other target. We will use SSIS XML Source to achieve this result.

About Oracle UCM Web Service

If you are not sure what is SOAP Web Service and how to call in SSIS, First read this article to understand general concepts of calling SOAP API in SSIS.

UCM Web Service offers calling various commands. You have to send XML Request and server will send you XML response with requested data. We will use Array Transformation Technique (Key / Value Pattern) to extract complex nested XML data.

Step-By-Step

Now let’s look at how to call Oracle UCM SOAP API in SSIS.

Read from Oracle UCM Web Service using XML Source

1. Drag Data Flow Task in SSIS Designer
2. Inside Data flow drag ZS SSIS XML Source from toolbox
3. First enter URL as below (Replace INSTANCE ID with your own instance)
   

   https://YOUR-INSTANCE-ID.oraclecloud.com/idcws/GenericSoapPort

4. Now Check Use Credentials and select new ZS-HTTP connection
5. Enter your User ID and Password to call Web Service using Basic Authentication. Click OK to save.
   

   Configure HTTP Connection – Call Oracle UCM Web Service (SOAP XML)

6. On XML Source UI Select Content Type = XML (text / xml, charset=UTF-8)
7. Select Request Method as POST
8. Click Enter Raw Edit for HTTP Headers and enter below
   

   SOAPAction: "urn:GenericSoap/GenericSoapOperation"

9. Click on Edit Body and enter your Request Body in XML format.For example to call GET_SEARCH_RESULTS service to retrieve the search results for the passed query text you can submit Request Body as below.
   NOTICE:  String in Search Text was surrounded by backtick (  `  ). Single tick / double quote didn’t work in our case.
   

   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ucm="http://www.oracle.com/UCM">
   <soapenv:Body>
     <ucm:GenericRequest webKey="cs">
       <ucm:Service IdcService="GET_SEARCH_RESULTS">
         <ucm:Document>
           <ucm:Field name="QueryText">dDocTitle &lt;starts&gt; `MY_sites_201`</ucm:Field>
         </ucm:Document>
       </ucm:Service>
     </ucm:GenericRequest>
   </soapenv:Body>
   </soapenv:Envelope>

   
10. Now Select Filter (Browse upto Fields under SearchResult node. Once you click OK try to edit your Expression so it looks like below. As you may notice we changed ResultSet[*] to ResultSet[?(@@name==’SearchResults’)]
    This will ensure that we extract records from only SearchResult Node and not other nodes such as UserAttribInfo or EnterpriseSearchResults
    

    $.env:Envelope.env:Body.ns2:GenericResponse.ns2:Service.ns2:Document.ns2:ResultSet[?(@@name=='SearchResults')].ns2:Row[*]

    

    Select Filter

11. Now go to 2D Array Tab. Select Key / Value Pattern Option.
12. Select Column Names / Value Filters as below. You can Use Browse Option to navigate to that field or just type below directly. Below expression will Transform Rows into Column. Foreach attribute name we will get one column.Column Name Filter =>  $.ns2:Field[*].@name
    Column Value Filter =>  $.ns2:Field[*].#text
13. Now Click Preview to confirm it works
    
14. That’s it, you can now click OK to close UI.

Loading Oracle UCM SOAP API data into SQL Server / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

In this post we saw how easy it is to consume XML SOAP Web Service in SSIS. We called SOAP Service for Oracle UCM Web Service and loaded data into target like SQL Server. You can download SSIS PowerPack and try many other scenarios like this using 70+ other SSIS Tasks /Components not mentioned in this article.

The post Call Oracle UCM Web Service in SSIS (Read XML SOAP API) appeared first on ZappySys Blog.

In this post we will lean SSIS Magento data read / write operations. Magento is a very popular eCommerce platform and they offer JSON based REST API and XML based SOAP API. You can use either API based on your need to automate common integration needs.

We recommend using REST API (JSON API) if possible because they are simpler and faster. We will focus only on JSON API in this article but we do have blog post on how to call SOAP API.

About Calling Magento REST API

If you are new to Magento API we highly recommend to get familiar with Magento REST API here. Magento REST API  offers two ways to authenticate.

1. Token based Authentication
2. OAuth Based Authentication (OAuth 1.0a)

In this article we will look at only Token based approach. In token based approach we have to get new token using admin / customer account information.

We will cover later how to access admin or customer level API later in this article (See HTTP connection configuration).

Configure Connection / Call Magento REST API in SSIS

Now lets look at very simple example how to use Token based method to call Magento REST API in SSIS. We will first configure HTTP connection and then call REST API.

Let’s get started.

1. First, open SSIS Package after installing SSIS PowerPack
2. Drag ZS REST API Task from SSIS Toolbox on Control Flow Designer. Let’s rename task to Get Magento Categories
3. Enter REST API URL you like to call. For example we can use below URL. Replace **your-magento-host** with your own name.
   

   http://**your-magento-host**/index.php/rest/V1/categories

4. Select Url from Connection mode
5. From Connection Dropdown select new ZS-HTTP connection
   

   Create new connection for SSIS REST API Task

6. Configure General Tab of HTTP Connection like below
   URL can be anything for now because we will use override option in next step (On REST API Task).
7. Select Credentials Type as Dynamic Token.
8. Enter Userid / password
   

   Configure Magento REST API Connect for Token based Authentication

9. Now goto Dynamic Token Tab (Or click Configure Link) and configure like below.
10. Enter Token URL for API Login URL field.
    NOTE: If you are a customer and want to access your own account (e.g. view your orders) then replace /admin with /customer  in URL)
    Replace **my-host** with your host name.
    

    http://***my-host***/index.php/rest/V1/integration/admin/token

11. Change Method to POST.
12. Enter Body as below (Placeholders are automatically replaced when /token endpoint is called.
    

    {"username":"[$userid$]", "password":"[$password$]"}

13. Select Content Type as Application/Json
    

    Configure Magento Token Request

14. Now click Dynamic Token – Response Tab and configure like below.
    Select Regex and enter Expression as below. This will remove double quotes around token in response.
    

    "(.*)"{{0,1}}

    

    Extract Token from response – Remove double quotes around value using Regular Expression

15. Click OK to save connection
16. Now on REST API Task check Use Direct URL Option
17. Click Test Request to confirm its working
    

    Call Magento REST API using SSIS REST API Task

Read data from Magento and Load into SQL Server

Now lets look at how to read Magento data and load into SQL Server. Assume that you like to read all products.

Configure JSON Source (Read Magento REST API)

1. Drag Data flow and double click to edit:
   

   Dragging and dropping Data Flow Task into Control Flow

2. Drag ZS JSON Source on data flow and double click to edit
3. Configure JSON Source like below if you like to get product list for example
   1. Enter URL as below (change http to https if your host support secure channel).
      

      http://***yourhost***/index.php/rest/V1/products?searchCriteria[pageSize]=1000
      --- to extract only specific fields do below way --
      http://***yourhost***/index.php/rest/V1/products?searchCriteria[pageSize]=1000&fields=items[sku,name]

   2. Check Use Credentials. Use same HTTP connection we created in previous section
   3. In the Filter enter $.items[*] or click Select Filter to Browse and select items node.
   4. Click Preview to confirm
4. Here is how your setup will look like after configuration.
   

   SSIS JSON Source Configuration – Read data from Magento REST API

5. If you expecting more than 1000 rows then you have to set up pagination. Now go to pagination tab and configure as below (Below options are only available in v2.9.1 or higher). If you have older version then check next section for workaround.
   1. Pagination Mode =URL Parameter Mode
   2. Page Num Indicator = searchCriteria[currentPage]
   3. Max Rows Expression = $.total_count
   4. Page Data Expression = $.items[*]
6. Here is how your Magento REST API Pagination Setup will look like.
   

   REST API Pagination using Max Row count Mode

7. Click OK to save JSON Source. In next section we will see how to load data into Target like SQL Server.

Configure OLEDB Destination (Load into SQL Server)

1. Drag OLEDB Destination.
2. Select / Create Connection
3. Select or create NEW target Table (Click NEW button)
4. Click Mapping Tab to map source columns to target table
5. Click OK to save
6. Execute Package

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Configure Magento Pagination (For Old Version)

In previous section we configured pagination using newer version (v2.9.1 or higher). But what if you have older version and you dont see that feature. Use below steps for workaround.

Now let’s look at another common scenario when you real lots of data. Let’s say you have 5000 products but API call we saw earlier returns max 1000 rows. In that case we have to keep reading next page until all rows are returned. ZappySys provides many settings to configure various REST APIs out there, unfortunately there is no standard around pagination methods. Since Magento API has some bug in API pagination we have to do following hack. Magento keep sending same data of last page if you try to access page which doesn’t exists. Here is how to overcome that bug.

Configure REST API task to get total row count

First step to configure magento pagination is get total rows returned in our request which you like to paginate (in our case /products).

1. Go to Control Flow designer
2. Drag ZS REST API Task from SSIS Toolbox
3. Let’s rename task to Get Magento Product Count
4. Double click it and configure like below (Change HTTP to HTTPS if needed)
   

   http://***yourhost***/index.php/rest/V1/products?searchCriteria[pageSize]=10

   
5. Go to Response Tab and configure like below. Enter expression as $.total_count
6. Click OK to save

Define Expression for JSON Source – MaxRows Property

Now next step is to define expression on MaxRows property of JSON Rows so we stop once all rows consumed from Paginated response. If you don’t do this it will keep paginating forever 🙁

1. On Control Flow designer select Data Flow which contains JSON Source for Magento
2. Right click and go to Properties of that data flow
3. Find Expression and click Button to add new expression
4. On Expression dialogbox select [Your JSON Source].[MaxRows] and for expression enter variable name which holds total count (e.g. @[User::MaxRows] )
5. Click OK to Save

Configure JSON Source for Magento Pagination

Now last thing we have to do is go to JSON Source and configure few things for pagination.

1. Go to data flow and double click JSON Source
2. Click on Pagination tab and select Pagination by URL Parameter Mode
3. Enter Page name as below
   

   searchCriteria[currentPage]

4. Click OK to save UI
5. Now run entire page to test (NOTE: Do not execute just data flow because if you do that way, it will not extract total row count)
   

Write data to Magento using SSIS Web API destination

Now let’s look at example to load data into Magento. In below example we will create few sample products.

1. Create new data flow and go to data flow designer
2. Drag and drop ZS CSV Source from SSIS toolbox. You can use any Data source but for simple demo we will use it.
3. Double click CSV source and select Direct Value mode from dropdown. Enter following sample data (Lets create 12 sample products.)
   

   SKU,ProductName
   PROD-1,Product 1
   PROD-2,Product 2
   PROD-3,Product 3
   PROD-4,Product 4
   PROD-5,Product 5
   PROD-6,Product 6
   PROD-7,Product 7
   PROD-8,Product 8
   PROD-9,Product 9
   PROD-10,Product 10
   PROD-11,Product 11
   PROD-12,Product 12

4. Now click OK to save UI
5. Now drag ZS Template Transform from toolbox.
6. Connect CSV Source to Template Transform
7. Enter following sample text to build request for new product. Refer to Magento API help file / tutorial to learn more.
   

   {
     "product": {
       "sku": "<%SKU%>",
       "name": "<%ProductName%>",
       "attribute_set_id": 4,
       "price": 25,
       "status": 1,
       "visibility": 1
      }
   }

8. Notice that when you click Insert Variable > Columns > select desired upstream column placeholder. You can also encode special characters in your data (e.g. new lines) using function like <%MyColumn,JSONENCODE%>
9. Click OK and save UI
10. Now drag ZS Web API Destination from SSIS Toolbox. Configure like below.Enter URL same as before (used to read product), change Method to POST, change content type to Application/JSON.
11. Now run entire flow (You can watch your API requests in Fiddler for debugging)

Debug Magento Web Requests using Fiddler

Click on below article to learn how to debug web requests

How to use Fiddler to analyze HTTP Web Requests

Magento Integration in Other Apps (e.g. Power BI / Informatica / SQL Server)

You can use techniques listed on this page with ODBC Drivers too for integration in other tools like Power BI, SQL Server code, Informatica etc.

Conclusion

In this post we saw how easy it is to perform Magento Integration in SSIS without doing any programming. In few steps you can read or write data in Magento using SSIS PowerPack REST API components. Download FREE Trial and explore many other scenarios.

The post SSIS Magento data Read / Write using REST API Call appeared first on ZappySys Blog.

Splunk is commonly used for searching, monitoring, and analyzing machine-generated big data, via a Web-style interface. In this post, you will learn how to implement Splunk API Integration with SQL Server or any other RDBMS (e.g. Oracle, MySQL, Postgresql) using SSIS in a few clicks. We will use SSIS XML Source to Read data from Splunk and Load into SQL Server / other targets (Using HTTP Connection).

We will discuss on How to Create an Intuit Developer Account, How to Create QuickBooks Online App for OAuth, We will also discuss reverse scenario to Write data to Splunk (API POST for Insert or Update in Splunk) using SSIS REST API Task

In nutshell, this post will focus on how to call Splunk API using SSIS.
So let’s get started.

What is Splunk

Splunk is a software platform to search, analyze and visualize the machine-generated data gathered from the websites, applications, sensors, devices etc. which make up your IT infrastructure and business.

The other benefits of implementing Splunk are:

• Input data can be in any format for e.g. .csv, or JSON or other formats
• Give Alerts / Events notification at the onset of a machine state
• Accurately predict the resources needed for scaling up the infrastructure
• Create knowledge objects for Operational Intelligence

Why Use Splunk REST API

So there will be a time when you want to automate certain things without using Splunk Web Portal. Here are some example scenarios which can be solved using the Splunk REST API.

• Reading/Update Splunk configuration files
• Creating Splunk searches using the REST API
• Create a new Splunk object for a specific context
• Edit a Splunk object

Step-By-Step – Import Splunk data into SQL Server

In order to start, we will show several examples. ZappySys includes an SSIS XML Source that will help you to call Splunk API, read search job results data from Splunk with SSIS, create a new Saved Search on Splunk with SSIS and do many more actions using REST API. To learn more about Splunk REST API check this help file.

To get Splunk data using the REST API call, you need to have a Free or Licensed Splunk Account. You can Download Splunk Enterprise or Register with Splunk see this link. You can find the videos for How to Install and Getting Data in on this link: Splunk Enterprise Videos.

Create / Configure Splunk Dataset

So first step in our demo would be make sure we have at least one Splunk Dataset which we can query using REST API. If you have already configured Dataset then skip this section.

1. First of all, let’s start the Splunk on your machine from the program menu.
   

   Start Splunk

2. Hence, Splunk opens into the browser. So let’s logged in and let’s Add Data for WinEvents to be searched in our next phase.
   

   Add Data

3. To start adding data first, choose a data source. For example, you can monitor WinEvents.
   

   Select Data Source

4. Select Search & Reporting in App Context and click next for review and click next to finish.
   

   Select Search & Reporting

5. That’s it local event logs input has been created successfully for searching.
   

   Start searching for local events

6. Therefore you will redirect to search page.
   

   Splunk search page

7. So, let’s import WinEvents search data into SQL Server in the next few sections.

Connect to Splunk in SSIS / Create Search JOB / Obtain JobID

Now once you have Splunk Dataset configured, next step is Call Splunk API. So most common thing you will do with splunk API is to call splunk search API. In splunk Search API is Job Style API means you perform following steps

1. First call /services/search/jobs/ endpoint with Search query (in POST) to create search Job. It returns you Job ID (also known as SID)
2. Wait until Job is done (This part may be tricky) … SO you may have to add timer task after first step and in the 3rd step enable Retry option on HTTP connection just incase it didnt finish within supplied delay.
3. Once Job finished you can read by calling below API endpoint.
   /services/search/jobs/{{your Job Id}}/results?output_mode=json

Thats it now let’s look at actual steps

1. Now let’s make the first call to search job using a POST method. If you are using a custom time range, pass it in with the POST request. Refer this link for more information: Export data using the Splunk REST API. If your URL is different than below HTTP connection then Check Use direct URL option on REST API Task and enter custom URL.
   

   URL:
   https://localhost:8089/services/search/jobs/
   
   Post Reuest Data:
   search=search source="WinEventLog:*" host="SCIFI08" earliest=-1d&output_mode=json

   

   Splunk POST Search request

2. So let’s get Response SID(Search job ID) into the variable to be used in the next call to get results.
   

   Store Splunk Response Search Job ID in a variable

3. Now add the Data Flow Task to retrieve Splunk search result data using JSON Source (REST API or File) in it. In below URL we are specifying count=5 but in real world change it to 100 or higher.
   

   TO Get Metadata deatils:
   https://localhost:8089/services/search/jobs/1549701697.1424/results?output_mode=json&offset=0&count=5

   

   Splunk Search JSON Results Data

4. Furthermore, Let’s set pagination to get all the results data.
   NOTE: In below screenshot we are specifying count=5 and Increment=5 just to show demo but in real world set both items to higher value (e.g. 100) so you request more records per page.
   

   Set Pagination

5. Let’s preview the results data.
   

   Preview Splunk Search Data

6. Edit JSON Source again and replace the below URL with it to get results By SID(Search Job ID) variable. And create the new HTTP connection as if the search job(SID) is newly created, it is possible that we will #get 204s(No Content) until the job is ready to respond.
   

   https://localhost:8089/services/search/jobs/{{User::v_sid}}/results?output_mode=json&offset=0&count=5

   

   204 No Content response error handling

7. Furthermore, click on OK button and make sure we need to click on NO.
   

   Splunk URL With variable

8. Finally, we are ready to Load Splunk data into the SQL Server.

Load Splunk data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

Above all, in this blog, we learned how to read Splunk data in SSIS. Furthermore we used XML Source Component and SSIS REST API Task to call Splunk REST API and load data into SQL server. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information about the use of Splunk Online REST API with our tools:

• REST API Task, you can also find Tutorial Video here.
• XML Source Component, you can also find Tutorial Video here.
• Help File: Documentation of REST API Task and XML Source Component.

The post How to read data from Splunk in SSIS appeared first on ZappySys Blog.

In this blog, we will learn how to call SOAP / REST API using Dynamic Token in SSIS (i.e. Two steps authentication approach – Fist Call Login API to get token and then call API).

In our previous blog post, we saw how to call Web API using some industry standards approaches, such as OAuth 2.0 Authorization or Basic Authorization. However, not all APIs follow the industry standard to authenticate API requests. After talking with many customers we noticed that very common non-standard technique to authenticate API is Dynamic Token Approach. In this approach first, you have to call Login request to get a temporary token (i.e. sessionid ). Once you get that temp token you can supply it along with your API requests. These temp token/session id usually expires in short time (i.e. expire in one hour or so).

Sounds complicated? Well not really if you use ZappySys SSIS PowerPack or ODBC Drivers for API. In this post, we will go through the steps, to authenticate using Dynamic Token approach. We will show you features of  ZappySys SSIS HTTP Connection Manager. For example purpose, we will call XML SOAP API using SSIS XML Source Connector but same concepts can be applied to read from JSON API using  SSIS JSON Source Connector or CSV API using CSV Source Connector.

So let’s get started.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS XML Source Connector and  SSIS JSON Source Connector that will help you to call SOAP / REST API, read various data from SOAP / REST API with SSIS.

To get token from login / authentication URL you have to gather a few pieces from your API documentation.

• How to call Login API for your API – (This is not the same URL as Vendor Site Login Page, but its usually API Login endpoint e.g. https://somecompany/api/v2/login)
• API Response Format (e.g. JSON, XML ,CSV or Random Text )

In our below example steps, we will use XML API which requires POST method first to pass Userid / password to authenticate. If Authentication is successful it may return a response in XML format which contains token.

In the next step, we will call other API (e.g. get-data) and pass token as Authorization Header. All these steps will be done automatically using  ZappySys SSIS HTTP Connection Manager.

Create HTTP Connection for Dynamic Token

To get API response data using SOAP / REST API call, you need to get an API Token from your login request URL. I am using ZappySys sample SOAP API to show you for the Demo purpose. Follow the steps mentioned below to get API Token:

Create in SSIS

1. Right Click on Connection Managers Window, and one Context menu appears for creating a new connection manager.
   

   Connection Manager Window: Create New Connection
2. Select ZS HTTP Connection Manager from the list of Connection Managers and, double-click on it or click on Add button to add a new Connection Manager. By Default, The created connection manager is of Package-Level. You can set Project Level Connection manager too If you want.
   

   List of SSIS Connection Managers – Select HTTP Connection Manager

Create in ODBC

To use HTTP Connection in ODBC Drivers you can change connection Type to HTTP like below and it will show similar UI discussed in this article.

Create an HTTP connection inside an ODBC driver

Configure HTTP Connection Credential Type

Once you add new SSIS ZS HTTP Connection Manager, Go to General Tab and Configure HTTP Connection Manager for Two Step Authorization Flow. Enter API URL, Login Credentials, and configure some basic settings for the same. We used Login URL as just example but you can enter any valid API URL endpoint here. Actual Login URL goes on Dynamic Token Tab (see next step)

SSIS ZS HTTP Connection Manager: General Tab Configurations Settings

Configure Request Settings

Once you enter Username / password details on General Tab now we need to setup some information like Login Request URL, Body, Headers. Lets look at few options.

Passing User ID /Password in Body

1. Go to Dynamic Token tab, Enter the following pieces.
   1. Token/API Login URL
      For our example enter API Login URL as:
      

      https://zappysys.com/downloads/files/test/soap-login.aspx

   2. Request Method as POST
   3. Request Body as below. Notice we used 

      [$userid$]

        and 

      [$password$]

        placeholders. These are replaced automatically at runtime so you do not have to hard code in Body and keep it secure. (See General Tab to enter UserID/Password)
      Pass User ID / Password in Body
      

      <soapenv:Envelope xmlns:soapenv=""http://schemas.xmlsoap.org/soap/envelope/"" xmlns:urn=""urn:partner.soap.sforce.com"">
         <soapenv:Body>
            <urn:login>
               <urn:username>[$userid$]</urn:username>
               <urn:password>[$password$]</urn:password>
            </urn:login>
         </soapenv:Body>
      </soapenv:Envelope>

   4. Select Request Body Content Type. For JSON data it may be application/json. For Key/Value Pair  key1=value1&key2=value2 leave it Default.Passing User ID /Password in Request BODY
      

      SSIS ZS HTTP Connection Manager: Dynamic Token Tab Configurations Settings of Request Tab

Passing User ID / Password in URL (GET Request)

Our Example API pass credentials in the POST body but some API may pass via URL as GET request.

In such case, you can enter like this

https://mysite.com/api/v2/login?userid=[$userid$]&password=[$password$]

[$userid$]

[$password$]

Passing User ID /Password in Header as Basic Authentication

Another scenario is API requires HTTP Basic authentication, (Pass User ID / Password in Header as Base64 encoded value) then you can pass credentials like this:

Authorization: Basic <<[$userid$]:[$password$],FUN_BASE64ENC>>


Here we have used FUN_BASE64ENC function but there are many other functions can be used anywhere in Body or Headers

Using Basic Authentication in Dynamic Token approach

Configure Response Settings

Now lets configure how to extract token (single value) from response. Which can be XML, JSON or sometimes Raw Text. We support 4 modes (default = None means full response text is treated as token). If you want to extract just single value and not full text then use below settings.

XML Expression Type (Use XPath)

1. Now, Go to Response Settings tab under a Dynamic Token tab, Enter Response Settings like Token Extract Mode, Expression Type, Content Filter Expression as below XPath Expression. The expression basically extracts content from sessionid XML node using below XPath Expression.

   //*[local-name()='sessionid']

   

   ODBC ZS HTTP Connection Manager: Dynamic Token Tab Configurations Settings of Response Tab

2. That’s it click on OK button to save your Connection Manager Configurations.

JSON Expression Type (Use JSONPath)

If you API returns JSON response (like Walmart API Example) then you can do something like below. Choose Expression Type=JSON and in Response Content Filter You can enter expression like $.some_property_name  (e.g. $.access_token). Check more about How to write JSON Path expression here.

SSIS ZS HTTP Connection Manager – Get a JSON response for the token

Regular Expression Type (Use Regex)

Another advanced mode to extract token from response is Regex Mode. You can write any valid regular expression like this one to extract just one value.

For example if your response returns raw string with token in double quotes but you want just token and no double quotes around it then try something like below.

Extract Token from response – Remove double quotes around value using Regular Expression

Call API in SSIS with Dynamic API Token

Now once you have setup HTTP connection manager to authenticate and get API token, we are ready for next step. Now we can use that token in SOAP / REST API Service call. So, let’s move to the SSIS Part.

1. Once SSIS Package is open, Goto Data Flow Section, For that Drag and Drop DataFlow Task from the SSIS Toolbox.
   

   Drag and drop Data flow task

2. Now, Drag and Drop SSIS ZS XML Source Component from the SSIS Toolbox (if your API is JSON based then drag ZS JSON Source).
   

   Drag and Drop ZS SSIS XML Source Component

3. Let’s double-click on XML Source Component to configure it’s Properties. Enter Request URL.
   

   https://zappysys.com/downloads/files/test/soap-getdata.aspx

   

   SSIS XML Source: Enter your SOAP Request URL

4. Check Use Credentials (Username/Password OR Token), Select HTTP Connection Manager from the drop-down list, we have created just before.
   

   ZS SSIS XML Source Component: Check Use Credentials and Select Connection Manager

5. Select the Request Method and Enter SOAP Request Body something like this. Notice that we used [$token$] as a placeholder. This is automatically replaced at runtime by ZS-HTTP Connection Manager.Passing Token in Body
   

   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:urn="urn:partner.soap.sforce.com">
      <soapenv:Body>
         <urn:sessionid>[$token$]</urn:sessionid>
      </soapenv:Body>
   </soapenv:Envelope>

   Passing Token in Header
   
   If your API expects token via HTTP Header (e.g. Authorization: xxxx-my-token-xxxxxxxx ) then make sure you configure Auth Scheme and Header name on HTTP Connection. By default it sends Token in Authorization header if you dont do anything.
   

   ZS SSIS XML Source Component: Enter SOAP Request Body

6. Select Content-Type as per your Request, here we are selecting

   text/XML;charset=UTF-8

     and adding 1 HTTP Header:

   SOAPAction: "https://zappysys.com/soap-getdata"

   

   ZS SSIS XML Source Component: Select Content-Type Add HTTP Request Header

7. select Filter Path: set as per your requirements. here for this example, we are setting:
   

   $.soapenv:Envelope.soapenv:Body.urn:Row[*]

   

   ZS SSIS XML Source Component: Select Filter

    

8. Check Include Parent Columns checkbox, for including parent Columns, and then click on Preview button to see the Preview data.
   

   SSIS XML Source Component: See the Preview data

9. That’s it. Click on OK button to Save your Configuration.
10. Finally drag any destination (i.e. OLEDB Destination) from your SSIS toolbox. If you want to load this data into SQL, then Drag OLE DB Destination Component, and configure it.

Conclusion

In this blog, we learned how to call any SOAP/REST API with Dynamic Token, retrieve Service response data with SSIS using XML Source Component OR JSON Source Component   in a very simple way. Furthermore, you can achieve many more functionalities with this tool. Check our blogs/articles on XML Source Component to find out what this tool is capable of more.

References

Finally, you can use the following links for more information about the call any SOAP / REST API with Dynamic Token feature with our tools:

• Landing Page for XML Source, you can also find Tutorial Video here.
• Help File: Documentation of XML Source.

The post How to call SOAP / REST API using Dynamic Token in SSIS / ODBC appeared first on ZappySys Blog.

Elasticsearch is a powerful engine that allows you to store, aggregate, and, most importantly, search data in a very analytical way. In this tutorial, you will learn how to bulk load data from SQL Server to Elasticsearch with SSIS (part of SQL Server) and ZappySys PowerPack.

The scope of this article will be to show how to import records from SQL Server into Elasticsearch index as JSON documents using Elasticsearch Bulk API. In a nutshell, we will retrieve IDs and names of the products from a SQL table, transform each row into a JSON and lastly, index each JSON in Elasticsearch under record’s corresponding ID. This tutorial is going to be your stepping stone to use any Elasticsearch API as a destination.

We will be using these SSIS components of PowerPack to make things work:

	JSON Generator Transform
	Web API Destination

Let’s begin?

Prerequisites

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Northwind database deployed on your machine (we will use it to load data from; download it).
4. Elasticsearch instance up and running.
5. ZappySys SSIS PowerPack installed.

<New ZS-OAUTH Connection>

Step-by-Step – Bulk loading data from SQL Server to Elasticsearch with SSIS

Load data from SQL Server database first

In this section, you will perform several basic steps to start the package: load data from SQL Server and prepare it for further steps.

1. Create a new SSIS package and drag a Data Flow Task into the Control Flow from the SSIS Toolbox.
   

   Dragging and dropping Data Flow Task into Control Flow

2. Open Data Flow Task and then add OLE DB Source.
3. Configure OLE DB Source to take data from Products table, Northwind database.
4. Choose ProductID and ProductName as columns you will use.
   

   Load data to Elasticsearch using SSIS and ZappySys Web API Destination component.

Transform data into JSON documents

To add data into Elasticsearch firstly we need to prepare it with JSON Generator Transform. It will help us convert table rows into JSON documents.

JSON String to Create Index data for the Bulk API call

1. Close the window and then drag Derived Column component from the SSIS Toolbox.
2. Furthermore, Add two new columns and name them 

   RowHeader

    and 

   RowFooter

   :
   

   Derived Column configuration to add prefix and header and footer to a JSON

   In the Expression column give them values:

   • RowHeader:
     

     "{ \"index\": { \"_index\": \"shop\", \"_type\": \"products\", \"_id\" : \"" + (DT_WSTR,100)ProductID + "\" } }\n"

   • RowFooter:  

     "\n"

3. Now drag JSON Generator Transform from the SSIS Toolbox and open it.
4. Right mouse click on item Mappings and select

   Add Element(s) (Below this node)

   :
   

   Convert SQL table rows into JSON using JSON Generator Transform

5. Once a dialog appears, select

   ProductName

    as Source Column and

   name

    as Output Alias:
   
   

   Map table column to JSON property using JSON Generator Transform

6. You should get a similar view:
   

   JSON Generator Transform configuration to create JSON from a SQL table

7. Then go to Header / Footer tab and:
   • Select

     Direct String

      option.
   • Input

     <%RowHeader%>

      in Header String option.
   • Input

     <%RowFooter%>

      in Footer String option.
   

   JSON Generator Transform configuration to convert rows into JSON suitable for Elasticsearch Bulk operation

8. Close the window.

JSON String to Update Index data for the Bulk API call

From the above section “JSON String to Create Index data for the Bulk API call” follows all the steps From Step-1 to Step-8.

Just for (2nd step) Step-2 use below expression for  

RowHeader

RowFooter

Derived Column configuration to add prefix and header and footer to a JSONIn the Expression column give them values:

• RowHeader:
  

  "{ \"update\": { \"_index\": \"shop\", \"_id\" : \"" + (DT_WSTR,100)ProductID + "\" } }\n{ \"doc\" : "

• RowFooter:  

  "}\n"

JSON String to Delete Index data for the Bulk API call

From the above section “JSON String to Create Index data for the Bulk API call” follows steps Step-1 and Step-2.

Just for (2nd step) Step-2 use below expression for  

RowHeader

Derived Column configuration to add prefix and header and footer to a JSON

In the Expression column give them values:

• RowHeader:
  

  "{ \"delete\": { \"_index\": \"shop\", \"_id\" : \"" + (DT_WSTR,100)ProductID + "\" } }\n"

Finally, Elasticsearch & SQL Server integration

1. Now drag Web API Destination SSIS component into the Data Flow and open it for editing.
2. In Select Connection section press

   <New ZS-HTTP Connection>

   .
   NOTE: If your ElasticSearch instance is hosted as AWS Managed Service then select  

   <New ZS-OAUTH Connection>

   . rather than ZS-HTTP. When OAuth UI launches select AWS v4 Provider. For more information on calling REST API on AWS check this article.
3. Once HTTP Connection Manager window opens configure connection to your Elasticsearch instance:
   • Set Web Url, which points to your Elasticsearch instance.
   • Set Credentials Type to

     Basic - UserID/Password

      (or other appropriate authentication method).
   • Finally, set User Name and Password:
     
     

     Configure SSIS HTTP Connection to connect to Elasticsearch

4. Close configuration window. Now it’s time to configure Web API Destination.
5. All that is separating you from getting those rows into Elasticsearch is:
   • Setting Input Column for Body to

     ZS_JSON_OUT

      for Create and Update Index Data and for Delete Index Data Input Column for Body to

     RowHeader [Derived Column]

     .
   • Setting URL to

     http://localhost:9200/_bulk

     .
   • Setting HTTP Request Method to

     POST

     .
   

   Index SQL Server Database data in Elasticsearch using SSIS component Web API Destination

6. Then open Batch Settings (For Body) tab:
   • Enable batch submission.
   • Set Body Batch Size, e.g. 1000. Make sure this number is even, otherwise you may get into problems (Elasticsearch Bulk request has special JSON request body format, as you perhaps noticed).
   

   Web API Destination batch settings configuration

7. Close the window and run the package! You should see green lights, telling you everything is OK:
   

   Successful data load from SQL Server to Elasticsearch

8. We can also query Elasticsearch by using one of its Search APIs – URI Search to see if we successfully indexed data:
   

   SQL Server data index in Elasticsearch

9. Rows number match in SSIS and in search results, thus everything is good. You are done.

What if I want more? After upserting data use Web API Destination further

Let’s say you have a requirement to do something with rows that were freshly indexed in Elasticsearch. Then you will need somehow to distinguish between created and updated records in Elasticsearch. And that won’t be difficult because Web API Destination acts not only as Destination but as Transformation as well. So one thing you have to do is connect JSON Parser Transform downstream to Web API Destination. It will parse Elasticsearch HTTP JSON response – which is returned by Web API Destination – into columns, which you can later easily redirect using Conditional Split:

1. Add JSON Parser Transform and connect it to Web API Destination:
   

   Use JSON Parser Transform to get Elasticsearch HTTP JSON response when integrating SQL Server & Elasticsearch

2. Make sure you have selected

   ResponseText

    as Select Input JSON Column.
3. Then go to Bulk API and copy/paste “the result of this bulk operation” JSON response:
   

   Using JSON Parser Transform to parse JSON response from REST HTTP request

4. Set

   $.items[*]

    as the filter.
5. Optional step. You won’t find a response sample for every Elasticsearch API call. In that case, you may want to use Postman to make an HTTP request to Elasticsearch and get a sample response:
   

   Use Postman to make a request to Elasticsearch and get a sample response to be used in JSON Parser Transform

   NOTE: Don’t forget to set the username and password in Authorization section!
6. Then use a Conditional Split and Trash Destination to redirect the rows:
   
   Input these clauses:
   • Inserted:
     

     REPLACENULL([index.result],"") == "created" || REPLACENULL([create.result],"") == "created"

   • Updated:
     

     REPLACENULL([index.result],"") == "updated" || REPLACENULL([create.result],"") == "updated"

7. As a result, new index records will be redirected to one destination, while updated records – to the other:
   

Overall you can use Web API Destination HTTP JSON response for other useful things as well, e.g. determine on how many replica shards record was indexed. JSON response will depend on which Elasticsearch API and which method you use.

Delete Index by making an API call.

If you want to delete the index by making Delete API call you can. Let’s make that call using the Rest API Task. Configure it like below screen and click on the Test Request button.

Rest API Delete Method

Things went bad: Error handling & debugging

Incidentally, you may incorrectly construct JSON for Web API Destination body. Elasticsearch nodes may go offline or go out of memory. In any case you may want to know when that happens and take actions accordingly. For that purpose you have to redirect failed requests from Web API Destination to some other destination:

1. Add a Derived Column above Web API Destination with expression

   (DT_WSTR,4000)ZS_JSON_OUT

    and name it

   JsonAsString

   . This will let you see what JSON you are actually passing.
2. After that, add database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don’t forget to set

   Redirect row

    option for both, Error and Truncation columns:
   
3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled in Elasticsearch

Call ElasticSearch API hosted on AWS

If your ElasticSearch instance is hosted as AWS Managed ElasticSearch Instance then select  

<New ZS-OAUTH Connection>

Call AWS Hosted ElasticSearch REST API in SSIS (V4 Request Signing)

Common Errors

Deployment to Production

Download a sample package

Be sure to download a sample SQL Server 2008 SSIS package, in case you want to try it right away (you can upgrade it to a higher version).

Conclusion. What’s next?

In conclusion, in this article we have learned how to bulk load data from SQL Server and upsert it in Elasticsearch index. We used JSON Generator Transform to help us convert database table rows into JSON documents. In addition, Web API Destination was very helpful in automatically creating and making HTTP requests to Elasticsearch instance and indexing our data as a result. This tutorial was quite simple and straightforward, yet it is a good starter to use other Elasticsearch APIs and their features. From now on Elasticsearch reference is your best friend in learning what APIs exist, what are their methods and how HTTP requests should look like.

Keywords

How to import data from SQL Server to Elasticsearch | How to load data into Elasticsearch | Using Elasticsearch to index SQL Server | Elasticsearch and SQL Server integration | How to use Elasticsearch together with SQL Server | Upsert SQL Server data into Elasticsearch

The post Loading data from SQL Server to Elasticsearch with SSIS appeared first on ZappySys Blog.