In this article, you will learn how to create simple or complex JSON file in Informatica just in a few clicks. We will use the Powerful Export JSON Command line Builder from ZappyShell. Using this Portable / xCopy style command line you can build nested/complex JSON / XML or even CSV from single / multiple SQL Queries as Input or Select Database Tables as Source. Later you can copy generated command line from Tool and execute in any tool like Informatica / Talend / DataStage or Automation Script like Python / PowerShell.

Prerequisite

This article assumes you have access to Informatica Developer Tools and basic knowledge of command-line tools.

1. Download ZappyShell Command line utility (Click on Download Button). You can get msi and Zip (Poratble / xCopy version)
2. If you want to use the Portable version just extract ZappyShell-xCopy zip to some folder lets say (C:\ZappyShell). If you installed MSI then its usually in C:\Program Files (x86)\ZappySys
3. Double click on DbCommandBuilder.exe as listed here  to get started. At the end of this exercise, you will be able to generate something like the below JSON File in few clicks in Informatica or any other Tool / Programming Language you prefer which has option to invoke command line…
4. For all possible command line options Refer this
5. So lets get started

Sample JSON Created by ZappyShell from Informatica

Sample JSON Created by Informatica Workflow (Create From Database SQL Query or Tables)

Create Command Line to Export JSON File from Database SQL Query / Tables

In this section, we will see how to build Command Line which you can then execute in Informatica Command Time and later Schedule Export Job to Produce JSON File on a predefined schedule or manually.

1. Open DbCommandBuilder.exe and configure the Connection / Data source(s) as listed here .
2. Select JSON Export Option from the Dropdown. You can also choose XML, Excel or CSV
3. You can use the Custom Layout Option if you wish to Export using custom Output Layout else just choose tables you like to export or type SQL query.
   
4. Configure your Input Dataset(s) (can be SQL Queries or Tables) and define Desired Layout for your Output File.
5. Here is just an example of how to define an Array for a nested layout in a JSON or XML file.
6. t the end of configuration you may see command line like below (more examples listed here)Example: Simple mode Export (Microsoft SQL Server)
   

   db.exe export customers --json -o c:\data\cust.json --connstr "Data Source=localhost;Initial Catalog=Northwind;Integrated Security=SSPI;" -y

   Example: For Simple mode Export (MySQL SQL Server – ODBC Driver)
   

   export customers --out c:\data\customers.json --json --overwrite --connstr "Driver={MySQL ODBC 5.2 ANSI Driver};Server=your_server_name_or_ip;Port=3306;Uid=user1;pwd=SomeP@ssword;Database=your_db_name"

   Example: For custom layout mode Export
   
   

   export --map c:\data\MySampleMap.xml -o c:\data\customers.json --json --connstr "Data Source=localhost;Initial Catalog=Northwind;Integrated Security=SSPI;" -y

7. Once the Command line is created you can click Run to test and then copy it into tools like Informatica Command Task (see next section).

Export / Create JSON File In Informatica

1. Once the command line is prepared (see previous section) we can open Informatica Developer (D)
2. Connect to your Repository and create a new workflow
3. Drag new Command Task, from Toolbox and double click to edit
   

   Informatica Command Task

4. Enter the sample command line we obtained from the previous step
5. Save the workflow and run it… That’s it you will see the JSON file generated

Export / Create XML File In Informatica

Same Steps as above except choose File format as XML and you will see output like the below once you run the command line.

Export / Create XML File In Informatica

Export / Create CSV File In Informatica

Same Steps as above except choose File format as CSV and you will see output like the below once you run the command line.

Generate CSV Files in Informatica

Export / Create Excel File In Informatica

Same Steps as above except choose File format as Excel and you will see output like the below once you run the command line.

Conclusion

In this post, we just saw how easy it is to generate JSON / XML / Excel or CSV files in tools like Informatica, Talend, python, PowerShell in few clicks without coding. You can schedule job to generate JSON / XML files from Database tables or SQL Query. Download ZappyShell Today and try it to see what else you can do with it.

The post Export / Create JSON File in Informatica (from SQL Query / Database Tables) appeared first on ZappySys Blog.

In our previous article we saw how to read Google Analytics data using SSIS. But what if you don’t use SSIS and you still like to Import Google Analytics data into SQL Server or Read inside other Reporting / ETL Apps (e.g. Excel, Power BI, MS Access … and many more). Not all BI tools comes with native driver / connectivity for Google Analytics. Sometimes they do offer Google Analytics connectivity but don’t offer feature you looking for (e.g. Sort, Segment Filter). In such case you have to look for solution outside.

Well not to worry we will show you how simple it is to connect to Google Analytics using ZappySys JSON Driver (For REST API / Files) in few mins. This driver is one of the best REST API drivers you can find in the market. It is so generic that it can virtually access any REST API you may find internally or on public sites (e.g. Facebook, Salesforce, Azure, AWS). Check these articles to see many more use cases of JSON Driver.

At the end of this article you will learn how to Query Google Analytics data in SSMS like below (Yes that’s correct .. Pretty Awesome!!! ).

Import Google Analytics Data into SQL Server Table (using T-SQL Code)

Testing Google Analytics API

If you are new to Google Analytics API then read little bit about Google Analytics REST API here. Google also offers a really good way to test Analytics API using Query Explorer Tool here. So please check that and understand how to craft correct REST API URL which will be used in the next section.

Using Google Analytics Query Explorer Tool – Create REST API Url (Dimension / Metrics Selection Browser)

Sample Google Analytics API Response

Behind the scene here is how API Request and Response Looks like for Google Analytics REST API. Notice that Authorization Header in below request is automatically added by ZappySys Driver or Query Testing tool above. Also notice nextLink attribute in response, its used to fetch more data. By default each response contains upto 10000 rows.

Request

GET https://www.googleapis.com/analytics/v3/data/ga?ids=ga:185737326&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration&dimensions=ga:date
Authorization: Bearer ya29.GlwUxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Response

{
    "kind": "analytics#gaData",
    "id": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:123456789&dimensions=ga:date&metrics=ga:avgSessionDuration&start-date=30daysAgo&end-date=yesterday",
    "query": {
        "start-date": "30daysAgo",
        "end-date": "yesterday",
        "ids": "ga:123456789",
        "dimensions": "ga:date",
        "metrics": ["ga:avgSessionDuration"],
        "start-index": 1,
        "max-results": 1000
    },
    "itemsPerPage": 1000,
    "totalResults": 30,
    "selfLink": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:123456789&dimensions=ga:date&metrics=ga:avgSessionDuration&start-date=30daysAgo&end-date=yesterday",
    "nextLink": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:123456789&dimensions=ga:date&metrics=ga:avgSessionDuration&start-date=30daysAgo&end-date=yesterday&start-index=10001&max-results=10000",
    "profileInfo": {
        "profileId": "123456789",
        "accountId": "33445566",
        "webPropertyId": "UA-33445566-3",
        "internalWebPropertyId": "454545454",
        "profileName": "All Web Site Data",
        "tableId": "ga:123456789"
    },
    "containsSampledData": false,
    "columnHeaders": [
        {
            "name": "ga:date",
            "columnType": "DIMENSION",
            "dataType": "STRING"
        },
        {
            "name": "ga:avgSessionDuration",
            "columnType": "METRIC",
            "dataType": "TIME"
        }
    ], "totalsForAllResults": {
        "ga:avgSessionDuration": "49.3779488037477"
    },
    "rows": [ 
		["20190425", "401.38912133891213"], 
		["20190426", "512.49723756906077"], 
		["20190427", "11.53488372093024"], 
		["20190428", "432.148648648648646"], 
		["20190429", "234.650735294117645"],
		.......
		.......
		.......
		.......
		["20190525", "74.123435294117645"]
	]
}

Creating Google API Project (Obtain Client ID / Secret)

Very first step to call any Google API including Google Analytics is to create Google API Project and register OAuth App to obtain Client ID and Client Secret. If you dont want to go through this hassle and start easy way then ZappySys offers Inbuilt Default App on OAuth Connection UI but we strongly recommend you create your own app rather than using Default App because in Default App API call limit is shared by many. Your data is never shared with ZappySys in any case even you use Default App.

Here is how to create API Project to call Google API. Once you create Google API Project make sure you enable Google Analytics API.

So once you have Client ID and Client Secret we can move forward to the next step.

Loading Google Analytics data into SQL Server

In our previous article we saw in depth general idea on how to import REST API data in SQL Server (T-SQL Script) without relying on any ETL tools.  Now lets look at step by step on how to load Google Analytics data into SQL Server.

In this example, our goal is to extract a Google Analytics report which shows average session duration and new users count by date (For last 30 days).

Setup ZappySys Data Gateway

Very first step to access any REST API Data inside SQL Server is to configure ZappySys Data Gateway. We covered this in previous article.  But here are high level steps for initial setup of Data Gateway.

1. Search for Gateway in Start menu and  Select ZappySys Data Gateway
   

   Open ZappySys Data Gateway

2. Create new User in Data gateway on Users tab. Enter username and password (we will use this when we create Linked Server) . Check Admin Option
   

   Add Data Gateway User

Setup Google Analytics API Data Source in Gateway / ODBC

Once gateway user is setup, now lets create a new Data Source for Google Analytics API. In this section we will talk how to create data source in gateway but most instructions can be used to create data source in ODBC too. At the end of this article we explained how to Launch ODBC Datasource UI. Step#1 and 2 are only different, all other steps same in ODBC DSN creation.

1. Click Add Data Source option (Select Native – JSON Driver )
   

   Add Gateway Data Source (Native JSON Driver)

2. Click Edit to configure data source
   

   Edit Gateway Data Source Settings

3. Now lets configure Driver settings.  Click on Load Connection String button.
   

   Load ZappySys Driver ConnectionString to configure UI

4. Enter the following ConnectionString to get started with predefined settings.
   Change ids (111223344 to your own Profile ID in DataPath URL). Other URL Parameters explained in next section.
   

   DRIVER={ZappySys JSON Driver};
   DataPath='https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date';
   DataConnectionType=OAuth;
   ScopeSeparator='{space}';
   Scope='https://www.googleapis.com/auth/analytics https://www.googleapis.com/auth/analytics.readonly';
   ServiceProvider=Google;
   Filter='$.rows[*]';
   ArrayTransformType=TransformSimpleTwoDimensionalArray;
   ArrayTransColumnNameFilter='$.columnHeaders[*].name';
   RequestMethod='GET';
   NextUrlAttributeOrExpr='$.nextLink';
   ClientId='xxxxxxxxxxxx';
   ClientSecret='yyyyyyyyyyyyyy';
   UseCustomApp='True'

    
5. URL used in DataPath is most important (parameters explained below) . You can use   Query Explorer Tool here to build same URL.
   

   https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date

   Parameters Explained 
   
   Here is some required parameters used in above REST API URL.
   ids=ga:11223344
   This is your Profile ID which you like to extract.  Read more
   start-date=30daysAgo
   This can be yyyy-MM-dd or some other some predefined date keywords (e.g today, yesterday 10daysAgo). Read more.
   end-date=yesterday
   Same as above
   metrics=ga:avgSessionDuration,ga:newUsers
   This can be comma separated list of metrics you like to extract (max 10). Read more.
   dimensions=ga:date
   This can be comma separated list of metrics you like to extract (max 7). Read more.
6. Now its time to configure OAuth settings. Click on Configure Connection. Enter Client ID and Client Secret Obtained in the previous section (see here) and then click Generate Token as below.
   

   Configure Google API Data Source / DSN (Google Analytics API Example)

7. Thats it now can click on Preview Tab and run sample query like below.
   

   select * from $

   –OR–
   (Select Table name from dropdown and generate default query and then you can add alias , remove unwanted columns)
   

   SELECT 
   "ga:date" date, 
   "ga:avgSessionDuration" avgSessionDuration,
   "ga:newUsers" newUsers
   FROM "rows"

   –OR–
   (Type custom query with your own URL generated from here, Change Profile ID – from 11223344 to your own id)
   

   SELECT * FROM $
   WITH(
   	 Src='https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date'
   	,IncludeParentColumns='False'
   )

   

Create Linked Server in SQL Server using T-SQL Script

Once our Gateway Data Source is configured we can now move to SQL Server Part to define Linked Server.

Here is how you can define linked server to access Google Analytics Data inside SQL Server. There are two ways to create Linked Server (1)  via T-SQL script (2) via SSMS UI)

Here is how to create using Script.

USE [master]
GO

--drop existing linked server for same name if already found
EXEC master.dbo.sp_dropserver @server=N'GOOGLE_ANALYTICS_LS', @droplogins='droplogins'
GO

EXEC master.dbo.sp_addlinkedserver 
	@server = N'GOOGLE_ANALYTICS_LS', 
	@srvproduct=N'', 
	@provider=N'SQLNCLI', 
	@datasrc=N'localhost,5000', --this is host name and port number where  ZappySys Data Gateway is running
	@catalog=N'ZS-GA' --this must be same name as defined on Data Source tab Grid on ZappySys Data Gateway

EXEC master.dbo.sp_addlinkedsrvlogin 
	@rmtsrvname=N'GOOGLE_ANALYTICS_LS',
	@useself=N'False',
	@locallogin=NULL,
	@rmtuser=N'SOME-GATEWAY-USER', -- user name created on ZappySys Data Gateway
	@rmtpassword='SOME-GATEWAY-PASSWORD' -- password for ZappySys Data Gateway user
GO

Create Microsoft SQL Server Linked Server using SSMS UI

Import Google Analytics Data into SQL Server Table (T-SQL Code)

Now lets see how to run sample query to import Google Analytics Data into SQL Server. Notice that how we have escaped quotes in OPENQUERY.
NOTE: Change id 11223344 with your own Id (use this tool to create URL)

select * into GaApiLoad from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
'SELECT * FROM "rows"
WITH 
 (
   SRC=''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date''
  ,IncludeParentColumns=''False''
 )
')
select * from GaApiLoad

Import Google Analytics Data into SQL Server Table (using T-SQL Code)

Expose Google Analytics Data as View

To make it simple you can expose above query as SQL Server views so you can access them from any Reporting / ETL Tools.

CREATE VIEW vw_Google_Analytics_BounceRate_PageViews_Last30Days
AS
--change ids to your own Profile ID
select * from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
'SELECT * FROM "rows"
WITH 
 (
   SRC=''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=2019-04-01&end-date=2019-04-30&metrics=ga:bounceRate,ga:uniquePageviews&dimensions=ga:date''
  ,IncludeParentColumns=''False''
 )
')
GO

Making things Dynamic using Stored Procedure with Parameters

Now let’s make few things dynamic. We will change above query so we can pass Id, Dimensions and Metrics as parameters.

Run below script to Create a stored proc and fetch google analytics data in SQL Server.

use MyDatabase
go
create proc usp_GoogleAnalyticsData
	@ids varchar(1000),
	@startDate varchar(50)='30daysAgo',
	@endDate varchar(50)='yesterday',
	@metrics varchar(200)='ga:avgSessionDuration,ga:newUsers',
	@dimensions varchar(200)='ga:date'	
as

declare @sql varchar(4000)

set @sql='select * from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
''SELECT * FROM "rows"
WITH 
 (
   SRC=''''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:' + @ids 
   +'&start-date='+ @startDate 
   +'&end-date='+ @endDate 
   +'&metrics='+ @metrics 
   +'&dimensions='+ @dimensions 
   +'''''
  ,IncludeParentColumns=''''False''''
 )
'')'

print @sql
execute(@sql)

go

--Example: query google analytics data
exec usp_GoogleAnalyticsData 
	@ids='11223344',  --Change this ID - Obtain from here https://ga-dev-tools.appspot.com/query-explorer/
	@startDate ='30daysAgo',  --or like this 2019-04-01
	@endDate   ='yesterday', --or like this 2019-04-30
	@dimensions='ga:date', 
	@metrics   ='ga:bounceRate,ga:uniquePageviews'

Insert Google Analytics Data into table (Save Stored Procedure Output)

Now what if you like to save Stored Proc output into a table? Well for that you must create table with same structure as stored proc output and then use INSERT INTO sql like below.

--import google analytics data in a table
if(object_id('GaDataLoad') is not null)
	drop table GaDataLoad
create table GaDataLoad([Date] date, BounceRate numeric(10,5), PageViews numeric(10,5))  

insert into GaDataLoad
exec usp_GoogleAnalyticsData 
	@ids='11223344',  --Change this ID - Obtain from here https://ga-dev-tools.appspot.com/query-explorer/
	@startDate ='30daysAgo',  --or like this 2019-04-01
	@endDate   ='yesterday', --or like this 2019-04-30
	@dimensions='ga:date', 
	@metrics='ga:bounceRate,ga:uniquePageviews'

select * from GaDataLoad

Performance Tips / Handling data errors due to missing rows

By default ZappySys API Drivers sends minimum 2 API requests. First one to fetch metadata and second, third… for Data. There will be a time when you wont have any data and it may throw error about no records found because it fails to parse metadata. You can avoid such issue by supplying metadata before hand so you can avoid expensive API calls. Check this article see how you can supply cached metadata in your SQL Query as below. This will avoid Metadata call and speedup significantly. It will also avoid errors due to no rows found for specified criteria.

select * from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
'SELECT * FROM "rows"
WITH 
 (
   SRC=''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:bounceRate,ga:uniquePageviews&dimensions=ga:date''
  ,IncludeParentColumns=''False''
  ,Meta=''[ 
     {"Name": "ga:date","Type": "String", "Length": 16 },
     {"Name": "ga:bounceRate", "Type": "String", "Length": 36 },
     {"Name": "ga:uniquePageviews", "Type": "String", "Length": 10 }
    ]''
 )
')

ODBC Connection – Google Analytics Data in Other Apps (e.g. SSRS / Power BI / Excel / Tableau)

So far we have talked accessing data inside SQL Server using Data Gateway Approach but what if you like to access in other apps ? Well you have three options.

1. Use Microsoft SQL Server Driver to call Linked Server Queries (OPENQUERY approach we saw earlier)
2. Access Google Analytics Data using ODBC connectivity
3. Use Microsoft SQL Server Driver to send direct SQL queries to Gateway (Bypass Linked Server – Use inner SQL query without OPENQUERY in this case)

Method-1 : Linked Server Approach

First approach most likely works in all cases because most apps will support connecting to SQL Server using OLEDB / ADO.net / ODBC / JDBC Drivers. So calling SQL Queries which uses Linked Server (i.e. OPENQUERY statement)  is advisable.

Method-2 : ODBC Driver Approach

However in some cases this may not be possible (e.g. You don’t have SQL Server inhouse or you don’t want to rely on SQL Server to access API). In such case you can use ODBC connectivity in your app. Since ODBC is widely adopted standard most app out there (Except JAVA apps) should support ODBC Drivers. If you like to use this approach then create ODBC DSN rather than Data Gateway Data source and use it in your Reporting / ETL / Custom Apps.

Open ODBC Data Sources (Create DSN)

ODBC User DSN Tab: Add new Driver Screen

Method-3 : Direct connection to Data Gateway (By pass Linked Server)

Last option we suggest for cases like JAVA Apps / Linux / Mac machines where you cannot install ZappySys ODBC Drivers and you dont have option to use SQL Server Linked Server either. In such case you can try to send SQL Queries to Gateway directly using Microsoft SQL Server Compatible Drivers (i.e. ODBC /JDBC). See this example how we called API Queries inside JAVA Apps.

Google Analytics / REST API Integration in various apps

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Conclusion

In this article we explored many ideas of JSON / API integration in SQL Server.  We saw how to create OAuth App for Google API and import Google Analytics data into SQL Server Table without doing any ETL using pure T-SQL code (Query / Views /Stored Procs). You can Download  FREE Trial of ODBC PowerPack and try yourself see how easy it is to query any API inside SQL Server and avoid expensive ETL processes.

The post Import Google Analytics data into SQL Server / Reporting / ETL appeared first on ZappySys Blog.

In this post, we will learn How to Load data into PostgreSQL from Excel file using SSIS PostgreSQL Destination.  We will use SSIS PowerPack to connect PostgreSQL. This article also covers how to read Excel file in SSIS.

SSIS PostgreSQL Destination can be used to perform bulk insert in PostgreSQL Table from any data source using SSIS.

In nutshell, this post will focus on how to Read Excel file and load data into PostgreSQL Table.

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Thirdly, You have obtained PostgreSQL Credential.
4. Finally, do not forget to install ZappySys SSIS PowerPack

Components Mentioned in this article

	PostgreSQL Destination
	Excel Source

What is PostgreSQL?

PostgreSQL is a powerful, open source object-relational database system that uses and extends the SQL language combined with many features that safely store and scale the most complicated data workloads. The origins of PostgreSQL date back to 1986 as part of the POSTGRES project at the University of California at Berkeley and has more than 30 years of active development on the core platform.

PostgreSQL has earned a strong reputation for its proven architecture, reliability, data integrity, robust feature set, extensibility, and the dedication of the open source community behind the software to consistently deliver performant and innovative solutions. PostgreSQL runs on all major operating systems, has been ACID-compliant since 2001, and has powerful add-ons such as the popular PostGIS geospatial database extender. It is no surprise that PostgreSQL has become the open source relational database of choice for many people and organisations.

Getting started with using PostgreSQL has never been easier – pick a project you want to build, and let PostgreSQL safely and robustly store your data.

Getting Started

In order to start, we will show several examples of how to connect the excel file and read data from the excel worksheet. ZappySys includes an SSIS Excel File Source Connector that will help you in reading excel sheet data from Excel File. Here we are showing you is, How to Load Excel File data in PostgreSQL using SSIS PostgreSQL Destination.

Getting Started – Read Excel File in SSIS

Let´s start with an example. Let’s use our SSIS Excel File Source Connector, we will read data from Microsoft Excel File.

1. First of all, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
2. Now Drag and Drop ZS Excel Source from the SSIS Toolbox.
3. Let’s Double Click on Excel Source for edit and select New Connection and click on New button.
   

   Edit Excel Source

4. Now select the excel file path in and test connection in opened excel connection manager window and click on OK.
   

   Select Excel File For Connection

5. Let’s read the Excel Workbook data using table mode and select sheet to read data from that excel sheet.
   

   Load Excel Sheet Data using Access mode as a Table

6.  We can also read Excel Workbook using SQL Query mode and filter the data.
   

   Read from Excel File using SQL Query Mode

7. You can do various other things also in the query mode, find our various example here to read Excel Workbook using SQL Query mode.
   

   Read from Excel File using SQL Query Mode Examples

    

8.  That’s it we are ready to load excel file data to PostgreSQL.

Load Excel File data into PostgreSQL

Now let’s look at how to load data into target like PostgreSQL database Table.

1. First of all, let’s create the PostgreSQL Database Connection From Connection Manager.
2. Right click in connection Window and select New Connection.
   

   Connection Manager Window: Create New Connection

3. Select the ZS-POSTGRESQL Connection type from the Connection Manager Window.
   

   Select ZS-PostgreSQL Connection

4. Let’s configure the connection for PostgreSQL Database and Test the Connection.
   

   Configure PostgreSQL Connection

5. Now lets’ Drag and Drop PostgreSQL Destination from SSIS Toolbox Inside the Data Flow.
   

   Add PostgreSQL Destination

6. Connect our Source component to PostgreSQL Destination.
   

   Connect Source Component to PostgreSQL Destination

7. Double click PostgreSQL Destination to configure it. And Select the PostgreSQL Connection.
   

   Select the PostgreSQL Connection

8. Furthermore, go to Component Properties Tab and Select the PostgreSQL Table to Load Excel File data into it.
   

   Select The PostgreSQL Table

9. Finally, let’s map the columns from the Column Mappings Tab.
   

   PostgreSql Destination : Column Mappings

10. That’s it we are ready to go. Finally execute the package and it will insert the excel file data into the PostgreSQL Table.

Conclusion

Above all, in this blog, we learned how to Load data into PostgreSQL from Excel file using SSIS PostgreSQL Destination using SSIS Pack. Therefore we used Excel File Source Connector to read the Excel file and load data into PostgreSQL using SSIS PostgreSQL Destination. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• About PostgreSQL
• PostgreSQL Documentation
• Help File : PostgreSQL Destination
• Help File: Excel Source

The post Load data into PostgreSQL from Excel file using SSIS appeared first on ZappySys Blog.

In this post, we will learn How to read excel file in SSIS – Load into SQL Server.  We will use SSIS PowerPack to connect Excel file. This article also covers how to read Excel file in SSIS.

SSIS Excel File Source Connector (Advanced Excel Source) can be used to read Excel files without installing any Microsoft Office Driver. You can extract using Table mode or query mode (SQL Command).

In nutshell, this post will focus on how to Read Excel file and load data into SQL Server Table.

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Make sure that Microsoft Excel installed
4. Finally, do not forget to install ZappySys SSIS PowerPack

Components Mentioned in this article

	Excel Source
	Excel Destination

What is Excel File Source Connector

SSIS Excel File Source Connector (Advanced Excel Source) can be used to read the Excel file without installing any Microsoft Office Driver. You can extract using Table mode or query mode (SQL Command). Support for flexible metadata detection mode so you can get accurate datatypes

Getting Started

In order to start, we will show several examples of how to connect the excel file and read data from the excel worksheet. ZappySys includes an SSIS Excel File Source Connector that will help you in reading excel sheet data from Excel File. Here we are showing you is, How to Load Excel File data in SQL Server using SSIS Pack.

Getting Started – Read Excel File in SSIS

Let´s start with an example. Let’s use our SSIS Excel File Source Connector, we will read data from Microsoft Excel File.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
2. Now add Excel Source from the SSIS Toolbox.
   

   Drag and Drop Excel Source

3. Double Click on Excel Source for edit and select New Connection and click on New button.
   

   Edit Excel Source

4. Now select the excel file path in and test connection in opened excel connection manager window and click on OK.
   

   Select Excel File For Connection

5. Let’s read the Excel Workbook data using table mode and select sheet to read data from that excel sheet.
   

   Load Excel Sheet Data using Access mode as a Table

6.  We can also read Excel Workbook using SQL Query mode and filter the data.
   

   Read from Excel File using SQL Query Mode

7. You can do various other things also in the query mode, find our various example here to read Excel Workbook using SQL Query mode.
   

   Read from Excel File using SQL Query Mode Examples

    

8.  That’s it we are ready to load excel file data to SQL Server.

Load Excel File data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

Above all, in this blog, we learned how to read excel file worksheet Data in Excel File using SSIS Pack. We used Excel File Source Connector to read the Excel file and load data into SQL Server. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• • Help File: Excel Source
  • Excel Source

The post How to read excel file in SSIS – Load into SQL Server appeared first on ZappySys Blog.

In this release we focused on mainly focused on fixing high priority bugs related to Upsert Destination. We also added High DPI Monitor Support with Auto Scale UI option (4K Resolution now supported)

Full Notes: https://zappysys.com/onlinehelp/ssis-powerpack/scr/release-notes.htm

Version 2.9.2.10321 [Mar 21, 2019]

New Features/Improvements

• NEW: Excel Source – Add Option to read first sheet and last sheet rather than fixed name ( $last_sheet$ and $first_sheet$ )
• NEW: General – Support for UI Auto Scaling when High DPI / Font Scaling Enabled on your OS (e.g. Win10) – 4K Monitors or High Resolution Display now supported
• NEW: Generator Tarnsforms / Export Tasks for JSON, XML, CSV – Add Option to set Date Format at column level rather than global option
• NEW: Upsert Destination – Add Create Table Option based on Upstream Metadata
• NEW: Upsert Destination – Add option to preview exising data from selected table
• NEW: Upsert Destination – Add option to show source and target column datatypes

Bug fixes

• FIX: All Source Components – After Sorting Available Columns you may lose Output column renames in the columns grid, also output columns sorted in reverse order
• FIX: All Source Components – After Sorting Columns you cannot remove or add columns from the grid
• FIX: Upsert Destination – Mapped Column status icon is not refreshing after column is mapped
• FIX: Upsert Destination – varchar, char, text datatypes detected as DT_WSTR / DT_NTEXT rather than DT_STR / DT_TEXT
• FIX: Upsert Destination – When you have zero rows in source you may get error – Invalid object name #zappysys_xxxxxx_yyyyyyyy
• FIX: Upsert Destination – When you map to column with scale=zero (e.g. numeric(N, 0)) you may get error on save
• FIX: Upsert Destination – You may get error when you change source column after save – An item with the same key has already been added
• FIX: Upsert Destination – You may get object reference not set error in advanced edit

The post SSIS PowerPack v2.9.2 released (Maintenance Release) appeared first on ZappySys Blog.

In this post, we will learn how to Create Excel File in SSIS from source like JSON / XML.  We will use SSIS PowerPack to connect and query a JSON or XML file. This article also covers creating Excel from JSON File.

JSON stands for JavaScript Object Notation and it is an Open and Standard format to read an object with attributes and values.  JSON is replacing XML because it is faster to read data, it is easier to parse data, it does not require end tags, it is shorter and it can use arrays.

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Make sure that Microsoft Excel installed
4. Finally, do not forget to install ZappySys SSIS PowerPack

Components Mentioned in this article

	JSON File/REST API Source
	XML File Source
	Excel Destination

What is JSON File Source

JSON Source Connector can be used to extract and output JSON data stored in local JSON files, JSON data coming from REST API web service calls (Web URL) or direct JSON String (variables or DB columns). JSON component also supports JSONPath to filter data from nested array/sub-documents. This component is optimized to work with very large files.

What is XML File Source

SSIS XML Source can be used to consume multiple XML files or XML formatted response from SOAP Web service or REST API Web Service. It supports advanced filtering along with a flexible approach to configuring request parameters for web service.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS JSON Source for JSON or XML File as well as REST API that will help you in reading JSON or XML Files from JSON or XML File to the Local machine. Here we are showing you is, How to Load JSON or XML Data in Excel File using SSIS Pack.

Getting Started – Create Excel File in SSIS

Let´s start with an example. In this SSIS JSON or XML Source for JSON or XML File task example, we will read JSON or XML files from Microsoft Excel File.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

2. Drag and Drop relevant JSON or XML Source(REST API and File) for JSON or XML File Task from the SSIS Toolbox.
   

   SSIS Add JSON or XML Source

3. Double Click on JSON or XML Source (REST API or File) for edit.
   

   SSIS Json/XML Source File

4. JSON and XML Source Task is able to do more various call of Rest API using Http and Oauth2 Connection also Please refer below articles link for the same
   • Various Articles for JSON Source
   • Various Articles for XML Source
5. Now let’s create the Excel File Connection. In the connection Manager Window right click and select new connection.
   

   Create Excel File Connection

    

6. And Create The Zs-Excel Type Connection. And click Add.
   

   Add Excel Connection

    

7. Now select the excel file path in and test connection.
   

   Select Excel File For Connection

    

8.  Now add Our Excel File Destination Connector and connect upper JSON or XML source with it
   https://zappysys.com/products/ssis-powerpack/ssis-excel-file-destination/
9.  And now edit the Excel destination and select that created excel file connection to load the data in excel file.
   

   SSIS Excel Connection Advanced Edit

    

10. Give the Range Start cell in Component Properties. (A1 for starting of the file) and select the table name.
    

    Give RangeStartCell and TableName

     

11.  Select the input columns.
    

    Select Input Columns

12. That’s run the package and it will load all the JSON or XML File data into the Excel file.
    

    run-package-for-json-into-excel-file

    

    Stored Data into Excel File

     

Conclusion

Above all, in this blog, we learned how to Load JSON or XML Data in Excel File using SSIS Pack. We used JSON or Excel Source(REST API or File) for JSON or Excel file, Excel Destination for Excel file to read the JSON or XML file and load data into Excel File. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• Our Various Articles for JSON Source
• Our Various Articles for XML Source
• Excel Destination

The post Create Excel File in SSIS (Read from JSON / XML) appeared first on ZappySys Blog.

In our previous post we saw how to create excel dynamically in SSIS. Now let’s advance further and find out how to create Excel reports in SSIS using Powerful Template engine (introduced in PowerPack 2.7.4)  In this post you will learn how to use ZappySys Export Excel Task to export data from Multiple SQL Tables to Single or Multiple Excel Sheets, we will use Formatting, Expressions and Filtering to export parent child datasets (Master-Detail) using Template approach which supports full customization as per your need.

We will be using following ZappySys SSIS PowerPack component to make things work in this tutorial:

SSIS Export Excel File Task (Dynamically generate)

You can check the following video to learn basic things about Export Excel Task. In future we will post video tutorial to show how Template engine works.

Requirements

1. First of all, MS Excel installed.
2. Secondly, an Excel template that you can download here.
3. In addition, you will need SQL Server installed.
4. Aso, the SQL Server Northwind sample database that you can download here.
5. Additionally, SSDT for Business Intelligence.
6. Finally, use the SSIS Excel package example that you can download here.

Concept about Excel Template Engine

ZappySys Export Excel Template engine is compatible with FlexCel engine described here. So read it carefully to get most details about how template engine works and what elements supported in Template engine. Basically you have to use Template file which will be used to generate final excel report in SSIS. Advantage of Template engine approach compared to Simple export is you can control every aspect of your report. Also you can create nested reports use multiple datasets in a single sheet (Master-Detail JOINS) which is not possible in default export mode. The only down side of this approach is… learning curve. However once you understand concepts of Template Engine it wont take you too much time to master it.

In Template file you can define bands (e.g. Customer Info, Order Info…). Each band can grow in Vertical or Horizontal direction. Band must be created by Named Range in Excel. You must name your range in a very specific way. Based on how you suffix and prefix your named range it will decide how to grow your band (i.e. Expand Vertical or Horizontal).

Few things to remember with Template Engine.

• Each Template file must have at least one Band which can be used to attach Dataset.
• Each Template file must have Config Sheet somewhere.
• You can use many functions in your Template using Tag system
• You can define Charts, Graphs any elements you like in Excel Template and link to Ranges you define.

Getting started

In next few section we will see step by step instructions on how to use export SQL server data to Excel using Template File approach.

Prepare Sample Tables (Source data)

In this example, we will create an Excel report based in two SQL Server tables from the Northwind database mentioned in the requirements. We are going to use Northwind dataset. You can download SQL Script file from here. Just download / extract and run it in SSMS to create necessary Tables with Sample Data.

1. First of all, we will see the data displayed in the Customer table:
   

   Northwind table in SQL Server

2. Secondly, we will check the data of the Orders table. Note that the CustomerID is included in this table, so we can create a master-detail report based on these 2 tables:
   

   SQL Server Orders

Prepare Excel Template

Now let’s create a  template file which will be used to share our exported file. To make demo simple we will use this already prepared template file. Lets understand how template file was created.

We assume you have read concepts of template engine by reading this link.

Basically few things to check in Template file

1. Defining correct Config Tab (Only needed if you are using special features such as reusable expression, or filters (i.e. subset of data).
2. Defining bands which is bound to dataset

Define Reusable Configuration Settings ( <#Config> tab )

In our example we are creating 2 extra tabs (UK, US) using filter expression (subset of dataset) so we need to use Config setting for that purpose. Notice that Config tab must be names as <#Config> and Reusable sections for Data, Format and Expression must be defined as below Cell A10 , Cell H10, Cell L10.

Define Config Settings for Excel Report

Create Bands

Now let’s look at how we define bands in template file. Read more about different types of bands in Excel Template Engine. Band are defined using Named ranges. In our example we use two bands in first Sheet1.

• For customer data __cust__  (start / end with double underscore)
• For orders data __ord__X (Read more on “X” ranges )
• For US tab  __cust_us__
• For UK tab __cust_uk__

Designing Excel Report Template (Define Ranges for Master-Detail Report)

Using Filter (Subset of data) in Excel Report

Create SSIS Package

Now let’s use Our sample package to get started. You can create your own Package but we have already created one to make demo simple.

1. In an SSIS project add the Demo SSIS package from the step 5 of the requirements:
   

   SSIS add package

2. In addition, in the SSIS project, drag and drop the ZS Export Excel task and Create a connection to the Northwind Database by clicking New for Source Connection.
3. and enter the source tables/queries you like to use in your report. In our example, we will use two tables.
   Syntax for Source data is

   alias1=Table1_Or_Query1 | alias2=Table2_Or_Query2 .... 

   In this example, we will use  cust= select * from dbo.Customers | ord=dbo.orders.
   You can also use just the table names for both e.g. cust=dbo.Customers | ord=dbo.orders
   
   

   Source table/query for Excel Report

    

4. Check the option to Use Template Engine To get the access to Excel Report Template Tab.
   

   Use Template Engine

5. Now go to Excel Report Template tab, enter the path of the template file which we downloaded in step 2 of the requirements and write the relationship between the customer and order tables (cust.CustomerID=ord.CustomerID):
   

   Specify template file for Excel report, define relationship for master-detail report

    

6. Finally, in the Target tab specify the path of the Excel Target and run the package:
   

   SSIS target in Excel

Checking the results of the package to create an Excel Report in SSIS using a template

1. Once that you run the package, you can check the results. First of all, we will check the Sheet1. It will contain the orders grouped by customerID:
   

   Excel report exported in SSIS based on a template

    

2. Secondly, in the US sheet in Excel, it will show the customers filtered by the country where the country is US:
   

   filter data by country in Excel

    

3. Also, in the UK sheet, it will show all the customers where the country is equal to UK:
   

   Show data where the country is the United Kingdom

Add Dynamic Comments

Sometimes you might want to add dynamic comments. Several cells may need comments, similar to tooltips, which come from another data column in the database.

To achieve this, add the comment to the cell in the template file and place a placeholder column under the specified column.

Place the placeholder column under the specified column.

This will allow you to achieve the desired output. The Excel file will be generated with comments in this format.

Excel File with Comments

Other Examples

There are many other examples of template engine use case found here. Refer to related Github link and download xlsx files from Github. You can use those template file path to run demo in SSIS.

Pivot Table Example

For example to run Pivot Table demo using Template Engine download below files

Pivot Tables.template.xlsx

ReportHeader.xls

Found under below path

https://github.com/tmssoftware/TMS-FlexCel.NET-demos/tree/master/csharp/VS2019/netframework/Modules/20.Reports/80.Pivot%20Tables

Conclusion

To conclude, we learned how to create a report from data in SQL Server and export based on a template. The template not only included a nice format in Excel, but also filter and relationships between two tables. If you liked our products, enjoy them HERE.

References

For more information about creating an Excel Report in SSIS using a template, refer to the following links:

• Getting Started with FlexCel Studio for the .NET Framework
• SSIS Export Excel File Task (Dynamically generate)
• SSIS Export Excel Task – Generate Excel Dynamically

The post Create Excel Report in SSIS using a template (Export Formatted xlsx File) appeared first on ZappySys Blog.

In this post, we will learn how to import JSON to Excel.  We will use ODBC PowerPack to connect and query a JSON file. This article also covers creating Excel from SOAP XML Web Service so read full article to learn about SOAP API Export.

JSON stands for Java Script Object Notation and it is an Open and Standard format to read object with attributes and values.  JSON is replacing XML because it is faster to read data, it is easier to parse data, it does not require end tags, it is shorter and it can use arrays.

We will load the data using the ZappySys JSON driver and then upload the data in Excel.

Requirements

This article assumes following requirements are met before you can follow the steps listed in this article.

1. Make sure that Microsoft Excel installed
2. Also, the driver ODBC PowerPack installed

Video Tutorial

Let’s take a look to this step by step tutorial.

An introduction to Rest API and OData

In this example, we will use OData (Open Data Protocol) to consume REST API. REST API (Representational State Transfer Application Program Interface) allows to handle the interoperability betwee computers and internet.

In REST API we can handle web services in different formats. In this example, we will work with the Northwind example. The Northwind example is available in this URL:

https://services.odata.org/V3/Northwind/Northwind.svc

1. By default the data is displayed in XML format. To show the data in JSON use this URL:

   https://services.odata.org/V3/Northwind/Northwind.svc/?$format=json

2. There are collections of data like Categories, CustomerDemographic, Customers, etc. For example the following URL will show the data of the categories collection:

   https://services.odata.org/V3/Northwind/Northwind.svc/Categories?$format=json

3. In the next steps, we will use ZappySys drivers to connect to this URL and query using OData.

Configure ODBC DSN for ZappySys JSON Driver

ODBC driver can be accessed in two modes.

1. Using DSN
2. Without DSN (Supply direct Connection String e.g. DRIVER={ZappySys JSON Driver}; …….. )

In this article, we will use DSN approach (User DSN). We will first add the ZappySys JSON Driver in the ODBC Data source Administrator.

Follow these steps to accomplish the task:

1. First, Windows search, write ODBC and select the ODBC Data sources (32 bits)
   

   Open ODBC Data Source

2. As a second step, in ODBC Data source Administrator press the Add button.
   

   Add ZappySys

    

3. In this step, create new data source, select ZappySys ODBC Driver.
   

   Add new zappysys json driver

4. Here we have several properties, write a data source name. In this example, the name will be ZappySys JSON to Excel.
5. The Data Source (URL or file path) can specify the URL of the source or if it is a local file, you can specify the local path. In this example, the URL is:
   

   https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json

   You can also specify local file path as Data Source

   For single file:  c:\data\myfile_1.json
   For multiple files: c:\data\myfile_*.json

6. Expand Other settings and in Data Format, select OData. Press OK.
   

   OData properties json driver

How to import REST API data to Excel

1. In Excel, go to Data Ribbon and select From Other Sources and From Microsoft Query.
   

   Excel Data Ribbon

2. In Choose Data Source, select ZappySys JSON for Excel and press OK.
   

   Select zappysys data source

3. Click Value and press the > to display all the attributes and press next.
   

   Create json query in excel

4. You can filter data and select columns and check if a columns is equal to, greater than or less than a specific value. In this example, we will not apply filters. Press next.
   

   filter json data in excel

5. Now if you want to export data to sheet without using custom Query then select first option. If you like to enter custom query then select  Microsoft Query (second option) once you close the wizard you may get an option to enter custom SQL (see the toolbar of Graphical Designer)We will use first option for now.
   

   Return data in Excel

6. In this step, Excel will show the Import Data window. Excel will let you select the sheet to insert the data. We will choose the existing sheet.
   

   Choose excel sheet

7. If everything is OK, you will be able to see the data.
   

   Excel rows with data

How to import data from REST API to CSV

If you already have your data in Excel with the previous steps, converting the data extracted from Rest API to Excel to a CSV is a straight forward process.

1. Using Excel, go to the File menú and select Save as and select a folder to store it.
   

   Save as in Excel

2. In order to save the data in CSV, save as type select CSV (Comma delimited)
   

   convert excel to csv

How to import JSON to Excel (From Local file)

1. To import data in Excel, the steps are the same for Rest API that for a physical json file. The only difference is that a local path is used instead of a URL. In order to get JSON data into Excel, you need to repeat the steps to add the ZappySys ODBC Driver in the ODBC Data source, but instead of specifying a URL, we just need to specify the local path.
2. To test the driver, let’s say that we have the following named sample.json file:
   

   {
     "odata.metadata": "https://services.odata.org/V3/Northwind/Northwind.svc/$metadata#Customers",
     "odata.nextLink": "Customers?$skiptoken='ERNSH'",
     "value": [
       {
         "CustomerID": "ALFKI",
         "CompanyName": "Alfreds Futterkiste",
         "ContactName": "Maria Anders",
         "ContactTitle": "Sales Representative",
         "Address": "Obere Str. 57",
         "City": "Berlin",
         "Region": null,
         "PostalCode": "12209",
         "Country": "Germany",
         "Phone": "030-0074321",
         "Fax": "030-0076545"
       },
       {
         "CustomerID": "ANATR",
         "CompanyName": "Ana Trujillo Emparedados y helados",
         "ContactName": "Ana Trujillo",
         "ContactTitle": "Owner",
         "Address": "Avda. de la Constitución 2222",
         "City": "México D.F.",
         "Region": null,
         "PostalCode": "05021",
         "Country": "Mexico",
         "Phone": "(5) 555-4729",
         "Fax": "(5) 555-3745"
       },
       {
         "CustomerID": "ANTON",
         "CompanyName": "Antonio Moreno Taquería",
         "ContactName": "Antonio Moreno",
         "ContactTitle": "Owner",
         "Address": "Mataderos 2312",
         "City": "México D.F.",
         "Region": null,
         "PostalCode": "05023",
         "Country": "Mexico",
         "Phone": "(5) 555-3932",
         "Fax": null
       },
       {
         "CustomerID": "AROUT",
         "CompanyName": "Around the Horn",
         "ContactName": "Thomas Hardy",
         "ContactTitle": "Sales Representative",
         "Address": "120 Hanover Sq.",
         "City": "London",
         "Region": null,
         "PostalCode": "WA1 1DP",
         "Country": "UK",
         "Phone": "(171) 555-7788",
         "Fax": "(171) 555-6750"
       },
       {
         "CustomerID": "BERGS",
         "CompanyName": "Berglunds snabbköp",
         "ContactName": "Christina Berglund",
         "ContactTitle": "Order Administrator",
         "Address": "Berguvsvägen 8",
         "City": "Luleå",
         "Region": null,
         "PostalCode": "S-958 22",
         "Country": "Sweden",
         "Phone": "0921-12 34 65",
         "Fax": "0921-12 34 67"
       },
       {
         "CustomerID": "BLAUS",
         "CompanyName": "Blauer See Delikatessen",
         "ContactName": "Hanna Moos",
         "ContactTitle": "Sales Representative",
         "Address": "Forsterstr. 57",
         "City": "Mannheim",
         "Region": null,
         "PostalCode": "68306",
         "Country": "Germany",
         "Phone": "0621-08460",
         "Fax": "0621-08924"
       },
       {
         "CustomerID": "BLONP",
         "CompanyName": "Blondesddsl père et fils",
         "ContactName": "Frédérique Citeaux",
         "ContactTitle": "Marketing Manager",
         "Address": "24, place Kléber",
         "City": "Strasbourg",
         "Region": null,
         "PostalCode": "67000",
         "Country": "France",
         "Phone": "88.60.15.31",
         "Fax": "88.60.15.32"
       },
       {
         "CustomerID": "BOLID",
         "CompanyName": "Bólido Comidas preparadas",
         "ContactName": "Martín Sommer",
         "ContactTitle": "Owner",
         "Address": "C/ Araquil, 67",
         "City": "Madrid",
         "Region": null,
         "PostalCode": "28023",
         "Country": "Spain",
         "Phone": "(91) 555 22 82",
         "Fax": "(91) 555 91 99"
       },
       {
         "CustomerID": "BONAP",
         "CompanyName": "Bon app'",
         "ContactName": "Laurence Lebihan",
         "ContactTitle": "Owner",
         "Address": "12, rue des Bouchers",
         "City": "Marseille",
         "Region": null,
         "PostalCode": "13008",
         "Country": "France",
         "Phone": "91.24.45.40",
         "Fax": "91.24.45.41"
       },
       {
         "CustomerID": "BOTTM",
         "CompanyName": "Bottom-Dollar Markets",
         "ContactName": "Elizabeth Lincoln",
         "ContactTitle": "Accounting Manager",
         "Address": "23 Tsawassen Blvd.",
         "City": "Tsawassen",
         "Region": "BC",
         "PostalCode": "T2F 8M4",
         "Country": "Canada",
         "Phone": "(604) 555-4729",
         "Fax": "(604) 555-3745"
       },
       {
         "CustomerID": "BSBEV",
         "CompanyName": "B's Beverages",
         "ContactName": "Victoria Ashworth",
         "ContactTitle": "Sales Representative",
         "Address": "Fauntleroy Circus",
         "City": "London",
         "Region": null,
         "PostalCode": "EC2 5NT",
         "Country": "UK",
         "Phone": "(171) 555-1212",
         "Fax": null
       },
       {
         "CustomerID": "CACTU",
         "CompanyName": "Cactus Comidas para llevar",
         "ContactName": "Patricio Simpson",
         "ContactTitle": "Sales Agent",
         "Address": "Cerrito 333",
         "City": "Buenos Aires",
         "Region": null,
         "PostalCode": "1010",
         "Country": "Argentina",
         "Phone": "(1) 135-5555",
         "Fax": "(1) 135-4892"
       },
       {
         "CustomerID": "CENTC",
         "CompanyName": "Centro comercial Moctezuma",
         "ContactName": "Francisco Chang",
         "ContactTitle": "Marketing Manager",
         "Address": "Sierras de Granada 9993",
         "City": "México D.F.",
         "Region": null,
         "PostalCode": "05022",
         "Country": "Mexico",
         "Phone": "(5) 555-3392",
         "Fax": "(5) 555-7293"
       },
       {
         "CustomerID": "CHOPS",
         "CompanyName": "Chop-suey Chinese",
         "ContactName": "Yang Wang",
         "ContactTitle": "Owner",
         "Address": "Hauptstr. 29",
         "City": "Bern",
         "Region": null,
         "PostalCode": "3012",
         "Country": "Switzerland",
         "Phone": "0452-076545",
         "Fax": null
       },
       {
         "CustomerID": "COMMI",
         "CompanyName": "Comércio Mineiro",
         "ContactName": "Pedro Afonso",
         "ContactTitle": "Sales Associate",
         "Address": "Av. dos Lusíadas, 23",
         "City": "Sao Paulo",
         "Region": "SP",
         "PostalCode": "05432-043",
         "Country": "Brazil",
         "Phone": "(11) 555-7647",
         "Fax": null
       },
       {
         "CustomerID": "CONSH",
         "CompanyName": "Consolidated Holdings",
         "ContactName": "Elizabeth Brown",
         "ContactTitle": "Sales Representative",
         "Address": "Berkeley Gardens 12 Brewery",
         "City": "London",
         "Region": null,
         "PostalCode": "WX1 6LT",
         "Country": "UK",
         "Phone": "(171) 555-2282",
         "Fax": "(171) 555-9199"
       },
       {
         "CustomerID": "DRACD",
         "CompanyName": "Drachenblut Delikatessen",
         "ContactName": "Sven Ottlieb",
         "ContactTitle": "Order Administrator",
         "Address": "Walserweg 21",
         "City": "Aachen",
         "Region": null,
         "PostalCode": "52066",
         "Country": "Germany",
         "Phone": "0241-039123",
         "Fax": "0241-059428"
       },
       {
         "CustomerID": "DUMON",
         "CompanyName": "Du monde entier",
         "ContactName": "Janine Labrune",
         "ContactTitle": "Owner",
         "Address": "67, rue des Cinquante Otages",
         "City": "Nantes",
         "Region": null,
         "PostalCode": "44000",
         "Country": "France",
         "Phone": "40.67.88.88",
         "Fax": "40.67.89.89"
       },
       {
         "CustomerID": "EASTC",
         "CompanyName": "Eastern Connection",
         "ContactName": "Ann Devon",
         "ContactTitle": "Sales Agent",
         "Address": "35 King George",
         "City": "London",
         "Region": null,
         "PostalCode": "WX3 6FW",
         "Country": "UK",
         "Phone": "(171) 555-0297",
         "Fax": "(171) 555-3373"
       },
       {
         "CustomerID": "ERNSH",
         "CompanyName": "Ernst Handel",
         "ContactName": "Roland Mendel",
         "ContactTitle": "Sales Manager",
         "Address": "Kirchgasse 6",
         "City": "Graz",
         "Region": null,
         "PostalCode": "8010",
         "Country": "Austria",
         "Phone": "7675-3425",
         "Fax": "7675-3426"
       }
     ]
   }

3. Finally, in the ZappySys driver, in Data Source (URL or File Path), write the path of your json file. In this example, c:\sql\sample.json. Also expand Data format and select Default.
   
   

   properties to query a local json file

   NOTE: When the Data format is set to OData and it is a local json file, the error message is “Data processing error: URI formats are not supported”. If the Data format is set to Original, the error message is “Query execution error: Requested value ‘Original’ was not found.” To solve this issue, make sure that in Other Settings, the Data format is set to Default.

How to query JSON or REST API

Our ZappySys driver is a very intuitive tool and you can write queries to data like a simple database table.

Let’s take a look to some examples:

In the ODBC Data Source Administrator, press Configure.

Configure ZappySys settings

On Preview page, write this query:

select 
"CustomerID",
"CompanyName",
"ContactName",
"ContactTitle",
"Address",
"City",
"Region",
"PostalCode",
"Country",
"Phone",
"Fax"
from [value] 
where City='Berlin'

The query will show the customers where the city is Berlin:

Query json information using filters

You can use in the where clause with double quotes:

WHERE City="Berlin"

or simple quotes:

WHERE City='Berlin'

It is also valid to comment code. The following example shows how to comment a single line (the where clause):

select "CustomerID", "CompanyName
from [value]
--where City='Berlin'

The following example shows how to comment multiple lines

/*
This example shows how 
to comment
multiple lines of code
*/

select "CustomerID", "CompanyName from [value]

Schedule / Automate Excel file creation using command line

There will be a time when you need to export REST API to Excel in an automated way (Scheduled Task). Possibly via Batch file, Python code, PowerShell script or Scheduler. Check this article to learn more.

You can use ZappyShell for DB (Command Line Tool) to export JSON / SOAP / REST API data to Excel Sheet. It supports any ODBC Connection string as a source and you can export to CSV, Excel, JSON and XML.

Here is a sample command line. You can automate it via Scheduled Job (e.g. Windows Scheduler or SQL Agent Job)

First create below file (name it script.txt)

export "SELECT * FROM $ LIMIT 20 WITH(Src='https://httpbin.org/json',Filter='$.slideshow.slides[*]' , RequestMethod='GET')"  -o "c:\data.xlsx" --connstr "DSN=MyAPIConnection;" -y

Then call this (assuming you have zappyshell in c:\zappyshell folder

c:\zappyshell>db.exe exec c:\script.txt

You can also use DSN less connection string such as below

export "SELECT * FROM $ LIMIT 20 WITH(Src='https://httpbin.org/json',Filter='$.slideshow.slides[*]' , RequestMethod='GET')"  -o "c:\temp\data.xlsx" --connstr "DRIVER={ZappySys JSON Driver};" -y

Export JSON / SOAP XML / REST API to Excel File – Create Excel File using Automated Command Line (Schedule)

Calling XML SOAP Web Service in Excel

So far we have looked at examples to consume data using JSON driver. Now let’s look at an example, to call XML SOAP Web Service in Excel.

To call SOAP API you need to know Request XML Body Structure. If you are not sure how to create SOAP Request body then no worries. Check this article to learn how to generate SOAP Request body using the Free tool SoapUI. Basically, you have to use SoapUI to generate Request XML and after that, you can replace parameters as needed in the generated body.

What is SOAP Web Service?

If you are new to SOAP Web Service sometimes referred as XML Web Service then please read some concept about SOAP Web service standard from this link There are two important aspects in SOAP Web service.

1. Getting WSDL file or URL
2. Knowing exact Web Service URL

What is WSDL

In very simple term WSDL (often pronounced as whiz-dull) is nothing but a document which describes Service metadata (e.g. Functions you can call, Request parameters, response structure etc). Some service simply give you WSDL as xml file you can download on local machine and then analyze or sometimes you may get direct URL (e.g. http://api.mycompany.com/hr-soap-service/?wsdl )

Example SQL Query for SOAP API call using ZappySys XML Driver

Here is an example SQL query you can write to call SOAP API. If you not sure about many details then check next few sections on how to use XML Driver User Interface to build desired SQL query to POST data to XML SOAP Web Service without any coding.

SELECT * FROM $
WITH(
	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
	,DataConnectionType='HTTP'
	,CredentialType='Basic' --OR SoapWss
	,SoapWssPasswordType='PasswordText'
	,UserName='myuser'
	,Password='pass$$w123'
	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
	,ElementsToTreatAsArray='HolidayCode'	
	,RequestMethod='POST'	
	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
	,RequestData='
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hol="http://www.holidaywebservice.com/HolidayService_v2/">
   <soapenv:Header/>
   <soapenv:Body>
      <hol:GetHolidaysAvailable>
         <!--type: Country - enumeration: [Canada,GreatBritain,IrelandNorthern,IrelandRepublicOf,Scotland,UnitedStates]-->
         <hol:countryCode>UnitedStates</hol:countryCode>
      </hol:GetHolidaysAvailable>
   </soapenv:Body>
</soapenv:Envelope>'
)

Now let's look at steps to create SQL query to call SOAP API. Later we will see how to generate code for your desired programming language (e.g. C# or SQL Server)

Video Tutorial - Introduction to SOAP Web Service and SoapUI tool

Before we dive into details about calling SOAP API using ZappySys XML Driver, lets first understand what is SOAP API and how to create SOAP requests using SoapUI tool. You will learn more about this process in the later section. The video contains some fragment about using SOAP API in SSIS but just ignore that part because we will be calling Soap API using ZappySys ODBC Driver rather than SSIS Components.  

Using SoapUI to test SOAP API call / Create Request Body XML

Assuming you have downloaded and installed SoapUI from here, now we are ready to use WSDL for your SOAP Web Service Calls. If you do not have WSDL file or URL handy then contact your API provider (sometimes you just have to add ?wsdl at the end of your Service URL to get WSDL so try that. Example: http://mycompany/myservice?wsdl ). If you don't know what is WSDL then in short, WSDL is Web service Description Language (i.e. XML file which describes your SOAP Service). WSDL helps to craft SOAP API request Body for ZappySys XML Driver. So Let's get started.

1. Open SoapUI and click SOAP button to create new SOAP Project
2. Enter WSDL URL or File Path of WSDLFor example WSDL for our sample service can be accessed via this URL

   http://www.dneonline.com/calculator.asmx?wsdl

   
   Create new SOAP API Project in SoapUI tool for SOAP API Testing
3. Once WSDL is loaded you will see possible operations you can call for your SOAP Web Service.
4. If your web service requires credentials then you have to configure it. There are two common credential types for public services (SOAP WSS or BASIC )
   1. To use SOAP WSS Credentials select request node and enter UserId, Password, and WSS-PasswordType (PasswordText or PasswordHash)
      
      Configure SOAP WSS Credentials for SoapUI (SOAP API Testing Tool)
   2. To use BASIC Auth Credentials select request node and double-click it. At the bottom click on Auth (Basic) and From Authorization dropdown click Add New and Select Basic.
      Configure Basic Authorization for SoapUI (SOAP API Testing Tool)
5. Now you can test your request first Double-click on the request node to open request editor.
6. Change necessary parameters, remove optional or unwanted parameters. If you want to regenerate request you can click on Recreate default request toolbar icon.
   Create SOAP Request XML (With Optional Parameters)
7. Once your SOAP Request XML is ready, Click the Play button in the toolbar to execute SOAP API Request and Response will appear in Right side panel. Test SOAP API using SoapUI Tool (Change Default XML Body / Parameters, Execute and See Response)

Create DSN using ZappySys XML Driver to call SOAP API

Once you have tested your SOAP API in SoapUI tool, we are ready to use ZappySys XML driver to call SOAP API in your preferred BI tool or Programming language.

1. First open ODBC Data Sources (search ODBC in your start menu or go under ZappySys > ODBC PowerPack > ODBC 64 bit)
2. Goto System DSN Tab (or User DSN which is not used by Service account)
3. Click Add and Select ZappySys XML Driver ZappySys ODBC Driver for XML / SOAP API
4. Configure API URL, Request Method and Request Body as below ZappySys XML Driver - Calling SOAP API - Configure URL, Method, Body
5. (This step is Optional) If your SOAP API requires credentials then Select Connection Type to HTTP and configure as below.
   ZappySys XML Driver - Configure SOAP WSS Credentials or Basic Authorization (Userid, Password)
6. Configure-Request Headers as below (You can get it from Request > Raw tab from SoapUI after you test the request by clicking the Play button) Configure SOAP API Request Headers - ZappySys XML Driver
7. Once credentials entered you can select Filter to extract data from the desired node. Make sure to select array node (see special icon) or select the node which contains all necessary columns if you don't have array node. Select Filter - Extract data from nested XML / SOAP API Response (Denormalize Hierarchy)
8. If prompted select yes to treat selected node as Array (This is helpful when you expect one or more record for selected node) Treat selected node as XML Array Option for SOAP API Response XML

Preview SOAP API Response / Generate SQL Code for SOAP API Call

Once you configure settings for XML Driver now you can preview data or generate example code for desired language (e.g. C#, Python, Java, SQL Server). Go to Preview tab and you will see default query generated based on settings you entered in previous sections. Attributes listed in WITH clause are optional. If you omit attribute in WITH clause it will use it from Properties tab.

Preview Data

Preview SOAP API Response in ZappySys XML Driver

Generate Code Option

Generate Example Code for ZappySys Driver

SOAP / REST API pagination in Excel

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

SOAP / REST API Error Handling in Excel

Sometimes errors occur... they just do and there is nothing you can do! Or can you? Actually, in ODBC PowerPack you can handle them in two ways.

METHOD 1 - Using Error Handling Options

When to use?

You may want to use them when your source is a resource located on the Internet; e.g. a file on a website, a file on an FTP server or just a plain API HTTP response. By default, when a remote server returns an error, data retrieval is stopped, an error is raised and no data is given back to you. This might not be always desirable.

Scenario 1

Imagine a scenario, that there is a web server which each day at 12 AM releases a new JSON file with that day's date as filename, e.g. http://www.some-server.com/data/2018-06-20.json. And, of course, you want to download it and use it daily in your Power BI report. But you have a problem: Power BI report data sources are refreshed each hour and you may get HTTP 404 status code (no file was found) when a file is not released yet. Which consequentially means other data sources won't be updated as well and you will see old and cached data on the report. That's where you could use Continue on any error or Continue when Url is invalid or missing (404 Errors) to avoid an error being raised and let other data sources to be updated.

Scenario 2

Another scenario is when you expect a web server to raise some kind of HTTP error when accessing a URL. You don't want ODBC Data Source to raise an error but instead, you want to get response data. That's where you can use Continue on any error or alike together with  Get response data on error to continue on an error and get the data:

METHOD 2 - Using Connection [Retry Settings]

Another scenario you may run into is a buggy web server. You ask it to give you some file or data and it, like a snotty kid, just doesn't give it to you! You have to ask twice or thrice before it does its job. If that's the case, you have to retry HTTP requests using Connection:

Security / Connection Types (Options for HTTP, OAuth, SOAP) in Excel

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

• HTTP
• OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

• SOAP WSS (when accessing a SOAP WebService)
• Static Token / API Key (when need to pass an API key in HTTP header)
• Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
• JWT Token (As per RFC 7519)

OAuth

If you are trying to access REST API resource, it is a huge chance, you will need to use OAuth Connection. Read this article to understand how OAuth authentication and authorization works and how to use it (article originally was written for SSIS PowerPack, but the concepts and UI stay the same): 
https://zappysys.com/blog/rest-api-authentication-with-oauth-2-0-using-ssis/

Conclusion

In this article, we learned how to use the ZappySys ODBC PowerPack to import JSON to Excel. We used the OData protocol and then we learned how to import from a JSON file to Excel.  With ZappySys ODBC Power Pack, you can query REST API information or JSON files using SQL and filter the information or write custom queries according to your needs.

References

• JSON – Introduction
• Learn REST: A RESTful Tutorial

Keywords: How to import JSON to Excel (Load File or REST API), how to convert JSON to Excel, Import JSON to Excel 2016, Load JSON to Excel 2013, Import REST API to Excel 2010

The post How to import JSON to Excel (Load File, REST API, SOAP XML) appeared first on ZappySys Blog.

SSIS comes with out-of-the-box support for reading and writing to Excel. However, it’s very restrictive if you want to make things dynamic, as any metadata inside DataFlow cannot be changed at runtime. Here are a few problems with using the native Excel Source or Destination.

• Metadata cannot be changed at runtime.
• Datatype guessing nightmare – only the first few rows are scanned, and the wrong datatype is selected.
• Cannot export multiple tables to Excel.
• Cannot split data into multiple tabs based on specific grouping criteria
• Cannot specify the location where data has to be written

SSIS export to Excel dynamically using Excel Export Task

If you want to generate Excel dynamically without any of the above hassle, then you can check the SSIS Excel Export Task. It comes with many advantages not found in the native Microsoft SSIS Excel Destination

Here is the list of a few features for the SSIS Excel Export Task

• Export multiple tables/views to Excel files (e.g., Sales% or do SalesJan|SalesFeb).
• Support for all major office versions (e.g, Office 2003, 2007, 2013+).
• Export SQL query output to an Excel file.
• A completely dynamic approach without limitations on DataFlow Strict Metadata.
• Option to export data to a specific starting location in an Excel Sheet (e.g, A5 will start writing to the  5th row, 1st column).
• Support for a password-protected Excel file.
• Option to clear a range of cells before writing data to an existing Excel workbook (e.g., A1:D5000 will clear A to D columns from 1st to 5000 rows).
• Automatically Split exported Excel data into multiple files by Size or Number of records.
• Automatically Split exported Excel data into multiple files or tabs when Split By Column (e.g., SplitBy=Country will create a new file for each country). If the SplitOnTab option is specified, then data is written to a new Excel tab rather than a file for each new group.
• Support for Compression and Append.
• Fully managed means it works out of the box on both 32-bit and 64-bit systems without any changes (Note that Native SSIS Excel functionality is 32-bit only).
• Support for SQL Server 2005, 2008, 2012, 2014 (32-bit and 64-bit).

Step-By-Step

SSIS Export to Excel File – Generate Excel files for selected tables/views

SSIS Export to Excel File – Generate Excel files for selected tables/views

SSIS Export to Excel File – Split Excel Data into Multiple Files

SSIS Export to Excel File – Split Excel Data into Multiple sheets

SSIS Export to Excel File – Date Formatting, Encoding Options

SSIS Export to Excel File – Execution Log

Conclusion

In this post, we have seen how Excel generation can be a headache when you have dynamic metadata. You can use the SSIS Excel Export Task to solve many of these challenges.

The post SSIS export to excel dynamically (supports multiple tables) appeared first on ZappySys Blog.