SQL Server guide

Reading data in SQL Server

This section covers steps for setting up Amazon S3 CSV File Connector in the ZappySys Data Gateway data source (based on API Driver) to read Amazon S3 CSV File data in SQL Server:

Create Data Source in Data Gateway based on ZappySys Amazon S3 CSV Driver

In this section we will create a data source for Amazon S3 CSV File in Data Gateway. Let's follow these steps to accomplish that:

Download and install ODBC PowerPack.
Search for gateway in Windows Start Menu and open ZappySys Data Gateway Configuration:
Go to Users tab and follow these steps to add a Data Gateway user:
- Click Add button
- In Login field enter username, e.g., john
- Then enter a Password
- Check Is Administrator checkbox
- Click OK to save
Now we are ready to add a data source:
- Click Add button
- Give Datasource a name (have it handy for later)
- Then select Native - ZappySys Amazon S3 CSV Driver
- Finally, click OK
AmazonS3CsvFileDSN

ZappySys Amazon S3 CSV Driver
Create and configure a connection for the Amazon S3 storage account.
You can use select your desired single file by clicking [...] path button.

mybucket/dbo.tblNames.csv

dbo.tblNames.csv

----------OR----------
You can also read the multiple files stored in Amazon S3 Storage using wildcard pattern supported e.g. dbo.tblNames*.csv.
```
Note: If you want to operation with multiple files then use wild card pattern as below 
(when you use wild card pattern in source path then system will treat target path as folder regardless you end with slash)

mybucket/dbo.tblNames.csv (will read only single .CSV file)
mybucket/dbo.tbl*.csv (all files starting with file name)
mybucket/*.csv (all files with .csv Extension and located under folder subfolder)
```
mybucket/dbo.tblNames*.csv

----------OR----------
You can also read the zip and gzip compressed files also without extracting it in using Amazon S3 CSV Source File Task.

mybucket/dbo.tblNames*.gz
Navigate to the Preview Tab and let's explore the different modes available to access the data.
1. --- Using Direct Query ---
  Click on Preview Tab, Select Table from Tables Dropdown and select [value] and click Preview.
2. --- Using Stored Procedure ---
```
Note : For this you have to Save ODBC Driver configuration and then again reopen to configure same driver. For more information click here.
```
  Click on the Custom Objects Tab, Click on Add button and select Add Procedure and Enter an appropriate name and Click on OK button to create.
  1. --- Without Parameters ---
    Now Stored Procedure can be created with or without parameters (see example below). If you use parameters then Set default value otherwise it may fail to compilation)
  2. --- With Parameters ---
```
Note : Here you can use Placeholder with Paramters in Stored Procedure.
Example : SELECT * FROM $ WHERE OrderID = '<@OrderID, FUN_TRIM>' or CustId = '<@CustId>' and Total >= '<@Total>'
```
3. --- Using Virtual Table ---
```
Note : For this you have to Save ODBC Driver configuration and then again reopen to configure same driver. For more information click here.
```
  ZappySys APi Drivers support flexible Query language so you can override Default Properties you configured on Data Source such as URL, Body. This way you don't have to create multiple Data Sources if you like to read data from multiple EndPoints. However not every application support supplying custom SQL to driver so you can only select Table from list returned from driver.
  
  Many applications like MS Access, Informatica Designer wont give you option to specify custom SQL when you import Objects. In such case Virtual Table is very useful. You can create many Virtual Tables on the same Data Source (e.g. If you have 50 Buckets with slight variations you can create virtual tables with just URL as Parameter setting).
```
vt__Customers
DataPath=mybucket_1/customers.csv

vt__Orders
DataPath=mybucket_2/orders.csv

vt__Products
DataPath=mybucket_3/products.csv
```
  1. Click on the Custom Objects Tab, Click on Add button and select Add Table and Enter an appropriate name and Click on OK button to create.
  2. Once you see Query Builder Window on screen Configure it.
  3. Click on Preview Tab, Select Virtual Table(prefix with vt__) from Tables Dropdown or write SQL query with Virtual Table name and click Preview.
Click OK to finish creating the data source
That's it; we are done. In a few clicks we configured the to Read the Amazon S3 CSV File data using ZappySys Amazon S3 CSV File Connector
Very important step. Now, after creating or modifying the data source make sure you:
- Click the Save button to persist your changes.
- Hit Yes, once asked if you want to restart the Data Gateway service.
This will ensure all changes are properly applied:

Skipping this step may result in the new settings not taking effect and, therefore you will not be able to connect to the data source.

Read data in SQL Server via Data Gateway

After configuring your data source using the ZappySys ODBC Driver, the next mandatory step to read that data in SQL Server is to create a Linked Server. SQL Server requires a Linked Server definition to access any ODBC-based source through the ZappySys Data Gateway, allowing the source driver data to be queried using standard T-SQL.

There are two ways to create the Linked Server:

Method 1: Using a SQL Script automatically generated by the Data Gateway
Method 2: Using SQL Server UI (SSMS) to manually configure the Linked Server

Method 1: Using a SQL Script automatically generated by the Data Gateway

The fastest and most reliable way to create the Linked Server is to use the SQL Script generated by the Data Gateway. This ensures all settings are applied correctly with minimal manual steps.

In the Data Gateway, open the App Integration tab.
Update the prefilled Linked Server Name if you want to use a custom name.
Select the AmazonS3CsvFileDSN data source which we created earlier as the Database.
Choose the correct SQL Server version for your environment.
- SQL 2019 or Lower (@provider='SQLNCLI11')
- SQL 2022 or Higher (@provider='MSOLEDBSQL')
Click Generate Code.
In the generated script scroll down to 4. Attach Gateway login with linked server step, enter your Data Gateway admin username and password.

'LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY'
Press Ctrl + A and Ctrl + C to copy the entire script.

LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY

AmazonS3CsvFileDSN
Paste the script into SQL Server Management Studio (SSMS) and run it.
That's it linked server is created in the SQL Server.

Finally, open a new query and execute a query we saved in one of the previous steps:

SELECT * FROM OPENQUERY([LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY], 'SELECT * FROM Orders')

Execute query at Linked Server to ZappySys Data Gateway in SSMS

SELECT * FROM OPENQUERY([LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY], 'SELECT * FROM Orders')

Sample SQL Script for Creating a Linked Server in SQL Server

USE [master]
GO
--///////////////////////////////////////////////////////////////////////////////////////
--Run below code in SSMS to create Linked Server and use ZappySys Drivers in SQL Server
--///////////////////////////////////////////////////////////////////////////////////////

-- Replace YOUR_GATEWAY_USER, YOUR_GATEWAY_PASSWORD
-- Replace localhost with IP/Machine name if ZappySys Gateway Running on different machine other than SQL Server
-- Replace Port 5000 if you configured gateway on a different port


--1. Configure your gateway service as per this article https://zappysys.com/links?id=10036

--2. Make sure you have SQL Server Installed. You can download FREE SQL Server Express Edition from here if you dont want to buy Paid version https://www.microsoft.com/en-us/sql-server/sql-server-editions-express

--Uncomment below if you like to drop linked server if it already exists
--EXEC master.dbo.sp_dropserver @server=N'LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY', @droplogins='droplogins'

--3. Create new linked server

EXEC master.dbo.sp_addlinkedserver
    @server = N'LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY'  --Linked server name (this will be used in OPENQUERY sql
, @srvproduct=N''
---- For MSSQL 2012, 2014, 2016, 2017, and 2019 use below (SQL Server Native Client 11.0)---
, @provider=N'SQLNCLI11'
---- For MSSQL 2022 or higher use below (Microsoft OLE DB Driver for SQL Server)---
--, @provider=N'MSOLEDBSQL'
, @datasrc=N'localhost,5000' --//Machine / Port where Gateway service is running
, @provstr=N'Network Library=DBMSSOCN;'
, @catalog=N'AmazonS3CsvFileDSN' --Data source name you gave on Gateway service settings

--4. Attach gateway login with linked server

EXEC master.dbo.sp_addlinkedsrvlogin
@rmtsrvname=N'LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY'  --linked server name
, @useself=N'False'
, @locallogin=NULL
, @rmtuser=N'YOUR_GATEWAY_USER' --enter your Gateway user name
, @rmtpassword='YOUR_GATEWAY_PASSWORD'  --enter your Gateway user's password
GO

--5. Enable RPC OUT (This is Optional - Only needed if you plan to use EXEC(...) AT YourLinkedServerName rather than OPENQUERY
EXEC sp_serveroption 'LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY', 'rpc', true;
EXEC sp_serveroption 'LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY', 'rpc out', true;

--Disable MSDTC - Below needed to support INSERT INTO from EXEC AT statement
EXEC sp_serveroption 'LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY', 'remote proc transaction promotion', false;

--Increase query timeout if query is going to take longer than 10 mins (Default timeout is 600 seconds)
--EXEC sp_serveroption 'LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY', 'query timeout', 1200;
GO

Method 2: Using SQL Server UI (SSMS) to manually configure the Linked Server

You can also create the Linked Server manually through SSMS if you prefer a visual setup. This method lets you configure the provider, data source, and security interactively.

First, let's open SQL Server Management Studio, create a new Linked Server, and start configuring it:

LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY

Microsoft OLE DB Driver for SQL Server

localhost,5000

AmazonS3CsvFileDSN

AmazonS3CsvFileDSN
- For SQL Server 2012, 2014, 2016, 2017, and 2019, choose SQL Server Native Client 11.0 as the provider.
- For SQL Server 2022 or higher, choose Microsoft OLE DB Driver for SQL Server as the provider.
Then click on Security option and configure username we created in ZappySys Data Gateway in one of the previous steps, e.g. john:
Optional step. Under the Server Options, Enable RPC and RPC Out and Disable Promotion of Distributed Transactions(MSDTC).

You need to enable RPC Out if you plan to use EXEC(...) AT [LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY] rather than OPENQUERY.
If don't enabled it, you will encounter the Server 'LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY' is not configured for RPC error.

Query Example:
```
DECLARE @MyQuery NVARCHAR(MAX) = 'SELECT * FROM Orders';

EXEC (@MyQuery) AT [LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY];
```
If you plan to use 'INSERT INTO <TABLE> EXEC(...) AT [LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY]' in that case you need to Disable Promotion of Distributed Transactions(MSDTC).
If don't disabled it, you will encounter the The operation could not be performed because OLE DB provider "SQLNCLI11" for linked server "MY_LINKED_SERVER_NAME" was unable to begin a distributed transaction. error.

Query Example:
```
INSERT INTO dbo.Products
DECLARE @MyQuery NVARCHAR(MAX) = 'SELECT * FROM Orders';

EXEC (@MyQuery) AT [LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY];
```

Finally, open a new query and execute a query we saved in one of the previous steps:

SELECT * FROM OPENQUERY([LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY], 'SELECT * FROM Orders')

SELECT * FROM OPENQUERY([LS_TO_AMAZON_S3_CSV_FILE_IN_GATEWAY], 'SELECT * FROM Orders')

Amazon S3 CSV File Connector Examples

The ZappySys API Driver is a user-friendly interface designed to facilitate the seamless integration of various applications with the Amazon S3 CSV File API. With its intuitive design and robust functionality, the ZappySys API Driver simplifies the process of configuring specific API endpoints to efficiently read or write data from Amazon S3 CSV File.

Click here to find more Amazon S3 CSV File Connector examples designed for seamless integration with the ZappySys API ODBC Driver under ODBC Data Source (36/64) or ZappySys Data Gateway, enhancing your ability to connect and interact with Prebuilt Connectors effectively.

Consume Data inside your App / Programming Language

Once you know how to load data from Amazon S3 CSV File Connector, you can click on one of the below links to learn the steps how to consume data inside your App / Programming Language from Amazon S3 CSV File Connector.

ODBC inside ETL / Reporting / BI Tools

ODBC inside Programming Languages

Key features of the ZappySys API Driver include:

The API ODBC driver facilitates the reading and writing of data from numerous popular online services (refer to the complete list here) using familiar SQL language without learning complexity of REST API calls. The driver allows querying nested structure and output as a flat table. You can also create your own ODBC / Data Gateway API connector file and use it with this driver.

Intuitive Configuration: The interface is designed to be user-friendly, enabling users to easily set up the specific API endpoints within Amazon S3 CSV File without requiring extensive technical expertise or programming knowledge.
Customizable Endpoint Setup: Users can conveniently configure the API endpoint settings, including the HTTP request method, endpoint URL, and any necessary parameters, to precisely target the desired data within Amazon S3 CSV File.
Data Manipulation Capabilities: The ZappySys API Driver allows for seamless data retrieval and writing, enabling users to fetch data from Amazon S3 CSV File and perform various data manipulation operations as needed, all through an intuitive and straightforward interface.
Secure Authentication Integration: The driver provides secure authentication integration, allowing users to securely connect to the Amazon S3 CSV File API by inputting the necessary authentication credentials, such as API tokens or other authentication keys.
Error Handling Support: The interface is equipped with comprehensive error handling support, ensuring that any errors or exceptions encountered during the data retrieval or writing process are efficiently managed and appropriately communicated to users for prompt resolution.
Data Visualization and Reporting: The ZappySys API Driver facilitates the seamless processing and presentation of the retrieved data from Amazon S3 CSV File, enabling users to generate comprehensive reports and visualizations for further analysis and decision-making purposes.

Overall, the ZappySys API Driver serves as a powerful tool for streamlining the integration of applications with Amazon S3 CSV File, providing users with a convenient and efficient way to access and manage data, all through a user-friendly and intuitive interface.

Reading data in SQL Server

Create Data Source in Data Gateway based on ZappySys Amazon S3 CSV Driver

--- Using Direct Query ---

--- Using Stored Procedure ---

--- Without Parameters ---

--- With Parameters ---

--- Using Virtual Table ---