This tutorial walks you through the process of extracting blog posts and comments data from Blogger using the REST API and Zappysys SSIS connectors. In this guide, you will learn how to authenticate with the Blogger REST API, configure a Zappysys REST data source, and load extracted data into SQL Server. This approach eliminates the need for custom code and provides a graphical interface for managing complex API integrations.

Zappysys provides a powerful REST connector that simplifies integrating cloud-based APIs into your SQL Server Integration Services packages. We will demonstrate a practical ETL pattern using ZappySys REST connectivity to extract data from Blogger and load it into SQL Server within SSIS.

Prerequisites

Before you begin, make sure you have the following:

• First, you will need to have SSIS installed
• Secondly, make sure to have SSDT
• Thirdly, access to a Blogger blog with appropriate permissions to read public posts and comments.
• Finally, do not forget to install ZappySys SSIS PowerPack.

Steps

Create/Select a Project in the Google API Console

1. Navigate to the Google API Console.
2. Click on the Project Dropdown at the top bar and either select an existing project or create a new one by clicking CREATE PROJECT.
   

   Google Console – Create or select a project

3. Once the project is set, click ENABLE APIS AND SERVICES.
4. Search for the Blogger API v3 and click ENABLE.
   

   Google Console – Enable Blogger API

5. Return to the main screen and click on the OAuth Consent Screen tab.
6. Select the External user type and click CREATE.
7. Provide the application name, user support email, and developer contact information, then click SAVE AND CONTINUE.
8. Click through the remaining scopes and summary pages (no additional scopes are required for this tutorial), and click SAVE AND CONTINUE on each.
9. Move to the Credentials tab.
10. Click CREATE CREDENTIALS in the top bar, choose OAuth Client ID, select Desktop App as the Application Type, and click Create to obtain your Client ID and Secret.
    

    Google Console – Create credentials

11. Download the credentials JSON file and save it in a secure location on your development machine.

Create the OAuth connection for Blogger

1. Create or select your SSIS project, and open the package.dtsx file.
2. Drag and drop the ZS REST API task from the Toolbox.
3. Inside Connection Manager, right-click and select New connection…
4. Inside the SSIS connectors, select the ZS-OAuth
   
   

   Create a new SSIS OAuth API Connection Manager

5. Select the Google API provider
6. Enter the Client ID and Client Secret you generated in the past step
7. Use Blogger URL scope: https://www.googleapis.com/auth/blogger
8. Generate the token
9. Test the connection to ensure it is working correctly.
10. Click OK to save the connection.
    

    OAuth connection – Blogger API

Get information from a Blog

1. Create a string variable {{User::BlogID}} and enter the ID from the blog you want to get information from.
2. From the SSIS Toolbox, drag the ZS REST API Task onto the Control Flow designer.
3. Double-click the task to open the editor.
4. Select the connection manager you created in the previous step.
5. In the URL Path field, enter: https://www.googleapis.com/blogger/v3/blogs/{{User::BlogID}}/posts
6. Set the HTTP method to GET.
7. Click the Preview tab to test the API connection and view sample data.
8. If the preview is successful, click OK to configure the task.
   

   REST API task – Blogger example

Save the information in a SQL table

1. Drag and drop a Data Flow into the Control Flow and double-click it.
   

   Drag and drop Data flow.

2. In the Data Flow, drag and drop the JSON source, then double-click it to configure it.
   

   SSIS JSON Source – Drag and Drop

3. Use the same configuration as in the REST API task, but first enable the Use credentials option.
4. Enter the URL, select the OAuth connection, and use the Method GET.
5. You can use the filter option to get the object you want to get data from: $.Items[*]
6. Preview the data and click OK to save the configuration.
   

   JSON Source – Blogger Example Request
7. Now drag and drop a destination component. In our case, we will use the Upsert destination component.
   

   SSIS – Data Flow – Drag and Drop Upsert Destination Component

8. Double-click in the Upsert destination component to configure it.
9. Select the Action you want to use.
10. Create or select a database connector.
11. Create or select a table.
12. Use the options Insert or/and Update.
13. Map the column; if the columns have the same name, it will be automatic.
14. Select at least one column for a primary key.
15. Click OK to save the configuration.
    

    Upsert destination – Blogger example

16. Run the package, and you will see the data from Blogger in your database
    

    Blogger – result

Code Examples

Sample REST API Request

Here’s an example of retrieving blog posts using the Blogger REST API:

GET https://www.googleapis.com/blogger/v3/blogs/{blogId}/posts
Authorization: Bearer {access_token}
Content-Type: application/json

Example response:

{
  "kind": "blogger#posts",
  "items": [
    {
      "kind": "blogger#post",
      "id": "7109703471293209437",
      "title": "Getting started with Blogger API",
      "content": "Learn how to integrate Blogger with your applications...",
      "published": "2026-03-15T10:30:00Z",
      "updated": "2026-03-20T14:15:00Z",
      "url": "http://example.blogspot.com/2026/03/getting-started.html",
      "author": {
        "displayName": "John Doe"
      }
    }
  ]
}

SQL Table Structure

Recommended table structure for storing Blogger posts data:

CREATE TABLE BloggerPosts (
    PostID BIGINT PRIMARY KEY,
    BlogID BIGINT NOT NULL,
    Title NVARCHAR(MAX),
    Content NVARCHAR(MAX),
    AuthorName NVARCHAR(255),
    PublishedDate DATETIME,
    UpdatedDate DATETIME,
    PostURL NVARCHAR(MAX),
    CreatedDate DATETIME DEFAULT GETDATE(),
    ModifiedDate DATETIME DEFAULT GETDATE()
);

Conclusion

You have successfully learned how to use Zappysys SSIS connectors to extract data from the Blogger
platform using the REST API. With ZappySys SSIS PowerPack, connecting Blogger to your SQL Server becomes
straightforward and reliable. By configuring the REST connector for Blogger and using the native SSIS destination,
you can build a robust ETL pipeline for extracting blog data.

Explore the full capabilities of the ZappySys SSIS PowerPack to optimize your data integration
tasks. By applying this pattern, you can extend your solution to include additional transformations, data
validation, error handling, and scheduling through SQL Server Agent.

Need Help?

If you encounter any issues, our support team is here to help:

1. Live Chat — Use the chat widget (bottom-right corner of this page)
2. Email — support@zappysys.com
3. Support Center — Visit the ZappySys Support Portal

The post SSIS tutorial: How to connect Blogger with SSIS appeared first on ZappySys Blog.

In our previous post, we explored how to read and write MongoDB data using the ZappySys MongoDB Source and Destination SSIS connectors. In this article, you will learn how to connect MariaDB to MongoDB using ZappySys SSIS PowerPack and transfer data between both systems.

We will demonstrate a practical ETL pattern using ZappySys ODBC/JDBC connectivity to extract data from MariaDB and load it into MongoDB within SSIS.

Prerequisites

Before you begin, make sure you have the following:

• First, you will need to have SSIS installed
• Secondly, make sure to have SSDT
• Thirdly, access credentials for both MariaDB and MongoDB
• Finally, do not forget to install ZappySys SSIS PowerPack and ZappySys ODBC PowerPack

Steps

Configure MariaDB Connection Using ODBC (JDBC Bridge)

Since MariaDB connectivity is handled via JDBC, we will use the ZappySys JDBC Bridge Driver to expose it as an ODBC data source.

1. Visit the official MariaDB website.
2. Download the latest MariaDB JDBC driver.
3. Save the .jar file locally, for example: D:\Zappysys\drivers\mariadb-java-client-3.5.7.jar
4. Download and install ZappySys ODBC PowerPack if not already installed.
5. Open ODBC Data Sources (64-bit).
   

   Open ODBC Data Sources

6. Create a User data source (User DSN) based on ZappySys JDBC Bridge Driver:
   

   Create a User data source

7. Configure the JDBC Connection. Fill in the connection details
   • Connection String  jdbc:mariadb://{host-name}:{port}/{database-name}

   • JDBC Driver File D:\Zappysys\drivers\mariadb-java-client-3.5.7.jar

   • Username root

   • Password ********

   • Connection Parameters[]
8. Click Test Connection. If configured correctly, you should see a success message.
   

   Zappysys JDBC – MariaDB

9. You can also use the Preview tab to select the table and execute SQL queries. For advanced SQL examples, refer to the JDBC Bridge documentation.
   

   Zappysys JDBC – MariaDB preview result

Connect MariaDB to SSIS

1. Open Visual Studio.
2. Create a New Project.
3. Select Integration Services Project.
4. Provide a project name and location, then click OK.
5. Drag and drop a Data Flow Task from the SSIS Toolbox onto the Control Flow surface.
   

   Drag and drop Data flow.

6. Double-click the task to open the Data Flow Designer.
7. Drag and drop an ODBC Source component into the Data Flow.
   

   Drag and drop the ODBC source.

8. Double-click the component to configure it.
9. Click New… to create a new ODBC Connection Manager.
   

   Creating an ODBC Connection Manager in SSIS

10. Select the MariaDB ODBC DSN you created earlier.
11. Click Test Connection and click OK to save the configuration.
    

    Creating an ODBC Connection Manager in SSIS from JDBC

12. Choose Data Access Mode: Table name or SQL command.
13. Select the desired table or enter a custom query.
14. Click Preview to validate the data, and click OK to save the configuration.
    
    

    Preview an ODBC Connection Manager in SSIS from JDBC

Configure MongoDB Destination

1. In the Connection Manager pane, click New Connection.
2. Select ZS-MONGODB.
3. Configure the connection:
   • Host
   • Username
   • Password
   • Database
4. Use the other tabs for more configuration if needed
5. Click Test Connection and Click OK.
   

   MongoDB connection configuration

6. Drag and drop the MongoDB Destination component.
   

   Drag and drop the MongoDB destination

7. Connect it to the ODBC Source and double-click to configure it.
8. Select the MongoDB connection.
   

   MongoDB Destination – Select connection

9. Go to the Properties tab and choose the operation and the target table.
   

   MongoDB Destination – Properties configuration

10. Go to the Column Mappings tab and map source columns to destination columns.
    

    MongoDB Destination – Map the columns.

11. Click OK to save the configuration.
12. Run the SSIS package. If everything is configured correctly, the data will be transferred from MariaDB to MongoDB.
    

    MongoDB Destination – Final Result

Conclusion

With ZappySys SSIS PowerPack and ODBC PowerPack, connecting MariaDB to MongoDB becomes straightforward and reliable.

By configuring the ODBC/JDBC bridge for MariaDB and using the native MongoDB destination in SSIS, you can build a robust ETL pipeline between the two databases.

Explore the full capabilities of the ZappySys SSIS PowerPack and ZappySys ODBC PowerPack to optimize your data integration tasks.

Need Help?

If you encounter any issues, our support team is here to help:

• Live Chat — Use the chat widget (bottom-right corner of this page)
• Email — support@zappysys.com
• Support Center — Support | ZappySys

The post SSIS tutorial: How to load MariaDB data into MongoDB appeared first on ZappySys Blog.

In this article, you’ll learn how to connect SSIS PowerPack (ZappySys) to AliPay OpenAPI to call REST endpoints from SSIS. We’ll focus on the practical integration pattern AliPay requires: sending requests to the OpenAPI gateway with the correct parameters and a valid RSA2 signature, then parsing JSON responses in SSIS.

AliPay OpenAPI basics (what SSIS must send)

AliPay OpenAPI calls typically go through a single gateway endpoint:

• Sandbox gateway:
  

  https://openapi-sandbox.dl.alipaydev.com/gateway.do

A request is built from:

• Standard parameters (for example: app_id, method, charset, sign_type, timestamp, version)
• biz_content (usually JSON)
• sign (a signature generated from the parameters using RSA/RSA2—AliPay recommends RSA2)

AliPay requires you to construct a “string to sign” from parameters and attach sign and sign_type to the request, then validate/verify responses using the AliPay public key.

High-level integration pattern in SSIS

Because AliPay signing is mandatory, the most reliable SSIS approach is a two-step pattern:

1. Generate the signature (RSA2) for the request parameters.
2. Call the AliPay gateway from ZappySys (REST API Task or REST API Source), passing the signed parameters.

ZappySys components support custom HTTP methods, headers, request bodies, and advanced options—so they work well once you have the signed payload ready.

Steps

Start in the AliPay Sandbox

AliPay strongly recommends using the sandbox for development and debugging. In the sandbox, you must:

• An AliPay Developer Account
• An AliPay App ID
• Use the sandbox gateway URL
• RSA2 Private Key (generated in AliPay developer console)
• AliPay Public Key (for verification)
• Access to the AliPay Sandbox environment (recommended for testing)

Decide where you will generate the RSA2 signature

You have three common options:

• Option A (Recommended): SSIS Script Task (C#) to generate signature
  Use a Script Task before the ZappySys call to build parameters and compute sign.
• Option B: Use an AliPay SDK in a small helper app
  Run it via Execute Process Task to output signed parameters (JSON/text), then pass them to ZappySys.
• Option C: Use a signing microservice
  SSIS calls your internal service to get sign, then calls AliPay.

AliPay’s documentation emphasizes the signature process and verification using the AliPay public key.

Example: What a signed AliPay request looks like

AliPay’s gateway model is “parameters + signature.” Conceptually, your final request includes:

• app_id
• method (the API name you’re calling)
• charset (often utf-8)
• sign_type (recommended RSA2)
• timestamp
• version
• biz_content (JSON)
• sign (computed over the canonicalized parameter string)

AliPay documents that sign and sign_type are appended after creating the string to be signed, and verification uses the AliPay public key.

Configure the ZappySys REST API Task (Control Flow)

Use this when operating (e.g., creating a payment, refunding, or querying).

1. Drag ZappySys REST API Web Service Task into Control Flow.
2. Set the task:
   • Method: POST (commonly used with AliPay gateway)
   • URL:
     

     https://openapi-sandbox.dl.alipaydev.com/gateway.do

   • Headers: use
     

     Content-Type: application/x-www-form-urlencoded

   • Body: form-encoded parameters; use this format: key1=value1&key2=value2…

     app_id=YOUR_APP_ID&method=alipay.trade.query&charset=utf-8&sign_type=RSA2&timestamp=2026-02-20 10:00:00&version=1.0&biz_content={"out_trade_no":"202602200001"}&sign=GENERATED_SIGNATURE

3. Click Test Request/Response to confirm you get a valid JSON response.
   

   Alipay REST API

Troubleshooting common AliPay + SSIS issues

• invalid-signature / missing-signature: most often caused by parameter ordering, incorrect URL-encoding, wrong key pair, or using sandbox keys against production (or vice versa).
• Wrong gateway: sandbox and production are fully isolated—ensure your app_id,keys, and gateway host match the environment.
• Encoding issues: confirm UTF-8 everywhere and ensure parameters are encoded exactly once before submission.
• Parsing JSON: use ZappySys JSON Source preview and JSONPath filters to flatten nested objects.

Security best practices

• Store private keys securely (SSIS variables protected, encrypted config, or secret vaults).
• Do not hardcode secrets in packages.
• Use a sandbox for development, then switch to production only after full validation.

Conclusion

Download SSIS PowerPack today and see for yourself how easy it is to connect SSIS PowerPack (ZappySys) to AliPay is absolutely achievable once you treat AliPay’s OpenAPI as a signed gateway integration. The key is generating a correct RSA2 signature for every request, then using ZappySys REST components to send the parameters and parse responses.

Still need help?

If the issue persists, please get in touch with our support team:

• Live Chat: Open the chat widget (bottom right of this page)
• Email: support@zappysys.com
• Support Center: Support | ZappySys

The post SSIS tutorial: How to connect to AliPay appeared first on ZappySys Blog.

In this article, we’ll learn how to integrate a documented API in Swagger with SQL Server Integration Services (SSIS) using ZappySys PowerPack.

Swagger enables us to visualize and test REST endpoints, while ZappySys SSIS provides advanced components for consuming APIs and transforming data in ETL processes.

Steps

Get the Swagger endpoints

1. Review the server’s endpoint list. In this example, we will explore the Pet category.
   

   Swagger – Pet category

2. From the list, we can see multiple endpoints and HTTP methods, including GET, POST, PUT, and DELETE.
3. Click on an endpoint to expand it. For this example, we will use Add a new pet to the store.
   
   

   Swagger – Create a new pet

4. Click Try it out. Swagger will allow you to edit the request body and choose the content type (JSON or XML).
5. Click Execute. Swagger will display
   • The generated request in cURL format
   • The Request URL
   • The Response body
   • The HTTP status code
6. In the Response JSON, we can see the new pet with status code 200, indicating success.
7. Now, with the CURL syntax, we can use that inside our tool.
   

   Swagger – Create a new pet CURL

POST request: Create Pet Request

1. Open Visual Studio and create a new Integration Services Project.
2. After the project is created, locate the SSIS Toolbox on the left panel.
3. Drag and drop the REST API Task onto the Control Flow surface and double-click it to configure
   

   Drag and Drop REST API Task

4. Use the cURL generated from Swagger. For reference, you can review our article about converting cURL to an SSIS API Task.
   

   curl -X 'POST' \
     'https://petstore.swagger.io/v2/pet' \
     -H 'accept: application/json' \
     -H 'Content-Type: application/json' \
     -d '{
     "id": 123,
     "category": {
       "id": 456,
       "name": "Friendly"
     },
     "name": "Zappy",
     "photoUrls": [
       "string"
     ],
     "tags": [
       {
         "id": 0,
         "name": "tech"
       }
     ],
     "status": "available"
   }'

5. -X POST represents the HTTP method. Set the Request Method to POST.
6. Enter the URL from the cURL command.
7. Each -H represents a Header. Add the corresponding headers inside the REST API Task.
8. The -d parameter represents the Body. Paste the JSON payload into the Body section.
9. Click Test Request/Response to validate the configuration.
   

   SSIS – Create a new pet

10. You can use additional components such as JSON Source or XML Source, depending on how you plan to process the response.

POST request: Upload a file

1. Next, we will upload an image for the pet we created earlier
   

   Swagger – upload image

2. Generate the CURL code.
   

   curl -X 'POST' \
     'https://petstore.swagger.io/v2/pet/123/uploadImage' \
     -H 'accept: application/json' \
     -H 'Content-Type: multipart/form-data' \
     -F 'file=@new.jpg;type=image/jpeg'

3. Configure the REST API Task with:
   • Request Method: POST
   • URL
   • Headers
   • File parameter
4. Check the option File upload.
5. Specify the full file path (e.g., file=@D:\Zappysys\new.jpg).
6. Click Test Request to validate the upload.
7. You can find an article about uploading files here.
   

   SSIS – upload an image

GET request: Get the Pet information

1.  Now we will get the pet we created earlier. This is the example
   

   Swagger – get pet by id

2. We generate the CURL
   

   curl -X 'GET' \
     'https://petstore.swagger.io/v2/pet/123' \
     -H 'accept: application/json'

3. Inside the Visual Studio project, drag and drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow designer.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

4. From the SSIS toolbox, drag and drop a JSON Source onto the data flow designer surface, and double-click on it to edit it
   

   SSIS JSON Source – Drag and Drop

5. We use the method GET
6. We enter the URL with the ID from the earlier response.
7. We can use the filter to get the object we need, and now we can check the data.
8. Preview the data, and you will see the Pet’s information
   

   SSIS – get a pet by ID

PUT request: Update an existing pet

1. We go to the updated pet example and run the example
   

   Swagger – update a pet

2. Generate the CURL
   

   curl -X 'PUT' \
     'https://petstore.swagger.io/v2/pet' \
     -H 'accept: application/json' \
     -H 'Content-Type: application/json' \
     -d '{
     "id": 0,
     "category": {
       "id": 0,
       "name": "string"
     },
     "name": "doggie2",
     "photoUrls": [
       "string"
     ],
     "tags": [
       {
         "id": 0,
         "name": "string"
       }
     ],
     "status": "available"
   }'

3. Inside the JSON source, we configure the HTTP method to PUT.
4. Change the url and the header.
5. Inside the Body we can change the values from the existing pet.
6. Test the connection and check the result.
   

   SSIS – update a pet

DELETE request: Remove an existing pet

1. Now we will delete the pet using this example.
   

   Swagger – delete a pet

2. Generate the CURL to delete the pet
   

   curl -X 'DELETE' \
     'https://petstore.swagger.io/v2/pet/1234' \
     -H 'accept: application/json' \
     -H 'api_key: special-key'

3. Inside the JSON source, configure the HTTP method to DELETE.
4. Change the URL to include the pet’s ID.
5. Inside the header, we need to insert an API key.
6. Test the connection, and we will delete the pet.
   

   aSSIS – delete a pet

Authentication method

1. Some APIs require authentication. You can configure authentication using an HTTP Connection Manager.
2. Enter the login URL if required.
3. Select the credential type (e.g., Basic – Username/Password).
4. Enter the credentials.
5. Click OK to save the configuration.
6. This connection can be reused across multiple REST API Tasks or JSON Sources.
7. For more details on basic authentication, click here. For more details on the OAuth connection, click here.
   

   Swagger – SSIS basic authentication

Conclusion

Integrating Swagger-documented APIs with ZappySys SSIS becomes significantly easier when using the generated cURL command as a reference. Swagger provides a clear and interactive way to understand API endpoints, request structures, headers, and payload formats, while cURL offers a direct, executable representation of the HTTP request.

By translating the cURL command into the REST API Task or JSON Source configuration, developers can confidently replicate API calls inside SSIS without guesswork. This approach ensures accuracy, reduces troubleshooting time, and provides a consistent workflow for handling GET, POST, PUT, DELETE operations, file uploads, and authentication scenarios.

Download SSIS PowerPack today and see for yourself how easy it is to combine Swagger and ZappySys SSIS, enabling organizations to seamlessly integrate modern REST APIs into enterprise ETL processes and bridge the gap between cloud-based services and traditional SQL Server data platforms.

Still need help?

If the issue persists, please get in touch with our support team:

• Live Chat: Open the chat widget (bottom right of this page)
• Email: support@zappysys.com
• Support Center: Support | ZappySys

The post SSIS tutorial: How to connect to Swagger appeared first on ZappySys Blog.

In this post, we will show how to save a list of files into a table a table using SSIS. The article will show how to get a list of files and then store the list of files into an Excel file.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from the Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Optional (If you want to Deploy and Schedule ) – Deploy and Schedule SSIS Packages

Step-by-step process to save the list of files and folders into an article

Save the information in a variable with the Advanced file system task

1. Drag and drop the Advance File System Task from the SSIS toolbox, chose to Get file list as ADO.net Data table and select an object variable type

Getting the file list in a variable

Save the result in a file with Logging task in order to save a list of files into a table

2. The next step is to save the result in a HTML file: select the file destination, uncheck append and prefix datetime. Select the file in log mode and none in message type, and insert the variable in the message as follows:

Save the value from the variable in a file without a prefix

Read the table and save it on a new file in order to save a list of files into a table

3. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox

4. Double click on the DataFlow task to see DataFlow designer surface.

5. From the SSIS toolbox drag and drop HTML source and insert the path for the new file, you can preview the file if it was previously created.

Preview the HTML file

7. Then you can use a destination like the Excel destination, JSON destination or CSV destination

Save the path list into an excel file

If everything is OK, the information will be stored in Excel. However, you can select any destination.

Conclusion

In this article, we show how to save a list of files in a table. Basically, we use the Advanced file system to save the list in a file. We use the Logging Task

The post How to save a list of files into a table using SSIS. appeared first on ZappySys Blog.

This time we will explain how to get all URLS from emails using MS Outlook. Sometimes we need get all URLs from emails. In this post, we will show how to do this. Step by step using SSIS.

Microsoft Graph API is a unified way to access many Microsoft services API including Office 365 API

Prerequisites

Before we perform steps listed in this article, you will need to make sure following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Optional (If you want to Deploy and Schedule ) – Deploy and Schedule SSIS Packages

Steps-by-step process to download images from HTML using SSIS

Register Application (OAuth2 App for Graph API)

First, check our article about how to get the OAuth2 connection, click here for the article

Get the information for the body content from the emails

1. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.

Drag and drop Data flow task

2. Double click on the DataFlow task to see the DataFlow designer surface.

3. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.

Drag and drop a JSON sourcec

4. Select the OAuth connection you created, then use this URL to get the body content

https://graph.microsoft.com/v1.0/me/messages?$select=subject,body,bodyPreview,uniqueBody

Getting the information from the emails

5. Drag and drop the trash destination and save the result in a file with the following configuration. Make sure to check the Overwrite target file if exists.

Save the body from the emails in a file

Now we have saved the email body in the file, to extract the links we need to go control flow and drag and drop the Regular Expression Parser Task and follow the steps from below section.

Getting all URLs from emails inside a variable

6. The next step is to save the URLs’. We will use the Regular Expression Parser Task for this.
Also, you need to use Regex and here is one example you can use, in this page Regex101 you can check more details about the expressions we are using:

Getting all URL from the emails

Expression:

href="(.*?)"{{*}}

Save the result in a file of the URLs from emails

7. Finally, use the Logging task to save the URLs in a file, use the following configuration

Uncheck all options and select message type as none

Conclusion

If everything is OK, you will be able to download the URLS from your emails. To do that, we read the list from the body of the emails. Then we get the URLs of the emails using expressions. Finally, we store them in a local file.

The post How to get all URLs from emails from Outlook appeared first on ZappySys Blog.

In our previous post we saw how to read API data in SSIS. This post we will cover how to call NetSuite RESTlet to read / write data in SSIS or ODBC Apps. For demo purpose we will use SSIS PowerPack REST API Task but concepts and many UI elements remain same for ODBC JSON Driver too. This article assumes you know how to write Netsuite RESTlet and you have obtained Credentials to call RESTlet from extranal app.

What is NetSuite RESTlets

NetSuite added a really nice feature to expose business data and workflows as API by writing Javascript functions which they call it RESTlets. You can secure and customize these restlets as per your need.

You can automate CRUD (Create /Read / Write / Delete) operations in NetSuite by writing special functions in JavaScript and expose those functions as API URL which can be called by any external Systems (i.e. SSIS Packages , ODBC Apps like Power BI, Informatica, SSRS).

RESTlet can be secured few ways such as OAuth 1.0 and OAuth 2.0 authentication. We will cover both approach briefly in this article. You can choose any one as per your need

Creating NetSuite RESTlet / Obtain OAuth Credentials (Client ID and Secret)

Below video shows how to obtain necessary credentials to call Netsuites API (RESTlet).

At mark 4:30 seconds it will show you How to use OAuth 2.0 for calling RESTlet

Create NetSuite RESTlet Connection

There are few ways to authenticate NetSuite RESTlet. If you are using ZappySys Products then it makes it super easy to use either approach. Lets look at both approaches.

Using OAuth 2.0 (Recommended)

As per above video if you obtained Client ID and Secret for OAuth 2.0 you can follow below steps to configure OAuth connection to call RESTlet.

What you need to use OAuth 2.0.

• Your Account ID (it goes in URL)
• Your Client ID and Secret (Obtained from Netsuite Portal) – See above video Mark 04:30
• Register Redirect URL in NetSuite portal (we recommend you use https://zappysys.com/oauth as call back Url for Automatic Code extraction)

Steps to configure OAuth Connection

1. Right click on Connection Panel and choose Choose “New Connection..”. Select ZS-OAUTH type
   
   
   

   Create new SSIS OAuth API Connection Manager

    

    

2.  Enter following things
   ————————-
   On General Tab
   ————————-
   ClientID and Secret: <get it from NetSuite>
   Authorization URL: https://<accountID>.app.netsuite.com/app/login/oauth2/authorize.nl
   Token URL: https://<accountID>.suitetalk.api.netsuite.com/services/rest/auth/oauth2/v1/token
   Scope: restlet
   

   Configure NetSuite OAuth 2.0 Connection for API call

    

   ————————-
   On Advanced Tab
   ————————-
   Callback / Redirect URL:  https://zappysys.com/oauth  (this must be entered same way in NetSuite Portal)

   

   OAuth2 Connection – Callback / Redirect URL

3. Now go back to General Tab and click Generate Token button
4. Follow login screen and try to finish the flow. At the end it will generate Tokens and populate on UI
5. Click OK to save

Using OAuth 1.0

Another approach is use OAuth 1.0 (Older version). If you have enabled OAuth 1.0 in NetSuite then follow below steps (Create connection is same as before).

1. Setup General Tab and Advanced Tab as below.
   On ==== General Tab====:
   Enter ClientID, ClientSecret, AccessToken, AccessToken Secret
   On ==== Advanced Tab ====:
   Check Do Not Include Empty Key/Value Pairs
   Select Signature Method as HMAC-SHA256
   For Extra Token Attribute enter realm=<your-account-number>_SB1
2. Click OK to Save

Call NetSuite RESTlet in SSIS

1. Create a new SSIS Package
2. Drag REST API Task from SSIS Toolbox
3. Double click the Task to configure
4. Select URL from Connection
5. From Connection dropdown select OAuth connection we created in the earlier section (either OAuth 1 or OAuth 2)
6. Enter the URL as below. Assuming you copied URL from NetSuite Replace script ID and account number
   

   https://<account-number>-sb1.restlets.api.netsuite.com/app/site/hosting/restlet.nl?script=<scriptid>&deploy=1

   Enter <account-number>
   Enter <scriptid>
7. Change Method to POST if your script supports Parameters (Else you can supply id=xxxx and recordtype=xxxxx in the URL for GET request)
8. If its POST request then chnage ContentType to ApplicationJSON (application/json)
9. Enter your Input in the body. In below example our RESTlet takes sql query as input so we enter just one parameter named query but you can have different parameter(s)
   

   Syntax:  { param1_name : <value> , param2_name : <value> ....  }

10. Now click Test see everything works
    

    Calling NetSuite RESTlet in SSIS (Using REST API Task)

Conclusion

We saw how easy it is to call NetSuite RESTlets or other NetSuite API in SSIS / other BI Apps using ODBC Drivers provided by ZappySys. Download SSIS PowerPack and try your self see how easy it is to consume virtually any API in SSIS.

The post How to call NetSuite RESTlet API in SSIS / ODBC appeared first on ZappySys Blog.

In our previous post, we saw how to read Salesforce data using SSIS Salesforce Source (Which uses SOAP API under the hood). This may be slow if you have millions of rows because Salesforce Source paginates every 2000 rows, so it’s not a good option to read millions of rows to iterate on a large Salesforce dataset (Table or Query). To increase the read speed, you can use BULK API. This article explains how to read salesforce data using BULK API and increase read speed.

NEW Version (v5.5+)

In the new version (v5.5), we have added an option to query in Bulk Mode. This means if you have millions of Salesforce object rows to read, no worries; it may be very fast in the new version. Previously, it used to take a long time because it read in batches of 2000 rows in each request. With the Bulk API, the batch size is really large.

Here is how to set Salesforce Bulk API Mode in SELECT query (Read operation).

1. Install new version of SSIS PowerPack. Make sure it is v5.5 or higher.
2. Open existing Salesforce Source UI
3. Make sure you check [Use Bulk API] option like below. if you do not see this option then most likely you are using older version.
   -OR-
   Just add #bulkmode prefix in the SQL query like below, it will use Bulk API
   

   Bulk API mode for SSIS Salesforce Source (Read Large amount of data using SOQL Query)

OLD Version

If you are using an older version of SSIS Power Pack (v5.4.0 or older), you can follow the steps mentioned in the following section.

We will see how to query using Salesforce Bulk API in SSIS. It involves three simple steps like the one below.

1. Call BULK API – Submit SOQL Query and receive a JobId back (Use SSIS Salesforce API Task)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query

2. Keep checking Job Status Until Status is JobComplete (Use SSIS Salesforce API Task)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}

3. Read Query Result  (Use SSIS CSV Source)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results

Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode / large dataset)

Now, let’s look at everything in detail.

Obtain Salesforce Credentials

The first step to call Salesforce API is to obtain credentials (security token or OAuth ClientId /Secret)

Method#1 – Check this article to use UserId/ Password / SecretToken
Method#2 – Check this article to use the OAuth Connection Option

Create Salesforce Connection

Once we have Salesforce Credentials, we can create a new Salesforce Connection (You can do this from Task UI or the below way)

1. Open Visual Studio ( Assuming you already have SSIS installed along with ZappySys PowerPack )
2. Create a New SSIS Package Project.
3. Right-click on the Connection Manager panel and click on New Connection.
   
4. Select ZS-SALESFORCE Connection Manager from the Connection Managers list and Click on the Add Button.
   
5. Now in Connection Manager UI, Enter your SalesForce Credentials (OAuth Client Id/. Secret –or— User Name, Password and Security Token), and leave all other properties as it is.
   

Call Create Job – Retrieve / Save JobId

Once the connection is created, we can create our first step. Basically, we will call this API to create Job (Bulk Read)

1. Here, in Visual Studio, drag and drop the ZS Salesforce API Task on the design surface.
2. Double click the Task and configure Request Settings Tab like below
   – Select Salesforce Connection (Created in the previous section)
   – Check Use direct URL
   – Enter URL like below (Replace {your-instance} with your own instance (i.e. na9) . For more information on how to find out my
   salesforce instance id check this article
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query

   – Request Method POST
   – Enter Body like below (You can use your own SOQL query). Keep in mind SQOL must be encoded for JSON format. This means the entire SQL must be in one line (replace new lines with \r\n , quotes with \” and tabs with \t)
   For example, to execute SELECT Id, Name FROM Accounts, you can write something like the one below in BODY.
   

   {
     "operation": "query",
     "query": "SELECT Id, Name\r\nFROM Account"
   }

   The above is for below (with a new line)
   

   SELECT Id, Name
   FROM Account

   

   Create Job – Call Bulk API to Read / Query using SOQL

3. Now Click Test Request (Copy the JobId which we will use in the Variable for Designtime Testing)
4. On the Response Tab, configure it like the below.
   – Select Response content Type = Json
   – Enter $.id as the expression
   – Check Save response option
   – Select Save to Variable and click New to create a variable. Name it JobId and enter a sample JobId value (copied from the previous step)
   

   Extract Salesforce JobId save to SSIS Variable

Do Status Check – Look for JobComplete

Once we create a job and store JobId in the variable we have to keep checking Job status every X seconds. Don’t worry you don’t have to use Loop or timer control for that. ZappySys made it so simple to check such thing without Loop /Timer.

1. Drag and drop the ZS Salesforce API Task in the design surface.
2. Double click and configure like below. URL will be different this time. Use variable as below and replace your instance ID
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}

   

   Check Salesforce Job Status Periodically (Without Loop + Timer Controls)

3. Now go to the Status Check Tab and configure it, as shown below. It will keep looking for the words below in the response to decide whether it is a Success or Failure. If those words are not found, they will be checked every 5 seconds (Maximum 1200 secs).
   – Check Enable Status Check option
   – For Success Value – Enter JobComplete
   -Check Fail task option
   – Enter Cancel|Failed|Abort in the text box
   

   Configure Job Status Check Loop using Salesforce API Task

Read data (Compressed CSV Format)

Last step is read the result from completed Salesforce Job. Its in CSV format so we will use Advanced SSIS CSV Source which supports API connection to Salesforce. For that, we will use the same Salesforce API Task used before.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
2. Double click on the DataFlow task to see DataFlow designer surface.
3. Here, In Visual Studio, drag and drop the ZS CSV Source (Web API or File) in the design panel
   
4. Double click on ZS CSV Source (Web API or File) Configure it like below.
   – Enter URL to read result like this (You can supply OPTIONAL maxRecords parameters to control how much data comes in a single response)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results
   --OR--
   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results?maxRecords=50000

   – Check Use Credentials and select Salesforce Connection
   – Click Preview (assuming JobId variable has a valid completed JobId)
   

   Read Salesforce Bulk API Job Result – Using SSIS CSV / API Source

5. Now, set up Pagination as below to read all records using an automated loop. You will need VERSION Released after Jun 20, 2022 to use Regular Expression feature in Header Value extractionBy default, Salesforce doesn’t send all records in response so we need to setup pagination. In below example we are reading Sforce-Locator attribute from Response and sending it to the next URL as locator=<continue-token> value
   – Choose Response Header Contains Continue Token 
   – Enter below expression in Page Num Indicator
   

   locator=Sforce-Locator(^((?!null\b).)*$)

   Basically, the above expression will keep looping until we find Sforce-Locator : null in the response header.
   

   Pagination for Salesforce Bulk Query API (Get Header Value from Sforce-Locator)

6. Click OK to save and now run entire package like below.

Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode / large dataset)

Conclusion

In this article we saw how to use various ZappySys Components / Task to achieve complex Salesforce API calls to read Query data using Bulk API. Download SSIS PowerPack and try yourself and also explore 70+ other tasks /components not discussed in this article.

The post How to query salesforce data using Bulk API in SSIS appeared first on ZappySys Blog.

In this post, we will see how to create SSIS Package Programmatically to use ZappySys SSIS PowerPack. This blog post assumes you have basic knowledge of Programming language like C# and you have installed Visual Studio Express or a similar product (VS Code not supported).

Add SSIS Dll References

The very first thing is to create a C# Project (i.e. Console or WinForm). Add SSIS dll reference.

You will need the following references in your project. The below example is for SSIS 2017 (Version 14.x.x)

• For SSIS 2012 change to Version=11.0.0.0
• For SSIS 2014 change to Version=12.0.0.0
• For SSIS 2016 change to Version=13.0.0.0
• For SSIS 2017 change to Version=14.0.0.0
• For SSIS 2019 change to Version=15.0.0.0
• And may be future versions same way ….

<Reference Include="Microsoft.SqlServer.Dts.Design"/>
<Reference Include="Microsoft.SqlServer.DTSPipelineWrap" />
<Reference Include="Microsoft.SQLServer.DTSRuntimeWrap" />
<Reference Include="Microsoft.SQLServer.ManagedDTS" />
<Reference Include="Microsoft.SqlServer.PipelineHost" />

Step-By-Step Example (Create SSIS Package Programmatically – C#)

Now let’s see each piece to create SSIS Package in code and later we will show the full code.

Import Namespaces

First step is at the top you need to import some namespaces.

using Microsoft.SqlServer.Dts.Runtime;
using Microsoft.SqlServer.Dts.Pipeline;
//using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
//using Microsoft.SqlServer.Dts.Runtime.Wrapper;

Create SSIS Package

Here how you can create SSIS Package instance in code

var p = new Package();

Add SSIS Connection

Now lets add connection manager to our package instance. To add connection you will need two pieces. Creation Name (i.e. ZS-SFTP) and ConnectionString. You can find creation name from exsing Dtsx file by opening in text editor.

var conn1=p.Connections.Add("ZS-SFTP");
conn1.Name = "MySftpConnection";
conn1.ConnectionString= "Protocol=Sftp;ServerName=127.0.0.1;Port=0;LogonType=Normal;UserName=zstest;Password=zstest123";

How to find CreationName of SSIS Connection manager

To find the creation name of your Connection Type open an existing DTSX file where you had similar connection type in the package. Locate node like below and check CrewationName. It can be a friendly name or can be fully qualified assembly name. Both ways is fine but a friendly name is easy to remember.

<DTS:ConnectionManagers>
  <DTS:ConnectionManager
    DTS:refId="Package.ConnectionManagers[RedshiftConn1]"
    DTS:CreationName="ZS-POSTGRESQL"
    .........
    .........

How to find ConnectionString

Now another thing you need to know is the exact connectionstring. You can go to SSIS Designer and right-click on Connection Icon and check Properties Grid. It will show you ConnectionString or you can use the above technique > Open DTSX file in notepad and find connectinstring there.

ConnectionString doesnt include any passwords / secrets. So manually append those properties in your connectionstring when you set in code.

Add SSIS Task

Now let’s add task in our SSIS Package instance. To create SSIS Task programmatically in code, you need to know the exact type name (i.e. creation name). As we discussed earlier check your dtsx file to know the creation name for the task you like to create. In the below code we used a friendly name which is not supported in SSIS 2012 so for that you must use full name. For SSIS 2014 or higher you can use friendly names.

You can also find it in file like this (example of SSIS 2017 and 2019 folders)
C:\Program Files (x86)\Microsoft SQL Server\140\DTS\UpgradeMappings\ZappySys.PowerPack140.Extensions.xml
C:\Program Files (x86)\Microsoft SQL Server\150\DTS\UpgradeMappings\ZappySys.PowerPack150.Extensions.xml

//var SSIS_VERSION = "110"; //110=SQL2012, 120=SQL2014, 130=SQL2016, 140=SQL2017, 150=SQL2019
//For SSIS 2012
var SFTP_TASK_2012 = "ZappySys.PowerPack.Task.SecureFtpTask.SecureFtpTask, ZappySys.PowerPack110, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502";
//For SSIS 2014 or higher
//Goto folder like this and find zappysys extension file 
//(e.g. 140) => C:\Program Files (x86)\Microsoft SQL Server\140\DTS\UpgradeMappings\ZappySys.PowerPack140.Extensions.xml
//Find Task creation name there
var SFTP_TASK = "ZappySys.SecureFtpTask";

var tsk1 = p.Executables.Add(SFTP_TASK) as TaskHost;
tsk1.Properties["SourceStorageConnection"].SetValue(tsk1, "MySftpConnection");
tsk1.Properties["StorageAction"].SetValue(tsk1, 10); //DownloadFileFromFtp [10] find numeric value for this enum from help file> properties https://zappysys.com/onlinehelp/ssis-powerpack/scr/ssis-sftp-ftp-ftps-task.htm
tsk1.Properties["SourcePathValue"].SetValue(tsk1, "subfolder/localsubfile1.csv");
tsk1.Properties["TargetPathValue"].SetValue(tsk1, @"c:\temp\");
tsk1.Properties["FileOverwriteAction"].SetValue(tsk1, 2); //OverwriteAll [2]

Execute SSIS Package

Now let’s execute the package using the below line

var status =p.Execute();

Check Execution Errors

At the end you need to check execution errors if any.

if (status != DTSExecResult.Success)
{
	string errs = p.Errors.Cast<DtsError>().Aggregate("",
           (current, err) => current + string.Format("Source={0}, Err={1}", err.Source, err.Description));
	throw new Exception("Package status was >> " + status + " >> " + errs);
}

Full Code

Here is the full code

using System;
using System.Linq;
using System.Runtime.InteropServices;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime;
/*
For SSIS 2012 - Compile with .net 4.0
For SSIS 2014-2017 - Compile with .net 4.5 
For SSIS 2019 - Compile with .net 4.6.2
 */

//current one is SSIS 2017 (140 dlls)
/*
 * References needed for SSIS 2017 , for 2019 change 14.0.0.0 to 15.0.0.0
 * 
    <Reference Include="Microsoft.SqlServer.DtsMsg, Version=14.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91, processorArchitecture=MSIL" />
    <Reference Include="Microsoft.SqlServer.Dts.Design, Version=14.0.0.0, PublicKeyToken=89845dcd8080cc91" />
    <Reference Include="Microsoft.SqlServer.DTSPipelineWrap, Version=14.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91, processorArchitecture=MSIL" />
    <Reference Include="Microsoft.SQLServer.DTSRuntimeWrap, Version=14.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91" />
    <Reference Include="Microsoft.SQLServer.ManagedDTS, Version=14.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91, processorArchitecture=MSIL" />
    <Reference Include="Microsoft.SqlServer.PipelineHost, Version=14.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91, processorArchitecture=MSIL" /> 
 */
namespace SSISCodeDemo
{
    class Program
    {
        static void Main(string[] args)
        {
            
            //Create_SSIS_Controlflow_Task_Demo();
            Create_SSIS_Dataflow_Source_Component_Demo();

            //Create_SSIS_Connection_Demo();
            //Create_SSIS_Variable_Demo();
            //Create_SSIS_Connect_Tasks_Demo();

        }

        private static void Create_SSIS_Controlflow_Task_Demo()
        {
            var p = new Microsoft.SqlServer.Dts.Runtime.Package();

            var conn1 = p.Connections.Add("ZS-SFTP");
            conn1.Name = "MySftpConnection";
            conn1.ConnectionString = "Protocol=Sftp;ServerName=127.0.0.1;Port=0;LogonType=Normal;UserName=zstest;Password=zstest123";

            //var SSIS_VERSION = "110"; //110=SQL2012, 120=SQL2014, 130=SQL2016, 140=SQL2017, 150=SQL2019
            //For SSIS 2012
            var SFTP_TASK_2012 = "ZappySys.PowerPack.Task.SecureFtpTask.SecureFtpTask, ZappySys.PowerPack110, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502";
            //For SSIS 2014 or higher
            //Goto folder like this and find zappysys extension file (e.g. 140) => C:\Program Files (x86)\Microsoft SQL Server\140\DTS\UpgradeMappings\ZappySys.PowerPack140.Extensions.xml
            //Find Task creation name there
            var SFTP_TASK = "ZappySys.SecureFtpTask";

            var tsk1 = p.Executables.Add(SFTP_TASK) as TaskHost;
            tsk1.Name = "SFTP_Task1";
            tsk1.Properties["SourceStorageConnection"].SetValue(tsk1, "MySftpConnection");
            tsk1.Properties["StorageAction"].SetValue(tsk1, 10); //DownloadFileFromFtp [10] find numeric value for this enum from help file> properties https://zappysys.com/onlinehelp/ssis-powerpack/scr/ssis-sftp-ftp-ftps-task.htm
            tsk1.Properties["SourcePathValue"].SetValue(tsk1, "subfolder/localsubfile1.csv");
            tsk1.Properties["TargetPathValue"].SetValue(tsk1, @"c:\temp\");
            tsk1.Properties["FileOverwriteAction"].SetValue(tsk1, 2); //OverwriteAll [2]


            /*
             Go to package file (*.dtsx) > Open in Notepad > and findout which properties set. Do same way (ignore some default proprties)
             Some enum properties needs numeric values > Check Task / Component help file to findout number for that

      <DTS:ObjectData>
        <SecureFtpTask
          StorageAction="DeleteFtpFile"
          FileOverwriteAction="OverwriteAll"
          TreatRegXForFullPath="False"
          ContinueOnError="False"
...................
...................
...................            
            PrefixTimestamp="False" />
      </DTS:ObjectData>
    </DTS:Executable>             
             */

            //Application.

            /* use below code to save to disk file 
            var app = new Microsoft.SqlServer.Dts.Runtime.Application();
            var filepath = @"c:\temp\demo.dtsx";
            app.SaveToXml(filepath, p, null);

            --load from disk to memory
            p = app.LoadPackage(filepath, null);

            */
            var status = p.Execute();

            if (status != DTSExecResult.Success)
            {
                var errs = "";
                foreach (var e in p.Errors)
                {
                    errs = errs + string.Format("Source={0}, Err={1}", e.Source, e.Description) + "\r\n";
                }

                throw new Exception("Package status was >> " + status + " >> " + errs);
            }

            Console.WriteLine("Execution Completed... Press any key to exit!!!");
            Console.ReadKey();
        }
        private static void Create_SSIS_Dataflow_Source_Component_Demo()
        {
            var p = new Microsoft.SqlServer.Dts.Runtime.Package();


            /*============ STEP-1: Create Dataflow Task =========== */

            var tsk1 = p.Executables.Add("STOCK:PipelineTask") as TaskHost;
            var dataFlowTask = tsk1.InnerObject as MainPipe;

            /*============ STEP-2: Create Connection =========== */
            var conn1 = p.Connections.Add("ZS-HTTP");
            conn1.Name = "MyConnection";
            conn1.ConnectionString = "Url=https://httpbin.org/get;CredentialType=Basic;UserName=test;Password=test123";


            /*============ STEP-3: Create Source =========== */
            var sourceComponent = dataFlowTask.ComponentMetaDataCollection.New();

            try
            {
                sourceComponent.ComponentClassID = "ZappySys.JsonSource";
                var sourceInstance = sourceComponent.Instantiate();
                sourceInstance.ProvideComponentProperties();

                sourceComponent.Name = "MySource";
                /**** Special Case for JSON Source - if user wants connection to use then we create Runtime connection. For all other components this is always created when ProvideComponentProperties() is called*/
                if (sourceComponent.RuntimeConnectionCollection.Count <= 0)
                {
                    var dtsRuntimeConnection = sourceComponent.RuntimeConnectionCollection.New();
                    dtsRuntimeConnection.Name = "Connection"; 
                    //dtsRuntimeConnection.Description = "This can be ZS-HTTP, ZS-OAUTH or ZS-SALESFORCE";
                }
                sourceComponent.RuntimeConnectionCollection[0].ConnectionManager = DtsConvert.GetExtendedInterface(conn1);
                sourceComponent.RuntimeConnectionCollection[0].ConnectionManagerID = conn1.ID;

                sourceInstance.SetComponentProperty("UseConnection", true);//JsonSource specific -- for all other components connection always used
                sourceInstance.SetComponentProperty("DirectPath", "https://httpbin.org/basic-auth/test/test123");
                //sourceInstance.SetComponentProperty("Filter", "$.value[*]");

                //make connection active before next step
                sourceInstance.AcquireConnections(null);

                //after connection + props set ... call ReInitialize meta... it creates columns / output / input etc
                sourceInstance.ReinitializeMetaData(); 
            }
            catch (COMException ex)
            {
                throw new Exception("Error Happened ==> " + sourceComponent.GetErrorDescription(ex.ErrorCode));
            }




            /*============ STEP-4: Create Target (without Reinitialize) =========== */

            var targetComponent = dataFlowTask.ComponentMetaDataCollection.New();
            CManagedComponentWrapper targetInstance=null;
            try
            {
                targetComponent.ComponentClassID = "ZappySys.TrashDestination";
                targetInstance = targetComponent.Instantiate();
                targetInstance.ProvideComponentProperties();

                targetComponent.Name = "MyDestination";

                targetInstance.SetComponentProperty("SaveDataToFile", true);
                targetInstance.SetComponentProperty("FilePath", @"c:\temp\dump.txt");
                targetInstance.SetComponentProperty("OverwriteFileIfExists", true);

                //--- Trash destination doesnt have connection but incase destination uses connection... set below way
                //if you have connection then set this way
                //var conn2 = p.Connections.Add("....."); //Creation name i.e. ZS-HTTP
                //conn2.Name = "MyConnection2";
                //conn3.ConnectionString = "Url=https://httpbin.org/get;CredentialType=Basic;UserName=test;Password=test123";
                //if (targetComponent.RuntimeConnectionCollection.Count <= 0)
                //{
                //    var dtsRuntimeConnection = targetComponent.RuntimeConnectionCollection.New();
                //    dtsRuntimeConnection.Name = "Connection"; 
                //}
                //sourceComponent.RuntimeConnectionCollection[0].ConnectionManager = DtsConvert.GetExtendedInterface(conn2);
                //sourceComponent.RuntimeConnectionCollection[0].ConnectionManagerID = conn2.ID;
                //make connection active before next step
                //targetInstance.AcquireConnections(null);
            }
            catch (COMException ex)
            {
                throw new Exception("Error Happened ==> " + targetComponent.GetErrorDescription(ex.ErrorCode));
            }

            /*============ STEP-5: Connect Source to Target =========== */

            //before we call reintializemetadata for dest .. lets attach src and dest so .. dest can use virtual inputs if needed
            var output = sourceComponent.OutputCollection[0]; //to make it simple lets pick first output but possible component can have more outputs
            var input = targetComponent.InputCollection[0];

            dataFlowTask.PathCollection.New().AttachPathAndPropagateNotifications(output, input);

            /*============ STEP-6: Configure Target =========== */
            try
            {
                //after connection + props set and source to target attached... call ReInitialize meta... it creates columns / output / input etc
                targetInstance.ReinitializeMetaData(); 
            }
            catch (COMException ex)
            {
                throw new Exception("Error Happened ==> " + targetComponent.GetErrorDescription(ex.ErrorCode));
            }


            /*============ STEP-6: Save Package =========== */

            //* use below code to save to disk file 
            var app = new Microsoft.SqlServer.Dts.Runtime.Application();
            var filepath = @"c:\temp\demo.dtsx";
            app.SaveToXml(filepath, p, null);

            /*============ STEP-7: Load Package =========== */
            //--load from disk to memory
            p = app.LoadPackage(filepath, null);

            //*/

            /*============ STEP-8: Execute Package =========== */
            var status = p.Execute();

            /*============ STEP-9: Print execution status / log =========== */
            if (status != DTSExecResult.Success)
            {
                var errs = "";
                foreach (var e in p.Errors)
                {
                    errs = errs + string.Format("Source={0}, Err={1}", e.Source, e.Description) + "\r\n";
                }

                throw new Exception("Package status was >> " + status + " >> " + errs);
            }

            Console.WriteLine("Execution Completed... Press any key to exit!!!");
            Console.ReadKey();
        }

        private static void Create_SSIS_Variable_Demo()
        {
            var p = new Microsoft.SqlServer.Dts.Runtime.Package();
            
            p.Variables.Add("MyVariableName1", false, "User", "Some value");
        }

        private static void Create_SSIS_Connection_Demo()
        {
            var p = new Microsoft.SqlServer.Dts.Runtime.Package();

            var conn1 = p.Connections.Add("ZS-SFTP");
            conn1.Name = "MySftpConnection";
            conn1.ConnectionString = "Protocol=Sftp;ServerName=127.0.0.1;Port=0;LogonType=Normal;UserName=zstest;Password=zstest123";

        }

        private static void Create_SSIS_Connect_Tasks_Demo()
        {
            var p = new Microsoft.SqlServer.Dts.Runtime.Package();

            var SFTP_TASK = "ZappySys.SecureFtpTask";

            var tsk1 = p.Executables.Add(SFTP_TASK) as TaskHost;
            tsk1.Name = "SFTP_Task1";
            /*set some props - see Create_SSIS_Controlflow_Task_Demo */

            var tsk2 = p.Executables.Add(SFTP_TASK) as TaskHost;
            tsk2.Name = "SFTP_Task2";
            /*set some props - see Create_SSIS_Controlflow_Task_Demo */
            
            //add link between task1-->task2
            var pc = p.PrecedenceConstraints.Add(tsk1, tsk2);
            pc.EvalOp = DTSPrecedenceEvalOp.Constraint;
            pc.Value = DTSExecResult.Success;
            pc.LogicalAnd = true; //only useful if multiple inputs coming and you want all previous steps to be Green vs Any One Green .. if its any use LogicalAnd=False

            //If you need to connect with expression use below
            //pc.EvalOp = DTSPrecedenceEvalOp.ExpressionAndConstraint;
            //pc.Expression = "@SomeVariable == true";
        }

       

     
    }
}

The post How to create SSIS Package Programmatically (Add / Execute Tasks) appeared first on ZappySys Blog.

OneDrive allows to store files on the cloud and and we have seen growing requirements in downloading/Uploading files from there. This post will provide step by step details of how to Upload/Download files from OneDrive.

Register Microsoft App for OAuth Authentication

In this section, you will learn how to register a custom app in Microsoft Azure portal that will allow access to the OneDrive. So, let’s get started –

1. Log into Microsoft Azure portal to register a custom app.
2. Register a new application by clicking New Registration link.
   

   New App Registration in Azure portal

3. Provide the name of the custom app and who can access the app in the organization.
   

   Register an OneDrive App

4. Go to the App overview and add a Redirect URL.
   

   Add a Redirect URL

5. Click on “Add a Platform” under Platform Configuration section and then select “Web” under Web applications section to enter a Redirect URL.
   

   https://zappysys.com/oauth

   

   Redirect URL

6. Create a Client Secret key which will be used to Authenticate the custom Azure app.
   

   Add a Client Secret

   

   Secret Key Expiration Period

   

   Specify Secret Key

   Note – Take a note of Client Secret, it will be required while configuring OAuth connection in the SSIS later.
7. Add API Permissions for OneDrive API.
   

   OneDrive API Permissions

8. Select the following permissions from the Delegated Permissions section.
   

   User.ReadBasic.All
   Files.Read
   offline_access

   

   Select Delegated Permissions

9. Take note of Client ID, it will be required while configuring OAuth connection in the SSIS later.
   

   Client ID

Get OneDrive File ID

In this section, you will learn how to get the File ID of the CSV file through OneDrive API. So, let’s get started –

1. Go to the Microsoft Graph Explorer.
2. Sign in to Graph Explorer.
   

   SSIS GraphExplorer SignIn

3. Run below OneDrive API to get a list of the files from the OneDrive.
   

   https://graph.microsoft.com/v1.0/me/drive/root/children

   

   Microsoft Graph Explorer

   NOTE: Take a note of File ID of the CSV file. For this example, we will download invoices.csv file.

Create OAuth Connection in SSIS

In this section, you will learn how to create an OAuth connection in SSIS to an Azure custom app. So, let’s get started –

1. Create a new ZS-OAuth connection in SSIS.
   

   ZappySys OAuth Connection in SSIS

2. Configure OAuth connection with following settings.

   Client ID	Provide Client ID from App Registration steps
   Client Secret	Provide Client Secret from App Registration steps
   Authorization URL	https://login.microsoftonline.com/organizations/oauth2/v2.0/authorize
   Access Token URL	https://login.microsoftonline.com/organizations/oauth2/v2.0/token
   Scope/Permissions	User.ReadBasic.All Files.Read.All offline_access

   

   OAuth Connection Configuration

3. Go to the Advanced tab and provide Return URL.
   

   https://zappysys.com/oauth

   

   Add Redirect URL

4. Click on Generate Token will give a login screen which will allow to generate access and refresh token for the app.
5. Approve the requested permissions bu Accepting it.
   

   Approve Permissions Requested

6. Token has been generate successfully.
   

   Refresh Token Successful

7. Let’s test the connection to make sure it is working fine.
   

   Connection Successful

Download CSV File Through OneDrive REST API in SSIS

In this section, you will learn how to download a CSV file through OneDrive API in SSIS and save it into a CSV file.

1. Here is the OneDrive API which allows to read content of the file.
   

   https://graph.microsoft.com/v1.0/me/drive/Items/<File ID>/content

2. Replace <File ID> in above step#1 with the file ID that we have got in the Get OneDrive File ID section. Final URL will look something like below.
   

   https://graph.microsoft.com/v1.0/me/drive/Items/01W7L3VCUBS27ILUQKENAK4LQGGTJ2R3TT/content

3. Let’s create a SSIS package with Data Flow Task from SSIS Toolbox.
4. Replace <File ID> in above step#1 with the file ID that we have got in the Get OneDrive File ID section. Final URL will look something like below.
   

   https://graph.microsoft.com/v1.0/me/drive/Items/01W7L3VCWMCGPYZLM2IRAYWROSH4XMQT7A/content

5. Configure REST API task as follows.
   

   Binary File Configuration in ZappySys REST API Task

6. Configure the responses setting as below.
   

   Binary File Configuration in ZappySys REST API Task

   NOTE: Ensure that the Treat response as the binary checkbox is checked otherwise the file will be downloaded but corrupted as zip/mp3 etc. file format is binary.
7. Check if the request is made correctly by clicking Test Request/Response.
   

   Example of Testing the Request

8. Let’s have a look at the folder where the file has been downloaded. Here is an example –
   

   Downloaded File Example

Download a Text File Through OneDrive API in SSIS

Here is another technique that can be used to download a CSV/text file through OneDrive API without knowing it’s file ID but file name. Please check this link for more details.

Download Excel Sheets Through OneDrive API in SSIS

JSON Source also supports reading API response which returns 2D arrays. Such as Office Excel Spreadsheet API to read excel data. Check this blog post to learn a similar scenario to parse 2D arrays. There are few other options for parsing 2D arrays which are documented here.

https://graph.microsoft.com/v1.0/me/drive/items/01CYZLFJDYxxxxxxx/workbook/worksheets('test')/range(address='A1:B200')

Upload a File Through OneDrive API in SSIS

In this section, you will learn how to upload a file through OneDrive API in SSIS. Uploading a file has few additional steps and has been documented separately. Please check this link for more details.

Delete a File Through OneDrive API in SSIS

In this section, you will learn how to delete a file from OneDrive through OneDrive APIs in SSIS.

1. Add a REST API task and configure as follows. Make sure to use the File ID of the file which needs to be deleted from OneDrive. Example URL –
   

   https://graph.microsoft.com/v1.0/me/drive/Items/01W7L3VCSEADQ457W54RBI3I47YGFJOD2I

   

   Delete OneDrive File

2. Execute the task and you will have file deleted from OneDrive.

Conclusion

In this article, we have learned how to Upload/Download/Delete data/file from OneDrive API in SSIS. We used SSIS REST API Task to extract data through OneDrive API in SSIS. Download SSIS PowerPack to try many other automation scenarios not discussed in this article.

The post How to call OneDrive API in SSIS (Upload, Download) appeared first on ZappySys Blog.