Using REST API with SharePoint in SSIS is a common requirement to Administer SharePoint. SharePoint 365 is a nice Microsoft application in the Cloud used to share documents and collaborate with the company. You can have schedules, projects, documents and more shared using a Web platform.

In addition, it is a very useful tool that your company may need. It is also possible to automate, administer task using Rest API.
This article, we will learn how to create REST API queries to SharePoint 365 using the Microsoft Graph API using SSIS. To learn about other Office 365 REST API in SSIS check this article (Outlook, Onedrive, mail, Excel API).

Also, we will show our ZappySys SSIS PowerPack that includes very powerful tools to export SharePoint data from REST API to any other source.

	SSIS REST API Web Service Task
	SSIS JSON Source (File, REST API, OData Connector)
	SSIS JSON File Destination (Create JSON File)

Requirements to REST API with SharePoint

1. First, you will need SSDT installed.
2. Second, SSIS PowerPack installed.
3. An Office 365 Account with SharePoint 365

Getting started

Microsoft included the Microsoft Graph to easily create API queries to different Microsoft components like OneDrive, Excel, Insights, Microsoft Team, Planner and more.

The Microsoft Graph is a gateway to access to 365 Data in an integrated platform. Let’s start with the Microsoft Graph Explorer and do some REST API queries to understand how to use REST API in SharePoint.

Microsoft Graph Explorer to SharePoint introduction

The best way to learn REST API with SharePoint 365 is to use the Graph Explorer. You can run API queries using the Graph Explorer and get your data. Here we will show how.

1. First, we will go to the Graph Explorer and sign-in with your Office 365 Account.
2. Secondly, in Run Query, you can specify your query, which will show SharePoint information.
   

   Graph Explorer

3. Let start with this link:
   

   https://graph.microsoft.com/v1.0/sites/root/lists

4. This shows properties about site resources. The response of the query is in JSON format:
   

   Graph API Microsoft JSON results

5. Also, it is possible to enable examples for SharePoint Sites, SharePoint Lists, and other Microsoft technologies:
   

   Graph API Applications

Create an OAuth connection to REST API with SharePoint in SSIS

1. 1. First, to get the Data from the Microsoft Graph, we will sign in to create an application in the Azure Portal app registration.
   2. Secondly, create a Microsoft Application. It will provide you an Application ID and a Password. For detailed steps about how to create a Microsoft Application, check our article related.
      

      Microsoft Graph API application id password

   3. Once that the application is created, go to SSDT or Visual Studio and create a new connection.
      

      SSIS OAuth connection

   4. Select the ZS-OAuth connection. Once selected, enter the Client ID (the Microsoft Graph Application ID), the client secret (application password).
   5. Add the Authorization URL is the following:
      

      https://login.microsoftonline.com/common/oauth2/v2.0/authorize

   6. Also, include the Access token URL is the following:
      

      https://login.microsoftonline.com/common/oauth2/v2.0/token

   7. The following permissions were applied in the example: User.Read
      

      offline_access
      Sites.ReadWrite.all
      Sites.Manage.All
      Sites.FullControl.All

      NOTE: It is advised to include offline_access scope in order to get a Refresh Token and to be free of any token restrictions.
   8. Once that the properties are entered, press the Generate token button:
      

      SSIS OAuth with REST API with SharePoint in SSIS

How to call SharePoint API in SSIS

1. Once that your OAuth Connection is created, we can use it in a REST API task. This task is included with the ZappySys SSIS PowerPack:
   

   SSIS REST API task

2. Secondly, in the REST API Task, select the option URL from connection and select the OAuth connection created before. We will first use the following URL to see the list of site properties:
   

   https://graph.microsoft.com/v1.0/sites/root/lists

3. Also, press Test request/Response button to get the results of the query:
   

   SSIS SharePoint rest API call

4. The response will be in JSON format:
   

   REST API SSIS Sharepoint 365 response

5. Also, it is possible to store the results in a file or a variable:
   

   SSIS save API SharePoint response to a file REST API with SharePoint

6. In Addition, you can run the following query to enumerate all the sites:
   

   https://graph.microsoft.com/beta/sites?select=siteCollection,webUrl&filter=siteCollection/root%20ne%20null

7. Also, to show the information on a specific site, the following query will be used:
   

   https://graph.microsoft.com/v1.0/sites/site-id

8. For example, if you site id is zappysys.sharepoint.com the URL would be the following:
   

   https://graph.microsoft.com/v1.0/sites/zappysys.sharepoint.com

9. To enumerate sites, the following example can be useful:
   

   https://graph.microsoft.com/beta/sites?select=siteCollection,webUrl&filter=siteCollection/root%20ne%20null

10. Also, to enumerate lists, you can use the following URL:
    

    https://graph.microsoft.com/beta/sites/zappysys.sharepoint.com/lists

How to read SharePoint lists in SSIS using REST API

We will learn how to show SharePoint lists using rest API.

In addition, you can get the list ids using this query:

https://graph.microsoft.com/beta/sites/zappysys.sharepoint.com/list

SharePoint list API

How to read data from a Sharepoint List

1. To get the list of ithems form a list use the following query:
   

   https://graph.microsoft.com/beta/sites/zappysys.sharepoint.com/lists/mylist/items/

2. Finally, to get metadata from a list, you can use the following query. To learn more how to import Sharepoint List into SQL Server use JSON / REST API Source.

https://graph.microsoft.com/beta/sites/zappysys.sharepoint.com/lists/fcf5fa2b-ceeb-4c9b-80e5-7bedd455fc9b

SSIS Sharepoint metadata list

How to create a SharePoint list item in SSIS using REST API (POST Method)

It is also possible to POST data using REST API. In the next example, we will create a new list in SharePoint 365 using REST API.

1. First, in SSDT, use the REST API Task used previously and specify the following URL:
   

   https://graph.microsoft.com/beta/sites/zappysys.sharepoint.com/lists/mynewlist/items/

   Where zappys.sharepoint.com is the site id and mynewlist is a new list created. Also, in the HTTP Request Method, use the POST method.
2. In addition, go to the Body Request, we will create a widget. We will use the following JSON data and make sure to select JSON (application/json) in the body:
   

   {
   "fields": {
   "Title": "Widget"
   }
   }

   

   SharePoint Post REST API

3. Finally, if everything is successful, you will be able to see the new item created:
   

   SSIS SharePoint list created with a POST

How to export SharePoint API response results to a file

We have a nice task named JSON Source used to get data from REST API or a JSON file and export to any source the results like SQL Server, Excel, a flat file, etc.
In this example, we will export data from the JSON Source to a JSON file using the JSON Destination Task.

1. First of all, we will use the Data Flow. Drag and drop the data flow to the design pane and double click to go to the Data Flow tab:
   

   SSIS Data Flow

2. Secondly, in the Data Flow tab, drag and drop the JSON Source, the JSON Destination and join them:
   

   SSIS JSON source to JSON destination

3. In addition, in the JSON Source, we will check the option Use Credentials and select the OAuth connection created and used before.
4. Enter Path or Web URL, we will use the following URL:
   

   https://graph.microsoft.com/beta/sites/zappysys.sharepoint.com/lists

5. Also, We will use the following filter:
   

   $.value[*].list

   

   SSIS JSON Source REST API with SharePoint in SSIS

6. In addition, go to JSON Destination task, select mappings and right click on add elements:
   

   SSIS JSON Destination

7. Also, select the attributes:
   

   SSIS add attributes

8. Finally, run the package to export the data.

Conclusion about REST API with SharePoint in SSIS

To conclude, we can say that SharePoint 365 is a very popular application to share documents and information. With REST API it is possible to get data properties and administer SharePoint. We used the REST API, JSON Source to get and send information. If you want to try yourself, you can download our tools here.

References

• Get access on behalf of a user
• Working with SharePoint sites in Microsoft Graph
• About SSIS PowerPack

The post Calling SharePoint 365 REST API in SSIS appeared first on ZappySys Blog.

SQL Server 2012 and all previous versions lacking native JSON support, same theory is true for their ETL Platform SSIS. If you are one of them who wants to export JSON from SQL Server or any other RDBMS like MySQL, Oracle then you can easily output complex JSON using any of the following components.

Five ways of creating JSON inside SSIS

ZappySys toolset provides multiple ways creating JSON inside SSIS. You can choose method suitable for your scenario. Each method has pros and cons so choose it as per your need.

JSON File Destination

SSIS JSON Destination can be used to generate simple or complex JSON files out of relational data source such as SQL Server, Oracle, MySQL. In few clicks you can create desired JSON Shape from single or multiple tables (Parent/Child Nested Documents).

PROS: Multiple data flow components can be attached. Which means unlike Export JSON Task you are not limited to OLEDB or ODBC Sources only.

CONS: When you remove upstream component it may break Dataset mappings. Also server side Lookup Option is not available. File Splitting Options not available.

Here are some articles for use case of this component.

JSON File Destination – Generate JSON in SSIS from multiple inputs

JSON Generator Transform

SSIS JSON Generator Transform can be used to generate single or multiple JSON documents from any type of data sources (e.g. SQL Server, MySQL, Flat File, Excel) inside data flow task. It takes multiple inputs and outputs JSON String Output Column which can be used to feed JSON to other downstream system.

PROS: Multiple data flow components can be attached. Which means unlike Export JSON Task you are not limited to OLEDB or ODBC Sources only.

CONS: When you remove upstream component it may break Dataset mappings. Also server side Lookup Option is not available. File Splitting Options not available. Output is JSON String Column.

Here are some articles for use case of this transform.

Preview JSON – Generate for BigQuery Table Insert

Export JSON Task

SSIS Export JSON File Task can be used to generate simple or complex JSON files out of relational data source such as SQL Server, Oracle, MySQL. It also support exporting directly to cloud (e.g. Azure Blob Storage, Amazon AWS S3, Secure FTP). In few clicks you can create desired JSON Shape from single or multiple tables (Parent/Child Nested Documents)

PROS: Simple to use with flexible options to export to file, variable, AWS S3, Azure Blob, FTP / SFTP location. Allows to split records. File Splitting Options available (Split By Size  / Row Count).

CONS: You can only use limited data sources as Dataset input (e.g. OLEDB / ODBC or SQL Server).

Here are some articles for use case of this task.

Template Transform

Template Transform can be used to create dynamic strings inside data flow. You can use upstream columns as placeholders or use SSIS variables anywhere in the template to create desired text (e.g. XML / JSON document).

PROS: Very Simple to use.

CONS: Cannot use multiple inputs. It may not be used to create Complex JSON and options like do not output NULL attributes is not possible.

Here are some articles for use case of this transform.

Create JSON for API Request (POST) using SSIS Template Transform

Logging Task (Control Flow Approach)

 Just like Template Transform you can use ZS Logging Task to create JSON string from Template and Variable Placeholders.

Here is how to use Logging Task to save JSON into a variable.

1. Select Variable which you like to Set on Logging Task
2. Enter your Template JSON String which you like to produce (see example below). Notice the use of yyyy-MM-dd format specifier and also JSONENCODE function.
   

   {
    ActionId: {{User::vActionId}},
    CreatedOn: "{{User::vOrderDate,yyyy-MM-ddTHH:mm:ss}}",
    Notes: "{{User::vNotes,JSONENCODE}}"
   }

3. Select Message Type = None
4. Uncheck Append, and Prefix options like below
   

   Creating JSON / XML Document using SSIS Logging Task

Here are some articles for use case of this task.

Create JSON using SSIS Export JSON Task

Now let’s look at Export JSON Task form example to create JSON. Steps listed in below sections are very similar to JSON File Destination or JSON Generator Transform.

ZappySys offers innovative Layout Editor for JSON / XML creation from multiple sources. You can use Layout Editor to produce desired JSON Layout (Layout is stored in XML format behind the scene) and then in later step you will see how to make it dynamic so values are changed at runtime.

You can read layout as direct value, read from variable or read from file from disk.

Select source connection for data

First lets create new OLEDB or ADO.net connection from where we will read data. In our case we reading from Northwind Database.

SSIS JSON Export Task – Select Source Database Connection

Create Datasets

Now add datasets which will be used to produce JSON File. One dataset will be always Root Dataset (Example: If your Root information is about customer then select customer table or query as your Main dataset. All other sections will be child sections)

Add SQL Dataset – Export JSON from SQL Query Option

Add Output Columns for Main JSON Document

Now its time to add columns for our output. If * is added by default then remove. * means output all columns. When you add columns first make sure you select correct parent node under which you want to add columns. In our case “Mappings” is parent node.

Add Output Columns for JSON Document – Column Selection Screen

Add JSON Array / JOIN Conditions for related document

Now let’s look at how to add JSON Array. For example Customer may have one or more Orders so we need to add node called Orders which may Produce JSON fragment like below

Orders : [ {...} , {...} ] 

Once customer related columns added, its time to add Orders section.

1. This is Array of documents so first highlight “Mappings” node
2. Click “Add Document Array” option from toolbar.
3. On Add/Edit Elements UI specify name for JSON Attribute and specify correct relationship to parent dataset. In our case Orders related to Customers using CustomerID key.

Add Child Section – nested JSON document , Define Parent-Child Relationship

Handling JSON Array with dummy root dataset (Multiple Arrays)

So in previous section we saw how to add array and define JOIN condition with parent dataset. Now what if you really have no parent dataset to join with ? For example we need to produce following sample JSON where Order array is related to Customer by common Column CustomerID as JOIN column. But Customer is not bound with any parent column. So how do you define JOIN condition? In such case you have to use below technique called dummy root dataset. (See next section on how to handle only one dataset using Single Dataset Array Mode)

Sample JSON (Dummy root pattern)

{
    "Customers": [
      {
        "CustomerID": "ALFKI",
        "CompanyName": "Alfreds Futterkiste",
        "Orders": [
          {
            "OrderID": 1000,
            "OrderDate": "2015-12-01"
          },
          {
            "OrderID": 1001,
            "OrderDate": "2015-12-01"
          }
        ]
      },

      {
        "CustomerID": "ANATR",
        "CompanyName": "Ana Trujillo Emparedados y helados",
        "Orders": [
          {
            "OrderID": 1002,
            "OrderDate": "2015-12-01"
          },
          {
            "OrderID": 1003,
            "OrderDate": "2015-12-01"
          }
        ]
      }

    ]
  }
}

Since root level always needs dataset, we must provide dummy dataset (only one row) even though there is no need for Parent Dataset JOIN in Customer Array. Typically you can write query like below to product dummy row in SQL Server or other database using OLEDB Source or Use Dummy Data Source with Max Rows=1 setting to produce one fake record.

SELECT 1 as Col1

Using Dummy DataSource as Input

For generating JSON which needs dummy root. Perform the following steps

1. Drag Dummy Data Source from SSIS Toolbox. Change How many rows to 1. Click OK
2. Now drag two more OLEDB sources and Configure (e.g. Customers and Orders)
3. Now drag ZS JSON FIle Destination component or ZS JSON Generator
4. Now connect Dummy datasource then Customers and then Orders
5. Double click JSON component to configure layout. You may notice that Dummy Input is set as Root by default.
6. You can now select Mappings node and click on Add Array Documents icon. Configure like below. Notice we have not selected any JOIN criteria because its joined with Dummy Root.
   

   Add array to join with dummy dataset

7. Now select newly added node and Click Add column icon to add CustomerID, CompanyName column under Customers array
8. Now select Customers array node and click Array Documents icon to Add Orders array.
9. Once Orders Array ia added you can Select Orders node and Insert new element to add OrderID and OrderDate columns
10. Finally it will look like below
    

    JSON File Destination – Generate JSON in SSIS from multiple inputs

Single dataset used as array

Now lets look at one more pattern just like described in the previous section but now we have just one nested level (Only customers and not using Orders). If you have such case then choose Single dataset pattern option in the output mode. On Export Task you may see as check box rather than dropdown. To produced desired JSON (see blow) perform these steps.

1. Select Output mode as Single Dataset Array
2. In your Layout builder click on Add Document Array Icon and name it “Customers”.
3. After that highlight newly added node > Right-Click > Add Elements under that node
4. That’s it you will now see preview like below.

{
    "Customers": [
      {
        "CustomerID": "ALFKI",
        "CompanyName": "Alfreds Futterkiste"
      },

      {
        "CustomerID": "ANATR",
        "CompanyName": "Ana Trujillo Emparedados y helados"
      }
    ]
  }
}

How to export JSON in 2D Array format

If you have to export JSON as 2D array format then you can change following option.

Here are examples of output for each option.
Select say you have source data with 2 columns id and name.

Default

[{id:1,name:"AA"}, {id:2,name:"BB"}]

Multicontent

{id:1,name:"AA"}{id:2,name:"BB"}

ArrayDocs

[{id:1,name:"AA"}, {id:2,name:"BB"}]

Array2D

[[1,"AA"],[2,"BB"]]

Array2DWithHeader

[["id","name"],[1,"AA"],[2,"BB"]]

ArrayLines

[1,"AA"][2,"BB"]

ArrayLinesWithHeader

["id","name"][1,"AA"][2,"BB"]

Preview JSON output using live preview

You can enable / disable Live preview of JSON layout as below.

JSON Output preview in SSIS Export JSON File Task

Make things dynamic (Using SSIS Variable for SQL or Layout)

There will be a time when you have use dynamic values in your JSON generation. You may have to Alias something from variable or you have to supply Variable in your SQL used for Dataset. Not to worry. ZappySys Supports multiple ways to make it dynamic. You can use SSIS Variables following 3 ways to make Layout XML dynamic.

1. Use SSIS variable inside Dataset SQL (Only for ZS Export JSON Task)
2. Anywhere inside Layout XML (Click Edit Code option)
3. Load Layout From Variable (Indirect Method)

Using Variable inside Dataset SQL

Here is how you can use Variable inside dataset SQL for ZS Export JSON Task

SSIS Export JSON Task – Using Variable for Dynamic SQL

Using Variable inside Layout XML

Here is how you can insert SSIS Variable anywhere in Layout XML. In below Example we are choosing Alias for one column from some SSIS Variable.

SSIS JSON File Destination – Using SSIS Variable for Dynamic Layout

Load Layout From Variable

If you have some need to load Layout XML from File or SSIS Variable. You can use Load Layout from Variable Option.

For example in below screenshot we are creating layout first and ave to some variable. Or you can load from external file using ZS Advanced File System Task.  Connect Logging task to Export JSON Task and you ready to produce JSON.

Set SSIS Variable value (multi-line content using placeholder)

Here is how to load Layout From any SSIS Variable. You can also load Layout from XML file or Variable which contains layout File Path.

JSON Layout from SSIS Variable

Other JSON Output Options

Execute package and verify generated JSON it should look like below (Date format may be different in your case if you have set different on JSON options tab). For clarity we have formatted JSON below but in your output it may be compact version so reduce file size.

Indent JSON Output

You can check Indent option to make JSON look pretty like below.

{
  "CustomerID": "ALFKI",
  "CompanyName": "Alfreds Futterkiste",
  "ContactName": null,
  "Orders": [
    {
      "OrderID": 10643,
      "OrderDate": "1997-08-25T00:00:00"
    },
    {
      "OrderID": 10692,
      "OrderDate": "1997-10-03T00:00:00"
    },
    {
      "OrderID": 10702,
      "OrderDate": "1997-10-13T00:00:00"
    },
    {
      "OrderID": 10835,
      "OrderDate": "1998-01-15T00:00:00"
    },
    {
      "OrderID": 10952,
      "OrderDate": "1998-03-16T00:00:00"
    },
    {
      "OrderID": 11011,
      "OrderDate": "1998-04-09T00:00:00"
    }
  ]
}

Hide NULL Attribute Option

There will be a time when you want to Hide NULL attribute rather than showing MyAttr: null this is also possible by checking Hide NULL Attribute Option on Options Tab.

Export JSON to Amazon S3, Azure Blob, Secure FTP (SFTP / FTPS)

Export JSON Task support some really good options to save file to other than local file system. If you want to export JSON directly to Amazon / Azure Cloud or FTP Server without saving to local disk then you can use Azure / FTP / Amazon Connection Manager on Target Tab of Export JSON Task. This option is only available for PRO Edition Users.

Export JSON / CSV / XML Files to Azure Blob, Amazon S3, Secure FTP Storage

Conclusion

If you want to output JSON from SQL Server or any other relational DB then its not simple task in SSIS specially when you have nested JSON documents (e.g. Customer->Orders). In this post you have seen how to use SSIS JSON Export Task to generate nested JSON from multiple datasets.

The post 5 Ways to Export JSON from SQL Server using SSIS appeared first on ZappySys Blog.