In our previous post, we saw how to read Salesforce data using SSIS Salesforce Source (Which uses SOAP API under the hood). This may be slow if you have millions of rows because Salesforce Source paginates every 2000 rows, so it’s not a good option to read millions of rows to iterate on a large Salesforce dataset (Table or Query). To increase the read speed, you can use BULK API. This article explains how to read salesforce data using BULK API and increase read speed.

NEW Version (v5.5+)

In the new version (v5.5), we have added an option to query in Bulk Mode. This means if you have millions of Salesforce object rows to read, no worries; it may be very fast in the new version. Previously, it used to take a long time because it read in batches of 2000 rows in each request. With the Bulk API, the batch size is really large.

Here is how to set Salesforce Bulk API Mode in SELECT query (Read operation).

1. Install new version of SSIS PowerPack. Make sure it is v5.5 or higher.
2. Open existing Salesforce Source UI
3. Make sure you check [Use Bulk API] option like below. if you do not see this option then most likely you are using older version.
   -OR-
   Just add #bulkmode prefix in the SQL query like below, it will use Bulk API
   

   Bulk API mode for SSIS Salesforce Source (Read Large amount of data using SOQL Query)

OLD Version

If you are using an older version of SSIS Power Pack (v5.4.0 or older), you can follow the steps mentioned in the following section.

We will see how to query using Salesforce Bulk API in SSIS. It involves three simple steps like the one below.

1. Call BULK API – Submit SOQL Query and receive a JobId back (Use SSIS Salesforce API Task)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query

2. Keep checking Job Status Until Status is JobComplete (Use SSIS Salesforce API Task)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}

3. Read Query Result  (Use SSIS CSV Source)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results

Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode / large dataset)

Now, let’s look at everything in detail.

Obtain Salesforce Credentials

The first step to call Salesforce API is to obtain credentials (security token or OAuth ClientId /Secret)

Method#1 – Check this article to use UserId/ Password / SecretToken
Method#2 – Check this article to use the OAuth Connection Option

Create Salesforce Connection

Once we have Salesforce Credentials, we can create a new Salesforce Connection (You can do this from Task UI or the below way)

1. Open Visual Studio ( Assuming you already have SSIS installed along with ZappySys PowerPack )
2. Create a New SSIS Package Project.
3. Right-click on the Connection Manager panel and click on New Connection.
   
4. Select ZS-SALESFORCE Connection Manager from the Connection Managers list and Click on the Add Button.
   
5. Now in Connection Manager UI, Enter your SalesForce Credentials (OAuth Client Id/. Secret –or— User Name, Password and Security Token), and leave all other properties as it is.
   

Call Create Job – Retrieve / Save JobId

Once the connection is created, we can create our first step. Basically, we will call this API to create Job (Bulk Read)

1. Here, in Visual Studio, drag and drop the ZS Salesforce API Task on the design surface.
2. Double click the Task and configure Request Settings Tab like below
   – Select Salesforce Connection (Created in the previous section)
   – Check Use direct URL
   – Enter URL like below (Replace {your-instance} with your own instance (i.e. na9) . For more information on how to find out my
   salesforce instance id check this article
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query

   – Request Method POST
   – Enter Body like below (You can use your own SOQL query). Keep in mind SQOL must be encoded for JSON format. This means the entire SQL must be in one line (replace new lines with \r\n , quotes with \” and tabs with \t)
   For example, to execute SELECT Id, Name FROM Accounts, you can write something like the one below in BODY.
   

   {
     "operation": "query",
     "query": "SELECT Id, Name\r\nFROM Account"
   }

   The above is for below (with a new line)
   

   SELECT Id, Name
   FROM Account

   

   Create Job – Call Bulk API to Read / Query using SOQL

3. Now Click Test Request (Copy the JobId which we will use in the Variable for Designtime Testing)
4. On the Response Tab, configure it like the below.
   – Select Response content Type = Json
   – Enter $.id as the expression
   – Check Save response option
   – Select Save to Variable and click New to create a variable. Name it JobId and enter a sample JobId value (copied from the previous step)
   

   Extract Salesforce JobId save to SSIS Variable

Do Status Check – Look for JobComplete

Once we create a job and store JobId in the variable we have to keep checking Job status every X seconds. Don’t worry you don’t have to use Loop or timer control for that. ZappySys made it so simple to check such thing without Loop /Timer.

1. Drag and drop the ZS Salesforce API Task in the design surface.
2. Double click and configure like below. URL will be different this time. Use variable as below and replace your instance ID
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}

   

   Check Salesforce Job Status Periodically (Without Loop + Timer Controls)

3. Now go to the Status Check Tab and configure it, as shown below. It will keep looking for the words below in the response to decide whether it is a Success or Failure. If those words are not found, they will be checked every 5 seconds (Maximum 1200 secs).
   – Check Enable Status Check option
   – For Success Value – Enter JobComplete
   -Check Fail task option
   – Enter Cancel|Failed|Abort in the text box
   

   Configure Job Status Check Loop using Salesforce API Task

Read data (Compressed CSV Format)

Last step is read the result from completed Salesforce Job. Its in CSV format so we will use Advanced SSIS CSV Source which supports API connection to Salesforce. For that, we will use the same Salesforce API Task used before.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
2. Double click on the DataFlow task to see DataFlow designer surface.
3. Here, In Visual Studio, drag and drop the ZS CSV Source (Web API or File) in the design panel
   
4. Double click on ZS CSV Source (Web API or File) Configure it like below.
   – Enter URL to read result like this (You can supply OPTIONAL maxRecords parameters to control how much data comes in a single response)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results
   --OR--
   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results?maxRecords=50000

   – Check Use Credentials and select Salesforce Connection
   – Click Preview (assuming JobId variable has a valid completed JobId)
   

   Read Salesforce Bulk API Job Result – Using SSIS CSV / API Source

5. Now, set up Pagination as below to read all records using an automated loop. You will need VERSION Released after Jun 20, 2022 to use Regular Expression feature in Header Value extractionBy default, Salesforce doesn’t send all records in response so we need to setup pagination. In below example we are reading Sforce-Locator attribute from Response and sending it to the next URL as locator=<continue-token> value
   – Choose Response Header Contains Continue Token 
   – Enter below expression in Page Num Indicator
   

   locator=Sforce-Locator(^((?!null\b).)*$)

   Basically, the above expression will keep looping until we find Sforce-Locator : null in the response header.
   

   Pagination for Salesforce Bulk Query API (Get Header Value from Sforce-Locator)

6. Click OK to save and now run entire package like below.

Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode / large dataset)

Conclusion

In this article we saw how to use various ZappySys Components / Task to achieve complex Salesforce API calls to read Query data using Bulk API. Download SSIS PowerPack and try yourself and also explore 70+ other tasks /components not discussed in this article.

The post How to query salesforce data using Bulk API in SSIS appeared first on ZappySys Blog.

In this section, we will discuss how to register OAuth App to call Salesforce API using Client ID / Secret rather than using your User Id / Password. Once you follow these steps below to register your Salesforce App (OAuth App), at the end you will get a Client ID (sometimes referred to as App Id) and Client Secret (or App Secret). These two pieces can be used later on to call Salesforce API using OAuth. So let's get started.

Register Salesforce App (OAuth)

1. Login into your account and go to Setup as shown below (click on gear icon at the top right corner):
2. Then in the search bar input "app" and open App Manager:
3. Then click New Connected App to create a new OAuth App: NOTE: In some version, it may say New External Client App

   Look for one of the following buttons:

   • New Connected App (Classic name)

   • New External Client App (Lightning name)

   
4. Give your App a name and: - Enable OAuth Settings - Set callback URL to following two URLs (one in each line) https://login.salesforce.com/services/oauth2/success https://zappysys.com/oauth - Add an OAuth scope you want or simply add "Full access". - Add "Perform requests on your behalf at any time (offline_token)" scope to get a refresh token so that you don't have to re-authenticate each time requesting an access token: IMPORTANT: You can use "Access and manage your data (api)" if you do not like to grant full permission to this app but make sure you request same permission from the caller app too  (Example on ZappySys Salesforce Connection UI -> Advanced Tab -> Set scope as "api offline_access" rather than "full offline_access"
5. Click "Save".

Get Client Id and Client Secret for Salesforce Connected App

Once the App is created, we need to find out the App Id / Secret so we can use it for API calls.

1. Go to App Manager
2. Find out the recently created App and click View (click on the small arrow to the right)
3. Once App page is loaded you can Copy Consumer Key and Consumer Secret (click on reveal to see). Also copy anyone Call back URL (We will need this for OAuth Connection > Advanced Tab > Call Back URL or Redirect URL property depending on which ZappySys Product / Driver you are using)
4. That's it, you can now use these 3 pieces to ZappySys OAuth connection UI or on Salesforce Connection UI (new version with OAuth support). The above screenshot shows where to enter these 3 pieces in the New Salesforce Connection. For the OAuth Connection, the redirect URL goes to the Advanced Tab > Callback URL textbox.

Get Client Id and Secret for Salesforce External Client App

If you created a new App by clicking on the New External Client App button, then your App may not be visible in the App Manager Screen. In that case, find it below way. [caption id="attachment_11637" align="alignnone" width="1222"] Get Client Id, Client Secret - For New External Client App[/caption]

Enable Client Credentials Grant (Non-Interactive Run-As)

If you like to enable Non-Interactive mode, just like UserId / Password Authentication, but without actually passing a real UserId or Password on the Connection UI, then you can enable Client Credentials Flow as below. NOTE: Only the New version of ZappySys Salesforce Connection supports Client Credentials Grant. Version released after Nov 1, 2025 supports this grant type. [caption id="attachment_11639" align="alignnone" width="419"] Enable Salesforce Client Credentials Grant (Run As)[/caption]

(OPTIONAL) Configure Salesforce OAuth refresh token validity

By default, Salesforce Refresh Token doesn't expire once you generate. If you want to change the default behavior, follow these steps.

1. Go to "Manage Connected Apps" menu item and press on your App name:
2. Click "Edit Policies" to configure refresh token validity:
3. Set "All users may self-authorize" and "Refresh token is valid until revoked" options:
4. Click "Save".

Using Salesforce Connection with OAuth App in SSIS / ODBC

Once you have Client ID / Secret, you can use it in the Salesforce Connection in SSIS Connector / ODBC Driver for Salesforce like below (Only the new version will have OAuth option)

Troubleshooting Known Issues

Let’s look at some common issues with Salesforce Connection and how to fix them.

OAuth Error: Missing Required Code Challenge

When you click Generate Token button, you may get an error like the one below.

Salesforce OAuth Generate Token Error – Missing Required Code Challenge

Solution

Here is how to fix it.

For Salesforce Connection UI: Go to Advanced Tab => Check Enable PKCE Code Verifier option
–OR–
For OAuth Connection UI: Go to OAuth2 Grant Options Tab => Check Enable Code Verifier Option

Enable PKCE code verifier option – OAuth Connection for Salesforce.

The post How to register Salesforce App and obtain Client ID / Secret (for API Call / OAuth) appeared first on ZappySys Blog.

In our previous posts we saw how to read and write Salesforce Data using SSIS. In this post we will see how to read Salesforce Metadata in SSIS using JSON Source.

Step-By-Step

In this section you will learn how to use ZappySys JSON Source Adapter to extract data from Salesforce REST API (JSON Format). You can call pretty much Any REST API to fetch data from salesforce but our example for this post would be read metadata (i.e. table list) from salesforce (we will call sobjects to get all table information) .

So lets get started.

1. In case you missed the previous section, You need to Download and Install SSIS ZappySys PowerPack.
2. Once you finished first step, Open Visual Studio and Create New SSIS Package Project.
3. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
4. Double click on the Data Flow task to see Data Flow designer surface.
5. From the SSIS toolbox drag and drop JSON Source on the Data Flow designer surface.
   
6. Now double click JSON Source enter any valid REST API URL like below. Make sure you used correct instance name. Using wrong instance name might slow down connection or fail your request. If you are not sure then just login to salesforce portal using credentials you like to use and observe browser URL.. It will have instance number.
   

   https://na34.salesforce.com/services/data/v20.0/sobjects/Account/describe/

7. Check Use Connection and select New ZS-Salesforce Connection from the list.
   

   Create new REST API Connection in SSIS (For JSON, XML or CSV Source)

8. Configure your salesforce connection
   
9. Select or type filter. For example in this case you will type 

   $.sobjects[*]

10. Click OK to Save UI and attach your source to some target like OLEDB Destination and run your data flow to test.
11. Here is full UI after all configurations.
    

    Read Salesforce Metadata in SSIS (Get Table List Example – SOObject REST API Call)

How to read Salesforce Table Fields Metadata

If you have need to read fields and its information for tables then use below steps

1. Use almost identical steps as previous section. Except two things. Our URL and Filter would be different.
2. Configure your API URL as below
   

   https://YOUR-INSTANCE.salesforce.com/services/data/v20.0/sobjects/YOUR-TABLE/describe/

   Example:
   

   https://demo.salesforce.com/services/data/v20.0/sobjects/Account/describe/

3. Configure Filter as a 

   $.fields[*]

Get Salesforce Table Fields Metadata using REST API call in SSIS

The post How to read Salesforce Metadata in SSIS using REST API (JSON Source) appeared first on ZappySys Blog.

In our previous posts we saw how to read and write Salesforce Data using SSIS Salesforce Connectors (Source /Destination). It covered how to read salesforce data from Tables and SOQL query but it does not support reading from reports. So now in this post we will see how to read Salesforce data from Custom Reports in SSIS which will use JSON Source rather than Salesforce Source.

Later in this post we will show you how to integrate Salesforce data in Apps like Power BI, SSRS, Informatica and languages like C#, Java, Python or T-SQL for MS SQL Folks.

So let’s get started!!!

Read Salesforce Custom Reports in SSIS

In this section you will learn how to use ZappySys JSON Source Adapter to extract data from Salesforce REST API (JSON Format). You can call pretty much Any REST API to fetch data from salesforce but our example for this post would be read metadata (i.e. table list) from salesforce (we will call sobjects to get all table information) .

So lets get started.

1. In case you missed the previous section, You need to Download and Install SSIS ZappySys PowerPack.
2. Once you finished first step, Open Visual Studio and Create New SSIS Package Project.
3. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
4. Double click on the Data Flow task to see Data Flow designer surface.
5. From the SSIS toolbox drag and drop JSON Source on the Data Flow designer surface.
   
6. Now double click JSON Source and enter your Report URL like below. Make sure you replace instance name and report ID as per the screenshot.
   

   https://YOUR-ORG-INSTANCE.salesforce.com/services/data/v64.0/analytics/reports/YOUR-REPORT-ID

   Example URLs
   

   Using Salesforce Org Domain:
   https://zappysys21.salesforce.com/services/data/v64.0/analytics/reports/00O61000001a97n
   
   Using Personal Lighting/My Domain (works when [My Domain] feature enabled):
   https://zappysys21.my.salesforce.com/services/data/v64.0/analytics/reports/00O61000001a97n
   
   Using Direct Pod/Instance URL:
   https://na34.salesforce.com/services/data/v64.0/analytics/reports/00O61000001a97n

7. Check Use Credentials and configure Salesforce Connection
   

   Create new REST API Connection in SSIS (For JSON, XML or CSV Source)

8. Configure your Salesforce connection
9. Select or type filter. For example in this case you will type 

   $.factMap.T!T.rows[*]

10. Uncheck Use Parents option (You can keep it but it will add many unwanted fields)
    NOTE: Please make sure that you’re configuring an API URL and not the HTML/Browser Page URL as both URLs will be different.Please make sure you replace instance name and report ID as per the below screenshot.
    

    https://YOUR-ORG-INSTANCE.salesforce.com/services/data/v64.0/analytics/reports/YOUR-REPORT-ID

    Example URLs 
    

    https://na34.salesforce.com/services/data/v64.0/analytics/reports/00O61000001a97n
    
    Using Salesforce Org Domain:
    https://zappysys21.salesforce.com/services/data/v64.0/analytics/reports/00O61000001a97n
    
    Using Personal Lighting/My Domain (works when [My Domain] feature enabled):
    https://zappysys21.my.salesforce.com/services/data/v64.0/analytics/reports/00O61000001a97n
    
    Using Direct Pod/Instance URL:
    https://na34.salesforce.com/services/data/v64.0/analytics/reports/00O61000001a97n

    

    Configure – Read Salesforce Custom Reports Data

11. Now Go to Array Transform Tab
    1. Select Transform Type :

       Complex 2-dimensional array

    2. Column name filter:

       $.reportMetadata.detailColumns[*]

    3. Row values filter:

       $.dataCells[*].label

12. Here is how your Array Transform Tab configuration should look like
    

    Configure – Advanced Array Transform (Read Salesforce Custom Reports Data in SSIS)

13. Now Click OK to Save UI and attach your source to some target like OLEDB Destination and run your data flow to test.
    

    Extract Data from Salesforce Custom Reports in SSIS – Load into Target like SQL Server, MySQL, PostgreSql, Oracle

Read Salesforce Custom Reports in ODBC Apps

So far we have seen how to read salesforce custom report data in SSIS using ETL approach, but what if you are not using SSIS and you want to pull data in programming languages like C#, Java, Python or apps like Power BI, Informatica, Excel, SSRS or using pure T-SQL way in SQL Server?

ZappySys Provides native Salesforce Driver to read / write Salesforce Object’s data (Its part of ODBC PowerPack). This driver can read/write standard objects (e.g. Accounts, Contacts) However reading from salesforce custom report is not supported by native driver. For this purpose you have to use JSON / REST Driver using this approach. With Generic REST / JSON API driver you can automate pretty much any Salesforce Operations. To get some advanced information on Report API you can refer this guide

So let’s get started.

1. Follow this blog post to install JSON / REST Driver and configure Salesforce OAuth Connection.
2. Once you can read some sample data described in previous step we are ready to run advanced queries.
3. Go to Preview Tab and Run below Query to pull data from Custom Report
   

   --change YOUR-INSTANCE and YOUR-REPORT-ID
   SELECT * FROM $
   WITH(
   Src='https://YOUR-ORG-INSTANCE.salesforce.com/services/data/v64.0/analytics/reports/YOUR-REPORT-ID'
   ,Filter='$.factMap.T!T.rows[*]'
   ,ArrayTransformType='TransformComplexTwoDimensionalArray'
   ,ArrayTransColumnNameFilter='$.reportMetadata.detailColumns[*]'
   ,ArrayTransRowValueFilter='$.dataCells[*].label'
   ,RequestMethod='GET'
   )

    

Integrate Salesforce Data in ODBC Apps (e.g. Power BI, SSRS, Excel, Informatica, Access, C#…)

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Conclusion

In this post we saw how easy it is to perform Salesforce Data Integration in SSIS or any other apps using ZappySys SSIS Connectors or ODBC Drivers. Download FREE Trial of SSIS PowerPack or if you need to integration in other apps then get ODBC PowerPack here

The post Read from Salesforce Custom Reports in SSIS / ODBC appeared first on ZappySys Blog.