In our previous post, we saw how to read Salesforce data using SSIS Salesforce Source (Which uses SOAP API under the hood). This may be slow if you have millions of rows because Salesforce Source paginates every 2000 rows, so it’s not a good option to read millions of rows to iterate on a large Salesforce dataset (Table or Query). To increase the read speed, you can use BULK API. This article explains how to read salesforce data using BULK API and increase read speed.

NEW Version (v5.5+)

In the new version (v5.5), we have added an option to query in Bulk Mode. This means if you have millions of Salesforce object rows to read, no worries; it may be very fast in the new version. Previously, it used to take a long time because it read in batches of 2000 rows in each request. With the Bulk API, the batch size is really large.

Here is how to set Salesforce Bulk API Mode in SELECT query (Read operation).

1. Install new version of SSIS PowerPack. Make sure it is v5.5 or higher.
2. Open existing Salesforce Source UI
3. Make sure you check [Use Bulk API] option like below. if you do not see this option then most likely you are using older version.
   -OR-
   Just add #bulkmode prefix in the SQL query like below, it will use Bulk API
   

   Bulk API mode for SSIS Salesforce Source (Read Large amount of data using SOQL Query)

OLD Version

If you are using an older version of SSIS Power Pack (v5.4.0 or older), you can follow the steps mentioned in the following section.

We will see how to query using Salesforce Bulk API in SSIS. It involves three simple steps like the one below.

1. Call BULK API – Submit SOQL Query and receive a JobId back (Use SSIS Salesforce API Task)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query

2. Keep checking Job Status Until Status is JobComplete (Use SSIS Salesforce API Task)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}

3. Read Query Result  (Use SSIS CSV Source)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results

Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode / large dataset)

Now, let’s look at everything in detail.

Obtain Salesforce Credentials

The first step to call Salesforce API is to obtain credentials (security token or OAuth ClientId /Secret)

Method#1 – Check this article to use UserId/ Password / SecretToken
Method#2 – Check this article to use the OAuth Connection Option

Create Salesforce Connection

Once we have Salesforce Credentials, we can create a new Salesforce Connection (You can do this from Task UI or the below way)

1. Open Visual Studio ( Assuming you already have SSIS installed along with ZappySys PowerPack )
2. Create a New SSIS Package Project.
3. Right-click on the Connection Manager panel and click on New Connection.
   
4. Select ZS-SALESFORCE Connection Manager from the Connection Managers list and Click on the Add Button.
   
5. Now in Connection Manager UI, Enter your SalesForce Credentials (OAuth Client Id/. Secret –or— User Name, Password and Security Token), and leave all other properties as it is.
   

Call Create Job – Retrieve / Save JobId

Once the connection is created, we can create our first step. Basically, we will call this API to create Job (Bulk Read)

1. Here, in Visual Studio, drag and drop the ZS Salesforce API Task on the design surface.
2. Double click the Task and configure Request Settings Tab like below
   – Select Salesforce Connection (Created in the previous section)
   – Check Use direct URL
   – Enter URL like below (Replace {your-instance} with your own instance (i.e. na9) . For more information on how to find out my
   salesforce instance id check this article
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query

   – Request Method POST
   – Enter Body like below (You can use your own SOQL query). Keep in mind SQOL must be encoded for JSON format. This means the entire SQL must be in one line (replace new lines with \r\n , quotes with \” and tabs with \t)
   For example, to execute SELECT Id, Name FROM Accounts, you can write something like the one below in BODY.
   

   {
     "operation": "query",
     "query": "SELECT Id, Name\r\nFROM Account"
   }

   The above is for below (with a new line)
   

   SELECT Id, Name
   FROM Account

   

   Create Job – Call Bulk API to Read / Query using SOQL

3. Now Click Test Request (Copy the JobId which we will use in the Variable for Designtime Testing)
4. On the Response Tab, configure it like the below.
   – Select Response content Type = Json
   – Enter $.id as the expression
   – Check Save response option
   – Select Save to Variable and click New to create a variable. Name it JobId and enter a sample JobId value (copied from the previous step)
   

   Extract Salesforce JobId save to SSIS Variable

Do Status Check – Look for JobComplete

Once we create a job and store JobId in the variable we have to keep checking Job status every X seconds. Don’t worry you don’t have to use Loop or timer control for that. ZappySys made it so simple to check such thing without Loop /Timer.

1. Drag and drop the ZS Salesforce API Task in the design surface.
2. Double click and configure like below. URL will be different this time. Use variable as below and replace your instance ID
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}

   

   Check Salesforce Job Status Periodically (Without Loop + Timer Controls)

3. Now go to the Status Check Tab and configure it, as shown below. It will keep looking for the words below in the response to decide whether it is a Success or Failure. If those words are not found, they will be checked every 5 seconds (Maximum 1200 secs).
   – Check Enable Status Check option
   – For Success Value – Enter JobComplete
   -Check Fail task option
   – Enter Cancel|Failed|Abort in the text box
   

   Configure Job Status Check Loop using Salesforce API Task

Read data (Compressed CSV Format)

Last step is read the result from completed Salesforce Job. Its in CSV format so we will use Advanced SSIS CSV Source which supports API connection to Salesforce. For that, we will use the same Salesforce API Task used before.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
2. Double click on the DataFlow task to see DataFlow designer surface.
3. Here, In Visual Studio, drag and drop the ZS CSV Source (Web API or File) in the design panel
   
4. Double click on ZS CSV Source (Web API or File) Configure it like below.
   – Enter URL to read result like this (You can supply OPTIONAL maxRecords parameters to control how much data comes in a single response)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results
   --OR--
   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results?maxRecords=50000

   – Check Use Credentials and select Salesforce Connection
   – Click Preview (assuming JobId variable has a valid completed JobId)
   

   Read Salesforce Bulk API Job Result – Using SSIS CSV / API Source

5. Now, set up Pagination as below to read all records using an automated loop. You will need VERSION Released after Jun 20, 2022 to use Regular Expression feature in Header Value extractionBy default, Salesforce doesn’t send all records in response so we need to setup pagination. In below example we are reading Sforce-Locator attribute from Response and sending it to the next URL as locator=<continue-token> value
   – Choose Response Header Contains Continue Token 
   – Enter below expression in Page Num Indicator
   

   locator=Sforce-Locator(^((?!null\b).)*$)

   Basically, the above expression will keep looping until we find Sforce-Locator : null in the response header.
   

   Pagination for Salesforce Bulk Query API (Get Header Value from Sforce-Locator)

6. Click OK to save and now run entire package like below.

Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode / large dataset)

Conclusion

In this article we saw how to use various ZappySys Components / Task to achieve complex Salesforce API calls to read Query data using Bulk API. Download SSIS PowerPack and try yourself and also explore 70+ other tasks /components not discussed in this article.

The post How to query salesforce data using Bulk API in SSIS appeared first on ZappySys Blog.

In this section, we will discuss how to register OAuth App to call Salesforce API using Client ID / Secret rather than using your User Id / Password. Once you follow these steps below to register your Salesforce App (OAuth App), at the end you will get a Client ID (sometimes referred to as App Id) and Client Secret (or App Secret). These two pieces can be used later on to call Salesforce API using OAuth. So let's get started.

Register Salesforce App (OAuth)

1. Login into your account and go to Setup as shown below (click on gear icon at the top right corner):
2. Then in the search bar input "app" and open App Manager:
3. Then click New Connected App to create a new OAuth App: NOTE: In some version, it may say New External Client App

   Look for one of the following buttons:

   • New Connected App (Classic name)

   • New External Client App (Lightning name)

   
4. Give your App a name and: - Enable OAuth Settings - Set callback URL to following two URLs (one in each line) https://login.salesforce.com/services/oauth2/success https://zappysys.com/oauth - Add an OAuth scope you want or simply add "Full access". - Add "Perform requests on your behalf at any time (offline_token)" scope to get a refresh token so that you don't have to re-authenticate each time requesting an access token: IMPORTANT: You can use "Access and manage your data (api)" if you do not like to grant full permission to this app but make sure you request same permission from the caller app too  (Example on ZappySys Salesforce Connection UI -> Advanced Tab -> Set scope as "api offline_access" rather than "full offline_access"
5. Click "Save".

Get Client Id and Client Secret for Salesforce Connected App

Once the App is created, we need to find out the App Id / Secret so we can use it for API calls.

1. Go to App Manager
2. Find out the recently created App and click View (click on the small arrow to the right)
3. Once App page is loaded you can Copy Consumer Key and Consumer Secret (click on reveal to see). Also copy anyone Call back URL (We will need this for OAuth Connection > Advanced Tab > Call Back URL or Redirect URL property depending on which ZappySys Product / Driver you are using)
4. That's it, you can now use these 3 pieces to ZappySys OAuth connection UI or on Salesforce Connection UI (new version with OAuth support). The above screenshot shows where to enter these 3 pieces in the New Salesforce Connection. For the OAuth Connection, the redirect URL goes to the Advanced Tab > Callback URL textbox.

Get Client Id and Secret for Salesforce External Client App

If you created a new App by clicking on the New External Client App button, then your App may not be visible in the App Manager Screen. In that case, find it below way. [caption id="attachment_11637" align="alignnone" width="1222"] Get Client Id, Client Secret - For New External Client App[/caption]

Enable Client Credentials Grant (Non-Interactive Run-As)

If you like to enable Non-Interactive mode, just like UserId / Password Authentication, but without actually passing a real UserId or Password on the Connection UI, then you can enable Client Credentials Flow as below. NOTE: Only the New version of ZappySys Salesforce Connection supports Client Credentials Grant. Version released after Nov 1, 2025 supports this grant type. [caption id="attachment_11639" align="alignnone" width="419"] Enable Salesforce Client Credentials Grant (Run As)[/caption]

(OPTIONAL) Configure Salesforce OAuth refresh token validity

By default, Salesforce Refresh Token doesn't expire once you generate. If you want to change the default behavior, follow these steps.

1. Go to "Manage Connected Apps" menu item and press on your App name:
2. Click "Edit Policies" to configure refresh token validity:
3. Set "All users may self-authorize" and "Refresh token is valid until revoked" options:
4. Click "Save".

Using Salesforce Connection with OAuth App in SSIS / ODBC

Once you have Client ID / Secret, you can use it in the Salesforce Connection in SSIS Connector / ODBC Driver for Salesforce like below (Only the new version will have OAuth option)

Troubleshooting Known Issues

Let’s look at some common issues with Salesforce Connection and how to fix them.

OAuth Error: Missing Required Code Challenge

When you click Generate Token button, you may get an error like the one below.

Salesforce OAuth Generate Token Error – Missing Required Code Challenge

Solution

Here is how to fix it.

For Salesforce Connection UI: Go to Advanced Tab => Check Enable PKCE Code Verifier option
–OR–
For OAuth Connection UI: Go to OAuth2 Grant Options Tab => Check Enable Code Verifier Option

Enable PKCE code verifier option – OAuth Connection for Salesforce.

The post How to register Salesforce App and obtain Client ID / Secret (for API Call / OAuth) appeared first on ZappySys Blog.

Download Salesforce Attachment in SSIS can be difficult the first time. In this new article, we will show how to download an Attachment from the Salesforce Source into a file using SSIS. Working with images is always tricky the first time. However, with the right article, this process is really simple. This article will show you how simple it is to do it.

Salesforce is an extremely powerful CRM based in the cloud. This company was created in California, San Francisco and it is a very common request to get the information from the cloud into a file automatically. These can be achieved with our ZappySys SSIS PowerPack plus some SSIS tasks.

Requirements

In order to do it, we will need the following components:

1. SSDT for SSIS installed
2. ZappySys SSIS Powerpack installed
3. A connection to Salesforce. In order to create a connection to Salesforce, check this video or check our article related.
4. Finally, we will need a file in Salesforce to download.

Getting Started

Creating the connection

1. First of all, in SSDT, create a new SSIS project
2. In order to create a connection in SSDT, go to the connection manager, and select a new connection.
   

   ssis-create-salesforce-connection

3. Also, Select the ZS Salesforce connection. This new connection is created by the ZS SSIS PowerPack plugin mentioned in the requirements. With this connection, we can easily connect to Salesforce and get the Information.
   

   Select SSIS Salesforce connection

4. Finally, write your email, password, and token. For more information about getting the token, please check our article related.
   

   Salesforce credentials in SSIS

Creating the package

1. First of All, Drag and drop the Data Flow Task from SSIS Toolbox and double-click it to edit.
   

   Drag and Drop : Data Flow Task from SSIS Toolbox

2. Secondly, in the Data Flow, we will use the Salesforce Source to connect to Salesforce and get the attachments.
   

   Drag and Drop: Salesforce Source from SSIS Toolbox

3. For this purpose, we will use the attachment table from Salesforce.
4. In addition, we will get the columns from the table. The image is in the body, however, we have additional columns like the id, name, ContenType, BodyLenght, createDate, and more:
   

   SSIS Salesforce Columns

5. Next, we will need to drag and drop SSIS Derived Column from the SSIS Toolbox below the Salesforce Source and connect it.
   

   Drag and drop the Derived Column

6. Double-click on the derived column, to specify the path to store the image in a local folder.
   

   SSIS-derived column download image salesforce

7. Now let’s, drag and drop SSIS Export Column from the SSIS Toolbox >> Other Transforms.
   

   Drag and drop Export Column

8. Double-click on the Export column to download an image of the body. In the Extract Column map the Body column of File data and the Path Column to where you want to save that file.
   

   SSIS export column salesforce image

9. Finally, we will use the trash destination to get the image and finish the flow.
   

   SSIS Trash destination

10. Now, we are ready. Your package should be something like this:
    

    ssis-full-download-package

11. Run the package to test it.
12. If everything is fine, we will be able to see the file. As you can see Downloading Salesforce Attachment in SSIS is a straightforward process.

Conclusion

In conclusion, we learned in this article how to Download Salesforce Attachment in SSIS. Basically, you need to use the ZappySys salesforce source, then use the derived column to specify the path and the Export column to get the image. Finally, the ZS Trash destination will be used  If you liked this article, do not hesitate to continue testing and using ZappySys PowerPack for SSIS.

The post Download Salesforce Attachment in SSIS appeared first on ZappySys Blog.

In our previous posts we saw how to read and write Salesforce Data using SSIS. In this post we will see how to read Salesforce Metadata in SSIS using JSON Source.

Step-By-Step

In this section you will learn how to use ZappySys JSON Source Adapter to extract data from Salesforce REST API (JSON Format). You can call pretty much Any REST API to fetch data from salesforce but our example for this post would be read metadata (i.e. table list) from salesforce (we will call sobjects to get all table information) .

So lets get started.

1. In case you missed the previous section, You need to Download and Install SSIS ZappySys PowerPack.
2. Once you finished first step, Open Visual Studio and Create New SSIS Package Project.
3. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
4. Double click on the Data Flow task to see Data Flow designer surface.
5. From the SSIS toolbox drag and drop JSON Source on the Data Flow designer surface.
   
6. Now double click JSON Source enter any valid REST API URL like below. Make sure you used correct instance name. Using wrong instance name might slow down connection or fail your request. If you are not sure then just login to salesforce portal using credentials you like to use and observe browser URL.. It will have instance number.
   

   https://na34.salesforce.com/services/data/v20.0/sobjects/Account/describe/

7. Check Use Connection and select New ZS-Salesforce Connection from the list.
   

   Create new REST API Connection in SSIS (For JSON, XML or CSV Source)

8. Configure your salesforce connection
   
9. Select or type filter. For example in this case you will type 

   $.sobjects[*]

10. Click OK to Save UI and attach your source to some target like OLEDB Destination and run your data flow to test.
11. Here is full UI after all configurations.
    

    Read Salesforce Metadata in SSIS (Get Table List Example – SOObject REST API Call)

How to read Salesforce Table Fields Metadata

If you have need to read fields and its information for tables then use below steps

1. Use almost identical steps as previous section. Except two things. Our URL and Filter would be different.
2. Configure your API URL as below
   

   https://YOUR-INSTANCE.salesforce.com/services/data/v20.0/sobjects/YOUR-TABLE/describe/

   Example:
   

   https://demo.salesforce.com/services/data/v20.0/sobjects/Account/describe/

3. Configure Filter as a 

   $.fields[*]

Get Salesforce Table Fields Metadata using REST API call in SSIS

The post How to read Salesforce Metadata in SSIS using REST API (JSON Source) appeared first on ZappySys Blog.

In this version we focused on many bug fixes. Among them most important bug was limitation of only 2000 characters for SQL Query using OPENQUERY statement. Now you can use up to 8000 characters for OPENQUERY and if that’s not enough use EXEC AT statement to send query to Data Gateway. We also made many improvements with our Salesforce Driver so now you can use it along with Data Gateway or any other Reporting / ETL tools.

Version 1.1.4.10904 [Sep 11, 2019]

New Features/Improvements

• NEW: All Drivers – New DSN dialogbox should suggest unique DSN name if already exists (Prevent accidental overwrite)
• NEW: All Drivers – Add support for named parameters for stored procs (e.g. exec webrequest @uri=’https://abc.com’ )
• NEW: Data Gateway – Give Export / Import Settings option
• NEW: Data Gateway – Provide list of database to caller App
• NEW: General – Use default log file name if only folder name is supplied
• NEW: Salesforce Driver – Add Placeholder Engine Support

Bug fixes

• FIX: All API Drivers – Query builders shows Password and Default Attributes on first time launch
• FIX: All API Drivers – RetryInfo not included in Connectionstring or generated SQL
• FIX: All API Drivers – URL JOIN may cause error for decimal values – Query execution error: Input string was not in a correct format.
• FIX: All API Drivers – URL on general tab is wiped out when you use save HTTP Connection
• FIX: All API Drivers – Field names are wrong StatucFieldxxxxxxx and StatucCheckxxxxxx rather than StatusCheckxxxxxx and StatusFieldxxxxxxx
• FIX: CSV Driver – JOIN query fails in CSV driver with invalid format detected
• FIX: Data Gateway – MS SQL Server – Support more than 2000 characters for SQL in OPENROWSET and EXEC – In old version you may get error – Data is invalid if text is longer than 2000 characters
• FIX: Data Gateway – Query with Caching Option hangs
• FIX: Data Gateway – Use of META in WITH clause prevents error in some cases
• FIX: Data Gateway – You may get “The data is invalid” error when the query is too long
• FIX: Data Gateway – You may get Data is invalid error for OAuth Authorization failure
• FIX: Salesforce Driver – When you execute INSERT via Linked Server Query it Inserts row but fails with error at the end
• FIX: Salesforce Driver – Load ConnectionString action throws Object reference not set to an instance of an object.
• FIX: Salesforce Driver – Click on Generate SQL throws Object reference not set to an instance of an object
• FIX: Salesforce Driver – Data is not returned in Excel when you use [From Microsoft Query] option
• FIX: Salesforce Driver – DELETE statement not working
• FIX: Salesforce Driver – Escape sequence not supported (e.g. WHERE name =’Sam\’s’ )
• FIX: Salesforce Driver – OPENQUERY / EXEC AT doesn’t work with DELETE / UPDATE / INSERT
• FIX: Salesforce Driver – SSIS Failed to Generate Metadata due to character encoding issue
• FIX: Salesforce Driver – Using * in SELECT with WHERE is not working

The post ODBC PowerPack v1.1.4 Released appeared first on ZappySys Blog.

In this article we’re going to focus on load data in Salesforce using SSIS (Bulk Insert, Update, Delete and Upsert). If you like to know how to read from Salesforce then refer to previous post here. For Demo purpose we will use AdventuresWorks sample database as our source but you can use your own Database for testing.

Salesforce developer account

To develop ETLs in Salesforce using SSIS you’ll need a Salesforce developer account. If you don’t have one, don’t worry, you can create a limited version FREE developer account. Click here to Sign up for FREE developer account.

Video Tutorial – Salesforce Insert, Upsert, Delete, Update

Getting Started

In this section, we will see how to perform bulk insert, update delete and upsert operations using ZS Salesforce Destination

Configure Salesforce Connection

First, we’ll need setup a connection to read or write Salesforce data is to make sure we have a working connection in SSIS.

1. Create new or open existing SSIS Project in Visual Studio.
2. Open SSIS Package and Go to SSIS designer.
3. Right click in the connection manager panel and click New connection
   

   Create a new SSIS Salesforce Connection

4. In the connection type selection, Select the ZS-SALESFORCE Connection.
5. We’ll use this information: Username, Password, Security Token (click the link learn how to obtain Salesforce Security Token )
6. Click OK to add the connection.
   

   Salesforce using SSIS, Salesforce connection

7. Save all changes

Bulk Insert – Import data into Salesforce using SSIS

Let’s start with the a bulk insert in Salesforce using SSIS. On this section, we’re going to generate retrieve data from and OLEDB Source and proceed to Insert the records into Salesforce using ZS Salesforce Destination.

Configure SQL Server Source (OLEDB)

1. Open SSIS Package and drag Data Flow task from Control Flow SSIS Toolbox
   

   Dragging and dropping Data Flow Task into Control Flow

2. Go to data flow designer. Drag and drop an OLE DB Source
3. Open the OLE DB Source and configure as follow:
   

   Configure SQL Server Source

Configure SSIS Salesforce Destination for Bulk Insert

Once SQL Server source is configure. Now let’s configure Salesforce Destination for Bulk Insert like below.

1. Drag and drop ZS Salesforce Destination
2. Open the destination component
3. Connection Manager
   

   Salesforce using SSIS, upsert destination connection manager

4. Component Properties
   

   salesforce-using-ssis-insert-properties

5. Column Mappings
   

   salesforce-using-ssis-insert-mapping

6. Click OK
7. Drag and drop two Trash Destination components
8. Connect ZS Salesforce Destination for: New records Output and Error Output
9. Right click on the joining arrows and enable data viewers on all previous connections
10. Save all changes
11. Right click in the data flow designer, click Execute Task
    

    salesforce-using-ssis-insert-results

Bulk Update – Modify data in Salesforce using SSIS

In this section we’re going to perform a bulk update on Salesforce using SSIS. We’re going to list some information from our source, list all our Salesforce Account data, merge it and only update the data we match. In below example what we will do is Find out matching records in Salesforce and our Source System based on some External Key field (e.g. AccountNumber in our case). And then obtain Salesforce Record Id for matching AccountNumber and use it to Update target record.

Configure SQL Server Source (OLEDB Source)

1. Open SSIS Package and drag Data Flow task from Control Flow SSIS Toolbox
2. Go to data flow designer. Drag and drop OLE DB Source
3. Open the OLE DB Source and configure
   

   Salesforce using SSIS, Update OLEDB source

Configure SSIS Salesforce Source

1. Now lets Drag and drop ZS Salesforce Source
2. Open the ZS Salesforce and configure. You can select Query Mode and enter query like this because we only care about two fields for now. Use query mode whenever possible because it transfers less data.
   

   select Id,AccountNumber from account

   

   Salesforce using SSIS, Update Salesforce source

Configure Sort Transforms

Now lets sort both previous datasets using Sort Transforms. This is needed to JOIN dataset using Merge Transform (See next section)

1. Drag and drop: 2 Sort and 1 Merge Join components
2. Connect the OLE DB Source to the first Sort component, and configure
   

   Salesforce using SSIS, Update OLEBD sort

3. Connect the Salesforce Source to the second Sort component, and configure
   

   Salesforce using SSIS,update Salesforce sort

Configure Merge Transforms

No next step would be to join previous datasets based on AccountNumber and then we will output Id from Salesforce which is needed for later step.

1. Drag and Drop Merge Transform from SSIS Toolbox like below.
   

   Drag SSIS Merge Join Transform from Toolbox

2. Connect Sorts to the Merge component (sort from OLEDB Source most be on the left)
3. Configure the Merge component
   

   Salesforce using SSIS, update merge

Configure SSIS Salesforce Destination for Bulk Update

1. Drag and drop ZS Salesforce Destination
2. Open the ZS Salesforce Destination and configure
3. Connection Manager tab
   

   Salesforce using SSIS, update destination connection manager

4. Component Properties tab
   

   Salesforce using SSIS, update destination component properties

5. For Update purposes, Salesforce Destinations needs to map the ID from the Account object. For more information, please use the following link SSIS Salesforce Destination – Update / Insert with Lookup Fields (Text to Id)
6. Column Mapping
   

   salesforce using SSIS, update destination column mappings

7. Click OK
8. Drag and drop three Trash Destination components
9. Create two connections from ZS Salesforce Destination as follows: Update records and Error Output
10. Right click on the joining arrows and enable data viewers on all previous connections
11. Save all changes
12. Right click in the data flow designer, click Execute Task
    

    Salesforce using SSIS, update results

Bulk Upsert – Import/Modify data in Salesforce using SSIS

In this section we’re going to perform a bulk upsert of 25 record’s on Salesforce using SSIS. Upsert is an operation that combines both insert an update.

ZS Salesforce Destination allows Upserts operations. You’ll need to select the Upsert as an action and indicate the Key value for the comparison.

Creating External ID field for Salesforce Upsert

For the Upsert action we have to use an Upsert Keyfield (i.e. External ID field). For more information read here External ID. Follow these steps to add a External ID field on Salesforce Accounts object:

1. Login to your Salesforce account.
2. Click on the configuration icon and select “Service Setup”
   

   Salesforce using SSIS, Salesforce edit object

3. On the new Tab, look for USER INTERFACE -> Objects And Fields -> Object Manager
4. Select Account object
5. Select Fields & Relationships
6. Click New, to create a field
   

   Salesforce using SSIS, Salesforce Fields & Configuration

7. For the field type, choose Number and click next
8. For Field Label and Field Name, type TableID. Check Unique and External ID boxes and click next
9. Leave field-level security as default, click next
10. Leave page layouts as default, click next.
11. After finishing the configuration, the field should appear in the Fields & Relationships page

Configure SQL Server Source

1. Open SSIS Package and drag Data Flow task from Control Flow SSIS Toolbox
   

   Dragging and dropping Data Flow Task into Control Flow

2. Go to data flow designer. Drag and drop OLE DB Source
3. Open the OLE DEB Source and configure as follow
   

   Salesforce using SSIS, Upsert OLE DB source

Configure SSIS Salesforce Destination for Upsert

1. Drag and drop ZS Salesforce Destination
2. Open the ZS Salesforce and configure
3. Connection Manager
   

   Salesforce using SSIS, upsert destination connection manager

4. Component Properties
   

   Salesforce using SSIS, upsert destination component properties

5. Column Mapping
   

   salesforce using SSIS, upsert destination column mapping

6. Click OK
7. Drag and drop three Trash Destination components

Configure Error Handling, Outputs for New, Updated, Failed Records

If you like to capture which rows are inserted and which rows are updated then you can drag Blue arrows coming from Salesforce destination component. You can also connect Red arrow to detect failed records and reason for error.

1. Connect ZS Salesforce Destination for: New records Output, Updated Records Output and Error Output
2. Right click on the joining arrows and enable data viewers on all previous connections
3. Right click in the data flow designer, click Execute Task
   

   Salesforce using SSIS, upsert results

Bulk Delete – Delete data in Salesforce using SSIS

In this section, we’re going to perform a bulk delete on Salesforce using SSIS ZS Salesforce Destination. In this example we’re going to list all our source customers (from previous sections), merge it wit the Salesforce Account object, retrieve the Salesforce Account Id and delete the rows. Delete Operation also requires to map Id field so we will use similar technique we used for Update.

Configure SQL Server Source (OLEDB Source)

1. Go to data flow designer. Drag and drop OLE DB Source
2. Open the OLE DEB Source and configure as follow
   

   salesforce-using-ssis-delete-oledb-source

Configure SSIS Salesforce Source

1. Drag and drop ZS Salesforce Source
2. Open the ZS Salesforce and configure.
3. You can select Query Mode and enter query like this because we only care about two fields for now. Use query mode whenever possible because it transfers less data.
   

   select Id,AccountNumber from account

Configure Sort Transforms

Now lets sort both previous datasets using Sort Transforms. This is needed to JOIN dataset using Merge Transform (See next section)

1. Drag and drop: 2 Sort and 1 Merge Join components
2. Connect the OLE DB Source to the first Sort component, and configure
   

   Salesforce using SSIS, Update OLEBD sort

3. Connect the Salesforce Source to the second Sort component, and configure
   

   Salesforce using SSIS,update Salesforce sort

Configure Merge Transforms

No next step would be to join previous datasets based on AccountNumber and then we will output Id from Salesforce which is needed for later step.

1. Now drag Merge Transform from SSIS toolbox
   

   Drag SSIS Merge Join Transform from Toolbox

2. Connect both Sort to the Merge component (sort from OLEDB Source most be on the left)
3. Configure the Merge component
   

   salesforce-using-ssis-delete-merge

Configure SSIS Salesforce Destination for Bulk Delete

1. Drag and drop ZS Salesforce Destination
2. Connect the Merge and ZS Salesforce Destination components. Right click the arrow and enable data viewer
3. Connection Manager tab
   

   Salesforce using SSIS, delete destination connection manager

4. Component Properties tab
   

   Salesforce using SSIS, delete destination component properties

5. For Delete purposes, Salesforce Destinations needs to map the ID from the Account object.For more information, please use the following link SSIS Salesforce Destination – Update / Insert with Lookup Fields (Text to Id) 
6. Column Mapping
   

   salesforce-using-ssis-delete-destination-column-mapping

7. Click OK
8. Save all changes
9. Right click in the data flow designer, click Execute Task
   

   salesforce-using-ssis-delete-results

Error Handling

When working on Salesforce with SSIS, we recommend Trash Destination component to help debugging outputs without performing DML on destinations.

Main benefits of usage:

1. Enable data viewers and get live feedback of the data
2. Save the results on different formats (XML, JSON, plain text files).

Conclusion

In this article, We have seen how easy it is to integrate Salesforce using SSIS. Using drag and drop connector you can perform Bulk insert, update, delete and Upsert operations.

Download ZappySys SSIS PowerPack to learn more about other possible scenarios which are not discussed in this article.

The post Load data in Salesforce using SSIS – Insert, Upsert, Delete, Update appeared first on ZappySys Blog.

In this post we will learn how to call Salesforce Marketing Cloud API using SSIS and load into SQL Server. We will show you use case of SSIS REST API Task  and SSIS JSON Source

Basically there are two steps to call Salesforce Marketing Cloud API

1. Obtain ClientID and ClientSecret ( Check these steps )
2. Get Access Token by calling requestToken API call
3. Once you have token you can call any other API endpoints to read / edit / update Marketing Cloud data. You must supply Token inside Authorization Header along with each call.

Requirements

Before we look into Step-By-Step section to call Salesforce Marketing API (Exacttarget) make sure you met the following requirements.

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download a FREE trial.
4. You have access to Salesforce Marketing Cloud and you have correct permission to create access key (or just ask Salesforce Admin to generate it for you Check these steps )

Getting Started

Now let’s look at step by step approach on how to read salesforce marketing data in SSIS.

Step-1 : Get Access Token for Salesforce Marketing Cloud API call

First step is to get token for Marketing Cloud API calls. We have to call this API to obtain token. This token typically lasts for one hour or so (3600 seconds).

Before we call below steps make sure you have obtained ClientID and ClientSecret ( Check these steps ).

1. Download and install SSIS PowerPack
2. Open SSIS Package
3. Drag ZS REST API Task from the SSIS Toolbox on Control Flow designer.
4. Rename task to call it Get Token
5. Double click to edit as below.
   Production URL:
   

   https://auth.exacttargetapis.com/v1/requestToken

   Sandbox URL:
   

   https://auth-test.exacttargetapis.com/v1/requestToken

   

   Get API Token for Salesforce Marketing Cloud API calls

6. Go to response setting tab and change Response Type to JSON and set expression as $.accessToken , Check Save response and Select Variable <New Variable> (e.g. vToken)
   

   Configure Response Tab – Extract API Token for Salesforce Marketing Cloud API calls

7.  Now Click Test Request/Response to confirm you get token. Copy Token from Bottom panel and click OK to save UI
8. Now right click in designer and click “Variables” and paste Token we got in previous step in the Value (This token is good for one hour only so update if it expires). This manual update is only needed while you designing page… and testing data Preview / get metadata (see Next Section).  When you run full package it will get fresh token each time and ignore Hardcoded token from Variable.

Step-2 : Read data from Salesforce Marketing Cloud API (SSIS JSON Source)

Now once we have token extract step done. We can move to next step which is to read actual data from Exacttarget API (i.e. Salesforce Marketing Cloud API)

We will us SSIS JSON Source to read data (JSON format) and save to SQL Server.

1. Drag data flow task from SSIS Toolbox and drop to control flow surface.
2. Rename Data flow to Get Data or Call Data
3. Connect first step (Get Token) to data flow
4. Double click task and from data flow toolbox drag  ZS JSON Source on the surface
5. Double click JSON Source and configure as below.
6. Click ON Select Filter Button and Select Correct filter… (Make sure select only Array Node icon to extract correct data)
7. Click Preview to confirm
8. Click OK to save
9. Connect JSON source to some target (e.g. OLEDB Destination for SQL Server)
10. Run your package

Read Salesforce Marketing Cloud data in SSIS and load into SQL Server (Using SSIS JSON Source)

Using HTTP Connection (New version – Single Step)

If you are using a new version (with HTTP – Dynamic Token Support), then you can try the following option.

1. On JSON Source Check Use Credentials and choose HTTP connection.
   

   Read Salesforce Martketing Cloud Data using REST API (in SSIS)

2. On the HTTP Connection, choose the following settings
   1. For URL enter

      https://auth.exacttargetapis.com/hub/v1/campaigns/

   2. Choose Dynamic Token Auth
   3. Enter Client ID (same as the previous section)
   4. Enter Client Secret  (same as the previous section)
   5. Click the Configure Dynamic Token link
      

      Salesforce Marketing Cloud API Call in SSIS Using HTTP Connection (Dynamic Token Method)

3.  Now on the Dynamic Token Tab configure like below (Request Settings Tab)
   1. Make sure you are on Request Settings Tab
   2. Enter token endpoint URL
      

      https://auth.exacttargetapis.com/v1/requestToken

   3. Set Request Method (POST)
   4. Select Content Type (application/json)
   5. Enter Body like below
      

      { clientId: "[$userid$]",clientSecret: "<<[$password$],FUN_JSONENC>>" }

   6. Click on Response Settings Tab
      

      Salesforce Marketing Cloud API Call in SSIS Using HTTP Connection (Dynamic Token – Request Settings)

4. On the Response Settings Tab configure like below
   1. Select Extract From Body
   2. Expression Type as Json
   3. Enter expression

      $.accessToken

      

      Salesforce Marketing Cloud API Call in SSIS Using HTTP Connection (Dynamic Token – Response Settings)

5. Click OK and you can preview your API data on JSON Source
   Example: https://auth.exacttargetapis.com/hub/v1/campaigns/
6. If you use HTTP COnnection with Dynamic Token you do not need two step approach we did in the previous section

Debug Web API call using Fiddler

To test how things looking behind the scene we strongly suggest to use tool like  fiddler. You can double click the URL entry (Right side) to see Request and Response Panels. Top panel is Request (URL, Headers, Body) and Bottom Panel is Response.

Debugging Web API call using Fiddler in SSIS

Loading data from REST API to SQL Server

Once you setup HTTP / OAUTH connection we can use JSON Source or XML Source to extract data from API and  load into SQL Server. If you are API is XML based API then use XML Source in below examples else use JSON Source. For example purpose we will load data into SQL Server but you can load into any Target (e.g. Flat file, Oracle, Excel) using Microsoft or ZappySys Destination connectors To load REST API data into SQL Server perform the following steps (Screenshots are generic and used for demonstration for purpose, your values can be different than below)

1. Go to SSIS Data Flow tab
2. Drag ZS JSON Source from the SSIS Toolbox (Use ZS XML Source if your API is XML based API)
3. Double click to edit Source
4. Set API URL you like to call, Check Use Credentials if you need to use HTTP or OAuth connection
5. Set Method / Body if its other than GET. For method other than GET make sure you select correct Content-Type from Dropdown (e.g. application/json). This indicate input data format in Body.
6. Set HTTP Headers if needed (in Most cases you won't need to set this)
7. Select Filter (many times this is not needed). If your API response has array node then select it here. For example if your response looks like this { rows: [ {...}, {...}, {...} ] } then filter would be $.rows[*]. Select Array node you like to extract in the Filter selection UI.
8. Configure Pagination if needed on the pagination tab. Check your API documentation see it supports pagination.
9. Click OK to Save component settings and generate desired metadata / columns for the output.
10. Now drag OLEDB destination from the SSIS toolbox and drop on the Data Flow surface (technically you can use any valid Destination but for demo we will use OLEDB Destination)
11. Connect Source to Destination Component
12. On OLEDB destination select / create new SQL Connection and then Click "New Table"
13. Click on Mappings tab to map columns by names and click OK to save
14. Execute the Package

Configure SSIS OLEDB Destination – Loading REST API Data into SQL Server Table

REST API to SQL Server Column Mappings for OLEDB Destination

Loading REST API to SQL Server in SSIS

POST / Write Data to API (Insert, Update or Delete)

There will be a time you like to automate POST actions (e.g. create new incident via API call). Check this article to learn more on how to POST , DELETE, PUT data using API calls

Common Errors

There are many error can occur during runtime. Here are most common errors you may face at runtime.

Truncation related error

The most common error you may face when you run a SSIS package is truncation error. During the design time only 300 rows are scanned from a source (a file or a REST API call response) to detect datatypes but at runtime it is likely you will retrieve far more records. So it is possible that you will get longer strings than initially expected. For detailed instructions how to fix common metadata related errors read an article "How to handle SSIS errors (truncation, metadata issues)".

Authentication related error

Another frequent error you may get is authentication error, which happens when you deploy/copy a package to another machine and run it there. Check Deployment to Production below to see why it happens and how to solve this problem.

Deployment to Production

In SSIS package sensitive data such as tokens and passwords are by default encrypted by SSIS with your Windows account which you use to create a package. So SSIS will fail to decrypt tokens/passwords, when you run it from another machine using another Windows account. To circumvent this when you are creating a SSIS package which uses authentication components (e.g. an OAuth Connection Manager or an HTTP Connection Manager with credentials, etc.), consider using parameters/variables to pass tokens/passwords. In this way you won’t face authentication related errors when package is deployed to a production server. Check our article on how to configure packages with sensitive data on your production or development server.

The post Read Salesforce Marketing Cloud data in SSIS (ExactTarget API) appeared first on ZappySys Blog.

In our previous posts we saw how to read and write Salesforce Data using SSIS Salesforce Connectors (Source /Destination). It covered how to read salesforce data from Tables and SOQL query but it does not support reading from reports. So now in this post we will see how to read Salesforce data from Custom Reports in SSIS which will use JSON Source rather than Salesforce Source.

Later in this post we will show you how to integrate Salesforce data in Apps like Power BI, SSRS, Informatica and languages like C#, Java, Python or T-SQL for MS SQL Folks.

So let’s get started!!!

Read Salesforce Custom Reports in SSIS

In this section you will learn how to use ZappySys JSON Source Adapter to extract data from Salesforce REST API (JSON Format). You can call pretty much Any REST API to fetch data from salesforce but our example for this post would be read metadata (i.e. table list) from salesforce (we will call sobjects to get all table information) .

So lets get started.

1. In case you missed the previous section, You need to Download and Install SSIS ZappySys PowerPack.
2. Once you finished first step, Open Visual Studio and Create New SSIS Package Project.
3. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
4. Double click on the Data Flow task to see Data Flow designer surface.
5. From the SSIS toolbox drag and drop JSON Source on the Data Flow designer surface.
   
6. Now double click JSON Source and enter your Report URL like below. Make sure you replace instance name and report ID as per the screenshot.
   

   https://YOUR-ORG-INSTANCE.salesforce.com/services/data/v64.0/analytics/reports/YOUR-REPORT-ID

   Example URLs
   

   Using Salesforce Org Domain:
   https://zappysys21.salesforce.com/services/data/v64.0/analytics/reports/00O61000001a97n
   
   Using Personal Lighting/My Domain (works when [My Domain] feature enabled):
   https://zappysys21.my.salesforce.com/services/data/v64.0/analytics/reports/00O61000001a97n
   
   Using Direct Pod/Instance URL:
   https://na34.salesforce.com/services/data/v64.0/analytics/reports/00O61000001a97n

7. Check Use Credentials and configure Salesforce Connection
   

   Create new REST API Connection in SSIS (For JSON, XML or CSV Source)

8. Configure your Salesforce connection
9. Select or type filter. For example in this case you will type 

   $.factMap.T!T.rows[*]

10. Uncheck Use Parents option (You can keep it but it will add many unwanted fields)
    NOTE: Please make sure that you’re configuring an API URL and not the HTML/Browser Page URL as both URLs will be different.Please make sure you replace instance name and report ID as per the below screenshot.
    

    https://YOUR-ORG-INSTANCE.salesforce.com/services/data/v64.0/analytics/reports/YOUR-REPORT-ID

    Example URLs 
    

    https://na34.salesforce.com/services/data/v64.0/analytics/reports/00O61000001a97n
    
    Using Salesforce Org Domain:
    https://zappysys21.salesforce.com/services/data/v64.0/analytics/reports/00O61000001a97n
    
    Using Personal Lighting/My Domain (works when [My Domain] feature enabled):
    https://zappysys21.my.salesforce.com/services/data/v64.0/analytics/reports/00O61000001a97n
    
    Using Direct Pod/Instance URL:
    https://na34.salesforce.com/services/data/v64.0/analytics/reports/00O61000001a97n

    

    Configure – Read Salesforce Custom Reports Data

11. Now Go to Array Transform Tab
    1. Select Transform Type :

       Complex 2-dimensional array

    2. Column name filter:

       $.reportMetadata.detailColumns[*]

    3. Row values filter:

       $.dataCells[*].label

12. Here is how your Array Transform Tab configuration should look like
    

    Configure – Advanced Array Transform (Read Salesforce Custom Reports Data in SSIS)

13. Now Click OK to Save UI and attach your source to some target like OLEDB Destination and run your data flow to test.
    

    Extract Data from Salesforce Custom Reports in SSIS – Load into Target like SQL Server, MySQL, PostgreSql, Oracle

Read Salesforce Custom Reports in ODBC Apps

So far we have seen how to read salesforce custom report data in SSIS using ETL approach, but what if you are not using SSIS and you want to pull data in programming languages like C#, Java, Python or apps like Power BI, Informatica, Excel, SSRS or using pure T-SQL way in SQL Server?

ZappySys Provides native Salesforce Driver to read / write Salesforce Object’s data (Its part of ODBC PowerPack). This driver can read/write standard objects (e.g. Accounts, Contacts) However reading from salesforce custom report is not supported by native driver. For this purpose you have to use JSON / REST Driver using this approach. With Generic REST / JSON API driver you can automate pretty much any Salesforce Operations. To get some advanced information on Report API you can refer this guide

So let’s get started.

1. Follow this blog post to install JSON / REST Driver and configure Salesforce OAuth Connection.
2. Once you can read some sample data described in previous step we are ready to run advanced queries.
3. Go to Preview Tab and Run below Query to pull data from Custom Report
   

   --change YOUR-INSTANCE and YOUR-REPORT-ID
   SELECT * FROM $
   WITH(
   Src='https://YOUR-ORG-INSTANCE.salesforce.com/services/data/v64.0/analytics/reports/YOUR-REPORT-ID'
   ,Filter='$.factMap.T!T.rows[*]'
   ,ArrayTransformType='TransformComplexTwoDimensionalArray'
   ,ArrayTransColumnNameFilter='$.reportMetadata.detailColumns[*]'
   ,ArrayTransRowValueFilter='$.dataCells[*].label'
   ,RequestMethod='GET'
   )

    

Integrate Salesforce Data in ODBC Apps (e.g. Power BI, SSRS, Excel, Informatica, Access, C#…)

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Conclusion

In this post we saw how easy it is to perform Salesforce Data Integration in SSIS or any other apps using ZappySys SSIS Connectors or ODBC Drivers. Download FREE Trial of SSIS PowerPack or if you need to integration in other apps then get ODBC PowerPack here

The post Read from Salesforce Custom Reports in SSIS / ODBC appeared first on ZappySys Blog.

Introduction

Salesforce is a popular CRM tool that lets you manage your sales, contacts, products, and other useful things. In this tutorial, you will learn how to call Salesforce REST API via ODBC driver using ZappySys JSON Driver. You can also use  ZappySys Salesforce Driver for salesforce data read / write but this article covers more more generic API calling usecase.

You will find out how to register a Connected App in Salesforce, set OAuth authentication and use it for data retrieval. We will cover a topic of how to load leads from Salesforce and use them virtually in any application which supports ODBC. We will not stop with that and touch a possibility of calling any method of any REST API-based Salesforce API; be it insert or simple retrieval.

Prerequisites

Make sure you meet these requirements:

1. ZappySys ODBC PowerPack installed.
2. Have a registered, working Salesforce account.

Register Salesforce App (for OAuth) 

In this section, we will discuss how to register OAuth App to call Salesforce API using Client ID / Secret rather than using your User Id / Password. Once you follow these steps below to register your Salesforce App (OAuth App), at the end you will get a Client ID (sometimes referred to as App Id) and Client Secret (or App Secret). These two pieces can be used later on to call Salesforce API using OAuth. So let's get started.

Register Salesforce App (OAuth)

1. Login into your account and go to Setup as shown below (click on gear icon at the top right corner):
2. Then in the search bar input "app" and open App Manager:
3. Then click New Connected App to create a new OAuth App: NOTE: In some version, it may say New External Client App

   Look for one of the following buttons:

   • New Connected App (Classic name)

   • New External Client App (Lightning name)

   
4. Give your App a name and: - Enable OAuth Settings - Set callback URL to following two URLs (one in each line) https://login.salesforce.com/services/oauth2/success https://zappysys.com/oauth - Add an OAuth scope you want or simply add "Full access". - Add "Perform requests on your behalf at any time (offline_token)" scope to get a refresh token so that you don't have to re-authenticate each time requesting an access token: IMPORTANT: You can use "Access and manage your data (api)" if you do not like to grant full permission to this app but make sure you request same permission from the caller app too  (Example on ZappySys Salesforce Connection UI -> Advanced Tab -> Set scope as "api offline_access" rather than "full offline_access"
5. Click "Save".

Get Client Id and Client Secret for Salesforce Connected App

Once the App is created, we need to find out the App Id / Secret so we can use it for API calls.

1. Go to App Manager
2. Find out the recently created App and click View (click on the small arrow to the right)
3. Once App page is loaded you can Copy Consumer Key and Consumer Secret (click on reveal to see). Also copy anyone Call back URL (We will need this for OAuth Connection > Advanced Tab > Call Back URL or Redirect URL property depending on which ZappySys Product / Driver you are using)
4. That's it, you can now use these 3 pieces to ZappySys OAuth connection UI or on Salesforce Connection UI (new version with OAuth support). The above screenshot shows where to enter these 3 pieces in the New Salesforce Connection. For the OAuth Connection, the redirect URL goes to the Advanced Tab > Callback URL textbox.

Get Client Id and Secret for Salesforce External Client App

If you created a new App by clicking on the New External Client App button, then your App may not be visible in the App Manager Screen. In that case, find it below way. [caption id="attachment_11637" align="alignnone" width="1222"] Get Client Id, Client Secret - For New External Client App[/caption]

Enable Client Credentials Grant (Non-Interactive Run-As)

If you like to enable Non-Interactive mode, just like UserId / Password Authentication, but without actually passing a real UserId or Password on the Connection UI, then you can enable Client Credentials Flow as below. NOTE: Only the New version of ZappySys Salesforce Connection supports Client Credentials Grant. Version released after Nov 1, 2025 supports this grant type. [caption id="attachment_11639" align="alignnone" width="419"] Enable Salesforce Client Credentials Grant (Run As)[/caption]

(OPTIONAL) Configure Salesforce OAuth refresh token validity

By default, Salesforce Refresh Token doesn't expire once you generate. If you want to change the default behavior, follow these steps.

1. Go to "Manage Connected Apps" menu item and press on your App name:
2. Click "Edit Policies" to configure refresh token validity:
3. Set "All users may self-authorize" and "Refresh token is valid until revoked" options:
4. Click "Save".

Calling Salesforce REST API with ODBC driver

To call Salesforce API, you need to have a Salesforce Connected App with enabled OAuth authentication. It will be used by ODBC Data Source, which we will create in the subsequent step. Then we will configure ODBC Data Source with Salesforce REST API URL and other parameters to access the leads.

How to Find Your Salesforce Instance ID

To call Salesforce API you have to supply Salesforce InstanceID in API Url. If you are not sure what is your Salesforce Instance ID then its very simple. Just look at your Salesforce URL after you login in Browser. First part in Domain would be your InstanceID (e.g. na34 if your URL looks like https://na34.salesforce.com/xxxxxxxxx ). You can also click Setup > Under Administrator > Click Company Profile > Company Information > Look for Instance

How to find your Salesforce Instance ID

Create an ODBC Data Source using ZappySys JSON Driver

1. Open ODBC Data Source Administrator
   

   Finding Windows ODBC Data Source Manager

2. Add a new ODBC Data Source based on ZappySys JSON Driver:
   

   Adding a new ODBC Data Source based on ZappySys JSON Driver

Configure the Data Source

1. Start configuring Data Source:
   – Enter URL:
   https://<your_instance>.salesforce.com/services/data/v20.0/query/?q=SELECT+id,name,email+FROM+Lead
   – Select “OAuth” for Connection Type.
   – You can set HTTP Request Method to “GET”.
   – Continue by clicking “Click to Configure”:
   

   Configuring ODBC Data Source to get leads from Salesforce

2. Once a window appears, go to Salesforce App Manager and view your App settings, so that you can use them for Data Source configuration:
   

   Opening Salesforce App OAuth settings to be used in ODBC Data Source

3. Then go back to ODBC configuration, select tab Advanced and set Callback/Return Url to “https://login.salesforce.com/services/oauth2/success“:
   

   Configuring Callback Url in ODBC Data Source OAuth options

4. Then go back to tab General and configure rest options using Salesforce App settings:
   – From Salesforce App settings copy Consumer Key to Client Id
   – Then copy Consumer Secret to Client Secret
   – Set Authorization Url to “https://login.salesforce.com/services/oauth2/authorize”
   – Set Access Token Url to “https://login.salesforce.com/services/oauth2/token”
   – Set Scopes to below (For complete list of possible scopes refer this link – scroll to OAuth Scopes section)
   

   full
   offline_access

    

    

   

   Configuring ODBC Data Source with Salesforce App OAuth settings

   On a sandbox organization you need to use “test.salesforce.com” domain
5. Click “Generate Token” button to generate the tokens. Once Salesforce login window appears, enter your Salesforce credentials and log in.
6. Click “OK” to close the window.

Load data from Salesforce using ODBC Data Source

We are ready to retrieve the data. So just open “Preview” tab in Data Source configuration window and hit “Preview Data” button:

Previewing Salesforce data in ODBC Data Source

As you see we have no actual lead names because they are present in column “records”. So replace “$” with column name “records” in “SELECT * FROM $” query to select actual lead records from JSON property “records” (which denotes a table in ODBC):

Changing table name in ODBC query to get the actual leads

Then click “Preview Data” again to get the leads:

Successfully retrieved Salesforce leads using Salesforce REST API and ODBC Driver

You are ready to use ODBC Data Source in an application!

Pagination in Salesforce REST API

Even you are ready to use the configured ODBC Data Source in an external application, the solution is not yet fully implemented. If you have too many leads or records than Salesforce can give you in one HTTP response you won’t get all the records. Thus you will need to use pagination. For that, go to “Pagination” tab and configure it like this:

Configuring ODBC Data Source to paginate through Salesforce results

If you need something more sophisticated then check the article about pagination (initially it was written for SSIS PowerPack but UI options and concepts apply to ODBC Driver too).

Using ODBC query in an application

Let’s say you want to access Salesforce Accounts instead of Leads. It seems the most logical thing to do is create another ODBC Data Source and change the URL. However, you can use the same ODBC Data Source and override its properties with ODBC query. Just copy the generated ODBC query, leave the things you want to override and use it in your application:

SELECT * FROM records
WITH(
Src='https://***yourinstance****.salesforce.com/services/data/v44.0/query/?q=SELECT+Id,name,Email+FROM+Lead'
)

In the example above, we are overriding “Url” property and accessing Accounts.

Check an article how to import data into Excel using ODBC Data Source with ZappySys to see how to use ODBC Data Source in your application. For more examples, check ODBC PowerPack blog category.

Field names for SELECT Query

One known issue in Salesforce REST API is you cannot use * to fetch all columns in SOQL. So you must specify column names manually. If you like to know all possible column names for given Table name then use below query.

https://na34.salesforce.com/services/data/v20.0/sobjects/Account/describe/

SELECT label,name,length,type,nillable,unique FROM $
WITH(
	 Src='https://**YOUR-INSTANCE**.salesforce.com/services/data/v44.0/sobjects/**YOUR-TABLE**/describe/'
	,Filter='$.fields[*]'
)

For example to get fields for Account table in ns34 instance you can use below URL

https://na34.salesforce.com/services/data/v44.0/sobjects/Account/describe/

Error Handling in Salesforce REST API

Sometimes errors occur... they just do and there is nothing you can do! Or can you? Actually, in ODBC PowerPack you can handle them in two ways.

METHOD 1 - Using Error Handling Options

When to use?

You may want to use them when your source is a resource located on the Internet; e.g. a file on a website, a file on an FTP server or just a plain API HTTP response. By default, when a remote server returns an error, data retrieval is stopped, an error is raised and no data is given back to you. This might not be always desirable.

Scenario 1

Imagine a scenario, that there is a web server which each day at 12 AM releases a new JSON file with that day's date as filename, e.g. http://www.some-server.com/data/2018-06-20.json. And, of course, you want to download it and use it daily in your Power BI report. But you have a problem: Power BI report data sources are refreshed each hour and you may get HTTP 404 status code (no file was found) when a file is not released yet. Which consequentially means other data sources won't be updated as well and you will see old and cached data on the report. That's where you could use Continue on any error or Continue when Url is invalid or missing (404 Errors) to avoid an error being raised and let other data sources to be updated.

Scenario 2

Another scenario is when you expect a web server to raise some kind of HTTP error when accessing a URL. You don't want ODBC Data Source to raise an error but instead, you want to get response data. That's where you can use Continue on any error or alike together with  Get response data on error to continue on an error and get the data:

METHOD 2 - Using Connection [Retry Settings]

Another scenario you may run into is a buggy web server. You ask it to give you some file or data and it, like a snotty kid, just doesn't give it to you! You have to ask twice or thrice before it does its job. If that's the case, you have to retry HTTP requests using Connection:

Using various Salesforce APIs

We used Salesforce REST API Search method to access the leads. The SOQL query we used looks identical to SQL in our case, so you can use other table names to access their data too. Or you can even use a more complex SOQL query in the same Search method. The Search method is not the only one that is available in Salesforce REST API, check other REST API methods too (you can even use the ones that insert/update the data).

In addition to Salesforce REST API, Salesforce has many more APIs, such as Salesforce Marketing Cloud API, Salesforce CPQ API, Salesforce IoT REST API, and others. You can use virtually any of them with ZappySys JSON Driver or ZappySys XML Driver since the drivers are highly configurable (you can call any API method with any HTTP method – be it POST, PUT or any other).

Reading data from Salesforce Custom Reports

So far we talked basic query to read data using SOQL / Standard Tables. But what if you have cuCheck this article to findout some techniques to read data from salesforce Custom Reports.

Resources

• Salesforce REST API Search method documentation
• Salesforce REST API methods
• Salesforce Table list
• Salesforce SOQL Documentation
• Salesforce API Explorer

Conclusion

We covered a whole process of how to access Salesforce data: starting from creating a Salesforce application, continuing by authenticating using OAuth and finishing by using an ODBC query in a target application. You learned how to retrieve the fields of Salesforce leads and paginate through the results. Now you are on a good path of using other API methods and even other Salesforce APIs 😉

The post How to call Salesforce REST API via ODBC driver appeared first on ZappySys Blog.

In this tutorial, you will learn how to read data from Salesforce to SQL Server using SSIS Salesforce Source.

In this article, we will extract Salesforce contacts and import them into a SQL Server database table. Also, you will learn how to do basic filtering on the contacts and get only a subset of them. Finally, we will briefly touch some additional advanced options of Salesforce Source component. Namely, you will learn how to convert datetime columns to different formats.

To achieve our goal we will use Salesforce Source SSIS component. Under the hood, it uses Salesforce API to make calls to, but you don’t have to be concerned about that:

Salesforce Source Connector

Prerequisites

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. A Salesforce account.
4. Any SQL Server database.
5. ZappySys SSIS PowerPack installed.

Step-by-Step – Read Salesforce data in SSIS

Get Salesforce Security Token

The first thing you will need to do to sync Salesforce with SQL Server is to get the Salesforce Security Token. You will need it for Salesforce Source Connector to connect to your Salesforce account:

1. Log in to your account using https://login.salesforce.com.
2. On the top right corner, press on your profile picture and click

   Settings

   :
   

   Getting Salesforce Security Token for use in Salesforce Connection Manager

3. On the top left corner, under My Personal Information select

   Reset My Security Token

    menu item:
   

   Resetting Salesforce Security Token

4. In the main window area click

   Reset Security Token

    button:
   

   Resetting Salesforce Security Token

5. Check your email for a new Security Token:
   

   Getting new Salesforce Security Token

Import Salesforce data into SQL Server

Once you have Security Token, you can start getting contacts from Salesforce and load them into SQL Server database table:

1. Open SSIS designer.
2. Create a new SSIS package and drag a Data Flow Task into the Control Flow from the SSIS Toolbox:
   

   Starting to create a SSIS package

3. Open Data Flow Task and then add Salesforce Source from SSIS toolbox:
   

   Using Salesforce Source to start exporting data

4. Start modifying Salesforce Source component and then press

   New

    button to create a new Salesforce Connection:
   

   Creating new Salesforce Connection for Salesforce Source

5. In a new window enter your Salesforce credentials and Security Token you have in your email. In advanced options group, you can select the environment in which the connection will work. Most likely, you want to use

   Production

   , so just select it in Service Type option.
   
   

   Salesforce Connection Manager configuration

6. Once you finished creating Salesforce Connection, select

   Contact

    as table you want to get data from. Be sure to have

   Table

    option selected in Access Mode. Also, below you can specify the size of the batch and the maximum number of rows (contacts in our case) to retrieve:
   

   Choosing Salesforce contacts table to export them to SQL Server

7. Close Salesforce Source configuration and in the Data Flow Task drag OLE DB Destination and connect it to Salesforce Source.
   

   Connecting Salesforce Source to OLE DB Destination

8. Open OLE DB Destination configuration, select or create a new connection; in this example, we are using LocalDB and tempdb database. Then select an existing table or create a new one by pressing

   New

    button. Finally, select

   Mappings

    option to map the columns and close the window:
   

   Choosing connection and table in OLE DB Destination

9. Run the package. In this example we extract 16 contacts and load them into a SQL table:
   

   Package execution result

10. Let’s open Salesforce page and take a look at the contacts source. We have 16 contacts here as well:
    

    Salesforce source contacts

11. After querying destination table using SSMS, we see that everything worked like it had to:
    

    Results: contacts exported from Salesforce into SQL Server

12. You are done!

What if things are more complicated? Queries and Data Filtering

Instead of accessing data using

Table

Query

Contact

• First of all, let’s create several variables, which we will use to get specific Salesforce contacts:
  

  Variables used to construct a dynamic query

  We will use

  Domain

   to filter contacts that have email with

  uog.com

   domain.
  

  From

    variable will help us filter the contacts based on the date, when they were created.
• Set

  Query

   in Access Mode option and input query below:
  

  Inputting a SOQL query into a Salesforce Source SSIS component

  Even this query looks much like T-SQL query, but in reality it is a SOQL query, an acronym for Salesforce Object Query Language.
  

  NOTE:

  • We are inputing {{User::From}} straight into a query and do not enclose it with single quotes, unlike in T-SQL.
  • You can explore more SOQL query examples by pressing « View Examples » button.
• Click

  Preview

   and you will see that only contacts that have domain “uog.com” were retrieved. Filtering on CreatedDate column was applied as well:
  

  Data preview for a subset of contacts

SOQL has some limitations, so check SOQL Limits on Objects page as well if you are writing a more complex query.

Advanced Options: Convert Datetime Columns

By default, datetime columns are outputted in

yyyy-MM-ddThh:mm:ss.fffZ

• Default format (e.g.

  2017-07-07T14:41:35.000Z

  )
• Local time (depends on your locale settings e.g.

  5/16/2017 14:41 PM

  )
• UTC time (depends on your locale settings, yet your time zone is taken into account, e.g.

  5/16/2017 11:41 PM

  )

Let’s take CreatedDate column and output it as UTC time:

1. Here is the preview of data, before conversion:
   

   Data preview before formatting datetime column

2. Now let’s output the CreatedDate column as UTC time:
   

   Salesforce Source Date/Time Handling

3. Let’s make another data preview. As you see CreatedDate time shifted to match UTC time:
   

   Data preview after formatting datetime column

Similarly, you can format input datetime columns, when you use

Query

Table

Common Errors

Deployment to Production

Download Sample Package

Be sure to download a sample SQL Server 2008 SSIS package, in case you want to try it right away (you can upgrade it to a higher version).

Conclusion

In this article, you learned how to export data from Salesforce to SQL Server. We used Salesforce Source to extract contacts from Salesforce and import them into a SQL Server database table. We did not have to configure much, just a few clicks and you were done. Also you learned how to use SOQL to write custom queries and filter out the data you don’t need. Finally, now you know how to convert datetime columns into Local and UTC time to match the format you expect in your database.

Keywords
Salesforce SQL Server Integration | Data loading Salesforce | Salesforce API SQL Server | Get data from Salesforce and load into SQL Server | Read data from Salesforce and write it to SQL Server  | Extract data from Salesforce to SQL Server

The post How to read Salesforce data in SSIS – Export to SQL Server appeared first on ZappySys Blog.