{"id":9507,"date":"2022-01-20T19:07:09","date_gmt":"2022-01-20T19:07:09","guid":{"rendered":"https:\/\/zappysys.com\/blog\/?p=9507"},"modified":"2024-06-30T19:10:21","modified_gmt":"2024-06-30T19:10:21","slug":"query-salesforce-data-using-bulk-api-ssis","status":"publish","type":"post","link":"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/","title":{"rendered":"How to query salesforce data using Bulk API in SSIS"},"content":{"rendered":"<h2>Introduction<\/h2>\n<p>In our previous post, we saw how to <a href=\"https:\/\/zappysys.com\/blog\/export-data-from-salesforce-to-sql-server-using-ssis\/\" target=\"_blank\" rel=\"noopener\">read Salesforce<\/a> data using SSIS Salesforce Source (Which uses SOAP API under the hood). This may be slow if you have millions of rows because Salesforce Source paginates every 2000 rows, so it&#8217;s not a good option to read millions of rows to iterate on a large Salesforce dataset (Table or Query). To increase the read speed, you can use BULK API. This article explains how to read salesforce data using BULK API and increase read speed.<\/p>\n<div class=\"content_block\" id=\"custom_post_widget-2523\"><h2><span id=\"Prerequisites\">Prerequisites<\/span><\/h2>\r\nBefore we perform the steps listed in this article, you will need to make sure the following prerequisites are met:\r\n<ol style=\"margin-left: 1.5em;\">\r\n \t<li><abbr title=\"SQL Server Integration Services\">SSIS<\/abbr> designer installed.\u00a0Sometimes it is referred to as <abbr title=\"Business Intelligence Development Studio\">BIDS<\/abbr> or <abbr title=\"SQL Server Data Tools\">SSDT<\/abbr> (<a href=\"https:\/\/docs.microsoft.com\/en-us\/sql\/ssdt\/download-sql-server-data-tools-ssdt\" target=\"_blank\" rel=\"noopener\">download it from the Microsoft site<\/a>).<\/li>\r\n \t<li>Basic knowledge of SSIS package\u00a0development using\u00a0<em>Microsoft SQL Server Integration Services<\/em>.<\/li>\r\n \t<li>Make sure\u00a0<span style=\"text-decoration: underline;\"><a href=\"https:\/\/zappysys.com\/products\/ssis-powerpack\/\" target=\"_blank\" rel=\"noopener\">ZappySys SSIS PowerPack<\/a><\/span>\u00a0is installed (<a href=\"https:\/\/zappysys.com\/products\/ssis-powerpack\/download\/\" target=\"_blank\" rel=\"noopener\">download it<\/a>, if you haven't already).<\/li>\r\n \t<li>(<em>Optional step<\/em>)<em>.<\/em>\u00a0<a href=\"https:\/\/zappysys.zendesk.com\/hc\/en-us\/articles\/360035974593\" target=\"_blank\" rel=\"noopener\">Read this article<\/a>, if you are planning to deploy packages to a server and schedule their execution later.<\/li>\r\n<\/ol><\/div>\n<h2>NEW Version (v5.5+)<\/h2>\n<p>In the new version (v5.5), we have added an option to query in Bulk Mode. This means if you have millions of Salesforce object rows to read, no worries; it may be very fast in the new version. Previously, it used to take a long time because it read in batches of 2000 rows in each request. With the Bulk API, the batch size is really large.<\/p>\n<p>Here is how to set Salesforce Bulk API Mode in SELECT query (Read operation).<\/p>\n<ol>\n<li>Install new version of SSIS PowerPack. Make sure it is v5.5 or higher.<\/li>\n<li>Open existing Salesforce Source UI<\/li>\n<li>Make sure you <strong>check [Use Bulk API]<\/strong> option like below. if you do not see this option then most likely you are using older version.<br \/>\n-OR-<br \/>\nJust add <strong>#bulkmode<\/strong> prefix in the SQL query like below, it will use Bulk API<\/p>\n<div id=\"attachment_11114\" style=\"width: 687px\" class=\"wp-caption alignnone\"><a href=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-bulk-api-option-soql-select-query.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-11114\" class=\"size-full wp-image-11114\" src=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-bulk-api-option-soql-select-query.png\" alt=\"\" width=\"677\" height=\"585\" srcset=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-bulk-api-option-soql-select-query.png 677w, https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-bulk-api-option-soql-select-query-300x259.png 300w\" sizes=\"(max-width: 677px) 100vw, 677px\" \/><\/a><p id=\"caption-attachment-11114\" class=\"wp-caption-text\">Bulk API mode for SSIS Salesforce Source (Read Large amount of data using SOQL Query)<\/p><\/div><\/li>\n<\/ol>\n<h2>OLD Version<\/h2>\n<p>If you are using an older version of SSIS Power Pack (v5.4.0 or older), you can follow the steps mentioned in the following section.<\/p>\n<p>We will see how to query using Salesforce Bulk API in SSIS. It involves three simple steps like the one below.<\/p>\n<ol>\n<li>Call BULK API &#8211; Submit SOQL Query and receive a JobId back (Use <a href=\"https:\/\/zappysys.com\/products\/ssis-powerpack\/ssis-salesforce-api-task\/\">SSIS Salesforce API Task<\/a>)\n<pre class=\"\">https:\/\/{your-instance}.salesforce.com\/services\/data\/v50.0\/jobs\/query<\/pre>\n<\/li>\n<li>Keep checking Job Status Until Status is <strong>JobComplete<\/strong> (Use <a href=\"https:\/\/zappysys.com\/products\/ssis-powerpack\/ssis-salesforce-api-task\/\">SSIS Salesforce API Task<\/a>)\n<pre class=\"\">https:\/\/{your-instance}.salesforce.com\/services\/data\/v50.0\/jobs\/query\/{{User::JobId}}<\/pre>\n<\/li>\n<li>Read Query Result\u00a0 (Use <a href=\"https:\/\/zappysys.com\/products\/ssis-powerpack\/ssis-csv-file-source-flat-file-web-api\/\" target=\"_blank\" rel=\"noopener\">SSIS CSV Source<\/a>)\n<pre class=\"\">https:\/\/{your-instance}.salesforce.com\/services\/data\/v50.0\/jobs\/query\/{{User::JobId}}\/results<\/pre>\n<\/li>\n<\/ol>\n<div id=\"attachment_9515\" style=\"width: 671px\" class=\"wp-caption alignnone\"><a href=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/salesforce-query-bulk-api-read-data.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-9515\" class=\"size-full wp-image-9515\" src=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/salesforce-query-bulk-api-read-data.png\" alt=\"Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode \/ large dataset)\" width=\"661\" height=\"330\" srcset=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/salesforce-query-bulk-api-read-data.png 661w, https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/salesforce-query-bulk-api-read-data-300x150.png 300w\" sizes=\"(max-width: 661px) 100vw, 661px\" \/><\/a><p id=\"caption-attachment-9515\" class=\"wp-caption-text\">Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode \/ large dataset)<\/p><\/div>\n<p>Now, let&#8217;s look at everything in detail.<\/p>\n<h3>Obtain Salesforce Credentials<\/h3>\n<p>The first step to call Salesforce API is to obtain credentials (security token or OAuth ClientId \/Secret)<\/p>\n<p>Method#1 &#8211; Check <a href=\"https:\/\/zappysys.com\/blog\/export-data-from-salesforce-to-sql-server-using-ssis\/#Get_Salesforce_Security_Token\">this article<\/a> to use <strong>UserId\/ Password \/ SecretToken<\/strong><br \/>\nMethod#2 &#8211; Check <a href=\"https:\/\/zappysys.com\/blog\/register-salesforce-app-obtain-client-id-secret-oauth-api-call\/\">this article<\/a> to use the <strong>OAuth<\/strong> Connection Option<\/p>\n<h3>Create Salesforce Connection<\/h3>\n<p>Once we have Salesforce Credentials, we can create a new Salesforce Connection (You can do this from Task UI or the below way)<\/p>\n<ol>\n<li>Open Visual Studio ( Assuming you already have SSIS installed along with\u00a0<a href=\"https:\/\/zappysys.com\/products\/ssis-powerpack\/download\/\" target=\"_blank\" rel=\"noopener\">ZappySys PowerPack<\/a>\u00a0)<\/li>\n<li>Create a New SSIS Package Project.<\/li>\n<li>Right-click on the Connection Manager panel and click on New Connection.<br \/>\n<img decoding=\"async\" class=\"figureimage\" title=\"SSIS Create New Connection\" src=\"https:\/\/zappysys.com\/onlinehelp\/ssis-powerpack\/scr\/images\/ssis-new-connection.png\" alt=\"SSIS Create New Connection\" \/><\/li>\n<li>Select <strong>ZS-SALESFORCE<\/strong> Connection Manager from the Connection Managers list and Click on the Add Button.<br \/>\n<img decoding=\"async\" class=\"figureimage\" title=\"SSIS Salesforce Connection Manager - ADD\" src=\"https:\/\/zappysys.com\/onlinehelp\/ssis-powerpack\/scr\/images\/salesforce-connection\/ssis-salesforce-api-task-connection-1.png\" alt=\"SSIS Salesforce Connection Manager - ADD\" \/><\/li>\n<li>Now in Connection Manager UI, Enter your SalesForce Credentials (OAuth Client Id\/. Secret &#8211;or&#8212; User Name, Password and Security Token), and leave all other properties as it is.<br \/>\n<img decoding=\"async\" class=\"figureimage\" title=\"SSIS Salesforce Connection Manager UI \" src=\"https:\/\/zappysys.com\/onlinehelp\/ssis-powerpack\/scr\/images\/salesforce-connection\/ssis-salesforce-connection-manager-ui.png\" alt=\"SSIS Salesforce Connection Manager UI \" \/><\/li>\n<\/ol>\n<h3>Call Create Job &#8211; Retrieve \/ Save JobId<\/h3>\n<p>Once the connection is created, we can create our first step. Basically, we will call this API to <a href=\"https:\/\/developer.salesforce.com\/docs\/atlas.en-us.api_asynch.meta\/api_asynch\/query_create_job.htm\" target=\"_blank\" rel=\"noopener\">create Job<\/a> (Bulk Read)<\/p>\n<ol>\n<li>Here, in Visual Studio, drag and drop the ZS Salesforce API Task on the design surface.<img decoding=\"async\" class=\"figureimage\" title=\"SSIS Salesforce API Task - Drag and Drop\" src=\"https:\/\/zappysys.com\/onlinehelp\/ssis-powerpack\/scr\/images\/salesforce-api-task\/ssis-salesforce-api-task-drag.png\" alt=\"SSIS Salesforce API Task - Drag and Drop\" \/><\/li>\n<li>Double click the Task and configure Request Settings Tab like below<br \/>\n&#8211; Select Salesforce Connection (Created in the previous section)<br \/>\n&#8211; Check <strong>Use direct URL<\/strong><br \/>\n&#8211; Enter URL like below (Replace <strong>{your-instance}<\/strong> with your own instance (i.e. na9) . For more information on how to find out my<br \/>\nsalesforce instance id\u00a0<a href=\"https:\/\/help.salesforce.com\/s\/articleView?id=000322728&amp;type=1\">check this article<\/a><\/p>\n<pre class=\"\">https:\/\/{your-instance}.salesforce.com\/services\/data\/v50.0\/jobs\/query\r\n<\/pre>\n<p>&#8211; Request Method <strong>POST<\/strong><br \/>\n&#8211; Enter Body like below (You can use your own <a href=\"https:\/\/developer.salesforce.com\/docs\/atlas.en-us.soql_sosl.meta\/soql_sosl\/sforce_api_calls_soql.htm\" target=\"_blank\" rel=\"noopener\">SOQL query<\/a>). Keep in mind SQOL must be <a href=\"https:\/\/www.freeformatter.com\/json-escape.html\" target=\"_blank\" rel=\"noopener\">encoded for JSON format<\/a>. This means the entire SQL must be in one line (replace new lines with \\r\\n , quotes with \\&#8221; and tabs with \\t)<br \/>\nFor example, to execute SELECT Id, Name FROM Accounts, you can write something like the one below in BODY.<\/p>\n<pre class=\"lang:js decode:true \">{\r\n  \"operation\": \"query\",\r\n  \"query\": \"SELECT Id, Name\\r\\nFROM Account\"\r\n}<\/pre>\n<p>The above is for below (with a new line)<\/p>\n<pre class=\"lang:tsql decode:true\">SELECT Id, Name\r\nFROM Account<\/pre>\n<div id=\"attachment_9516\" style=\"width: 690px\" class=\"wp-caption alignnone\"><a href=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-call-salesforce-bulk-api-query-soql-sql.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-9516\" class=\"size-full wp-image-9516\" src=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-call-salesforce-bulk-api-query-soql-sql.png\" alt=\"Create Job - Call Bulk API to Read \/ Query using SOQL\" width=\"680\" height=\"547\" srcset=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-call-salesforce-bulk-api-query-soql-sql.png 680w, https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-call-salesforce-bulk-api-query-soql-sql-300x241.png 300w\" sizes=\"(max-width: 680px) 100vw, 680px\" \/><\/a><p id=\"caption-attachment-9516\" class=\"wp-caption-text\">Create Job &#8211; Call Bulk API to Read \/ Query using SOQL<\/p><\/div><\/li>\n<li>Now Click Test Request (Copy the <strong>JobId<\/strong> which we will use in the Variable for Designtime Testing)<\/li>\n<li>On the Response Tab, configure it like the below.<br \/>\n&#8211; Select Response content Type = Json<br \/>\n&#8211; Enter <strong>$.id<\/strong> as the expression<br \/>\n&#8211; Check Save response option<br \/>\n&#8211; Select Save to Variable and click New to create a variable. Name it <strong>JobId<\/strong> and enter a sample <strong>JobId<\/strong> value (copied from the previous step)<\/p>\n<div id=\"attachment_9517\" style=\"width: 459px\" class=\"wp-caption alignnone\"><a href=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-call-salesforce-bulk-api-extract-jobid-from-result.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-9517\" class=\"size-full wp-image-9517\" src=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-call-salesforce-bulk-api-extract-jobid-from-result.png\" alt=\"Extract Salesforce JobId save to SSIS Variable\" width=\"449\" height=\"236\" srcset=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-call-salesforce-bulk-api-extract-jobid-from-result.png 449w, https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-call-salesforce-bulk-api-extract-jobid-from-result-300x158.png 300w\" sizes=\"(max-width: 449px) 100vw, 449px\" \/><\/a><p id=\"caption-attachment-9517\" class=\"wp-caption-text\">Extract Salesforce JobId save to SSIS Variable<\/p><\/div><\/li>\n<\/ol>\n<h3>Do Status Check &#8211; Look for JobComplete<\/h3>\n<p>Once we create a job and store JobId in the variable we have to keep <a href=\"https:\/\/developer.salesforce.com\/docs\/atlas.en-us.api_asynch.meta\/api_asynch\/query_get_one_job.htm\" target=\"_blank\" rel=\"noopener\">checking Job status<\/a> every X seconds. Don&#8217;t worry you don&#8217;t have to use Loop or timer control for that. ZappySys made it so simple to check such thing without Loop \/Timer.<\/p>\n<ol>\n<li>Drag and drop the ZS Salesforce API Task in the design surface.<img decoding=\"async\" class=\"figureimage\" title=\"SSIS Salesforce API Task - Drag and Drop\" src=\"https:\/\/zappysys.com\/onlinehelp\/ssis-powerpack\/scr\/images\/salesforce-api-task\/ssis-salesforce-api-task-drag.png\" alt=\"SSIS Salesforce API Task - Drag and Drop\" \/><\/li>\n<li>Double click and configure like below. URL will be different this time. Use variable as below and replace your instance ID\n<pre class=\"\">https:\/\/{your-instance}.salesforce.com\/services\/data\/v50.0\/jobs\/query\/{{User::JobId}}<\/pre>\n<div id=\"attachment_9518\" style=\"width: 720px\" class=\"wp-caption alignnone\"><a href=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-api-job-status-check.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-9518\" class=\"size-full wp-image-9518\" src=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-api-job-status-check.png\" alt=\"Check Salesforce Job Status Periodically (Without Loop + Timer Controls) \" width=\"710\" height=\"566\" srcset=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-api-job-status-check.png 710w, https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-api-job-status-check-300x239.png 300w\" sizes=\"(max-width: 710px) 100vw, 710px\" \/><\/a><p id=\"caption-attachment-9518\" class=\"wp-caption-text\">Check Salesforce Job Status Periodically (Without Loop + Timer Controls)<\/p><\/div><\/li>\n<li>Now go to the Status Check Tab and configure it, as shown below. It will keep looking for the words below in the response to decide whether it is a Success or Failure. If those words are not found, they will be checked every 5 seconds (Maximum 1200 secs).<br \/>\n&#8211; Check Enable Status Check option<br \/>\n&#8211; For Success Value &#8211; Enter <strong>JobComplete<\/strong><br \/>\n-Check Fail task option<br \/>\n&#8211; Enter <strong>Cancel|Failed|Abort<\/strong> in the text box<\/p>\n<div id=\"attachment_9519\" style=\"width: 709px\" class=\"wp-caption alignnone\"><a href=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-api-job-status-check-2.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-9519\" class=\"size-full wp-image-9519\" src=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-api-job-status-check-2.png\" alt=\"Configure Job Status Check Loop using Salesforce API Task\" width=\"699\" height=\"640\" srcset=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-api-job-status-check-2.png 699w, https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-api-job-status-check-2-300x275.png 300w\" sizes=\"(max-width: 699px) 100vw, 699px\" \/><\/a><p id=\"caption-attachment-9519\" class=\"wp-caption-text\">Configure Job Status Check Loop using Salesforce API Task<\/p><\/div><\/li>\n<\/ol>\n<h2><\/h2>\n<h3>Read data (Compressed CSV Format)<\/h3>\n<p>Last step is <a href=\"https:\/\/developer.salesforce.com\/docs\/atlas.en-us.api_asynch.meta\/api_asynch\/query_get_job_results.htm\" target=\"_blank\" rel=\"noopener\">read the result<\/a> from completed Salesforce Job. Its in CSV format so we will use Advanced\u00a0<a href=\"https:\/\/zappysys.com\/products\/ssis-powerpack\/ssis-csv-file-source-flat-file-web-api\/\" target=\"_blank\" rel=\"noopener\">SSIS CSV Source<\/a> which supports API connection to Salesforce. For that, we will use the same Salesforce API Task used before.<\/p>\n<ol>\n<li>Drag and Drop SSIS Data Flow Task from SSIS Toolbox.<br \/>\n<img decoding=\"async\" class=\"figureimage\" title=\"SSIS Data Flow Task - Drag and Drop\" src=\"https:\/\/zappysys.com\/onlinehelp\/ssis-powerpack\/scr\/images\/drag-and-drop-data-flow-task.png\" alt=\"SSIS Data Flow Task - Drag and Drop\" \/><\/li>\n<li>Double click on the DataFlow task to see DataFlow designer surface.<\/li>\n<li>Here, In Visual Studio, drag and drop the ZS CSV Source (Web API or File) in the design panel<br \/>\n<img decoding=\"async\" class=\"figureimage\" title=\"SSIS CSV Source (Web API or File) - Drag and Drop\" src=\"https:\/\/zappysys.com\/onlinehelp\/ssis-powerpack\/scr\/images\/csv-source\/ssis-csv-file-source-drag.png\" alt=\"SSIS CSV Source (Web API or File) - Drag and Drop\" \/><\/li>\n<li>Double click on ZS CSV Source (Web API or File) Configure it like below.<br \/>\n&#8211; Enter URL to read result like this (You can supply OPTIONAL\u00a0maxRecords parameters to control how much data comes in a single response)<\/p>\n<pre class=\"\">https:\/\/{your-instance}.salesforce.com\/services\/data\/v50.0\/jobs\/query\/{{User::JobId}}\/results\r\n--OR--\r\nhttps:\/\/{your-instance}.salesforce.com\/services\/data\/v50.0\/jobs\/query\/{{User::JobId}}\/results?maxRecords=50000<\/pre>\n<p>&#8211; Check Use Credentials and select Salesforce Connection<br \/>\n&#8211; Click Preview (assuming JobId variable has a valid completed JobId)<\/p>\n<div id=\"attachment_9520\" style=\"width: 689px\" class=\"wp-caption alignnone\"><a href=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/read-salesforce-data-bulk-api-csv-result.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-9520\" class=\"size-full wp-image-9520\" src=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/read-salesforce-data-bulk-api-csv-result.png\" alt=\"Read Salesforce Bulk API Job Result - Using SSIS CSV \/ API Source\" width=\"679\" height=\"663\" srcset=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/read-salesforce-data-bulk-api-csv-result.png 679w, https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/read-salesforce-data-bulk-api-csv-result-300x293.png 300w\" sizes=\"(max-width: 679px) 100vw, 679px\" \/><\/a><p id=\"caption-attachment-9520\" class=\"wp-caption-text\">Read Salesforce Bulk API Job Result &#8211; Using SSIS CSV \/ API Source<\/p><\/div><\/li>\n<li>Now, set up Pagination as below to read all records using an automated loop. You will need <strong>VERSION Released after Jun 20, 2022<\/strong> to use Regular Expression feature in Header Value extractionBy default, Salesforce doesn&#8217;t send all records in response so we <a href=\"https:\/\/zappysys.com\/blog\/ssis-rest-api-looping-until-no-more-pages-found\/#Method-7_REST_APIPagination_using_Cursor_Continuation_Token_from_Response_Header\" target=\"_blank\" rel=\"noopener\">need to setup pagination<\/a>. In below example we are reading <strong>Sforce-Locator<\/strong> attribute from Response and sending it to the next URL as <strong>locator=&lt;continue-token&gt;<\/strong> value<br \/>\n&#8211; Choose <strong>Response Header Contains Continue Token\u00a0<\/strong><br \/>\n&#8211; Enter below expression in <strong>Page Num Indicator<\/strong><\/p>\n<pre class=\"lang:default highlight:0 decode:true\">locator=Sforce-Locator(^((?!null\\b).)*$)<\/pre>\n<p>Basically, the above expression will keep looping until we find <strong>Sforce-Locator : null<\/strong> in the response header.<\/p>\n<div id=\"attachment_9723\" style=\"width: 872px\" class=\"wp-caption alignnone\"><a href=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/salesforce-bulk-query-api-pagination-read-csv-data.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-9723\" class=\"size-full wp-image-9723\" src=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/salesforce-bulk-query-api-pagination-read-csv-data.png\" alt=\"Pagination for Salesforce Bulk Query API (Get Header Value from Sforce-Locator)\" width=\"862\" height=\"523\" srcset=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/salesforce-bulk-query-api-pagination-read-csv-data.png 862w, https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/salesforce-bulk-query-api-pagination-read-csv-data-300x182.png 300w, https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/salesforce-bulk-query-api-pagination-read-csv-data-768x466.png 768w\" sizes=\"(max-width: 862px) 100vw, 862px\" \/><\/a><p id=\"caption-attachment-9723\" class=\"wp-caption-text\">Pagination for Salesforce Bulk Query API (Get Header Value from Sforce-Locator)<\/p><\/div><\/li>\n<li>Click OK to save and now run entire package like below.<\/li>\n<\/ol>\n<div id=\"attachment_9515\" style=\"width: 671px\" class=\"wp-caption alignnone\"><a href=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/salesforce-query-bulk-api-read-data.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-9515\" class=\"size-full wp-image-9515\" src=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/salesforce-query-bulk-api-read-data.png\" alt=\"Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode \/ large dataset)\" width=\"661\" height=\"330\" srcset=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/salesforce-query-bulk-api-read-data.png 661w, https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/salesforce-query-bulk-api-read-data-300x150.png 300w\" sizes=\"(max-width: 661px) 100vw, 661px\" \/><\/a><p id=\"caption-attachment-9515\" class=\"wp-caption-text\">Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode \/ large dataset)<\/p><\/div>\n<h2>Conclusion<\/h2>\n<p>In this article we saw how to use various ZappySys Components \/ Task to achieve complex Salesforce API calls to read Query data using Bulk API. Download <a href=\"https:\/\/zappysys.com\/products\/ssis-powerpack\/\" target=\"_blank\" rel=\"noopener\">SSIS PowerPack<\/a>\u00a0and try yourself and also explore 70+ other tasks \/components not discussed in this article.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction In our previous post, we saw how to read Salesforce data using SSIS Salesforce Source (Which uses SOAP API under the hood). This may be slow if you have millions of rows because Salesforce Source paginates every 2000 rows, so it&#8217;s not a good option to read millions of rows to iterate on a [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":11114,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[447,209,466],"tags":[591,160,161,95,210],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.3 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\r\n<title>How to query salesforce data using Bulk API in SSIS | ZappySys Blog<\/title>\r\n<meta name=\"description\" content=\"How to query salesforce data using Bulk API in SSIS - ZappySys Blog\" \/>\r\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\r\n<link rel=\"canonical\" href=\"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/\" \/>\r\n<meta property=\"og:locale\" content=\"en_US\" \/>\r\n<meta property=\"og:type\" content=\"article\" \/>\r\n<meta property=\"og:title\" content=\"How to query salesforce data using Bulk API in SSIS | ZappySys Blog\" \/>\r\n<meta property=\"og:description\" content=\"How to query salesforce data using Bulk API in SSIS - ZappySys Blog\" \/>\r\n<meta property=\"og:url\" content=\"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/\" \/>\r\n<meta property=\"og:site_name\" content=\"ZappySys Blog\" \/>\r\n<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/ZappySys\/\" \/>\r\n<meta property=\"article:published_time\" content=\"2022-01-20T19:07:09+00:00\" \/>\r\n<meta property=\"article:modified_time\" content=\"2024-06-30T19:10:21+00:00\" \/>\r\n<meta property=\"og:image\" content=\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-bulk-api-option-soql-select-query.png\" \/>\r\n\t<meta property=\"og:image:width\" content=\"677\" \/>\r\n\t<meta property=\"og:image:height\" content=\"585\" \/>\r\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\r\n<meta name=\"author\" content=\"ZappySys\" \/>\r\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\r\n<meta name=\"twitter:creator\" content=\"@https:\/\/twitter.com\/zappysys\/\" \/>\r\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"ZappySys\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"10 minutes\" \/>\r\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/\",\"url\":\"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/\",\"name\":\"How to query salesforce data using Bulk API in SSIS | ZappySys Blog\",\"isPartOf\":{\"@id\":\"https:\/\/zappysys.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-bulk-api-option-soql-select-query.png\",\"datePublished\":\"2022-01-20T19:07:09+00:00\",\"dateModified\":\"2024-06-30T19:10:21+00:00\",\"author\":{\"@id\":\"https:\/\/zappysys.com\/blog\/#\/schema\/person\/2756c237457fbc95d82cb38962f81f82\"},\"description\":\"How to query salesforce data using Bulk API in SSIS - ZappySys Blog\",\"breadcrumb\":{\"@id\":\"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/#primaryimage\",\"url\":\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-bulk-api-option-soql-select-query.png\",\"contentUrl\":\"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-bulk-api-option-soql-select-query.png\",\"width\":677,\"height\":585,\"caption\":\"Bulk API mode for SSIS Salesforce Source (Read Large amount of data using SOQL Query)\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/zappysys.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to query salesforce data using Bulk API in SSIS\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/zappysys.com\/blog\/#website\",\"url\":\"https:\/\/zappysys.com\/blog\/\",\"name\":\"ZappySys Blog\",\"description\":\"SSIS \/ ODBC Drivers \/ API Connectors for JSON, XML, Azure, Amazon AWS, Salesforce, MongoDB and more\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/zappysys.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/zappysys.com\/blog\/#\/schema\/person\/2756c237457fbc95d82cb38962f81f82\",\"name\":\"ZappySys\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/zappysys.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/5c9be148088ba9b8af8e955c5f7c22b5?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/5c9be148088ba9b8af8e955c5f7c22b5?s=96&d=mm&r=g\",\"caption\":\"ZappySys\"},\"sameAs\":[\"http:\/\/www.zappysys.com\/\",\"https:\/\/www.facebook.com\/ZappySys\/\",\"https:\/\/twitter.com\/https:\/\/twitter.com\/zappysys\/\"],\"url\":\"https:\/\/zappysys.com\/blog\/author\/admin\/\"}]}<\/script>\r\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to query salesforce data using Bulk API in SSIS | ZappySys Blog","description":"How to query salesforce data using Bulk API in SSIS - ZappySys Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/","og_locale":"en_US","og_type":"article","og_title":"How to query salesforce data using Bulk API in SSIS | ZappySys Blog","og_description":"How to query salesforce data using Bulk API in SSIS - ZappySys Blog","og_url":"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/","og_site_name":"ZappySys Blog","article_author":"https:\/\/www.facebook.com\/ZappySys\/","article_published_time":"2022-01-20T19:07:09+00:00","article_modified_time":"2024-06-30T19:10:21+00:00","og_image":[{"width":677,"height":585,"url":"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-bulk-api-option-soql-select-query.png","type":"image\/png"}],"author":"ZappySys","twitter_card":"summary_large_image","twitter_creator":"@https:\/\/twitter.com\/zappysys\/","twitter_misc":{"Written by":"ZappySys","Est. reading time":"10 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/","url":"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/","name":"How to query salesforce data using Bulk API in SSIS | ZappySys Blog","isPartOf":{"@id":"https:\/\/zappysys.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/#primaryimage"},"image":{"@id":"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/#primaryimage"},"thumbnailUrl":"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-bulk-api-option-soql-select-query.png","datePublished":"2022-01-20T19:07:09+00:00","dateModified":"2024-06-30T19:10:21+00:00","author":{"@id":"https:\/\/zappysys.com\/blog\/#\/schema\/person\/2756c237457fbc95d82cb38962f81f82"},"description":"How to query salesforce data using Bulk API in SSIS - ZappySys Blog","breadcrumb":{"@id":"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/#primaryimage","url":"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-bulk-api-option-soql-select-query.png","contentUrl":"https:\/\/zappysys.com\/blog\/wp-content\/uploads\/2022\/01\/ssis-salesforce-bulk-api-option-soql-select-query.png","width":677,"height":585,"caption":"Bulk API mode for SSIS Salesforce Source (Read Large amount of data using SOQL Query)"},{"@type":"BreadcrumbList","@id":"https:\/\/zappysys.com\/blog\/query-salesforce-data-using-bulk-api-ssis\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/zappysys.com\/blog\/"},{"@type":"ListItem","position":2,"name":"How to query salesforce data using Bulk API in SSIS"}]},{"@type":"WebSite","@id":"https:\/\/zappysys.com\/blog\/#website","url":"https:\/\/zappysys.com\/blog\/","name":"ZappySys Blog","description":"SSIS \/ ODBC Drivers \/ API Connectors for JSON, XML, Azure, Amazon AWS, Salesforce, MongoDB and more","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/zappysys.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/zappysys.com\/blog\/#\/schema\/person\/2756c237457fbc95d82cb38962f81f82","name":"ZappySys","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/zappysys.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/5c9be148088ba9b8af8e955c5f7c22b5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/5c9be148088ba9b8af8e955c5f7c22b5?s=96&d=mm&r=g","caption":"ZappySys"},"sameAs":["http:\/\/www.zappysys.com\/","https:\/\/www.facebook.com\/ZappySys\/","https:\/\/twitter.com\/https:\/\/twitter.com\/zappysys\/"],"url":"https:\/\/zappysys.com\/blog\/author\/admin\/"}]}},"_links":{"self":[{"href":"https:\/\/zappysys.com\/blog\/wp-json\/wp\/v2\/posts\/9507"}],"collection":[{"href":"https:\/\/zappysys.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/zappysys.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/zappysys.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/zappysys.com\/blog\/wp-json\/wp\/v2\/comments?post=9507"}],"version-history":[{"count":9,"href":"https:\/\/zappysys.com\/blog\/wp-json\/wp\/v2\/posts\/9507\/revisions"}],"predecessor-version":[{"id":11116,"href":"https:\/\/zappysys.com\/blog\/wp-json\/wp\/v2\/posts\/9507\/revisions\/11116"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/zappysys.com\/blog\/wp-json\/wp\/v2\/media\/11114"}],"wp:attachment":[{"href":"https:\/\/zappysys.com\/blog\/wp-json\/wp\/v2\/media?parent=9507"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/zappysys.com\/blog\/wp-json\/wp\/v2\/categories?post=9507"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/zappysys.com\/blog\/wp-json\/wp\/v2\/tags?post=9507"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}