Jira Connector for Azure Data Factory (Pipeline) How to Read Issues

Introduction

In this article we will delve deeper into Jira and Azure Data Factory (Pipeline) integration, and will learn how to read issues. We are continuing from where we left off. By this time, you must have installed ODBC PowerPack, created ODBC Data Source, and configured authentication settings in your Jira account .

So, let's not waste time and begin.

Use Query Builder to generate SQL query

The first thing you have to do is open Query Builder:

ZappySys API Driver - Jira

Read and write Jira data effortlessly. Track, manage, and automate issues, projects, worklogs, and comments — almost no coding required.

JiraDSN
Then simply select the Read Issues endpoint (action).
Continue by configuring the Required parameters. You can also set optional parameters too.
Move on by hitting Preview Data button to preview the results.

If you see the results you need, simply copy the generated query:

Read Issues

Optional Parameters
Fields	*all
Search by	JQL
Project Key(s) - Comma Separated (CBS,BRK)
JQL Expression (For Issue Search)	created >= 1970-01-01
Issue Key(s) or Ids - Comma Separated (CBS-10,PRA-13)
CustomColumnsRegex
Metadata Mode (Change if columns missing) SLOW
Advanced Properties
PagingMaxRowsExpr	$.total
PagingMaxRowsDataPathExpr	$.issues[*]
Continue on Error String	False
Error String To Match	does not exist
PagingMode	ByResponseAttribute
NextUrlAttributeOrExpr	$.nextPageToken
NextUrlSuffix	nextPageToken=<%nextlink%>

SELECT * FROM Issues

--//Query single issue by numeric Issue Id
--SELECT * FROM Issues Where Id=101234

--//Query issue by numeric Issue Ids (multiple)
--SELECT * FROM Issues WITH(SearchBy='Key', Key='101234,101235,101236')

--//Query issue by Issue Key(s) (alpha-numeric)
--SELECT * FROM Issues WITH(SearchBy='Key', Key='PROJ-11')
--SELECT * FROM Issues WITH(SearchBy='Key', Key='PROJ-11,PROJ-12,PROJ-13')

--//Query issue by project(s)
--SELECT * FROM Issues WITH(SearchBy='Project', Project='PROJ')
--SELECT * FROM Issues WITH(SearchBy='Project', Project='PROJ,KAN,CS')

--//Query issue by JQL expression
--SELECT * FROM Issues WITH(SearchBy='Jql', Jql='status IN (Done, Closed) AND created > -5d' )

That's it! You can use this query in Azure Data Factory (Pipeline).

Let's not stop here and explore SQL query examples, including how to use them in Stored Procedures and Views (virtual tables) in the next steps.

SQL query examples

Use these SQL queries in your Azure Data Factory (Pipeline) data source:

How to List issues

Lists all issues

SELECT * FROM Issues

--//Query single issue by numeric Issue Id
--SELECT * FROM Issues Where Id=101234

--//Query issue by numeric Issue Ids (multiple)
--SELECT * FROM Issues WITH(SearchBy='Key', Key='101234,101235,101236')

--//Query issue by Issue Key(s) (alpha-numeric)
--SELECT * FROM Issues WITH(SearchBy='Key', Key='PROJ-11')
--SELECT * FROM Issues WITH(SearchBy='Key', Key='PROJ-11,PROJ-12,PROJ-13')

--//Query issue by project(s)
--SELECT * FROM Issues WITH(SearchBy='Project', Project='PROJ')
--SELECT * FROM Issues WITH(SearchBy='Project', Project='PROJ,KAN,CS')

--//Query issue by JQL expression
--SELECT * FROM Issues WITH(SearchBy='Jql', Jql='status IN (Done, Closed) AND created > -5d' )

How to List a single issue by Id

List a single issue by Key (e.g. CS-123) or Numeric Id (e.g. 10001).

SELECT * FROM Issues WITH(SearchBy='Key', Key='10001')

How to List a single issue by Id - Continue on a specific error message

By default if issue is not found or search condition is bad you may get an error but you can continue by setting ContineOnErrorForMessage=1 and message you like to ignore in ErrorSubstringToMatch.

SELECT * FROM Issues WITH(SearchBy='Key', Key='10001', ContineOnErrorForMessage=1, ErrorSubstringToMatch='Issue does not exist')

How to List a single issue by Key

List a single issue by Key (e.g. CS-123) or Numeric Id (e.g. 10001).

SELECT * FROM Issues WITH(SearchBy='Key', Key='CS-1')

How to List multiple issues by Id or Key

List multiple issues by comma separated Key(s) or Numeric Id(s).

SELECT * FROM Issues WITH(SearchBy='Key', Key='CS-1,CS-2,10003,10004')

How to List all issues for a specific project

List all issues for a specified project code. To query multiple project you can use comma separated list. Example: SELECT * FROM Issues WITH(SearchBy='Project', Project='PROJ1,PROJ2,PROJ3')

SELECT * FROM Issues WITH(SearchBy='Project', Project='CS')

How to List issues (fetch specific fields only rather than all)

Lists all issues and fetch only specified fields rather than all fields (useful to speed up data fetch if you only need handful fields)

SELECT * FROM Issues WITH(Fields='id,key,summary,status')

How to Search issues using Advanced JQL query expression

List issues using JQL query expression

SELECT * FROM Issues WITH (Jql='status IN (Done, Closed) AND created > -5d' )
/*
Useful links:
  https://support.atlassian.com/jira-work-management/docs/jql-fields/
  https://www.atlassian.com/software/jira/guides/jql/tutorials#advanced-search
  https://www.atlassian.com/blog/jira/jql-the-most-flexible-way-to-search-jira-14
  
Other Possible JQL expressions:

Ids (IN):       key IN(10001, 10002, 10003);
Keys (IN):       key IN(CS-1, CS-2, CS-3);
Projects (IN):       project IN(PROJ1, PROJ2, PROJ3);
Status (EQUAL):       status='Done';
Date (Expression 1):       created >=-5d;
Date (Expression 2):       created >=startOfMonth() AND created <=now();
Date (Expression 3):       created >=startOfYear() AND created <=startOfDay();
Date (Static):       created >= '2008/12/31';
Date (Static with time):       created >= '2008/12/31 23:59';
Project and Status (AND + IN):       project=CS AND status NOT IN ('Done', 'Open', 'Closed');
Assignee and Created Date:    assignee is NOT EMPTY and created < -1d;
Text (Contains - Fuzzy):       Summary ~ 'some words' OR description ~ 'some words';
Text (Contains - Fuzzy Wildcard):       Summary ~ 'some*' OR description ~ 'some*';
Text (Contains - Exact):       Summary ~ '\"exact words\"' OR description ~ '\"exact words\"';
Text (Does Not Contain - Fuzzy):       Summary !~ 'some words' OR description !~ 'some words';
Empty OR Null:   fixVersion is empty OR fixVersion is null;
Is Not Empty OR Is Not Null:    fixVersion is not empty OR fixVersion is not null;
WAS Operator (previous value):       status WAS "Resolved" BY (jsmith,srogen) BEFORE "2019/02/02";
WAS IN Operator:   status WAS IN ("Resolved", "In Progress");
WAS NOT IN Operator:    status WAS NOT IN ("Resolved", "In Progress");
WAS + BY + DURING (date range):       status WAS "Resolved" BY (jsmith,srogen) DURING("2019/02/02", "2020/02/02");
CHANGED operator:    assignee CHANGED;
CHANGED operator (multiple):       status CHANGED FROM "In Progress" TO "Open"

*/

How to Query Issues with All Columns (Including SLA / Nested Custom Fields)—No META

Simpler approach: set Metadata Mode to MergeStaticDynamic so the driver scans sample data and exposes all columns (static + dynamic). No META parameter needed. Slower due to extra requests used to guess column types from data.

-- No META needed. Set MetaDetectionOrder to MergeStaticDynamic so nested/SLA columns are discovered automatically.
-- Slower: driver makes extra requests to scan data and merge with static metadata.

SELECT *
FROM Issues
WITH(
	  "JQL"='project IN(SUP)'
	  ,"MetaDetectionOrder"='MergeStaticDynamic'
)

How to Query Custom Fields with Nested Document Structures (SLA Fields)

This example shows how to query custom fields that contain nested document structures (not arrays), such as SLA fields. Use @OverrideMode:1 to merge static and dynamic metadata, allowing you to query both standard fields and nested custom field properties.

-- NOTE: Replace customfield_10084, customfield_10085, customfield_10086 with your own custom field IDs.
-- You can run [SELECT * FROM Fields] in preview tab to get the field IDs. 
-- OR you can find field IDs in Jira: Project Settings > Issue types > Edit field > field ID in URL or API.

SELECT 
-- static fields

  id
, key
, summary
, statusname

-- dynamic fields

-- SLA: Time to Resolution (replace customfield_10084 with your SLA field ID) 

,[fields.customfield_10084.id] as c10084_id
,[fields.customfield_10084.name] as c10084_name
,[fields.customfield_10084.ongoingCycle.startTime.friendly] as c10084_startTime_friendly
,[fields.customfield_10084.ongoingCycle.startTime.jira] as c10084_startTime_jira
,[fields.customfield_10084.ongoingCycle.breachTime.friendly] as c10084_breachTime_friendly
,[fields.customfield_10084.ongoingCycle.breachTime.jira] as c10084_breachTime_jira
,[fields.customfield_10084.ongoingCycle.elapsedTime.friendly] as c10084_elapsedTime_friendly
,[fields.customfield_10084.ongoingCycle.remainingTime.friendly] as c10084_remainingTime_friendly
,[fields.customfield_10084.ongoingCycle.goalDuration.friendly] as c10084_goalDuration_friendly
,[fields.customfield_10084.ongoingCycle.breached] as c10084_breached
,[fields.customfield_10084.ongoingCycle.paused] as c10084_paused

-- SLA: Time to First Response (replace customfield_10085 with your SLA field ID)

,[fields.customfield_10085.id] as c10085_id
,[fields.customfield_10085.name] as c10085_name
,[fields.customfield_10085.ongoingCycle.startTime.friendly] as c10085_startTime_friendly
,[fields.customfield_10085.ongoingCycle.startTime.jira] as c10085_startTime_jira
,[fields.customfield_10085.ongoingCycle.breachTime.friendly] as c10085_breachTime_friendly
,[fields.customfield_10085.ongoingCycle.breachTime.jira] as c10085_breachTime_jira
,[fields.customfield_10085.ongoingCycle.elapsedTime.friendly] as c10085_elapsedTime_friendly
,[fields.customfield_10085.ongoingCycle.remainingTime.friendly] as c10085_remainingTime_friendly
,[fields.customfield_10085.ongoingCycle.goalDuration.friendly] as c10085_goalDuration_friendly
,[fields.customfield_10085.ongoingCycle.breached] as c10085_breached
,[fields.customfield_10085.ongoingCycle.paused] as c10085_paused

-- SLA: Time to Close After Resolution (replace customfield_10086 with your SLA field ID)

,[fields.customfield_10086.id] as c10086_id
,[fields.customfield_10086.name] as c10086_name

-- Other nested custom fields (replace IDs as needed) 
,[fields.customfield_10024.id] as c10024_id
,[fields.customfield_10024.name] as c10024_name
,[fields.customfield_10075.languageCode] as c10075_languageCode
,[fields.customfield_10075.displayName] as c10075_displayName

FROM Issues
WITH(
	  "JQL"='project IN(SUP)'
	  -- @OverrideMode:1 merges metadata so you can use static + dynamic fields together
	  ,META='@OverrideMode:1
	  ;fields.customfield_10084.id : string(10)
	  ;fields.customfield_10084.name : string(180)
	  ;fields.customfield_10084.ongoingCycle.startTime.friendly : string(130)
	  ;fields.customfield_10084.ongoingCycle.startTime.jira : DateTime
	  ;fields.customfield_10084.ongoingCycle.breachTime.friendly : string(130)
	  ;fields.customfield_10084.ongoingCycle.breachTime.jira : DateTime
	  ;fields.customfield_10084.ongoingCycle.elapsedTime.friendly : string(30)
	  ;fields.customfield_10084.ongoingCycle.remainingTime.friendly : string(70)
	  ;fields.customfield_10084.ongoingCycle.goalDuration.friendly : string(30)
	  ;fields.customfield_10084.ongoingCycle.breached : Boolean
	  ;fields.customfield_10084.ongoingCycle.paused : Boolean
	  ;fields.customfield_10085.id : string(10)
	  ;fields.customfield_10085.name : string(220)
	  ;fields.customfield_10085.ongoingCycle.startTime.friendly : string(130)
	  ;fields.customfield_10085.ongoingCycle.startTime.jira : DateTime
	  ;fields.customfield_10085.ongoingCycle.breachTime.friendly : string(130)
	  ;fields.customfield_10085.ongoingCycle.breachTime.jira : DateTime
	  ;fields.customfield_10085.ongoingCycle.elapsedTime.friendly : string(30)
	  ;fields.customfield_10085.ongoingCycle.remainingTime.friendly : string(70)
	  ;fields.customfield_10085.ongoingCycle.goalDuration.friendly : string(30)
	  ;fields.customfield_10085.ongoingCycle.breached : Boolean
	  ;fields.customfield_10085.ongoingCycle.paused : Boolean
	  ;fields.customfield_10086.id : string(10)
	  ;fields.customfield_10086.name : string(300)
	  ;fields.customfield_10024.id : string(40)
	  ;fields.customfield_10024.name : string(80)
	  ;fields.customfield_10075.languageCode : string(20)
	  ;fields.customfield_10075.displayName : string(70)
	  ;--add more nested custom field properties here as needed--
	  '
)

get_issues endpoint belongs to Issues table(s), and can therefore be used via those table(s).

Stored Procedures and Views

Create Custom Stored Procedure

You can create procedures to encapsulate custom logic and then only pass handful parameters rather than long SQL to execute your API call.

Steps to create Custom Stored Procedure in ZappySys Driver. You can insert Placeholders anywhere inside Procedure Body. Read more about placeholders here

Go to Custom Objects Tab and Click on Add button and Select Add Procedure:
Enter the desired Procedure name and click on OK:
Select the created Stored Procedure and write the your desired stored procedure and Save it and it will create the custom stored procedure in the ZappySys Driver:
Here is an example stored procedure for ZappySys Driver. You can insert Placeholders anywhere inside Procedure Body. Read more about placeholders here
```
CREATE PROCEDURE [usp_get_orders]
    @fromdate = '<<yyyy-MM-dd,FUN_TODAY>>'
 AS
    SELECT * FROM Orders where OrderDate >= '<@fromdate>';
```
That's it now go to Preview Tab and Execute your Stored Procedure using Exec Command. In this example it will extract the orders from the date 1996-01-01:
```
Exec usp_get_orders '1996-01-01';
```

Create Custom Virtual Table

ZappySys API Drivers support flexible Query language so you can override Default Properties you configured on Data Source such as URL, Body. This way you don't have to create multiple Data Sources if you like to read data from multiple EndPoints. However not every application support supplying custom SQL to driver so you can only select Table from list returned from driver.

If you're dealing with Microsoft Access and need to import data from an SQL query, it's important to note that Access doesn't allow direct import of SQL queries. Instead, you can create custom objects (Virtual Tables) to handle the import process.

Many applications like MS Access, Informatica Designer wont give you option to specify custom SQL when you import Objects. In such case Virtual Table is very useful. You can create many Virtual Tables on the same Data Source (e.g. If you have 50 URLs with slight variations you can create virtual tables with just URL as Parameter setting.

Go to Custom Objects Tab and Click on Add button and Select Add Table:
Enter the desired Table name and click on OK:
And it will open the New Query Window Click on Cancel to close that window and go to Custom Objects Tab.
Select the created table, Select Text Type AS SQL and write the your desired SQL Query and Save it and it will create the custom table in the ZappySys Driver:
Here is an example SQL query for ZappySys Driver. You can insert Placeholders also. Read more about placeholders here
```
SELECT
  "ShipCountry",
  "OrderID",
  "CustomerID",
  "EmployeeID",
  "OrderDate",
  "RequiredDate",
  "ShippedDate",
  "ShipVia",
  "Freight",
  "ShipName",
  "ShipAddress",
  "ShipCity",
  "ShipRegion",
  "ShipPostalCode"
FROM "Orders"
Where "ShipCountry"='USA'
```
That's it now go to Preview Tab and Execute your custom virtual table query. In this example it will extract the orders for the USA Shipping Country only:
```
SELECT * FROM "vt__usa_orders_only"
```

Read Issues in Azure Data Factory (Pipeline)

Sign in to Azure Portal
- Open your browser and go to: https://portal.azure.com
- Enter your Azure credentials and complete MFA if required.
- After login, go to Data factories.
Under Azure Data Factory Resource - Create or select the Data Factory you want to work with.
Inside the Data Factory resource page, click Launch studio.
Create a New Linked service:
- In the Manage section (left menu).
- Under Connections, select Linked services.
- Click + New to create a new Linked service based on ODBC.
Select ODBC service:
Configure new ODBC service. Use the same DSN name we used in the previous step and copy it to Connection string box:

JiraDSN

DSN=JiraDSN
For created ODBC service create ODBC-based dataset:
Go to your pipeline and add Copy data connector into the flow. In Source section use OdbcDataset we created as a source dataset:
Then go to Sink section and select a destination/sink dataset. In this example we use precreated AzureBlobStorageDataset which saves data into an Azure Blob:
Finally, run the pipeline and see data being transferred from OdbcDataset to your destination dataset: