Introduction
BIML or Business Intelligence Markup Language is a nice technology that lets you generate SSIS packages using a bit of scripting and parameters. Check for more info if you are new to it:
In this article we will cover the points how to create BIML custom task or component. For this purpose we will use ZappySys PowerPack Task or Data Flow component as an example. At the end of this article you will find sample BIML files you can download and fiddle around.
Use BIML to create SSIS packages that use PowerPack tasks or components
BIML Elements
BIML uses CustomTask, CustomComponent, and CustomSsisConnection elements to create custom SSIS connectors. In each of them you will need to specify CreationName or a similar attribute/element, which defines what kind of component you want to create. E.g. for LoggingTask it would look like this:
1 |
ZappySys.PowerPack.Task.LoggingTask.LoggingTask, ZappySys.PowerPack120, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502 |
You will find the connector name in DTSX. For Tasks you have to specify CreationName attribute, for components it is UserComponentTypeName. For Rest API Task it looks similar to this:
1 2 3 |
<DTS:Executable DTS:refId="Package\Sequence Container\ZS REST API Task 1.EventHandlers[OnError]\ZS REST API Task" DTS:CreationName="ZappySys.PowerPack.Task.RestApiTask.RestApiTask, ZappySys.PowerPack110, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502" |
For different SQL Server SSIS package version you will need to use a different DLL name, e.g.:
- SQL Server 2008: ZappySys.PowerPack100
- SQL Server 2012: ZappySys.PowerPack110
- SQL Server 2014: ZappySys.PowerPack120
- SQL Server 2016: ZappySys.PowerPack130
- SQL Server 2017: ZappySys.PowerPack140
So to generate SQL Server 2017 SSIS package custom connector creation name becomes like this:
1 |
ZappySys.PowerPack.Task.LoggingTask.LoggingTask, ZappySys.PowerPack140, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502 |
Create BIML custom task with CustomTask
You will need to use CustomTask element to create PowerPack Tasks in your Control Flow. So open your DTSX where you use a PowerPack Task, find out the CreationName and then copy child element in <DTS:ObjectData> element. (e.g. LoggingTask). You will need to escape it and use it in your BIML as well.
DTSX snippet
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
... <DTS:Executable DTS:refId="Package\ZS Logging Task" DTS:CreationName="ZappySys.PowerPack.Task.LoggingTask.LoggingTask, ZappySys.PowerPack110, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502" ... <DTS:Variables /> <DTS:ObjectData> <LoggingTask Message="Hello from {{System::PackageName}}. I started on {{System::StartTime,yyy-MM-dd hh:mm:ss tt}}" LogVariableName="" MessageType="Information" LogMode="ExecutionLog" LogFilePath="" LogFileAppend="True" LogDateTime="True" DateTimeFormat="HH:mm:ss" LogLevel="Normal" PrefixTimestamp="False" /> </DTS:ObjectData> </DTS:Executable> ... |
BIML snippet
A Logging Task under Tasks element will look like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
... <CustomTask Name="Logging Task" CreationName="ZappySys.PowerPack.Task.LoggingTask.LoggingTask, ZappySys.PowerPack120, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502" TaskContact="ZappySys"> <ObjectData> <LoggingTask Message="Hello from {{System::PackageName}}. I started on {{System::StartTime,yyy-MM-dd hh:mm:ss tt}}" LogVariableName="" MessageType="Information" LogMode="ExecutionLog" LogFilePath="" LogFileAppend="True" LogDateTime="True" DateTimeFormat="HH:mm:ss" LogLevel="Normal" PrefixTimestamp="False" /> </ObjectData> </CustomTask> ... |
Create BIML custom components using CustomComponent
Use CustomComponent BIML element to generate these components in DataFlow:
- Sources
- Transformations
- Destinations
If you need a connection then use CustomSsisConnection element.
Again, you will have to copy UserComponentTypeName property value and use it in BIML to initialize a component. For custom connection it is a bit different — just take CreationName attribute value from DTSX and use it in BIML element. Also, don’t forget to copy property values from DTSX and mirror them in BIML.
Below is DTSX and BIML snippets where a Data Flow contains JSON Source which uses HTTP Connection and is connected to Trash Destination.
DTSX snippet
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 |
... <DTS:ConnectionManagers> <DTS:ConnectionManager DTS:refId="Package.ConnectionManagers[ZS-HTTP Connection]" DTS:CreationName="ZS-HTTP" DTS:DTSID="{E57D23E9-E50A-4C4A-BB19-B91E776887CF}" DTS:ObjectName="ZS-HTTP Connection"> <DTS:ObjectData> <ConnectionManager ConnectionString="Url=https://httpbin.org/anything;CredentialType=Notset;UserName=;Timeout=100000;UseProxy=False;ProxyUrl=;UseProxyCreds=False;ProxyUserName=;DoNotUseDefaultProxy=False;UseClientCertificate=False;CertificatePath=;CertificateStorageType=CurrentUser;CertificateStoreName=My;CertificateThumbprint=;RetryMode=None;RetryCountMax=3;RetryWaitTimeMs=2000;RetryMultiplyWaitTime=False;RetryMessageList=;RetryStatusCodeList=" /> ... <DTS:Executable DTS:refId="Package\Dataflow" DTS:CreationName="SSIS.Pipeline.3" DTS:DTSID="{C59866C4-9732-4B3A-8F80-7784E52C63F5}" DTS:ExecutableType="SSIS.Pipeline.3" DTS:LocaleID="-1" DTS:ObjectName="Dataflow" DTS:TaskContact="Performs high-performance data extraction, transformation and loading;Microsoft Corporation; Microsoft SQL Server v10; (C) 2007 Microsoft Corporation; All Rights Reserved;http://www.microsoft.com/sql/support/default.asp;1"> <DTS:ObjectData> <pipeline BLOBTempStoragePath="" bufferTempStoragePath="" version="1"> <components> <component refId="Package\Dataflow\JsonSource" componentClassID="DTS.ManagedComponentWrapper.3" name="JsonSource" usesDispositions="true" version="25"> <properties> <property dataType="System.Int32" name="AccessMode" typeConverter="ZappySys.Common.Ssis.FileValueAccessMode, ZappySys.Common, Version=1.0.2017.10830, Culture=neutral, PublicKeyToken=1903265a93eae502">2</property> <property dataType="System.Boolean" name="UseConnection">true</property> <property dataType="System.String" name="UserComponentTypeName">ZappySys.PowerPack.Adapter.JsonSource.JsonSource, ZappySys.PowerPack110, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502</property> ... </properties> <connections> <connection refId="Package\Dataflow\JsonSource.Connections[HTTP 1]" connectionManagerID="Package.ConnectionManagers[ZS-HTTP Connection]" connectionManagerRefId="Package.ConnectionManagers[ZS-HTTP Connection]" name="HTTP 1" /> </connections> <outputs> <output refId="Package\Dataflow\JsonSource.Outputs[JsonSourceOutput]" errorRowDisposition="FailComponent" name="JsonSourceOutput" truncationRowDisposition="FailComponent"> <outputColumns> <outputColumn refId="Package\Dataflow\JsonSource.Outputs[JsonSourceOutput].Columns[Url]" dataType="wstr" externalMetadataColumnId="Package\Dataflow\JsonSource.Outputs[JsonSourceOutput].ExternalColumns[url]" length="50" lineageId="Package\Dataflow\JsonSource.Outputs[JsonSourceOutput].Columns[Url]" name="Url" /> </outputColumns> <externalMetadataColumns isUsed="True"> <externalMetadataColumn refId="Package\Dataflow\JsonSource.Outputs[JsonSourceOutput].ExternalColumns[url]" dataType="wstr" length="50" name="url" /> </externalMetadataColumns> </output> ... </outputs> </component> <component refId="Package\Dataflow\TrashDestination" componentClassID="DTS.ManagedComponentWrapper.3" name="TrashDestination" version="2"> <properties> <property dataType="System.String" name="UserComponentTypeName">ZappySys.PowerPack.Adapter.TrashDestination.TrashDestination, ZappySys.PowerPack110, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502</property> ... </properties> <inputs> <input refId="Package\Dataflow\TrashDestination.Inputs[TrashInput]" name="TrashInput"> <externalMetadataColumns /> </input> </inputs> </component> </components> <paths> <path refId="Package\Dataflow.Paths[TrashInput]" endId="Package\Dataflow\TrashDestination.Inputs[TrashInput]" name="TrashInput" startId="Package\Dataflow\JsonSource.Outputs[JsonSourceOutput]" /> </paths> ... |
BIML snippet
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
... <Connections> <CustomSsisConnection Name="ZS-HTTP Connection" CreationName="ZS-HTTP" ObjectData="<ConnectionManager ConnectionString="Url=https://httpbin.org/anything;CredentialType=Notset;UserName=;Timeout=100000;UseProxy=False;ProxyUrl=;UseProxyCreds=False;ProxyUserName=;DoNotUseDefaultProxy=False;UseClientCertificate=False;CertificatePath=;CertificateStorageType=CurrentUser;CertificateStoreName=My;CertificateThumbprint=;RetryMode=None;RetryCountMax=3;RetryWaitTimeMs=2000;RetryMultiplyWaitTime=False;RetryMessageList=;RetryStatusCodeList=" />"/> </Connections> <Packages> <Package Name="DataFlow with Connection Sample" ConstraintMode="Linear"> <Tasks> <Dataflow Name="Dataflow"> <Transformations> <CustomComponent Name="JsonSource" UsesDispositions="true" ComponentTypeName="ZappySys.PowerPack.Adapter.JsonSource.JsonSource, ZappySys.PowerPack130, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502"> <CustomProperties> <CustomProperty Name="AccessMode" DataType="Int32">2</CustomProperty> <CustomProperty Name="UseConnection" DataType="Boolean">true</CustomProperty> </CustomProperties> <OutputPaths> <OutputPath Name="JsonSourceOutput" ErrorRowDisposition="FailComponent" TruncationRowDisposition="FailComponent"> <ExternalColumns> <ExternalColumn Name="url" DataType="String" Length="50" /> </ExternalColumns> <OutputColumns> <OutputColumn Name="Url" ExternalMetadataColumnName="url" DataType="String" Length="50" /> </OutputColumns> </OutputPath> <OutputPath Name="Json Error Output" IsErrorOutput="true"> <OutputColumns> <OutputColumn Name="Url" /> </OutputColumns> </OutputPath> </OutputPaths> <Connections> <Connection Name="HTTP 1" ConnectionName="ZS-HTTP Connection" /> </Connections> </CustomComponent> <CustomComponent Name="TrashDestination" ComponentTypeName="ZappySys.PowerPack.Adapter.TrashDestination.TrashDestination, ZappySys.PowerPack130, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502"> <InputPaths> <InputPath OutputPathName="JsonSource.JsonSourceOutput" Identifier="TrashInput" SsisName="TrashInput"/> </InputPaths> </CustomComponent> </Transformations> ... |
Resources
- BIML documentation
- CustomTask documentation
- CustomComponent documentation
- CustomSsisConnection documentation
Download sample BIMLs
- Sample BIML with PowerPack DataFlow Components
- Sample BIML with PowerPack DataFlow Components with Connection
- Sample BIML with PowerPack Task