|
Amazon Storage Connection Manager
|
 
|
|
Amazon Storage Connection Manager can be used to connect to AWS Storage Services such as S3, DynamoDB, Redshift Cluster. You can supply your Accesskey and Secret Key (Or use Temporary credentials with Session Key) to connect to various services.
|
| Property Name | Description | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AccessKey | |||||||||||||
| UseDualStackEndPoint | |||||||||||||
| ForcePathStyle | This option is useful for S3 based services which are hosted on cloud or locally. By default /mybucket/myfile Path is translated to https://mybucket.some-service-host/myfile API call but this can cause issue for non AWS services (i.e. Mini.IO, Veritas, Cloudian, Ceph...). Setting ForcePathStyle=True will call APi like https://some-service-host/mybucket/myfile which is desired while calling 3rd party S3 Protocol compatible service. | ||||||||||||
| UseAcceleratedEndPoint | |||||||||||||
| MaxUploadSpeedKbps | Maximum upload speed in KB/second (0=Unlimited). When you enable this option only single thread will be used for larger filer uploads so only set this option if you really need to slow down uploads for good reason. | ||||||||||||
| DisableProxyAutoDetect | Check this option to turn off proxy detection. This may speed up web requests in some cases. | ||||||||||||
| ServiceUrl | |||||||||||||
| UseRegionSpecificEndpoint | Set this option to true if you connecting to specific region rather than US Standard. By default all requests made to US Standard region first and then it obtains correct region information for your request but in some rare case AWS server doesn't return correct region and it throws Bad Request error. Setting this option to true can help you to fix such errors. Leave this option to false if you are not sure about your region. | ||||||||||||
| Encrypt | If you want to enable encryption for content uploaded or downloaded to and from Amazon S3 then check this option. This is not valid for ServiceType=DynamoDB | ||||||||||||
| EncryptionAlgorithm | Algorithm for encryption. Supported algorithms are AES256, RSA1024 and KMS. If S3 connection is used for Redshift data transfer then only AES256 is supported.
|
||||||||||||
| EncryptionKeyFile | Path of secret encryption key file when EncryptionAlgorithm=RSA1024 or AES256. If EncryptionAlgorithm=KMS then this must be Key ID (Obtained from KMS Console). Key file content is in base64 format for AES and for RSA encryption key file content is XML format. You can open it in notepad and validate if key is in right form. | ||||||||||||
| ServiceType |
|
||||||||||||
| Region | |||||||||||||
| UseHttp | |||||||||||||
| SecretKey | |||||||||||||
| CredentialType |
|
||||||||||||
| SessionToken | |||||||||||||
| ProxyPassword | |||||||||||||
| ProxyHost | Proxy Server or IP (e.g. MyProxy01 --or-- 192.168.0.1 --or-- proxy1.mydomain.com) | ||||||||||||
| ProxyUser | UserName for proxy authentication | ||||||||||||
| ProxyPort | Proxy server port number | ||||||||||||
| ConnectionString | |||||||||||||
| ProgressUpdateInterval | Update interval in KB. System will report upload/download progress based on this setting. By default progress is reported every 1000 KB. | ||||||||||||
| Timeout | |||||||||||||
| MaxRetryOnError | |||||||||||||
| RetryInterval | |||||||||||||
| MaxParallelThreads |
|
Download the Latest File from Amazon S3 Storage using SSISIntroduction In our previous article, we saw SSIS Amazon S3 Storage Task examples. Now let’s look at how to Download the Latest File from Amazon S3 Storage using SSIS. Microsoft SSIS includes the Amazon S3 Storage Task that allows us to upload files to Amazon S3 Storage, download files, creating local and remote directories an more. In […] |
|
|
SSIS Amazon S3 Storage Task examples (Download, Upload, Delete Files / Folders)Introduction In this article, we will show SSIS Amazon S3 Storage task examples. Uploading files using Amazon S3 Storage Task or downloading files are common requirements in SSIS. In order to work with Amazon S3 Storage, Microsoft SSIS includes the Amazon S3 Storage Task that allows to upload files to Amazon S3 Storage, download files, creating […] |
|
Setting Metadata key / value in SSIS Amazon S3 / Azure Storage TaskIntroduction In this post we will see how to set Metadata key/value in Azure Blob Storage Task and Amazon Storage Task (S3 Task). We will also see how to set them dynamically at runtime (Use SSIS Expression). Set Amazon S3 Task Metadata (Key / Value Pairs) Here is how you can set Metadata in Amazon […] |
|
Copy Amazon S3 Files in SSIS to different AWS AccountIntroduction In our previous blog we saw how to upload data to Amazon S3 now let’s look at how to Copy Amazon Files from one AWS account to another AWS account (Server Side Copy) using SSIS Amazon Storage Task. (Must be version v2.7.9 or higher) If you are doing file copy within same account then there […] |
|
How to delete Amazon S3 File / Folder / Bucket in SSISIntroduction In our previous posts we looked at how to use SSIS Amazon Storage Task (Amazon S3 Task) to automate some Amazon S3 related tasks in SSIS. In this blog post will continue discovering more use cases. Let’s learn how to delete Amazon S3 File, delete Amazon S3 Folder and delete Amazon S3 Bucket. Deleting S3 Files […] |
|
Using Regular Expressions in SSISIntroduction In this short article you will learn how to write Regular expressions in SSIS (i.e. Regex) and what tool to use to test them. You will also find helpful resources on how to write more sophisticated expressions and learn more about them. For demo purpose we will use FREE SSIS Regex Parser Task to parse and […] |
|
Import / Export data from Amazon Athena using SSISIntroduction In our previous post we explored unlimited possibilities to call Amazon AWS API using SSIS. Now in this post we will learn how to import / export data from Amazon Athena using SSIS. Amazon Athena is very similar service like Google BigQuery which we documented already. To read data from Amazon Athena we will use […] |
|
Filter by file size, name, date for Amazon / Azure Storage SSIS TaskIntroduction Sometimes you may need to make an advanced filtering on file path, file date, size etc in SSIS Amazon S3 Task or SSIS Azure Blob Storage Task . For example, you have a directory “Customers“, which contains client directories. Each of them contains a directory named “Invoices“, which is partitioned by year. E.g. “Customers/Microsoft/Invoices/2015”. So […] |
|
SSIS – Copy Amazon S3 files from AWS to AzureIntroduction Azure and AWS both are most popular Cloud Platforms. In this blog post we will learn how to copy or move Amazon S3 files to Azure Blob Storage without any coding or scripting (AWS to Azure File Copy / Migration Scenario). To achieve this objective we will use following Drag and Drop SSIS Tasks (i.e. […] |
|
Loading data from SQL Server to Amazon S3 in SSIS (Split Files, GZip)Introduction In this blog post you will see how easy it is to load large amount of data from SQL Server to Amazon S3 Storage. For demo purpose we will use SQL Server as relational source but you can use same steps for any database engine such as Oracle, MySQL, DB2. In this post we […] |
