Export REST API to CSV is in some cases necessary to process the data because many tools can handle CSV files. In this new article, we will show different ways to export the data. The first example will do it using C#. The second example with use Python. Let’s take a look at these examples.

Requirements to export REST API to CSV

1. First of all, you will need ZappySys ODBC PowerPack installed.
2. Secondly, you will require Visual Studio installed for the C# example.
3. Finally, it is necessary Python installed for the Python example.

Export REST API to CSV using C#

C# is a pretty popular programing language. In the first example, we will show how to display the REST API information to a CSV file named. Let’s take a look at the code:

1. First of all, we will connect to REST API using a connection to the following Data Path:
   

   https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json

   using (OdbcConnection conn = 
         new OdbcConnection("Driver={ZappySys JSON Driver};DataPath='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json';"))

2. Secondly, we create a connection using a file stream. We will save the results to a file in the c:\sql\sample.csv:
   

   FileStream fs = new FileStream("C:\\sql\\sample.txt", FileMode.Create);
   StreamWriter writer = new StreamWriter(fs);
   StringBuilder output = new StringBuilder();

3. Also, we will add a query to REST API. Note that with the ZappySys ODBC PowerPack, you can do a simple SQL query to get REST API data. This is pretty simple and intuitive:
   

   OdbcCommand cmd = new OdbcCommand(
        @"SELECT  CustomerID,CompanyName FROM value", conn);

4. In addition, we will read the data and close the connections:
   

   conn.Close();  
   writer.Close(); 
   fs.Close();

5. The complete code will be the following:
   

   using System;
   using System.IO;
   using System.Data.Odbc;
   
   public class Program
   {
       public static void Main()
       {
           var outpath = @"C:\temp\sample.txt";
   
           using (var conn = new OdbcConnection("Driver={ZappySys JSON Driver};DataPath='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json';"))
           {
               conn.Open();
               var cmd = new OdbcCommand(@"SELECT CustomerID,CompanyName FROM $ WITH(Filter='$.value[*]')", conn);
               
               //Increases the timeout duration from the default 30 seconds, which may be insufficient in certain scenarios.
               cmd.CommandTimeout=600; // 600-seconds
               
               var rdr = cmd.ExecuteReader();
   
               using (var fs = new FileStream(outpath, FileMode.Create))
               {
                   using (var writer = new StreamWriter(fs))
                   {
                       //write file header
                       writer.WriteLine("CustomerID,CompanyName");
   
                       while (rdr.Read())
                       {
                           //write file row
                           writer.WriteLine("{0},{1}", rdr["CustomerID"], rdr["CompanyName"]);
                       }
                       conn.Close(); //close connection
                       writer.Close();
                       fs.Close();
                   }
               }
           }
   
           //Read from file and display the content
           Console.Write(File.ReadAllText(outpath));
   
           Console.WriteLine("\r\n===== Press any key to end the program =====\r\n");
           Console.Read();
       }
   }

    
6. Finally, you will be able to see the file created:
   

   REST API into CSV

Export REST API to CSV using Python

Python is another really popular programming language. The popularity is growing a lot. In this example, we will learn how to Export REST API to CSV using Python.

1. First of all, you will need to install Pip if not included in Python. Pip is Package Installer.
   For instructions about the installation, refer to this link.
2. Secondly, you will also need the pyodbc. The pyodbc allows connecting to ODBC using Python. To install it go to the scripts folder of Python where Python is installed and run this command:pip install pyodbc.
3. Once that pyodbc is installed, we will run the following code:Full Code
   

   import csv
   import pyodbc
   
   conn = pyodbc.connect(
       r'DRIVER={ZappySys JSON Driver};'
       )
   cursor = conn.cursor()	
   rows = cursor.execute("SELECT CustomerID,CompanyName FROM value WHERE COUNTRY='Germany' WITH (SRC='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json')") 
   with open(r'C:\sql\cus2.csv', 'w', newline='') as csvfile:
       writer = csv.writer(csvfile)
       writer.writerow([x[0] for x in cursor.description])  
       for row in rows:
           writer.writerow(row)

4. Now lets’s understand parts of above code. We have used the csv and pyodbc modules in the code:
   

   import csv
   import pyodbc

5. Also, we connect to the ZappySys ODBC Driver:
   

   conn = pyodbc.connect(
   r'DRIVER={ZappySys JSON Driver};'
   )

6. In addition, we have used a cursor to get the rows and send a SQL query to get data from the REST API:
   

   cursor = conn.cursor() 
   rows = cursor.execute("SELECT CustomerID,CompanyName FROM value 
   WHERE Country='Germany' WITH 
   (SRC='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json')")

7. Following code is to open the CSV file stream:
   

   with open(r'C:\sql\customer.csv', 'w', newline='') as csvfile:

8. Finally, we will write the data from REST API into the CSV file:
   

   writer = csv.writer(csvfile)
   writer.writerow([x[0] for x in cursor.description]) 
   for row in rows:
   writer.writerow(row)

9. To conclude, if everything is OK, you will be able to see the created CSV file:
   

   Python code REST API

Using ODBC DSN in Connection String

So far we have seen DSN less connection string approach for ODBC Driver but now lets look at another way to use ODBC Driver in your C# or Python code. You can define many settings on DSN Datasource rather than setting in the ConnectionString.

Configure DSN for REST API Connection

1. First of all, we will access the following URL:
   

   https://services.odata.org/V3/Northwind/Northwind.svc/Orders?$format=json

2. Secondly, in the windows start menu, Search for “ODBC” open the ODBC Data Sources.
3. Also, in the ODBC Administrator, press Add and select the ZappySys JSON
   Driver:
   

   Add ODBC JSON

4. Finally, specify the URL of step 1 and save the configuration:
   

   URL ODBC

Using ODBC DSN in C# Code

Now to use ODBC DSN you created simply change our previous C# Code as below (Just one line)

using (var conn = new OdbcConnection("DSN=Your-DSN-Name-Goes-Here"))
{
    conn.Open();
    ...........
    ...........
    ...........

Using ODBC DSN in C# code to call REST API

Using ODBC DSN in Python Code

Now to use ODBC DSN you created simply change our previous C# Code as below (Just one line)

conn = pyodbc.connect(
    r'DSN=Your-DSN-name-Goes-Here;'
    )
	
	...........
	...........
	...........

ZappySys JSON /REST API Driver Query Examples

Reading from XML files or API can be done using the same way as previous sections except you have to use ZappySys XML Driver. Read help file here to see json query examples.

ZappySys XML / SOAP Driver Query Examples

Reading from XML files or API can be done using the same way as previous sections except you have to use ZappySys XML Driver. Read help file here to see xml query examples.

Conclusion

To conclude, in this article, we show how to access REST API using C# and Python. We used the ZappySys ODBC PowerPack that allows accessing to REST API data and JSON files using SQL queries. It is possible to create simple SQL queries and access the data. It is also possible to access to XML files and Web API with this tool. If you liked this tool you can test the ZappySys ODBC PowerPack here.

References

Finally, if you want to read more about this topic, refer to these links:

• Download ZappySys ODBC PowerPack Installer
• How To Use the ODBC .NET Managed Provider in Visual C# .NET and Connection Strings
• Python ODBC Wiki

The post How to export REST API to CSV using c# or Python appeared first on ZappySys Blog.

Below script assume that you have column count based on separator=comma. if you use different separator then just change script accordingly

public void Main()
        {
            // TODO: Add your code here

            var colCnt = GetFileColumns("file1",',');
            MessageBox.Show("Your file has total " + colCnt + " columns");
            Dts.TaskResult = (int)ScriptResults.Success;
        }
        private int GetFileColumns(string connName,char colSeperator= ',')
        {
            var filePath=Dts.Connections[connName].ConnectionString;
            string firstline;
            using (System.IO.StreamReader sr = new System.IO.StreamReader(filePath))
            {
                 firstline= sr.ReadLine();
            }              
            var colArr=firstline.Split(new char[]{colSeperator});
            return colArr.Length;
        }

The post How to count file columns in SSIS using C# Script appeared first on ZappySys Blog.

In this small blog post you will learn How to move files using SSIS Advanced File System Task and How to wait until file is unlocked using C# Script Task.

How to check if file is locked (SSIS C# Script Task)

If you want to check if file is locked in C# then below code will do the trick. But no worry if you dont know C#. ZappySys SSIS PowerPack comes with two tasks which can solve this issue without coding. You can use SSIS Advanced File System Task with Get file lock status action or Use SSIS Validation Task which has option to throw error on lock condition or you can continue by saving lock status into variable and continue without throwing error.

// Attempt to open the file exclusively. -- If you get erro means file is locked
using (FileStream fs = new FileStream(fullPath,
	FileMode.Open, FileAccess.ReadWrite,
	FileShare.None, 100))
{
	fs.ReadByte();

	// If we got this far the file is ready
	break;
}

How to wait until file is unlocked (SSIS C# Script Task)

Now lets check real example which will first check for locked file using SSIS Validation Task and then if file is locked then wait until file is unlocked (with some timeout hardcoded in script).

If specified wait time is reached then script will throw error. Once file is unlocked SSIS Advanced File System Task will copy file to target.

SSIS – How to check if file is locked. Wait until file is unlocked (C# Code – SSIS Script Task).

SSIS Validation Task -Store file lock status into variable

SSIS Validation Task -Check if file is locked. Save status to variable or throw error.

SSIS Advanced File System Task – Copy, Move, Rename, Delete multiple files

SSIS Advanced File System Task. Copy, Move, Rename, Delete multiple files

SSIS C# Script Task – Check file is locked, wait until file is unlocked

#region Namespaces
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.IO;
#endregion

namespace ST_334a75922c6a47a5b4ac21ee8e4e5fff
{
	[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
	public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
	{
		public void Main()
		{
			// TODO: Add your code here
            try
            {
                var file = Dts.Variables["User::filePath"].Value.ToString();

                bool continueIfFileMissing = false;
                if (!System.IO.File.Exists(file))
                {
                    if (continueIfFileMissing)
                    {
                        Dts.TaskResult = (int)ScriptResults.Success;
                        return;
                    }
                    LogError("File not found: " + file);
                    Dts.TaskResult = (int)ScriptResults.Failure;
                }
                    
                WaitForFile(file, maxWaitInSec:30);
                Dts.TaskResult = (int)ScriptResults.Success;
            }
            catch (Exception ex)
            {
                LogError(ex.Message);
                Dts.TaskResult = (int)ScriptResults.Failure;
            }           
			
		}
        /// <summary>
        /// Blocks until the file is not locked any more.
        /// </summary>
        /// <param name="fullPath"></param>
        bool WaitForFile(string fullPath,int maxWaitInSec)
        {
            int numTries = 0;
            while (true)
            {
                ++numTries;
                try
                {
                    // Attempt to open the file exclusively.
                    using (FileStream fs = new FileStream(fullPath,
                        FileMode.Open, FileAccess.ReadWrite,
                        FileShare.None, 100))
                    {
                        fs.ReadByte();

                        // If we got this far the file is ready
                        break;
                    }
                }
                catch (Exception ex)
                {
                   if(numTries==1 ||  numTries % 20 ==0 )
                      LogWarning(string.Format("WaitForFile {0} failed to get an exclusive lock: {1}",fullPath, ex.Message));

                    //if (numTries > 10)
                    //{
                    //    LogWarning(string.Format(
                    //        "WaitForFile {0} giving up after 10 tries",
                    //        fullPath));
                    //    return false;
                    //}

                   if (numTries >= maxWaitInSec * 2)
                   {
                       throw new Exception("Max wait time reached for file : " + fullPath + ". Waited for " + maxWaitInSec + " seconds but lock not released");
                   }

                    // Wait for the lock to be released
                    System.Threading.Thread.Sleep(500);
                }

                
            }

            LogInformation( string.Format( "WaitForFile {0} returning true after {1} tries",fullPath, numTries));
            return true;
        }

        private void LogInformation(string msg)
        {
            bool again = false;
            Dts.Events.FireInformation(0, "ScriptTask", msg, "", 0,ref again);
        }
        
        private void LogError(string msg)
        {
            Dts.Events.FireError(0, "ScriptTask", msg, "", 0);
        }
        private void LogWarning(string msg)
        {
            Dts.Events.FireWarning(0, "ScriptTask", msg, "", 0);
        }
        #region ScriptResults declaration
        /// <summary>
        /// This enum provides a convenient shorthand within the scope of this class for setting the
        /// result of the script.
        /// 
        /// This code was generated automatically.
        /// </summary>
        enum ScriptResults
        {
            Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
            Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
        };
        #endregion

	}
}

Download Sample Package

Below sample will work only if you have SSIS PowerPack Installed. Download it from here it will take only 1 minute to install
Download SSIS 2012 – Sample Package (Process Locked file)

Conclusion

Processing and detecting locked files in SSIS can be tricky but using small C# script it can save you some headache. Download Advanced File System Task to try many options not available in native File System Task.

Other Keywords:
Check locked files in SSIS
How to check whether file is locked or not in SSIS
Detect locked file in SSIS
Wait until file is unlocked using C# script
How to handle file locking issue in SSIS using C# script

The post SSIS check file is locked and wait until file is unlocked (C# Script) appeared first on ZappySys Blog.

If you using SSIS JSON Source or our REST API Task then if URL/Document you trying to read not found then server may return 404 (Not Found) Exception. In order to handle this error gracefully you can use following Script to store status in a variable and then you can use conditional Control Flow to Skip data loading from Web Service.

Step1: Drag Script Task from toolbox
Step2: Select Language as C#
Step3: Select Readonly variable (In this case “User::url” – String Type)
Step4: Select ReadWrite variable (In this case “User::found” – Boolean Type)
Step5: Edit code and paste following script in your script task

SSIS C# Code – Check URL Exists (Catch 404 Not Found Error)

C# Code: Check URL Exists (Detect 404 Error)

public void Main()
	{
		// TODO: Add your code here
            Dts.Variables["found"].Value = UrlFound(Dts.Variables["url"].Value.ToString());
		Dts.TaskResult = (int)ScriptResults.Success;
	}

        private bool UrlFound(string url)
        {
            try
            {
                var request = WebRequest.Create(url);
                var response = request.GetResponse();
                return true;
            }
            catch (WebException we)
            {
                HttpWebResponse errorResponse = we.Response as HttpWebResponse;
                if (errorResponse != null)
                    if (errorResponse.StatusCode == HttpStatusCode.NotFound)
                    {
                        return false;
                    }
                    else
                        throw; //we only look for 404 Error
            }
            return false;
        }

The post SSIS Code – Check URL Exists using C# Script – 404 Error appeared first on ZappySys Blog.