CloudTier Storage Tiering Filter Driver SDK

Download CloudTier Storage Tiering SDK Setup File
Download CloudTier Storage Tiering SDK Zip File

What is the CloudTier Storage Tiering SDK

The CloudTier Storage Tiering SDK is a Hierarchical Storage Management (HSM) file system filter driver development kit. It is a data storage technique that automatically moves data between high-cost and low-cost storage media. It allows you to free up on-premise storage capacity transparently, by moving out cooler data to the cloud storage, thereby reducing capital and operational expenditures.

 The CloudTier Storage Tiering SDK provides you a simple and low-cost solution to integrate your on-premise storage to cloud storage infrastructure in a seamless, secure, and transparent fashion. There is no interruption to migrate your on-premise files to the remote cloud storage, so you don't need to change your existing applications and infrastructure. It uses the on-premises storage as a tier 0 storage (hot storage), uses the public cloud storage as a tier 1 storage (clod storage). So you can connect the cloud storage as a second tier. Your application can use the cloud storage just like the local storage without any changes.

Understand The File System Filter Driver

A file system filter driver intercepts requests targeted at a file system or another file system filter driver. By intercepting the request before it reaches its intended target, the filter driver can extend or replace functionality provided by the original target of the request. File system filtering services are available through the filter manager in Windows. The Filter Manager provides a framework for developing File Systems and File System Filter Drivers without having to manage all the complexities of file I/O.

How Does The Storage Tiering Work

The storage tiering is a comprehensive technology to access the remote storage transparently. The CloudTier storage SDK integrates with the local file system. When an application accesses a stub file, the CloudTier Filter Driver will bring back the remote storage’s data back to the application or rehydrate the stub file as a normal physical file based on the recall policy.

A stub file looks and acts like a regular file, it has the same file attributes with the original physical file (file size, creation time, last write time, last access time), it also keeps the original file’s security. A stub file is a file with sparse file and reparse point attributes, your customized tag data can be embedded to the stub file. A stub file doesn’t take the storage space for the file data, it only keeps the meta data of the file.

To implement the storage tiering solution, first you need to transfer the files to the remote storage, it can be your private or public cloud server. After the files were transferred to the server, you can replace them with the stub file based on your ILM (Information Lifecycle Manager) policy, after the file was replaced with the stub file, the file data space in your premise storage was freed up.

The Use Cases of CloudTier Storage Tiering

Cloud storage tiering can be widely used in telecommunications, government, oil, medical and other industries.

1. Healthcare Data Archiving. Healthcare providers are increasingly required to retain patient medical record data for multi-year periods. Storage requirements for patient data can quickly escalate when the records include high resolution images and ultrasound content.
2. Business Policy Mandated Data Archiving. Companies can generate PB’s of data in the course of day to day operations and to meet legal compliance requirements. Archive Storage can work with Smart Archiving offerings from ISV’s to create a low cost, content archiving solution.
3. Digital Media Content Retention. Creators can generate PB’s worth of video and picture content that is used in the development of original digital media. Archive Storage gives creators a low-cost storage repository for original source content. File-level tiering makes it easy to shift from cold to hot storage should the need arise to use that content for another project.
4. Security/Public Safety data retention. As the number and sophistication of threats to personal and business safety continue to increase, so does the demand for video surveillance. Public and now private sector companies generate TB’s of surveillance footage daily in the course of protecting their citizens and assets. Archive Storage is a low-cost option for storing that data.

The CloudTier Storage Tiering Demo

To use the CloudTier Storage Tiering SDK is very simple, we have the C++/C# demo source code, it demonstrates how to use the SDK. The first step is generate the stub file with the API “CreateStubFileEx” as below:

  	
[DllImport("FilterAPI.dll", SetLastError = true)]
public static extern bool CreateStubFileEx(
	 [MarshalAs(UnmanagedType.LPWStr)]string fileName,
	 long fileSize,
	  uint fileAttributes,
	  uint tagDataLength,
	  IntPtr tagData,
	  long creationTime,
	  long lastWriteTime,
	  long lastAccessTime,
	  bool overwriteIfExist,
	  ref IntPtr fileHandle);

After the stub file was generated, you need to handle the read request of the stub file, you just need to register the callback function for the file system filter driver. When the stub file was accessed, the callback function will be invoked, the callback function will retrieve the data from the remote server and send back to the filter driver. In your application you need to start the filter driver service and register the callback function as below:

  	
if(!filterControl.StartFilter(numberOfThreads,connectionTimeOut, licenseKey, ref lastError))
{
	return false;
}

if (!filterControl.SendConfigSettingsToFilter(ref lastError))
{
   return false;
}

filterControl.OnFilterRequest += OnFilterRequestHandler;

The OnFilterRequestHandler callback function will handle the stub file read request. The parameters passing to the callback function includes the user name, process name, file name, file size, file attributes, file time, read offset and read length. To handle the read request in the callback function, you can return the block data or the whole file data from your remote server based on the read request type.

  	
static void OnFilterRequestHandler(object sender, FilterRequestEventArgs e)
{
	Boolean ret = true;

	try
	{

		//here the data buffer is the reparse point tag data, in our test, we added the test source file name of the stub file as the tag data.
		string cacheFileName = Encoding.Unicode.GetString(e.TagData);
		//remove the extra data of the file name.
		cacheFileName = cacheFileName.Substring(0, e.TagDataLength / 2);

		if (e.MessageType == FilterAPI.MessageType.MESSAGE_TYPE_RESTORE_FILE_TO_CACHE)
		{
			//for the write request, the filter driver needs to restore the whole file first,
			//here we need to download the whole cache file and return the cache file name to the filter driver,
			//the filter driver will replace the stub file data with the cache file data.

			//for memory mapped file open( for example open file with notepad in local computer )
			//it also needs to download the whole cache file and return the cache file name to the filter driver,
			//the filter driver will read the cache file data, but it won't restore the stub file.

			e.ReturnCacheFileName = cacheFileName;

			//if you want to rehydrate the stub file, please return with REHYDRATE_FILE_VIA_CACHE_FILE
			if (GlobalConfig.RehydrateFileOnFirstRead)
			{
				e.FilterStatus = FilterAPI.FilterStatus.REHYDRATE_FILE_VIA_CACHE_FILE;
			}
			else
			{
				e.FilterStatus = FilterAPI.FilterStatus.CACHE_FILE_WAS_RETURNED;
			}

			e.ReturnStatus = (uint)FilterAPI.NTSTATUS.STATUS_SUCCESS;
		}
		else if (e.MessageType == FilterAPI.MessageType.MESSAGE_TYPE_RESTORE_BLOCK_OR_FILE)
		{

			e.ReturnCacheFileName = cacheFileName;

			//for this request, the user is trying to read block of data, you can either return the whole cache file
			//or you can just restore the block of data as the request need, you also can rehydrate the file at this point.

			//if you want to rehydrate the stub file, please return with REHYDRATE_FILE_VIA_CACHE_FILE
			if (GlobalConfig.RehydrateFileOnFirstRead)
			{
				e.FilterStatus = FilterAPI.FilterStatus.REHYDRATE_FILE_VIA_CACHE_FILE;
			}
			else if (GlobalConfig.ReturnCacheFileName)
			{
				e.FilterStatus = FilterAPI.FilterStatus.CACHE_FILE_WAS_RETURNED;
			}
			else
			{
				//we return the block the data back to the filter driver.
				FileStream fs = new FileStream(cacheFileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
				fs.Position = e.ReadOffset;

				int returnReadLength = fs.Read(e.ReturnBuffer, 0, (int)e.ReadLength);
				e.ReturnBufferLength = (uint)returnReadLength;                        

				e.FilterStatus = FilterAPI.FilterStatus.BLOCK_DATA_WAS_RETURNED;                       

				fs.Close();

			}

			e.ReturnStatus = FilterAPI.NTSTATUS.STATUS_SUCCESS;
		}
		else
		{
			EventManager.WriteMessage(158, "ProcessRequest", EventLevel.Error, "File " + e.FileName + " messageType:" + e.MessageType + " unknow.");

			e.ReturnStatus = FilterAPI.NTSTATUS.STATUS_UNSUCCESSFUL;

			ret = false;
		}

					
	}
	catch (Exception ex)
	{
		EventManager.WriteMessage(181, "ProcessRequest", EventLevel.Error, "Process request exception:" + ex.Message);
	}

}