CloudTier Storage Tiering Demo

Download CloudTier Storage Tiering Demo Setup File
Download CloudTier Storage Tiering Demo Zip File

C# Storage Tiering Demo

The CloudTier Storage Tiering Demo is a C# demo project, it demonstrates how to use the CloudTier Storage Tiering SDK. The CloudTier Storage Tiering SDK is a Hierarchical Storage Management (HSM) file system filter driver development kit. It is a data storage technique that automatically moves data between high-cost and low-cost storage media. It allows you to free up on-premise storage capacity transparently, by moving out cooler data to the cloud storage, thereby reducing capital and operational expenditures.

The CloudTier SDK includes the kernel mode filter driver “cloudtier.sys”, a user mode library FilterAPI.dll which exports the APIs from the filter driver, and communication between the filter driver and the user mode application. The C# CloudTier demo project includes project “FilterControl”, “CommObjects” and “CloudTierDemo. You must reference the project “FilterControl” which is wrapper project for FilterAPI, it controls the communication between the filter driver and the application.

When you run the “CloudTierDemo.exe”, it will generate the test source files in folder “TestSourceFolder” and the associated test stub files in folder “TestStubFolder”, when you read the test stub file “TestStubFolder\testFile.1.txt”, it will get the file data from the source file “TestSourceFolder\testFile.1.txt”.

A stub file looks and acts like a regular file. It has the same file attributes with the original physical file (file size, creation time, last write time, last access time). It also keeps the original file's security. The difference between the stub file and the normal physical file is the stub file doesn't take any physical space, looks like a 0 kb file.

You also can create your own test stub file based on your own custom source files if you have. You can specify your test source file folder first, then you can choose a folder to create the test stub files based on the source file as below:

Here is the settings which related to the stub file reading.

1. Exclude process Ids, you can exclude the process to read the stub file. The excluded processes can’t read the stub files.
2. The number of the threads to handle the stub file reading.
3. The connection timeout which the filter driver waits for the response from the user mode application.
4. The return data type: a. Return block data on read, this return method allows you only return the read data back to the filter driver, i.e. if the application only read 10 bytes, you just need to return a block of data includes the read data. b. Return cache file on read, this return method allows you to return the whole cache file with file data to the filter driver. c. Rehydrate file on read, this return method allows you to rehydrate the stub file after it was read.

Understand The CloudTier Storage Tiering Demo

To use the CloudTier Storage Tiering SDK is very simple, we have the C++/C# demo source code, it demonstrates how to use the SDK. The first step is generate the stub file with the API “CreateStubFileEx” as below:

  	
[DllImport("FilterAPI.dll", SetLastError = true)]
public static extern bool CreateStubFileEx(
	 [MarshalAs(UnmanagedType.LPWStr)]string fileName,
	 long fileSize,
	  uint fileAttributes,
	  uint tagDataLength,
	  IntPtr tagData,
	  long creationTime,
	  long lastWriteTime,
	  long lastAccessTime,
	  bool overwriteIfExist,
	  ref IntPtr fileHandle);

After the stub file was generated, you need to handle the read request of the stub file, you just need to register the callback function for the file system filter driver. When the stub file was accessed, the callback function will be invoked, the callback function will retrieve the data from the remote server and send back to the filter driver. In your application you need to start the filter driver service and register the callback function as below:

  	
if(!filterControl.StartFilter(numberOfThreads,connectionTimeOut, licenseKey, ref lastError))
{
	return false;
}

if (!filterControl.SendConfigSettingsToFilter(ref lastError))
{
   return false;
}

filterControl.OnFilterRequest += OnFilterRequestHandler;

The OnFilterRequestHandler callback function will handle the stub file read request. The parameters passing to the callback function includes the user name, process name, file name, file size, file attributes, file time, read offset and read length. To handle the read request in the callback function, you can return the block data or the whole file data from your remote server based on the read request type.

  	
static void OnFilterRequestHandler(object sender, FilterRequestEventArgs e)
{
	Boolean ret = true;

	try
	{

		//here the data buffer is the reparse point tag data, in our test, we added the test source file name of the stub file as the tag data.
		string cacheFileName = Encoding.Unicode.GetString(e.TagData);
		//remove the extra data of the file name.
		cacheFileName = cacheFileName.Substring(0, e.TagDataLength / 2);

		if (e.MessageType == FilterAPI.MessageType.MESSAGE_TYPE_RESTORE_FILE_TO_CACHE)
		{
			//for the write request, the filter driver needs to restore the whole file first,
			//here we need to download the whole cache file and return the cache file name to the filter driver,
			//the filter driver will replace the stub file data with the cache file data.

			//for memory mapped file open( for example open file with notepad in local computer )
			//it also needs to download the whole cache file and return the cache file name to the filter driver,
			//the filter driver will read the cache file data, but it won't restore the stub file.

			e.ReturnCacheFileName = cacheFileName;

			//if you want to rehydrate the stub file, please return with REHYDRATE_FILE_VIA_CACHE_FILE
			if (GlobalConfig.RehydrateFileOnFirstRead)
			{
				e.FilterStatus = FilterAPI.FilterStatus.REHYDRATE_FILE_VIA_CACHE_FILE;
			}
			else
			{
				e.FilterStatus = FilterAPI.FilterStatus.CACHE_FILE_WAS_RETURNED;
			}

			e.ReturnStatus = (uint)FilterAPI.NTSTATUS.STATUS_SUCCESS;
		}
		else if (e.MessageType == FilterAPI.MessageType.MESSAGE_TYPE_RESTORE_BLOCK_OR_FILE)
		{

			e.ReturnCacheFileName = cacheFileName;

			//for this request, the user is trying to read block of data, you can either return the whole cache file
			//or you can just restore the block of data as the request need, you also can rehydrate the file at this point.

			//if you want to rehydrate the stub file, please return with REHYDRATE_FILE_VIA_CACHE_FILE
			if (GlobalConfig.RehydrateFileOnFirstRead)
			{
				e.FilterStatus = FilterAPI.FilterStatus.REHYDRATE_FILE_VIA_CACHE_FILE;
			}
			else if (GlobalConfig.ReturnCacheFileName)
			{
				e.FilterStatus = FilterAPI.FilterStatus.CACHE_FILE_WAS_RETURNED;
			}
			else
			{
				//we return the block the data back to the filter driver.
				FileStream fs = new FileStream(cacheFileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
				fs.Position = e.ReadOffset;

				int returnReadLength = fs.Read(e.ReturnBuffer, 0, (int)e.ReadLength);
				e.ReturnBufferLength = (uint)returnReadLength;                        

				e.FilterStatus = FilterAPI.FilterStatus.BLOCK_DATA_WAS_RETURNED;                       

				fs.Close();

			}

			e.ReturnStatus = FilterAPI.NTSTATUS.STATUS_SUCCESS;
		}
		else
		{
			EventManager.WriteMessage(158, "ProcessRequest", EventLevel.Error, "File " + e.FileName + " messageType:" + e.MessageType + " unknow.");

			e.ReturnStatus = FilterAPI.NTSTATUS.STATUS_UNSUCCESSFUL;

			ret = false;
		}

					
	}
	catch (Exception ex)
	{
		EventManager.WriteMessage(181, "ProcessRequest", EventLevel.Error, "Process request exception:" + ex.Message);
	}

}