CloudTier Storage Tiering SDK

Download CloudTier Storage Tiering SDK Setup File
Download CloudTier Storage Tiering SDK Zip File

The Introduction

For most companies, data – especially unstructured data, continues to grow by 50 percent annually. The impact of spending more every year on storage, and on protecting and managing information has often pushed IT departments to their limits. Combined with longer mandated retention periods, today’s information management challenges are forcing IT staff, from the vice president to the systems administrator, to reduce complexity and cost without putting their organization’s infrastructure, information, and intellectual property at risk. The transparent cloud storage tiering is a simple and low cost solution to integrate your on-premise storage to cloud seamlessly.

Transparent Cloud Tiering SDK

Transparent Cloud Storage Tiering SDK provides you a simple solution to develop the cloud archiving software, it allows you to integrate your existing on-premises applications with the remote cloud storage infrastructure in a seamless, secure, and transparent fashion. The cloud tiering is a data storage technique which automatically moves data between high-cost on-premise storage and low-cost remote cloud storage, it provides a native cloud storage tier, it allows you to free up on-premise storage capacity, by moving out cooler data to the cloud storage, thereby reducing capital and operational expenditures.

Cloud Tiering

Transparent Cloud Data File Access

By creating ILM (Information Lifecycle Manager) policy, a stub file will be created after the file was migrated to the cloud storage. A stub file looks and acts like a regular file, it has the same file attributes with the original physical file (file size, creation time, last write time, last access time), it also keeps the original file's security. A stub file is a file with sparse file and reparse point attributes, your customized tag data can be embedded to the stub file. A stub file doesn't take the storage space for the file data, it only keeps the meta data of the file. When the stub file is accessed, the file data in the cloud storage will be transparently recalled to the stub file, you can rehydrate the stub file or just return the data to the application based on the recall policy.

stub file property

Cloud Archiving Solution For Unstructured Data

Cloud archiving is the process of moving data to secondary storage in the cloud, the potential benefits of cloud archiving includes lower costs and easier access, no interruption and change to your existing applications and infrastructure. Automatically archive, manage and secure all your organization’s files to the cloud, transparently access your archived files.

CloudTier

The Main Advantages Of Smart Storage Tiering Are:

  • Lower Storage Costs

If you have two terabytes of expensive server storage where 50% of the data is never or rarely accessed, with the smart storage tiering, you can transfer a terabyte data to the NAS storage or cloud storage, you can save a terabyte storage space in SAN RAID storage.

  • Maximize the server's hard disk available space

Reclaim storage space without disrupting users. Set up policies to automatically remove older files from file servers, cleaning up disk space, and replace them with an intelligent shortcut "stub" that invisibly retrieves the original file from the archive. 

  • Improve efficiency

When the user needs these data, it can be accessed transparently in real time. If you need to recover and restore a file that was accidently deleted or modified in error, you can restore the file or even a whole folder from the repository.

  • Reduce the server backup time and recovery time, only need to backup frequently used files.

  • Improve data security

the data in the server can be encrypted, and access these data through the storage management software, only authorized users can access the data, and can log the access activities.

  • Remove duplicate data, the storage server only keep single instance

The Use Cases Of Cloud Storage Tiering

Cloud storage tiering can be widely used in telecommunications, government, oil, medical and other industries. Tiered storage is the first choice of medical PACS (Picture Archiving and Communication System, medical imaging storage and transmission systems), a lot of data in such applications are rarely visit, these data are transferred to a less expensive network storage. When users and applications access the stub files in the local storage, it is completely transparent, the system will automatically restore the data back to the stub file from the remote storage server. The network attached storage is scalable, tiered storage products provide users with an infinite online data space.

An Example Demonstrates How To Use The Transparent Cloud Storage Tiering SDK In C#

The following example will generate some stub files. To handle the read request of the stub file, we need to register the callback function for the file system filter driver. When the stub file was accessed, the callback function will be invoked, the callback function will retrive the data from the remote server and send back to the filter driver.

  
 public static Boolean ProcessRequest(MessageSendData messageSend, ref MessageReplyData messageReply)
        {
            Boolean ret = false;

            try
            {

		//here the data buffer is the reparse point tag data, in our test, 
		//we assume the reparse point tag data is the cache file name of the stub file.
                string cacheFileName = Encoding.Unicode.GetString(messageSend.DataBuffer);
                cacheFileName = cacheFileName.Substring(0, (int)messageSend.DataBufferLength / 2);

                if (messageSend.MessageType == (uint)MessageType.MESSAGE_TYPE_RESTORE_FILE_TO_CACHE)
                {
                    //for the first write request, the filter driver needs to restore the whole file first,
                    //here we need to download the whole cache file and return the cache file name to the filter driver,
                    //the filter driver will replace the stub file data with the cache file data.

                    //for memory mapping file open( for example open file with notepad in local computer,
                    //it also needs to download the whole cache file and return the cache file name to the filter driver,
                    //the filter driver will read the cache file data, but it won't restore the stub file.

                    ret = DownloadCacheFile(messageSend,cacheFileName, ref messageReply);
                }
                else if (messageSend.MessageType == (uint)MessageType.MESSAGE_TYPE_RESTORE_BLOCK_OR_FILE)
                {

                    //for this request, the user is trying to read block of data, you can either return the whole cache file
                    //or you can just restore the block of data as the request need, you also can rehydrate the file at this point.

                    //if the whole cache file was restored, you better to return the cache file instead of block data.
                    if (GlobalConfig.RehydrateFileOnFirstRead || GlobalConfig.ReturnCacheFileName)
                    {
                        ret = DownloadCacheFile(messageSend, cacheFileName, ref messageReply);
                    }
                    else
                    {
                        ret = GetRequestedBlockData(cacheFileName, messageSend.Offset, messageSend.Length, ref messageReply);
                    }
                }
                else
                {                 
                    messageReply.ReturnStatus = (uint)NTSTATUS.STATUS_UNSUCCESSFUL;
                }

                messageReply.MessageId = messageSend.MessageId;
                messageReply.MessageType = messageSend.MessageType;

                EventLevel eventLevel = EventLevel.Information;
                if (messageReply.ReturnStatus != (uint)NTSTATUS.STATUS_SUCCESS)
                {
                    eventLevel = EventLevel.Error;
                }

                ret = true;

            }
            catch (Exception ex)
            {
                EventManager.WriteMessage(181, "ProcessRequest", EventLevel.Error, "Process request exception:" + ex.Message);
                return false;
            }

            return ret;

        }

The example can monitor the stub file access, the user name and process name which accessed the stub file will be displayed in the console.

CloudTier Console

About EaseFilter Inc.

EaseFilter Inc. is a company who specializes in windows file system filter driver development. It can provide architect, implement and test file system filter drivers for a wide range of functionalities. It also can offer several levels of assistance to meet your specific needs: Provide consulting service for your existing file system filter driver; Customize the SDK to meet your requirement; Create your own filter driver with SDK source code.