How to Combine Azure File Sync with Azure Archive

October 05, 2017 Geoff Bourgeois Azure Archive Storage, Azure File Sync 0 Comments

Azure File Sync was announced by Klaas Langhout of the Azure Storage team at the Microsoft Ignite 2017 conference. In this post, we will talk about how to combine Azure File Sync with an Azure Archive strategy.

What is Azure File Sync?

Azure File Sync (AFS) is an easy way to integrate multiple Windows Servers into a global namespace. In other words, you can have multiple instances of Windows Servers connecting to Azure File Services and synchronizing to provide a distributed workgroup a common file server with synchronized directory structure, permissions, and data.

Setting up AFS is incredibly simple. The first step is to create and configure a Sync Group in Azure. Next, you need to enroll one or more Windows Server instances into the Sync Group. The enrolment process includes running an installer on the Windows Server host machine which will install the Microsoft sync engine software and a filter-driver component.

Global Namespace for Workgroups, but no File Locking

The global file share concept is particularly useful for remote and branch office scenarios.

At the time of launch, Azure File Sync will support five (5) Windows Server instances in a sync group. Expect Microsoft to increase this number in the future.

The sync engine does not include the concept of global file locking. If two users access the same item, modify it, and attempt to save changes at nearly the same time, AFS will prevent collisions by saving one of the versions with a new name (appending the file name with the host server name). Obviously, this is inconvenient for users, so if file locking is important to you, Microsoft recommends using a partner solution called Talon Software.

The Storage Tiering Advantage

In addition to providing a global namespace, the filter-driver component of Azure File Sync performs seamless storage tiering. Original files can be converted to offline files with reparse points that support intercepting file open requests.

This method of storage tiering is handy because any user or application will continue to see tiered files as if they are local, and they can open and interact with files as normal.

Since the data must exist in the cloud to sync across all the Windows Servers in the Sync Group, it makes sense to virtualize the older data because it isn’t being accessed. The result is a lean and mean on-prem Windows Server and significantly less load on the network connection – the sync engine creates the older data on-prem as offline files as opposed to downloading everything to all Windows Servers.

How Pricing Works

AFS uses Azure File Services storage in Azure. Coinciding with the public preview announcement, Microsoft included a 25% price reduction of Azure File Services when configured as Locally-Redundant Storage (LRS), bring it to $0.06/GB/month. As usual, prices can vary depending on the Azure region you use, and Geo-Redundant Storage (GRS) is a different pricing tier.

Azure File Sync vs. Cloud Gateways vs. Cloud Archiving

Azure File Sync will take a significant portion of the cloud storage gateway market. For smaller organizations that already have a Windows Server storage infrastructure, AFS is particularly attractive compared to the prospect of acquiring and deploying new cloud storage gateway appliances that require a lift and shift.

However, without global file locking support, many organizations will hesitate with AFS (or, they will add Talon).

For small businesses with just two or three terabytes, we think Azure File Sync will, in most cases, replace the need for using HubStor cloud archiving. A small business case easily cloud-enable their Windows Servers and, since their storage volumes are so low, not worry too much about the higher storage cost of Azure Files.

However, in scenarios with storage requirements exceeding 30 TB, we think IT administrators will need to consider their options more closely. AFS only supports five Windows Servers in a Sync Group at this time, and, as we'll explore in the cost analysis section below, the storage costs of Azure Files scale linearly. 

 

Azure File Sync

Cloud Gateways

HubStor Cloud Archive

Global namespace

Y

Y

 

File locking

Y – with 3rd party add on

Y

 

Storage tiering for Windows

Y

 

Y

Storage tiering for CIFS/NFS

 

 

Y

Deterministic tiering policies

 

 

Y

Rapid RTO recovery/migrations using pointers

Y

 

Y

Cloud data management

 

 

Y

  

Why Azure File Sync Needs an Azure Archive Strategy

Just like any collaboration solution, you will need to think about storage management and archiving because aging data eventually clutters and burdens the workspace.

As you consider using AFS, we recommend planning your Azure archive storage strategy in parallel, which will be driven by three factors:

  • Cost management – moving older data from Azure Files storage to lower-cost Azure Blob Storage will reduce cloud storage spending. In the archive, HubStor’s storage analytics and policy engine make it easy to manage data between Hot, Cool, and Archive tiers for further cost optimization.
  • Performance and efficiency – minimizing the file object count that the sync engine needs to manage will eventually become a priority. At Ignite, Klaas said that the largest scale test of AFS was 30 million files. With an average file size of 2 MB, that is 60 TB. The object count limits of the sync engine (whatever they are) will necessitate an archiving strategy because it will remove inactive data from the scope of AFS, thus allowing the sync engine to run more efficiently in the scenarios where it must run against the entire corpus of data in the sync group.
  • Data governance requirements -- AFS uses Azure Files which is readily available storage for other services to run against the data. However, a proactive archiving strategy may be better for some clients. For instance, search and data management features support fast discovery. Other features such as chargeback, data classification, and the eventual deletion of content are turn key. Features like WORM storage and litigation hold are also available with archiving.

Ideas on How to Combine Azure Archiving with Azure File Sync

As usual in enterprise IT, there are multiple ways to reach a solution.

Idea #1: Add an ‘Archive’ Windows Server On-premises that is Cloud-enabled with HubStor

We think the surefire approach to integrate an Azure Archive strategy to AFS is to stand up a Windows Server instance which will be the front-end for the archive. This ‘archive’ Windows Server would be cloud-enabled using HubStor.

You would simply copy folders for archiving to the archive Windows Servers. HubStor would then perform seamless storage tiering from the archive Windows Server to Azure Blob Storage with synchronization of folders and security ACLs.

HubStor’s seamless storage tiering for Windows Server works exactly like the storage tiering of AFS – offline files with reparse points and a filter driver that handles recalls. However, unlike AFS, HubStor has a deterministic policy engine that gives you greater control over tiering and local caching. If the ‘archive’ server needs to be distributed, this can easily be accomplished with a standard DFS configuration.

Idea #2: Run Policy-based Tiering from Azure Files to Azure Blob Storage using HubStor

The other approach, which is one we will discuss in more depth in a follow up post, has ingestion to the archive happening from the content in Azure File Services. In other words, the policy engine for archiving captures content from Azure Files.

NOTE: This is something we are still validating in our labs because it may not play nicely with the sync engine of Azure File Sync.

The idea here is to crawl Azure Files and, based on policies, migrate content into the archive because this approach does not involve egress and ingress activity to get the data from AFS to your Azure archive.

In this approach, there are three methods to access archived content in HubStor:

  • User Portal – a Web-based interface which users authenticate to via Azure AD. Since Azure File Sync now supports security ACLs, HubStor would sync the data-level authorization so that the User Portal to the archive would provide users with self-service access to their folders and items.
  • Privileged users – IT admins can always recover data back to Azure Files or on-premises using the HubStor Recovery Utility.
  • Restore pointers to an on-premises Windows Server – the other approach involves using HubStor’s Recovery Utility to generate the archived data locally as pointers with the same directory and permissions.

Capturing data into the archive from Azure Files is a more efficient approach because it avoids egress and a lot less impact on the network connection. With HubStor’s Recovery Utility, you can still present the archived data locally to users and apps without having to download anything from the archive.

Cost Analysis: Azure File Sync vs. HubStor

We said that cost management would be a driver for archiving data from the scope of Azure File Sync.

There is quite a spread in pricing between Azure Files and Azure Blob Storage. The pricing for Azure Files is well suited for active data; whereas, Azure Blob’s Cool and Archive tiers are nicely suited for long-term retention of data that is low-touch.

To illustrate this difference, we have run the monthly figures comparing the pricing of AFS and HubStor at data volumes from 1 TB up to 210 TB.

Azure File Sync

Assumptions:

  • Azure pricing is based on East US 2 region in USD, pay-as-you-go rates.
  • No storage access/activity or data egress costs are factored.
  • For Azure File Sync:
    • Five (5) Windows Servers in the sync group up to 105 TB.
    • At 105 TB and beyond, 10 Windows Servers in the sync group (we assume Microsoft will support larger sync groups in the future).
  • For HubStor:
    • At 15 TB, the performance class of the PaaS configuration is increased to S2.
    • At 105 TB, the performance class of the PaaS configuration is increased to S3.
    • The figures are all inclusive of Azure costs and HubStor’s standard markup.

It is important to also call out the fact that the above price examples do not factor the activity costs.

Storage activity costs are often overlooked, and they can be significant with object counts into the hundreds of millions or billions of items.

As you can see in the chart below, a sample of the storage access costs for Write, List, and Read operations is significantly lower in Azure Files than what we see with Azure Blob.

 

AFS - LRS

AFS - GRS

Hot Blob - LRS

Hot Blob - GRS

Cool Blob - LRS

Cool Blob - GRS

Archive Blob - LRS

Write Operations (per 10,000)

$0.015

$0.03

$0.05

$0.10

$0.10

$0.20

$0.30

List Operations (per 10,000)

$0.015

$0.015

$0.05 

$0.10

$0.10

$0.20

$0.30

Read Operations (per 10,000)

$0.0015

$0.0015

$0.004 

$0.004

$0.01

$0.01

$0.15

 

Microsoft is pricing higher activity costs on the lower-cost storage tiers.

A pricing structure like this means that you need to manage data intelligently to have optimally storage spend in the cloud. Analytics and a sophisticated rules engine allow you to place data on the appropriate tier at the best time.

Conclusion

So far, we have looked at the cool new Azure File Sync from Microsoft. We think it is a great fit for many types of organizations and scenarios, and ultimately will help drive adoption of the Azure cloud platform.

Unfortunately, for cloud storage gateway vendors, AFS is another bite into their total addressable market. In a previous post, I commented on Gartner’s negative view on the CSG market. AFS is just another reason why it is a tough to be a CSG vendor these days.

Although Azure File Sync is a great solution for a cloud file share and global namespace, it is not yet ready for scalability scenarios and the storage costs of Azure Files becomes prohibitive as your data volumes grow. For these reasons, we think AFS gets more mileage with an Azure-based archive solution.

In our next post, we will explore other aspects of Azure File Sync and provide our results on the analysis of the archive grabbing content directly from Azure Files as it ages.

 

 Request a Demo

 

 

Stay Informed
Stay Informed!

Get HubStor success stories, product news, and announcements delivered to your inbox!

Join our mailing list to start receiving exclusive content only our newsletter members have access to.



Back to blog list


About the Author

Subscribe to Email Updates