Setting Archive Tier on Azure storage

In my article

I spoke about using Azure Archive storage as a good location for long term data retention. The way that you configure this is basically to set up a storage account as usual and initially configure it as ‘Cool’ storage (since you can’t do Archive storage directly). You then upload files there (typically using Azure Storage Explorer). The final piece of the puzzle is to change the access tier from ‘Cool’ to ‘Archive’ by right mouse clicking on the item.

You can do the same using Azure Storage Explorer.

The challenge becomes when you want to do more than a single file at a time.

You’ll see that you now don’t get the option to set a tier any more once you have two items or more selected. The same happens with Azure Storage Explorer as well.

Thanks to Marc Kean who pointed me in the right direction, the solution lies in changing this programmatically. Marc has a script on his site and I found another on GitHub as well but decided to write my own anyway which you’ll find here:

https://github.com/directorcia/Azure/blob/master/az-blob-tierset.ps1

with mine you’ll need to set the following variable first at the top of the script:

$storageaccountname = “<your storage account name here>”

$storageresourcegroup = “<your storage account resource group name here>”

$storagetier = “<your desired storage tier level here>” # Hot, Cool or Archive

You’ll also need to connect to you Azure account beforehand which you can do with script of mine:

https://github.com/directorcia/Azure/blob/master/az-connect.ps1

My script will, get the storage account via:

$storageaccount = Get-AzStorageAccount -name $storageaccountname -ResourceGroupName $storageresourcegroup

Get the access for that account via

$key = (get-azstorageaccountkey -ResourceGroupName $storageaccount.ResourceGroupName -Name $storageaccount.StorageAccountName).value[0]

Get the context via:

$context = New-AzstorageContext -StorageAccountName $storageaccount.StorageAccountName -StorageAccountKey $key

get the actual container via:

$storagecontainers = get-azstoragecontainer -Context $context

It will then build an array of all the objects in that container. It will then cycle through all these items changing their tier level via:

$blob.icloudblob.SetStandardBlobTier($StorageTier)

This therefore effectively changes all the items in the container to the tier level you select. This is why I like to set up containers for specific tiers rather than intermingling.

Just remember to run this script AFTER you upload your files to swap them to the cheaper Archive tier. You could also use this script to swap them back at a later stage if you need.

Moving to the Cloud–Part 2

This is part of a multi part examination of the options of moving to the Microsoft cloud. If you missed the first episode, you’ll find it here:

Moving to the Cloud – Part 1

which covered off setting up a site to site VPN to Azure.

The next piece of the puzzle that we’ll add here is storage.

Storage in the Microsoft cloud comes in many forms, SharePoint, Teams, OneDrive for Business and Azure. We’ll get to stuff in Microsoft 365 like SharePoint, Teams and OneDrive later, but to start off with we want to take advantage of the site to site VPN that was set up in Part 1.

In Azure there are three different access tiers of storage; hot, cool and archive. They all vary by access speed and cost. The slower the access, the cheaper it is. Hot is the fastest access, followed by cool, then archive. You can read more about this here:

Azure Blob storage: hot, cool, and archive access tiers

The other variable here with Azure storage is the performance tier; standard or premium. You can read more here:

Introduction to Azure storage

Basically, standard performance tier uses HDD while Premium uses SSD. Apart from performance, the major difference is how the storage cost is actually calculated. With the standard tier, you are only billed for the space you consume BUT you are also billed for access (read, write, delete) operations. With premium, you are billed for the total capacity of the storage you allocate immediately BUT, you are not billed for any access operations.

So the key metrics you need to keep in mind when you are designing a storage solution in Azure is firstly the access tier (hot, cool or archive) the performance tier (standard or premium) and the capacity you desire for each. You may find some combinations are unavailable, so check out the document linked above for more details on what is available with all these options.

The easiest approach to Azure storage is to create an Azure SMB Share and map these directly on a workstation which I have previously detailed here:

Creating an Azure SMB Share

as well as an overview on pricing:

Clarification on Azure SMB file share transactions

Azure SMB files currently only supports hot and cool tiers. You can use archive storage but only via blob access, not SMB files. So what good are all of these you may ask? Well, if you read my article:

Data discovery done right

You’ll find that I recommend dividing up your data into items to be deleted, archived and to be migrated.

So we need to ask ourselves the question, what data makes sense where?

Let’s start with Azure archive storage. What makes sense in here, given that Azure archive storage is aimed at replacement of traditional long term storage (think tape drives)? Into this, you want to put data that you aren’t going to access very often, and that doesn’t make sense going into Teams, SharePoint and OneDrive. What sort of data doesn’t make sense going into SharePoint? Data that can’t be indexed such as large image files without text, Outlook PST backups, custom file types SharePoint indexing doesn’t support (think some types of CAD files and other third party file types). In my case, Azure archive storage is a great repository for those PST backups I’ve accumulated over the years.

Here is the guidance from Microsoft:

Hot – Optimized for storing data that is accessed frequently.
Cool – Optimized for storing data that is infrequently accessed and stored for at least 30 days.
Archive – Optimized for storing data that is rarely accessed and stored for at least 180 days with flexible latency requirements (on the order of hours).

We now repeat this again but with the cool tier storage, remember that this tier now directly supports Azure SMB files. So, what makes sense here? There is obviously no hard and fast rule but again, what doesn’t make sense going into SharePoint? Stuff that can’t be indexed, is typically large, is not accessed that often but more often than archive storage AND you also want to be accessible via a mapped drive letter. In my case, that data that springs to mind are my desktop utility apps (like robocopy), ISO images (of old versions of SharePoint server I keep in case I need to do a migration) and copies of my podcast recordings in MP3 format.

We repeat this again for the hot tier which is fastest and most expensive storage option. Initially here I’m going to place the user profile data when I get around to configuring Windows Virtual Desktop (WVD) in this environment. That needs to be quick, however most other current data files I have will go into Microsoft 365. Being the most expensive tier of storage, I want to keep this as small as possible and only REALLY put data on here that makes sense.

You don’t have to use all three tiers as I do. You can always add more storage later if you need to, but I’d recommend you work out what capacity you want for each tier and then implement it. For me, I’m going for 100GB Archive, 100GB cool and 50GB hot as a starting point. Your capacities will obviously vary depending on how much data you plan to put in each location. That why you need to have some idea idea where all your data is going to go BEFORE you set all this stuff up. Some will go to Azure, some will go to Microsoft 365, some will deleted and so on.

As for performance tiers, I’m going to stick with standard across all storage accounts for now to keep costs down and only pay for the capacity I actually use.

Let’s now look at some costs by using the Azure pricing calculator:

I’ll firstly work out the price for each based on 1TB total storage for comparisons between the tiers and to SharePoint and OneDrive for Business.

All the storage calculations are in AU$, out of the Australian East data center, on the standard performance tier and locally redundant unless otherwise stated.

You can see that 1TB or archive storage is only AU$2.05, but it ain’t that simple.

There are other operations, as you can see above that need to be taken into account. I have adjusted these to what I believe makes sense for this example but as you can see, variations here can significantly alter the price (especially the read operations).

The estimated total for 1TB of archive storage on the standard performance tier = AU$27.05 per month.

Now, as a comparison, if I change the performance tier to Premium I get:

The price of the storage goes way up, while the price of operations goes way down. So, if you want to minimise costs and you have lots of operations on your storage, standard tier is your best option.

The estimated total for 1TB of archive storage on the premium performance tier = AU$224.22 per month.

Basically 10 x the cost above the standard tier.

In my case, I don’t need 1TB of storage, I only want 100GB of storage.

When I now do the estimation of 100GB of archive storage, the cost of just the storage falls by 10x (as expected) to AU$0.20, Don’t forget however about the storage operations which remain the same. So, my storage cost went down but my operation costs remained the same. Thus,

The estimated total for my 100GB of archive storage on the standard performance tier = AU$25.95 per month.

While premium is:

The estimated total for my 100GB of archive storage on the premium performance tier = AU$22.78 per month.

As outlined before, as a general rule of thumb with archive storage, premium performance tier is better value for low storage capacity and also low data operations. Once the capacity increases with premium performance, the price ramps ups.

So why would I recommend staying with the standard performance tier? Although, I ‘estimate’ that my archive will be small, I want the flexibility to grow the capacity if I need it. Remember, that we don’t set a storage capacity quota for block storage, it can just grow as needed and the bigger the storage capacity the more it will cost me if I go premium. Given that storage capacity here is more important than working with the data, I want the cheapest storage costs I can get as the data capacity increases. Thus, I’ll stick with the standard access tier. Also, remember that I’m estimating when my storage reaches 100GB here I’ll be billed AU$25.95 per month but until I reach that capacity and the less operations I do on files there, the cheaper this storage will be. I therefore expect my ‘real world’ costs to in fact be much less than this AU$25.95 figure over time.

Let’s now look at the next two storage locations, which will be Azure SMB file shares.

Unfortunately, the pricing calculator doesn’t allow us to easily calculate the price for an SMB Share on a cool access tier (Azure SMB files doesn’t currently support being on the archive tier). However, the pricing is only an estimate, so I know if I place it on a cool access tier it will be cheaper anyway, so I’m going to keep it simple.

Thus, for reference:

The estimated total for 1TB of SMB file storage on the standard performance tier = AU$106.58 per month.

remembering that for the standard tier we need to take into account the cost of operations as shown.

and for Premium:

The estimated total for 1TB of SMB file storage on the premium performance tier = AU$348.00 per month.

With premium storage, you don’t need to worry about operations, however don’t forget, if you go premium you’ll be paying for the total allocated capacity no matter how much you are actually using. Thus, I’ll again be sticking with standard storage.

So, for my 50GB Azure SMB files hot tier I calculate the following:

The estimated total for my 50GB of hot SMB file storage on the standard performance tier = AU$32.40 per month.

Now how can I get an idea of what the cool SMB file price will be? Although it is not this simple, I’m going to use a ratio from:

Azure Block blob pricing

So, by my super rough rule of thumb maths I get:

cool/hot = 0.02060/0.0275 = 0.75

Thus, cool storage is 75% the cost of hot storage say.

The estimated total for my 100GB of cool SMB file storage on the standard performance tier = AU$32.40 per month x 2 x 0.75 = AU$48.60 per month

The 2 x is because the hot price I have is only for 50GB and I want 100GB of cool storage.

In summary then, I will create 3 x storage repositories for my data:

– 100GB blob archive storage = AU$25.95 per month

– 100GB SMB file cool storage = AU$48.60 per month

– 50GB SMB file hot storage = AU$32.40 per month

250GB total storage estimated cost = AU$106.95 per month

Again remember, this is my estimated MAXIMUM cost, I expect it to be much lower until the data capacities actually reach these levels.

Now that I have the costs, how do I actually go about using these storage locations?

Because archive storage is blob storage I’ll need to access it via something like Azure Storage Explorer as I can’t easily use Windows Explorer. I’m not expecting all users to work with this data so Azure Storage Explorer will work fine to upload and manipulate data if needed by a select few.

As for the SMB file cool and hot storage I’m going to map these to two drives across my VPN as I have detailed previously:

Azure file storage private endpoints

This means they’ll just appear as drive letter on workstations and I can copy data up there from anything local, like a file server. The great thing is that these Azure SMB file shares are only available across the VPN and not directly from elsewhere as the article shows. That can be changed if desired, but for now that’s they way I’ll leave it. I can also potentially get to these locations via Azure Storage Explorer if I need to. The flexibility of the cloud.

So far we now have:

– Site to Site VPN to Azure (<5GB egress from/unlimited ingress to Azure)= $36.08 per month

– 100GB blob archive storage = AU$25.95 per month

– 100GB SMB file cool storage (mapped to Y: Drive) = AU$48.60 per month

– 50GB SMB file hot storage (mapped to Z: Drive) = AU$32.40 per month

Total maximum infrastructure cost to date = AU$143.03 per month

So we now have in place the ability to start shifting data that doesn’t make sense going into Microsoft 365 SharePoint, Teams and OneDrive for Business. Each of the three new storage locations has their advantages and disadvantages. That is why I created them all, to give me the maximum flexibility at the minimum cost

We continue to build from here in upcoming articles. Stay tuned.

Azure file storage private endpoints

I’ve previously detailed how to create an Azure SMB File Share:

Creating an Azure SMB file share

as a way to create a ‘cloud USB’ drive that you can map to just about any desktop quickly and easily. All of this is accomplished securely but many remain hesitant to do this across the Internet directly. Luckily, there is now an option to map this SMB share to an IP address inside an Azure VNet to restrict access if desired.

Before you set this up you will need to have an existing Azure Vnet created as well as a paid Azure subscription. You can add a Private Endpoint to an existing Azure storage account or create one at the same time you create a new Azure Storage account. In this case, I’m going to an existing account.

In the Azure portal search for “private link”, which should then take you to the Private Link Center as shown above. Select the Add button on the right.

You’ll need to select a Resource Group as well as a Name as shown above.

You’ll then to select the Azure Storage account and the file option to connect to an existing SMB file share as shown above.

Next, you’ll need to connect to an existing Vnet and if you want to access the resource privately by a name, then you’ll need to integrate it with a private DNS zone, which will also be set up for you as part of this process.

You can then add tags. Note – when I created mine, if I assigned tags here I couldn’t create the Private Endpoint, which appears to be a bug. So, if for some reason you find the same issue, create the Private Endpoint without tags and then add them later.

With all that done, select the Create button to finish the configuration on the Review + Create page.

When the set up process is complete you’ll now see your endpoint as shown above with an allocated IP address on the Vnet you selected.

If you then look at your Vnet, as shown above, you will see that the Storage Account is seen as a connected device.

If you now visit the Storage Account and select Firewalls and virtual networks as shown above, you can configure what networks can access this new Private Endpoint.

Leaving the option set to All networks means that you can still map to that SMB share directly across the Internet, which you may want.

However, in the above case, I have selected to restrict the access to the Vnet only.

Doing so means that the ONLY way I can now access that SMB Share is via the selected Vnet. I can’t get to it using the Azure portal on my remote desktop machine as shown above.

If I wanted to access this from a remote location, outside the Vnet across the Internet, I could add those details below. However, I have chosen not to do this.

My Azure SMB File share now has a dedicated IP address that is restricted to access via an Azure Vnet, how do I work with this share directly on premises? Easy. I set up an Azure Site to Site VPN to that same Vnet and now I can access that Azure SMB File share from my local machines by mapping to something like the IP address.

Thus, the only way that Azure SMB file share can be access is across a Site to Site VPN, making even more secure.

Private Endpoints support connection to a number of PaaS Azure services as shown above. This is handy as it allows you to connected you Azure IaaS services (like VMs) directly to Azure PaaS (like storage) quickly and easily as shown. What’s the benefit? Remember, IaaS is typically billed on time used, while PaaS is billed on resource consumption. Thus, why should I pay for a VM to store my data and pay the time it runs (typically 24/7), plus disk storage where I could use Azure Storage and most be billed just for the data capacity?

PaaS is the future and has many benefits over IaaS. You should be looking to shift as much of you infrastructure to PaaS to take advantage of things like reduce maintenance, cost savings, etc. Private Endpoints is an easy way to start doing just that. For more information on Azure Private Endpoint visit:

What is Azure Private Endpoint?