Azure Storage Account Diagnostics (classic) hidden Data

Looking at Azure Storage Accounts a little bit in more detail, I noticed something that was surprising to me.

Connecting to a storage account via the Microsoft Azure Storage Explorer (at least version 1.12.0 or 1.13.0) and looking at the content, I noticed a Blob Container named $logs and some tables named $MetricsHourPrimaryTransactionsType and $MetricsCapacityBlob.

Storage Account Metrics Tables

Searching the Internet lead me to the description of the Azure Storage analytics metrics (Classic), which describes these tables.

What I did not knew: By default, diagnostics is turned on for hour metrics for all subservices (blob, table, …) when creating a storage account. The popup information of the portal states that this is the case for storage accounts created via the portal. The docs do not mention this restriction, means metrics (classic) are enabled no matter how you create the storage account. From what I saw the diagnostics were at least also enabled when using Azure CLI to create a storage account.

Storage Account Default Diagnostics Settings

Why hidden?

Well, the answer is quite simple. I was only able to see these tables and the blob container using the Azure Storage Explorer client app. They were not listed in the portal, by the Storage Explorer preview inside the portal, not returned by PowerShell’s Get-AzStorageTable, C# CloudTableClient.ListTables or Azure CLI az storage table list.
Unfortunately, at the time of writing, the source code of the Microsoft Azure Storage Explorer was not open (see this issue at GitHub), so I was not able to figure out what the explorer is doing to also show theses tables.

Disable using ARM Template, PowerShell, or Azure CLI?

I thought it might be a good idea to disable diagnostics automatically by either using an Azure Resource Management (ARM) template, Azure CLI or PowerShell.

Unfortunately, at the time of writing, it seems like this is not possible, at least not the way I would prefer it.

You can use Set-AzStorageServiceLoggingProperty or an ARM template to enable / disable the metrics for each subservice (e.g. blob, table, …, see this GitHub issue), but you cannot disable entire diagnostics programmatically. This is only possible via the portal.

Cost

Seems like Microsoft is trying to earn some extra money 😉 You will be charged for the storage required by the analytics data and some transactions. For details, please refer to Billing on Storage Metrics.

Delete outdated Data

You can define a retention time for the diagnostics data. Unfortunately, disabling the diagnostics seems to also disable the housekeeping. Means, outdated data that was created before disabling diagnostics will not be deleted. One can delete the rows manually using the Azure Storage Explorer. But this doesn’t seems like a real solution to me.

Suggestion

In case-you do not want / need to analyze the access details on your storage accounts, turn off the diagnostics as soon as you created the account. At the time of writing it seems the only way to do so is manually via the portal.

Before disabling it, think twice if some critical / sensible data is stored by the storage account which requires it to be able to analyze the access history.

Links

Azure Storage Account Overview
Azure Storage analytics metrics (Classic)
Microsoft Azure Storage Explorer download
Microsoft Azure Storage Explorer GitHub issue “Where is the source code? #138”
Set the storage service logging level via PowerShell Set-AzStorageServiceLoggingProperty
GitHub issue “how to configure the storage account’s diagnostics logs using an arm template. can you please point me to any doc on that info #13195”
Billing on Storage Metrics