Sources of billing information and export methods of billing data

In this section, we will go through several basic concepts of cloud cost representations and ways of getting cost information, with the related pros and cons.

As mentioned, before jumping into each of the recommended ways of checking your cloud spending, you need to take a little time to clarify what are the targets you want to achieve (it's very useful to also involve the financial controller team of your company in this).

You could start with some of the main questions you typically have to answer, as follows:

Question 5: Do you need a technical representation of the costs, a business representation, or both?

A technical representation is easily achievable since it is based on the infrastructure you've built on Azure and represents the costs for a technical team. A business view of the costs should link the costs to a business object (for example, an application or a service to the customer) that may be composed of different Azure resources. Tagging cloud resources is an important part of this representation and will be covered in the upcoming chapters.

Long story short, you may think about the technical representation as how much we spent on storage, grouped by storage type and storage transactions (typical for a cloud service team) and about the business representation, as we have four applications in our Azure infrastructure, as how much does application A cost on Azure, considering all the Azure resources involved?

Question 6: Which level of detail do you need to reach? You can choose from the following:

• High level: Is it enough to know that VMs cost X, licenses cost Y, and an internet of things (IoT) hub costs Z?
• Detailed: I need to know the costs of every component—for example, I need to know the costs of the storage, split into the space used and access transactions.

Question 7: Is the Azure Cost Management tool enough (the filtering, grouping features, drill-down capabilities, and so on) for your purposes?

Once you have grasped the basic concepts of Azure billing and what information is available with the included tools, you should be able to understand if a different tool is needed for your company.

Question 8: Do you need to work on the raw data to add business value or maybe integrate into the company dashboarding and a total cost of ownership (TCO) control system?

This will again depend on how your organization usually maps, displays, and accounts for IT costs. You might already have a billing dashboard, and integrating the Azure information into your own portal or view is the right way of adding cloud spend to your overall picture.

Question 9: Is the Azure technical terminology right for your audience?

For example: Does your audience know what a storage transaction is, or the difference between bandwidth and inputs/outputs per second (IOPS)?

If you need to export costs to a non-technical audience, you will need to aggregate costs in higher views with custom labels and forget the Azure Cost Management tool, or plan to rename technical terminology with a custom dictionary for non-technical users (for example, general-purpose storage, LRS (locally redundant storage), and (GRS) (geo-redundant storage)) into more company-friendly labels such as storage, local storage, and geo-redundant storage.

Having briefly covered how to organize your subscription, and the importance of your hierarchy and initial setup to successful cost governance, one of the first steps to establish full control is to export, analyze, and save your Azure billing information.

Export methods

Keep in mind that this could not be a one-size-fits-all approach: you should identify which case is better for you and your organization. And the answer may change over time, prompting you to rework and evolve the original approach.

Let's now see the different ways you can get your cost insights.

The Azure Cost Management tool

This is the right tool to start learning about the terminology and how Azure allocates costs and what are the main cost-splitting criteria or cost categories you can find.

The Azure Cost Management tool is integrated into your Azure account and it's ready to use.

You can find the tool in Azure by searching for Cost Management + Billing in the search bar of the Azure portal (https://portal.azure.com).

You can reach the analytics graphs by clicking on Cost analysis on the left pane, as illustrated in the following screenshot:

Figure 1.18 – Azure portal: Cost analysis

The default view is a cumulated view of the current month.

Important Note:

The Azure portal features and views are constantly changing; therefore, you might find your screen a bit different from our screenshots. However, the contents and capabilities are typically only added, so the information will be consistent with ours at the time of writing.

Using the controls in the zone above the graph, you can add filtering clause(s) and change the grouping criteria, the Granularity setting, and the rendered graphs, as illustrated in the following screenshot:

Figure 1.19 – Azure portal: Cost analysis (Accumulated option)

In my experience, the most useful filtering and grouping criteria to start with are these:

• By subscription (remember that the best practices suggest you split different business services/applications into different subscriptions, as we will see in the next chapter): For example, Production, Disaster Recovery, and so on.
• Service name: Simply, the service you bought from Azure. Every service has different MeterCategory types.
• Meter category: Let's call it cost-Macrocategory. It identifies the high-level cost type according to how a specific service is billed (for example, VMs, disks, storage account, IoT hub, Structured Query Language (SQL) servers, SQL databases, and so on).
• Meter subcategory: Useful for a detailed drill-down of cost analysis, it identifies different cost types inside MeterCategory (for example, MeterCategory Disks has different subcategories such as provisioned IOPS, provisioned bandwidth, storage, and so on).
• Resource group: This allows you to filter the cost representation for a slice, reflecting the cost of one or more resource groups. If you implemented resource groups by application, you could have a first very interesting high-level cost split. Please also note that the cost analysis pane is also available directly on the specific resource group page.
• Tag: This allows you to filter based on custom tags applied to the instances. The prerequisite of this view is that you've already implemented a tagging policy, otherwise you may lose instances (and costs!) in the report. We will dig deeper into tagging in the next chapter.

If you are a reseller or provider (for CSP contracts), additional scopes are available to your analysis, such as the following:

• Billing account will help visualize billing (pre-tax) for all customers and billing profiles.
• Billing profile will help visualize billing (pre-tax) of a billing invoice that can then be filtered by customer or InvoiceID.
• End customer will display all the costs (pre-tax) associated with a specific customer.

  Tip:

  Additional information can be found here: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/get-started-partners.

In the following screenshot, you will find three pie charts that enable you to have three different views for three additional data dimensions. For example, you may have filtered by virtual machines in the main graph and have a different drill-down for Service name, Location, and Enrollment account name:

Figure 1.20 – Azure Cost Management (Cost analysis)

Hint: Azure cost control is a two-dimensional (2D) tool, and the pie charts add information but can only represent a static view and not a trend, so they might not help you in identifying unplanned extra costs if you have very dynamic cloud usage. We will learn how to deal with this later in the book.

To identify what is going on with your Azure infrastructure and costs and represent trends and spikes, you need to represent the costs on a timeline, therefore using the main graph area and switching between different visualizations to obtain a more comprehensive way to represent the same data, as follows:

• Filtering by meter category, grouped by resource group, to identify which resource type is more expensive in which resource group.
• Filtering by resource group, grouped by meter category, to identify, inside the resource group you chose, the resource type that is causing any extra costs.
• You can now filter by resource group, filter by meter category, and group by resource to identify the instance(s) that are causing any extra costs.

You should always consider switching graph filtering/grouping due to the representational limits of the main graph until you identify which resource is causing cost anomalies.

In the following subsections, we'll work on different visualization types for the cost analysis that will get you up to speed with fully understanding the cloud bill and all the related details.

Azure data visualization types

Using the rightmost dropdown in the Azure Cost Management main graph, you can change the graph type between the following options:

• Line chart: Trend is represented by a line.
• Area chart: Trend is represented by a colored area.
• Column (stacked): Each item for the grouping criteria is stacked on top of the other.
• Column (grouped): Each item for the grouping criteria is presented side by side.
• Table

An example of an Area chart is provided here:

Figure 1.21 – Azure portal: Cost analysis (Accumulated area view)

This visualization is useful to display a trend, and specifically in the Accumulated view, since it allows you to identify if the cost is linear (for example, X Euros (€) per day, for every day) or if there are spikes or unplanned extra-consumption.

Now, let's look at an example of a Column (stacked) bar chart:

Figure 1.22 – Azure portal: Cost analysis (stacked bar daily view)

This visualization is very useful for period-by-period comparison (for example, with Granularity: Daily, to plot every day in the chosen period).

In the previous screenshot, for example, we can identify and compare the costs of VMs for every resource group; it's very easy to identify the spike and the resource group responsible for the spike.

Spending forecast

Another very useful feature of Azure Cost Management is the analysis of your Azure consumption in the previous period and a calculation of your spending forecast.

The calculated spending for the upcoming future and until the next invoice is typically displayed in a shaded color and will represent your forecasted spending of a future timeframe until the next invoice. Please note you will need at least 10 days of spending data for a forecast to be generated.

Azure and AWS costs

Azure Cost Management allows customers who have a multi-cloud environment to display Amazon Web Services (AWS) costs in the same format as Azure spending. This is done via a management group configuration and will allow consistency in billing reports—for example, to your internal departments, regardless of the cloud used.

For more information about how to configure Azure Cost Management to import AWS account billing information, please refer to the official documentation at https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/aws-integration-set-up-configure.

Amortized versus actual cost

As you open the Cost Management tool, the default view is the ACTUAL COST view, with all the costs and usage made only in the selected period, as illustrated in the following screenshot:

Figure 1.23 – Azure portal: Cost analysis (ACTUAL COST)

Important Note:

Please note that the actual cost will still be a few hours behind the actual resource usage, as mentioned earlier.

If you click on Actual cost, you are presented with an option of changing it to Amortized cost. You can switch to an amortized view by clicking on Amortized cost from the dropdown, as illustrated in the following screenshot:

Figure 1.24 – Azure portal: Cost analysis (Amortized cost option)

If you've previously bought upfront reservations, you'll see that the amount will change according to the calculated monthly cost of the upfront reservations.

The following example will clarify this topic.

Let's say you bought, 3 months ago, a VM 1-year pre-paid reservation with a cost of 100 US Dollars (USD) per month, and an upfront payment of 1,200$. The actual and amortized view will be as follows:

• The actual view will show a spend of 1,200$ the day you bought the reservation.
• The amortized view will show a spend of 100$ each month from the month you bought the reservation up to the chosen timeframe.

We will dig deeper into reservations in the following chapters, but for now, this is only needed to understand the Azure portal's different views.

All costs versus usage only

Another very useful filter when you need to separate the cost representation from reservations (both upfront and monthly) is the Charge type filter, which allows you to choose between the following:

• refund: If you refunded a previously bought reservation, you can isolate the refund credit with this filter.
• purchase: All the purchases you made during the period (for example, licenses, reservations both upfront and monthly, and other services).
• usage: The net resource usage, without any kind of reservation purchase(s).

The following screenshot shows the same:

Figure 1.25 – Azure portal: Cost analysis (charge type)

Data export

Every company has its own deck of slides or reports in which you're asked to insert insight and evidence about cloud spending, so you need to export data, tables, or images, and import them into your deck.

There's no need to use the snipping tool: the Azure Cost Management page allows you to export data and graphs in Excel, comma-separated values (CSV), and Portable Network Graphics (PNG) image formats.

You just need to click on the Download icon in the upper bar, as indicated in the following screenshot:

Figure 1.26 – Azure portal: Cost analysis (Download data)

An option section will then appear on the right, as illustrated in the following screenshot:

Figure 1.27 – Azure portal: Cost analysis (Download data options)

Here, you can choose the export method (PNG, Excel, or CSV).

Downloading raw data

In the previous method, we learned that the Azure cost control tool is already available for all customers and allows you to understand your costs and the meaning of the various terms such as Service name, Meter category, Meter subcategory, and so on.

This will allow you to explore the costs, but not to integrate this into any of your company's billing dashboards or cost control strategies. Furthermore, Azure cost control does not allow you to customize the data visualization, aggregate service names, or rename labels to export a shared view to management or the cost control division.

Fortunately, there is a way, and Azure has made it quite simple for you. When you download your report file in CSV format, you'll see a complete breakdown of all the charges that were invoiced, per used service.

You can export the raw data in a standard .csv file format, open it with your preferred software, and elaborate upon your personal cost report.

For example, you can aggregate VMs, licenses, disks, and NICs in one macro-category called virtual machines (in the Azure Cost Management tool, you are bound to the predefined categories).

Having all the raw data, it's completely up to you how to aggregate the costs and how to chart them.

Important Note:

Understanding Azure costs starting from the raw CSV data can be difficult for technical people but practical for a cost controller. Consider practicing using Azure Cost Management first, and then double-check your hypothesis on the raw data with Azure Cost Management to ensure your understanding is correct and you're exporting the right cost representation.

To export the raw data, go to the Azure Cost Management page, select the Usage + charges blade in the right menu, and then download the CSV file with the rightmost download icon, as illustrated in the following screenshot:

Figure 1.28 – Azure Cost Management: Usage + charges view

If you have an analytics pipeline or data warehouse for cost analysis, you can download CSV files periodically and load them into a cost database to integrate the Azure costs in your cost control system, and implement your personal representation, aligned with your company standard.

Important Note:

Viewing and downloading the cost report in the middle of the billing month period allows you to better control costs (since you must optimize your spending on a daily basis, you should not wait until the end of the month) but I strongly advise you to not to count on the month-to-date export to build the monthly cost by adding each delta, since this may not be so reliable.

A daily export is extremely useful for identifying issues and spikes and behaviors, but the best way to load the final stable view for the monthly spend (corresponding to the invoice) is to wait until at least the second—or, better—the third day of the following month (for example, consolidated, stable raw data from January will be available on February 3). This is due to a delay in processing Azure consumption and different time zones. In my personal experience, the third day of the following month is safe.

Automatically exporting daily costs

If you want to integrate all Azure costs in your company cost control pipeline, a manual daily export to ingest in your cost control system is not practical: you need to automatically have the costs imported into your system.

Azure helps you with the Exports feature, which allows you to schedule an automatic job that exports the CSV file in a storage account. Your cost control system should only download the last CSV file from that storage account and ingest the data automatically.

Just go to the Exports blade—here, you can see the already defined export job or create a new job, as illustrated in the following screenshot:

Figure 1.29 – Azure Cost Management (Exports)

To create a new job, you have to click on the + Add button, and then fill out the form displayed, as follows:

Figure 1.30 – Azure Cost Management (Export options)

Once you named your job, the most important things to set are these:

• Metric: Actual costs (without upfront reservations) or amortized costs (with the monthly cost of the upfront reservation)
• Export type: Daily export (month-to-date); weekly report (last 7 days), monthly report (last month's costs); one-time report

Other fields let you point the export job to the right storage account, using an existing one or creating a new one, but the most important ones are these two.

Hint: I suggest implementing two exports, as follows:

• Actual costs; daily export: To have Azure consumption with monthly reservation and Azure usage costs
• Amortized costs; daily export: To have Azure consumption with upfront reservation too, in order to know (by difference) the amount of upfront reservation monthly

  Important Note:

  With your tagging strategy if you need to represent costs by custom tags, you'll find a field named Tags in the exported CSV file, with a JavaScript Object Notation (JSON) data structure with all the tags (the ones you added and other service tags, normally hidden).

Using the command-line interface

Another way to get Azure detailed costs is the command-line interface (CLI) (both the PowerShell and the Linux AZ CLI). This is a very interesting option to easily generate reports that you can send to key people, directly with simple batch files, without having to struggle with exporting from Azure, importing the CSV file into a database, and configuring a business intelligence (BI) tool.

Tip:

To install the previously mentioned tools, please refer to the official documentation for the Azure CLI (https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) and PowerShell (https://docs.microsoft.com/en-us/powershell/azure/install-az-ps?view=azps-6.5.0).

Please note that using the Azure CLI (fired within the web browser, from the portal) will incur costs derived from the storage you use and the data transfers, for it to work. Using the Azure Cloud Shell will always have the latest tools available for Azure CLI and PowerShell. Since you need to login through the browser to use the cloud shell, your session is always authenticated. There is a very minimal charge for the static data stored in directories in cloud shell, which uses a storage account in Azure.

The documentation starting points are listed here:

• AZ CLI—Billing (https://docs.microsoft.com/en-us/cli/azure/costmanagement): Lets you manage invoices, billing profiles and details, configure roles for users, list products for billing account, and so on
• AZ CLI—Consumption (https://docs.microsoft.com/en-us/cli/azure/consumption): Lets you manage usage details, reservations, and budgets, and view price sheets
• AZ CLI—Cost management (https://docs.microsoft.com/en-us/cli/azure/costmanagement): Lets you manage Azure Cost Management exports (refer to the Automatically export daily cost section)
• PowerShell (https://docs.microsoft.com/en-us/powershell/module/az.billing)

Though the official documentation may point out a lot of different parameters and queries for managing the cost control by command line, I will focus your attention on a couple of commands for extracting and managing the usage details.

Let's focus on the consumption subcommands, which allow us to download information on currently generated costs on our account.

Here are some AZ CLI examples.

To get Azure usage details for a specific period (a day or a month in the past), you can use the following command:

az consumption usage list --include-meter-details --include-
additional-properties --start-date 2020-10-01 
--end-date 2020-10-02

This command will return JSON with useful information (among other useful information) such as the location, instance ID, subscription, resource group, and net costs (without taxes). This command is supported only with EA, Web Direct, and MCA offer types.

Important Note:

If you prefer to have the output in a tab-separated values (TSV) format, you may add --output tsv in the command line, but you'll lose any nested information. Please try executing the command for the JSON and TSV formats and compare the output: you'll find that the JSON format is far more detailed than the other one.

Important things to keep in mind are listed here:

• The command line allows you to have the net prices.
• The command line allows you to specify a period start/end date.
• The command line allows you to get the top N spending resources.

So, it's very useful to have a quick overview of the costs and to work directly in batch scripting, but it's not comprehensive and it does not represent the final invoice.

Please note that on the tagging strategy, you'll find a JSON element with all the instance tags. This information is lost in the TSV format.

Using application programming interfaces

If you want to better control data ingestion, instead of downloading a CSV and ingesting it in your cost control system, you can do it programmatically by using application programming interfaces (APIs). Microsoft has different APIs that let you download the billing details, forecast, budget, and so on.

The main documentation entry point is https://docs.microsoft.com/en-us/rest/api/consumption/: this is where you should start reading documentation about REpresentational State Transfer (REST) entry point(s) you need to invoke to get useful information about your billing, such as the following:

• Usage details—the full CSV or JSON structure for the billing of the desired period (this allows you to customize the period start and end dates)
• Reservation details
• Pricelist
• Budgets
• Forecasts (only for Enterprise customers)

In addition, at the https://docs.microsoft.com/en-us/rest/api/billing/ link, you will find information about other operations on billing objects (for example, transfer billing ownership, payment methods, and so on). Usually, if you're searching for a way to control your costs and integrate the costs in your pipeline, this is not the first point from where you should start.

Important Note:

If you searched online, you may find old Reporting APIs (https://docs.microsoft.com/it-it/rest/api/billing/enterprise/billing-enterprise-api-usage-detail). Despite being officially discontinued, they still work, but we recommend avoiding these APIs for building an injection.

As you can see from the official documentation, using APIs allows you to deeply integrate billing data into your dashboard and cost management software, but it may be a little complex at the beginning, considering you don't know what to expect from each API and you'll probably end up searching in different exports for missing billing details you're desperately seeking.

Question 10: But APIs are a bit complex—why are they so useful?

Unfortunately, there is no simple answer, but APIs are generally very smart for integrating analysis, feedback, and decisions based on an external tool. Just think about an enterprise that uses an integrated authorization flow for any action that could generate costs: creating a VM will generate costs, adding a disk generates costs, creating a new SQL database or an Azure Kubernetes cluster generates costs, and all should be authorized internally.

Your company probably has a centralized governance team that can provide users with internal processes that automate the provisioning of cloud resources. So, everyone should compile some form in a software tool to ask and wait for approval, which will then trigger deployment. The tool will use APIs to start the resource creation process in an automated fashion.

The same applies to buying reserved instances or to cost accounting inside the enterprise: you might need to extract costs based on the department or management group.

Important Note:

You cannot perform this kind of operation without using APIs: it's the only way to integrate Azure actions and extractions, programmatically, to an already existing software (generally an enterprise-scale software).

In the end, please keep in mind that Microsoft itself suggests you use the method we discussed in the Automatically exporting daily costs section for large data handling in its official documentation, which can be found at https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation.

In conclusion, my personal suggestion is to start from the first method—the Azure Cost Management online web page—to better understand the terminology and how costs are grouped and exported by Azure.

The next step will be downloading a full monthly invoice, opening it with your preferred spreadsheet, and studying each row, eventually pivoting and comparing what you find in the CSV file and what you've found with the Azure Cost Management tool. Then, you can import the CSV file into a database (a simple open source database is more than enough) and start querying the database, grouping results, and constructing tables to translate the technical terms into something more understandable for non-technical people (such as cost controllers or C-levels).

One more step in automating cost control ingestion is to schedule one or more exports and download the file(s) from the target storage account.

The last, more complex but extremely flexible way to integrate cost management and implement a feedback loop is to use the Consumption and Billing APIs. Once you have your fully automated database ingestion, custom tables, and whatever you need to customize cost representation, you can build your own BI dashboard on it and master cost control.