This blog is part 2 of a series that covers relevant Azure fundamentals - concepts/terminology you need to know, in the context of Hadoop. Much off the content is a copy of Azure documentation (full credit to the Azure documentation team). I have merely compiled relevant information into a single post, to create a one stop shop for those new to Azure and thinking Hadoop.
Special thanks to fellow Data Solution Architects, Thomas Abraham and Ryan Murphy for their review and feedback.
Here's what's covered in the post:
Section 05: Azure storage at a high level
Section 06: Azure blob storage
Section 07: Azure disk storage options at a high level
Section 08: Azure managed disks
5. Azure Storage
Azure has many offerings from a storage perspective. This section will touch on ones relevant from the Hadoop perspective.
Azure storage - offerings:
Azure Storage offers non-relational data storage including Blob Storage, Table Storage, Queue Storage, and Files.
Only those relevant to the Hadoop context are covered in this section.
6. Azure Blob Storage
Is Azure's object store PaaS service and includes three types - block blobs, append blobs, page blobs.
6.1. Blob Service Components:
Is an entity within which your provisioned Azure storage sits. All access to storage services takes place through the storage account. The storage account is the highest level of the namespace for accessing each of the fundamental services. It is also the basis for authentication.
A storage account has limits - IOPS limits, throughput limits, and a subscription has number of storage accounts limit. This details scalability limits.
This storage account can be a General-purpose storage account or a Blob storage account which is specialized for storing objects/blobs.
A container provides a grouping of a set of blobs. All blobs must be in a container. An account can contain an unlimited number of containers. A container can store an unlimited number of blobs. Note that the container name must be lowercase.
A file of any type and size. Azure Storage offers three types of blobs: block blobs, page blobs, and append blobs.
6.2. Types of blob
1. Block blob:
Block blobs are ideal for storing text or binary files, such as documents and media files.
Block blobs are relevant in the context of Hadoop. Block blobs are HDFS storage compatible and a block blob storage account can be attached to Hadoop cluster as auxiliary HDFS - see section below called "WASB"
2. Append blobs:
Append blobs are similar to block blobs in that they are made up of blocks, but they are optimized for append operations, so they are useful for logging scenarios.
3. Page blogs:
Page blobs are disk abstractions - can be up to 1 TB in size, and are more efficient for frequent read/write operations. Azure Virtual Machines use page blobs as OS and data disks. Page blobs are relevant in the context of Hadoop when you use unmanaged disks - covered further down in this blog.
6.3. Azure Storage redundancy options:
LRS: Locally redundant storage
LRS offers intra-datacenter redundancy - keeps 3 replicas of your data within a single facility within a single region for durability
GRS: Geo-redundant storage
This is the default option for redundancy when a storage account is created. With GRS your data is replicated across two regions, and 3 replicas of your data are kept in each of the two regions.
RA-GRS: Read-access geo-redundant storage
This is the recommended option for production services relying on Azure Storage. For a GRS storage account, you have the ability to leverage higher availability by reading the storage account’s data in the secondary region.
ZRS: Zone-redundant storage
ZRS fits between LRS and GRS in terms of durability and price. ZRS stores 3 replicas of your data across 2 to 3 facilities. It is designed to keep all 3 replicas within in a single region, but may span across two regions. ZRS currently only supports block blobs. ZRS allows customers to store blobs at a higher durability than a single facility can provide with LRS
6.4. Azure Blob Storage Feature Summary
- Is a massively scalable object store - structured and unstructured data
- Is highly available - three copies are maintained with customer charged for only one copy
- Offers redundancy options detailed in section above
- Offers strong consistency
- Offers encryption at rest with Microsoft managed keys or customer managed keys
- Offers tiers based on data temperature - hot cool, archive - with cost versus performance trade-off.
6.5. WASB and HDFS:
Storage accounts - Block blobs can be leveraged as auxiliary HDFS - and are supported by all Hadoop distributions for secondary HDFS. Microsoft contributed towards the Hadoop-Azure module of the Apache Hadoop project.
- Read and write data stored in an Azure Blob Storage account like it were from attached disks.
- Present a hierarchical file system view by implementing the standard Hadoop FileSystem interface.
- Supports configuration of multiple Azure Blob Storage accounts.
- Supports both block blobs (suitable for most use cases, such as Spark, MapReduce) and page blobs (suitable for continuous write use cases, such as an HBase write-ahead log).
- Reference file system paths using URLs using the wasb scheme.
- Also reference file system paths using URLs with the wasbs scheme for SSL encrypted access.
- Tested on both Linux and Windows.
- Tested at scale
6.6. Attaching a storage account to your Hadoop cluster:
You need to create a storage account, you will need to make an entry into core-site.xml - the storage account name, and associated access key, save and restart HDFS and dependent services.
You can encrypt the key for added security.
6.7. Interacting with your storage account from HDFS shell/Spark etc - WASB URI:
Always use wasbs scheme - wasbs utilizes SSL encrypted HTTPS access.
In general, you have provide the fully qualified blob storage URI as detailed in the example below.
% hdfs dfs -mkdir wasbs://
@ .blob.core.windows.net/dummyDir % hdfs dfs -put dummyFile wasbs:// @.blob.core.windows.net/dummyDir /dummyFile % hdfs dfs -cat wasbs:// @.blob.core.windows.net/dummyDir/dummyFile
"Do what you love and love what you do"
6.78 Encryption of storage accounts attached to your Hadoop cluster:
Azure offers storage service encryption (AES 256 bit) - server side, transparent encryption with Microsoft managed keys or customer provided keys secured in Azure Key Vault.
When you use wasbs scheme (https), data transfer is over SSL.
6.9. Blocking wasb scheme (http) in storage accounts attached to your Hadoop cluster:
You can configure your storage account to allow only wasbs scheme.
6.10. Blocking external access to storage accounts attached to your Hadoop cluster:
Configure VNet service endpoint to your storage account and specify subnets to allow access to and configure to not allow access from elsewhere.
6.11. Distcp to WASB
You can distcp from your on-premise cluster to your Azure storage account
% hadoop distcp hdfs://
:9001/user/ / wasbs:// @ .blob.core.windows.net/ /
6.12. Architectural considerations/best practices with using blob storage as auxiliary HDFS:
This section is specific to block blobs.
- Encrypt the storage credentials in core-site.xml; Periodically regenerate keys for added security
- Encrypt at rest with Storage Service Encryption
- Encrypt during transit - Make wasbs (/https) the only mode supported for data transfer
- Block external access with VNet service endpoint
Out of the box
Plan for replication to DR datacenter, just like you would with primary HDFS
With blob storage, size for the actual amount of data you want to store (without replicas).
E.g. Lets say you have 5 TB, with disks you would size for 5 TB * 4 (3 for replicas, 1 for other)
Choose between hot/cool/archive based on your requirements
Microsoft does not charge for the replicas, and you pay per use.
Constraints and workarounds:
Storage accounts come with IOPS limits - be cognizant of the same, and shard data across multiple storage accounts to avoid throttling and outages from the same
500 TB limit of storage account limit can be increased with support ticket
200 storage account per subscription limit can be increased with support ticket
Understand performance versus cost tradeoff with using Azure blob storage as your secondary HDFS
Understand that you will incur egress charges for data leaving your storage account.
6.13. Apache Ranger plug-in for WASB
Microsoft developed a Ranger plug-in for fine-grained RBAC for WASB. Will cover this in detail in subsequent blogs.
7. Azure Disks
Azure offers two flavors of disks you can attach to your Hadoop virtual machines - unmanaged and managed disks, in two performance tiers - standard and premium.
7.1. Unmanaged disks:
At a very high level, with unmanaged disks, you have to create a storage account, and have to deal with the constraints/limits of storage account like IOPS, it offers lower fault tolerance and a higher management overhead than its managed counterpart. I wont be covering any more on unmanaged disks - prefer managed disks over unmanaged as detailed in this video.
7.2. Managed disks:
Azure Managed Disks simplifies disk management for Azure IaaS VMs by managing the storage accounts associated with the VM disks. You only have to specify the type (Premium or Standard) and the size of disk you need, and Azure creates and manages the disk for you.
7.3. Premium Disks:
They are high performance Solid State Drive (SSD)-based storage designed to support I/O intensive workloads with significantly high throughput and low latency. There are several SKUs of premium disks and you choose the option which best meets your required storage size, IOPs, and throughput. They are supported by DS-series, DSv2-series, FS-series, and GS-series virtual machine sizes, at the time of writing this blog. There are no transaction costs for premium disks.
7.4. Standard Disks:
Standard Disks use Hard Disk Drive (HDD)-based storage media (are backed by regular spinning disks). They are best suited for dev/test and other infrequent access workloads that are less sensitive to performance variability. There are several SKUs of standard disks and you choose the option which best meets your required storage size, IOPs, and throughput. There are no transaction costs for standard disks.
8. Azure Managed Disks
8.1. IOPS and throughput
Azure managed disk SKUs come with IOPS and throughput limits to be cognizant of. The compute section will elaborate on this again. Here is an example of what to expect..
Value proposition and features..
8.2. Enhanced availability over unmanaged disks
Azure managed disks offer enhanced storage availability. Lets say if you have an availability set with multiple VMs, with managed disks, managed disks will be placed in separate storage units per VM for higher availability.
8.3. Simplifies creation of multiple VMs with same custom image
With unmanaged disks, if you have a custom image you want to propagate across multiple VMs, you have to copy the same across multiple storage accounts, as there is a cap on number of VMs per storage account - effectively an administrative overhead.
With managed disks, you can create a managed disk image, then capture a VM you need to replicate - this will capture all the VM metadata, it gets saved to the same resource group by default, you can then use it to provision multiple VMs with the same VM image. Some examples are - Cloudera Centos 7.3 image in the marketplace, a base VM image of a Hadoop slave node etc
Creating a Linux VM image - documentation
8.4. Create managed disk snapshots independent of the disks themselves
You can create a snapshot of managed disks and persist the same, and can even delete the managed disks after snapshotting.
This is helpful for backup/DR scenarios or for testing scenarios (e.g. financial month-end/quarter-end/year-end point-in-time snapshots of data)
8.5. Simplified upgrade/downgrade between standard and premium managed disks
The picture says it all - stop the VM, update storage type, reboot.
Where would this be useful:
Lets say - you build a cluster for a PoC/dev/test environment, and once done, want to upgrade that to production, and while in the lower environments, for cost optimization, you went for standard disks, you want the best performance in production with premium managed disks...
8.6. Converting from unmanaged to managed disks
No worries if you have already provisioned unmanaged disks, we have a conversion process that will switch your disks from unmanaged to managed with a few commands.
8.7. Simplifies scaling VMs
Managed Disks handles storage for you behind the scenes. Previously, you had to create storage accounts to hold the disks (VHD files) for your Azure VMs. When scaling up, you had to make sure you created additional storage accounts so you didn't exceed the IOPS limit for storage with any of your disks. With Managed Disks handling storage, you are no longer limited by the storage account limits (such as 20,000 IOPS / account). You also no longer have to copy your custom images (VHD files) to multiple storage accounts. You can manage them in a central location – one storage account per Azure region – and use them to create hundreds of VMs in a subscription.
Managed Disks will allow you to create up to 10,000 VM disks in a subscription, which will enable you to create thousands of VMs in a single subscription.
8.8. Disk-level RBAC
You can use Azure Role-Based Access Control (RBAC) to assign specific permissions for a managed disk to one or more users. Managed Disks exposes a variety of operations, including read, write (create/update), delete, and retrieving a shared access signature (SAS) URI for the disk. You can grant access to only the operations a person needs to perform his job. For example, if you don't want a person to copy a managed disk to a storage account, you can choose not to grant access to the export action for that managed disk. Similarly, if you don't want a person to use an SAS URI to copy a managed disk, you can choose not to grant that permission to the managed disk.
8.9. Azure Backup Service Support
Use Azure Backup service with Managed Disks to create a backup job with time-based backups, easy VM restoration and backup retention policies. Managed Disks only support Locally Redundant Storage (LRS) as the replication option; this means it keeps three copies of the data within a single region. For regional disaster recovery, you must backup your VM disks in a different region using Azure Backup service and a GRS storage account as backup vault.
8.10. Managed Disk Snapshot
A Managed Snapshot is a read-only full copy of a managed disk which is stored as a standard managed disk by default. With snapshots, you can back up your managed disks at any point in time. These snapshots exist independent of the source disk and can be used to create new Managed Disks. They are billed based on the used size. For example, if you create a snapshot of a managed disk with provisioned capacity of 64 GB and actual used data size of 10 GB, snapshot will be billed only for the used data size of 10 GB.
8.11. Managed disk image
Managed Disks also support creating a managed custom image. You can create an image from your custom VHD in a storage account or directly from a generalized (sys-prepped) VM. This captures in a single image all managed disks associated with a VM, including both the OS and data disks. This ties into #2 above.
8.12. Image versus Snapshot
With Managed Disks, you can capture a VM image of a generalized VM that has been deallocated. This image will include all of the disks attached to the VM. You can use this image to create a new VM, and it will include all of the disks.
A snapshot is a copy of a single disk at the point in time it is taken. If a VM has only one disk, you can create a VM with either tyhe image or the snapshot. If you have multiple disks attached to a VM, and they are striped - the snapshot feature does not support this scenario yet, you have to create an image to replicate the VM.
8.13. Encryption at rest
The two kinds of encryption are available - Storage Service Encryption (SSE) - at the storage service level, and Azure Disk Encryption (ADE) at the disk level (OS and data disks). Both persist keys to Azure Key Vault.
SSE is enabled by default for all Managed Disks, Snapshots and Images in all the regions where managed disks is available.
WRT to ADE, for Linux, the disks are encrypted using the DM-Crypt technology
Three replicas are maintained, and are designed for 99.999% availability. .
8.15. Pricing and billing
Disk tier - premium versus standard
Number of transactions - applicable for standard disks only
To be continued...
This post first appeared on MSDN Blogs | Get The Latest Information, Insights, Announcements, And News From Microsoft Experts And Developers In The MSDN Blogs., please read the originial post: here