Part 3 - AWS - EBS (Elastic Block Storage)

EBS: (Elastic Block Storage) (Disk in a Cloud)

Provides block-level storage volumes: -

Amazon EBS allows you to create storage volumes and attach them to Amazon EC2 instances. Once attached, you can create a file system on top of these volumes, run a database, or use them in any other way you would use a block devices.  Supports up to 16 TB.

Cost is calculated from GB/month of total allocated size

·        Even if nothing is stored on it.

Ability to store information off an EC2 instance.

·        Volumes can be attached and detached to EC2 instances

·        Volumes persist after terminating an instance.

Amazon EBS volumes are placed in a specific Availability Zone

Can only be attached to instances within the same zone

·        Can only be attached to one instance at a time

·        Many volumes can be attached to a single instance

Volumes are automatically replicated across AWS servers within the zone

·        To improve reliability, availability, and durability

·        Prevents data loss due to single hardware failure

o   Not designed for durability or considered highly durable

o   Designed to be better than storing data on a single hard drive

·        Snapshots can be used to provide highly durable backups

EBS Volumes Types

·        General Purpose SSD (GP2)

o   Designed for 99.999% availability

o   Ratio of 3 IOPS per GB with up to 10000 IOPS and the ability to burst up to 3000 IOPS for short periods for volumes under 1 Gib.

·        Provisioned IOPS SSD (I01) (Input output Per Second)

o   Designed for I/O intensive applications such as large relational or NoSQL databases. Use if you need more than 10,000 IOPS

·        Magnetic (Standard)

o   Lowest cost per gigabyte to all EBS volumes types. Magnetic volumes are ideal for workloads where data is accessed infrequently, and applications where the lowest storage cost is important.

Creating EBS Volume

Attaching an EBS Volume

Mounting an EBS Volume

·        Once attached, a volume must be mounted in the actual instance

o   Depends on OS

·        Volume must also be formatted before use

o   New volume appear as zeroed-out raw block-level storage

o   Can format as any file system desired

§  Depends on OS

Snapshot

·        EBS volumes can be backed up

•        A point-in-time snapshot

·        Stored in S3

•        Provides very high durability

•        More on S3 shortly

·        Snapshots are incremental backups to reduce storage requirements

•        Snapshots only store data that changed since last snapshot

•        All snapshots have access to all data to restore to that snapshot

•        Deleting a snapshot only deletes data unique to that snapshot

·        A snapshot can be restored to a new EBS volume

Volume Vs Snapshots

·       Volumes exist on EBS

o   Virtual Hard Disk

·        Snapshots exist on S3

·        You can take a snapshot of volume, this will store that volume on S3.

·        Snapshots are point in time copies of Volumes.

·        Snapshots are incremental, this means that only the blocks that have changed since your last snapshot are moved to S3.

·        If this is your first snapshot, it may take some time to create.

Volumes vs Snapshots - Security

·       Snapshots of encrypted volumes are encrypted automatically.

·        Volumes restored from encrypted snapshots are encrypted automatically.

·        You can share snapshots, but only if they are unencrypted.

o   These snapshots can be shared with other AWS accounts or made public.

Snapshots of Root Device Volumes

·       To create a snapshot for Amazon EBS volumes that server as root devices, you should stop the instance before taking the snapshot.

RAID, Volume & Snapshots

·       RAID=Redundant Array of Independent Disks

o   RAID 0 - Striped, No Redundancy, Good Performance.

o   RAID 1 - Mirrored, Redundancy

o   RAID 5 - Good for reads, bad of writes, AWS does not recommend every putting RAID 5's on EBS

o   RAID 10 - Striped & Mirrored, Good Redundancy, Good Performance.

How can I take a Snapshot of a RAID Array?

·       Problem - Take a snapshot, the snapshot excludes data held in the cache by applications and the OS. This tends not to matter on a single volume, however using multiple volumes in a RAID array, this can be a problem due to interdependencies of the array.

·        Solution - Take an application consistent snapshot.

o   Stop the application from writing to disk

o   Flush all caches to the disk

·        How can we do this?

o   Freeze the file system

o   Unmount the RAID array

o   Shutting down the associated EC2 instances

Creating Snapshots

·        Management Console

•        Three ways to create a snapshot

–       From EBS Volumes page, right-click volume and select Create Snapshot

–       From EBS Volumes page, select Actions | Create Snapshot

–       From EBS Snapshots page, select the Create Snapshot button

·        Command line:

·        Creating Snapshots

Syntax: #aws ec2 create-snapshot –-volume-id <vol_id> –-description <description>

Example: #aws ec2 create-snapshot –-volume-id vol-cf674da4 –-description "June05Snap"

Creating Volumes from Snapshots

·        From Management Console

•        Can specify a snapshot when creating a new volume (shown earlier)

•        Or

•        Same as before, except add a –-snapshot-id parameter

# aws ec2 create-volume --size <size> --snapshot-id <snapshot> --availability-zone <zone> --volume-type <type>

Increasing EBS Volumes Size

·        A volume size cannot be increased directly, but it can be done easily

·        Increasing volume size requires four steps

o   Create a snapshot of the volume

o   Create a new volume from that snapshot with a larger size

o   Attach the new volume to an instance

o   Configure the file system for the new volume size

–       This is the only new step

·        Performing Step 4 will depend on the operating system

o   The following slides assume that a new 3-GB volume was created from a 2-GB snapshot

Reducing EBS Volume Size

·        Reducing volume size is different than increasing size

o   Volumes created from a snapshot cannot be smaller than the snapshot

·        Reducing volume size is more of a manual process

·        Create a new EBS volume of the desired size

o   The new volume must be big enough to hold the amount of data!

·        Attach both the original and new volume to the same instance

o   Format the new volume as needed

·        Copy the contents from the original volume to the new volume

o   Linux: Use cp or rsync 

o   Windows: Use Windows Explorer or xcopy

·        Recommended to then create a snapshot of the new volume

Moving EBS Volumes across Zones

Moving volumes across availability zones within the same region is easy

·        Create a snapshot of the original volume

·        Create a new volume in a different zone from the snapshot

o   Snapshots are not associated with an availability zone

Moving EBS Volumes Across Region

·        Moving volumes across regions is also easy

o   Create a snapshot of the original volume

·        From the Snapshots screen, select the snapshot, click Actions, and select Copy

o   Select the region to copy to

·        This will incur bandwidth charges

Encrypting EBS Volumes

·        EBS supports encrypting volumes

o   From the Create Volume screen, select “Encrypt this volume”

§  Available keys are managed from the IAM console

·        Encryption/decryption performed by Amazon

·        Can also encrypt the volume manually

o   EBS volumes are just block-level devices

o   Can use operating system modules to encrypt files or entire file systems

§  Windows Encrypting File System (EFS)

§  Various Linux/UNIX cryptographic file systems

EBS Limits

·        Each account has an EBS limit of a total size of 20 TB for each volume type

·        Higher limits can be requested at

o   http://aws.amazon.com/contact-us/ebs_volume_limit_request

·        Requests are for a specific region

Exam Tips EBS

·        EBS Consists of;

o   General Purpose SSD - GP2 - (Up to 10,000 IOPS)

o   Provisioned IOPS SSD - IO1-(More than 10,000 IOPS)

o   Magnetic - cheap, infrequently accessed storage)

·        You cannot mount 1 EBS volume to multiple EC2 instances, instead use EFS.

Part 2 - AWS - Elastic Compute Cloud EC2

EC2 (Elastic Compute Cloud)

·        Resizable compute capacity

·        Complete control of your computing resources

·        Reduces the time required to obtain and boot new service instance to minutes.

·        Scale capacity as your computing requirement changes

·        Pay only for capacity that you actually use.

·        Choose Linux or Windows

·        Deploy across AWS regions and Availability Zones for reliability.

Basic Terminology:

Instance:

·        Running virtual Server

·        Created from AMI Template

Elastic IP

·        Static IP address assigned to an account.

·        Static IP address may be assigned/reassigned to instances

o   That is why they are called elastic

Security Group

·        Like a network segment behind a firewall

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. Amazon EC2 reduces the time required to obtain and boot new server instances to minutes, allowing you to quickly scale capacity, both up and down, as your computing requirements change. 

EC2 Options:

·       On Demand Instance- allow you to pay a fixed rate by the hour with no commitment.

o   Users that want the low cost and flexibility of Amazon EC2 without any up-front payment or long-term commitment

o   Application with short term, spiky, or unpredictable workloads that cannot be interrupted.

o   Applications being developed or tested on Amazon EC2 for the first time

·        Reserved Instance - provide you with a capacity reservation, and offer a significant discount on the hourly charge for an instance. 1 Year or 3 Year Terms

o   Applications with steady state or predictable usage

o   Applications that require reserved capacity

o   Users able to make upfront payments to reduce their total computing costs even further.

·        Spot Instance- enable you to bid whatever price you want for instance capacity providing for even greater savings if your applications have flexible start and end times.

o   Applications that have flexible start and end times

o   Applications that are only feasible at very low compute prices

o   Users with urgent computing needs for large amounts of additional capacity.

o   If the Spot instance is terminated by Amazon EC2, you will not be charged for a partial hour of usage. However, if you terminate the instance yourself, you will be charged for any hour in which the instance ran.

EC2 Instance Type:

 

 How I remember them;

·        D for Density

·        I for IOPS

·        R for RAM

·        T cheap general purpose (think T2 Micro)

·        M - main choice for general purpose apps

·        C for Compute

·        G – Graphics

AWS EC2 Instance Purchasing Option

Overview

·        Amazon provides different ways to pay for the EC2 instances

§  On-Demand Instances

§  Reserved Instances

§  Spot Instances

·        Light, Medium, and Heavy Utilization Reserved Instances are no longer available for purchase and were part of the Previous Generation AWS EC2 purchasing model

Instance Purchasing Options

On-Demand Instances

·        No Upfront costs or commitments-Simple Pay by the hour.

·        Instance runs until you stop or terminate it.

·        Instances can be scaled accordingly as per the demand

·        Although AWS makes effort to have the capacity to launch On-Demand instances, there might be instances during peak demand where the instance cannot be launched

·        Well suited for

§  Users that want the low cost and flexibility of Amazon EC2 without any up-front payment or long-term commitment

§  Applications with short term, spiky, or unpredictable workloads that cannot be interrupted

§  Applications being developed or tested on Amazon EC2 for the first time

Reserved Instances

·        Reserved Instances provides lower hourly running costs by providing a billing discount as well as capacity reservation that is applied to instances and there would never be a case of insufficient capacity from AWS

·        Discounted usage price is fixed for as long as you own the Reserved Instance, allowing you to predict compute costs over the term of the reservation.

·        Reserved instances are best suited if consistent, heavy, use is expected and they can provide savings over owning your own hardware or running only On-Demand instances.

·        Well Suited for

§  Applications with steady state or predictable usage

§  Applications that require reserved capacity

§  Users able to make upfront payments to reduce their total computing costs even further

·        Reserved instance is not a physical instance that you launch, but it’s just an accounting term applied to the instance usage during billing

·        Reserved Instances do not renew automatically, and the EC2 instances can be continued to be used but charged On-Demand rates

·        Auto Scaling or other AWS services can be used to launch the On-Demand instances that use the Reserved Instance benefits

·        With Reserved Instances

§  You pay for the entire term whether or not you use it

§  Once purchased, the reservation cannot be cancelled but can be sold

§  Reserved Instance pricing tier discounts only apply to purchases made from AWS, and not to the third party Reserved instances

How Reserved Instances work

Billing Benefits & Payment Options

·        Reserved Instance purchase reservation is automatically applied to running instances that match the specified parameters

·        Reserved Instance can also be utilized by launching On-Demand instances with the same configuration as to the purchased reserved capacity

Payment Options

·        No Upfront

o   No upfront payment is required and the account is charged on a discounted hourly rate for every hour, regardless of the usage

o   Only available as 1-year reservation

·        Partial Upfront

o   A portion of the cost is paid upfront and the remaining hours in the term are charged at an hourly discounted rate, regardless of the usage

·        Full Upfront

o   Full payment is made at the start of the term, with no costs for the remainder of the term, regardless of the usage

Understanding Hourly Billing

·        Reserved Instances are billed for every hour during the term that you select, regardless of whether the instance is running or not.

·        Reservations and discounted rates only apply to one instance-hour per hour. If an instance restarts during the first hour of a reservation and runs for two hours before stopping, the first instance-hour is charged at the discounted rate but three instance-hours are charged at the On-Demand rate. If the instance restarts during one hour and again the next hour before running the remainder of the reservation, one instance-hour is charged at the On-Demand rate but the discounted rate is applied to previous and subsequent instance-hours.

Spot Instances

·        Spot instances enables bidding on unused EC2 instances, and are launched whenever the bid price exceeds the current market spot price

·        Amazon EC2 sets up the hourly price which fluctuates depending upon the demand and supply of spot instances

·        Spot instances are a cost-effective choice and can bring the EC2 costs significantly

·        Spot instances can be used for applications flexible in the timing when they can run and also able to handle interruption by storing the state externally for e.g. they are well-suited for data analysis, batch jobs, background processing, and optional tasks

·        Well Suited for

§  Applications that have flexible start and end times

§  Applications that are only feasible at very low compute prices

§  Users with urgent computing needs for large amounts of additional capacity

·        Spot instances differ from the On-Demand instances

§  they are not launched immediately

§  they can be terminated anytime

§  prices vary as per the demand and supply of spot instances

·        Usual strategy involves using Spot instances with On-Demand or Reserved instances, which provide a minimum level of guaranteed compute resources, while spot instances provide an additional computation boost

·        Spot instances can also be launched with a required duration (also known as Spot blocks), which are not interrupted due to changes in the Spot price

·        Amazon EC2 provides a data feed, sent to an Amazon S3 bucket specified during subscription, that describes the Spot instance usage and pricing

·        T2 and HS1 instance class types are not supported for Spot instances

Spot Concepts

·        Spot pool – Pool of EC2 instances with the same instance type, availability zone, operating system and network platform

·        Spot price – Current market price of a spot instance per hour as set by Amazon EC2 based on the last fulfilled bid

·        Spot bid – is the maximum bid price the bidder is willing to pay for the spot instance

·        Spot fleet – is the set of instances launched based on the criteria the bidder

·        Spot instance interuption – Amazon EC2 terminates the spot instances whenever the bid price is lower than the current market price or the supply has reduced

·        Bid status – provides the current state of the spot bid

Spot Pricing & How it works

·        Amazon EC2 sets up an hourly spot price which fluctuates depending upon the demand and supply.

·        If the bid price exceeds the current market spot price, the request is fulfilled by Amazon till either the spot instance is terminated or the spot price increases beyond the bid price

·        Everyone pays the same market price for tha period irrespective of the bid price given the bid price is more than the spot price for e.g. if the spot price is $0.20 and there are 2 bids from Customers with bid price $0.25 and $0.30, both customers would still pay $0.20 only

·        If the Spot instance is terminated by Amazon, you are not billed for the partial hour. However, if the spot instance are terminated by you, you will be charged for the partial hour

·        Spot instances with a predefined duration use a fixed hourly price that remains in effect for the Spot instance while it runs

·        Amazon EC2 can interrupt the Spot instance when the Spot price rises above your bid price, when the demand for Spot instances rises, or when the supply of Spot instances decreases.

·        When Amazon EC2 marks a Spot instance for termination, it provides a Spot instance termination notice, which gives the instance a two-minute warning before it terminates.

·        Termination notice warning is made available to the applications on the Spot instance using an item in the instance metadata and needs to check periodically (aws recommends every 5 seconds)

·        Amazon EBS-backed instance if it is a Spot instance cannot be stopped and started, but only rebooted or terminated

Pricing Example

·        State 1 – Starting with Amazon EC2 has 5 Spot instances available

§  6 bids available for Spot instances

§  Amazon EC2 picks up the top five priced bids and allocates a Spot instance to them

§  Spot Price is $0.10

§  Bid with the price of $0.05 is not served

·        State 2 – Supply of Amazon EC2 Spot instances reduce to 3

§  Amazon EC2 terminates the 2 spot instances with $0.10 ( the order in which the instances are terminated is determined at random )

§  Rest of the Spot instances continue

·        State 3 – New bid for Spot Instance is placed with Price $0.15 is placed

§  Spot instance with price $0.15 is fullfilled

§  Amazon EC2 terminates the single spot instances with $0.10

§  Spot Price changed to $0.15

·        State 4 New bid for Spot Instance is placed with Price $2 is placed

§  Spot instance with price $2 is fulfilled

§  Amazon EC2 terminates the single spot instances with $0.15

§  Spot Price changed to $1.00

Spot Instances best practices

·        Choose a reasonable bid price, which is low enough to suit you budget and high enough for the request to be fulfilled and should not be higher than the On-Demand bid price

·        Ensure the instances and up and ready as soon as the request is fulfilled, by provisioning a AMI with all the required softwares and load application data from user data

·        Store important data regularly and externally in a place that won’t be affected when the Spot instance terminates for e.g., you can use Amazon S3, Amazon EBS, or DynamoDB.

·        Divide the work into smaller finer tasks so that they can be completely and state saved more frequently

·        Use Spot termination notice warning to monitor instance status regularly

·        Test applications to ensure that it handles unexpected termination gracefully. This can be tested by using On-Demand instances and terminating them

Exam Tips EC2

·        Know the differences between;

o   On Demand

o   Spot

o   Reserved

·        Remember with spot instances;

o   If you terminate the instance, you pay for the hours

o   If AWS terminates the spot instance, you get the hour it was terminated in for free.

EC2 Lab Exam Tips

·        Termination Protection is turned off by default, you must turn it on.

·        On an EBS-backed instance, the default action is for the root EBS volume to be deleted when the instance is terminated.

·        Root volumes cannot be encrypted by default, you need a third party tool (such as bit locker etc.) to encrypt the root volume.

·        Additional volumes can be encrypted.

EC2 Container Services (Not in exam as of now)

·        Amazon EC2 Container Service (Amazon ECS) is a highly scalable, fast, container management service that makes it easy to run, stop, and manage Docker containers on a cluster of Amazon Elastic Compute Cloud (Amazon EC2) instances.

·        Amazon ECS lets you launch and stop container-based applications with simple API calls, allows you to get the state of your cluster from a centralized service, and gives you access to many familiar Amazon EC2 features.