Lots of applications require shared file storage that can be accessed by multiple servers at the same time. But building your own file storage system will take time and can be very expensive and once your file storage system is deployed and operational, you have to do complex maintenance and backup operations to make sure its performing well and the data is secure. What if there was a better way ? So let’s get started on Amazon Efs Performance Considerations and Overview.

Amazon Elastic File System or Amazon EFS is a shared file system for use with Amazon EC2.

With Amazon EFS , applications running on multiple EC2 instances can access the file system at the same time and the service used the industry standard NFSv4 version file access protocol so the applications and tools that you today work seamlessly.

It is easy to create an Amazon Elastic File System with a few clicks in the AWS Management Console or using the AWS SDK or the AWS command line interface. Once your file system is created EC2 instances in your Amazon VPC can mount your file system and begin storing data in the EFS.

Amazon EFS manages all the infrastructure for you and automatically scales your file system storage capacity up or down as you add or remove files, ensuring you always have the storage you need while reducing time consuming administration tasks and with Amazon EFS you only pay for the space you files and directories use. There is no minimum fee and there are no setup costs.

Making sure that your data is available and secure is really important. Amazon EFS is designed to be durable and highly available and automatically replicate your data across multiple availability zones in the region. Amazon EFS gives you the flexibility to control who can access your file systems and their content. Amazon VPC security groups and network access control lists allow you to manage network access to your Amazon EFS file systems.

At the file and directory level, Amazon EFS supports user and group read,write and execute permissions and Amazon EFS is integrated with AWS identity and access management to control access to Amazon EFS’s administrative API.

Amazon EFS lets you run applications that you shared file storage in the cloud and automates time consuming administration tasks freeing you to focus on your application and business.

Mount Amazon EFS on EC2

Log into Amazon AWS management console and click on EFS. If this is the first time you are using EFS then you should see the welcome screen.

Click on create file system. The next page show “configure file system access”. chose the desired Security group from the drop down. Be sure to select the Security Group where you have a opened ports so that EC2 can connect with EFS “Next Step”.

Click Next Step and we go to Configure optional settings. You can add key-value tags on this page. The tags can be used later for reports or billing breakdown.

The last step is to review and create the file system.

Once the EFS has been created you’ll be able to see the message that your EFS has been created and the necessary details along with it.

During the EFS creation, I had mentioned about selecting the desired Security Group.We want to be able to mount an EFS on EC2, which means we are looking at rules that would allow connections both ways and also perform all the normal inbound and outbound operations in an EC2 Instance.

Let’s see how to create a Security Group. Please go the EC2 Dashboard and on the left side of the panel, you will be able to see the Security Group option. Click on that, and on the next page, you will see the “Create Security Group” option.

Once you have clicked on that option, a pop-up will occur where can specify all the necessary details. You can add the security group name, description and select the desired VPC for it as well. Add an InBound rule with type as NFS, it should auto populate protocol and port range; Please select a source of ‘Anywhere’ from the dropdown but you can limit it to a particular IP or a subnet of your choice. You can add various Inbound and Outbound rules and also can limit it to an IP or a subnet. Please click on “Create” once you have finished setting up all your rules.

This Security group can be added to the EFS that was created. If the EFS was created with default Security group, we can edit the same and replace with new Security Group. Go to the EFS group and click on ‘Manage File System Access’. Remove the default Security Groups and add the one we created and click on save.

Next we will mount the file system on EC2. In the Dashboard you will see the “Connect” option. Click on it and a Pop-up jumps out explaining how to connect to your EC2 Instance via SSH. To connect via SSH, you must specify the rule to allow connection to the SSH port. Follow the Instructions in the Pop-up and you’ll be able to login to your EC2 Instance.

If you go to the file system dashboard, you should see a link that has instructions to mount the file system. Click on the link to bring up up this window

If you have created the EC2 instance using the Amazon linux then you should have the required tools to mount the file system. If not, follow the instructions on the popup; create a folder called ‘efs’ on the EC2 instance and fire the command specified on step 3 of the popup to mount it.

I created the efs folder in the /mnt directory after switching from normal user to root user. The mount command can be copied over from the pop-up instructions window. To permanently mount the EFS add the entry to /etc/fstab file

Once you mount the file system, you can start creating files to it. Also, If you unmount the file system and mount it to a different machine the files are all still there.

Performance Tips for Amazon EFS

If a project needed a shared network file system that was available to all servers, and able to scale horizontally to anywhere between 1 and 100 servers, with low-latency file access, and also needed to be able to handle small file writes and file locks synchronously, Amazon EFS, which uses NFSv4.1 is the best way to go as it checks most of these check boxes to an extent. I would like to point out some areas which you should concentrate to increase performance.

Monitor EFS Burst Credits

You can usually start out using the free/cheap tier of a service, but once you start putting production loads on the service, you’ll have your first major outage with no visible cause. This happens when you are not monitoring your burst credits.

Imagine you are running a CMS based system, MySQL connections were stacking up, application threads were clogged, and extremely few requests were getting responses. You noticed that file operations were really slow. Upon further investigation, you would be able to see that your Burst Credits were expired, and your Permitted Throughput went from 50 Mbps to 0.5 Mbps. So make sure that you monitor your Burst Credit Balance or add alerts in Cloudwatch to check them.

Write dummy data to get better performance

As there is no ‘instance class’ in EFS, there is no way to go for an upgrade(similar to CPU-optimized, RAM-optimized etc) The only real control you have over I/O limits is to switch between ‘General Purpose‘ ( for low latency and high horizontal scalability), or ‘Max I/O‘ ( for larger volumes of data transfer, but with slightly higher latency). When you create a new EFS volume, you get a paltry .5 MB/s sustained transfer rate, and 7.2 minutes worth of burst credits (up to 100 MB/s). You write a giant file to the filesystem, and EFS takes up to an hour to increase your limits, according to this chart.

So it is always, better to create a dummy large file, as soon as EFS volume is created.
Note: Exclude these large test files from backup.

Do not run app codes from EFS

Use NFS/EFS for storage of things like media assets, exported data files, and asynchronous logs that don’t require extremely low-latency to work well.

Use the correct mount options

AWS recommends the use of certain mount points when setting up EFS on Instances. It is best to go ahead with these recommendations unless you have very specific needs. Also, use NFS v4.1 if at all possible.

Average I/O Size

Amazon EFS file systems are distributed across large number of storage servers and this enables high level of availability,scalability and durability. This carefully designed structure results in a small latency overhead for each file operation. Due to this low latency, overall throughput increases because the overhead is spread over a large amount of data.This distributed desgin helps one to cut out bottlenecks and constraints that are normally associated with file servers.

Simultaneous Connections

EFS file systems can be mounted upto thousands of EC2 instances and simultaneously, the key to higher throughput is to distribute our application across more instances to drive higher output levels.

Request Model

Enabling asynchronous writes to the file system, the pending write operations are buffered on the EC2 instance before they are written to the Amazon EFS. Practicing this techniques typically has lower latencies. The kernel uses additional memory for caching while performing asynchronous writes. Accessing a file system that has been mounted with the sync option or opening files using an option that bypasses the cache issues synchronous requests to Amazon EFS.

Amazon EC2 Instances

When launching your Amazon EC2 instances, choose instance types that meets your requirements for the smooth running of your application. Applications that perform huge number of read and writes needs more memory and computing capacity than applications that does not perform such high numbers of read and write.

Performance Modes

Amazon EFS are meant to support a wide variety of cloud storage workloads. Amazon EFS offers two performance modes, which at the time of creation has to be selected.

The recommended mode is the General Purpose performance mode. This mode is ideal for latency-sensitive use cases, web hosting and content management systems. By default Amazon EFS selects General Purpose performance mode at the start of the creation of file system.

File systems in the Max I/O performance mode can scale to higher levels of aggregate throughput. Highly distributed applications such as Big data analysis and media processing can benefit from this mode.

Conclusion

Amazon EFS is a great fit for many use-cases. It is advised to consider the above tuning recommendations to achieve maximum performance.

Thanks for dropping by. Ready for the next blog?

Git Version Control in cPanel

The post Amazon Efs Performance Considerations and Overview appeared first on Sysally.