Wecan
mount an S3 bucket onto an AWS instance as a file system known as S3fs. It is a
FUSE filesystem application backed by amazon web services, that allows you to
mount an Amazon S3 bucket as a local file-system. We can use system commands
with this drive just like as any other Hard Disk in the system. On s3fs mounted
files systems we can simply use cp, mv and ls the basic Unix commands similar to
run on locally attached disks.
Filesystem in
Userspace (FUSE)
is a software interface for Unix and Unix-like computer operating systems that
lets non-privileged users create their own file systems without editing kernel
code. This is achieved by running the file system code in user space while the
FUSE module provides only a “bridge” to the actual kernel interfaces.
Why
S3 Bucket?
We can consider
NFS sort of solution, even now we have EFS from Amazon but it’s costly and even
the same data were used for their analytics solution. So we thought to use S3 to
satisfy both the requirement.
Follow the below steps to mount your S3 bucket to
Your Linux Instance.
We are assuming
that you have a running Linux EC2(Red Hat/Centos) instance on AWS with root
access and a bucket created in S3 which is to be mounted on your Linux
Instance.
Step-1: Using new instance of CentOS or Red
Hat.Update the system.
#sudo yum update
Step-2: Install
Required Packages
First, we will
install all the dependencies for fuse and s3cmd. Install the required packages
to system use following command.
# sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
Step-3: Download
s3fs source code from git.
# git clone
Step-4 :Now
Compile and install the code.
Following the set
of command will compile fuse and add fuse module in the kernel.
# cd s3fs-fuse# ./autogen.sh # ./configure — prefix=/usr — with-openssl# make # sudo make install
Step-5: Use below command to check where s3fs
command is placed in os.
# which s3fs
Step-6: Creating a IAM role for s3
bucket
Create one IAM role with policy having appropriate access to particular bucket.For example :- My IAM role name is s3fsmountingrole and bucket created is s3fs-demobucketPolicy attached should be read/ write access for bucket s3fs-demobucketEnter policy name Description and Policy Document as given below
{
“Version”: “2012–10–17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: [
“s3:GetBucketLocation”,
“s3:ListAllMyBuckets”
],
“Resource”: “arn:aws:s3:::*”
},
{
“Effect”: “Allow”,
“Action”: [“s3:ListBucket”],
“Resource”: [“arn:aws:s3:::s3fs-demobucket”]
},
{
“Effect”: “Allow”,
“Action”: [
“s3:PutObject”,
“s3:GetObject”,
“s3:DeleteObject”
],
“Resource”: [“arn:aws:s3:::s3fs-demobucket/*”]
}
]
}
Attach IAM Role to the running Instance or Launching new Instance
Step-7: Now create a directory or provide the
path of an existing directory and mount S3bucket in it.
#sudo mkdir -p /var/s3fs-demofs
Step-8:
Now mount the s3 bucket using IAM role enter following command :
#s3fs -o iam_role=”s3fsmountingrole” -o url=”
" -o endpoint=eu-central-1 -o dbglevel=info -o curldbg -o allow_other -o use_cache=/tmp s3fs-demobucket/var/s3fs-demofs
Step-9: Check mounted s3 bucket. The output
will be similar as shown below but Used size may differ.
#df -h
df -h shows the
mounted file system, here you can see we have successfully mounted the S3 bucket
on your EC2 Instance.
Note: If
you already had some data in s3bucket and it is not visible, then you have to
set permission in ACL at the S3 AWS management console for that s3
bucket.
Congrats!! You have successfully mounted your S3
bucket to your EC2 instance.
Conclusion
Here, I explained
how to mount AWS s3 bucket on EC2 Linux instance, and for demo purpose, I used
RedHat machine and created one IAM role for access to s3 bucket and attached it
to running instance. You can also get access to s3 bucket from EC2 instance by
providing AWS access key and secret key.
S3FS is a FUSE (File System in User Space) will mount Amazon S3 as a local file system. S3FS has an ability to manipulate Amazon S3 bucket in many useful ways. If you wish to access your Amazon S3 bucket without mounting it on your server, you can use s3cmd command line utility to manage S3 bucket.
What is an Amazon S3 bucket?
Amazon S3 is a cloud based web service interface that you can used to store and retrieve any amount of data. To upload your data, first you need to create an S3 bucket in one of the Amazon regions.
Creating a Bucket
S3 provides an API for creating and managing buckets. You can create a maximum of 100 buckets from your AWS console. When you create a bucket, you need to provide a name and AWS region where you want to create the bucket. In each bucket, you can store any number of objects. You can use your AWS account root credentials to create a bucket, but it is not recommended. Instead just create an IAM user and add full permission to that user on S3 bucket. You can access your S3 bucket from your Amazon S3 console.
Please follow the below steps to mount s3 bucket on your server.
1) Remove Existing Packages
Before installing any package, first you need to check if you have any existing fuse or S3FS on your server. If it is already existing, then remove it from your server to avoid further conflicts. Use the following command to check if you have any existing fuse or S3FS on your server
CentOS users:
$ yum remove fuse fuse-s3fsUbuntu Users:
$ apt-get remove fuse
2) Install Packages
Install all dependency packages for fuse and s3cmd using the below command.
CentOS users:
$ yum install gcc libstdc++-devel gcc-c++ curl-devel libxml2-devel openssl-devel mailcapUbuntu Users:
$ apt-get install build-essential libcurl4-openssl-dev libxml2-dev mime-support
3) Download and Compile Fuse
Move to /usr/src then download and compile fuse source code. After compiling, add fuse to kernel. In my case the latest version of fuse is fuse-3.0.0
$ cd /usr/src/
$ wget https://github.com/libfuse/libfuse/releases/download/fuse-3.0.0/fuse-3.0.0.tar.gz
$ tar xzf fuse-3.0.0.tar.gz
$ cd fuse-3.0.0
$ ./configure –prefix=/usr/local
$ make && make install
$ export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
$ ldconfig
$ modprobe fuse
4) Download and compile S3FS
Navigate to /usr/src, Download and compile s3fs source code.
$ git clone https://github.com/s3fs-fuse/s3fs-fuse.git
$ cd s3fs-fuse
$ ./autogen.sh
$ ./configure
$ make
$ make install
5) Setup Access Key
Both access key and secret key of your s3 AWS account is required for configuring S3FS. Replace the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with your actual key values.
$ vi /etc/passwd-s3fsMake sure that the file has proper permission.
AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY
$ chmod 600 /etc/passwd-s3fs
6) Mount S3 Bucket
You can run the below command to mount s3fs.
$ s3fs mybucket /path/to/mountpoint -o passwd_file=/etc/passwd-s3fsYou can also mount the s3 bucket on boot by following below commands.
$ mkdir /tmp/cacheCongratulations you have successfully mounted s3 bucket on your server.
$ mkdir /path/to/mountpoint
$ chmod 777 /tmp/cache /path/to/mountpoint
$ vi /etc/fstab
s3fs#<mybucket> /path/to/mountpoint fuse allow_other,use_cache=/tmp/cache,uid=userid,gid=groupid 0 0
$ mount -a