CCA131 – Create an HDFS user’s home directory

In the exam, you may be asked to create a home directory for an existing local user onto HDFS. You may further be asked to set a specific ownership or permission to the home directory. The process basically involves:

1. Create a local user if not already present
2. Create a home directory into HDFS for the user
3. Assign appropriate ownership and permissions to the home directory

1. Create a local user

Please note that you will have to create a local user in the exam only if it’s not already present on the system. Let’s see and example to create home directory in HDFS for user “test” and assigning requested permissions.

1. First, verify if the user “test” is already present in the local system:

# id test
id: test: no such user

2. Let’s create a user named test on the local system.

# useradd test

3. Verify the user creation with the “id” command again.

$ id test
uid=1001(test) gid=1001(test) groups=1001(test)

2. Create home directory into HDFS

1. Next step is to create a home directory for the user “test” into HDFS. This can be done with the user “hdfs” which has a superuser priviledges in HDFS filesystem. Infact all the commands given below are to be executed with the user hdfs.

# su - hdfs
$ hdfs dfs -mkdir /user/test

2. Verify if the directory is created:

$ hdfs dfs -ls /user
drwxr-xr-x   - hdfs   supergroup          0 2018-09-01 19:18 /user/test

3. Assign ownership and permission

The last and important step in to assign desired ownership and permissions to the home directory. Without this you can not use the directory for any purpose.

1. First, verify the directory and its default permissions.

$ hdfs dfs -ls /user
drwxr-xr-x   - hdfs   supergroup          0 2018-09-01 19:18 /user/test

2. Change the ownership of the directory “/user/test” to test:test.

$ hdfs dfs -chown test:test /user/test

$ hdfs dfs -ls /user
drwxr-xr-x   - test   test                0 2018-09-01 19:18 /user/test

3. The current permissions of the directory are 755. Let’s change the permissions to 766 using the command shown below:

$ hdfs dfs -chmod 766 /user/test

$ hdfs dfs -ls /user
drwxrw-rw-   - test   test                0 2018-09-01 19:18 /user/test

Copying files to and from HDFS filesystem

To verify if we have performed all the steps properly, we can create a file locally and upload it to HDFS and vice versa using the user “test”. Let create a file locally first using the test user.

# su - test
$ touch /home/test/test_file

Now, copy this file to the HDFS home directory of user test. If you want to copy a file from any other location, make sure the parent directory has the permission of 755.

$ hdfs dfs -put /home/test/test_file /user/test/

Verify:

$ hdfs dfs -ls /user/test
Found 1 items
-rw-r--r--   3 test test          0 2018-09-01 20:03 /user/test/test_file

Let’s now copy the same file from HDFS to local filesystem.

$ hdfs dfs -get /user/test/test_file /tmp/

Verify:

$ ls -l /tmp/test_file
-rw-r--r-- 1 test test 0 Sep  1 20:05 /tmp/test_file

The command “hdfs dfs -get” and “hdfs dfs -CopyToLocal” are equivalent and can be used interchangeably.