Показаны сообщения с ярлыком HDFS. Показать все сообщения
Показаны сообщения с ярлыком HDFS. Показать все сообщения
, ,

Permission denied when loading data into hive table

вторник, 8 октября 2013 г. 0 коммент.

I got an error:


[root ~]# sudo -u hive hive
hive> LOAD DATA INPATH '/user/hdfs/d_product_type_hive_exp.csv' OVERWRITE INTO TABLE D_PRODUCT_TYPE_HIVE;
Loading data to table default.d_product_type_hive
rmr: DEPRECATED: Please use 'rm -r' instead.
Moved: 'hdfs://xxxxx:8020/user/hive/warehouse/d_product_type_hive' to trash at: hdfs://xxxxx:8020/user/hive/.Trash/Current
Failed with exception Permission denied: user=hive, access=WRITE, inode="/user/hdfs":hdfs:supergroup:drwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:224)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:204)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4716)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4698)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkParentAccess(FSNamesystem.java:4666)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameToInternal(FSNamesystem.java:2698)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameToInt(FSNamesystem.java:2666)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:2645)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename(NameNodeRpcServer.java:589)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:381)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44964)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1701)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1697)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1695)

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
hive>

Cause: Command "LOAD DATA"(without LOCAL) moves the data to the final location and the user hive don't have the rights to delete the file.
Action: Give the right to "write" on the directory with the file.


[root ~]# sudo -u hdfs hadoop fs -ls -d /user/hdfs/
Found 1 items
drwxr-xr-x - hdfs supergroup 0 2013-10-08 18:28 /user/hdfs
[root ~]# sudo -u hdfs hadoop fs -chmod o+w /user/hdfs/
[root ~]# sudo -u hdfs hadoop fs -ls -d /user/hdfs/
Found 1 items
drwxr-xrwx - hdfs supergroup 0 2013-10-08 18:28 /user/hdfs

Try again:


[root ~]# sudo -u hive hive
hive> LOAD DATA INPATH '/user/hdfs/d_product_type_hive_exp.csv' OVERWRITE INTO TABLE D_PRODUCT_TYPE_HIVE;
Loading data to table default.d_product_type_hive
rmr: DEPRECATED: Please use 'rm -r' instead.
Moved: 'hdfs://xxxxx:8020/user/hive/warehouse/d_product_type_hive' to trash at: hdfs://xxxxx:8020/user/hive/.Trash/Current
Table default.d_product_type_hive stats: [num_partitions: 0, num_files: 1, num_rows: 0, total_size: 354, raw_data_size: 0]
OK
Time taken: 5.638 seconds
hive>
Читать полностью

,

HDFS: Failed to move to trash

понедельник, 24 июня 2013 г. 0 коммент.

I got an error:

[root@vm1 tmp]# hadoop fs -rmr /tmp/my_dir
13/06/21 17:51:40 WARN fs.TrashPolicyDefault: Can't create trash directory: hdfs://vm2:8020/user/root/.Trash/Current/tmp
rmr: Failed to move to trash: hdfs://vm2:8020/tmp/my_dir. Consider using -skipTrash option

I checked if there is a directory "/user/root" and tried to create one.

[root@vm1 tmp]# hadoop fs -ls /
Found 2 items
drwxrwxrwt - hdfs supergroup 0 2013-06-21 17:50 /tmp
drwxr-xr-x - hdfs supergroup 0 2013-06-20 14:25 /user
[root@vm1 tmp]# hadoop fs -ls /user
Found 2 items
drwxrwxr-t - hive hive 0 2013-06-20 14:24 /user/hive
drwxrwxr-x - oozie oozie 0 2013-06-20 14:25 /user/oozie
[root@vm1 tmp]# hadoop fs -mkdir /user/root
mkdir: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x

The owner of the directory "/ user" is hdfs:supergroup.
Let's create a home directory for the root.

[root@v1 tmp]# sudo -u hdfs hadoop fs -mkdir /user/root 
[root@v1 tmp]# sudo -u hdfs hadoop fs -chown root:root /user/root

Check now delete the file.

[root@v1 tmp]# hadoop fs -rm -r /tmp/my_dir
Moved: 'hdfs://vm2:8020/tmp/my_dir' to trash at: hdfs://vm2:8020/user/root/.Trash/Current

Читать полностью