Error: Permission denied: user=dr.who, access=READ_EXECUTE, inode="/user/spark":spark:spark:drwxr-x--x or Spark-shell or Spark-submit

ERROR SparkContext: Error initializing SparkContext.
org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
        at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
        at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
        at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:216)
        at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:145)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6609)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6591)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:6543)



Solutions:

sudo -u hdfs hadoop fs -mkdir /user/root
sudo -u hdfs hadoop fs -chown root /user/root

Flume Maven Build Error: Mysql Source

Tests run: 20, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 0.285 sec <<< FAILURE! - in org.keedio.flume.source.SQLSourceHelperTest
checkStatusFileCorrectlyUpdated(org.keedio.flume.source.SQLSourceHelperTest)  Time elapsed: 0.024 sec  <<< FAILURE!
java.lang.AssertionError: expected:<10> but was:<0>
 at org.junit.Assert.fail(Assert.java:88)
 at org.junit.Assert.failNotEquals(Assert.java:834)
 at org.junit.Assert.assertEquals(Assert.java:645)
 at org.junit.Assert.assertEquals(Assert.java:631)
 at org.keedio.flume.source.SQLSourceHelperTest.checkStatusFileCorrectlyUpdated(SQLSourceHelperTest.java:170)

getCustomQuery(org.keedio.flume.source.SQLSourceHelperTest)  Time elapsed: 0.005 sec  <<< ERROR!
org.apache.flume.conf.ConfigurationException: status.column.name property not set
 at org.keedio.flume.source.SQLSourceHelper.checkMandatoryProperties(SQLSourceHelper.java:313)
 at org.keedio.flume.source.SQLSourceHelper.<init>(SQLSourceHelper.java:100)
 at org.keedio.flume.source.SQLSourceHelperTest.getCustomQuery(SQLSourceHelperTest.java:239)


Results :

Failed tests: 
  SQLSourceHelperTest.checkStatusFileCorrectlyUpdated:170 expected:<10> but was:<0>
Tests in error: 
  SQLSourceHelperTest.getCustomQuery:239 ? Configuration status.column.name prop...

Solution:
Use this commands
#mvn clean package -DskipTests=true

Error: GPG key retrieval failed: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'archive.cloudera.com'"

Cloudera Manager Installation Problem


[root@master cloudera-manager-installer]# tail -f 2.install-oracle-j2sdk1.7.log
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package oracle-j2sdk1.7.x86_64 0:1.7.0+update67-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================
 Package             Arch       Version              Repository            Size
========================================================================
Installing:
 oracle-j2sdk1.7     x86_64     1.7.0+update67-1     cloudera-manager     135 M

Transaction Summary
========================================================================
Install       1 Package(s)

Total download size: 135 M
Installed size: 279 M
Downloading Packages:
warning: rpmts_HdrFromFdno: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Retrieving key from http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera


GPG key retrieval failed: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'archive.cloudera.com'"

Solutions:

Install Perl


[root@master cloudera-manager-installer]# yum install perl -y

Hadoop Cluster:Passwordless SSH setup on CentOs In 3 Steps

Step 1:

[root@pravi ~]# yum install openssh-clients -y 

[root@pravi ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
e7:55:ae:cd:e0:b3:91:4a:36:53:a7:d7:62:d2:9a:d8 root@master.techvantage.com
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|              .  |
|             o   |
|        S . + o  |
|         o + X . |
|          * O B .|
|         o * X . |
|          o E    |
+-----------------+


~/.ssh/id_rsa [private key]
~/.ssh/id_rsa.pub [public key]

Step 2:

[root@master ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub <slave-ip>
Now try logging into the machine, with "ssh 'slave'", and check in:

  .ssh/authorized_keys


to make sure we haven't added extra keys that you weren't expecting.

Step 3

[root@master ~]# ssh <slave-hostname>

Spark History Server Error

Error:

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

Exception in thread "main" java.lang.reflect.InvocationTargetException
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
 at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:233)
 at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: java.lang.IllegalArgumentException: Log directory specified does not exist: hdfs://master.example.com:8020/user/spark/applicationHistory.
 at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:172)
 at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:123)
 at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:116)
 at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:48)
 ... 6 more

Solutions:

[root@master ~]# sudo -u hdfs hadoop fs -mkdir /user/spark
[root@master ~]# sudo -u hdfs hadoop fs -mkdir /user/spark/applicationHistory
[root@master ~]# sudo -u hdfs hadoop fs -chown -R spark:spark /user/spark
[root@master ~]# sudo -u hdfs hadoop fs -chmod 1777 /user/spark/applicationHistory