Monday, December 7, 2015

Spark History Server Error

Error:
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Exception in thread "main" java.lang.reflect.InvocationTargetException
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
 at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:233)
 at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: java.lang.IllegalArgumentException: Log directory specified does not exist: hdfs://master.example.com:8020/user/spark/applicationHistory.
 at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:172)
 at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:123)
 at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:116)
 at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:48)
 ... 6 more
Solutions:
[root@master ~]# sudo -u hdfs hadoop fs -mkdir /user/spark
[root@master ~]# sudo -u hdfs hadoop fs -mkdir /user/spark/applicationHistory
[root@master ~]# sudo -u hdfs hadoop fs -chown -R spark:spark /user/spark
[root@master ~]# sudo -u hdfs hadoop fs -chmod 1777 /user/spark/applicationHistory



No comments:

Post a Comment