13518219792
这篇文章主要介绍了Spark中会遇到什么坑，具有一定借鉴价值，感兴趣的朋友可以参考下，希望大家阅读完这篇文章之后大有收获，下面让小编带着大家一起了解一下。
十载专注建站、设计、互联网产品按需定制制作服务，业务涵盖高端网站设计、商城网站开发、重庆小程序开发、软件系统开发、app软件定制开发等。凭借多年丰富的经验，我们会仔细了解每个客户的需求而做出多方面的分析、设计、整合，为客户设计出具风格及创意性的商业解决方案，创新互联公司更提供一系列网站制作和网站推广的服务，以推动各中小企业全面信息数字化，并利用创新技术帮助各行业提升企业形象和运营效率。
1.1 Scala与Intellij集成报错
在Scala安装成功后，准备到Intellij上写Scala代码，发现Scala都配好了(关于如何配置，网上资料很多)，结果运行Scala程序时报错。
错误：
Error:scalac: Multiple 'scala-library*.jar' files (scala-library.jar, scala-library.jar, scala-library.jar) in Scala compiler classpath in Scala SDK scala-sdk-2.12.2
解决方法：在OverStackflow上找到了思路。在Intellij中打开project structure，删除已有的Scala的路径(我的Scala是安装在/usr/local/Cellar/scala/2.12.2路径下的),重新添加/usr/local/Cellar/scala/2.12.2/idea/lib目录即可。
改动前
改动后
1.2 Scala语法Intellij不认
在Intellij中写了一个Scala的HelloWorld，代码如下
/** * Created by jackie on 17/5/7. */ package com.jackie.scala.s510 object HelloWorld { def main(args: Array[String]): Unit = { println("hello world") println(increaseAnother(5)); println(Array(1,2,3,4).map{(x:Int)=>x+1}.mkString(",")); println(Array(1,2,3,4) map{(x:Int)=>x+1} mkString(",")); println(Array(1,2,3,4) map{(x:Int)=>x+1} mkString(",")); // test object var person = new Person() person.name_=("john") // name_=()对应java中的setter方法 println("Person name:" + person.name) person.name = "Jackie" println("Person name:" + person.name) var mp = new MyPerson() mp.name_("alihaha") println("MyPerson name:" + person.name) var pwp = new PersonWithParam("Jackie", 18) println("PersonWithParam:" + pwp.toString()) } def increaseAnother(x: Int): Int = x + 1 }运行的时候，报错mkString无法识别。
错误：mkString can't be resolved
解决方法：需要交代下我各个环境的版本参数，Intellij-14.0， jdk-8， scala-2.12.2。但是在Intellij中能选择的Scala***版本只有2.11，所有后来将Intellij升级到2017.1版本，这时候还报错Error:scalac: Error: org.jetbrains.jps.incremental.scala.remote.ServerException，然后在Intellij中打开project structure，将scala由2.12.2换成2.11.7，问题解决。
1.3 Spark与Intellij集成的问题
Spark环境都安装好了，所以想在Intellij中运行Spark程序，但是在添加了Spark的相关依赖后，发现无法编译通过。
错误：
Exception NoSuchMethodError: com.google.common.collect.MapMaker.keyEquivalence
解决方法：实现声明，之前在maven中一直引用的都是spark-core2.10，这时候报错，我定位问题出在Guava上，然后找到所有间接依赖了Guava的jar，都exclude，问题还是没有解决。期间添加了Spark的很多依赖，试了都不行，***试了下Spark-core2.11，问题解决(有的时候版本的兼容性真的很坑)。
1.4 hadoop上传本地文件到HDFS
如果想将本地文件上传到HDFS，使用hadoop fs -put localDir hdfsDir，前提是保证hadoop启动。
错误：
jackie@jackies-MacBook-Pro:~|⇒ hadoop fs -put ~/Documents/doc/README.md / 17/05/13 10:56:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/05/13 10:56:40 WARN ipc.Client: Failed to connect to server: localhost/127.0.0.1:8020: try once and fail. java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:681) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:777) at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:409) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1542) at org.apache.hadoop.ipc.Client.call(Client.java:1373) at org.apache.hadoop.ipc.Client.call(Client.java:1337) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:787) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335) at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1700) at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1436) at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1433) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1433) at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:64) at org.apache.hadoop.fs.Globber.doGlob(Globber.java:282) at org.apache.hadoop.fs.Globber.glob(Globber.java:148) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1685) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326) at org.apache.hadoop.fs.shell.CommandWithDestination.getRemoteDestination(CommandWithDestination.java:195) at org.apache.hadoop.fs.shell.CopyCommands$Put.processOptions(CopyCommands.java:256) at org.apache.hadoop.fs.shell.Command.run(Command.java:164) at org.apache.hadoop.fs.FsShell.run(FsShell.java:315) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.FsShell.main(FsShell.java:378) put: Call From jackies-macbook-pro.local/192.168.73.56 to localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
解决方法：进入hadoop安装目录(我的是/usr/local/Cellar/hadoop)进入sbin下执行./start-all.sh启动hadoop服务。
1.5 Spark启动
上篇在配置Spark时没有配置spark-defaults.conf文件，所以在Spark安装目录下(我的是/usr/local/Spark)启动./start-all.sh出错。
错误：
spark-shell Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). 17/05/13 13:42:49 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/05/13 13:42:51 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master 192.168.73.56:7077 org.apache.spark.SparkException: Exception thrown in awaitResult at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77) at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167) at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83) at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:88) at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:96) at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1$$anon$1.run(StandaloneAppClient.scala:106) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Failed to connect to /192.168.73.56:7077
解决方法：将Spark安装目录下的conf中的spark-defaults.conf.template拷贝一份出来，重命名为spark-defaults.conf，按照https://sanwen8.cn/p/3bac5Bj.html配置好，再启动Spark，发现还是报错
https://sanwen8.cn/p/3bac5Bj.html Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). 17/05/13 14:19:12 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/05/13 14:19:15 ERROR SparkContext: Error initializing SparkContext. java.net.ConnectException: Call From jackies-MacBook-Pro.local/192.168.73.56 to 192.168.73.56:8021 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
于是按照StackOverflow，将spark-defaults.conf中的spark.eventLog.enabled由true改为false，之后再启动成功。
注意：这里我反复配置了localhost和自己的ip，来回切换，最终证明只要在/etc/hosts中配置好ip对应映射的名称，可以直接用名称即可，不用写ip，而且要保持hadoop中的配置文件和spark中的配置文件要一致，否则针对会精疲力尽。
1.6 将运算任务交给Spark运行的报错
运行下面的一个Demo程序
package com.jackie.scala.s513; import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaPairRDD; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.api.java.function.FlatMapFunction; import org.apache.spark.api.java.function.Function2; import org.apache.spark.api.java.function.PairFunction; import scala.Tuple2; import java.util.Arrays; import java.util.Iterator; import java.util.List; import java.util.regex.Pattern; /** * Created by jackie on 17/5/13. */ public class Simple { private static final Pattern SPACE = Pattern.compile(" "); public static void main(String[] args) throws Exception { //创建一个RDD对象 SparkConf conf=new SparkConf().setAppName("Simple").setMaster("local"); //创建spark上下文对象，是数据的入口 JavaSparkContext spark=new JavaSparkContext(conf); //获取数据源 JavaRDD lines = spark.textFile("hdfs://jackie:8020/"); /** * 对于从数据源得到的DStream，用户可以在其基础上进行各种操作， * 对于当前时间窗口内从数据源得到的数据首先进行分割， * 然后利用Map和ReduceByKey方法进行计算，当然***还有使用print()方法输出结果； */ JavaRDD words = lines.flatMap(new FlatMapFunction() { @Override public Iterator call(String s) { return Arrays.asList(SPACE.split(s)).iterator(); } }); //使用RDD的map和reduce方法进行计算 JavaPairRDD ones = words.mapToPair( new PairFunction() { @Override public Tuple2 call(String s) { return new Tuple2(s, 1); } }); JavaPairRDD counts = ones.reduceByKey( new Function2() { @Override public Integer call(Integer i1, Integer i2) { return i1 + i2; } }); List> output = counts.collect(); for (Tuple2 tuple : output) { //输出计算结果 System.out.println(tuple._1() + ": " + tuple._2()); } spark.stop(); } } 这个程序需要读取HDFS上根目录下的README.md文件，但是在此之前我执行了"hadoop namenode -format"(注意，这个操作引起了后面的一系列问题)。所以就准备重新使用hadoop fs -put localDir hdfsDir上传README.md，结果这时候报错
错误：
hadoop fs -put /Users/jackie/Documents/doc/README.md / 17/05/13 15:47:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/05/13 15:47:16 WARN hdfs.DataStreamer: DataStreamer Exception org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /README.md._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1733) at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:265) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2496) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:828)
后来发现是datanode没有启动，然后开始找datanode没有启动的原因，在这里http://www.aboutyun.com/thread-7931-1-1.html
文中解释：当我们执行文件系统格式化时，会在namenode数据文件夹(即配置文件中dfs.name.dir在本地系统的路径)中保存一个current/VERSION文件，记录namespaceID，标识了所格式化的 namenode的版本。如果我们频繁的格式化namenode，那么datanode中保存(即配置文件中dfs.data.dir在本地系统的路径)的current/VERSION文件只是你***次格式化时保存的namenode的ID，因此就会造成datanode与namenode之间的id不一致。
解决方法：采取的做法是根据执行hadoop namenode –format得到成功的提示。
这时候再执行jps命令，我们就可以看到datanode了
类似的，同样是在执行hadoop fs -put /Users/jackie/Documents/doc/README.md /是报错如下
hadoop fs -put /Users/jackie/Documents/doc/README.md / 17/05/15 09:51:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/05/15 09:51:05 WARN ipc.Client: Failed to connect to server: jackie/192.168.73.56:8020: try once and fail. java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:681) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:777) at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:409)
一开始以为是ip的配置问题，但是反复修改无果，后来发现使用jps时，没有启动namenode，于是在网上找http://blog.csdn.net/bychjzh/article/details/7830508
于是在/usr/local/Cellar/hadoop/hdfs下删除原来在core-site.xml中配置的tmp目录，然后新建了hadoop_tmp目录，并在core-site.xml中修改成
hadoop.tmp.dir /usr/local/Cellar/hadoop/hdfs/hadoop_tmp A base for other temporary directories.
并执行hadoop namenode –format，***在使用start-all.sh启动所有的服务，执行上传文件成功
感谢你能够认真阅读完这篇文章，希望小编分享的“Spark中会遇到什么坑”这篇文章对大家有帮助，同时也希望大家多多支持创新互联，关注创新互联行业资讯频道，更多相关知识等着你来学习!
