13518219792
hadoop2.7.0集群在使用中遇到的bug及解决办法是什么，很多新手对此不是很清楚，为了帮助大家解决这个难题，下面小编将为大家详细讲解，有这方面需求的人可以来学习下，希望你能有所收获。
韩城ssl适用于网站、小程序/APP、API接口等需要进行数据传输应用场景，ssl证书未来市场广阔！成为成都创新互联公司的ssl证书销售渠道，可以享受市场价格4-6折优惠！如果有意向欢迎电话联系或者加微信：18982081108（备注：SSL证书合作）期待与您的合作！
hadoop环境是2.7.0的集群环境，使用sqoop 1.4.6执行从MySQL向hive的数据导入。
执行过程中报错，如下方的日志信息。但是查询hive中的数据，发现实际数据已经过来了，但因为mysql的数据表较多，不能一一对应的比对一下。所以为了确保同步数据成功，需要重新的正确的执行一次。
15/09/28 10:22:01 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /data/hadoop/share/hadoop/mapreduce Note: /tmp/sqoop-hadoop/compile/60bb7ee51d4794512d28b8efc4029fbc/QueryResult.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 15/09/28 10:22:06 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/60bb7ee51d4794512d28b8efc4029fbc/QueryResult.jar 15/09/28 10:22:09 INFO tool.ImportTool: Destination directory /tmp/wfpuser_t0301 is not present, hence not deleting. 15/09/28 10:22:09 INFO mapreduce.ImportJobBase: Beginning query import. 15/09/28 10:22:09 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 15/09/28 10:22:09 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 15/09/28 10:22:09 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 15/09/28 10:22:09 INFO client.RMProxy: Connecting to ResourceManager at zhebuduan-bd-3/192.168.1.113:8032 15/09/28 10:22:15 INFO db.DBInputFormat: Using read commited transaction isolation 15/09/28 10:22:15 INFO mapreduce.JobSubmitter: number of splits:1 15/09/28 10:22:16 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1443364253801_0310 15/09/28 10:22:17 INFO impl.YarnClientImpl: Submitted application application_1443364253801_0310 15/09/28 10:22:18 INFO mapreduce.Job: The url to track the job: http://zhebuduan-bd-3:8088/proxy/application_1443364253801_0310/ 15/09/28 10:22:18 INFO mapreduce.Job: Running job: job_1443364253801_0310 15/09/28 10:22:31 INFO mapreduce.Job: Job job_1443364253801_0310 running in uber mode : false 15/09/28 10:22:31 INFO mapreduce.Job: map 0% reduce 0% 15/09/28 10:22:34 INFO mapreduce.Job: Task Id : attempt_1443364253801_0310_m_000000_0, Status : FAILED Rename cannot overwrite non empty destination directory /data/hadoop/data/tmp/nm-local-dir/usercache/hadoop/filecache/60 java.io.IOException: Rename cannot overwrite non empty destination directory /data/hadoop/data/tmp/nm-local-dir/usercache/hadoop/filecache/60 at org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:735) at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:236) at org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:678) at org.apache.hadoop.fs.FileContext.rename(FileContext.java:958) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:366) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 15/09/28 10:22:38 INFO mapreduce.Job: Task Id : attempt_1443364253801_0310_m_000000_1, Status : FAILED Rename cannot overwrite non empty destination directory /data/hadoop/data/tmp/nm-local-dir/usercache/hadoop/filecache/62 java.io.IOException: Rename cannot overwrite non empty destination directory /data/hadoop/data/tmp/nm-local-dir/usercache/hadoop/filecache/62 at org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:735) at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:236) at org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:678) at org.apache.hadoop.fs.FileContext.rename(FileContext.java:958) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:366) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 15/09/28 10:22:42 INFO mapreduce.Job: map 100% reduce 0% 15/09/28 10:22:42 INFO mapreduce.Job: Task Id : attempt_1443364253801_0310_m_000000_2, Status : FAILED Rename cannot overwrite non empty destination directory /data/hadoop/data/tmp/nm-local-dir/usercache/hadoop/filecache/64 java.io.IOException: Rename cannot overwrite non empty destination directory /data/hadoop/data/tmp/nm-local-dir/usercache/hadoop/filecache/64 at org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:735) at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:236) at org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:678) at org.apache.hadoop.fs.FileContext.rename(FileContext.java:958) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:366) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 15/09/28 10:22:43 INFO mapreduce.Job: map 0% reduce 0% 15/09/28 10:23:00 INFO mapreduce.Job: map 100% reduce 0% 15/09/28 10:23:00 INFO mapreduce.Job: Job job_1443364253801_0310 completed successfully 15/09/28 10:23:00 INFO mapreduce.Job: Counters: 31 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=140349 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=87 HDFS: Number of bytes written=3712573 HDFS: Number of read operations=4 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Failed map tasks=3 Launched map tasks=4 Other local map tasks=4 Total time spent by all maps in occupied slots (ms)=20017 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=20017 Total vcore-seconds taken by all map tasks=20017 Total megabyte-seconds taken by all map tasks=20497408 Map-Reduce Framework Map input records=12661 Map output records=12661 Input split bytes=87 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=177 CPU time spent (ms)=8810 Physical memory (bytes) snapshot=175165440 Virtual memory (bytes) snapshot=880988160 Total committed heap usage (bytes)=197132288 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=3712573
在网上查了一下问题原因，最终找到一个解决办法，删除/data/hadoop/data/tmp/nm-local-dir/usercache/hadoop/filecache目录下的缓存文件，直接进入该目录，执行rm -rf *，本来想备份一下，tar命令打包怎么也执行不完，所以直接删掉了。注意在集群关闭的时候执行。启动集群后，操作不再报错。
但是还有个问题，执行hadoop dfsadmin -report后，提示datanode节点的状态为
Decommission Status : Normal Configured Capacity: 1055816155136 (983.31 GB) DFS Used: 267768670295 (249.38 GB) Non DFS Used: 59758983081 (55.65 GB) DFS Remaining: 728288501760 (678.27 GB) DFS Used%: 25.36% DFS Remaining%: 68.98% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 10 Last contact: Mon Sep 28 15:43:20 CST 2015
cache used和cache remaining都是0了，这个不知道怎么搞定，虽然当前没看出什么问题，看着膈应啊。谁知道帮个忙告诉我吧~
我在官网找到了这个bug，在2.7.1版本中已经修复了这个bug，对集群进行升级。
看完上述内容是否对您有帮助呢？如果还想对相关知识有进一步的了解或阅读更多相关文章，请关注创新互联行业资讯频道，感谢您对创新互联的支持。
