我正在尝试在Hadoop上执行map reduce程序。
当我将工作提交到hadoop单节点集群时。正在创建作业,但消息失败
“被ApplicationMaster杀死的容器”
使用的输入大小为10 MB。
当我使用输入文件400 KB的相同脚本时,它成功了。但是输入大小为10 MB的输入文件失败。
在我的终端中显示的完整日志如下。
15/05/29 09:52:16 WARN util.NativeCodeLoader: Unable to `load native- hadoop library for your platform... using builtin-java classes where applicable Submitting job on the cluster... 15/05/29 09:52:17 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 15/05/29 09:52:18 INFO input.FileInputFormat: Total input paths to process : 1 15/05/29 09:52:18 INFO mapreduce.JobSubmitter: number of splits:1 15/05/29 09:52:19 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1432910768528_0001 15/05/29 09:52:19 INFO impl.YarnClientImpl: Submitted application application_1432910768528_0001 15/05/29 09:52:19 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1432910768528_0001/ 15/05/29 09:52:19 INFO mapreduce.Job: Running job: job_1432910768528_0001 15/05/29 09:52:29 INFO mapreduce.Job: Job job_1432910768528_0001 running in uber mode : false 15/05/29 09:52:29 INFO mapreduce.Job: map 0% reduce 0% 15/05/29 09:52:41 INFO mapreduce.Job: map 100% reduce 0% 15/05/29 10:03:01 INFO mapreduce.Job: map 0% reduce 0% 15/05/29 10:03:01 INFO mapreduce.Job: Task Id : attempt_1432910768528_0001_m_000000_0, Status : FAILED AttemptID:attempt_1432910768528_0001_m_000000_0 Timed out after 600 secs Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143
我的映射器在这里触发另一个程序,该程序将在这里处理我的输入文件。由映射器触发的程序通常会占用大量内存。
因此,请在这方面帮助我。
在以下属性中添加yarn-site.xml并重新启动VM,
yarn-site.xml
VM
<property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> <description>Whether virtual memory limits will be enforced for containers</description> </property> <property> <name>yarn.nodemanager.vmem-pmem-ratio</name> <value>4</value> <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description> </property>