位置：首页-资讯-后端开发

Hadoop配置机架感知(python脚

2023-01-31 07:29

短信预约 -IT技能 免费直播动态提醒

昨天QQ群里提了一个hadoop运行效率分配的问题，总结一下，写个文章。集群使用hadoop-1.0.3

有些hadoop集群在运行的时候，不完全是绝对平均的分配，不过需要尽可能平均的分配任务，避免某一台或者某几台服务器任务过重，其他服务器无事可做。这个，一方面是需要用到balancer，一个就是机架感知了。

通常，balancer是自动启动的。而机架感知则需要单独配置和编写脚本。不过，机架感知，不是说是感知哪个服务器坏了，是根据机架位置的拓扑结构来选取服务器进行任务的权重分配。

机架感知需要自己写一个脚本，然后放到hadoop的core-site.xml配置文件中，用namenode和jobtracker进行调用。

python代码摘自竹叶青的博客

#!/usr/bin/python
#-*-coding:UTF-8 -*-
import sys

rack = {"hadoop-node-31":"rack1",
                                "hadoop-node-32":"rack1",
                                "hadoop-node-33":"rack1",
                                "hadoop-node-34":"rack1",
                                "hadoop-node-49":"rack2",
                                "hadoop-node-50":"rack2",
                                "hadoop-node-51":"rack2",
                                "hadoop-node-52":"rack2",
                                "hadoop-node-53":"rack2",
                                "hadoop-node-54":"rack2",
                                "192.168.1.31":"rack1",
                                "192.168.1.32":"rack1",
                                "192.168.1.33":"rack1",
                                "192.168.1.34":"rack1",
                                "192.168.1.49":"rack2",
                                "192.168.1.50":"rack2",
                                "192.168.1.51":"rack2",
                                "192.168.1.52":"rack2",
                                "192.168.1.53":"rack2",
                                "192.168.1.54":"rack2",
                                }

if __name__=="__main__":
        print "/" + rack.get(sys.argv[1],"rack0")

按照老赵的博客，由于hadoop没有明确的说明机架感知是感知IP还是感知主机名，所以就都写上了。然后将脚本赋予可执行权限chmod +x RackAware.py，并放到bin/目录下。

然后打开conf/core-site.html

放入一下代码

<property>
        <name>topology.script.file.name</name>
        <value>/opt/modules/hadoop/hadoop-1.0.3/bin/RackAware.py</value>

    </property>
    <property>
        <name>topology.script.number.args</name>
        <value>20</value>

    </property>

然后重启hadoop的namenode和jobtracker，可以在logs里看下namenode和jobtracker的日志，看到机架感知功能已经启用了。

namenode日志

2012-06-08 14:42:19,174 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.49:50010 storage DS-1155827498-192.168.1.49-50010-1338289368956
2012-06-08 14:42:19,204 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/192.168.1.49:50010
2012-06-08 14:42:19,205 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.53:50010 storage DS-1773813988-192.168.1.53-50010-1338289405131
2012-06-08 14:42:19,226 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/192.168.1.53:50010
2012-06-08 14:42:19,226 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.34:50010 storage DS-2024494948-127.0.0.1-50010-1338289438983
2012-06-08 14:42:19,242 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack1/192.168.1.34:50010
2012-06-08 14:42:19,242 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.54:50010 storage DS-767528606-192.168.1.54-50010-1338289412267

......

2012-06-08 14:42:49,492 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 2 racks and 10 datanodes
2012-06-08 14:42:49,492 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks
2012-06-08 14:42:49,642 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: First cycle completed 0 blocks in 0 msec
2012-06-08 14:42:49,642 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: Queue flush completed 0 blocks in 0 msec processing time, 0 msec clock time, 1 cycles

jobtracker 日志

2012-06-08 14:50:42,728 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/hadoop-node-52
2012-06-08 14:50:42,731 INFO org.apache.hadoop.mapred.JobTracker: Adding tracker tracker_hadoop-node-52:localhost.localdomain/127.0.0.1:33999 to host hadoop-node-52
2012-06-08 14:50:42,753 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack1/hadoop-node-34
2012-06-08 14:50:42,754 INFO org.apache.hadoop.mapred.JobTracker: Adding tracker tracker_hadoop-node-34:localhost.localdomain/127.0.0.1:37025 to host hadoop-node-34
2012-06-08 14:50:42,773 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/hadoop-node-51

免责声明：

① 本站未注明“稿件来源”的信息均来自网络整理。其文字、图片和音视频稿件的所属权归原作者所有。本站收集整理出于非商业性的教育和科研之目的，并不意味着本站赞同其观点或证实其内容的真实性。仅作为临时的测试数据，供内部测试之用。本站并未授权任何人以任何方式主动获取本站任何信息。

② 本站未注明“稿件来源”的临时测试数据将在测试完成后最终做删除处理。有问题或投稿请发送至: 邮箱/279061341@qq.com QQ/279061341

机架 Hadoop python

阅读原文内容投诉