共計(jì) 4281 個(gè)字符，預(yù)計(jì)需要花費(fèi) 11 分鐘才能閱讀完成。
丸趣 TV 小編給大家分享一下 hadoop 2 偽分布式如何搭建，希望大家閱讀完這篇文章之后都有所收獲，下面讓我們一起去探討吧！
單點(diǎn)偽分布式
1. 下載 hadoop
2. 安裝 jdk，設(shè)置環(huán)境變量
export JAVA_HOME=/usr/local/java/jdk1.7.0_79
export JRE_HOME=$JAVA_HOME/jre
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=./:/$JAVA_HOME/lib:$JAVA_HOME/jre/lib
[root@iZ281cu2lqjZ etc]# source /etc/profile
3. 創(chuàng)建用于和組
groupadd hadoop
useradd -g hadoop yarn
useradd -g hadoop hdfs
useradd -g hadoop mapred
4. 創(chuàng)建數(shù)據(jù)和日志目錄
mkdir -p /var/data/hadoop/hdfs/nn
mkdir -p /var/data/hadoop/hdfs/snn
mkdir -p /var/data/hadoop/hdfs/dn
chown hdfs:hadoop /var/data/hadoop/hdfs -R
mkdir -p /var/log/hadoop/yarn
chown yarn:hadoop /var/log/hadoop/yarn -R
進(jìn)入 yarnhadoop 的目錄中
mkdir logs
chmod g+w logs
chown yarn:hadoop . -R
5. 配置 core-site.xml
property
name fs.default.name /name
value hdfs://localhost:9000 /value
/property
property
name hadoop.http.staticuser.user /name
value hdfs /value
/property
fs.default.name 是 Namenode 指定了主機(jī)名和請(qǐng)求端口號(hào)；
hadoop.http.staticuser.user 是 hdfs 的默認(rèn)用戶名
6. 配置 hdfs.site.xml
property
name dfs.replication /name
value 1 /value — 默認(rèn)為 3
/property
property
name dfs.namenode.name.dir /name
value file:/var/data/hadoop/hdfs/nn /value
/property
property
name fs.checkpoint.dir /name
value file:/var/data/hadoop/hdfs/snn /value
/property
property
name fs.checkpoint.edit.dir /name
value file:/var/data/hadoop/hdfs/snn /value
/property
property
name dfs.datanode.data.dir /name
value file:/var/data/hadoop/hdfs/db /value
/property
7. 配置 mapred-site.xml
property
name mapreduce.framework.name /name
value yarn /value
/property
為 mapreduce 制定框架名，使用 yarn
8 配置 yarn-site.xml
property
name yarn.nodemanager.aux-services /name
value mapreduce_shuffle /value
/property
property
name yarn.nodemanager.aux-services.mapreduce.shuffle.class /name
value org.apache.hadoop.mapred.ShuffleHandler /value
/property
主要是配置 shuffle，默認(rèn)是沒有配置 shuffle
9. 調(diào)整 JAVA 堆大小
hadoop-env.sh
HADOOP_HEAPSIZE= 500
yarn-env.sh
YARN_HEAPSIZE=500
10. 格式化 HDFS
切換到 hdfs 用戶，進(jìn)入到 hadoop bin 目錄執(zhí)行
./hdfss namenode -format
碰到問題
11../hdfs namenode -format
12.[hdfs@localhost sbin]$ ./hadoop-daemon.sh start namenode
starting namenode, logging to /opt/yarn/hadoop-2.7.1/logs/hadoop-hdfs-namenode-localhost.out
[hdfs@localhost sbin]$ ./hadoop-daemon.sh start secondarynamenode
starting secondarynamenode, logging to /opt/yarn/hadoop-2.7.1/logs/hadoop-hdfs-secondarynamenode-localhost.out
[hdfs@localhost sbin]$ ./hadoop-daemon.sh start datanode
starting datanode, logging to /opt/yarn/hadoop-2.7.1/logs/hadoop-hdfs-datanode-localhost.out
使用 JPS 可以檢查進(jìn)程
結(jié)果：
[hdfs@localhost sbin]$ jps
3915 SecondaryNameNode
3969 DataNode
3833 NameNode
4047 Jps
12. 啟動(dòng) yarn
[yarn@localhost sbin]$ ./yarn-daemon.sh start nodemanager
/opt/yarn/hadoop-2.7.1/etc/hadoop/yarn-env.sh: line 121: unexpected EOF while looking for matching `
/opt/yarn/hadoop-2.7.1/etc/hadoop/yarn-env.sh: line 124: syntax error: unexpected end of file
starting nodemanager, logging to /opt/yarn/hadoop-2.7.1/logs/yarn-yarn-nodemanager-localhost.out
/opt/yarn/hadoop-2.7.1/etc/hadoop/yarn-env.sh: line 121: unexpected EOF while looking for matching `
/opt/yarn/hadoop-2.7.1/etc/hadoop/yarn-env.sh: line 124: syntax error: unexpected end of file
[yarn@localhost sbin]$ jps
4132 ResourceManager
4567 Jps
4456 NodeManager
13 驗(yàn)證
訪問: ip:50070
ip:8088
最后可以運(yùn)行 hadoop 包中的例子進(jìn)行檢驗(yàn)。這就是偽分布式簡單的安裝步驟。
問題：如果遇到 50070 端口不能訪問就是 yarn 沒有成功啟動(dòng)。
去 sbin 啟動(dòng) yarn ./start-yarn.sh 時(shí)報(bào)
localhost: Error: JAVA_HOME is not set and could not be found.
需要修改 hadoop-env.xml 中 java_home 設(shè)置為絕對(duì)路徑。
重啟 yarn 解決問題。
配置 hbase：
修改 hbase-env.sh
修改# export JAVA_HOME=/usr/local/java/jdk1.7.0_79
修改 hbase-site.xml
configuration
property
name hbase.rootdir /name
value hdfs://localhost:9000/hbase /value
/property
property
name dfs.replication /name
value 1 /value
/property
property
name hbase.cluster.distributed /name
value true /value
/property
/configuration
啟動(dòng) hbase：
[root@iZ281cu2lqjZ bin]# ./start-hbase.sh
root@localhost s password:
localhost: starting zookeeper, logging to /usr/local/hbase/hbase-1.1.4/bin/../logs/hbase-root-zookeeper-iZ281cu2lqjZ.out
starting master, logging to /usr/local/hbase/hbase-1.1.4/bin/../logs/hbase-root-master-iZ281cu2lqjZ.out
starting regionserver, logging to /usr/local/hbase/hbase-1.1.4/bin/../logs/hbase-root-1-regionserver-iZ281cu2lqjZ.out
[root@iZ281cu2lqjZ bin]# jps
1597 DataNode
3180 ResourceManager
3463 NodeManager
1462 NameNode
8680 HRegionServer
1543 SecondaryNameNode
8536 HQuorumPeer
8597 HMaster
8729 Jps
完成
看完了這篇文章，相信你對(duì)“hadoop 2 偽分布式如何搭建”有了一定的了解，如果想了解更多相關(guān)知識(shí)，歡迎關(guān)注丸趣 TV 行業(yè)資訊頻道，感謝各位的閱讀！
