最近公司搞国产化，项目里用到的hadoop 2.10.1版本 不支持国产化，hadoop 3.1.1 版本支持ARM架构的国产CPU。
hadoop的版本修改势必会影响项目里对API、restfulAPI、flink with hadoop相关使用，所以本地先搭建一个hadoop 3.1.1 测试一下版本升级对项目的影响。

参考官网搭建一个本地的hadoop
官网：Hadoop: Setting up a Single Node Cluster.

1. Required Software

java 和 ssh

Java™ must be installed. Recommended Java versions are described at HadoopJavaVersions

ssh must be installed and sshd must be running to use the Hadoop scripts thatmanage remote Hadoop daemonsif the optional start and stop scripts are to be used.
Additionally, it is recommmended that pdsh also be installed for better ssh resource management.

2. Download

To get a Hadoop distribution, download a recent stable release from one of the Apache Download Mirrors.

3. Prepare to Start the Hadoop Cluster

解压并在 hadoop-env.sh 设置java home。

Unpack the downloaded Hadoop distribution. In the distribution, edit the file etc/hadoop/hadoop-env.sh to define some parameters as follows:

#setto the root of your Java installationexportJAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_271.jdk/Contents/Home

尝试命令

$ bin/hadoop

This will display the usage documentation for the hadoop script.

Now you are ready to start your Hadoop cluster in one of the three supported modes:
我们这次关注单节点的安装

Pseudo-Distributed Operation (伪集群模式)
Hadoop can also be run on a single-node in a pseudo-distributed mode where each Hadoop daemon runs in a separate Java process.

4. HDFS on a Single Node

4.1 configuration

a. etc/hadoop/core-site.xml:

要配置一个数据目录，用于存储数据，否则可能hadoop默认选择的目录没有权限使用。

org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:Directory/private/tmp/hadoop-lianggao/dfs/name is in an inconsistent state:storage directory doesnotexistorisnotaccessible.at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:392)at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:243)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1197)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:779)at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:677)at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:764)at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1018)at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:991)at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1767)at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1832)

<configuration><property><name>fs.defaultFS</name><value>hdfs://localhost:9000</value></property><property><name>hadoop.tmp.dir</name><value>/Users/lianggao/MyWorkSpace/002install/hadoop-3.3.1/hadoop_repo</value></property></configuration>

b. etc/hadoop/hdfs-site.xml:

<configuration><property><name>dfs.replication</name><value>1</value></property></configuration>

Setup passphraseless ssh (省略)

如需要参考官网：官网

4.2. Execution

a. Format the filesystem:

$ bin/hdfs namenode-format

看到这些信息格式化成功

2022-08-1217:43:11,039INFO common.Storage:Storage directory/Users/lianggao/MyWorkSpace/002install/hadoop-3.3.1/hadoop_repo/dfs/name has been successfully formatted.2022-08-1217:43:11,069INFO namenode.FSImageFormatProtobuf:Saving image file/Users/lianggao/MyWorkSpace/002install/hadoop-3.3.1/hadoop_repo/dfs/name/current/fsimage.ckpt_0000000000000000000usingno compression2022-08-1217:43:11,200INFO namenode.FSImageFormatProtobuf:Image file/Users/lianggao/MyWorkSpace/002install/hadoop-3.3.1/hadoop_repo/dfs/name/current/fsimage.ckpt_0000000000000000000 of size403bytes saved in0seconds.

如果格式化失败需要先删除nn的管理目录

因为格式化的时候是创建了nn文件的管理目录 common.Storage: Storage directory /data/hadoopdata/name has been successfully formatted.

b. Start NameNode daemon and DataNode daemon:

$ sbin/start-dfs.sh

c. Browse the web interface for the NameNode; by default it is available at:

NameNode - http://localhost:9870/

d. When you’re done, stop the daemons with:

$ sbin/stop-dfs.sh

5. YARN on a Single Node

5.1. Configure parameters as follows:

etc/hadoop/mapred-site.xml:

<configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property><property><name>mapreduce.application.classpath</name><value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value></property></configuration>

etc/hadoop/yarn-site.xml:

<configuration><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><property><name>yarn.nodemanager.env-whitelist</name><value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value></property></configuration>

5.2. Start ResourceManager daemon and NodeManager daemon:

$ sbin/start-yarn.sh

5.3. Browse the web interface for the ResourceManager; by default it is available at:

ResourceManager - http://localhost:8088/

5.4. When you’re done, stop the daemons with:

$ sbin/stop-yarn.sh