概述:本次实验中,在搭建Spark Standalone模式分布式集群时花了大量时间,除此之外,按照教程没遇到任何问题,都能成功简单复现.

Standalone 分布式集群搭建遇到的麻烦

ulimit -f 1024000
cd /home

注意到wrapdocker会出现iptable相关错误,并且尝试更新,修复模块失败(推测是禁止了外网网络连接)
所以采用禁止iptable方式启动docker

sudo dockerd --iptables=false > /var/log/docker.log 2>&1 &

加载镜像

docker load -i hbase-ssh2_v1.0.tar

由于禁用iptable,所以为了让容器能互相访问,我选择自建一个网络

编辑docker-compose.yml
vi docker-compose.yml

1,%d 清空后粘贴下面代码

version: '2'
services:
  master:
    image: hbase-ssh:v1.1
    container_name: master
    hostname: master
    networks:
      spark-network:
        ipv4_address: 172.18.0.2  # 静态IP
    ports:
      - "19001:22"
      - "50070:50070"
      - "8088:8088"
      - "16010:16010"
    restart: always

  slave1:
    image: hbase-ssh:v1.1
    container_name: slave1
    hostname: slave1
    networks:
      spark-network:
        ipv4_address: 172.18.0.3  # 静态IP
    depends_on:
      - master
    ports:
      - "19002:22"
    restart: always

  slave2:
    image: hbase-ssh:v1.1
    container_name: slave2
    hostname: slave2
    networks:
      spark-network:
        ipv4_address: 172.18.0.4  # 静态IP
    depends_on:
      - slave1
    ports:
      - "19003:22"
    restart: always

# 定义自定义网络
networks:
  spark-network:
    driver: bridge
    ipam:
      config:
        - subnet: 172.18.0.0/16
          gateway: 172.18.0.1

启动三个容器

docker-compose up -d

开三个终端,分别进入三个节点

docker exec -it master bash
docker exec -it slave1 bash
docker exec -it slave2 bash


在master节点
echo "172.18.0.3 slave1" >> /etc/hosts 

echo "172.18.0.4 slave2" >> /etc/hosts

在slave1 slave2节点

echo "172.18.0.2 master" >> /etc/hosts 

通过echo $JAVA_HOME得到
/usr/lib/jvm/jdk1.8.0_111

其余内容,包括节点间的ssh免密登录.spark配置文件编辑修改,分发等等都不算困难.

主要是开头遇到问题,容器之间的连接遇到麻烦,解决后便一马平川.