Kubernetes部署大数据组件系列一:一键部署Zookeeper集群

发布于 2019-09-26 作者 风铃 75次 浏览 版块 后端


目前的大数据平台存在的问题:



  • 通过手工方式直接部署在物理机上,过程复杂、效率低,不适合大规模集群的部署和动态扩容。

  • 由于平台组件繁多,对宿主机底层库依赖过高,一旦安装完成,很难进行迁移,也无法做成模版进行重复使用。

  • 组件升级不够灵活,往往会对有相同底层依赖的其他组件带来影响。

  • 采用虚机部署时,会对宿主机产生资源浪费、开销大。

  • 物理机或虚机关启等管理操作速度较慢。

  • 无法在同一套物理集群上部署多套大数据平台,使用虚机的方案太耗费资源。


针对上述问题,我们打算用时下流行的Docker容器来封装大数据各个组件,用Google的神器Kubernetes来编排这些Docker容器,最终达到:



  • 平台能够自动化的快速部署(小规模集群能够达到秒级)、动态扩容。

  • 平台构建一次,就能够形成模版重复使用、灵活迁移。

  • 平台构建一次,就能够灵活部署在不同的环境中(比如公有云、私有云、裸机,虚拟机)。

  • 组件能够灵活升级、灵活替代。

  • 消除线上线下环境不一致导致的问题,再也不会有“在我的机器上运行的好好的”这种托词。

  • 因为其多资源的低消耗,在物理集群上能同时部署多套大数据平台,供不同部门独
    享。

  • 很适合持续集成/持续部署(CI/CD)的环境,因为可以自动的下载、绿色的安装和彻底清除应用程序,不会影响宿主机的环境。


本篇是该系列的第一篇,手把手教你通过K8s部署zookeeper集群,涉及到的概念,各位看官自行脑补,我只讲一下部署过程,直接上干货了。


- 使用Kargo搭建Kubernetes 1.6集群:























地址 角色
10.0.7.14 Ansible
Client (Kargo用到的工具)
10.0.8.182 Master, NODE 1
10.0.8.183 NODE 2

[下文提到的kargo.tg、kargo_images_v1.6.1.tar.gz压缩包]http://pan.baidu.com/s/1kUKyfBL


1,我用笔记本(macOS系统),作为Ansible client。先新建一个文件夹kargo,把kargo.tgz放进去并解压,目录结构如下:






2,需要安装python 2.7、pip 9.0.1、python-netaddr、ansible 2.3.1.0等工具
[ansible安装]http://valdhaus.co/writings/ansible-mac-osx/


3,修改 inventory/inventory.cfg

wangliangdeMacBook-Pro:kargo wangliang$ pwd
	/Users/wangliang/kargo/kargo
	wangliangdeMacBook-Pro:kargo wangliang$ vim inventory/inventory.cfg
	
	[all]
	node1    ansible_user=app ansible_host=10.0.8.182  ip=10.0.8.182 access_ip=10.0.8.182
	node2    ansible_user=app ansible_host=10.0.8.183  ip=10.0.8.183 access_ip=10.0.8.183
	
	[kube-master]
	node1    ip=10.0.8.182 access_ip=10.0.8.182
	
	[kube-node]
	node1
	node2
	
	[etcd]
	node1
	
	[k8s-cluster:children]
	kube-node
	kube-master
	
	[calico-rr]

4,分别在10.0.8.182(Node 1)、10.0.8.183(Node 2)上做如下操作:
首先安装Docker 1.13.1,并启动服务
[Ubuntu安装Docker]http://www.linuxidc.com/Linux/2017-01/139985.htm


其次把kargo_images_v1.6.1.tar.gz 拷贝到要安装k8s的节点上,放到k8s文件夹内,并解压,目录结构如下:











5,在kargo_images_v1.6.1目录下,进入bash,执行:

➜  k8s cd kargo_images_v1.6.1
	➜  kargo_images_v1.6.1 pwd
	/data/k8s/kargo_images_v1.6.1
	➜  kargo_images_v1.6.1 bash
	app@yuyan1:/data/k8s/kargo_images_v1.6.1$images=$(ls -l ../kargo_images_v1.6.1|awk -F' ' '{ print $9 }')
	
	app@yuyan1:/data/k8s/kargo_images_v1.6.1$ for x in ${images[*]}; do sudo docker load -i $x; done

将目录下的镜像放入docker images。






6, Node1、Node2和Mac间都要SSH免密,Mac上还要打开共享里面的允许远端访问



Node1、Node2、Mac都在用户目录下执行

➜  ~ cd /home/app
	ssh-keygen -t rsa  #一路回车就行
	Generating public/private rsa key pair.
	Enter file in which to save the key (/home/app/.ssh/id_rsa):
	Created directory '/home/app/.ssh'.
	Enter passphrase (empty for no passphrase):
	Enter same passphrase again:
	Your identification has been saved in /home/app/.ssh/id_rsa.
	Your public key has been saved in /home/app/.ssh/id_rsa.pub.
	The key fingerprint is:
	7c:f4:2f:21:f2:85:7f:20:38:02:1d:ef:79:39:b6:be root@dcos1
	The key's randomart image is:
	+--[ RSA 2048]----+
	|      .          |
	|     . o         |
	|    . . . .      |
	|     . o + +     |
	|      . S X =    |
	|       . B B +   |
	|          o o o  |
	|         .   o   |
	|          E.     |
	+-----------------+
	
	➜  ~ cd /home/app/.ssh
	➜  .ssh ssh-copy-id Node2
	➜  .ssh ssh-copy-id Mac
	
	每次执行,步骤都如下类似:
	The authenticity of host 'Node1 (10.0.8.183)' can't be established.
	ECDSA key fingerprint is 91:1a:13:a8:57:2b:a0:42:4d:aa:c9:83:c3:33:16:f9.
	Are you sure you want to continue connecting (yes/no)? yes
	/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
	/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
	root@dcos1's password:
	
	Number of key(s) added: 1
	
	Now try logging into the machine, with:   "ssh 'Node2'"
	and check to make sure that only the key(s) you wanted were added.

所有虚机都执行完成后,可以通过ssh hostname来查看之间是否ssh连接已经免密了,如下测试

➜  ~ ssh 10.0.8.183
	Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-62-generic x86_64)
	
	 * Documentation:  https://help.ubuntu.com
	 * Management:     https://landscape.canonical.com
	 * Support:        https://ubuntu.com/advantage
	
	180 packages can be updated.
	75 updates are security updates.
	
	
	Last login: Wed Jun 28 09:16:54 2017 from 10.0.7.14
	➜  ~

7,在Ansible client(Mac)上执行


$ ansible-playbook -i ~/kargo/inventory/inventory.cfg cluster.yml -b -v –user=username –ask-sudo-pass

安装成功,在Mac上可以看到如下输出:


在Node 1上可以看到:


遇到的错误:
1,有如下错误:


kubectl The connection to the server localhost:8080 was refused

修改了vim inventory/group_vars/k8s-cluster.ym 里面的8080为28080,避免和别人冲突,又有如下错误:


kubernetes/master : Master | wait for the apiserver to be running

原因是apiserver的镜像起不了,无法提供服务
怀疑可能是不支持docker 1.17版本,所以更换为docker 1.13.1版本子再次安装后没有上述错误了,安装成功。切记,要用docker 1.13.1版本。


2,如果按照上述操作始终没有成功,可以把节点上的


/run/kubernetes
/etc/kubernetes

都删除,然后把docker也都卸载重新装。然后在Ansible client节点上重新部署就可以啦。



  • 使用Dockerfile制作包含zkui 2.0 的Zookeeper 3.4.10版本的镜像:
    1,建立你要做docker的目录,结构如下:



2,其中Dockerfile如下:

FROM openjdk:8-jre-alpine
	MAINTAINER Wang Liang <wangl8@knownsec.com>
	
	# Install required packages
	RUN apk add --no-cache 
		bash 
		su-exec
	
	ENV ZOO_USER=zookeeper 
		ZOO_CONF_DIR=/conf 
		ZOO_DATA_DIR=/data 
		ZOO_UI_DIR=/zkui 
		ZOO_DATA_LOG_DIR=/datalog 
		ZOO_PORT=2181 
		ZOO_TICK_TIME=2000 
		ZOO_INIT_LIMIT=5 
		ZOO_SYNC_LIMIT=2
	
	# Add a user and make dirs
	RUN set -x 
		&& adduser -D "$ZOO_USER" 
		&& mkdir -p "$ZOO_DATA_LOG_DIR" "$ZOO_DATA_DIR" "$ZOO_CONF_DIR" 
		&& chown "$ZOO_USER:$ZOO_USER" "$ZOO_DATA_LOG_DIR" "$ZOO_DATA_DIR" "$ZOO_CONF_DIR"
	
	ARG GPG_KEY=C823E3E5B12AF29C67F81976F5CECB3CB5E9BD2D
	ARG DISTRO_NAME=zookeeper-3.4.10
	
	# Download Apache Zookeeper, verify its PGP signature, untar and clean up
	RUN set -x 
		&& apk add --no-cache --virtual .build-deps 
			gnupg 
		&& wget -q "http://www.apache.org/dist/zookeeper/$DISTRO_NAME/$DISTRO_NAME.tar.gz" 
		&& wget -q "http://www.apache.org/dist/zookeeper/$DISTRO_NAME/$DISTRO_NAME.tar.gz.asc" 
		&& export GNUPGHOME="$(mktemp -d)" 
		&& gpg --keyserver ha.pool.sks-keyservers.net --recv-key "$GPG_KEY" 
		&& gpg --batch --verify "$DISTRO_NAME.tar.gz.asc" "$DISTRO_NAME.tar.gz" 
		&& tar -xzf "$DISTRO_NAME.tar.gz" 
		&& mv "$DISTRO_NAME/conf/"* "$ZOO_CONF_DIR" 
		&& rm -r "$GNUPGHOME" "$DISTRO_NAME.tar.gz" "$DISTRO_NAME.tar.gz.asc" 
		&& apk del .build-deps
	
	ADD zkui-master $ZOO_UI_DIR
	
	WORKDIR $DISTRO_NAME
	VOLUME ["$ZOO_DATA_DIR", "$ZOO_DATA_LOG_DIR"]
	
	EXPOSE $ZOO_PORT 2888 3888
	
	ENV PATH=$PATH:/$DISTRO_NAME/bin:$ZOO_UI_DIR 
		ZOOCFGDIR=$ZOO_CONF_DIR
	
	COPY docker-entrypoint.sh /
	ENTRYPOINT ["/docker-entrypoint.sh"]

3,docker-entrypoint.sh如下:

#!/bin/bash
	
	set -e
	
	# Allow the container to be started with `--user`
	if [ "$1" = 'zkServer.sh' -a "$(id -u)" = '0' ]; then
		chown -R "$ZOO_USER" "$ZOO_DATA_DIR" "$ZOO_DATA_LOG_DIR"
		exec su-exec "$ZOO_USER" "$0" "$@"
	fi
	
	# Generate the config only if it doesn't exist
	if [ ! -f "$ZOO_CONF_DIR/zoo.cfg" ]; then
		CONFIG="$ZOO_CONF_DIR/zoo.cfg"
	
		echo "clientPort=$ZOO_PORT" >> "$CONFIG"
		echo "dataDir=$ZOO_DATA_DIR" >> "$CONFIG"
		echo "dataLogDir=$ZOO_DATA_LOG_DIR" >> "$CONFIG"
	
		echo "tickTime=$ZOO_TICK_TIME" >> "$CONFIG"
		echo "initLimit=$ZOO_INIT_LIMIT" >> "$CONFIG"
		echo "syncLimit=$ZOO_SYNC_LIMIT" >> "$CONFIG"
	
		for server in $ZOO_SERVERS; do
			echo "$server" >> "$CONFIG"
		done
	fi
	
	if [ -f "$ZOO_UI_DIR/config.cfg" ]; then
		CONFIG="$ZOO_UI_DIR/config.cfg"
	
		echo "serverPort=$ZOO_UI_PORT" >> "$CONFIG"
		echo "zkServer=$ZOO_UI_SERVER" >> "$CONFIG"
	fi
	
	# Write myid only if it doesn't exist
	if [ ! -f "$ZOO_DATA_DIR/myid" ]; then
		echo "${ZOO_MY_ID:-1}" > "$ZOO_DATA_DIR/myid"
	fi
	
	cd $ZOO_UI_DIR
	exec nohup java -jar zkui-2.0-SNAPSHOT-jar-with-dependencies.jar &
	exec zkServer.sh start-foreground

[制作过程需要的文件]http://pan.baidu.com/s/1i5IG6sH


执行


➜  wldocker sudo docker build -t zookeeper_3.4.10_zkui_2.0:0.0.1 .

就可以制作出包含ZKUI的zookeeper 3.4.10的docker镜像了






把该镜像上传到HARBOR库,稍后k8s编排时会用到。



  • 通过Kubernetes
    编排启动Zookeeper集群:

    我们采取的方案是每个zkserver属于一个pod,每个pod绑定一个物理主机。
    只需要完成yaml文件即可:


zk-1.yaml:

---
	apiVersion: v1
	kind: Service
	metadata:
	  name: zk-1-svc
	  labels:
		app: zk-1-svc
	spec:
	  ports:
	  - port: 2888
		name: server
	  - port: 3888
		name: leader-election
	  - port: 2181
		name: client
	  - port: 9999
		name: zkui
	  selector:
		app: zk-1
	  type: NodePort
	---
	apiVersion: v1
	kind: ConfigMap
	metadata:
	  name: zk-1-cm
	data:
	  jvm.heap: "1G"
	  tick: "2000"
	  init: "10"
	  sync: "5"
	  client.cnxns: "60"
	  snap.retain: "3"
	  purge.interval: "0"
	---
	apiVersion: extensions/v1beta1
	kind: Deployment
	metadata:
	  name: zk-1
	spec:
	  replicas: 1
	  template:
		metadata:
		  labels:
			app: zk-1
		spec:
		  nodeSelector:
			zk: zk-1
		  containers:
		  - name: myzk
			imagePullPolicy: IfNotPresent
			image: registry.k8s.intra.knownsec.com/bigdata/zookeeper_3.4.10_zkui_2.0:0.0.1
			resources:
			  requests:
				memory: "2Gi"
				cpu: "500m"
			ports:
			- containerPort: 2181
			  name: client
			- containerPort: 2888
			  name: server
			- containerPort: 3888
			  name: leader-election
			- containerPort: 9999
			  name: zkui
			env:
			- name : ZK_HEAP_SIZE
			  valueFrom:
				configMapKeyRef:
					name: zk-1-cm
					key: jvm.heap
			- name : ZK_TICK_TIME
			  valueFrom:
				configMapKeyRef:
					name: zk-1-cm
					key: tick
			- name : ZK_INIT_LIMIT
			  valueFrom:
				configMapKeyRef:
					name: zk-1-cm
					key: init
			- name : ZK_SYNC_LIMIT
			  valueFrom:
				configMapKeyRef:
					name: zk-1-cm
					key: tick
			- name : ZK_MAX_CLIENT_CNXNS
			  valueFrom:
				configMapKeyRef:
					name: zk-1-cm
					key: client.cnxns
			- name: ZK_SNAP_RETAIN_COUNT
			  valueFrom:
				configMapKeyRef:
					name: zk-1-cm
					key: snap.retain
			- name: ZK_PURGE_INTERVAL
			  valueFrom:
				configMapKeyRef:
					name: zk-1-cm
					key: purge.interval
			- name: ZK_CLIENT_PORT
			  value: "2181"
			- name: ZK_SERVER_PORT
			  value: "2888"
			- name: ZK_ELECTION_PORT
			  value: "3888"
			- name: ZOO_MY_ID
			  value: "1"
			- name: ZOO_SERVERS
			  value: "server.1=0.0.0.0:2888:3888 server.2=zk-2-svc:2888:3888"
			- name: ZOO_UI_PORT
			  value: "9999"
			- name: ZOO_UI_SERVER
			  value: "zk-1-svc:2181,zk-2-svc:2181"
	#        readinessProbe:
	#          exec:
	#            command:
	#            - "zkok.sh"
	#          initialDelaySeconds: 10
	#          timeoutSeconds: 5
	#        livenessProbe:
	#          exec:
	#            command:
	#            - "zkok.sh"
	#          initialDelaySeconds: 10
	#          timeoutSeconds: 5
			volumeMounts:
			- name: data
			  mountPath: /data
			- name: datalog
			  mountPath: /datalog
		  volumes:
		  - name: data
			hostPath:
			  path: /data/k8s/zookeeper/data
		  - name: datalog
			hostPath:
			  path: /data/k8s/zookeeper/datalog

zk-2.yaml:

---
	apiVersion: v1
	kind: Service
	metadata:
	  name: zk-2-svc
	  labels:
		app: zk-2-svc
	spec:
	  ports:
	  - port: 2888
		name: server
	  - port: 3888
		name: leader-election
	  - port: 2181
		name: client
	  - port: 9999
		name: zkui
	  selector:
		app: zk-2
	  type: NodePort
	---
	apiVersion: v1
	kind: ConfigMap
	metadata:
	  name: zk-2-cm
	data:
	  jvm.heap: "1G"
	  tick: "2000"
	  init: "10"
	  sync: "5"
	  client.cnxns: "60"
	  snap.retain: "3"
	  purge.interval: "0"
	---
	apiVersion: extensions/v1beta1
	kind: Deployment
	metadata:
	  name: zk-2
	spec:
	  replicas: 1
	  template:
		metadata:
		  labels:
			app: zk-2
		spec:
		  nodeSelector:
			zk: zk-2
		  containers:
		  - name: myzk
			imagePullPolicy: IfNotPresent
			image: registry.k8s.intra.knownsec.com/bigdata/zookeeper_3.4.10_zkui_2.0:0.0.1
			resources:
			  requests:
				memory: "2Gi"
				cpu: "500m"
			ports:
			- containerPort: 2181
			  name: client
			- containerPort: 2888
			  name: server
			- containerPort: 3888
			  name: leader-election
			- containerPort: 9999
			  name: zkui
			env:
			- name : ZK_HEAP_SIZE
			  valueFrom:
				configMapKeyRef:
					name: zk-2-cm
					key: jvm.heap
			- name : ZK_TICK_TIME
			  valueFrom:
				configMapKeyRef:
					name: zk-2-cm
					key: tick
			- name : ZK_INIT_LIMIT
			  valueFrom:
				configMapKeyRef:
					name: zk-2-cm
					key: init
			- name : ZK_SYNC_LIMIT
			  valueFrom:
				configMapKeyRef:
					name: zk-2-cm
					key: tick
			- name : ZK_MAX_CLIENT_CNXNS
			  valueFrom:
				configMapKeyRef:
					name: zk-2-cm
					key: client.cnxns
			- name: ZK_SNAP_RETAIN_COUNT
			  valueFrom:
				configMapKeyRef:
					name: zk-2-cm
					key: snap.retain
			- name: ZK_PURGE_INTERVAL
			  valueFrom:
				configMapKeyRef:
					name: zk-2-cm
					key: purge.interval
			- name: ZK_CLIENT_PORT
			  value: "2181"
			- name: ZK_SERVER_PORT
			  value: "2888"
			- name: ZK_ELECTION_PORT
			  value: "3888"
			- name: ZOO_MY_ID
			  value: "2"
			- name: ZOO_SERVERS
			  value: "server.1=zk-1-svc:2888:3888 server.2=0.0.0.0:2888:3888 "
			- name: ZOO_UI_PORT
			  value: "9999"
			- name: ZOO_UI_SERVER
			  value: "zk-1-svc:2181,zk-2-svc:2181"
	#        readinessProbe:
	#          exec:
	#            command:
	#            - "zkok.sh"
	#          initialDelaySeconds: 10
	#          timeoutSeconds: 5
	#        livenessProbe:
	#          exec:
	#            command:
	#            - "zkok.sh"
	#          initialDelaySeconds: 10
	#          timeoutSeconds: 5
			volumeMounts:
			- name: data
			  mountPath: /data
			- name: datalog
			  mountPath: /datalog
		  volumes:
		  - name: data
			hostPath:
			  path: /data/k8s/zookeeper/data
		  - name: datalog
			hostPath:
			  path: /data/k8s/zookeeper/datalog

[yaml共享]http://pan.baidu.com/s/1jIyvGPK


执行:

➜  zookeeper kubectl create -f zk-1.yaml
	service "zk-1-svc" created
	configmap "zk-1-cm" created
	deployment "zk-1" created
	➜  zookeeper kubectl create -f zk-2.yaml
	service "zk-2-svc" created
	configmap "zk-2-cm" created
	deployment "zk-2" created


  • 功能验证:

➜  zookeeper kubectl get po  #看po情况
	NAME                    READY     STATUS    RESTARTS   AGE
	zk-1-1238243890-phcm8   1/1       Running   0          1m
	zk-2-4022999611-x27vq   1/1       Running   0          1m
	➜  zookeeper kubectl get po -o wide #看绑定物理节点情况
	NAME                    READY     STATUS    RESTARTS   AGE       IP               NODE
	zk-1-1238243890-phcm8   1/1       Running   0          1m        11.233.112.152   yuyan1
	zk-2-4022999611-x27vq   1/1       Running   0          1m        11.233.77.83     yuyan2
	➜  zookeeper kubectl get service  #看端口映射情况
	NAME         CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                       AGE
	kubernetes   11.233.0.1      <none>        443/TCP                                                       4d
	zk-1-svc     11.233.30.168   <nodes>       2888:31292/TCP,3888:30513/TCP,2181:31570/TCP,9999:30476/TCP   1m
	zk-2-svc     11.233.27.68    <nodes>       2888:30906/TCP,3888:30842/TCP,2181:32473/TCP,9999:31490/TCP   1m
	➜  zookeeper kubectl exec  zk-2-4022999611-x27vq -t -i -- bash #进入容器看服务情况
	bash-4.3# zkServer.sh status
	ZooKeeper JMX enabled by default
	Using config: /conf/zoo.cfg
	Mode: leader  #选举正常
<p>通过宿主机ip加映射后的端口访问zkui <a href="http://10.0.8.183:31490" rel="nofollow">http://10.0.8.183:31490</a><br /></p>
<div class="image-package"><img src="" alt="" /><br />
    <div class="image-caption">提供对外访问正常</div>
</div>
<p><strong>参考</strong><br />[Kargo部署K8s]<a
        href="http://oo3p.com/2017/04/17/kargo%E5%AE%B9%E5%99%A8%E5%8C%96%E9%83%A8%E7%BD%B2kubernetes%E9%AB%98%E5%8F%AF%E7%94%A8%E9%9B%86%E7%BE%A4/"
        rel="nofollow">http://oo3p.com/2017/04/17/kargo容器化部署kubernetes高可用集群/</a><br />[zookeeper docker]<a
        href="https://github.com/31z4/zookeeper-docker/tree/f12428ab7c6ea263ef037cf258129b83276c009c"
        rel="nofollow">https://github.com/31z4/zookeeper-docker/tree/f12428ab7c6ea263ef037cf258129b83276c009c</a><br />[k8s
    pod]<a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector"
        rel="nofollow">https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector</a><br />[k8s编排zookeeper集群]<a
        href="https://github.com/kubernetes/contrib/tree/master/statefulsets/zookeeper"
        rel="nofollow">https://github.com/kubernetes/contrib/tree/master/statefulsets/zookeeper</a></p>




作者:俺是亮哥
链接:http://www.jianshu.com/p/315e18c2f3a2
來源:简书
著作权归作者所有。商业转载请联系作者获得授权,非商业转载请注明出处。

收藏
暂无回复