[Linux] CentOS7 에서 K8S 클러스터, ArgoCD 구성하기

기존에 올려둔 가이드 내용을 바탕으로 업데이트하여 작성한 신규 가이드

기존 가이드 : https://khs960508.tistory.com/10

[Linux] CentOS7 에서 K8S 클러스터 구축

 

쿠버네티스?

Linux 컨테이너 작업을 자동화하는 오픈소스 플랫폼을 뜻함 이 플랫폼에서는 컨테이너화된 애플리케이션을 배포하고 확장하는 데 수동 프로세스가 필요하지 않음 Linux 컨테이너를 실행하는 호스트 그룹을 함께 클러스터링할 수 있으며 쿠버네티스를 통해 이러한 클러스터를 쉽고 효율적으로 관리할 수 있다 클러스터는 퍼블릭 클라우드, 프라이빗 클라우드 또는 하이브리드 클라우드 전체로 호스트를 확장할 수 있다 쿠버네티스는 Apache Kafka를 통한 실시간 데이터 스트리밍과 같이 신속한 확장을 요하는 클라우드 네이티브 애플리케이션을 호스팅하는 데 이상적인 플랫폼 이다.

 

GitOps?

쿠버네티스의 구성요소들을 관리하고 배포하기 위해서는 Manifest파일을 구성하여 실행해야하는데 이러한 파일들은 계속해서 변경되기 때문에 지속적인 관리가 필요한데 이를 편하게 Git으로 관리하는 방식

 

ArgoCD?

한마디로 쿠버네티스를 위한 CD(Continuous Delivery)툴 GitOps방식으로 관리되는 Manifest 파일의 변경사항을 감시, 현재 배포된 환경의 상태와 Git에 정의된 Manifest 상태를 동일하게 유지하는 역할을 수행

 

테스트환경의 대략적인 구성도

 

1.서버 스펙

Connect(Bootstrap)용 서버1대 : CentOS 7.8 / 2 vCPU, 4GB Mem

Master용 서버 1대 : CentOS 7.8 / 2 vCPU, 4GB Mem  // Master노드라고 생각했는데 진행하다 보니 얘도 Worker노드같다

Worker용 서버 1대 : CentOS 7.8 / 2 vCPU, 4GB Mem

 

2.클러스터 구성

모든 서버에서 진행 [Connect / Master1 / Worker1]

# ssh-keygen –t rsa

# systemctl restart sshd

 

Connect(Bootstrap) 서버에서만 진행

# vi /etc/hosts

각 노드서버들ip 적고 master1, worker1 .... worker3 까지 입력후 저장

 

# ssh-copy-id -i ~/.ssh/id_rsa.pub root@master1 (~worker3까지)

# ssh root@master1 hostname (~worker3까지)

# yum repolist

# vi /etc/yum.repos.d/docker-ce.repo

아래 내용을 docker-ce.repo에 작성

[docker-ce]

name=Docker-CE Repository

baseurl= https://download.docker.com/linux/centos/7/$basearch/stable

enabled=1

gpgcheck=1

keepcache=0

gpgkey= https://download.docker.com/linux/centos/gpg

 

# yum install nfs-utils nfs-utils-lib

 

모든 서버에서 진행 [Connect / Master1 / Worker1]

# echo 1 > /proc/sys/net/ipv4/ip_forward

# systemctl stop firewalld && systemctl disable firewalld

# sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/sysconfig/selinux && cat /etc/sysconfig/selinux

# setenforce 0

# swapoff -a

 

Connect(Bootstrap) 서버에서만 진행

# yum install epel-release

# yum -y update

 

# yum install python3 (파이썬 설치) 

 

# git clone https://github.com/kubernetes-sigs/kubespray.git

# cd kubespray/

# vi requirements.txt

(ansible과 core의 값을 아래와 같이 변경)

ansible==4.10.0

ansible-core==2.11.12

cryptography==3.4.8

jinja2==2.11.3

netaddr==0.7.19

pbr==5.4.4

jmespath==0.9.5

ruamel.yaml==0.16.10

ruamel.yaml.clib==0.2.6

MarkupSafe==1.1.1

 

# pip3 install --upgrade pip

# pip3 install -r requirements.txt

 

※ requirements.txt 내용을 변경하지 않으면 아래와 같은에러들 발생

 

 

# cp -rfp inventory/sample inventory/saascluster

# declare -a IPS=( 각 노드들의 IP주소를 입력, 각IP별 띄어쓰기로 구분 )

# CONFIG_FILE=inventory/saascluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

 

*내용이 정상적으로 출력되는지 확인*

# cat inventory/saascluster/group_vars/all/all.yml

# cat inventory/saascluster/group_vars/k8s_cluster/k8s-cluster.yml

 

# ansible-playbook -i inventory/saascluster/hosts.yaml  --become --become-user=root cluster.yml

 

Connect(BootStrap) 서버에서만 진행

# vi /etc/yum.repos.d/kubernetes.repo

아래내용을 kubernetes.repo에 작성

[kubernetes]

name=Kubernetes

baseurl= https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64

enabled=1

gpgcheck=0

repo_gpgcheck=0

gpgkey= https://packages.cloud.google.com/yum/doc/yum-key.gpg  https://packages.cloud.google.com/yum/doc/rpm-package-key.gp

 

# yum install kubectl

 

Master1 노드에서 아래 명령어 실행

# mkdir -p $HOME/.kube

# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

# chown $(id -u):$(id -g) $HOME/.kube/config

# vi .kube/config

 

Connect(BootStrap) 노드에서 아래 명령어 실행

# mkdir -p $HOME/.kube

# vi .kube/config

(아래 더보기 내용 복붙)

apiVersion: v1

clusters:

- cluster:

    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeE1EY3lPVEF3TlRnd01Gb1hEVE14TURjeU56QXdOVGd3TUZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBSlp1CnZLaFB0SUFGV1pPU1RDemlyOWh4VHNudEpWTnZZUE1PUnoyRzNPclNNQlRzWVNyQ2YzQ0haeDZNOEI3UmlHK28KQUFWZjFLczhzem91VHZDQ0J1NWtMRFdzTXhlQm1JbmpDTDZkc3E3VDllN0RjVGoyekZWdGFIM0xwejZ6K1F4VwpsTjUzR3BxZWRTWmc3cEhybFJuQklVOW1wcWNrN3RkUmdWanNnSHVrdVpMdVhpM1ljZ2ZqMFcra0JRZ2NCaTYyCkN5b2pUTllhUXBacEZSaGE1czF4NUUyWlJMY1hoZisrL3V2OWh4YUFzNjRkWGtaUVBNeW5XekVNdHlvSGhJVkEKc3YrejNEY3RqZWNXZUo3alpiTTRjaE1RZVF5dVVBSEUveW1uWVVwbVgvZTJILzY0T0pyTGhRS1NhNzJzTk42agp5KzB6REJxZE5TNkdKbnRYMzVzQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZFY01jYVZEREZzMlpVNDArbTkydHFlVjVnUWVNQTBHQ1NxR1NJYjMKRFFFQkN3VUFBNElCQVFBVDFqdU02ZWtRTDc2ak9NSFZTZzhGS0pIc1I4cmxpdld5eGNMbXNFTGZzL0t1UVdzdwo1Rkw3Z2pnSVpWZi9NbEVtTFhETXZxQ0YyOHg2eHN0cG1nQnFUczlaRFFGSjZqZURWUlp1S3VwZG1QbDNFYU1DCm5LZW9Tb1RqOWdjV29oTzZhZlZtN0xtTGlHeDlLeVBHS1BFOWlpWUlYTHNTNi8vL1B0MmduSW1TejVQTzNmMDUKdjNJc1JSOUhtSTJnQUxLZUJvdnRCYklvY2tCak50VHVOUWJWeUNMUUZLK0RuSU1Dc0hKNWtqaFhQSkZlcVFZTwpEZU1pK25qS0psRnQ0QVhLQVNKNFFZUDZleTZGc0NsQjlQZ2U2TnV3RGdHWFZ4eHBwZDVWVFZzeE5nSWVidDhsCnEzUjRab0JRdFZFa1p6MklTQ09GVWFpdzhDb1B4RjZuY2xPcQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

    server: https://127.0.0.1:6443

  name: cluster.local

contexts:

- context:

    cluster: cluster.local

    user: kubernetes-admin

  name: kubernetes-admin@cluster.local

current-context: kubernetes-admin@cluster.local

kind: Config

preferences: {}

users:

- name: kubernetes-admin

  user:

    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJVENDQWdtZ0F3SUJBZ0lJY01ubWpNQTY2Z2d3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TVRBM01qa3dNRFU0TURCYUZ3MHlNakEzTWprd01EVTRNREZhTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXlrbVkzVHFZbndzSTZmaEQKbGhNUC9NMEFzRWJ6aTZyUWpaekN0MlpMZ3pnWUZIOG1tM2ZWbENqZEVKMkJScUNFbG84QlRJeE1DUVBtSGtDcwpLdTJYZGtnUG9DUTNRY0RkQkZ1Y2o0dzYwdmxod2NOY1VxNW9JSnZRZFM4eDZJa0ljdDZGWEJBU1o4MXFiVnhyCjc2OVhuOEFwNHBXMmRVdlNndE9GcGRFSVFWd1RvckhwSi90OUdQc1ZQM3VLWkttaG1CbEFZdUlUbjR4RE1CZ00KdEJFUGRzbnpkVnE4dUVsS09Ec05idG1TRncxd01Sa1c0MFJtU2haT0xTMVZsT1NQNnhBNFova0FsZ3RpaUFUNgppTWw2UnhGdW1reHBIQ1dUYWE3STQzSTc0NG5ndU44NmhmeW5lUEZpQjVRNzNKRkxya1NHWTBJclhXQnBhd2hNCnJQL056d0lEQVFBQm8xWXdWREFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RBWURWUjBUQVFIL0JBSXdBREFmQmdOVkhTTUVHREFXZ0JSSERIR2xRd3hiTm1WT05QcHZkcmFubGVZRQpIakFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBVUJGNUFpS2kwWXIzejRoaDd1TlBxNjM4NmI2UVJzQ3VOaHVqClFjbnIrRXUwVVp0ajl3MGJkZEFtVVR3UTRRMEpxZEs1UkJrbUVuVGpjRHRDUm8wcmtjNG5DWG9wcG5GTFZMbloKL0ZlSVNsa3lTS1JNYVpDcDRtaTZweHpwTFh2WU1GL2JDZFBpWTVDSEhCVnVLUW1XUUtoSGN3djA0MjFoV0c0Qwp2dHovOEFObUlOaFZLNlN2Z2lxQ20veTZhMmFldVQxR2ZaY3RuVmZGR3BPZVJXdFhlVitmbzI0eUJBUnptRWg2CmtGc0drVlJPUHp1Y3h1Z0NFME44VlVhby9WbDhSSkpFNFZMZWNXOUJZb200dWlRWmRjNGZWdGlaZWQ2cUloVk4KRVQveDBWaTQrQ21adEVNaEZvbzJuK2RGeXFaKzkwODBQTEpaZkxhczkxb01TZEdETUE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBeWttWTNUcVlud3NJNmZoRGxoTVAvTTBBc0Viemk2clFqWnpDdDJaTGd6Z1lGSDhtCm0zZlZsQ2pkRUoyQlJxQ0VsbzhCVEl4TUNRUG1Ia0NzS3UyWGRrZ1BvQ1EzUWNEZEJGdWNqNHc2MHZsaHdjTmMKVXE1b0lKdlFkUzh4NklrSWN0NkZYQkFTWjgxcWJWeHI3NjlYbjhBcDRwVzJkVXZTZ3RPRnBkRUlRVndUb3JIcApKL3Q5R1BzVlAzdUtaS21obUJsQVl1SVRuNHhETUJnTXRCRVBkc256ZFZxOHVFbEtPRHNOYnRtU0Z3MXdNUmtXCjQwUm1TaFpPTFMxVmxPU1A2eEE0Wi9rQWxndGlpQVQ2aU1sNlJ4RnVta3hwSENXVGFhN0k0M0k3NDRuZ3VOODYKaGZ5bmVQRmlCNVE3M0pGTHJrU0dZMElyWFdCcGF3aE1yUC9OendJREFRQUJBb0lCQUF0U1d5L21tNndySGZDYQphSTM1dXlkWEFmYmJ3YjNMYmFpL1lWYUtieWVFK0RRTmVjZXh5VTVsaXhubTl0bFJHcHFpbmx2K2JqTUMrTm93Cm5ISTZvZ09JdlBjbGo4SXFYVFJsUERuOGl6NUxudUVOWWhsdWtYMWh1eVVUNHlNMHFEaGxvK2tuTFNDS0txNjgKUVg1RXZnNiszcGxENzlaWXpBdHVjMGg0anBHNTBBQzJDTStsOEZESnFmVHlyMkdVSTJmd2NtYU9OaGY5UUFsNQowT1VmOU40ZUIrV3lENjNyQnVQQ3BUMW03anhSWWJhUUplZ1lYT1E3bWtZQXJ2OWcrQXZORmMwSWExR3IrVmUwCnpCdGhJM3RWODNDbjVJQ3RGSGtReEM0MDY0VEx2ZWpwK3JYR1pOR2drb1lRK2tPUGNqNnZGditNQzI4M1c4UE8KM1NjcGQ2RUNnWUVBMGNDengyTzkwREVKWlRtTXY2ZTZlVjQwMzUvL0Z4RWUrNEs2b1Y0T3lNL29SRWJSb29NagpPMU5ac3NKOVczb1krWWs0aHJNVkVHYnZwQU12Umsyd3VpRDRrMlJsL1BIeXIwUFhLNUs5TUVyZm43eEdwVXFmCkJqOFZ5WEo1TGVSakxIaUZrNGRUTDMxdHdCMmJxZHpsSHdBM043dnBoaUFING1mNDQzMERNdVVDZ1lFQTl1T0cKaWxhZEo0MFRScGZrLzN1a1JVRXBLNEphL0Ira1JsN0oyZll4bUpWZTVENVB4cnlqRWd5eDAwV2RBWjkrZ3ZGeQowUEJnSHlEUU9kTStNd0g0YVdyUTV3ZWJYV05neTVPWDZCTCtzR2hwdWs1THdZdWZVVVltSythMDBRU0J6ZU9zClVEQ09mTG9YVGJNQ1VYaitIL2Z3YjJsdjQwUm95bG15OENvNGJxTUNnWUJuS01kQ2dTT1JpUXg5NUZmOHlwbEQKcDdtSllqOHJNUmh6eWsvUG5WcllHS2k3SzdsZW0vQnY5UVpnakpicXJwb3AzL3NWQkc2UHFwQ2hzWTlHbkY4VwpxN0NWalFNNzdDTFhZQjRySi9nMEVCNzdpTFdXQmFWQWhUWko1U3NFRHhkWGY4N3lSOVA1djA2Qzlnemc5ZXNaCjNCbE1KL2NxS0FhNHF6b1ZhQjV5b1FLQmdRQ1FoZ2hRR3Z2TUYwMWRXYlYzeVEwVWJWeUlWN3FqTkhLWnJ3dEMKU0Znb0ZlaUNlK2QvVXhXeWp6UFpsVHFmcEpvT1ZRRDlSbmsrUDVzay9uKytwRHlpbVNESTNTOFpyLzhOck15VAowZXE5b2FHaXFncTJ0ZUVCVnFzcHI5ZlJNMjQ0OURuK2h5NGgzVjJlNGxkTkVpTFEyL2NuZ1BVNmVPVk83c3RNCnc2cGRPUUtCZ1FDVFVqTmFrLzJqQnJMeDlOVGI4WTc4WElJTHJLblFRYlBIMmVyK0pFWGF1LytSUXN3NnBHUDUKSnlzc08rdVdPZzhCL01FTmFkL2cwZVVTdDhlYjlPcExNZENiT3ZzczFNbkwzQ0J2S0RjMktCRjVueTUyWWNubwovb3lMYVlaYzRPZ3RDdVpvOUdKbFJGc2JvVDg2WWcwTXlSSHY3YkhFQlIwUFMrc2VUWFVTVEE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

 

Master1 노드에서 아래 명령어 실행

# kubectl proxy --port=8080 &

# curl http://localhost:8080/api

 

Connect(BootStrap)서버의 config내용중 ip Address 부분을 Master1 노드에서 curl을 쳐서 나온 아이피주소로 수정

Master1 노드에서 kubectl version , kubectl cluster-info 정상작동 확인

 

3.Helm 차트& 도커 설치

Master1 노드에서 진행

# curl  https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 > get_helm.sh

# chmod 700 get_helm.sh

# ./get_helm.sh

# helm version

 

# helm repo add stable  https://charts.helm.sh/stable

# helm repo add bitnami  https://charts.bitnami.com/bitnami

# helm repo add incubator  https://charts.helm.sh/incubator

# helm repo add ibm-helm  https://raw.githubusercontent.com/IBM/charts/master/repo/ibm-helm

# helm repo add gitlab  https://charts.gitlab.io/

# helm plugin install  https://github.com/chartmuseum/helm-push.git

 

# helm install nfs-server stable/nfs-server-provisioner

# kubectl get storageclass

 

# yum install -y docker

# systemctl start docker

# systemctl enable docker

 

4.ArgoCD 설치

Master1 노드에서 진행 (2022.11.03 수정사항 - Worker 노드에서 진행 후, 워커노드IP:7000 으로 접속 가능)

# kubectl create namespace argocd

# kubectl apply -n argocd -f  https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

 

*삭제명령어*

# kubectl delete -n argocd -f  https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

 

argocd cli 설치하기

#VERSION=$(curl --silent " https://api.github.com/repos/argoproj/argo-cd/releases/latest" | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/')

# curl -sSL -o /usr/local/bin/argocd  https://github.com/argoproj/argo-cd/releases/download/$VERSION/argocd-linux-amd64

# chmod +x /usr/local/bin/argocd

 

# kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2

# kubectl -n argocd get secret argocd-secret

 

초기비밀번호 확인

# kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

 

포트포워딩

# kubectl port-forward --address 0.0.0.0 svc/argocd-server -n argocd 7000:80 &

 

ArgoCD 접속

웹브라우저에서 http://마스터노드IP:7000