Cocktail Private Hub 서버 설치

1.필요 소프트웨어 다운로드 및 설치

• Docker, docker-compose 설치 및 harbor 압축파일 다운로드 및 압축 해제

Cocktail Private Hub를 설치하기 위한 설치하고자 하는 장비상에서 아래 init.sh파일을 upload하고 실행하면 docker, docker-compose설치 및 harbor 압축파일 다운로드& 압축해제를 수행한다.

작업 디렉토리 생성 및 설치 script upload

// root 사용자 변경
# sudo su - root

// 방화벽 중지
# systemctl stop firewalld
# systemctl disable firewalld
# vi /etc/sysconfig/selinux
SELINUX=disabled

// 작업 디렉토리 생성
# mkdir -p /root/cocktail/cert/harbor
# mkdir -p /root/cocktail/cert/docker

//docker, docker-compose, harbor 설치 및 다운로드 
# vi /root/init.sh 
yum install -y yum-utils
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum makecache fast
yum install -y --setopt=obsoletes=0 \
  docker-ce-17.03.1.ce-1.el7.centos \
  docker-ce-selinux-17.03.1.ce-1.el7.centos
systemctl enable docker
systemctl start docker

curl -L https://github.com/docker/compose/releases/download/1.12.0/docker-compose-`uname -s`-`uname -m` > docker-compose
chmod +x docker-compose
mv docker-compose /usr/local/bin/
docker-compose --version

yum install -y wget
wget https://github.com/vmware/harbor/releases/download/v1.1.1-rc4/harbor-online-installer-v1.1.1-rc4.tgz
tar -zxvf harbor-online-installer-v1.1.1-rc4.tgz

# chmod 755 /root/init.sh

// openssl.cnf 작성
# vi /root/cert/openssl.cnf
[ req ]
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_ca ]
basicConstraints = critical, CA:TRUE
keyUsage = critical, digitalSignature, keyEncipherment, keyCertSign

[ v3_req_server ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names_harbor

[ v3_builder_server ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names_builder

[ v3_builder_client ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth

[ alt_names_harbor ]
DNS.1 = localhost
IP.1 = 172.30.1.56 // registry server IP 입력
IP.2 = 127.0.0.1

[ alt_names_builder ]
DNS.1 = localhost
IP.1 = 172.30.1.56 // registry server IP 입력
IP.2 = 127.0.0.1

# chmod 755 /root/cert/openssl.cnf

// harbor 인증서 만들기
# vi /root/cert/harbor/make_harbor_cert.sh
#!/bin/bash

# Harbor Template 의 root ca 파일 경로
#!/bin/bash

# Harbor Template 의 root ca 파일 경로
HARBOR_CA_CERT_DIR=/root/cocktail/harbor/common/templates/registry
TARGET_DIR=/data/cert

openssl genrsa -out ca.key 4096
openssl req -x509 -new -sha256 -nodes -key ca.key -days 10000 -subj "/CN=harbor-ca" -out ca.crt -extensions v3_ca -config ../openssl.cnf

openssl genrsa -out harbor.key 4096
openssl req -new -key harbor.key -out harbor.csr -subj "/CN=harbor" -config ../openssl.cnf
openssl x509 -req -in harbor.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out harbor.crt -days 10000 -extensions v3_req_server -extfile ../openssl.cnf

mkdir -p $TARGET_DIR

# Harbor Template 의 root ca 파일 교체
cp ca.crt $HARBOR_CA_CERT_DIR/root.crt
cp harbor.crt $TARGET_DIR
cp harbor.key $TARGET_DIR

# chmod 755 /root/cert/harbor/make_harbor_cert.sh

// docker 인증서 만들기
# vi /root/cert/docker/make_docker_cert.sh
#!/bin/bash

HARBOR_CA=/root/cocktail/cert/harbor/ca.crt
HARBOR_CA_KEY=/root/cocktail/cert/harbor/ca.key

set -e
if [ "$#" -gt 0 ]; then
  DOCKER_HOST="$1"
else
  echo " => ERROR: You must specify the docker FQDN as the first arguement to this scripts! <="
  exit 1
fi

if [ "$USER" == "root" ]; then
  echo " => WARNING: You're running this script as root, therefore root will be configured to talk to docker"
  echo " => If you want to have other users query docker too, you'll need to symlink /root/.docker to /theuser/.docker"
fi

echo " => Using Hostname: $DOCKER_HOST  You MUST connect to docker using this host!"

echo " => Ensuring config directory exists..."
mkdir -p "$HOME/.docker"

echo " => Verifying ca.srl"
if [ ! -f "ca.src" ]; then
  echo " => Creating ca.srl"
  echo 01 > ca.srl
fi

echo " => Generating server key"
openssl genrsa -out server.key 2048

echo " => Generating server CSR"
openssl req -new -key server.key -out server.csr -subj "/CN=$DOCKER_HOST" -config ../openssl.cnf

echo " => Signing server CSR with Harbor CA"
openssl x509 -req -in server.csr -CA $HARBOR_CA -CAkey $HARBOR_CA_KEY -out server.crt -days 3650 -extensions v3_builder_server -extfile ../openssl.cnf


echo " => Generating client key"
openssl genrsa -out client.key 2048

echo " => Generating client CSR"
openssl req -new -key client.key -out client.csr -subj "/CN=docker.client" -config ../openssl.cnf

echo " => Creating extended key usage"

echo " => Signing client CSR with Harbor CA"
openssl x509 -req -in client.csr -CA $HARBOR_CA -CAkey $HARBOR_CA_KEY -out client.crt -days 3650 -extensions v3_builder_client -extfile ../openssl.cnf

if [ -d "/etc/profile.d" ]; then
  echo " => Creating profile.d/docker"
  sudo sh -c "echo '#!/bin/bash
export DOCKER_CERT_PATH=$HOME/.docker
export DOCKER_HOST=tcp://$DOCKER_HOST:2376
export DOCKER_TLS_VERIFY=1' > /etc/profile.d/docker.sh"
  sudo chmod +x /etc/profile.d/docker.sh
  source /etc/profile.d/docker.sh
else
  echo " => WARNING: No /etc/profile.d directory on your system."
  echo " =>   You will need to set the following environment variables before running the docker client:"
  echo " =>   DOCKER_HOST=tcp://$DOCKER_HOST:2376"
  echo " =>   DOCKER_TLS_VERIFY=1"
fi

OPTIONS="--tlsverify --tlscacert=$HOME/.docker/ca.crt --tlscert=$HOME/.docker/server.crt --tlskey=$HOME/.docker/server.key -H=0.0.0.0:2376"
if [ -f "/etc/sysconfig/docker" ]; then
  echo " => Configuring /etc/sysconfig/docker"
  BACKUP="/etc/sysconfig/docker.$(date +"%s")"
  sudo mv /etc/sysconfig/docker $BACKUP
  sudo sh -c "echo '# The following line was added by ./create-certs docker TLS configuration script
OPTIONS=\"$OPTIONS\"
# A backup of the old file is at $BACKUP.' >> /etc/sysconfig/docker"
  echo " => Backup file location: $BACKUP"
else
  echo " => WARNING: No /etc/sysconfig/docker file found on your system."
  echo " =>   You will need to configure your docker daemon with the following options:"
  echo " =>   $OPTIONS"
fi

# Copy docker server certificate
cp $HARBOR_CA /root/.docker/ca.crt
cp ./server.crt /root/.docker/server.crt
cp ./server.key /root/.docker/server.key
# Copy docker client certificate
cp $HARBOR_CA /root/.docker/ca.pem
cp ./client.crt /root/.docker/cert.pem
cp ./client.key /root/.docker/key.pem

export DOCKER_HOST=tcp://DOCKER_HOST:2376
export DOCKER_TLS_VERIFY=1
echo " => Done! You just need to restart docker for the changes to take effect"

# chmod 755 vi /root/cert/docker/make_docker_cert.sh

init.sh 를 실행하면 docker, docker-compose설치 및 harbor 압축파일 다운로드& 압축해제를 수행한다.

# cd ~/root/
# ./init.sh

2.Harbor 설정

• Harbor Self Signing Certificate 생성하기

Harbor를 설치하는 서버에서 작업한다.

make_harbor_cert.sh 실행하면 harbor.crt, harbor.key 파일이 생성된다.

# cd /root/cert/harbor
# ./make_harbor_cert.sh "도메인 또는 IP"

Generating a 4096 bit RSA private key
.......................................................................................++
...................................................++
writing new private key to 'ca.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:JP
State or Province Name (full name) []:
Locality Name (eg, city) [Default City]:
Organization Name (eg, company) [Default Company Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:192.168.50.105
Email Address []:
Generating a 4096 bit RSA private key
.....................................................................................................................................................................................................++
...............................................................................................................................................................................................................................++
writing new private key to 'harbor.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:JP
State or Province Name (full name) []:
Locality Name (eg, city) [Default City]:
Organization Name (eg, company) [Default Company Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:192.168.50.105
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
Signature ok
subject=/C=JP/L=Default City/O=Default Company Ltd/CN=192.168.50.105
Getting CA Private Key

# ls -al   // 인증서가 정상적으로 생성되었는지 확인한다.

• Harbor.cfg 파일 수정하기

# vi ~/cocktail/harbor/harbor.cfg
...
hostname = 서버 ip
ui_url_protocol = https
ssl_cert = /root/cert/harbor/harbor.crt     // harbor crt 파일 경로
ssl_cert_key = /root/cert/harbor/harbor.key // harbor 인증서 key 파일 경로
harbor_admin_password = C0ckt@1lAdmin
...

• Harbor 기동

./prepare 스크립트를 실행해서 harbor 설정 파일들을 generation한 후 docker-compse를 기동한다

# cd /root/harbor
# ./install.sh

// Harbor가 정상기동 했는지 확인. (총 7개의 container가 running 상태이면 정상임)
# docker ps

• Harber 설치 장비에서 Docker client로 Harbor 로그인이 정상적으로 되는지 확인하기.

Harbor 서버에서 생성한 ca.crt파일을 docker client가 인증서를 확인하는 디렉토리인 /etc/docker/certs.d/serverip/ 밑에 복사한다.

폴더 생성은 반드시 아래와 같은 형식을 따른다.

# mkdir -p /etc/docker/certs.d/xxx.xxx.xxx.xxx
# cp /root/cocktail/cert/harbor/ca.crt /etc/docker/certs.d/xxx.xxx.xxx.xxx
# docker login https://xxx.xxx.xxx.xxx

< 참고 - Harbor 실행/중단/설정변경시 >

Harbor는 부팅시에 자동 실행되며 수동으로 기동, 종료 및 설정 변경시에 아래 명령을 실행하면 된다.

# cd ~/cocktail/harbor
# docker-compose start
# docker-compose stop

설정을 변경한 경우에는 install.sh 파일을 다시 실행한다.

3. Cocktail Buid Server 설치

Cocktail build Server는 docker상에 container로 실행되며 Cocktail Builder-api component와 통신하게 된다.

일반적으로 Docker Server와 Docker Client간 로컬이 아닌 원격지에서는 반드시 TLS로 통신해야 함으로 사설인증서를 생성하여 설정해 준다.

이를 편하게 설정하기 위해 "make_docker_cert.sh" 파일을 설치하고자 하는 VM 또는 machine에 upload 또는 작성하여 실행한다.

# cd /root/cert/doker
# ./make_docker_cert.sh "도메인 또는 IP"

# cd /root/.docker
# ls -al                   // 실제 인증서가 생성되었는지 확인해 본다.
drwx------. 2 root root  211 12월 19 15:18 .
dr-xr-x---. 6 root root  204 12월 19 15:15 ..
-rw-r--r--. 1 root root 1980 12월 19 15:18 ca.crt
-rw-r--r--. 1 root root    3 12월 19 15:18 ca.srl
-rw-r--r--. 1 root root 1464 12월 19 15:18 cert.pem
-rw-r--r--. 1 root root  895 12월 19 15:18 client.csr
-rw-------. 1 root root  159 12월 19 15:15 config.json
-rw-r--r--. 1 root root   30 12월 19 15:18 extfile.cnf
-rw-r--r--. 1 root root 1679 12월 19 15:18 key.pem
-rw-r--r--. 1 root root 1428 12월 19 15:18 server-cert.pem
-rw-r--r--. 1 root root 1679 12월 19 15:18 server-key.pem
-rw-r--r--. 1 root root  899 12월 19 15:18 server.csr

// 시스템 전역 환경설정 파일 수정 및 실행.
# cd /etc/profile.d
# vi docker.sh
  export DOCKER_CERT_PATH 의 값을 /root/.docker 로 변경
# ./docker.sh

// docker service 파일 수정.
# vi /lib/systemd/system/docker.service

#ExecStart=/usr/bin/dockerd  // 주석으로 처리    
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --tlsverify --tlscacert=/root/.docker/ca.crt --tlscert=/root/.docker/server.crt --tlskey=/root/.docker/server.key

// Docker service restart
# systemctl daemon-reload
# systemctl restart docker
*
// Docker daemon이 정상적으로 실행됬는지 확인
# docker ps

4. k8s cluster를 구성하는 모든 node에서 harbor에 저장된 image를 pull할 수 있도록 harbor의 CA파일을 전 node에 복사.

k8s cluster를 구성하는 모든 node에서 harbor로 부터 image를 pull하기 위해 harbor 설치시에 생성한 ca.crt파일을 모든 node의 /etc/docker/certs.d/serverip/ 밑에 복사한다.

mkdir -p /etc/docker/certs.d/xxx.xxx.xxx.xxx  (중요. xxx.xxx.xxx.xxx는 harbor를 설치한 ip임)
이후 ca.crt를 복사한다.

5. Cocktail builder-api, client-api의 환경 변수 수정.

k8s dashboard에서 builder-api의 환경변수 REGISTRYURL, DOCKER_URL, SERVER_TYPE, CA_PEM, CERT_PEM, KEY_PEM 를 수정하고, cocktail-api의 환경변수 REGISTRY_URL을 수정한다.

• builder-api

• cocktail-api