v2ray服务器搭建(谷歌云搭建v2ray配置)

zzxiexin 3 0

本文参照红帽官方文档,在裸机安装Openshift4.3文档进行。因为只有一台64G内存的PC机,安装vmware vsphere 6.7免费版进行本测试,所以尝试在OCP官方文档要求的最低内存需求基础上,内存减半安装,记录如下。

发现头条号不支持markdown,对程序员太不又好了,拷贝过来的代码格式都没了,简书的这一点就要好一些。

https://www.jianshu.com/p/7c0c2affadb8

1、ocp安装的过程

红帽官方文档记载的安装过程如下:

bootstrap启动并从准备好master需要的资源master从bootstrap获取需要的资源并完成启动master通过bootstrap构建etcd集群bootstrap使用刚才构建的etcd集群启动一个临时的kubernetes control plane临时control plane在master节点启动生产control plane临时control plane关闭并将控制权移交给生产control planebootstrap将ocp组建注入到生产control plane安装程序关闭bootstrapcontrol plane 部署计算节点control plane 通过operator方式安装其他服务2、准备服务器资源

服务器规划如下:

3台control plane节点,安装etcd、control plane组件和infras基础组件,因为资源紧张,不部署dns服务器,通过hosts文件解析域名;2台compute 节点,运行实际负载;1台bootstrap节点,执行安装任务;1台misc/lb节点,用于准备安装资源、启动bootstrap,并作为lb节点使用。

Hostname vcpu ram hdd ip fqdn misc/lb 4 8g 120g 192.168.128.30 misc.ocptest.ipingcloud.com/lb.ocptest.ipincloud.com bootstrap 4 8g 120g 192.168.128.31 bootstrap.ocptest.ipincloud.com master1 4 8g 120g 192.168.128.32 master1.ocptest.ipincloud.com master2 4 8g 120g 192.168.128.33 master2.ocptest.ipincloud.com master3 4 8g 120g 192.168.128.34 master3.ocptest.ipincloud.com worker1 2 4g 120g 192.168.128.35 worker1.ocptest.ipincloud.com worker2 2 4g 120g 192.168.128.36 worker2.ocptest.ipincloud.com

3、准备网络资源

api server和ingress公用一个lb,即misc/lb 以为dns配置记录,ocptest是cluster名,ipingcloud.com是基础域名.这些配置,需要修改ansi-playbook文件的tasks/相应模板。 参见 https://github.com/scwang18/ocp4-upi-helpernode.git

dns配置

组件 dns记录 描述 Kubernetes API api.ocptest.ipincloud.com 该DNS记录指向control plane节点的负载平衡器。群集外部和群集中所有节点都必须可以解析此记录。 Kubernetes API api-int.ocptest.ipincloud.com 该DNS记录指向control plane节点的负载平衡器。该记录必须可从群集中的所有节点上解析。 Routes *.apps.ocptest.ipincloud.com 通配符DNS记录指向ingress slb。群集外部和群集中所有节点都必须可以解析此记录。 etcd etcd-.ocptest.ipincloud.com DNS记录指向etcd节点,群集所有节点都必须可以解析此记录。 etcd _etcd-server-ssl._tcp.ocptest.ipincloud.com 因为etcd使用2380对外服务,因此,需要建立对应每台etcd节点的srv dns记录,优先级0,权重10和端口2380,如下表

etcd srv dns记录表

#一下激怒是必须的,用于bootstrap创建etcd服务器上,自动配置etcd服务解析

#_service._proto.name. TTL class SRV priority weight port target. _etcd-server-ssl._tcp.<cluster_name>.<base_domain> 86400 IN SRV 0 10 2380 etcd-0.<cluster_name>.<base_domain>. _etcd-server-ssl._tcp.<cluster_name>.<base_domain> 86400 IN SRV 0 10 2380 etcd-1.<cluster_name>.<base_domain>. _etcd-server-ssl._tcp.<cluster_name>.<base_domain> 86400 IN SRV 0 10 2380 etcd-2.<cluster_name>.<base_domain>.

创建ssh私钥并加入ssh agent

通过免登陆ssh私钥,可以用core用户身份登录到master节点,在集群上进行安装调试和灾难恢复。

(1)在misc节点上执行一下命令创建sshkey

ssh-keygen -t rsa -b 4096 -N ''

以上命令在~/.ssh/文件夹下创建id_rsa和id_rsa.pub两个文件。

(2)启动ssh agent进程并把将无密码登录的私钥加入ssh agent

eval "$(ssh-agent -s)"ssh-add ~/.ssh/id_rsa

下一步安装ocp时,需要将ssh公钥提供给安装程序配置文件。

因为我们采用自己手动准备资源方式,因此,需要将ssh公钥放到集群各节点,本机就可以免密码登录集群节点

#将刚才生成的 ~/.ssh目录中的 id_rsa.pub 这个文件拷贝到你要登录的集群节点 的~/.ssh目录中scp ~/.ssh/id_rsa.pub root@192.168.128.31:~/.ssh/#然后在集群节点上运行以下命令来将公钥导入到~/.ssh/authorized_keys这个文件中cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys4、获取安装程序

需要注册红帽官网账号,下载测试版安装程序,下载链接具体过程略。 https://cloud.redhat.com/openshift/install/metal/user-provisioned

下载安装程序rm -rf /data/pkgmkdir -p /data/pkgcd /data/pkg#ocp安装程序#wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-install-linux-4.3.0.tar.gz#ocp 客户端#wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-client-linux-4.3.0.tar.gz#rhcos安装程序wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest/latest/rhcos-4.3.0-x86_64-installer.iso#rhcos bios raw文件wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest/latest/rhcos-4.3.0-x86_64-metal.raw.gz#如果采用iso文件方式安装,相面两个文件都不需要下载#rhcos安装程序内核文件,用于使用ipex方式安装wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest/latest/rhcos-4.3.0-x86_64-installer-kernel#rhcos初始化镜像文件,用于使用ipex方式安装wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest/latest/rhcos-4.3.0-x86_64-installer-initramfs.img5、准备工具机misc

参照王征的脚本修改的工具机准备工具,可以方便的在工具机上启动 LB、DHCP、PXE、DNS和HTTP服务 (1)安装ansible和git

yum -y install ansible git

(2)从github拉取playbook

cd /data/pkggit clone https://github.com/scwang18/ocp4-upi-helpernode.git

(3)修改playbook的参数文件 根据自己的网络规划修改参数文件

[root@centos75 pkg]# cd /data/pkg/ocp4-upi-helpernode/[root@centos75 ocp4-upi-helpernode]# cat vars-static.yaml[root@misc pkg]# cat vars-static.yaml---staticips: truenamed: truehelper: name: "helper" ipaddr: "192.168.128.30" networkifacename: "ens192"dns: domain: "ipincloud.com" clusterid: "ocptest" forwarder1: "192.168.128.30" forwarder2: "192.168.128.30" registry: name: "registry" ipaddr: "192.168.128.30" yum: name: "yum" ipaddr: "192.168.128.30"bootstrap: name: "bootstrap" ipaddr: "192.168.128.31"masters: - name: "master1" ipaddr: "192.168.128.32" - name: "master2" ipaddr: "192.168.128.33" - name: "master3" ipaddr: "192.168.128.34"workers: - name: "worker1" ipaddr: "192.168.128.35" - name: "worker2" ipaddr: "192.168.128.36"force_ocp_download: falseocp_bios: "file:///data/pkg/rhcos-4.3.0-x86_64-metal.raw.gz"ocp_initramfs: "file:///data/pkg/rhcos-4.3.0-x86_64-installer-initramfs.img"ocp_install_kernel: "file:///data/pkg/rhcos-4.3.0-x86_64-installer-kernel"ocp_client: "file:///data/pkg/openshift-client-linux-4.3.0.tar.gz"ocp_installer: "file:///data/pkg/openshift-install-linux-4.3.0.tar.gz"ocp_filetranspiler: "file:///data/pkg/filetranspiler-master.zip"registry_server: "registry.ipincloud.com:8443"[root@misc pkg]#

(4)执行ansible安装

ansible-playbook -e @vars-static.yaml tasks/main.yml6、准备docker env# 在可以科学上网的机器上打包必要的镜像文件#rm -rf /data/ocp4mkdir -p /data/ocp4cd /data/ocp4# 这个脚本不好用,不下载,使用下面自己修改过# wget https://raw.githubusercontent.com/wangzheng422/docker_env/dev/redhat/ocp4/4.3/scripts/build.dist.shyum -y install podman docker-distribution pigz skopeo docker buildah jq python3-pip pip3 install yq# https://blog.csdn.net/ffzhihua/article/details/85237411wget http://mirror.centos.org/centos/7/os/x86_64/Packages/python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpmrpm2cpio python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm | cpio -iv --to-stdout ./etc/rhsm/ca/redhat-uep.pem | tee /etc/rhsm/ca/redhat-uep.pemsystemctl start dockerdocker login -u wuliangye2019 -p Red@123! registry.redhat.iodocker login -u wuliangye2019 -p Red@123! registry.access.redhat.comdocker login -u wuliangye2019 -p Red@123! registry.connect.redhat.compodman login -u wuliangye2019 -p Red@123! registry.redhat.iopodman login -u wuliangye2019 -p Red@123! registry.access.redhat.compodman login -u wuliangye2019 -p Red@123! registry.connect.redhat.com# to download the pull-secret.json, open following link# https://cloud.redhat.com/openshift/install/metal/user-provisionedcat << 'EOF' > /data/pull-secret.json{"auths":{"cloud.openshift.com":{"auth":"xxxxxxxxxxx}}}EOF

创建 build.dist.sh文件

#!/usr/bin/env bashset -eset -xvar_date=$(date '+%Y-%m-%d')echo $var_date#以下不用每次都执行#cat << EOF >> /etc/hosts#127.0.0.1 registry.ipincloud.com#EOF#mkdir -p /etc/crts/#cd /etc/crts#openssl req \# -newkey rsa:2048 -nodes -keyout ipincloud.com.key \# -x509 -days 3650 -out ipincloud.com.crt -subj \# "/C=CN/ST=GD/L=SZ/O=Global Security/OU=IT Department/CN=*.ipincloud.com"#cp /etc/crts/ipincloud.com.crt /etc/pki/ca-trust/source/anchors/#update-ca-trust extractsystemctl stop docker-distributionrm -rf /data/registrymkdir -p /data/registrycat << EOF > /etc/docker-distribution/registry/config.ymlversion: 0.1log: fields: service: registrystorage: cache: layerinfo: inmemory filesystem: rootdirectory: /data/registry delete: enabled: truehttp: addr: :8443 tls: certificate: /etc/crts/ipincloud.com.crt key: /etc/crts/ipincloud.com.keyEOFsystemctl restart dockersystemctl enable docker-distributionsystemctl restart docker-distributionbuild_number_list=$(cat << EOF4.3.0EOF)mkdir -p /data/ocp4cd /data/ocp4install_build() { BUILDNUMBER=$1 echo ${BUILDNUMBER} mkdir -p /data/ocp4/${BUILDNUMBER} cd /data/ocp4/${BUILDNUMBER} #下载并安装openshift客户端和安装程序 第一次需要运行,工具机ansi初始化时,已经完成这些动作了 #wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/${BUILDNUMBER}/release.txt #wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/${BUILDNUMBER}/openshift-client-linux-${BUILDNUMBER}.tar.gz #wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/${BUILDNUMBER}/openshift-install-linux-${BUILDNUMBER}.tar.gz #解压安装程序和客户端到用户执行目录 第一次需要运行 #tar -xzf openshift-client-linux-${BUILDNUMBER}.tar.gz -C /usr/local/bin/ #tar -xzf openshift-install-linux-${BUILDNUMBER}.tar.gz -C /usr/local/bin/ export OCP_RELEASE=${BUILDNUMBER} export LOCAL_REG='registry.ipincloud.com:8443' export LOCAL_REPO='ocp4/openshift4' export UPSTREAM_REPO='openshift-release-dev' export LOCAL_SECRET_JSON="/data/pull-secret.json" export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=${LOCAL_REG}/${LOCAL_REPO}:${OCP_RELEASE} export RELEASE_NAME="ocp-release" oc adm release mirror -a ${LOCAL_SECRET_JSON} \ --from=quay.io/${UPSTREAM_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-x86_64 \ --to-release-image=${LOCAL_REG}/${LOCAL_REPO}:${OCP_RELEASE} \ --to=${LOCAL_REG}/${LOCAL_REPO}}while read -r line; do install_build $linedone <<< "$build_number_list"cd /data/ocp4#wget -O ocp4-upi-helpernode-master.zip https://github.com/wangzheng422/ocp4-upi-helpernode/archive/master.zip#以下注释,因为quay.io/wangzheng422这个仓库的registry版本是v1不能与v2共存#podman pull quay.io/wangzheng422/filetranspiler#podman save quay.io/wangzheng422/filetranspiler | pigz -c > filetranspiler.tgz#podman pull docker.io/library/registry:2#podman save docker.io/library/registry:2 | pigz -c > registry.tgzsystemctl start dockerdocker login -u wuliangye2019 -p Red@123! registry.redhat.iodocker login -u wuliangye2019 -p Red@123! registry.access.redhat.comdocker login -u wuliangye2019 -p Red@123! registry.connect.redhat.compodman login -u wuliangye2019 -p Red@123! registry.redhat.iopodman login -u wuliangye2019 -p Red@123! registry.access.redhat.compodman login -u wuliangye2019 -p Red@123! registry.connect.redhat.com# 以下命令要运行 2-3个小时,耐心等待。。。# build operator catalogpodman login registry.ipincloud.com:8443 -u root -p Scwang18oc adm catalog build \ --appregistry-endpoint https://quay.io/cnr \ --appregistry-org redhat-operators \ --to=${LOCAL_REG}/ocp4-operator/redhat-operators:v1 oc adm catalog mirror \ ${LOCAL_REG}/ocp4-operator/redhat-operators:v1 \ ${LOCAL_REG}/operator#cd /data#tar cf - registry/ | pigz -c > registry.tgz#cd /data#tar cf - ocp4/ | pigz -c > ocp4.tgz

执行build.dist.sh脚本

这里有个巨坑,因为从quay.io拉取image镜像到本地时,拉取的文件有5G多,通常一次拉取不完,会出错,每次出错后,重新运行build.dist.sh会把以前的registry删除掉,从头再来,浪费很多时间,实际上可以不用删除,执行oc adm release mirror时会自动跳过已经存在的image。血泪教训。

bash build.dist.sh

oc adm release mirror执行完毕后,回根据官方镜像仓库生成本地镜像仓库,返回的信息需要记录下来,特别是imageContentSource信息,后面 install-config.yaml 文件里配置进去

SuccessUpdate image: registry.ipincloud.com:8443/ocp4/openshift4:4.3.0Mirror prefix: registry.ipincloud.com:8443/ocp4/openshift4To use the new mirrored repository to install, add the following section to the install-config.yaml:imageContentSources:- mirrors: - registry.ipincloud.com:8443/ocp4/openshift4 source: quay.io/openshift-release-dev/ocp-release- mirrors: - registry.ipincloud.com:8443/ocp4/openshift4 source: quay.io/openshift-release-dev/ocp-v4.0-art-devTo use the new mirrored repository for upgrades, use the following to create an ImageContentSourcePolicy:apiVersion: operator.openshift.io/v1alpha1kind: ImageContentSourcePolicymetadata: name: examplespec: repositoryDigestMirrors: - mirrors: - registry.ipincloud.com:8443/ocp4/openshift4 source: quay.io/openshift-release-dev/ocp-release - mirrors: - registry.ipincloud.com:8443/ocp4/openshift4 source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

以下命令不需要执行,在build.dish.sh里已经执行了

oc adm release mirror -a /data/pull-secret.json --from=quay.io/openshift-release-dev/ocp-release:4.3.0-x86_64 --to-release-image=registry.ipincloud.com:8443/ocp4/openshift4:4.3.0 --to=registry.ipincloud.com:8443/ocp4/openshift4 podman login registry.ipincloud.com:8443 -u root -p Scwang18oc adm catalog build \ --appregistry-endpoint https://quay.io/cnr \ --appregistry-org redhat-operators \ --to=registry.ipincloud.com:8443/ocp4-operator/redhat-operators:v1 oc adm catalog mirror \ registry.ipincloud.com:8443/ocp4-operator/redhat-operators:v1 \ registry.ipincloud.com:8443/operator#如果oc adm catalog mirror执行不成功,会生成一个mapping.txt的文件,可以根据这个文件,执行不成功的行删除,再以下面的方式执行oc image mirror -a /data/pull-secret.json -f /data/mapping-ok.txtoc image mirror quay.io/external_storage/nfs-client-provisioner:latest registry.ipincloud.com:8443/ocp4/openshift4/nfs-client-provisioner:latestoc image mirror quay.io/external_storage/nfs-client-provisioner:latest registry.ipincloud.com:8443/quay.io/external_storage/nfs-client-provisioner:latest#查看镜像的shacurl -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET https://registry.ipincloud.com:8443/v2/ocp4/openshift4/nfs-client-provisioner/manifests/latest 2>&1 | grep Docker-Content-Digest | awk '{print ($3)}'#删除镜像摘要curl -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X DELETE https://registry.ipincloud.com:8443/v2/ocp4/openshift4/nfs-client-provisioner/manifests/sha256:022ea0b0d69834b652a4c53655d78642ae23f0324309097be874fb58d09d2919#回收镜像空间podman exec -it mirror-registry /bin/registry garbage-collect /etc/docker/registry/config.yml7、创建installer配置文件

(1)创建installer文件夹

rm -rf /data/installmkdir -p /data/installcd /data/install

(2)定制install-config.yaml文件

补充pullSecret[root@misc data]# cat /data/pull-secret.json{"auths":{"cloud.openshift.com":{"auth":"省略"}}}添加sshKey(3.1创建的公钥文件内容)cat ~/.ssh/id_rsa.pubadditionalTrustBundle(Mirror registry创建是生成的csr)[root@misc crts]# cat /etc/crts/ipincloud.com.crt-----BEGIN CERTIFICATE-----xxx省略-----END CERTIFICATE-----添加代理

生产环境可以不用直连外网,通过在install-config.yaml文件为集群设置代理。

本次测试,为了加速外网下载,我在aws上事先搭建了一个v2ray server,misc服务器作为v2ray客户端,具体搭建过程另文叙述。

在反复试验时,比如 install-config.yaml 所在的目录是 config,必须 rm -rf install 而不是 rm -rf install/*,后者未删除其中的隐藏文件 .openshift_install_state.json,有可能引起:x509: certificate has expired or is not yet valid。在文档和博客示例中 install-config.yaml 的 cidr 配置为 10 网段,由于未细看文档理解成了节点机网段,这造成了整个过程中最莫名其妙的错误:no matches for kind MachineConfig。最终文件内容如下:[root@centos75 install]# vi install-config.yamlapiVersion: v1baseDomain: ipincloud.comproxy: httpProxy: http://192.168.128.30:8001 httpsProxy: http://192.168.128.30:8001compute:- hyperthreading: Enabled name: worker replicas: 0controlPlane: hyperthreading: Enabled name: master replicas: 3metadata: name: ocptestnetworking: clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 networkType: OpenShiftSDN serviceNetwork: - 172.30.0.0/16platform: none: {}fips: falsepullSecret: '{"auths":{"省略'additionalTrustBundle: | -----BEGIN CERTIFICATE----- 省略,注意这里要前面空两格 -----END CERTIFICATE-----imageContentSources:- mirrors: - registry.ipincloud.com:8443/ocp4/openshift4 source: quay.io/openshift-release-dev/ocp-release- mirrors: - registry.ipincloud.com:8443/ocp4/openshift4 source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

(3)备份定制install-config.yaml文件,便于以后可以重复使用

cd /data/installcp install-config.yaml ../install-config.yaml.202002058、创建Kubernetes manifest和Ignition配置文件

(1)生成Kubernetes manifests文件

openshift-install create manifests --dir=/data/install

注意:指定install-config.yaml所在目录是,需要使用绝的路径

(2)修改 manifests/cluster-scheduler-02-config.yml文件以防止pod调度到control plane节点

红帽官方安装文档说明,kubernetes不支持ingress的load balancer访问control-plane节点的pod

a.打开manifests/cluster-scheduler-02-config.ymlb.找到mastersSchedulable参数,设置为Falsec.保存并退出。vi /data/install/manifests/cluster-scheduler-02-config.yml

(3)创建Ignition配置文件

注意:创建Ignition配置文件完成后,install-config.yaml文件将被删除,请务必先备份此文件。

openshift-install create ignition-configs --dir=/data/install

(4)将Ignition配置文件拷贝到http服务器目录,待安装时使用

cd /data/install\cp -f bootstrap.ign /var/www/html/ignition/bootstrap.ign\cp -f master.ign /var/www/html/ignition/master1.ign\cp -f master.ign /var/www/html/ignition/master2.ign\cp -f master.ign /var/www/html/ignition/master3.ign\cp -f worker.ign /var/www/html/ignition/worker1.ign\cp -f worker.ign /var/www/html/ignition/worker2.igncd /var/www/html/ignition/chmod 755 *.ign

至此,已完成必要的配置文件设置,开始进入下一步创建节点。

9、定制RHCOS ISO

安装时需要修改启动参数,只能手动录入,每台机器修改很麻烦,容易出错,因此我们采用genisoimage来定制每台机器的安装镜像。

#安装镜像创建工具yum -y install genisoimage libguestfs-toolssystemctl start libvirtd#设置环境变量export NGINX_DIRECTORY=/data/pkgexport RHCOSVERSION=4.3.0export VOLID=$(isoinfo -d -i ${NGINX_DIRECTORY}/rhcos-${RHCOSVERSION}-x86_64-installer.iso | awk '/Volume id/ { print $3 }')#生成一个临时文件目录,用于放置过程文件TEMPDIR=$(mktemp -d)echo $VOLIDecho $TEMPDIRcd ${TEMPDIR}# Extract the ISO content using guestfish (to avoid sudo mount)#使用guestfish可以将不用sudo mount将iso文件解压出来guestfish -a ${NGINX_DIRECTORY}/rhcos-${RHCOSVERSION}-x86_64-installer.iso \ -m /dev/sda tar-out / - | tar xvf -#定义修改配置文件的函数modify_cfg(){ for file in "EFI/redhat/grub.cfg" "isolinux/isolinux.cfg"; do # 添加恰当的 image 和 ignition url sed -e '/coreos.inst=yes/s|$| coreos.inst.install_dev=sda coreos.inst.image_url='"${URL}"'\/install\/'"${BIOSMODE}"'.raw.gz coreos.inst.ignition_url='"${URL}"'\/ignition\/'"${NODE}"'.ign ip='"${IP}"'::'"${GATEWAY}"':'"${NETMASK}"':'"${FQDN}"':'"${NET_INTERFACE}"':none:'"${DNS}"' nameserver='"${DNS}"'|' ${file} > $(pwd)/${NODE}_${file##*/} # 修改参数里的启动等待时间 sed -i -e 's/default vesamenu.c32/default linux/g' -e 's/timeout 600/timeout 10/g' $(pwd)/${NODE}_${file##*/} done}#设置url,网关、dns等iso启动通用参数变量URL="http://192.168.128.30:8080"GATEWAY="192.168.128.254"NETMASK="255.255.255.0"DNS="192.168.128.30"#设置bootstrap节点变量NODE="bootstrap"IP="192.168.128.31"FQDN="bootstrap"BIOSMODE="bios"NET_INTERFACE="ens192"modify_cfg#设置master1节点变量NODE="master1"IP="192.168.128.32"FQDN="master1"BIOSMODE="bios"NET_INTERFACE="ens192"modify_cfg#设置master2节点变量NODE="master2"IP="192.168.128.33"FQDN="master2"BIOSMODE="bios"NET_INTERFACE="ens192"modify_cfg#设置master3节点变量NODE="master3"IP="192.168.128.34"FQDN="master3"BIOSMODE="bios"NET_INTERFACE="ens192"modify_cfg#设置master4节点变量NODE="worker1"IP="192.168.128.35"FQDN="worker1"BIOSMODE="bios"NET_INTERFACE="ens192"modify_cfg#设置master5节点变量NODE="worker2"IP="192.168.128.36"FQDN="worker2"BIOSMODE="bios"NET_INTERFACE="ens192"modify_cfg# 为每个节点创建不同的安装镜像# https://github.com/coreos/coreos-assembler/blob/master/src/cmd-buildextend-installer#L97-L103for node in bootstrap master1 master2 master3 worker1 worker2; do # 为每个节点创建不同的 grub.cfg and isolinux.cfg 文件 for file in "EFI/redhat/grub.cfg" "isolinux/isolinux.cfg"; do /bin/cp -f $(pwd)/${node}_${file##*/} ${file} done # 创建iso镜像 genisoimage -verbose -rock -J -joliet-long -volset ${VOLID} \ -eltorito-boot isolinux/isolinux.bin -eltorito-catalog isolinux/boot.cat \ -no-emul-boot -boot-load-size 4 -boot-info-table \ -eltorito-alt-boot -efi-boot images/efiboot.img -no-emul-boot \ -o ${NGINX_DIRECTORY}/${node}.iso .done# 清除过程文件cdrm -Rf ${TEMPDIR}cd ${NGINX_DIRECTORY}9、在节点机器上安装RHCOS

(1)将定制的ISO文件拷贝到vmware esxi主机上,准备装节点

[root@misc pkg]# scp bootstrap.iso root@192.168.128.200:/vmfs/volumes/hdd/iso[root@misc pkg]# scp m*.iso root@192.168.128.200:/vmfs/volumes/hdd/iso[root@misc pkg]# scp w*.iso root@192.168.128.200:/vmfs/volumes/hdd/iso

(2)按规划创建master,设置从iso启动安装

进入启动界面后,直接点击安装,系统自动回自动下载bios和配置文件,完成安装安装完成后,需要将iso文件退出来,避免再次进入安装界面安装顺序是bootstrap,master1,master2,master3,待master安装并启动完成后,再进行worker安装安装过程中可以通过proxy查看进度 http://registry.ipincloud.com:9000/安装过程中可以在misc节点查看详细的bootstrap进度。openshift-install --dir=/data/install wait-for bootstrap-complete --log-level debug

注意事项:

ignition和iso文件的正确匹配我在安装的时候,master1提示etcdmain: member ab84b6a6e4a3cc9a has already been bootstrapped,花了很多时间分析和解决问题,因为master1在安装完成后,etcd组件会自动安装并注册为member,我再次使用iso文件重新安装master1后,etcd自动安装注册时,会检测到etcd及集群里已经有这个member,无法重新注册,因此这个节点的etcd一直无法正常启动,解决办法是:

手工修改-aster1节点的etcd的yaml文件,在exec etcd命令末尾增加–initial-cluster-state=existing参数,再删除问题POD后,系统会自动重新安装etcd pod,恢复正常。 正常启动以后,要把这个改回去,否则machine-config回一直无法完成

#[root@master1 /]# vi /etc/kubernetes/manifests/etcd-member.yaml exec etcd \ --initial-advertise-peer-urls=https://${ETCD_IPV4_ADDRESS}:2380 \ --cert-file=/etc/ssl/etcd/system:etcd-server:${ETCD_DNS_NAME}.crt \ --key-file=/etc/ssl/etcd/system:etcd-server:${ETCD_DNS_NAME}.key \ --trusted-ca-file=/etc/ssl/etcd/ca.crt \ --client-cert-auth=true \ --peer-cert-file=/etc/ssl/etcd/system:etcd-peer:${ETCD_DNS_NAME}.crt \ --peer-key-file=/etc/ssl/etcd/system:etcd-peer:${ETCD_DNS_NAME}.key \ --peer-trusted-ca-file=/etc/ssl/etcd/ca.crt \ --peer-client-cert-auth=true \ --advertise-client-urls=https://${ETCD_IPV4_ADDRESS}:2379 \ --listen-client-urls=https://0.0.0.0:2379 \ --listen-peer-urls=https://0.0.0.0:2380 \ --listen-metrics-urls=https://0.0.0.0:9978 \ --initial-cluster-state=existing [root@master1 /]# crictl podsPOD ID CREATED STATE NAME NAMESPACE ATTEMPTc4686dc3e5f4f 38 minutes ago Ready etcd-member-master1.ocptest.ipincloud.com openshift-etcd 5 [root@master1 /]# crictl rmp xxx检查是否安装完成如果出现INFO It is now safe to remove the bootstrap resources,表示master节点安装完成,控制面转移到master集群。[root@misc install]# openshift-install --dir=/data/install wait-for bootstrap-complete --log-level debugDEBUG OpenShift Installer v4.3.0DEBUG Built from commit 2055609f95b19322ee6cfdd0bea73399297c4a3eINFO Waiting up to 30m0s for the Kubernetes API at https://api.ocptest.ipincloud.com:6443...INFO API v1.16.2 upINFO Waiting up to 30m0s for bootstrapping to complete...DEBUG Bootstrap status: completeINFO It is now safe to remove the bootstrap resources[root@misc install]#

(3)安装worker

进入启动界面后,直接点击安装,系统自动回自动下载bios和配置文件,完成安装安装完成后,需要将iso文件退出来,避免再次进入安装界面安装顺序是bootstrap,master1,master2,master3,待master安装并启动完成后,再进行worker安装安装过程中可以通过proxy查看进度 http://registry.ipincloud.com:9000/也可以在misc节点是查看详细安装节点[root@misc redhat-operators-manifests]# openshift-install --dir=/data/install wait-for install-complete --log-level debugDEBUG OpenShift Installer v4.3.0DEBUG Built from commit 2055609f95b19322ee6cfdd0bea73399297c4a3eINFO Waiting up to 30m0s for the cluster at https://api.ocptest.ipincloud.com:6443 to initialize...DEBUG Cluster is initializedINFO Waiting up to 10m0s for the openshift-console route to be created...DEBUG Route found in openshift-console namespace: consoleDEBUG Route found in openshift-console namespace: downloadsDEBUG OpenShift console route is createdINFO Install complete!INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/data/install/auth/kubeconfig'INFO Access the OpenShift web-console here:https://console-openshift-console.apps.ocptest.ipincloud.comINFO Login to the console with user: kubeadmin, password: pubmD-8Baaq-IX36r-WIWWf需要审批worker节点的加入申请

查看待审批的csr

[root@misc ~]# oc get csrNAME AGE REQUESTOR CONDITIONcsr-7lln5 70m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issuedcsr-d48xk 69m system:node:master1.ocptest.ipincloud.com Approved,Issuedcsr-f2g7r 69m system:node:master2.ocptest.ipincloud.com Approved,Issuedcsr-gbn2n 69m system:node:master3.ocptest.ipincloud.com Approved,Issuedcsr-hwxwx 13m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pendingcsr-ppgxx 13m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pendingcsr-wg874 70m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issuedcsr-zkp79 70m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued[root@misc ~]#

执行审批

oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve

(3)在misc上启动nfs

bash /data/pkg/ocp4-upi-helpernode/files/nfs-provisioner-setup.sh#查看状态oc get pods -n nfs-provisioner(4)ocp内部registry使用nfs作为存储oc patch configs.imageregistry.operator.openshift.io cluster -p '{"spec":{"storage":{"pvc":{"claim":""}}}}' --type=mergeoc get clusteroperator image-registry10 配置登录

(1)配置普通管理员账号

#在misc机器上创建admin tokenmkdir -p ~/authhtpasswd -bBc ~/auth/admin-passwd admin scwang18#拷贝到本地mkdir -p ~/authscp -P 20030 root@218.89.67.36:/root/auth/admin-passwd ~/auth/#在 OAuth Details 页面添加 HTPasswd 类型的 Identity Providers 并上传admin-passwd 文件。https://console-openshift-console.apps.ocptest.ipincloud.com#授予新建的admin用户集群管理员权限oc adm policy add-cluster-role-to-user cluster-admin admin

标签: #v2ray服务器搭建

  • 评论列表

留言评论

 
QQ在线咨询
售前咨询电话
173-175-32776
技术支持电话
173-175-32776
嘿,欢迎咨询