Connor's Site

【私有镜像仓库(Harbor)】【高可用】

2019-08-25T06:51:40.347000Z

Registry高可用性设计

Registry高可用性（HA）是多数生产系统需要关心的问题，基本要求就是没有单点故障。下面是2种常见的HA方案：

（1）共享存储

一种比较标准的方案，就是多个的Registry实例共享同一个后端存储，任何一个实例持久化到存储的镜像，都可被其他实例中读取。通过前置LB进来的请求，可以分流到不同的实例中去处理，实现了负载均衡，也避免了单点故障。

（2）双主复制

在两个节点间采用双主复制策略，互相复制镜像。即使有一个实例失效，另一个实例仍然可以提供服务，从而在一定程度上可以满足HA的需求。

相比而言采用共享存储的方式更为合理，后端即可采用ceph、swift共享存储方式，亦可使用阿里云等云存储方式。

存在的问题：

（1）session 保存

请求经过LB转发以后，每次请求都会被平均分配至各个harbor。这就导致了使用浏览器登录时，会出现session在不同的harbor间失效的问题。可以通过设置LB将指定IP的请求发送到固定的harbor可以解决上述问题，也可以通过共享session的方法解决。共享session的话，目前harbor支持使用redis来共享session。

（2）数据库数据一致性

因为高可用就得需要启动至少两个harbor，而每个harbor都有自己的mysql，其数据库数据的一致性问题就有待解决。解决方案可以让多个harbor使用同一个mysql数据库，数据库采用高可用的方式搭建。这样就屏蔽掉了数据不一致问题。

Harbor 如何使用外部数据库？

操作步骤如下：

（1）harbor数据库导出

docker exec -it harbor-db bash

mysqldump -u root -p --databases registry > registry.dump #数据库的默认密码在cat common/config/adminserver/env 中查看

docker cp harbor-db:/tmp/registry.dump . #退出container,将备份的数据copy 出来

（2）将数据导入到外部mysql

mysql -uharbor -h $MYSQLHOST -p

source ./registry.dump;

（3）配置外置数据库

修改docker-compose.yml 文件，将mysql部分删除

（4）修改harbor导入的环境变量

共享session设置：

Harbor对session共享已经给予了支持，只需要将_REDIS_URL这个环境变量配置到各个节点harbor ui组件的env文件中即可：

vi common/config/adminserver/env

LOG_LEVEL=debug

CONFIG_PATH=/etc/ui/app.conf

UI_SECRET=LuAwkKUtYjF4l0mQ

JOBSERVICE_SECRET=SmsO1kVo4SrmgOIp

GODEBUG=netdns=cgo

_REDIS_URL=redis_ip:6379,100,redis_password,0

【私有镜像仓库(Harbor)】【安装配置】

2019-08-25T06:48:46.043000Z

安装过程参考： https://github.com/vmware/harbor/blob/master/docs/installation_guide.md

Harbor的所有服务组件都是在Docker中部署的，所以官方安装使用Docker-compose快速部署，所以需要首先安装Docker、Docker-compose。由于Harbor是基于Docker Registry V2版本，所以就要求Docker版本不小于1.10.0，Docker-compose版本不小于1.6.0。

Harbor 安装分为在线安装和离线安装，这里我们采用在线安装版本(v1.2.2)，机器：devops-k8s-2-c.hb2

(1): 下载并解压 harbor-online-installer-v1.2.2.tgz

(2): 配置 harbor.cfg

# hostname设置访问地址，可以使用ip、域名

hostname = registry.intra.xxxx.com

# 邮件设置，发送重置密码邮件时使用

email_identity =

email_server = smtp.mxhichina.com

email_server_port = 465

email_username = connor@xxxx.com

email_password = *******

email_from = Harbor @xxxx.com>

email_ssl = true

# 认证方式，这里支持多种认证方式，如LADP、本次存储、数据库认证。默认是db_auth，mysql数据库认证

auth_mode = ldap_auth

# LDAP认证时配置项

ldap_url = ldap://127.0.0.1:389

ldap_searchdn = cn=Manager,dc=xxxx,dc=com

ldap_search_pwd = ********

ldap_basedn = dc=xxxx,dc=com

ldap_uid = uid

ldap_scope = 3

ldap_timeout = 5

(3) 安装启动harbor:

sudo ./install.sh

(4) 配置nginx (host: monitor, monitor2 file:/opt/nginx/conf/vhosts/harbor.conf)

server {

listen 80;

server_name registry.intra.xxxx.com;

error_log /data/logs/nginx/error-nile.log;

client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image uploads

location ^~ /proxy/ {

internal; #指定规则为internal规则，防止外部请求命中此规则

rewrite '^/proxy/(http?)/([^/]+)/(\d+)/(.*)' /$4 break;

proxy_pass $1://$2:$3;

}

location / {

proxy_pass http://devops-k8s-2-c.hb2;#harbor所在机器

}

重新加载nginx 配置： nginx -s reload

测试地址： http://registry.intra.xxxx.com/

注：(1): 数据默认持久化在 /data 目录（secretkey_path配置），日志目录在/var/log/harbor

（2）: harbor通过docker-compose 的启动命令参考： https://github.com/vmware/harbor/blob/master/docs/installation_guide.md#managing-harbors-lifecycle

【私有镜像仓库(Harbor)】

2019-08-25T06:45:37.711000Z

Harbor 简介：

Harbor 是 Vmwar 公司开源的企业级的 Docker Registry 管理项目, 是对官方开源的 Docker Registry 的扩展，增加了一些企业需要的功能，如安全、复制和管理。

github: https://github.com/vmware/harbor

Harbor 功能及特点：

（1）基于角色的访问控制（2）基于策略的镜像复制（3）镜像扫描

（4）LDAP/AD 支持（5）镜像删除、垃圾回收（6）UI 界面

（7）日志审计（8）RESTful API （9）易于部署

架构介绍

Habor 在架构上主要由五个组件构成：

（1）Proxy: Harbor 的registry, UI, token 等服务，通过一个前置的反向代理统一接收浏览器、Docker客户端的请求，并将请求转发给后端不同的服务。

（2）Registry: 负责存储Docker 镜像，并处理docker push/pull 命令。由于要对用户进行访问控制（不同用户对docker image有不同的读写权限），Registry 会指向一个token服务，强制用户每次docker push/pull 请求都要携带一个合法的token, Registry 会进行验证。

（3）Core services: Harbor的核心功能，主要提供以下服务：

UI: 提供图形化界面，帮助用户管理registry 上的镜像，并对用户进行授权

webhook: 监控Registry上的image状态变化，把状态及时传递给UI 模块

token 服务：根据用户权限给每个docker push/pull 命令签发token

(4) Database: 负责存储用户权限、审计日志、Docker image 分组信息等数据

(5) Log collector: 负责收集其他组件的log

部署配置：

安装过程参考： https://github.com/vmware/harbor/blob/master/docs/installation_guide.md

配置如下：

# hostname设置访问地址，可以使用ip、域名

hostname = registry.intra.xxxx.com

# 邮件设置，发送重置密码邮件时使用

email_identity =

email_server = smtp.mxhichina.com

email_server_port = 465

email_username = connor@xxxx.com

email_password = *******

email_from = Harbor @xxxx.com>

email_ssl = true

# 认证方式，这里支持多种认证方式，如LADP、本次存储、数据库认证。默认是db_auth，mysql数据库认证

auth_mode = ldap_auth

# LDAP认证时配置项

ldap_url = ldap://127.0.0.1:389

ldap_searchdn = cn=Manager,dc=xxxx,dc=com

ldap_search_pwd = ********

ldap_basedn = dc=xxxx,dc=com

ldap_uid = uid

ldap_scope = 3

ldap_timeout = 5

测试地址： http://registry.intra.xxxx.com

上传/下载镜像

在Docker daemon的启动配置中加入以下参数 --insecure-registry registry.intra.xxxx.com

拉取镜像： docker pull registry.intra.xxxx.com/library/busybox:latest

上传镜像：

(1) 登录： docker login -u $USER -p $PWD registry.intra.xxxx.com

(2) 给镜像打tag: docker tag docker.io/nginx:1.9.1 registry.intra.xxxx.com/library/nginx:1.9.1

(3) 上传镜像：docker push registry.intra.xxxx.com/library/nginx:1.9.1

【openshift教程-部署】【集群扩容、缩容及版本升级】

2019-08-25T06:35:53.228000Z

一、集群扩容

通过Ansible进行扩容需要编辑Ansible的Inventory，即/etc/ansible/hosts文件。在相应的角色下添加新节点的域名及配置信息。然后执行Ansible的Playbook实施具体的扩容动作。

执行node扩容命令：

ansible-playbook openshift-ansible/playbooks/byo/openshift-node/scaleup.yml

主控节点扩容通过执行openshift-master/scaleup.yml Playbook实现：

ansible-playbook openshift-ansible/playbooks/byo/openshift-master/scaleup.yml

二、集群缩容

缩容是指减少集群的计算资源。缩容主要考虑如何将计算资源抽离集群而不影响现有的业务。

在实际部署中，大量的容器应用都运行在计算节点之上。所幸的是，OpenShift是一个多节点的集群云平台，集群内存在多个计算节点，提供了冗余。

在缩容的场景中，集群管理员需要保证：新的容器不会再创建于要缩减的计算节点之上；当前运行在计划缩减的计算节点之上的容器能迁移到其他计算节点之上。

OpenShift提供了相关的机制帮助集群管理员控制一个计算节点是否可用于容器的编排调度。

例如，主控节点Master默认情况下是不运行容器的。查看节点的状态，可以看到Master节点的状态是SchedulingDisabled，即不可调度的。

停止调度：

oc adm manage-node node1.example.com --schedulable=false

要注意的是，将节点设置成不可调度之后，OpenShift的调度器将停止往该节点上分配新的容器，但是原先在该节点上运行的容器，仍然在该节点上运行。

当一个计算节点因故希望退出集群时，集群管理员应该先将其上运行的容器调度到其他的计算节点之上，避免计算节点下线导致应用服务中断。

可以通过oc adm manage-node --evacuate命令将指定的Pod迁移到其他的计算节点上。例如，下面的例子将node1节点上所有带有app=httpd标签的Pod都迁移出该计算节点。

oc adm manage-node node1.example.com --evacuate --pod-selector="app=httpd"

三、版本升级

从3.7升级到3.9

wget https://github.com/openshift/openshift-ansible/archive/openshift-ansible-3.10.0-0.7.0.tar.gz

tar -zxf openshift-ansible-3.10.0-0.7.0.tar.gz

ansible-playbook -i ~/ansible-inventory openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade.yml

【openshift教程-部署】【生产环境部署】

2019-08-25T06:31:32.146000Z

一、准备

wget https://github.com/openshift/openshift-ansible/archive/openshift-ansible-3.7.39-1.tar.gz

tar zxf openshift-ansible-3.7.39-1.tar.gz

hosts文件

hosts

[OSEv3:children]
masters
nodes
etcd
#nfs
lb
 
new_nodes
 
#[nfs]
#openshift-harbor-0-b.hb2
 
[OSEv3:vars]
openshift_node_kubelet_args={'experimental-allowed-unsafe-sysctls': ['net.core.somaxconn']}
ansible_ssh_user=root
deployment_type=origin
#openshift_deployment_type=origin
openshift_release=v3.7
openshift_image_tag=v3.7.0
openshift_pkg_version=-3.7.0
openshift_logging_image_version=v3.7.0
openshift_disable_check=disk_availability,docker_storage,memory_availability,docker_image_availability,package_version,package_availability
openshift_master_identity_providers=[{'name': 'my_ldap_provider', 'challenge': 'true', 'login': 'true', 'kind': 'LDAPPasswordIdentityProvider', 'attributes': {'id': ['uid'], 'email': ['mail'], 'name': ['uid'], 'preferredUsername': ['uid'
]}, 'bindDN': 'cn=Manager,dc=xxxx,dc=com', 'bindPassword': 'XXXXXXXXXX', 'ca': '', 'insecure': 'true', 'url': 'ldap://127.0.0.1:389/ou=TECH,o=xxxxx,dc=xxxx,dc=com?uid'}]
 
openshift_enable_service_catalog=false
template_service_broker_install=false
 
openshift_master_default_subdomain=apps.intra.xxxx.com
openshift_master_cluster_hostname=openshift.intra.xxxx.com
openshift_master_cluster_public_hostname=openshift.intra.xxxx.com
 
 
debug_level=2
openshift_hosted_registry_replicas=3
openshift_docker_insecure_registries=registry.intra.xxxx.com
#openshift_hosted_registry_selector={ 'type': 'registry'}
openshift_hosted_registry_cert_expire_days=720
openshift_hosted_registry_storage_kind=nfs
openshift_hosted_registry_storage_access_modes=['ReadWriteMany']
openshift_hosted_registry_storage_nfs_directory=/data/exports
openshift_hosted_registry_storage_nfs_options='*(rw,root_squash)'
openshift_hosted_registry_storage_volume_name=registry
openshift_hosted_registry_storage_volume_size=50Gi
 
openshift_metrics_install_metrics=true
openshift_metrics_storage_kind=nfs
openshift_metrics_storage_access_modes=['ReadWriteOnce']
openshift_metrics_storage_nfs_directory=/data/exports
openshift_metrics_storage_nfs_options='*(rw,root_squash)'
openshift_metrics_storage_volume_name=metrics
openshift_metrics_storage_volume_size=50Gi
openshift_metrics_image_version=v3.7
#openshift_metrics_storage_labels={'region': 'infra'}
 
#openshift_hosted_router_selector={'type': 'router'}
#openshift_hosted_router_replicas=1
 
[masters]
openshift-master-0-b.hb2
openshift-master-1-c.hb2
openshift-master-2-d.hb2
 
 
[nodes]
openshift-master-0-b.hb2
openshift-master-1-c.hb2
openshift-master-2-d.hb2
openshift-services-1-b.hb2 openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-b'}"
openshift-services-1-c.hb2 openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-c'}"
openshift-services-1-d.hb2  openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-d'}"
openshift-services-0-b.hb2 openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-b'}"
openshift-services-0-c.hb2 openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-c'}"
openshift-services-0-d.hb2 openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-d'}"
openshift-base-0-b.hb2 openshift_node_labels="{'region' : 'infra' , 'type' : 'router', 'servicetype': 'base', 'zone': 'zone-b'}"
openshift-base-1-c.hb2 openshift_node_labels="{'region' : 'infra' , 'type' : 'registry', 'servicetype': 'base', 'zone': 'zone-c'}"
openshift-base-2-d.hb2 openshift_node_labels="{'region' : 'infra' , 'type' : 'registry', 'servicetype': 'base', 'zone': 'zone-d'}"
openshift-gateway-1-c.hb2 openshift_node_labels="{'nodetype': 'node', 'servicetype': 'gateway', 'zone': 'zone-c'}"
openshift-gateway-0-b.hb2 openshift_node_labels="{'nodetype': 'node', 'servicetype': 'gateway', 'zone': 'zone-b'}"
openshift-inf-0-c.hb2  openshift_node_labels="{'nodetype': 'node', 'servicetype': 'inf', 'zone': 'zone-c'}"
openshift-inf-0-d.hb2  openshift_node_labels="{'nodetype': 'node', 'servicetype': 'inf', 'zone': 'zone-d'}"
openshift-services-2-c.hb2   openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-c'}"
#openshift-services-2-b.hb2   openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-b'}"
openshift-services-3-b.hb2   openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-b'}"
openshift-services-3-c.hb2   openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-c'}"
openshift-services-4-c.hb2   openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-c'}"
openshift-services-5-c.hb2   openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-c'}"
openshift-services-4-b.hb2  openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-b'}"
openshift-services-2-d.hb2  openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-d'}"
openshift-services-3-d.hb2  openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-d'}"
openshift-services-0-e.hb2  openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-e'}"
openshift-services-1-e.hb2  openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-e'}"
 
#[new_nodes]
#openshift-services-4-b.hb2  openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-b'}"
#openshift-services-2-d.hb2  openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-d'}"
#openshift-services-3-d.hb2  openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-d'}"
#openshift-services-0-e.hb2  openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-e'}"
#openshift-services-1-e.hb2  openshift_node_labels="{'nodetype': 'node', 'servicetype': 'online', 'zone': 'zone-e'}"
 
 
 
[etcd]
openshift-etcd-2-d.hb2
openshift-etcd-1-c.hb2
openshift-etcd-0-b.hb2
 
[lb]
openshift.intra.xxxx.com

二、安装依赖

所有机器上执行：

pre_install.sh

systemctl stop firewalld
systemctl disable firewalld
yum install -y wget git net-tools bind-utils iptables-services bridge-utils bash-completion curl vim openssl python-passlib conntrack-tools socat nfs-utils
yum install -y yum-utils device-mapper-persistent-data lvm2 docker-1.13.1
yum install -y NetworkManager
#pip uninstall -y urllib3
#yum install -y python-urllib3-1.10.2-3.el7.noarch
systemctl restart NetworkManager
systemctl enable NetworkManager
rm -rf /etc/docker/daemon.json
tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://docker.mirrors.ustc.edu.cn/"]
}
EOF
 
mkdir -p /data/docker
rm -rf /data/docker/*
rm -rf /var/lib/docker/
ln -s /data/docker /var/lib/
systemctl restart docker
pip install ansible==2.4.3.0
 
yum install -y cockpit cockpit-*
systemctl restart cockpit
systemctl enable cockpit.socket

rpm_install.sh

rpm包地址：

http://mirror.centos.org/centos/7/paas/x86_64/openshift-origin/

rpm_install.sh

yum install -y conntrack-tools socat nfs-utils
yum localinstall -y origin-clients-3.7.0-1.0.7ed6862.x86_64.rpm
yum localinstall -y origin-3.7.0-1.0.7ed6862.x86_64.rpm
yum localinstall -y tuned-profiles-origin-node-3.7.0-1.0.7ed6862.x86_64.rpm
yum localinstall -y openvswitch-2.6.1-0.el7.x86_64.rpm
rpm -ivh origin-node-3.7.0-1.0.7ed6862.x86_64.rpm
rpm -ivh origin-master-3.7.0-1.0.7ed6862.x86_64.rpm
rpm -ivh origin-docker-excluder-3.7.0-1.0.7ed6862.noarch.rpm
rpm -ivh origin-excluder-3.7.0-1.0.7ed6862.noarch.rpm
rpm -ivh origin-sdn-ovs-3.7.0-1.0.7ed6862.x86_64.rpm
rm -rf /etc/origin/*

rm_config.sh

mv /etc/origin/master /etc/origin/master.bk
mv /etc/origin/node /etc/origin/node.bk

pre_img.sh

docker pull docker.io/openshift/origin-docker-registry:v3.7.0
docker pull docker.io/openshift/origin-sti-builder:v3.7.0
docker pull docker.io/openshift/origin-deployer:v3.7.0
docker pull docker.io/openshift/origin-haproxy-router:v3.7.0
docker pull docker.io/openshift/origin-pod:v3.7.0
docker pull docker.io/cockpit/kubernetes

etcd集群上执行：

ssh -t openshift-etcd-0-b.hb2 'yum install -y etcd'

ssh -t openshift-etcd-1-c.hb2 'yum install -y etcd'

ssh -t openshift-etcd-2-d.hb2 'yum install -y etcd'

三、执行集群部署

ansible-playbook -i inventory/hosts openshift-ansible-openshift-ansible-3.7.39-1/playbooks/byo/config.yml

四、备注

执行完成后，添加router的环境变量（默认是30s超时，需要fix满足业务需求）

ROUTER_DEFAULT_CLIENT_TIMEOUT 120s

ROUTER_DEFAULT_SERVER_TIMEOUT 120s

【openshift教程-部署】【一主多Node】

2019-08-25T06:26:35.506000Z

阿里云上准备三台机器

master hostname: master.openshift.yqb.com ip address: 10.25.158.97

node1 hostname: node1.openshift.yqb.com ip address: 10.25.91.65

node2 hostname: node2.openshift.yqb.com ip address: 10.25.199.113

1、在所有机器上关闭firewalld

close firewalld

systemctl stop firewalld
systemctl disable firewalld

2、在所有机器上安装基础的包

install base

yum install -y wget git net-tools bind-utils iptables-services bridge-utils bash-completion curl vim openssl

3、在master机器上安装dns服务，如果使用已经有的可以忽略

install dns

yum install -y bind

vim /etc/named.conf；修改两行：

allow-query     { 0.0.0.0/0; };
listen-on port 53 { any; };

并添加dns转发

如下：

/etc/named.conf

//
// named.conf
//
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
// server as a caching only nameserver (as a localhost DNS resolver only).
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//
// See the BIND Administrator's Reference Manual (ARM) for details about the
// configuration located in /usr/share/doc/bind-{version}/Bv9ARM.html
options {
        /*listen-on port 53 { 127.0.0.1; };*/
        listen-on port 53 { any; };
        listen-on-v6 port 53 { ::1; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        allow-query     { 0.0.0.0/0; };
        /*allow-query     { localhost; };*/
        /*
         - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.
         - If you are building a RECURSIVE (caching) DNS server, you need to enable
           recursion.
         - If your recursive DNS server has a public IP address, you MUST enable access
           control to limit queries to your legitimate users. Failing to do so will
           cause your server to become part of large scale DNS amplification
           attacks. Implementing BCP38 within your network would greatly
           reduce such attack surface
        */
        recursion yes;
        dnssec-enable no;
        dnssec-validation no;
        /* Path to ISC DLV key */
        bindkeys-file "/etc/named.iscdlv.key";
        managed-keys-directory "/var/named/dynamic";
        pid-file "/run/named/named.pid";
        session-keyfile "/run/named/session.key";
        forwarders{
                10.143.22.116;
                10.143.22.118;
        };
};
logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};
zone "." IN {
        type hint;
        file "named.ca";
};
include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";

vim /etc/named.rfc1912.zones ；添加如下内容

/etc/named.rfc1912.zones

// named.rfc1912.zones:
//
// Provided by Red Hat caching-nameserver package
//
// ISC BIND named zone configuration for zones recommended by
// RFC 1912 section 4.1 : localhost TLDs and address zones
// and http://www.ietf.org/internet-drafts/draft-ietf-dnsop-default-local-zones-02.txt
// (c)2007 R W Franks
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//
zone "localhost.localdomain" IN {
        type master;
        file "named.localhost";
        allow-update { none; };
};
zone "localhost" IN {
        type master;
        file "named.localhost";
        allow-update { none; };
};
zone "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa" IN {
        type master;
        file "named.loopback";
        allow-update { none; };
};
zone "1.0.0.127.in-addr.arpa" IN {
        type master;
        file "named.loopback";
        allow-update { none; };
};
zone "0.in-addr.arpa" IN {
        type master;
        file "named.empty";
        allow-update { none; };
};
zone "openshift.yqb.com" IN {
    type master;
    file "named.openshift.yqb.com";
    allow-update { none; };
};
zone "25.10.in-addr.arpa" IN {
        type master;
        file "10.25.arpa";
        allow-update { none; };
};

vim /var/named/named.openshift.yqb.com；添加文件

/var/named/named.openshift.yqb.com

$TTL 1D
@       IN SOA openshift.yqb.com. rname.invalid. (
                                        0       ; serial 
                                        1D      ; refresh 
                                        1H      ; retry 
                                        1W      ; expire 
                                        3H )    ; minimum 
@       IN NS @
        A  10.25.158.97
master  IN A 10.25.158.97
node1  IN A 10.25.91.65
node2  IN A 10.25.199.113

vim /var/named/10.25.arpa ；添加文件

/var/named/10.25.arpa

$TTL 1D
@       IN SOA  openshift.yqb.com. rname.invalid. (
                                        0       ; serial 
                                        1D      ; refresh 
                                        1H      ; retry 
                                        1W      ; expire 
                                        3H )    ; minimum 
        NS      @
        AAAA    ::1
158.97      PTR      10.25.158.97.openshift.yqb.com.
91.65      PTR      10.25.91.65.openshift.yqb.com.
199.113      PTR      10.25.199.113.openshift.yqb.com.

启动服务：

start named

systemctl enable named
systemctl start named
systemctl status named

将10.25.158.97 添加到所有机器的 /etc/resolv.conf，并注释掉其它nameserver

/etc/resolv.conf

nameserver 10.25.158.97
#nameserver 10.143.22.116
#nameserver 10.143.22.118
options timeout:2 attempts:3 rotate single-request-reopen

4、安装Openshift基础包

install openshift base

yum install -y docker
vim /etc/sysconfig/docker
OPTIONS=' --selinux-enabled --log-driver=json-file --log-opt max-size=50m'
  
yum install -y centos-release-openshift-origin

5、在master上安装master的包，并设置

install master

yum install -y origin-master origin-pod origin-sdn-ovs origin-dockerregistry

vim /etc/origin/master/master-config.yaml；添加如下

/etc/origin/master/master-config.yaml

  
****
  
corsAllowedOrigins:
- //10\.25\.158\.97:8443$
- //127\.0\.0\.1(:|$)
- //localhost(:|$)
- kubernetes.default
- kubernetes.default.svc.cluster.local
- kubernetes
- openshift.default
- openshift.default.svc
- 172.30.0.1
- openshift.default.svc.cluster.local
- kubernetes.default.svc
- openshift
  
***
  
schedulerConfigFile: "/etc/origin/master/scheduler.json"
  
***
  
networkPluginName: "redhat/openshift-ovs-multitenant"
  
***
  
routingConfig:
    subdomain: openshift.yqb.com
  
***
  

创建调度策略文件scheduler.json，vim /etc/origin/master/scheduler.json

/etc/origin/master/scheduler.json

{
    "apiVersion": "v1",
    "kind": "Policy",
    "predicates": [
        {
            "name": "MatchNodeSelector"
        },
        {
            "name": "PodFitsResources"
        },
        {
            "name": "PodFitsPorts"
        },
        {
            "name": "NoDiskConflict"
        },
        {
            "name": "NoVolumeZoneConflict"
        },
        {
            "name": "MaxEBSVolumeCount"
        },
        {
            "name": "MaxGCEPDVolumeCount"
        },
        {
            "argument": {
                "serviceAffinity": {
                    "labels": [
                        "region"
                    ]
                }
            },
            "name": "Region"
        }
    ],
    "priorities": [
        {
            "name": "LeastRequestedPriority",
            "weight": 1
        },
        {
            "name": "SelectorSpreadPriority",
            "weight": 1
        },
        {
            "argument": {
                "serviceAntiAffinity": {
                    "label": "zone"
                }
            },
            "name": "Zone",
            "weight": 2
        }
    ]
}

启动主节点。并配置kub配置文件：

start master

systemctl enable origin-master
systemctl start origin-master
  
mkdir .kube
ln -s /etc/origin/master/admin.kubeconfig .kube/config
  
oc login -u system:admin
oc get all

6、在所有node节点上安装node的包

install node

yum install -y origin-node origin-pod origin-sdn-ovs origin-dockerregistry

7、在master节点上生产node的配置文件

node config

mkdir /etc/origin/node1.openshift.yqb.com
mkdir /etc/origin/node2.openshift.yqb.com
 
ln -s /etc/origin/ openshift.local.config
  
oc adm create-node-config --node-dir='/etc/origin/node1.openshift.yqb.com/' --dns-domain='openshift.yqb.com' --dns-ip='10.25.158.97' --hostnames='node1.openshift.yqb.com' --master='https://10.25.158.97:8443' --network-plugin='redhat/openshift-ovs-multitenant' --node='node1.openshift.yqb.com'
  
oc adm create-node-config --node-dir='/etc/origin/node2.openshift.yqb.com/' --dns-domain='openshift.yqb.com' --dns-ip='10.25.158.97' --hostnames='node2.openshift.yqb.com' --master='https://10.25.158.97:8443' --network-plugin='redhat/openshift-ovs-multitenant' --node='node2.openshift.yqb.com'
  
scp /etc/origin/node1.openshift.yqb.com/*root@node1.openshift.yqb.com:/etc/origin/node/
scp /etc/origin/node2.openshift.yqb.com/*root@node2.openshift.yqb.com:/etc/origin/node/

8、在所有节点上

vim /etc/origin/node/node-config.yaml，在kind后面添加kubeletArguments

kubeletArguments

kind: NodeConfig
kubeletArguments:
  node-labels:
  - region=primary
  - zone=west

将/etc/origin/node/ca.crt内容添加到/etc/ssl/certs/ca-bundle.crt最后

执行启动命令

start node

systemctl enable iptables
systemctl start iptables
 
systemctl enable docker
systemctl start docker
 
systemctl enable origin-node
systemctl start origin-node
  
docker pull openshift/origin-sti-builder     
docker pull openshift/origin-deployer         
docker pull openshift/origin-docker-registry  
docker pull openshift/origin-haproxy-router  
docker pull openshift/origin-pod
  
iptables -N OS_FIREWALL_ALLOW
iptables -I INPUT 8 -j OS_FIREWALL_ALLOW
iptables -I OS_FIREWALL_ALLOW -p tcp -m tcp --dport 10250 -j ACCEPT
iptables -I OS_FIREWALL_ALLOW -p udp -m udp --dport 10250 -j ACCEPT
iptables -I OS_FIREWALL_ALLOW -p tcp -m tcp --dport 10255 -j ACCEPT
iptables -I OS_FIREWALL_ALLOW -p tcp -m tcp --dport 80 -j ACCEPT
iptables -I OS_FIREWALL_ALLOW -p tcp -m tcp --dport 443 -j ACCEPT
iptables -I OS_FIREWALL_ALLOW -p udp -m udp --dport 4789 -j ACCEPT
iptables -I OS_FIREWALL_ALLOW -p udp -m udp --dport 10255 -j ACCEPT

9、在master上执行操作

添加registry

add registry

oc create serviceaccount registry -n default
oadm policy add-scc-to-user privileged system:serviceaccount:default:registry
oadm registry --service-account=registry --mount-host=/opt/openshift-registry
oc create route passthrough --service docker-registry -n default
oc get svc
oc get route

获取到registry的服务IP，这里为172.30.126.243

config registry

oc adm ca create-server-cert --signer-cert=/etc/origin/master/ca.crt  --signer-key=/etc/origin/master/ca.key --signer-serial=/etc/origin/master/ca.serial.txt --hostnames="172.30.126.243,docker-registry-default.openshift.yqb.com" --cert=/etc/origin/master/registry.crt --key=/etc/origin/master/registry.key
  
oc secrets new registry-certificates /etc/origin/master/registry.crt /etc/origin/master/registry.key -n default
  
oc secrets add registry registry-certificates -n default
oc secrets add default registry-certificates -n default
  
 
oc env dc/docker-registry REGISTRY_HTTP_TLS_CERTIFICATE=/etc/secrets/registry.crt REGISTRY_HTTP_TLS_KEY=/etc/secrets/registry.key -n default
oc patch dc/docker-registry -p '{"spec":{"template":{"spec":{"containers":[{"name":"registry","livenessProbe":{"httpGet":{"scheme":"HTTPS"}}}]}}}}'  -n default
oc patch dc/docker-registry -p '{"spec":{"template":{"spec":{"containers":[{"name":"registry","readinessProbe":{"httpGet":{"scheme":"HTTPS"}}}]}}}}'  -n default
oc volume dc/docker-registry --add --type=secret --secret-name=registry-certificates -m /etc/secrets -n default
oc create serviceaccount router -n default
oadm policy add-scc-to-user hostnetwork system:serviceaccount:default:router
oadm policy add-cluster-role-to-user cluster-reader system:serviceaccount:default:router
oadm router router --replicas=1 --service-account=router
  
chown 1001:root /opt/openshift-registry
  

10、添加模板

官方模板地址：https://github.com/openshift/origin/tree/master/examples

例如:https://github.com/openshift/origin/tree/master/examples/quickstarts中有很多常用的示例模板，git clone到本地即可，进入example目录

导入方法：

import template

oc create -f image-streams-centos7.json -n openshift

打开浏览器：https://10.25.158.97:8443

用户名密码，都是使用中自动创建的（自己随便写，用户会被自动创建保存）

【openshift教程-部署】【单节点部署】

2019-08-25T06:25:38.894000Z

一、准备

1、centos7，最好是最新版

selinux保持打开，默认即可（不要修改）

2、修改hostname

hostnamectl set-hostname master.openshift.yqb.com

3、安装docker

yum install -y docker

systemctl start docker

二、安装Openshift origin

各版本地址：https://github.com/openshift/origin/releases

例如：

1、下载文件

cd /opt/

wget https://github.com/openshift/origin/releases/download/v3.7.0/openshift-origin-server-v3.7.0-7ed6862-linux-64bit.tar.gz

tar -zxf openshift-origin-server-v3.7.0-7ed6862-linux-64bit.tar.gz

ln -s openshift-origin-server-v3.7.0-7ed6862-linux-64bit /opt/openshift

vim /etc/profile 在文件末尾添加path路径：PATH=$PATH:/opt/openshift/

source /etc/profile

cd /opt/

2、启动

cd /opt/openshift

openshift start

3、配置管理员

cp ~

mkdir -p ~/.kube

cp /opt/openshift/openshift.local.config/master/admin.kubeconfig ~/.kube/config

oc get all

三、配置

htpasswd -b /etc/origin/master/htpasswd dev dev

htpasswd -b /etc/origin/master/htpasswd admin admin

oc adm policy add-cluster-role-to-user cluster-admin admin

route的添加

oadm policy add-scc-to-user privileged system:serviceaccount:default:router

oadm router router --replicas=1 --service-account=router

registry的添加

oadm registry --config=/opt/openshift/openshift.local.config/master/admin.kubeconfig --service-account=registry

模板的添加

【openshift教程-19.4】【监控-prometheus-grafana展示页面】

2019-08-25T06:31:34.470000Z

访问地址：http://grafana-monitoring.apps.intra.XXXXXXXX.com

支持自定义，可以在Gitlab对应项目里添加（https://gitee.com/promethues-samples/prometheus-deploy/blob/master/manifests/grafana/grafana-dashboard-definitions-nginx.yaml）

【openshift教程-19.3】【监控-prometheus-查询页面】

2019-08-25T06:31:34.654000Z

访问地址：http://prometheus-k8s-monitoring.apps.intra.XXXX.com

基本上支持所有所需要的指标，需要熟悉PMSQL查询语句

比如：

min(kube_deployment_status_replicas_available{deployment="$deployment_name",namespace="$deployment_namespace"}) without (instance, pod)

【openshift教程-19.2】【监控-prometheus-部署】

2019-08-25T06:31:29.357000Z

1、下载部署项目

git clone https://gitee.com/promethues-samples/prometheus-deploy.git

2、执行一键部署

cd prometheus-operator/ && ./deploy

成功后：

查看所有的route：

【openshift教程-19.1】【监控-prometheus-权限设置】

2019-08-25T06:31:29.499000Z

开通使用宿主机网络及目录挂载权限

oc edit scc restricted

所有节点放开9100的访问权限

iptables -I OS_FIREWALL_ALLOW -p tcp -m tcp --dport 9100 -j ACCEPT

【openshift教程-18】【敏感信息管理secrets】

2019-08-25T03:18:27.813000Z

openshift可以通过secrets来管理敏感信息。例如数据库用户名、密码

还可以通过不同环境下的secrets来管理不同环境下的服务。例如测试环境的app-demo和线上的app-demo，使用不同的secrets来使用测试环境的库或者线上的库

【openshift教程-17】【内核参数调优】

2019-08-25T03:17:05.814000Z

官方文档：https://docs.openshift.org/latest/admin_guide/sysctls.html

例如支持队列个数参数调优

vim /etc/origin/node/node-config.yaml

experimental-allowed-unsafe-sysctls:

- "net.core.somaxconn"

systemctl restart origin-node.service

在配置模板中，添加systcl调优的注释

或者在安装过程中制定kubernetes的参数：

【openshift教程-16】【一次性任务、定时任务】

2019-08-25T03:14:50.113000Z

openshift支持的Job完全是k8s的任务机制。

可以通过查看容器的执行状态，还可以查看执行日志

一次性任务：myjob.yml

myjob.yml

apiVersion: batch/v1
kind: Job
metadata:
  name: myjob
spec:
  template:
    metadata:
      name: myjob
    spec:
      containers:
      - name: hello
        image: busybox
        command: ["echo", "Hello k8s job!"]
      restartPolicy: Never

oc create -f myjob.yml -n hello

定时任务：mycronjob.yml

mycronjob.yml

apiVersion: batch/v2alpha1
kind: CronJob
metadata:
  name: cronjob
spec:
  schedule: "*/720 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            command: ["echo", "Hello k8s job!"]
          restartPolicy: OnFailure

oc create -f mycronjob.yml -n hello

【openshift教程-15】【自动扩容、缩容】

2019-08-25T03:05:25.077000Z

Horizontal Pod Autoscaling可以根据CPU使用率或应用自定义metrics自动扩展Pod数量（支持replication controller、deployment和replica set）。

oc project hello-world

oc autoscale deploymentconfig cakephp-mysql-persistent --max=5 --min=2 --cpu-percent=60

注意，要自动触发hpa，需要给资源设置限额，当超过限额后才会触发

一、自动扩容

测试，在两个pod中，分别打yes命令，占满CPU

二、自动缩容

去掉yes命令

设置触发限额为60%

3分钟后，会自动触发

【openshift教程-14】【应用数据持久化】

2019-08-25T03:00:12.779000Z

核心概念：持久化卷（persistentvolume）和持久化请求（persistentvolumeclaim）

创建一个持久化卷，要有一个共享存储，如Glusterfs、ceph、OpenStack cinder、nfs等等

以nfs为例

一、创建一个nfs共享存储

例如在Openshift集群外的一台机器上，nile-0：

yum -y install nfs-utils rpcbind

echo '/data/exports/pv0002 *(rw,sync,all_squash)' >> /etc/exports

systemctl restart rpcbind

exportfs -r

systemctl restart nfs

则查看showmount -e

二、在Openshift集群，使用管理员账户， pv0003.json

创建pv

oc create -f pv0003.json

状态STATUS，Bound表示已被挂载，Available表示可用

创建pvc

三、使用

只需将项目内的PVC挂载到容器目录即可

四、注意事项

一定要注意nfs共享目录的权限

或者，可以写成应用要使用的权限，例如MySQL用户的权限，等等

【openshift教程-13】【挂载宿主机目录】

2019-08-25T02:52:57.249000Z

宿主机上：

mkdir -p /data/logs
chown -R 1001:0 /data/

因为所有启动用户都是1001用户（default）,所以设置权限给default

添加权限，设置运行所用用户可以使用mount

$ oc edit scc restricted

Change runAsUser.Type to RunAsAny.

Add allowHostDirVolumePlugin: true.

Save the changes.

【openshift教程-12.4】【设置serviceaccount，永久token】

2019-08-25T02:49:56.489000Z

1、添加serviceaccount

oc create serviceaccount cronjob-shanhai

等于创建了system:serviceaccount:shanhai:cronjob-shanhai

2、查询产生的secret

3、获取token

oc describe secret cronjob-shanhai-token-trtvh

4、添加权限

oc policy add-role-to-user cronjob-edit system:serviceaccount:shanhai:cronjob-shanhai -n shanhai

5、使用token

oc login --token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3**********************

oc project shanhai

【openshift教程-12.3】【添加template权限】

2019-08-25T02:44:01.700000Z

创建一个clusterrole，并授权给所有的认证用户

template-view.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
{
    "kind": "ClusterRole",
    "apiVersion": "v1",
    "metadata": {
        "name": "template-view",
        "creationTimestamp": null,
        "annotations": {
            "openshift.io/description": "A user who can view but not edit any resources within the project. They can not view secrets or membership.",
            "openshift.io/reconcile-protect": "false"
        }
    },
    "rules": [
        {
            "verbs": [
                "get"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                ""
            ],
            "resources": [
                "namespaces",
                "namespaces/status"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "",
                "template.openshift.io"
            ],
            "resources": [
                "templates"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "",
                "image.openshift.io"
            ],
            "resources": [
                "imagestreamimages",
                "imagestreams",
                "imagestreamtags"
            ]
        },
        {
            "verbs": [
                "get"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "",
                "image.openshift.io"
            ],
            "resources": [
                "imagestreams/layers"
            ]
        }
    ]
}

oc create -f template-view.json

oc policy add-role-to-group template-view system:authenticated -n openshift

【openshift教程-12.2】【自定义权限】

2019-08-25T02:42:41.032000Z

导出edit和view的clusterrole

oc export clusterrole edit -o json >>edit.json

oc export clusterrole view -o json >>view.json

以view.json为基础，创建自定义role命名为sfg-view.json，添加一个verbs

资源为pod和pod/exec

创建自定义clusterrole

oc create -f sfg-view.json

使用sfg-view，使用户具有对某一个项目打开pod终端的权限

oc policy add-role-to-user sfg-view wanghuafeng -n hello-world

实例：

sfg-cluster-view.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
{
    "kind": "ClusterRole",
    "apiVersion": "v1",
    "metadata": {
        "name": "sfg-cluster-view",
        "creationTimestamp": null,
        "annotations": {
            "authorization.openshift.io/system-only": "true",
            "openshift.io/description": "A super-user that can view all cluster.",
            "openshift.io/reconcile-protect": "false"
        }
    },
    "rules": [
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "*"
            ],
            "resources": [
                "*"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": null,
            "resources": [],
            "nonResourceURLs": [
                "*"
            ]
        }
    ]
}

sfg-view.json

sfg-view

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
{
    "kind": "ClusterRole",
    "apiVersion": "v1",
    "metadata": {
        "name": "sfg-view",
        "creationTimestamp": null,
        "annotations": {
            "openshift.io/description": "A user who can view but not edit any resources within the project. They can not view secrets or membership.",
            "openshift.io/reconcile-protect": "false"
        }
    },
    "rules": [
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                ""
            ],
            "resources": [
                "pods",
                "pods/exec"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                ""
            ],
            "resources": [
                "configmaps",
                "endpoints",
                "persistentvolumeclaims",
                "pods",
                "replicationcontrollers",
                "serviceaccounts",
                "services"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                ""
            ],
            "resources": [
                "bindings",
                "events",
                "limitranges",
                "namespaces",
                "namespaces/status",
                "pods/log",
                "pods/status",
                "replicationcontrollers/status",
                "resourcequotas",
                "resourcequotas/status"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "autoscaling"
            ],
            "resources": [
                "horizontalpodautoscalers"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "batch"
            ],
            "resources": [
                "cronjobs",
                "jobs",
                "scheduledjobs"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "extensions"
            ],
            "resources": [
                "deployments",
                "deployments/scale",
                "horizontalpodautoscalers",
                "replicasets",
                "replicasets/scale"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "extensions"
            ],
            "resources": [
                "daemonsets"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "apps"
            ],
            "resources": [
                "deployments",
                "deployments/scale",
                "statefulsets"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "",
                "build.openshift.io"
            ],
            "resources": [
                "buildconfigs",
                "buildconfigs/webhooks",
                "builds"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "",
                "build.openshift.io"
            ],
            "resources": [
                "builds/log"
            ]
        },
        {
            "verbs": [
                "view"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "build.openshift.io"
            ],
            "resources": [
                "jenkins"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "",
                "apps.openshift.io"
            ],
            "resources": [
                "deploymentconfigs",
                "deploymentconfigs/scale"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "",
                "apps.openshift.io"
            ],
            "resources": [
                "deploymentconfigs/log",
                "deploymentconfigs/status"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "",
                "image.openshift.io"
            ],
            "resources": [
                "imagestreamimages",
                "imagestreammappings",
                "imagestreams",
                "imagestreamtags"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "",
                "image.openshift.io"
            ],
            "resources": [
                "imagestreams/status"
            ]
        },
        {
            "verbs": [
                "get"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "",
                "project.openshift.io"
            ],
            "resources": [
                "projects"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "",
                "quota.openshift.io"
            ],
            "resources": [
                "appliedclusterresourcequotas"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "",
                "route.openshift.io"
            ],
            "resources": [
                "routes"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "",
                "route.openshift.io"
            ],
            "resources": [
                "routes/status"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "",
                "template.openshift.io"
            ],
            "resources": [
                "processedtemplates",
                "templateconfigs",
                "templateinstances",
                "templates"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                "",
                "build.openshift.io"
            ],
            "resources": [
                "buildlogs"
            ]
        },
        {
            "verbs": [
                "get",
                "list",
                "watch"
            ],
            "attributeRestrictions": null,
            "apiGroups": [
                ""
            ],
            "resources": [
                "resourcequotausages"
            ]
        }
    ]
}