GitLab Runner Operator 安装与配置

1. 安装 k8s

版本：v1.23.5

参考：k8s 测试环境搭建（k3s） - leffss - 博客园 (cnblogs.com)

在集群 coredns 中添加 gitlab 主机 hosts 解析

$ kubectl -n kube-system edit configmaps coredns
...
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts /etc/coredns/NodeHosts {
          10.10.10.60 gitlab.leffss.cn
          ttl 60  
          reload 15s 
          fallthrough
        }             
...

# 删除 coredns pod 生效
$ kubectl get pod -n kube-system
NAME                                      READY   STATUS      RESTARTS   AGE
metrics-server-86cbb8457f-hqt9p           1/1     Running     0          5h6m
local-path-provisioner-5ff76fc89d-cz9hj   1/1     Running     0          5h6m
helm-install-traefik-lmt7r                0/1     Completed   0          5h6m
coredns-854c77959c-w4t7j                  1/1     Running     0          5h6m
svclb-traefik-fp7f9                       2/2     Running     0          5h5m
traefik-6f9cbd9bd4-l7h45                  1/1     Running     0          5h5m

$ kubectl -n kube-system delete pod coredns-854c77959c-w4t7j
pod "coredns-854c77959c-nm28h" deleted

2. 安装 gitlab runner operator

安装 Operator Lifecycle Manager (OLM),

$ curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.20.0/install.sh | bash -s v0.20.0

安装 cert-manager

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.yaml

安装 operator

$ kubectl create -f https://operatorhub.io/install/gitlab-runner-operator.yaml

• 当前版本 1.8.0

可以使用以下命令查看当前进度

$ kubectl get csv -n operators

kubectl get pod -n operators
NAME                                                READY   STATUS             RESTARTS   AGE
gitlab-runner-controller-manager-54ddcd566d-hrsc5   1/2     ImagePullBackOff   0          5m38s

Back-off pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0"

查看 pod，发现报错：

$ kubectl -n operators describe pod gitlab-runner-controller-manager-54ddcd566d-hrsc5
...
...
...
  Warning  Failed     3m47s                  kubelet            Failed to pull image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0": rpc error: code = Unknown desc = Error response from daemon: Get "https://gcr.io/v2/": context deadline exceeded
  Normal   BackOff    81s (x14 over 5m38s)   kubelet            Back-off pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0"

原因是无法拉取镜像：gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0

临时解决方法：

docker pull kubesphere/kube-rbac-proxy:v0.8.0
docker tag kubesphere/kube-rbac-proxy:v0.8.0 gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0

3. 部署 gitLab runner

3.1 创建 gitlab runner

创建 runner Registration token secret 文件：

cat > gitlab-runner-secret.yml << EOF
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-runner-secret
type: Opaque
stringData:
  runner-registration-token: <runner Registration token> # your project runner secret
EOF

创建 runner crd yaml 文件：

cat > gitlab-runner.yml << EOF
apiVersion: apps.gitlab.com/v1beta2
kind: Runner
metadata:
  name: gitlab-runner
spec:
  gitlabUrl: https://gitlab.leffss.cn
  buildImage: alpine
  token: gitlab-runner-secret
EOF

创建：

kubectl apply -f gitlab-runner-secret.yml
kubectl apply -f gitlab-runner.yml

查看：

$ kubectl get pod
NAME                                    READY   STATUS    RESTARTS   AGE
gitlab-runner-runner-587447dfbc-hk2q9   1/1     Running   0          2m18s

$ kubectl get runner
NAME            AGE
gitlab-runner   2m21s

注册成功：

3.2 运行任务测试

创建 .gitlab-ci.yml

stages:
  - test

test_script-section:
  script:
    - echo foo
  needs: []
  variables:
    FF_SCRIPT_SECTIONS: "true"

test:
  stage: test
  script:
  - env

运行时会为每个 job 创建一个 pod

$ kubectl get pod -A
NAMESPACE      NAME                                                              READY   STATUS      RESTARTS   AGE
default        gitlab-runner-runner-587447dfbc-hk2q9                             1/1     Running     0          8m11s
default        runner-8hnnqcxg-project-7-concurrent-04x5gv                       0/2     Init:0/1    0          18s
default        runner-8hnnqcxg-project-7-concurrent-19tn8m                       0/2     Init:0/1    0          18s

4. 卸载 gitlab runner operator

1. 删除 crd

   kubectl delete -f gitlab-runner.yml

2. 删除 secret

   kubectl delete -f gitlab-runner-secret.yml

3. 删除 Operator subscription

   kubectl delete subscription my-gitlab-runner-operator -n operators

4. 查看 clusterserviceversion

   $ kubectl get clusterserviceversion -n operators
   NAME                            DISPLAY         VERSION   REPLACES                        PHASE
   gitlab-runner-operator.v1.8.0   GitLab Runner   1.8.0     gitlab-runner-operator.v1.7.0   Succeeded

5. 删除 clusterserviceversion

   kubectl delete clusterserviceversion gitlab-runner-operator.v1.8.0 -n operators

5. 配置 gitlab runner operator

参考：Configuring GitLab Runner on OpenShift | GitLab

5.1 支持的配置

配置项	Operator版本	描述
gitlabUrl	all	GitLab 示例地址，例如：https://gitlab.example.com
token	all	Secret 名称，其中包含 runner-registration-token
tags	all	设置 runner tags
concurrent	all	任务并发数设置。0 无限制，默认 10
interval	all	interval 设置，默认 30
locked	1.8	是否锁定，默认 false
runUntagged	1.8	是否能运行 untags 的 job，如果未定义 tags 则默认 true，否则 false
protected	1.8	是否只运行保护分支 job，默认 false
cloneURL	all	覆盖 gitlabUrl 。 在 runner 不能连接 gitlabUrl 时使用
env	all	ConfigMap 名称，其中包含的 key-value 键值对将在 pod 作为环境变量
runnerImage	1.7	设置 gitlab runner 镜像。默认是 operator 版本绑定的镜像
helperImage	all	设置 GitLab Runner helper 默认镜像
buildImage	all	设置 build job 默认镜像
cacheType	all	设置 cache 类型，可选项：gcs, s3, azure
cachePath	all	设置 cache 目录
cacheShared	all	设置 cache shared 模式
s3	all	S3 cache 设置。 关联Cache properties
gcs	all	GCS cache 设置。关联Cache properties
azure	all	Azure cache设置。 关联Cache properties
ca	all	TLS secret 名称，其中包括自签 CA 证书
serviceAccount	all	Runner pod serviceAccount 设置
config	all	configmap名称，其中包含 configuration template.

5.2 设置 HTTP_PROXY 环境变量

1. 新增 custom-env.yml

   apiVersion: v1
   data:
     NO_PROXY: 172.21.0.1
     HTTP_PROXY: example.com
   kind: ConfigMap
   metadata:
     name: custom-env

2. 应用

   kubectl apply -f custom-env.yaml

3. 更新 gitlab-runner.yml

   apiVersion: apps.gitlab.com/v1beta2
   kind: Runner
   metadata:
     name: dev
   spec:
     gitlabUrl: https://gitlab.example.com
     token: gitlab-runner-secret
     env: custom-env

5.3 自定义 config.toml

配置参考：Registering runners | GitLab

1. 新增模板文件 custom-config.toml

   [[runners]]
     [runners.kubernetes]
       [runners.kubernetes.volumes]
         [[runners.kubernetes.volumes.empty_dir]]
           name = "empty-dir"
           mount_path = "/path/to/empty_dir"
           medium = "Memory"

2. 创建 ConfigMap

   kubectl create configmap custom-config-toml --from-file config.toml=custom-config.toml

3. 更新 gitlab-runner.yml

   apiVersion: apps.gitlab.com/v1beta2
   kind: Runner
   metadata:
     name: dev
   spec:
     gitlabUrl: https://gitlab.example.com
     token: gitlab-runner-secret
     config: custom-config-toml

4. 查看配置

   $ kubectl exec -it gitlab-runner-runner-5ff5b95967-z9nbp -- cat /home/gitlab-runner/.gitlab-runner/config.toml
   ...
   ...
   ...
     executor = "kubernetes"
     [runners.custom_build_dir]
     [runners.cache]
       [runners.cache.s3]
       [runners.cache.gcs]
       [runners.cache.azure]
     [runners.kubernetes]
       host = ""
       bearer_token_overwrite_allowed = false
       image = "alpine"
       namespace = "default"
       namespace_overwrite_allowed = ""
       helper_image = "registry.gitlab.com/gitlab-org/ci-cd/gitlab-runner-ubi-images/gitlab-runner-helper-ocp:x86_64-v14.10.0"
       poll_timeout = 180
       service_account_overwrite_allowed = ""
       pod_annotations_overwrite_allowed = ""
       [runners.kubernetes.affinity]
       [runners.kubernetes.pod_security_context]
       [runners.kubernetes.build_container_security_context]
         [runners.kubernetes.build_container_security_context.capabilities]
       [runners.kubernetes.helper_container_security_context]
         [runners.kubernetes.helper_container_security_context.capabilities]
       [runners.kubernetes.service_container_security_context]
         [runners.kubernetes.service_container_security_context.capabilities]
       [runners.kubernetes.volumes]
   
         [[runners.kubernetes.volumes.empty_dir]]
           name = "empty-dir"
           mount_path = "/path/to/empty_dir"
           medium = "Memory"
       [runners.kubernetes.dns_config]
       [runners.kubernetes.container_lifecycle]

5.4 配置自签 TLS 证书

1. 创建包含 ca 证书的 secret：custom-tls-ca-secret.yml

   apiVersion: v1
   kind: Secret
   metadata:
     name: custom-tls-ca
   type: Opaque
   stringData:
     tls.crt: |
     -----BEGIN CERTIFICATE-----
     MIIEczCCA1ugAwIBAgIBADANBgkqhkiG9w0BAQQFAD..AkGA1UEBhMCR0Ix
     .....
     7vQMfXdGsRrXNGRGnX+vWDZ3/zWI0joDtCkNnqEpVn..HoX
     -----END CERTIFICATE-----

2. 应用

   kubectl apply -f custom-tls-ca-secret.yaml

3. 更新 gitlab-runner.yml

   apiVersion: apps.gitlab.com/v1beta2
   kind: Runner
   metadata:
     name: dev
   spec:
     gitlabUrl: https://gitlab.example.com
     token: gitlab-runner-secret
     ca: custom-tls-ca

5.5 配置 cpu 和 mem 限制

可以通过 config.toml 配置 cpu limit 与 mem limit ，具体参考 5.3 章节