K8S 10: Setup Gitlab Self Hosted on GKE Cluster

Tạo Gitlab với yêu cầu là: disable MinIO, dùng GCS. Disable cert-manager, dùng cert-manager mình tự tạo riêng. Các config về resource của gitlab ở mức minimal. Gitlab sẽ vào qua link `gitlab.your-subdomain.your-domain.net` và có HTTPS

Posted on by hoangmnsd 5 mins read
Table of Contents

Tạo Gitlab với yêu cầu là:

  • disable MinIO, dùng GCS,
  • disable cert-manager, dùng cert-manager mình tự tạo riêng,
  • các config về resource của gitlab ở mức minimal,
  • gitlab sẽ vào qua link gitlab.your-subdomain.your-domain.net và có HTTPS

Yêu Cầu

Cách Làm

1. Setup environment variables

export PROJECT_ID="YOUR_PROJECT_ID"
export CLUSTER_NAME="YOUR_CLUSTER_NAME"
export SUBDOMAIN="your-subdomain.your-domain.net"
gcloud config set project ${PROJECT_ID}
gcloud config set compute/region asia-northeast1
gcloud config set compute/zone asia-northeast1-a

2. Tạo namespace gitlab và add helm repo

kubectl create namespace gitlab
helm repo add gitlab https://charts.gitlab.io/
helm repo update

3. Dùng Kubed để synchronize TLS secret giữa các namespaces

helm repo add appscode https://charts.appscode.com/stable/
helm repo update
helm install -n kubed appscode/kubed --version 0.12.0 --namespace kube-system --wait --set config.clusterName=${CLUSTER_NAME}

Giả sử nhờ bài trước mà mình đã issue được 1 TLS Secret thành công tên là echo-tls-secret-prod ở namespace default,

k get secret -A
default           default-token-7hsrc                              kubernetes.io/service-account-token   3      3h11m
default           echo-tls-secret-prod                             kubernetes.io/tls                     3      157m
default           echo-tls-secret-staging                          kubernetes.io/tls                     3      164m
gitlab            default-token-t728z                              kubernetes.io/service-account-token   3      99s

giờ mình cần sync secret đó sang namespace gitlab khác thì cần làm như sau:
annotate secret mà mình muốn sync giữa các namespaces

# Đánh label `app=kubed` cho namespace gitlab
kubectl label namespace gitlab app=kubed
# Annotate để secret mình muốn sẽ sync sang các namespace có label `app=kubed` 
kubectl annotate secret echo-tls-secret-prod -n default kubed.appscode.com/sync="app=kubed"

check: (Bạn sẽ thấy namespace gitlab cũng có secret echo-tls-secret-prod)

k get secret -A
default           default-token-7hsrc                              kubernetes.io/service-account-token   3      3h15m
default           echo-tls-secret-prod                             kubernetes.io/tls                     3      161m
default           echo-tls-secret-staging                          kubernetes.io/tls                     3      168m
gitlab            default-token-t728z                              kubernetes.io/service-account-token   3      5m43s
gitlab            echo-tls-secret-prod                             kubernetes.io/tls                     3      6s

4. GCP - Tạo Service Account

gcloud iam service-accounts create gitlab-gcs \
  --display-name "Gitlab Cloud Storage"

5. GCP - Tạo Service Account Key

gcloud iam service-accounts keys create --iam-account \
  gitlab-gcs@${PROJECT_ID}.iam.gserviceaccount.com /tmp/gitlab-gcs-key.json

6. GCP - Binding role Storage Admin vào Service Account

gcloud projects add-iam-policy-binding --role roles/storage.admin ${PROJECT_ID} \
  --member=serviceAccount:gitlab-gcs@${PROJECT_ID}.iam.gserviceaccount.com

7. GCP - Create GCS Buckets

gsutil mb -c standard gs://${PROJECT_ID}-gitlab-uploads
gsutil mb -c standard gs://${PROJECT_ID}-gitlab-artifacts
gsutil mb -c standard gs://${PROJECT_ID}-gitlab-lfs
gsutil mb -c standard gs://${PROJECT_ID}-gitlab-packages
gsutil mb -c standard gs://${PROJECT_ID}-gitlab-registry
gsutil mb -c standard gs://${PROJECT_ID}-gitlab-runner-cache
gsutil mb -c nearline gs://${PROJECT_ID}-gitlab-backups

8. Create a Secret to hold your Cloud Storage credentials

kubectl create secret generic google-application-credentials \
  --from-file=gcs-application-credentials-file=/tmp/gitlab-gcs-key.json \
  --namespace gitlab

9. Tạo file rails.yaml

cat > /tmp/rails.yaml <<EOF
provider: Google
google_project: ${PROJECT_ID}
google_client_email: gitlab-gcs@${PROJECT_ID}.iam.gserviceaccount.com
google_json_key_string: '$(cat /tmp/gitlab-gcs-key.json)'
EOF

10. Tạo secret từ file rails.yaml

kubectl create secret generic gitlab-rails-storage --from-file=connection=/tmp/rails.yaml --namespace gitlab

11. (Optional) Xóa key đi cho secure

rm -rf /tmp/gitlab-gcs-key.json
rm -rf /tmp/rails.yaml

12. Tạo Gitlab helm values file

cat > ./gitlab-values.yaml <<EOF
# Values for gitlab/gitlab chart on GKE
global:
  edition: ce

  ## doc/charts/globals.md#configure-ingress-settings
  ingress:
    configureCertmanager: false
    tls:
      enabled: true
      secretName: echo-tls-secret-prod

  hosts:
    domain: ${SUBDOMAIN}
    https: false
    gitlab:
      https: false
    registry:
      https: false
    minio:
      https: false

  ## doc/charts/globals.md#configure-minio-settings
  minio:
    enabled: false

  registry:
    bucket: ${PROJECT_ID}-gitlab-registry

  ## doc/charts/globals.md#configure-appconfig-settings
  ## Rails based portions of this chart share many settings
  appConfig:
    ## doc/charts/globals.md#general-application-settings
    enableUsagePing: false

    defaultTheme: 2

    defaultProjectsFeatures:
      issues: true
      mergeRequests: true
      wiki: true
      snippets: true
      builds: true
      containerRegistry: true

    ## doc/charts/globals.md#lfs-artifacts-uploads-packages
    backups:
      bucket: ${PROJECT_ID}-gitlab-backups
    lfs:
      bucket: ${PROJECT_ID}-gitlab-lfs
      connection:
        secret: gitlab-rails-storage
        key: connection
    artifacts:
      bucket: ${PROJECT_ID}-gitlab-artifacts
      connection:
        secret: gitlab-rails-storage
        key: connection
    uploads:
      bucket: ${PROJECT_ID}-gitlab-uploads
      connection:
        secret: gitlab-rails-storage
        key: connection
    packages:
      bucket: ${PROJECT_ID}-gitlab-packages
      connection:
        secret: gitlab-rails-storage
        key: connection

gitlab:  
  webservice:
    minReplicas: 1
    resources:
      limits:
       memory: 1.5G
      requests:
        cpu: 100m
        memory: 900M
    workhorse:
      resources:
        limits:
          memory: 100M
        requests:
          cpu: 10m
          memory: 10M
  task-runner:
    backups:
      objectStorage:
        backend: gcs
        config:
          secret: "google-application-credentials"
          key: gcs-application-credentials-file
          gcpProject: ${PROJECT_ID}
  sidekiq:
    minReplicas: 1
    resources:
      limits:
        memory: 1.5G
      requests:
        cpu: 50m
        memory: 625M
  gitlab-shell:
    minReplicas: 1

nginx-ingress:
  controller:
    replicaCount: 1
    minAvailable: 0
    resources:
      requests:
        cpu: 50m
        memory: 100Mi
  defaultBackend:
    replicaCount: 1
    minAvailable: 0
    resources:
      requests:
        cpu: 5m
        memory: 5Mi

redis:
  resources:
    requests:
      cpu: 10m
      memory: 64Mi

certmanager:
  install: false

prometheus:
  install: false

registry:
  enabled: true
  certificate:
    secret: gitlab-rails-storage
    key: connection
  ingress:
    tls:
      enabled: true
      secretName: echo-tls-secret-prod

gitlab-runner:
  install: false
  rbac:
    create: true
  runners:
    locked: false
    cache:
      cacheType: gcs
      gcsBucketname: ${PROJECT_ID}-gitlab-runner-cache
      secretName: gitlab-rails-storage
      cacheShared: true
EOF

13. Install Gitlab helm chart

helm upgrade --install gitlab gitlab/gitlab \
  --namespace gitlab \
  --values gitlab-values.yaml

Check gitlab trên link gitlab.your-subdomain.your-domain.net được như sau là OK:

Đăng nhập vào gitlab bằng user “root”, pass lấy từ command sau:

kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath={.data.password} -n gitlab | base64 --decode ; echo

Check xem có thể Backup và Restore Gitlab bình thường ko là OK:

kubectl exec gitlab-task-runner-7d686c6474-w8mnt -it backup-utility -n gitlab
WARNING: This version of GitLab depends on gitlab-shell 11.0.0, but you're running Unknown. Please update gitlab-shell.
2020-02-21 09:26:19 +0000 -- Dumping database ... 
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2020-02-21 09:26:22 +0000 -- done
WARNING: This version of GitLab depends on gitlab-shell 11.0.0, but you're running Unknown. Please update gitlab-shell.
2020-02-21 09:26:32 +0000 -- Dumping repositories ...
2020-02-21 09:26:32 +0000 -- done
Dumping registry ...
empty
Dumping uploads ...
empty
Dumping artifacts ...
empty
Dumping lfs ...
empty
Dumping packages ...
empty
WARNING: This version of GitLab depends on gitlab-shell 11.0.0, but you're running Unknown. Please update gitlab-shell.
Packing up backup tar
Copying file:///srv/gitlab/tmp/backup_tars/1582277165_2020_02_21_12.7.6_gitlab_backup.tar [Content-Type=application/x-tar]...
/ [1 files][150.0 KiB/150.0 KiB]
Operation completed over 1 objects/150.0 KiB.
[DONE] Backup can be found at gs://xxx-gitlab-backups/1582277165_2020_02_21_12.7.6_gitlab_backup.tar

Nếu backup bị lỗi đỏ hoặc có lỗi kiểu Bucket not found: registry. Skipping backup of registry thì có nghĩa là registry bucket không nhìn thấy, cần check lại file values nhé

DONE! 🎉🎉🎉

14. Clean up

  • Nên chắc chắn rằng bạn đã xóa Cluster, VPC
  • Vào Service Cloud DNS, xóa các record được sinh ra bởi ExternalDNS
  • Vào IAM - Service Account xóa service account gitlab mà bạn đã tạo
  • Vào Compute Engine - Disk xóa các disk được tạo bởi Cluster
  • Vào Storage, xóa các bucket *-gitlab-* bạn đã tạo ra

CREDIT

https://docs.gitlab.com/charts/ https://docs.gitlab.com/charts/charts/globals.html#connection https://ruzickap.github.io/k8s-harbor/part-03/#install-nginx-ingress https://gitlab.com/gitlab-org/charts/gitlab/issues/1464 https://docs.gitlab.com/charts/backup-restore/backup.html

Thank You!

Your comment has been submitted. It will appear on this page shortly! OK

Yikes, Sorry!

Error occured. Couldn't submit your comment. Please try again. Thank You! OK

Leave a comment