Multi-Cloud Infrastructure

Comprehensive documentation of 31 VMs across 4 cloud accounts

31
Total VMs
4
Cloud Accounts
26
Tools/Services
HA
All Critical

Complete Global Architecture

End-to-end infrastructure overview showing all 31 VMs across 4 cloud accounts, networking, storage, and service connectivity

Complete Global Infrastructure Architecture

Tailscale VPN Mesh Network

Tailscale VPN Mesh Network Architecture

How It Works

  • Installation: Tailscale daemon on all 31 VMs
  • Authentication: OAuth/SSO (no passwords)
  • Key Exchange: Secure WireGuard key distribution
  • P2P Encryption: Direct encrypted connections
  • MagicDNS: ssh hostname (e.g., ssh master-01)
  • Network: 100.64.0.0/10 private address space

Benefits

  • ✅ 28 VMs completely private (no public IPs)
  • ✅ Encrypted P2P between all nodes
  • ✅ NAT traversal (works across cloud providers)
  • ✅ No firewall rules needed
  • ✅ Zero-trust security model
  • ✅ Easy service discovery

Load Balancer Cluster

Load Balancer Cluster Architecture

HAProxy + Keepalived

UP
Location: DigitalOcean Account 1
VMs: 2 nodes (MASTER + BACKUP)
Version: HAProxy
Floating IP: VRRP VIP
Failover: Automatic sub-5s
Status: ✅ Active

LB-01 (Master)

RUNNING
Public IP: Primary node
Private IP: Tailscale VPN
Resources: Dedicated compute
Memory: Sufficient for HAProxy
Role: VRRP Master
Health Checks: Active monitoring

LB-02 (Backup)

RUNNING
Public IP: Backup node
Private IP: Tailscale VPN
Resources: Dedicated compute
Memory: Sufficient for HAProxy
Role: VRRP Backup
Health Checks: Active monitoring

Kubernetes K3s Cluster

3 Master Nodes

UP
Version: K3s Latest
Location: DO Account 1
Specs: Optimized for HA
etcd Quorum: 2/3 Consensus
Control Plane:
  • • master-01
  • • master-02
  • • master-03

5 Worker Nodes

READY
Status: Ready
Specs: Optimized for workloads
Total Pods: Running
Ingress: Nginx Controller Active
Nodes:
  • • worker-01 to worker-05

Core Components

ACTIVE
Storage: NFS-based persistent volumes
Namespaces:
  • • argocd
  • • ingress-nginx
  • • velero
  • • kube-system
Apps: ArgoCD, Velero
K3s Architecture

PostgreSQL HA Cluster

PostgreSQL + Patroni

UP
Location: DO Account 2
Nodes: 3 (1 Primary + 2 Standby)
Replication: Streaming (sync/async)
Auto-Failover: Enabled via Patroni
Backups: Daily automated

Primary Node (pg-01)

MASTER
Public IP: Primary database node
Specs: Optimized for database
Role: Primary (Writes)
Replication Lag: Minimal
Status: ✅ Accepting connections

Standby Nodes

REPLICAS
Standby 1: Replica node
Standby 2: Replica node
Specs: Optimized for replication
Role: Standby (Reads only)
Sync Mode: Synchronous
Status: ✅ Both in sync
PostgreSQL Architecture

Redis Sentinel Cluster

Redis Sentinel

UP
Version: Redis with Sentinel
Location: DO Account 3
Nodes: 3 (1 Master + 2 Replicas)
Quorum: 2/3
Auto-Failover: Enabled

Master + Sentinel

RUNNING
Node: Master node
Specs: Optimized for cache
Role: Master (Writes)
Sentinel Port: Active monitoring
Status: ✅ Serving traffic

Replicas + Sentinel

SYNCED
Replica 1: Replica node
Replica 2: Replica node
Specs: Optimized for replication
Role: Replicas (Read-only)
Replication Lag: Minimal
Status: ✅ Both in sync
Redis Architecture

MinIO Erasure Coding

MinIO Cluster

UP
Location: DO Account 3
Nodes: 3 (Distributed cluster)
Erasure Coding: EC:2
Durability: Survives 1 node failure
URLs: Console + API HTTPS

3 Data Nodes

HEALTHY
minio-01: Data node
minio-02: Data node
minio-03: Data node
Tailscale IPs: Private network
Health: 3/3 UP ✅
Drive Status: All healthy

Access & Backups

CONFIGURED
Console: https://minio-console.yhakkache.tech
API: https://minio.yhakkache.tech
Admin Creds: Configured
Velero Backups: K3s integrated
GitLab Backups: Automated
Data: Multiple buckets active
MinIO Architecture

Monitoring & Observability

Prometheus + Grafana

UP
Location: DO Account 3
Specs: Optimized for metrics
Prometheus: Targets monitoring
Grafana URL: https://grafana.[DOMAIN]
Prometheus URL: https://prometheus.[DOMAIN]
Retention: Long-term storage

Loki + Jaeger

UP
Location: GCP
Specs: Optimized for logging
Loki URL: https://loki.[DOMAIN]
Jaeger URL: https://jaeger.[DOMAIN]
Log Agents: Deployed on all VMs
Tracing: Distributed tracing enabled

Exporter Coverage

COMPLETE
Node Exporter: 31/31 VMs
HAProxy Exporter: 2/2 nodes
PostgreSQL Exporter: 3/3 nodes
Redis Exporter: 3/3 nodes
K3s Metrics: All components
Custom Dashboards: 15+ dashboards
Monitoring Architecture

CI/CD Pipeline

GitLab CE

UP
Location: GCP
Specs: Optimized for Git services
URL: https://gitlab.[DOMAIN]
Features: Git, CI/CD, Issues, Wiki
Runners: K3s cluster integrated
Status: ✅ Online

Harbor Registry

UP
Location: GCP
Specs: Optimized for image registry
URL: https://harbor.[DOMAIN]
Features: Docker/OCI registry + scanning
Containers: All healthy
Status: ✅ Online

SonarQube

UP
Location: GCP
Specs: Optimized for code analysis
URL: https://sonarqube.[DOMAIN]
Features: Code quality + SAST
Access: TLS via HAProxy
Status: ✅ Online

Vault HA Secrets Management

Vault Cluster

SEALED
Location: DO Account 3
Nodes: 2 (HA Raft consensus)
URL: https://vault.[DOMAIN]
Consensus: Raft (etcd replacement)
Status: ⚠️ Sealed (needs unseal)

vault-01 (Primary)

UP
Private IP: Primary vault node
Role: Primary Leader (Raft)
Storage: Raft-integrated backend
Restart Policy: always
Status: Container running

vault-02 (Standby)

UP
Private IP: Standby vault node
Role: Standby Replica
Replication: Real-time sync
Auto-Failover: Enabled
Status: Waiting for unsealing

Features

  • ✓ Secrets management (passwords, API keys)
  • ✓ Encryption as a service
  • ✓ Dynamic secrets generation
  • ✓ Audit logging
  • ✓ HA with Raft consensus
  • ✓ Shamir secret sharing (3 keys, threshold 2)

Status

  • Current: SEALED ⚠️
  • Requires: 2 of 3 unseal keys
  • HA: Both nodes UP ✅
  • Network: Accessible via Tailscale
  • Storage: Raft backend healthy
  • Action: Unseal via UI or CLI

Velero Backup & Recovery

Velero Deployment

UP
Location: K3s Cluster (velero namespace)
Operator: Velero controller deployed
Backend Storage: MinIO (S3-compatible)
Bucket: velero-backups
Schedule: Daily automatic backups
Status: ✅ Active and monitoring

Backup Configuration

CONFIGURED
Backup Scope: All namespaces
PVC Storage: Snapshots included
Schedule: Daily backups
Retention: Long-term storage
Compression: Enabled
Storage: MinIO (3-node cluster)

Recovery Options

AVAILABLE
Full Cluster: Complete K3s restore
Selective: Specific namespaces
Applications: GitLab, Harbor, etc.
Data: NFS, MinIO buckets
RTO: < 15 minutes
RPO: Daily backups

Backup Strategy

Incremental Backups

  • • Daily snapshots (K3s cluster)
  • • PVC data included
  • • Stored in MinIO (EC:2)
  • • Distributed across nodes

Data Protection

  • • MinIO Erasure Coding (survives 1 node failure)
  • • Backup servers (redundant copies)
  • • HTTPS encryption in transit
  • • Access control via Vault

ArgoCD GitOps Deployment

ArgoCD Deployment

UP
Location: K3s Cluster (argocd namespace)
Operator: ArgoCD controller deployed
Git Repository: GitLab CE
Sync Policy: Auto-sync enabled
Health Check: Continuous
Status: ✅ Monitoring Git repos

Application Management

CONFIGURED
Sync Interval: Automatic detection
Deployment: Automatic on Git push
Rollback: Single-click revert
Multi-cluster: Single ArgoCD instance
Applications: Monitoring, CI/CD tools

Features & Benefits

ACTIVE
✓ Declarative Git-driven deployments
✓ Real-time application sync status
✓ RBAC integration with GitLab
✓ Webhook-based notifications
✓ Multi-environment support
✓ Application health monitoring

Supported Application Types

  • • Helm Charts
  • • Kustomize overlays
  • • Jsonnet
  • • Plain Kubernetes manifests
  • • Custom plugins
  • • ArgoCD AppSets (multi-repo)

GitOps Workflow

  • 1. Push code to GitLab
  • 2. GitLab CI builds Docker image
  • 3. Harbor registry stores image
  • 4. Update K3s manifests in Git
  • 5. ArgoCD detects change
  • 6. Auto-sync to K3s cluster

Infrastructure Statistics

CLOUD DISTRIBUTION
DO Account 1 10 VMs
DO Account 2 3 VMs
DO Account 3 10 VMs
GCP Account 8 VMs
SERVICE STATUS
UP Services 29/31
Operational 95%
HA Components 7/7
SSL Domains 13/13
NETWORKING
Tailscale Nodes 31
Public VMs 3
Private VMs 28
Failover Time <5s

Complete Tech Stack (26 Tools)

DATA

  • ✓ PostgreSQL 16
  • ✓ Redis 7.x
  • ✓ MinIO (S3)
  • ✓ Patroni (HA)
  • ✓ Sentinel
  • ✓ NFS v4

ORCHESTRATION

  • ✓ K3s v1.34.3
  • ✓ Traefik
  • ✓ etcd
  • ✓ HAProxy 2.4
  • ✓ Keepalived
  • ✓ Nginx Ingress

MONITORING

  • ✓ Prometheus
  • ✓ Grafana
  • ✓ Loki
  • ✓ Jaeger
  • ✓ Node Exporter
  • ✓ Promtail

SECURITY

  • ✓ Vault (HA)
  • ✓ Tailscale VPN
  • ✓ Let's Encrypt
  • ✓ cert-manager
  • ✓ Bastion Host
  • ✓ WireGuard

CI/CD

  • ✓ GitLab CE
  • ✓ Harbor Registry
  • ✓ SonarQube
  • ✓ ArgoCD
  • ✓ Docker
  • ✓ Velero

NETWORKING

  • ✓ Nginx
  • ✓ HAProxy
  • ✓ Keepalived
  • ✓ Tailscale
  • ✓ VRRP
  • ✓ Floating IP

Access & Credentials

SSH Access

# Via Bastion
ssh [BASTION_USER]@[BASTION_IP]
# Direct (Tailscale)
ssh hostname # e.g., ssh master-01

Web Interfaces

GitLab: https://gitlab.[DOMAIN]
Harbor: https://harbor.[DOMAIN]
SonarQube: https://sonarqube.[DOMAIN]
Grafana: https://grafana.[DOMAIN]
MinIO: https://minio-console.[DOMAIN]

Kubernetes

# SSH to master and run kubectl
ssh master-01 'kubectl get nodes'
# Config location
/etc/rancher/k3s/k3s.yaml

Important URLs

Prometheus: https://prometheus.[DOMAIN]
Jaeger: https://jaeger.[DOMAIN]
HAProxy Stats: https://stats.[DOMAIN]/stats
Vault: https://vault.[DOMAIN] ( SEALED)

High Availability Components

Load Balancers (2 nodes)

Keepalived VRRP | Failover < 5s |

Kubernetes Masters (3 nodes)

etcd quorum 2/3 | API Server HA | Controller HA | Scheduler HA

PostgreSQL (3 nodes)

Patroni auto-failover | 1 Primary + 2 Standby | Streaming replication

Redis (3 nodes)

Sentinel quorum 2/3 | 1 Master + 2 Replicas | Auto-failover

MinIO (3 nodes)

Erasure Coding EC:2 | Survives 1 node failure | Distributed storage

Vault (2 nodes)

Raft consensus | Shamir 3 keys, threshold 2 | Sealed (needs unseal)

NFS (2 nodes)

rsync + inotify replication | Real-time sync | 550GB each

Backup Redundancy

2 backup servers | DO Account 3 + GCP | Multiple destinations