Hello everyone,
I need some help to configure a service to access at my proxmox cluster.
I have already access to the WebUI and the VNC console of our VMs, but when I want to upload an iso or to launch a shell console it’s going like chaos.
If I open a shell console and don’t type on my keyboard in the last 10 seconds I have a connection closed error with code 1006.
If I upload an ISO image on one of our three nodes the progress bar went to 100% but when it need to be pushed on the nodes I have an error with error code (0) Invalid document.
I don’t have this problem if I went through the WebUI behind the haproxy or direct access with IP address so the problem is like stuck on BunkerWeb.
To recapitulate my configuration :
BunkerWeb dockerized in clustering mode (One manager, two workers nodes)
HaProxy in High Availability mode with Keepalived (Two servers One VIP)
Proxmox nodes in cluster modes
To schematize the client access it’s look like this :
Client –> Bunkerweb Service –> BunkerWeb Worker –> HaProxy Bunkerweb Worker TCP mode –> HaProxy Proxmox node http mode –> Proxmox nodes in backend
I went through this problem like 2 weeks ago and I can’t solve this.
This my actual configuration :
Bunkerweb service :
IS_DRAFT=no
SERVER_NAME=cluster-proxmox.domain.tld
USE_TEMPLATE=low
SECURITY_MODE=detect
BAD_BEHAVIOR_STATUS_CODES=400 401 403 404 429 500
BAD_BEHAVIOR_THRESHOLD=30
BAD_BEHAVIOR_BAN_TIME=3600
BLACKLIST_COMMUNITY_LISTS=ip:danmeuk-tor-exit ua:mitchellkrogza-bad-user-agents ip:laurent-minne-data-shield-aggressive ip:laurent-minne-data-shield-critical
CLIENT_CACHE_ETAG=no
CORS_DENY_REQUEST=no
USE_DNSBL=no
GRPC_INTERCEPT_ERRORS=no
GZIP_PROXIED=expired no-cache no-store private auth
KEEP_UPSTREAM_HEADERS=*
REFERRER_POLICY=no-referrer-when-downgrade
AUTO_LETS_ENCRYPT=yes
EMAIL_LETS_ENCRYPT=infrasmtp@nuagis.cloud
LETS_ENCRYPT_CHALLENGE=dns
LETS_ENCRYPT_DNS_PROVIDER=infomaniak
LETS_ENCRYPT_DNS_PROPAGATION=90
USE_LETS_ENCRYPT_WILDCARD=yes
LETS_ENCRYPT_DNS_CREDENTIAL_ITEM=dns_infomaniak_token=DDDCCCXXX
USE_LIMIT_REQ=no
USE_LIMIT_CONN=no
LIMIT_CONN_MAX_HTTP1=200
LIMIT_CONN_MAX_HTTP2=500
LIMIT_CONN_MAX_HTTP3=500
LIMIT_REQ_RATE=80r/s
MAX_CLIENT_SIZE=50000m
USE_MODSECURITY=no
USE_MODSECURITY_CRS=no
USE_MODSECURITY_CRS_PLUGINS=no
USE_REAL_IP=yes
REAL_IP_FROM=192.168.XXX.XXX/XX 10.255.XXX.XXX/XX 192.168.XXX.XXX/XX 10.255.XXX.XXX/XX 192.255.XXX.XXX/XX 10.255.XXX.XXX/XX
USE_REVERSE_PROXY=yes
PROXY_NO_CACHE=no
REVERSE_PROXY_HOST=https://10.255.XXX.XXX:20443
REVERSE_PROXY_WS=yes
REVERSE_PROXY_BUFFERING=no
REVERSE_PROXY_KEEPALIVE=yes
REVERSE_PROXY_CONNECT_TIMEOUT=3600s
REVERSE_PROXY_READ_TIMEOUT=3600s
REVERSE_PROXY_SEND_TIMEOUT=3600s
REVERSE_PROXY_MODSECURITY=no
USE_ROBOTSTXT=yes
ROBOTSTXT_COMMUNITY_LISTS=ai-robots-txt
SSL_PROTOCOLS=TLSv1 TLSv1.1 TLSv1.2 TLSv1.3
SSL_CIPHERS_LEVEL=intermediate
WHITELIST_IP=192.168.XXX.XXX/XX 10.255.XXX.XXX/XX 192.168.XXX.XXX/XX 10.255.XXX.XXX/XX 192.255.XXX.XXX/XX 10.255.XXX.XXX/XX
HaProxy configuration for bunkerweb and proxmox (frontend and backend config) :
FRONTEND PROXMOX
frontend web-proxmox
# Bind VIP (Keepalived)
bind 10.255.XXX.XXX:20480
bind 10.255.XXX.XXX:20443 ssl crt /etc/haproxy/certs/nuagis.cloud.pem
maxconn 1000
option http-keep-alive
tcp-request inspect-delay 5s
# Redirect ssl
http-request return status 200 content-type text/plain lf-string "%[path,field(-1,/)].${ACCOUNT_THUMBPRINT}\n" if { path_beg '/.well-known/acme-challenge/' }
#ACL API WS et VNC
acl is_api path_beg /api2/json
acl is_vnc path_beg /api2/json/nodes
# Détection WebSocket (VNC console)
acl hdr_upgrade_ws hdr(Upgrade) -i WebSocket
acl hdr_connection_upgrade hdr(Connection) -i upgrade
# limite stricte pour web classique
http-request deny if !is_api !hdr_upgrade_ws !hdr_connection_upgrade { sc_http_req_rate(0) gt 300 }
# limite plus large pour API
http-request deny if is_api { sc_http_req_rate(0) gt 500 }
# quasi libre pour websocket (sinon VNC KO)
http-request deny if hdr_upgrade_ws hdr_connection_upgrade { sc_http_req_rate(0) gt 2000 }
# Protection slowloris
option http-buffer-request
# Headers WebSocket
http-request set-header Connection "upgrade" if hdr_upgrade_ws hdr_connection_upgrade
http-request set-header Upgrade %[req.hdr(Upgrade)] if hdr_upgrade_ws hdr_connection_upgrade
# =========================
# 🔁 REDIRECTION HTTPS
# =========================
http-request redirect scheme https unless { ssl_fc }
# Ajout IP client dans header (traçabilité)
option forwardfor
# =========================
# 🔐 HSTS (forcer HTTPS navigateur)
# =========================
http-response set-header Strict-Transport-Security max-age=63072000
# =========================
# 🌐 ROUTING FQDN
# =========================
acl cluster_proxmox hdr(host) -i cluster-proxmox.domain.tld:20443
acl cluster_proxmox hdr(host) -i cluster-proxmox.domain.tld
# Envoi vers backend si correspondance
use_backend cluster-pprod-nuagis if cluster_proxmox
BACKEND PROXMOX
backend cluster-pprod-nuagis
mode http
# Répartition de charge simple
balance leastconn
timeout tunnel 1h
timeout server 1h
option http-keep-alive
http-reuse safe
option http-server-close
option forwardfor
fullconn 6000
# =========================
# ❤️ HEALTHCHECK
# =========================
option httpchk GET /
# =========================
# 🍪 PERSISTENCE SESSION
# =========================
cookie SERVERID insert indirect nocache
# =========================
# 🔁 HEADERS PROXY
# =========================
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
# Headers WebSocket
http-request set-header Connection "upgrade" if { req.hdr(Upgrade) -i WebSocket } { req.hdr(Connection) -i upgrade }
http-request set-header Upgrade %[req.hdr(Upgrade)] if { req.hdr(Upgrade) -i WebSocket } { req.hdr(Connection) -i upgrade }
# =========================
# 🖥 NODES PROXMOX
# =========================
server domaintld-m620-pve1 172.255.XXX.XXX:8006 cookie S1 maxconn 2000 check ssl verify none
server domaintld-m620-pve2 172.255.XXX.XXX:8006 cookie S2 maxconn 2000 check ssl verify none
server domaintld-m620-pve3 172.255.XXX.XXX:8006 cookie S3 maxconn 2000 check ssl verify none
BunkerWeb Configuration
defaults
timeout connect 5s
timeout client 5s
timeout server 5s
Frontend configuration Bunkerweb
frontend bunkerweb_http_front
mode tcp
bind 10.255.XXX.XXX:30880
default_backend bunkerweb_worker_backend_http
frontend bunkerweb_https_front
mode tcp
bind 10.255.XXX.XXX:30443
default_backend bunkerweb_worker_backend_https
Backend configuration bunkerweb :
backend bunkerweb_worker_backend_http
mode tcp
balance roundrobin
server domaintld-wafworker-d-p01 192.255.XXX.XXX:40880 check send-proxy-v2
server domaintld-wafworker-d-p02 192.255.XXX.XXX:40880 check send-proxy-v2
backend bunkerweb_worker_backend_https
mode tcp
balance roundrobin
server domaintld-wafworker-d-p01 192.255.XXX.XXX:40443 check send-proxy-v2
server domaintld-wafworker-d-p02 192.255.XXX.XXX:40443 check send-proxy-v2
Let me know if you need futher information, all help will be welcome.
Best regards,
Laygon