Superset Cluster Tricks

Running experiments on the Superset Cluster project.

I built superset-cluster: a highly available Apache Superset deployment backed by a MySQL InnoDB Cluster, with no Kubernetes or any other orchestrator anywhere in the stack. This article explains how the nodes work together and includes practical validation experiments for each key mechanism, demonstrating that the advertised guarantees hold under real conditions.

Why I did it

The goal was a production-grade, highly available business intelligence tool with no application-level single point of failure and no maintenance downtime.

Kubernetes could support this architecture, but I avoided it for reasons specific to the project. The cluster is small, fixed at five nodes by default, and introducing Kubernetes would add an operational control plane with failure modes more complex than the application itself.

Architecture

Under normal operation, the primary management node owns the VRRP floating VIP. Nginx terminates HTTPS and forwards requests to Gunicorn (which serves Superset UI). Superset accesses MySQL through SQLAlchemy and the local MySQL Router. Router uses InnoDB Cluster metadata for direct writes to the current MySQL primary. Celery executes asynchronous tasks, while Redis provides application cache. MySQL Shell provides admin console that enable managing MySQL Router and InnoDB Cluster lifecycle.

InnoDB Cluster runs three MySQL replicas in single-primary mode by default. One primary and two secondaries are synchronized through Group Replication. The two management nodes provide application and routing redundancy. If the primary management node fails: Keepalived transfers the VIP to the secondary node. If the MySQL primary fails: Group Replication promotes an eligible secondary, and MySQL Router directs new connections to the new primary.

The topology therefore defines failover paths for application ingress, database routing, and MySQL primary loss. Below I recorded validation test results demonstrating the intended high-availability design.

See it Work

The topology defines failover paths for application ingress, database routing, and MySQL primary loss. Below I recorded validation test results demonstrating the intended high-availability design.

All tests were run against the superset-cluster project at commit 7e00624.

TLS everywhere

Traffic on ports 443, 3306, and between InnoDB Cluster nodes is TLS-wrapped end to end (the one exception is Keepalived’s heartbeat).

Raw test output

Test 1: User ↔ Superset - TLS handshake on the VIP, port 443, confirming no plaintext HTTP data (GET/Host/cookies) ever crosses the link

$ curl -vk https://172.18.0.10/ -o /dev/null -s 2>&1 | head -15
*   Trying 172.18.0.10:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / RSASSA-PSS
* Server certificate:
*   subject: CN=172.18.0.10
> GET / HTTP/1.1
> Host: 172.18.0.10

$ tcpdump -r /tmp/tls_redo.pcap -nn -X 'port 443 and tcp[tcpflags] & tcp-push != 0' 2>/dev/null | head -6
16:42:42.278675 IP 172.18.0.1.44332 > 172.18.0.10.443: Flags [P.], seq ..., length 1551
	0x0030:  c36e 1ce7 1603 0106 0a01 0006 0603 035a  .n.............Z
	0x0040:  2b54 a8a3 64d8 dec8 e970 454a 362a dcb6  +T..d....pEJ6*..

TLS 1.3 from the first byte (16 03 01... = Handshake/ClientHello) - no GET, Host:, or cookies visible.

Test 2: Management Nodes ↔ MySQL InnoDB Cluster - MySQL Router’s proxied connection from the mgmt node to the cluster on port 3306 upgrades to TLS via SSLRequest before any query data flows

$ docker exec node-0 docker exec mysql-mgmt sh -c 'ss -tnp' | grep 3306
ESTAB 0 0  172.18.0.10:40850  172.18.0.4:3306  users:(("mysqlrouter",pid=40,fd=76))

$ tcpdump -r /tmp/internal.pcap -nn -X 'host 172.18.0.10 and host 172.18.0.4 and port 3306 and tcp[tcpflags] & tcp-push != 0' 2>/dev/null | head -10
17:12:24.652914 IP 172.18.0.4.3306 > 172.18.0.10.37434: Flags [P.], seq ..., length 78
	0x0030:  1973 8fe8 4a00 0000 0a38 2e30 2e34 3500  .s..J....8.0.45.
	0x0070:  696e 675f 7368 6132 5f70 6173 7377 6f72  ing_sha2_passwor
	0x0080:  6400                                     d.
17:12:24.653020 IP 172.18.0.10.37434 > 172.18.0.4.3306: Flags [P.], seq 1:37, length 36
	0x0030:  5ac2 6a8d 2000 0001 85aa be19 0000 0040  Z.j............@
17:12:24.658122 IP 172.18.0.10.37434 > 172.18.0.4.3306: Flags [P.], seq 37:638, length 601
	0x0030:  5ac2 6a8d 1603 0102 5401 0002 5003 0318  Z.j.....T...P...

Plaintext MySQL greeting (8.0.45) → 36-byte SSLRequest → TLS ClientHello (16 03 01...) in the next packet.

Test 3: Between Management Nodes - the only traffic between node-0/node-1 is Keepalived’s VRRP heartbeat.

$ tcpdump -r /tmp/internal.pcap -nn -X vrrp 2>/dev/null | head -8
17:12:24.275390 IP 172.18.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20
	0x0000:  45c0 0028 1542 0000 ff70 193d ac12 0002  E..(.B...p.=....
	0x0010:  e000 0012 2133 6401 0001 cead ac12 000a  ....!3d.........
	0x0020:  0000 0000 0000 0000                      ........

tcpdump itself decodes authtype none - one-way IP-multicast heartbeat, no query/credential/session data carried.

Test 4: Between MySQL InnoDB Cluster - group replication traffic between node-2/3/4 on port 3306, confirmed TLS-wrapped with zero plaintext schema/query leakage

$ tcpdump -r /tmp/internal.pcap -nn -X 'host 172.18.0.4 and host 172.18.0.5 and port 3306 and tcp[tcpflags] & tcp-push != 0' 2>/dev/null | head -8
17:12:24.301788 IP 172.18.0.5.48702 > 172.18.0.4.3306: Flags [P.], seq ..., length 474
	0x0030:  8324 731b 1703 0301 d501 c2f5 8f13 7292  .$s...........r.
	0x0040:  97b0 4dcc b5d4 cb3d 388d ec08 821c cfd7  ..M....=8.......

$ tcpdump -r /tmp/internal.pcap -A 'port 3306' 2>/dev/null | grep -ciE 'CREATE TABLE|INSERT INTO|group_replication'
0

Connection already established (long-lived group-replication link), so all that’s visible is 17 03 03... - TLS ApplicationData carrying live GTID/binlog events. Zero plaintext matches despite constant real replication activity during the capture.

VRRP failover with `nopreempt`

When Keepalived is killed on the master, the backup takes over the VIP in 3.46s. nopreempt holds: 20s after the master recovers, the backup still holds the VIP instead of necessarily switching back to the recovered node.

Raw test output

Test 1: Failover time - kill keepalived on the MASTER (node-0) with SIGKILL and measure how fast the BACKUP (node-1) picks up the VIP

$ docker exec node-0 docker exec mysql-mgmt sh -c 'ip addr show eth0 | grep 172.18.0.10'
    inet 172.18.0.10/16 scope global secondary eth0

$ docker exec node-0 docker exec -u root mysql-mgmt kill -9 9 10
killed at t=0.00s

[poll node-1 every 0.1s for the VIP]
node-1 acquired VIP at t=3.46s

$ docker exec node-1 docker exec mysql-mgmt sh -c 'ip addr show eth0 | grep -E "inet |UP"'
    inet 172.18.0.3/16 brd 172.18.255.255 scope global eth0
    inet 172.18.0.10/16 scope global secondary eth0

$ ip neigh show 172.18.0.10
172.18.0.10 dev br-78a79621eb28 lladdr 9a:c7:24:59:09:30 REACHABLE

$ docker exec node-1 docker exec mysql-mgmt sh -c 'ip link show eth0 | grep link/ether'
    link/ether 9a:c7:24:59:09:30 brd ff:ff:ff:ff:ff:ff link-netnsid 0

$ ping -c 2 -W 1 172.18.0.10
64 bytes from 172.18.0.10: icmp_seq=1 ttl=64 time=0.061 ms
64 bytes from 172.18.0.10: icmp_seq=2 ttl=64 time=0.063 ms
2 packets transmitted, 2 received, 0% packet loss

3.46s: matching the documented ~3× advert_int. The ARP entry for the VIP resolves to node-1’s MAC (gratuitous ARP sent on takeover), and the 2-packet ping burst during the window landed with 0% loss.

Test 2: No flap-back after recovery - restart keepalived on the original higher-priority master and confirm nopreempt stops reclaiming the VIP

$ docker exec node-0 docker exec -u root mysql-mgmt sh -c \
    'ip addr del 172.18.0.10/16 dev eth0; setsid keepalived --use-file /opt/default/keepalived.conf </dev/null >/tmp/keepalived_restart.log 2>&1 & sleep 1; pgrep keepalived'
9
10

$ sleep 20   # past vrrp_startup_delay (15s) + settle time

$ docker exec node-0 docker exec mysql-mgmt sh -c 'ip addr show eth0 | grep -E "inet |UP"'
    inet 172.18.0.2/16 brd 172.18.255.255 scope global eth0

$ docker exec node-1 docker exec mysql-mgmt sh -c 'ip addr show eth0 | grep -E "inet |UP"'
    inet 172.18.0.3/16 brd 172.18.255.255 scope global eth0
    inet 172.18.0.10/16 scope global secondary eth0

$ ip neigh show 172.18.0.10
172.18.0.10 dev br-78a79621eb28 lladdr 9a:c7:24:59:09:30 STALE

$ ping -c 2 -W 1 172.18.0.10
64 bytes from 172.18.0.10: icmp_seq=1 ttl=64 time=0.054 ms
64 bytes from 172.18.0.10: icmp_seq=2 ttl=64 time=0.063 ms
2 packets transmitted, 2 received, 0% packet loss

20s after node-0 (priority 100) came back, node-1 (priority 90) still holds the VIP and still serves traffic. nopreempt means a recovering higher-priority node rejoins passively as BACKUP.

InnoDB Cluster election

Hard-stopping the primary triggers auto-election of a new PRIMARY within ~25s, with the dead node marked (MISSING) rather than silently dropped. It rejoins as SECONDARY on restart with no automatic (necessary) re-election.

Raw test output

Test 1: Cluster status before killing the primary - baseline topology, node-2 PRIMARY, node-3/node-4 SECONDARY, cluster status OK and tolerant of one failure.

$ docker exec node-0 docker exec mysql-mgmt sh -c \
    'mysqlsh --login-path=node-3 --interactive --execute="dba.getCluster(\"superset\").status();"'
{
    "clusterName": "superset",
    "defaultReplicaSet": {
        "primary": "node-2:3306",
        "ssl": "REQUIRED",
        "status": "OK",
        "statusText": "Cluster is ONLINE and can tolerate up to ONE failure.",
        "topology": {
            "node-2:3306": {"memberRole": "PRIMARY", "mode": "R/W", "status": "ONLINE"},
            "node-3:3306": {"memberRole": "SECONDARY", "mode": "R/O", "status": "ONLINE"},
            "node-4:3306": {"memberRole": "SECONDARY", "mode": "R/O", "status": "ONLINE"}
        },
        "topologyMode": "Single-Primary"
    }
}

Test 2: Cluster status after killing the primary - hard-stop node-2 entirely and confirm automatic re-election

$ docker stop node-2
node-2

[poll `dba.getCluster('superset').status()` from node-3 every 5s]
t= 20.0s  status flips to OK_NO_TOLERANCE_PARTIAL, primary field still stale ("node-2:3306")
t=~25s    primary reassigned to node-3:3306

$ docker exec node-0 docker exec mysql-mgmt sh -c \
    'mysqlsh --login-path=node-3 --interactive --execute="dba.getCluster(\"superset\").status();"'
{
    "defaultReplicaSet": {
        "primary": "node-3:3306",
        "status": "OK_NO_TOLERANCE_PARTIAL",
        "statusText": "Cluster is NOT tolerant to any failures. 1 member is not active.",
        "topology": {
            "node-2:3306": {
                "memberRole": "SECONDARY", "mode": "n/a", "status": "(MISSING)",
                "shellConnectError": "MySQL Error 2005: Could not open connection to 'node-2:3306'..."
            },
            "node-3:3306": {"memberRole": "PRIMARY", "mode": "R/W", "status": "ONLINE"},
            "node-4:3306": {"memberRole": "SECONDARY", "mode": "R/O", "status": "ONLINE"}
        }
    }
}

node-3 auto-elected new PRIMARY, node-2 shows (MISSING) with a connect error rather than being silently removed from the topology.

Test 3: Cleanup - restoring the third node and confirming it rejoins as SECONDARY

$ docker start node-2
node-2

$ docker exec --user=root node-2 /bin/bash -c 'service ssh start && service docker start'
 * Starting OpenBSD Secure Shell server sshd    [ OK ]
 * Starting Docker: docker                      [ OK ]
[nested mysql container auto-starts, group_replication: "This server was declared online within the replication group."]

$ docker exec node-0 docker exec mysql-mgmt sh -c \
    'mysqlsh --login-path=node-3 --interactive --execute="dba.getCluster(\"superset\").status();"'
{
    "defaultReplicaSet": {
        "primary": "node-3:3306",
        "status": "OK",
        "statusText": "Cluster is ONLINE and can tolerate up to ONE failure.",
        "topology": {
            "node-2:3306": {"memberRole": "SECONDARY", "mode": "R/O", "status": "ONLINE"},
            "node-3:3306": {"memberRole": "PRIMARY", "mode": "R/W", "status": "ONLINE"},
            "node-4:3306": {"memberRole": "SECONDARY", "mode": "R/O", "status": "ONLINE"}
        }
    }
}

node-2 rejoined on its own (recovery channel config survived on disk) as SECONDARY, not PRIMARY - node-3 keeps the "R/W" role until it is the one that fails.

Blocking potentially dangerous commands via seccomp

The seccomp profile on the MySQL containers removes the fatal syscalls from the allowed set, even for root.

Raw test output

Test 1: Try to signal processes from inside the container - every kill signal number rejected identically at the syscall boundary, even for root.

$ docker exec node-2 docker exec mysql sh -c 'kill -9 7; echo exit=$?'
sh: 1: kill: Operation not permitted
exit=1

$ docker exec node-2 docker exec mysql sh -c 'kill -0 7; echo exit=$?'
sh: 1: kill: Operation not permitted
exit=1

$ docker exec node-2 docker exec mysql sh -c 'kill -TERM 7; echo exit=$?'
sh: 1: kill: Operation not permitted
exit=1

$ docker exec node-2 docker exec --user root mysql sh -c 'kill -9 7; echo exit=$?'
sh: 1: kill: Operation not permitted
exit=1

$ docker exec node-2 docker exec --user root mysql sh -c 'kill -9 1; echo exit=$?'
sh: 1: kill: Operation not permitted
exit=1

The filter blocks the syscalls itself, not particular signals (no kill signals combinations go through). Same EPERM for root as for the unprivileged user - the block happens in the kernel’s seccomp filter, beneath any in-container privilege level. mysqld (pid 7) and pid 1 remained alive and unchanged after every kill attempt above, and the container stayed healthy throughout.

Test 2: External supervision from the host still works (a different code path than in-container kill(2)), confirming the container is still manageable from the host.

$ docker exec node-2 docker kill -s HUP mysql
mysql

$ docker exec node-2 docker ps --filter name=mysql --format '{{.Names}}\t{{.Status}}'
mysql	Up 37 minutes (healthy)

If MySQL fails internally, the container should become unhealthy rather than restart repeatedly and risk entering a CrashLoopBackOff state.

Swarm secrets never touch disk

Redacted secrets are mounted as read-only tmpfs and never show up in docker inspect or docker logs.

Raw test output

Test 1: Confirming the two claims - secrets are mounted as ro tmpfs, and none appear in docker inspect

$ docker exec node-0 docker exec superset.1.y4j2s00c8cle629wjtrxihxjd mount | grep -i secret
tmpfs on /run/secrets/superset_secret_key type tmpfs (ro,relatime,inode64)
tmpfs on /run/secrets/mysql_superset_password type tmpfs (ro,relatime,inode64)

$ docker exec node-0 docker inspect superset.1.y4j2s00c8cle629wjtrxihxjd --format '{{.Config.Env}}'
[VIRTUAL_IP_ADDRESS=172.18.0.10 PATH=... LANG=C.UTF-8 GPG_KEY=... PYTHON_VERSION=3.10.14 ...
 SUPERSET_ENV=production FLASK_APP=superset.app:create_app() PYTHONPATH=/app/pythonpath
 SUPERSET_HOME=/app/superset_home SUPERSET_PORT=8088 SUPERSET_CONFIG_PATH=/app/superset_config.py]

Secrets are mounted read-only from tmpfs, not bind-mounted from disk, and never show up among the container’s docker inspect.

Test 2: Database password is redacted before it reaches any log driver

$ docker exec node-0 docker logs <container> 2>&1 | grep -i 'mysql+mysqlconnector\|SQLAlchemy URI|<mysql password>'
SQLAlchemy URI:
mysql+mysqlconnector://superset:<redacted>@172.18.0.10:6446/superset

Password appears only as the literal string <redacted> in the connection URI logged at startup, the real credential never reaches the log driver. The Superset cluster’s own password is not checked here, since it is a fixed "cluster" default value.

MySQL root password: permission tightening and a zero-trace credential handoff

The password is world-readable only during the ~150-second MySQL initialization, then locked to 400 root:root. Wrapper scripts transmit it via a dedicated PTY, keeping it out of arguments, history, and logs, and subsequently store it in an obfuscated format.

Raw test output

Test 1: The permission transition, 444 → 400 - the root password file is world-readable only for the ~150s window mysqld needs to init, then locked down

$ docker exec node-2 docker exec mysql stat -c '%a %U:%G' /var/run/mysqld/mysql_root_password
400 root:root                                    # steady state, pre-restart

$ docker exec node-2 docker restart mysql
mysql

t=10s   container-side: 444 mysql:mysql   |   node-2 host-side: 444 superset:superset
t=31s   container-side: 444 mysql:mysql   |   node-2 host-side: 444 superset:superset
t=91s   container-side: 444 mysql:mysql   |   node-2 host-side: 444 superset:superset
t=140s  container-side: 444 mysql:mysql   |   node-2 host-side: 444 superset:superset
t=155s  container-side: 444 mysql:mysql   |   node-2 host-side: 444 superset:superset
t=166s  container-side: 400 root:root     |   node-2 host-side: 400 root:root

Transition happens at t=155s→166s, matching the 150s HEALTHCHECK_START_PERIOD. Inode number identical on both sides throughout (real bind mount, not a container-local copy).

Test 2: Confirming credential distribution leaves no password traces in logs, shell history, or process arguments

$ <run store_credentials.exp, polling /proc/[pid]/comm every 50ms + tailing docker logs>
AAAAAB4XBgENHAQGCxYEHAMLHRUPEAMBEAAAAIZlUzklRHD70eFQH05oYiYQAAAAvoUuMVuMAtaH...

$ <100x /proc poll @ 50ms during the run for mysql_config_editor>
(no matching process captured - the interactive exchange is sub-50ms)

$ docker exec node-2 docker logs mysql --since 2m
(empty)

$ docker exec node-2 docker exec --user root mysql sh -c 'find / -iname "*history*" -not -path "/proc/*" -not -path "/sys/*"'
(no .bash_history, no .mysql_history anywhere)

Password never appears in logs, history, or process args - only output is an obfuscated (not encrypted) .mylogin.cnf base64 blob, decodable offline via MySQL’s public, keyless obfuscation format.

HTTP security headers, hidden server tokens

Nginx heads in front of Flask-Talisman (Apache Superset). Each required security header should be present, and appear exactly once on the superset-cluster VIP, on every response type.

Raw test output

Test 1: Confirming the expected security headers appear on the superset-cluster VIP, with correct values, on both response types

$ curl -Ik https://172.18.0.10/
HTTP/1.1 302 FOUND
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Referrer-Policy: strict-origin-when-cross-origin

$ curl -Ik https://172.18.0.10/this-path-does-not-exist-xyz     # 404
HTTP/1.1 404 NOT FOUND
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Referrer-Policy: strict-origin-when-cross-origin

$ docker exec node-0 docker exec <container> grep -n 'proxy_hide_header\|add_header' /etc/nginx/nginx.conf
24:        proxy_hide_header Strict-Transport-Security;
25:        proxy_hide_header X-Content-Type-Options;
26:        proxy_hide_header X-Frame-Options;
27:        proxy_hide_header X-XSS-Protection;
28:        proxy_hide_header Referrer-Policy;
30:        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
31:        add_header X-Content-Type-Options nosniff always;
32:        add_header X-Frame-Options DENY always;
33:        add_header X-XSS-Protection "1; mode=block" always;
34:        add_header Referrer-Policy "strict-origin-when-cross-origin" always;

HSTS, X-Content-Type-Options, X-Frame-Options, X-XSS-Protection, and Referrer-Policy are all present, each with the intended value on both the 302 and 404 paths - preload survives, X-Frame-Options is unambiguous, and Referrer-Policy uses the strict setting.

Encrypted overlay network (VXLAN + IPsec)

Redis and Superset are always located on the same node, so their traffic stays on the local bridge and never enters the IPsec path. A temporary two-node Swarm confirms the mechanism works correctly: traffic appears as ESP-encrypted ciphertext between hosts and becomes plaintext only after decryption at the vxlan0 interface.

Raw test output

Test 1: Confirming encryption is active between the two hosts (since colocated services never cross the VXLAN interface) and that the traffic is unreadable in transit

$ ip -s xfrm state          # on scratch-a, now that scratch-b has a task on the same encrypted network
src 172.18.0.8 dst 172.18.0.7
	proto esp spi 0x0ab1a88e reqid 13681891 mode transport
	aead rfc4106(gcm(aes)) ... (160 bits)
src 172.18.0.7 dst 172.18.0.8
	proto esp spi 0x1f945555 reqid 13681891 mode transport
	aead rfc4106(gcm(aes)) ... (160 bits)
[... 2 more SAs, one pair per direction ...]

$ ip -s xfrm policy
src 172.18.0.7/32 dst 172.18.0.8/32 proto udp dport 4789 uid 0
	dir out action allow ...
	tmpl src 172.18.0.7 dst 172.18.0.8 proto esp ... mode transport

# outer link between the two hosts (eth0, where the real bytes travel):
$ tcpdump -i eth0 -A '(host 172.18.0.8 and (udp port 4789 or esp))'
10:48:06.598678 IP 172.18.0.7 > 172.18.0.8: ESP(spi=0x1f945555,seq=0xd), length 140
E.......@2Io..........UU....])^C...xW..i.!c....Sl2.oX....,.e.........Y.>G2Z......DW	F8.....y..!	...
                                                    # unreadable - no ports, no payload, nothing

# same traffic, inside the network sandbox, at the vxlan0 device (post-decrypt/decapsulate):
$ nsenter --net=/var/run/docker/netns/1-7f3e91c2a5 tcpdump -i vxlan0 -A
10:48:06.598678 IP 10.0.1.4.38044 > 10.0.1.2.5000: Flags [P.], seq 1:33, ... length 32
E..Tb.@.@...
.........1v..e....@.L.....
4@.."...HELLO_ACROSS_HOSTS_SECRET_MSG_1

Same packets, two vantage points: unreadable ESP ciphertext on the outer host-to-host link, plain IP/TCP with the readable payload only after decryption at the vxlan0 tunnel endpoint.

Parallel replication

At idle, only one of the four applier workers is active. Under a burst of 32 000 transactions across eight tables, all four workers (the default) begin applying concurrently until the burst clears.

Raw test output

Test 1: Idle baseline versus a write burst - confirming only worker 1 applies anything at idle, while all four workers apply concurrently during a 32 000 transaction burst across eight non-overlapping tables, before returning to idle

$ mysql --login-path=node-2 -e "SELECT WORKER_ID, SERVICE_STATE, APPLYING_TRANSACTION, LAST_APPLIED_TRANSACTION FROM performance_schema.replication_applier_status_by_worker WHERE CHANNEL_NAME='group_replication_applier';"
WORKER_ID  SERVICE_STATE  APPLYING_TRANSACTION  LAST_APPLIED_TRANSACTION
1          ON                                   9309cd8a-...:3782
2          ON
3          ON
4          ON

$ mysql --login-path=node-3 -e "
    CREATE DATABASE paralleltest;
    CREATE TABLE paralleltest.t1 (id INT AUTO_INCREMENT PRIMARY KEY, val INT);
    CREATE TABLE paralleltest.t2 (id INT AUTO_INCREMENT PRIMARY KEY, val INT);
    CREATE TABLE paralleltest.t3 (id INT AUTO_INCREMENT PRIMARY KEY, val INT);
    CREATE TABLE paralleltest.t4 (id INT AUTO_INCREMENT PRIMARY KEY, val INT);
    CREATE TABLE paralleltest.t5 (id INT AUTO_INCREMENT PRIMARY KEY, val INT);
    CREATE TABLE paralleltest.t6 (id INT AUTO_INCREMENT PRIMARY KEY, val INT);
    CREATE TABLE paralleltest.t7 (id INT AUTO_INCREMENT PRIMARY KEY, val INT);
    CREATE TABLE paralleltest.t8 (id INT AUTO_INCREMENT PRIMARY KEY, val INT);
  "

$ for n in 1 2 3 4 5 6 7 8; do
    ( for i in $(seq 1 4000); do echo "INSERT INTO t$n VALUES (NULL,$i);"; done \
      | mysql --login-path=node-3 paralleltest ) &
  done; wait

[polling node-2 every ~0.3s during the burst]
poll #0:  W1 APPLYING :5990   W2 APPLYING :5993   W3 APPLYING :5991   W4 APPLYING :5992
poll #1:  W1 APPLYING :6068   W2 APPLYING :6070   W3 APPLYING :6069   W4 APPLYING :6071
poll #2:  W1 APPLYING :6153   W2 APPLYING :6150   W3 APPLYING :6151   W4 APPLYING :6152
...
poll #13: W1 APPLYING :7130   W2 APPLYING :7129   W3 APPLYING :7128   W4 APPLYING :7131

$ mysql --login-path=node-2 -e "SELECT WORKER_ID, APPLYING_TRANSACTION, LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP FROM performance_schema.replication_applier_status_by_worker WHERE CHANNEL_NAME='group_replication_applier';"
WORKER_ID  APPLYING_TRANSACTION  LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP
1                                2026-07-16 10:11:16.031435
2                                2026-07-16 10:10:47.406484
3                                2026-07-16 10:10:46.368626
4                                2026-07-16 10:10:46.383622

Idle baseline shows only worker 1 with a LAST_APPLIED_TRANSACTION.

Mid-burst, all four workers report distinct, advancing APPLYING_TRANSACTION GTIDs on every one of 14 polls.

Post-burst, all four report a LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP within milliseconds of each other, confirming sustained parallel apply rather than a momentary spike.

Celery fair scheduling and the slow query log

One 15s query submitted alongside five ~0.01s ones: the fast queries finish in ~210–260ms each while the slow one runs uninterrupted. The slow query log correctly captures the slow query and excludes the fast ones.

Raw test output

Test 1: Submitting one slow query alongside five fast ones - well after the fast queries have finished, only the slow task still appears in the active pool

$ celery -A superset.tasks.celery_app:app inspect stats
"pool": {"max-concurrency": 4, "processes": [203, 204, 205, 206], "writes": {"strategy": "fair", ...}}

$ TOKEN=$(curl -sk -X POST https://172.18.0.10/api/v1/security/login -H 'Content-Type: application/json' \
    -d '{"username":"admin","password":"admin","provider":"db","refresh":true}' | jq -r .access_token)
$ CSRF=$(curl -sk -X GET https://172.18.0.10/api/v1/security/csrf_token/ -H "Authorization: Bearer $TOKEN" | jq -r .result)

$ curl -sk -X POST https://172.18.0.10/api/v1/sqllab/execute/ -H "Authorization: Bearer $TOKEN" -H "X-CSRFToken: $CSRF" -H "Content-Type: application/json" \
    -d '{"database_id":1,"schema":"superset","sql":"SELECT SLEEP(15)", "runAsync": true, "client_id":"slowburstB"}'      # t=0.00s
$ curl -sk -X POST https://172.18.0.10/api/v1/sqllab/execute/ -H "Authorization: Bearer $TOKEN" -H "X-CSRFToken: $CSRF" -H "Content-Type: application/json" \
    -d '{"database_id":1,"schema":"superset","sql":"SELECT 1, SLEEP(0.01)", "runAsync": true, "client_id":"fastburstB1"}'   # t=0.34s
$ curl -sk -X POST https://172.18.0.10/api/v1/sqllab/execute/ -H "Authorization: Bearer $TOKEN" -H "X-CSRFToken: $CSRF" -H "Content-Type: application/json" \
    -d '{"database_id":1,"schema":"superset","sql":"SELECT 2, SLEEP(0.01)", "runAsync": true, "client_id":"fastburstB2"}'   # t=0.50s
$ curl -sk -X POST https://172.18.0.10/api/v1/sqllab/execute/ -H "Authorization: Bearer $TOKEN" -H "X-CSRFToken: $CSRF" -H "Content-Type: application/json" \
    -d '{"database_id":1,"schema":"superset","sql":"SELECT 3, SLEEP(0.01)", "runAsync": true, "client_id":"fastburstB3"}'   # t=0.69s
$ curl -sk -X POST https://172.18.0.10/api/v1/sqllab/execute/ -H "Authorization: Bearer $TOKEN" -H "X-CSRFToken: $CSRF" -H "Content-Type: application/json" \
    -d '{"database_id":1,"schema":"superset","sql":"SELECT 4, SLEEP(0.01)", "runAsync": true, "client_id":"fastburstB4"}'   # t=0.85s
$ curl -sk -X POST https://172.18.0.10/api/v1/sqllab/execute/ -H "Authorization: Bearer $TOKEN" -H "X-CSRFToken: $CSRF" -H "Content-Type: application/json" \
    -d '{"database_id":1,"schema":"superset","sql":"SELECT 5, SLEEP(0.01)", "runAsync": true, "client_id":"fastburstB5"}'   # t=1.00s

$ celery -A superset.tasks.celery_app:app inspect active     # t≈5.8s
->  celery@6bdc0cb9aeaf: OK
    * {'id': '4b4dd1db-...', 'name': 'sql_lab.get_sql_results', 'args': [8, 'SELECT SLEEP(15)'],
       'worker_pid': 203, 'time_start': 1784197202.98, ...}

$ celery -A superset.tasks.celery_app:app inspect active     # t≈10.5s
    * {'id': '4b4dd1db-...', 'name': 'sql_lab.get_sql_results', 'args': [8, 'SELECT SLEEP(15)'], 'worker_pid': 203, ...}

$ celery -A superset.tasks.celery_app:app inspect active     # t≈15.1s
    * {'id': '4b4dd1db-...', 'name': 'sql_lab.get_sql_results', 'args': [8, 'SELECT SLEEP(15)'], 'worker_pid': 203, ...}

$ docker exec node-3 docker exec mysql grep -B3 SLEEP /var/log/mysql/slow-queries.log
# User@Host: superset[superset] @  [172.18.0.10]  Id: 22525
# Query_time: 15.000301  Lock_time: 0.000000 Rows_sent: 1  Rows_examined: 1
SET timestamp=1784197203;
SELECT SLEEP(15)

At every check, the only task reported is the slow one, pinned to worker_pid 203. All 5 fast queries ran in 210-260ms each, done within ~1.03s total, while the slow query ran the full 15,222ms uninterrupted.

Redis cache performance

Response time drops from roughly 2.19 seconds cold to 0.06 seconds cached (about 35x faster). Confirmed by a real cache key with the expected TTL.

Raw test output

Test 1: Confirming Superset stores query results in the cache and measuring the benefit - cache hit timing plus direct Redis key inspection

$ curl -sk -X POST https://172.18.0.10/api/v1/dataset/ -H "Authorization: Bearer $TOKEN" -H "X-CSRFToken: $CSRF" -H "Content-Type: application/json" \
    -d '{"database":1,"schema":"superset","table_name":"cache_test_vds","sql":"SELECT SLEEP(2) AS delay_flag, 1 AS cnt"}'
# dataset id: 1

$ curl -sk -w '\nTIME_TOTAL:%{time_total}' -X POST https://172.18.0.10/api/v1/chart/data -H "Authorization: Bearer $TOKEN" -H "X-CSRFToken: $CSRF" -H "Content-Type: application/json" \
    -d '{"datasource":{"id":1,"type":"table"},"queries":[{"columns":["delay_flag","cnt"],"metrics":[],"row_limit":10}],"result_type":"full","result_format":"json"}'
# first (cold):  {"cache_key": "ee154b211049e45623868f0f23fdd440", "cached_dttm": null, "is_cached": null}   TIME_TOTAL:2.190370
# repeated 3x:    {"cache_key": "ee154b211049e45623868f0f23fdd440", "cached_dttm": "2026-07-16T10:37:00", "is_cached": true}
#                 TIME_TOTAL:0.045243 / 0.078710 / 0.064761

$ python3 -c "import redis; r=redis.Redis(host='redis'); print([k for k in r.keys('*') if b'cache' in k])"
[b'superset_data_cacheee154b211049e45623868f0f23fdd440']

$ python3 -c "import redis; r=redis.Redis(host='redis'); k=b'superset_data_cacheee154b211049e45623868f0f23fdd440'; print(r.ttl(k), r.type(k), r.strlen(k))"
3556 string 1060

$ python3 -c "import redis; r=redis.Redis(host='redis'); k=b'superset_data_cacheee154b211049e45623868f0f23fdd440'; print(r.get(k)[:60])"
b'!\x80\x05\x95\x18\x04\x00\x00\x00\x00\x00\x00}\x94(\x8c\x02df\x94\x8c\x11pandas.core.frame\x94...'

~2.19s cold vs. ~0.06s cached (~35x faster), same cache_key and stable cached_dttm confirming cache hits. The key is a pickled pandas.DataFrame with a TTL matching the configured 3600s timeout.

References

szachovy/superset-cluster

Why I did it

Architecture

See it Work

TLS everywhere

VRRP failover with nopreempt

InnoDB Cluster election

Blocking potentially dangerous commands via seccomp

Swarm secrets never touch disk

MySQL root password: permission tightening and a zero-trace credential handoff

HTTP security headers, hidden server tokens

Encrypted overlay network (VXLAN + IPsec)

Parallel replication

Celery fair scheduling and the slow query log

Redis cache performance

References

VRRP failover with `nopreempt`