How we moved CrossPost from cPanel to Kubernetes in one afternoon

CrossPost ran on a cPanel shared host from the day it launched. It worked, but the cracks had been showing for a while: websockets returned HTTP 500 through Apache, so live updates never worked; there was no ext-ffi and no libvips, so HEIC image conversion was impossible; the shell PHP was 8.4 while the actual daemons ran 8.3; and every secret sat in a plaintext .env. This month we moved the whole production app to our Talos Kubernetes cluster — and we let an AI agent, Devin CLI, drive the migration while we supervised.

The short version: about three and a half hours from a written plan to a verified production system, an eleven-minute maintenance window, and exactly one bug that users could have noticed. Here is the longer version, including the parts that went wrong.

The plan: rehearse everything first

CrossPost is a paying SaaS, so a single-shot cutover was never on the table. The target was one dedicated namespace on our existing Talos cluster: the vendor’s official, digest-pinned image as a single pod; KubeBlocks MySQL 8.0 in semi-synchronous replication with inline backups and point-in-time recovery; KubeBlocks Redis; secrets synced from Infisical, never touching Git; ingress through our shared Cilium Gateway with cert-manager for TLS.

Everything was written as guarded Taskfile targets and plain scripts before we touched anything live: one task builds the data plane, one applies the app, one prepares the route, one issues the certificate. The database move was a script that dumps MariaDB on cPanel, strips the bits MySQL 8 does not accept, and streams straight into the new pod. Then we rehearsed the entire thing on the cluster’s internal address — real database, real files, real certificate — before DNS ever changed.

One artifact got treated as more important than the database itself: the Laravel APP_KEY. Every OAuth token, billing detail and integration credential in the database is encrypted with it. It migrated byte-for-byte, which is why every connected social account survived the move without re-authorization.

Cutover day

At 18:33 UTC the old site went into maintenance mode. Final database dump and import, final sync of uploaded files, pod restart. At 18:40 the authoritative DNS answer changed; two and a half minutes later Let’s Encrypt had issued the new certificate; at 18:44 the login page returned 200 on the new cluster. Maintenance mode to first healthy public response: eleven minutes.

After that we moved the media bucket: 17,020 objects, about a gigabyte, mirrored into a new bucket on the cluster’s MinIO in 58 seconds. A closer look at that number afterwards told a different story — most of those objects were orphaned thumbnails from years of imported social posts that nobody had ever deleted. Roughly 15,000 of them are gone now; the live library is the imported-post thumbnails, our media uploads and account avatars, under two thousand objects and about 133 MB. We also updated the eleven old login links on this site. Total tool calls for the migration day core: 802.

What actually broke

The honest list is more interesting than the timeline. Three things worth writing down:

• Every callback URL broke for 53 minutes. On cPanel, Apache sent everything under /app to PHP. In the vendor image, nginx reserves the /app prefix for the websocket server, so every OAuth return, inbox webhook and payment webhook hit the wrong process and got a 500. Our smoke tests checked the login page, not a callback. The fix was a patched nginx config mounted over the vendor’s; the lesson is to compare route:list with the web server config before the window, and to test a callback URL, not just a page.
• The websocket “bug” that was never a bug. Testing websockets through the Gateway returned 500, and we spent real time on it — twice, because the conclusion was lost to a context compaction. The test client was negotiating HTTP/2, which drops the Upgrade header. Browsers always use HTTP/1.1 for upgrades, so users were never affected. Test websockets with explicit HTTP/1.1, and write conclusions down.
• Policy whack-a-mole. The pod could not reach MinIO (our own egress rules excluded RFC1918, and MinIO’s default-deny had no rule for the app), then the new bucket refused anonymous reads the old one allowed. Two network rules and one bucket policy later, media flowed.

There was also a two-minute public 503 right after the flip whose root cause we never pinned down, and DNS turned out to be the least cloud-like part of the day: a PowerDNS front end over a BIND zone file managed by cPanel, changed by editing the zone file directly and pushing it to all three authoritative servers. The right preparation would have been lowering the TTL days ahead.

What we got for it

Websockets work for the first time. HEIC conversion works. Secrets live in Infisical instead of a plaintext file. Upgrades are now a digest bump and a rolling restart instead of a maintenance script on a shared box, and the database takes its own backups with point-in-time recovery. The old cPanel account stays untouched and warm as a rollback until the soak period ends.

Honest ledger for an agent-driven migration: it wrote all the manifests, caught its own callback bug before a customer did, and left three clean commits. It also cated a file it should not have, reached for the forceful flag faster than we would have, and lost a solved problem to a compacted context. Supervision is not optional — but neither, it turns out, is it slow.