Define the recovery target before deploying
Rollback works only when the prior release is identifiable, compatible and still available. Use immutable, version-labeled application artifacts and verify them before activation. Define which signals trigger rollback: failed health checks, rising errors, user-visible regressions, security exposure or broken dependencies.
A rollback of application files is not a database rollback. Schema changes and live writes may make an earlier application version incompatible even when its files are preserved. Review forward/backward schema compatibility and backup recovery procedures separately.
Build and validate a restricted artifact
The JoeC Apache VM workflow packages allowlisted public HTML, JavaScript, CSS and the PHP contact handler. Its build script deliberately excludes private repository files such as runbooks, article Markdown sources, CI files and deployment credentials.
A safe release sequence starts with local checks:
python3 scripts/verify_site.py
python3 scripts/build_release.py
python3 scripts/validate_release.py dist/joec-public-vm.zipThe validator checks ZIP members, expected file paths, total and per-file limits, symlinks and CRC. It is intended to detect malformed or unexpected artifacts before they reach the active web root. Do not bypass validation to accelerate an urgent change; diagnose the failed gate.
Activate atomically and check multiple layers
The VM deployer extracts files into a new versioned release directory before switching a current symlink. The previous release pointer remains available for rollback. This reduces the window in which visitors can see a partially copied site.
sudo bash scripts/deploy-vm.sh
python3 scripts/smoke_origin.pyThe smoke suite checks loopback health, HTML, deployed tools, security headers and PHP handler execution. These checks verify the origin but do not prove external Cloudflare routing, email delivery, every browser flow or user experience. Follow with a public-URL check and explicit functional acceptance tests.
Roll back deliberately
If the release fails its defined acceptance criteria, record the observed issue, affected version and who authorized the recovery decision. On the JoeC VM, the release manager provides a rollback script:
sudo bash scripts/rollback-vm.sh
python3 scripts/smoke_origin.pyDo not assume success from the command exit alone: repeat both the origin and external verification checks. Preserve failing-release logs and the artifact hash for postmortem analysis, subject to privacy and log-retention policy.
Where deployment state includes persistent data, reversing web assets may be insufficient. For those systems, use a migration strategy explicitly designed for rollback or roll-forward and practice disaster recovery outside production.
Improve the next release
Capture the trigger signal, detection time, decision time, rollback duration and verification evidence. Add or strengthen preflight tests when a preventable failure is discovered. Prefer a small, repeatable release process with a clear owner rather than a complicated one that cannot be exercised routinely.