Skip to content

Rollbacks and day-to-day operations

Undo a bad release, retry a failed one, find the right log, and plan the next improvements.

About 15 min · Verified 8 October 2026

0 of 4 steps done0%
On this page

How a bad release gets undone#

SituationWhat happensWhat you do
A test fails in BuildPipeline stops. Production untouchedFix the code and push
The new version crashes or fails ValidateServiceCodeDeploy automatically rolls back to the last good revisionRead the failed event, fix, push again
The release succeeds but behaves wronglyNothing is automaticRevert in Git (best), or redeploy a known-good commit
The very first deployment failsThere is no previous revision to restoreUse the manual checkout and saved unit (see below), fix, retry

Roll back on purpose#

Your computerCreate a commit that undoes the bad one
git log --oneline -n 5
git revert --no-edit <COMMIT_SHA>
git push origin main

Replace <COMMIT_SHA> with your own value.

The pipeline tests the revert and deploys it like any other change. You keep an honest history of what happened.

Option 2: run the pipeline on a specific known-good commit#

Your computerRelease an older commit through the same pipeline
aws codepipeline start-pipeline-execution --region ap-south-1 --name shortlink-api-prod \
  --source-revisions actionName=Source,revisionType=COMMIT_ID,revisionValue=<COMMIT_SHA>

Replace <COMMIT_SHA> with your own value.

Use this when you need the old version back right now and will sort out the Git history afterwards. Remember that the next push to main will deploy whatever is at the tip of main again.

Option 3: redeploy a previous CodeDeploy deployment#

Open CodeDeployDeployments, select an earlier Succeeded deployment and use its Retry deployment (or Copy deployment) action to deploy the same revision again. This skips the build and tests, so use it only for a revision you already trust.

If the first release failed and the service is down#

Restore the hand-made setup you backed up in guide 1:

On the EC2 instancePut the original unit back
sudo cp -p /etc/systemd/system/shortlink-api.service.before-codedeploy /etc/systemd/system/shortlink-api.service
sudo systemctl daemon-reload
sudo systemctl restart shortlink-api
curl -fsS http://127.0.0.1:3000/health

This only works if /opt/shortlink/repo still exists, which is why the first-release chapter told you to delete it only after success.

Retry without starting over#

When one action in a stage fails and you have fixed the cause, retry just that action so the successful ones are reused:

CodePipelineshortlink-api-prod(failed stage)Retry stageRetry failed actions
Your computerThe same thing from the CLI
aws codepipeline retry-stage-execution --region ap-south-1 \
  --pipeline-name shortlink-api-prod --stage-name Deploy \
  --pipeline-execution-id <EXECUTION_ID> --retry-mode FAILED_ACTIONS

Replace <EXECUTION_ID> with your own value.

If you changed the structure of the pipeline after the failure, start a new execution instead.

Where to look when something fails#

SymptomFirst place to look
Pipeline did not start after a pushThe push touched no file matching the filters, the push was to another branch, or the connection is not Available
Red SourceDeveloper ToolsConnections status; repository and branch names in the Source action
Red BuildThe CodeBuild log (View details → Logs). Tests, npm ci or the buildspec
Red DeployCodeDeployDeploymentsthe deploymentView events, then the script's output
Deployment stuck on BlockTrafficThe target group is draining. It waits for the deregistration delay (30 s here, 300 s by default)
Deployed, but the app is downsudo journalctl -u shortlink-api -n 80 --no-pager on the server, and curl http://127.0.0.1:3000/health
Agent never receives the deploymentsudo systemctl status codedeploy-agent and /var/log/aws/codedeploy-agent/codedeploy-agent.log
Your computerEverything about the last execution in one command
aws codepipeline get-pipeline-state --region ap-south-1 --name shortlink-api-prod

Good habits#

  • Merge, do not push straight to main. Use short-lived branches and pull requests (into your own fork) so a typo does not deploy.
  • Turn on branch protection for main in GitHub (SettingsBranches) to require pull requests.
  • Add a notification. In the pipeline choose NotifyCreate notification rule and send Pipeline execution failed to an SNS topic with your email.
  • Never put secrets in the repository or the buildspec. The database password lives only in /etc/shortlink/shortlink.env on the server.
  • Watch the cost. CodePipeline and CodeBuild are cheap for a workshop, but the infrastructure from guide 1 is not. Run the clean-up chapter when you finish.

What to improve next#

TodayBetter
One server, in-place, ~1 minute of 503s per releaseTwo servers in an Auto Scaling group with CodeDeployDefault.OneAtATime so one is always serving
Tests onlyAdd a lint step, a dependency audit and a smoke test after deploy
Manual approval never requiredAdd a manual approval action before Deploy for production
Database schema applied on every deployIntroduce versioned migrations for destructive changes
Separate API and web pipelinesOne pipeline with both paths (see guide 3, Operations)

Next: troubleshooting.

Found a mistake? Edit this page on GitHub.