Gradual restart for Cloud Foundry apps

January 21, 2018

Just recently I needed to test a change to an environment variable of an app running in Cloud Foundry without knowing whether this change would need to be persisted in the deployment manifest of that app. Instead of deploying the same app again just because of one environment variable (temporarily) changing, you can change the environment variable on the deployed app and restart it. However the default cf restart will cause your app to be temporarily unavailable. Read along if you are interested in restarting Cloud Foundry deployed apps without causing a downtime.

When deploying apps to the cloud, maybe microservices as part of a distributed system, it makes sense to adhere to the twelve-factor app methodology. By doing so you design for scalability, because once your app runs as an independent, stateless process, you can easily run as many instances of your app as you need. With that in mind I find it useful to have more than one instance of every app running. This way, even if one instance crashes, your app will still be available because of the second instance still running, which is great. Your users had no negative impact by that app instance crash and yet you have the opportunity to find and fix the crash reason based on real production data.

Motivation for gradual app restarts

Now there will be situations when I want to restart an application, like the one pictured above where the restart is necessary for the app to pick up a changed environment variable.

In Cloud Foundry you usually have two options here: You could perform a blue-green deployment. Or instead you just issue a cf restart command.

The first option looks like an overkill when all you want to do is to try something out by temporarily changing an environment variable. Than again the second option will restart all app instances at once, thus causing downtime of your app, which is something we like to avoid.

Now assuming that we run at least two instances of each and every app I felt like there is a better, third option. That option is to perform a gradual restart. By that I mean restarting one app instance after another, ensuring that there is always at least one app instance running.
This way once every instance has been restarted, the changed environment variable is fully taken into account by the app and we had no downtime for that app, too!

CF CLI’s possibilites

It turns out that a gradual restart can be done with CF CLI’s cf restart-app-instance command.
However I find there is one major caveat of using this command for a gradual app restart:
You would need to perform this command as many times as you have app instances of an app you want to restart. Furthermore you might want to wait between issuing the commands to give the restarted app instance the time it needs to become available again.
This makes the process of gradually restarting an app with just this command a very manual and repetitive task.

Improving on CF CLI’s capabilities

So instead of manually performing gradual restarts in this tedious and repetitive manner I created a function that makes use of the cf restart-app-instance command. In the end even a basic function is better than the manual process.
So all we really need is a function that takes the name of the app we want to restart, the number of app instances this affects and how long we want to wait before restarting the next instance.
The following function does take exactly those three inputs:

function cf-restart-app-gradually() { index=0; while [ $index -lt $2 ]; do cf restart-app-instance $1 $((index++)); sleep $3; done }

After adding this to my .bashrc file, I can easily perform a gradual restart for a deployed app by using this command:

cf-restart-app-gradually APP_NAME HIGHEST_APP_INSTANCE_INDEX DELAY_BETWEEN_RESTARTS_IN_SECONDS

I find this quite convenient and maybe this is helpful for someone else, too. 🙂