docker-compose wait for dependencies

Sometimes, when you have an over-optimized system leveraging asynchronization, you have to jump through hoops to synchronize things again to make them work in practice. This is the case with docker-compose currently.

First, let me mention that this is with the docker-compose file format 3.x. Things are different with other versions.

Here’s the issue. Say you’ve got 2 microservices you want to dockerize, a tomcat container and a mysql container. docker-compose will start the two containers in arbitrary order, but you and I know that the tomcat webapp probably needs the database to be available. How do you do it?

The docker compose file allows for an argument called depends_on.
depends_on argument orders the services but it doesn’t actually wait for a service to be “ready”. I believe it just waits for the container’s service port to be exposed. The database may not have started up yet even if the ports are exposed.

Stackoverflow has this solution
Unfortunately, the depends_on’s condition argument that was introduced in compose file version 2.x is deprecated in 3.x.

I believe Docker’s official philosophy is that your application is to be created in a robust manner. That is, if the database is not available (either it’s not started yet or it has gone down temporarily or permanently since starting), the application should handle it gracefully. It’s also very difficult for them to know what it means for a service to be started. e.g. What does it mean for mysql to be started, vs postgres, vs rabbitmq, vs tomcat, vs nodejs, vs some random service you wrote?

Nevertheless, I believe there ought to be a standard way that allows users to order container startup based on readiness.

So let’s talk about the solution that my coworker and I put together, built on other open source utilities.

The first thing I’d like you to take a look at is ufoscout’s docker-compose-wait project. His wait.sh script will wait for a service to be ready before returning. It uses the netcat util to ping the service’s port until it’s available.

In your docker-compose.yml file, you just need to specify the container names and port of the other services that you depend on. See the WAIT_HOSTS environment variable. For example

version: '3'
services:

  mysql:
    image: "mysql:5.7"
    container_name: mysql
    ports:
      - "3306:3306"

  my_app:
    image: "mySuperApp:latest"
    environment:
      WAIT_HOSTS: mysql:3306

The wait.sh file is the key, but there are some caveats to using it. For instance, your container needs the nc (netcat) util. So unless your image is ubuntu, you’re not likely to have that in the standard tomcat image from Docker Hub. You’ll also have to make sure you call the wait.sh script before calling the service startup script. e.g. “CMD /wait.sh && /MySuperApp.sh”

Here’s what I did for the tomcat container. First, I located the the tomcat image’s Dockerfile. It’s referenced in the README for me so that’s nice.

Then I made several modifications to it

ADD https://raw.githubusercontent.com/ufoscout/docker-compose-wait/1.0.0/wait.sh /$CATALINA_HOME/wait.sh
RUN chmod +x /$CATALINA_HOME/wait.sh
RUN apt-get update && apt-get install -y netcat 
# CMD ["catalina.sh", "run"]
CMD ./wait.sh && catalina.sh run

Now I build the image.

docker build -t kanesee/tomcat-wait:9.0 .

The entire set of files and modifications can be found in this github repo.

So now we have a version of tomcat that will wait for other services to be ready before it starts. We just need to add the WAIT_HOSTS argument to docker-compose.yml to specify the name of the database container and port and then it’s ready to go.
If you need longer than the default 30 seconds, you can set WAIT_HOSTS_TIMEOUT to a longer period.

For your benefit, I pushed my tomcat-wait image to Docker Hub so that you don’t need to create your own image if you’re looking for a tomcat v9.0 that does this.

Otherwise, please follow this recipe and create your own “waiting” services. If you create one, I’d love to know about it. Please leave a link to your image in the comments section.

Advertisements
Tagged , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: