Docker container frequently hangs

gwerbin-tive · August 23, 2023, 7:11pm

When running with Docker in a local developer environment, the Liquibase container frequently hangs, seemingly forever. Sometimes it gets to a specific point in the logs before hanging, other times the container completely hangs without any output, getting stuck while Docker is attempting to attach to the container. Sometimes it responds to Ctrl-C, other times it must be killed with docker kill.

Deleting the container and image and then rebuilding before running seems to reduce the frequency of the problem, although it doesn’t eliminate it entirely.

This happens at least half the time we run it with docker compose run or docker compose up, and it can be frustrating while trying to work on the application from a clean start, which we often want to do during testing.

Ideally, we’d be able to run docker compose up liquibase and never have a problem.

Hopefully this just a simple misconfiguration on our end! I have tried to search for this problem here and on the Liquibase issue tracker and have never found anyone else with the same issue.

In case that’s relevant, our team is all using Mac M1 machines, using Docker & Compose as installed from Docker Desktop. Note that we are running in production using AWS CloudBuild and (as far as I know) have never had a problem, this is only a problem in our local developer environments.

Any debugging suggestions, insight, etc. would be much appreciated here.

Here is the relevant section of docker-compose.yml:

version: '3.9'

services:

  liquibase:
    platform: 'linux/amd64'
    build:
      context: '../liquibase'
    command: ['update']
    environment:
      - LIQUIBASE_HEADLESS=true
      - LIQUIBASE_SHOW_BANNER=false
      - LIQUIBASE_DRIVER=org.postgresql.Driver
      - LIQUIBASE_COMMAND_CHANGELOG_FILE=db.change-log.xml
      - LIQUIBASE_COMMAND_URL=jdbc:postgresql://postgres:5432/${POSTGRES_DB:-postgres}
      - LIQUIBASE_COMMAND_USERNAME=${POSTGRES_USER:-admin}
      - LIQUIBASE_COMMAND_PASSWORD=${POSTGRES_PASSWORD:-admin}
      - LIQUIBASE_LOG_LEVEL=FINE
    volumes:
      - type: 'bind'
        source: '${PWD}/liquibase/changelog'
        target: '/liquibase/changelog'
        read_only: true
    depends_on:
      postgres:
        condition: service_healthy

  postgres:
    image: 'timescale/timescaledb:latest-pg15'
    ports:
      - target: 5432
        published: 5432
    volumes:
      - type: volume
        source: postgres
        target: '/app/postgres'
      - type: bind
        source: '${PWD}/docker/init_test_db.sh'
        target: '/docker-entrypoint-initdb.d/init_test_db.sh'
    environment:
      - POSTGRES_HOST
      - POSTGRES_DB
      - POSTGRES_USER
      - POSTGRES_PASSWORD
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}']
      start_period: '10s'
      interval: '10s'
      timeout: '5s'
      retries: 5

And here is liquibase/Dockerfile:

# syntax=docker/dockerfile:1.4

FROM liquibase/liquibase:4.18

# Download Postgress Driver
# Maven artifact: https://mvnrepository.com/artifact/org.postgresql/postgresql/42.5.1
# SHA1: https://repo1.maven.org/maven2/org/postgresql/postgresql/42.5.1/postgresql-42.5.1.jar.sha1
# hadolint ignore=DL4006
RUN wget --progress=dot:giga 'https://repo1.maven.org/maven2/org/postgresql/postgresql/42.5.1/postgresql-42.5.1.jar' \
  && printf 'ac2f61eb3b1b4e47ea45de47e73d2e92f49e3ce1 *postgresql-42.5.1.jar' | sha1sum -c - \
  && mv 'postgresql-42.5.1.jar' /liquibase/lib/

CMD ["history"]

Here is liquibase/changelog/db.change-log.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
	xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
	xmlns:pro="http://www.liquibase.org/xml/ns/pro"
	xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
		http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd
		http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd
		http://www.liquibase.org/xml/ns/pro http://www.liquibase.org/xml/ns/pro/liquibase-pro-latest.xsd">

	<!-- Changesets -->
  <includeAll path="schema-changelog" relativeToChangelogFile="true" />

</databaseChangeLog>

We have two .sql scripts in liquibase/changelog/schema-changelog right now, with some very straightforward CREATE ROLE and CREATE TABLE commands.

Finally, below is an excerpt of the FINE logs (can’t post the full thing, too long for Discourse) from a recent invocation that resulted in an infinite hang:

[2023-08-23 18:26:48] FINE [liquibase.executor] CREATE TABLE public.databasechangeloglock (ID INTEGER NOT NULL, LOCKED BOOLEAN NOT NULL, LOCKGRANTED TIMESTAMP WITHOUT TIME ZONE, LOCKEDBY VARCHAR(255), CONSTRAINT databasechangeloglock_pkey PRIMARY KEY (ID))
[2023-08-23 18:26:48] FINE [liquibase.executor] 0 row(s) affected
[2023-08-23 18:26:48] FINE [liquibase.lockservice] Created database lock table with name: public.databasechangeloglock
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Initialize Database Lock Table
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] DELETE FROM public.databasechangeloglock
[2023-08-23 18:26:48] FINE [liquibase.executor] 0 row(s) affected
[2023-08-23 18:26:48] FINE [liquibase.executor] INSERT INTO public.databasechangeloglock (ID, LOCKED) VALUES (1, FALSE)
[2023-08-23 18:26:48] FINE [liquibase.executor] 1 row(s) affected
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.executor] Lock Database
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] INFO [liquibase.lockservice] Successfully acquired change log lock
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.changelog.ChangeLogHistoryService instance liquibase.changelog.MockChangeLogHistoryService
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.changelog.ChangeLogHistoryService instance liquibase.changelog.StandardChangeLogHistoryService
[2023-08-23 18:26:48] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.parser.ChangeLogParser instance liquibase.parser.core.formattedsql.FormattedSqlChangeLogParser
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.parser.ChangeLogParser instance liquibase.parser.core.json.JsonChangeLogParser
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.parser.ChangeLogParser instance liquibase.parser.core.sql.SqlChangeLogParser
[2023-08-23 18:26:48] FINE [liquibase.configuration] No configuration value for liquibase.secureParsing found
[2023-08-23 18:26:48] FINE [liquibase.configuration] Configuration liquibase.secureParsing is using the default value of true
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.parser.ChangeLogParser instance liquibase.parser.core.xml.XMLChangeLogSAXParser
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.parser.ChangeLogParser instance liquibase.parser.core.yaml.YamlChangeLogParser
[2023-08-23 18:26:48] FINE [liquibase.resource] Path db.change-log.xml in /liquibase does not exist
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.CustomPreconditionWrapper
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.AndPrecondition
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.ChangeLogPropertyDefinedPrecondition
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.ChangeSetExecutedPrecondition
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.ColumnExistsPrecondition
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.DBMSPrecondition
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.ForeignKeyExistsPrecondition
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.IndexExistsPrecondition
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.NotPrecondition
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.ObjectQuotingStrategyPrecondition
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.OrPrecondition
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.PreconditionContainer
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.PrimaryKeyExistsPrecondition
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.RowCountPrecondition
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.RunningAsPrecondition
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.SequenceExistsPrecondition
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.SqlPrecondition
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.TableExistsPrecondition
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.TableIsEmptyPrecondition
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.UniqueConstraintExistsPrecondition
[2023-08-23 18:26:48] FINE [liquibase.servicelocator] Loaded liquibase.precondition.Precondition instance liquibase.precondition.core.ViewExistsPrecondition
[2023-08-23 18:26:48] FINE [liquibase.parser] Resolving XML entity name='null', publicId='null', baseURI='null', systemId='http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd'

The last message in the output above is:

Resolving XML entity name='null', publicId='null', baseURI='null', systemId='http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd'

It seems like this hang point is frequent, but there might be others I haven’t seen because usually we run with log level INFO, not FINE.

Topic		Replies	Views
Execution of Liquibase in Java hangs … and solution. Patch needed	0	263	May 10, 2024
Maven Plugin Hangs when using SSH Tunnel General Discussion	1	655	June 24, 2011
Liquibase and Docker General Discussion	15	4330	April 10, 2020
Liquibase is stuck on 'Reading from [dbo].[DATABASECHANGELOG]' General Discussion	0	1184	December 14, 2016
Using Liquibase via Docker Community Content and Articles	0	2320	October 27, 2021

Docker container frequently hangs

Related topics