Possible minor bug: Service port definition syntax in production.yml

[felix-caboff]

Describe the bug
I regularly get errors when using the CLI that tells me that the port definitions are invalid. They are relatively simple to fix by hand but I thought I would ask the question in case I am missing something / this is actually a real bug (albeit a minor one).

An example of the YAML in question is:

    build:
      context: .
...SNIP...
    environment:
      - NO_PROXY=django,graphql_engine
      - POSTGRES_DB=${POSTGRES_DB}
      - POSTGRES_HOST=${POSTGRES_HOST}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_PORT=${POSTGRES_PORT}
      - POSTGRES_USER=${POSTGRES_USER}
...SNIP...
    expose:
      - "${POSTGRES_PORT}:5432"

The error is:

Error response from daemon: invalid JSON: invalid port '5432:5432': invalid syntax

So it seems that ${POSTGRES_PORT} is set to 5432 and it also has ":5432" appended which makes it invalid.

To Reproduce
Steps to reproduce the behavior:

1. Login to the machine and access the CLI
2. Execute ./ghostwriter-cli-linux backup or similar
3. See error

Expected Behavior
I think this should "just work".

Our setup
We are running this on an Ubuntu 22.04 host with Docker version 29.0.1 all patched and up to date.

[felix-caboff]

A few bits of extra information in case some reads this whilst the changes that are in the Ghostwriter-CLI repo are not yet integrated into the main Ghostwriter repo. The above still possibly applies but dealing with it was further complicated by the mis-match between the OS installed Docker and its minimum requirements and the docker handling within Ghostwriter itself. Essentially, running most commands within the ghostwriter-cli resulted in errors such as:

Failed to get container list from Docker: Error response from daemon: client version 1.41 is too old. Minimum supported API version is 1.44, please upgrade your client to a newer version

This was possible to resolve by downloading the current binary from the CLI repo and putting it within the Ghostwriter directory. Catch here though, is that you can't do a straightforward git pull to then update Ghostwriter. I effectively did a backup using the new CLI, removed it, did a git pull, then put another copy of the new CLI into the right place and then continued the upgrade process like normal from there.

Given the CLI repo had an update just a couple of days ago, I suggest that the main repo needs an update fairly soon to remove this extra complication. This is a bit more time pressured here because the current version of Docker on Ubuntu (and other Debian based?) does not work with the current main repo CLI executable.

And for what it is worth, I swap lines such as this:

    expose:
      - "${POSTGRES_PORT}:5432"

With this:

    expose:
      - "5432"

In the YAML files to get it to be happy with the syntax. The best way to approach this probably needs confirming.

Hopefully this is helpful for anyone who is currently working through this.

[chrismaddalena]

Docker has changed the syntax over the years between versions. I think it was v1.26 that last changed them. That was when they altered how they passed env variables. The current YAML files appear fine if I check them with a Docker and YAML linter or perform any operations with them.

Docker v29 made some significant changes last week. Docker v29 may prefer a different syntax, but I have not been able to update yet. Docker still indicates that v28 is the latest version for macOS. I'll see if I can manually update and reproduce the issue. For now, Docker has no complaints with Docker v28.3.3.

Update: I do not see any errors when running ./ghostwriter-cli-linux backup or other commands (docker compose or with Ghostwriter CLI) with the local.yml or production.yml files.

The final expose value being 5432:5432 is correct. That tells Docker to expose the container's port 5432 to the host on port 5432. If you drop the mapping (e.g., 5432), that exposes the port only to other containers. That could still be a beneficial change for postgres and collab-server, as we don't need external services to access them.

[felix-caboff]

Thanks for looking into this @chrismaddalena. I think there may be something else a bit strange here but I can't put my finger on it. What I do know though is with the unaltered production.yml file I get this error when running CLI actions:

Error response from daemon: invalid JSON: invalid port '5432:5432': invalid syntax

For what it is worth, the official (?) specification for docker compose states this:

Syntax is <portnum>/[<proto>] or <startport-endport>/[<proto>] for a port range.

The syntax you have is apparently listed as used by the ports configuration section:

You can either specify both ports (HOST:CONTAINER), or just the container port. In the latter case, the container runtime automatically allocates any unassigned port of the host.

I am also reading other articles on the same from earlier this year which suggest the same. Happy to be proven wrong. Just aware that there appears to be something not quite right.

In any case, please can I encourage the integration of the latest release of ghostwriter-cli into this repo?

[cobbr]

I ran into the same issue and I downgraded docker-ce to v28 as a workaround for now, which temporarily fixes the issue:

$ apt install docker-ce=5:28.5.2-1~ubuntu.22.04~jammy

I would concur with @felix-caboff that it appears that we just need to update the CLI binaries in this repo to get this working on docker v29. I think the current versions of the CLI in this repo do not use a docker API supported by docker v29.

[chrismaddalena]

@cobbr Thanks for confirming it's related to v28 » v29. The current Ghostwriter CLI binaries in the main branch aren't compatible with Docker v29 yet, but Ghostwriter CLI v0.3.0 is available and works with Docker v29. I'll tag a release to bump up the binaries here.

[smcgu]

Just to update here, the 0.3.0 binary of the CLI does not resolve this issue with v29 of Docker.

See: GhostManager/Ghostwriter_CLI#23 (comment)

[felix-caboff]

Hey @smcgu I see from your issue that you are experiencing the same issue as I originally posted about in this thread. I think it might be two different issues in roughly the same place so easily conflated. Have you had success by changing the syntax like I did?

[chrismaddalena]

Yes, the issue mentioned in the other thread appears to be related to this thread about ports. I still haven't run into any problems with Docker v29, the API changes, or the port syntax, but some of you are seeing different behavior with Docker v29 and the YAML. Maybe try some of the changes we are proposing in this draft PR: #770 — specifically the port sections (mostly deletions).

[felix-caboff]

I can confirm that implementing f3dc9f0 works for me. I didn't do the rest of the PR but this seems to be fine.