Deploy a standalone Docker application on the server

This title might seem odd: If you're already using Docker, why deploy it inside a VM?

Yes, generally speaking, few people have this kind of need: When deploying Docker containers at scale, we typically opt for more mature container orchestration solutions, such as K8S.

However, your data center might still be stuck in a virtual machine (VM)-based deployment model. If a certain application doesn't provide a VM deployment method and only offers a Docker image, you're in an awkward position. Setting up K8S just for one container is like using a sledgehammer to crack a nut. On the other hand, installing Docker specifically inside a VM loses the benefits of Docker's shared kernel and lightweight nature.

This pain often leads to a compromise of hybrid deployment, resulting in a situation where an operations engineer might have to create a dedicated virtual machine just for a single Docker image. In this case, the container effectively becomes a dedicated VM.

Although awkward, there are still some considerations to keep in mind in this scenario. This blog post briefly discusses this situation.

1 Challenge

This scenario is a very typical example of a hybrid deployment: a single data center contains two types of resource isolation units. Therefore, once this deployment model is adopted, it is no longer possible to manage resources directly from a single perspective, such as per VM or per container. Some workloads run in VMs, while others run in containers.

In this scenario, virtual machines and containers must be planned and managed appropriately to ensure system stability and reliability. At the same time, it is essential to immediately define a clear future direction and gradually transition toward a container-based deployment model.

2 Installing Docker

This step is quite straightforward: we are still discussing operations within the VM, and the VM requires Docker to be installed first.

On the target VM (assuming it is Ubuntu), run the command to install Docker:

curl -fsSL get.docker.com -o get-docker.sh
CHANNEL=stable sh get-docker.sh
rm get-docker.sh

3 Install Docker Desktop (Optional)

In some cases, you might need to debug quickly. If your environment has a GUI, Docker Desktop is a good tool.

Just be aware that this is very difficult to install. (It is almost the hardest thing to configure I have ever seen.)

Reference: https://github.com/docker/docker-credential-helpers/issues/102

3.1 Ensure Docker is Installed

First, make sure Docker is installed:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt install docker-ce docker-ce-cli docker-compose

3.2 Download and Install Docker Desktop

First, visit the Docker Desktop website: https://www.docker.com/products/docker-desktop/ to download the corresponding version. For example, the latest Ubuntu version is:

https://desktop.docker.com/linux/main/amd64/docker-desktop-4.22.0-amd64.deb

Install it using the following command:

sudo apt install ./docker-desktop-4.22.0-amd64.deb

This often fails to install. If it fails:

sudo apt install --fix-broken
sudo apt install --fix-missing

3.3 Install GPG and Pass

After installation, don't forget to install gpg and pass

sudo apt install gnupg2 pass

3.4 Installing Docker Credentials Helpers

You also need to install docker-credential-helpers separately. Download it here:

https://github.com/docker/docker-credential-helpers/releases

Download the version suitable for your operating system. For example, the latest download link for Linux is:

https://github.com/docker/docker-credential-helpers/releases/download/v0.8.0/docker-credential-pass-v0.8.0.linux-amd64

Grant read permissions to the downloaded file and copy it to /usr/bin/docker-credential-pass.

cd ~/Downloads
sudo mv ./docker-credential-pass-v0.8.0.linux-amd64 ./docker-credential-pass
sudo chmod u+x docker-credential-pass
sudo mv docker-credential-pass /usr/bin

3.5 Import or Generate a GPG Key

Don't forget, if you already have a gpg key, remember to import it:

Export the key on the old computer:

SIGNKEY=$(gpg --list-secret-keys --keyid-format LONG | grep sec | awk '{print $2}' | awk -F/ '{print $2}')
gpg --export-secret-keys $SIGNKEY > private.key

After copying the private.key file to the new computer or syncing it via a cloud storage service (note: this poses a security risk), import the key on the new computer:

gpg --import ~/Nextcloud/Storage/GPG/private.key

If you have never used gpg or don't have a gpg key, you need to generate one immediately:

gpg --generate-key

During the process above, you will need to enter your Email, Name, and the password for the new Key.

For detailed usage of gpg key, please refer to: My Blog

3.6 Trusting New Keys

If you encounter an error such as gnupg: There is no assurance this key belongs to the named user, you need to manually trust this key:

When using gpg --list-keys, you will see the following output:

pub   rsa3072 2018-10-07 [SC] [expires: 2020-10-06]
      1234567890ABCDEF1234567890ABCDEF12345678

Note that the long string is the Key ID! Copy it carefully.

Use the command:

gpg --edit-key <KEY_ID>
gpg> trust

Here, gpg will ask you:

1 = I don't know or won't say
2 = I do NOT trust
3 = I trust marginally
4 = I trust fully
5 = I trust ultimately
m = back to the main menu

Select 5. Trust is now complete.

gpg> quit

3.7 Setting the password

Run the command below to set the password

pass init 1234567890ABCDEF1234567890ABCDEF12345678

Note that the string above is your Key ID.

Then insert an item:

pass insert docker-credential-helpers/docker-pass-initialized-check

In the steps above, pass will prompt you for a password. Enter: pass is initialized

You can now check your configuration:

pass show docker-credential-helpers/docker-pass-initialized-check

3.8 Setting Up Docker

Now, modify the Docker login settings. Create or edit: vim ~/.docker/config.json:

{
    "credsStore": "pass"
}

You should be able to log in now. Use the command:

docker login

Simply enter your username and password.

After completing the login, check the status:

docker-credential-pass list

Docker configuration is now complete.

You can now use docker-desktop.

3.9 Open Docker Desktop

Open Docker Desktop. It should not prompt you to log in and should be ready to use directly.

Understanding the Volumes That Containers Need to Access

Before actually running the target container, be sure to understand what content the container needs to persist.

By "content that needs to be persisted," we refer to data or files that the container needs to access, such as configuration files, log files, databases, and so on. This data needs to remain consistent within the container and also needs to be shared between containers.

Therefore, before deploying the container, you need to determine which volumes need to be passed through to the VM. You can use Docker's -v or --mount options to mount directories on the VM into the container, allowing the container to access these volumes. For example:

mkdir -p /var/www/remotely
docker run -d --name remotely --restart unless-stopped -p 5000:5000 -v /var/www/remotely:/remotely-data immybot/remotely:latest 

This will create a container named "remotely" and mount the "/var/www/remotely" directory from the VM to the "/remotely-data" directory inside the container.

The directories commonly required by the containers I use are:

On a personal computer, it is not recommended to run containers with root privileges. This can cause issues when using certain GUI management tools in the future. Therefore, I use the following script to run external third-party containers:

echo "Container name?"
read C_NAME
I_NAME=$(echo $C_NAME | cut -d'/' -f2)

echo "Container path?"
read C_PATH

echo "Container port?"
read C_PORT


echo "Running:"
cd ~
docker run -d --name $I_NAME --restart unless-stopped -p $C_PORT:$C_PORT -v $I_NAME-storage:$C_PATH $C_NAME:latest

After mapping directories within the container that require persistence, such as /remotely-data, to volumes, updating the container becomes very easy. This is because the data and files inside the container are stored on the host rather than within the container itself. This makes updating and maintaining the container highly convenient, as operations can be performed directly on the host without worrying about data loss inside the container.

In addition, storing container data and files on the host also enables data sharing between containers. Multiple containers can share the same host directory, thereby facilitating data sharing and collaboration.

It is important to note that when using the -v or --mount options to mount a host directory into the container, you must ensure that the permissions and ownership of the host directory match the user and group inside the container. Otherwise, the container may be unable to access the mounted directory, which could prevent the application from running properly.

Determine Port Mapping

Similar to volumes, containers also require port mapping to access the external network. Containers running in a VM need to obtain an IP address within the VM's network stack, and their ports must also be mapped to the VM's ports so that the external network can access the containers.

You can use Docker's -p option to map container ports to VM ports. For example:

docker run --restart unless-stopped -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=<YourStrong!Passw0rd>' -p 1433:1433 -v sqlserver-instance-1:/var/opt/mssql -d mcr.microsoft.com/mssql/server:latest

This will map port 1433 of the container to port 1433 of the VM.

Managing Containers and VMs

In a hybrid deployment, containers and VMs must be managed separately. You can use Docker commands to manage containers, for example:

docker ps # 查看正在运行的容器
docker stop <container-id> # 停止容器
docker rm <container-id> # 删除容器

For VM management, you need to use VM management tools, such as VMware, VirtualBox, etc.

Ensure the container starts with the host

Sometimes, the host may unexpectedly restart. In such cases, it is necessary to ensure that containers also start automatically with the host. This can be achieved using Docker's --restart option to enable automatic container restarts. For example:

docker run -d --name my-container --restart unless-stopped my-image:latest

This will create a container named "my-container" and automatically start the container when the host boots.

If you have already created the container and need to set it to start automatically, you need to use the docker update command to update the container's options. Specifically, you can use the following command to set the container to restart automatically when the host starts:

docker update --restart unless-stopped my-container

This will update the options for the container named "my-container" so that it automatically restarts when the host starts, unless the container has been manually stopped. Note that this command only updates the container's options and does not restart the container. If you want to restart the container immediately to apply the new options, you can use the following command:

docker restart my-container

This will restart the container named "my-container" and apply the new options, ensuring it automatically restarts when the host starts.

Upgrading Containers

Although upgrading containers is straightforward, we must consider that the actual deployment unit remains the VM. Therefore, we still need to adopt some "brute-force" methods. Specifically, this involves first deleting the container, then downloading the new version, and finally running it with the same script. This approach ensures that the new container version correctly loads and utilizes persistent volumes while maintaining compatibility with previous versions. Additionally, this method makes container updates and maintenance highly convenient, as operations can be performed directly on the host without worrying about data loss within the container.

For example, the upgrade method for the Remotely container is as follows:

sudo docker stop remotely
sudo docker rm remotely
sudo docker pull immybot/remotely:latest
sudo docker run -d --name remotely --restart unless-stopped -p 5000:5000 -v /var/www/remotely:/remotely-data immybot/remotely:latest

This will stop and remove the old container named "remotely", then download the latest version of the "immybot/remotely" image from Docker Hub, and restart the container using the same script. When restarting the container, we also specify a persistent volume "/var/www/remotely" to ensure that data is not lost during container upgrades. This volume will be mounted to the "/remotely-data" directory inside the container.

It is important to note that the container's configuration and environment variables must also be correctly passed to the new version of the container to ensure its proper operation.

Note! This is only a very simplified approach for upgrading a single container on a single machine! If you are upgrading a container that is part of a larger application stack, this may involve coordinating the upgrade order of multiple containers to ensure the stability of the entire application stack, which is not covered in this blog post. There are many best practices for container upgrades, such as using a rolling upgrade strategy instead of upgrading all containers at once to minimize application downtime. Additionally, you can search for further information on how to use container orchestration tools (such as Kubernetes) to coordinate container upgrades.

Direct Container Operations

In emergency situations, you may need to perform operations inside the database container. In this case, you can use Docker commands to enter the container's interactive terminal to execute the necessary operations. Below is the command to enter the container:

docker exec -it <container_name> bash

Here, <container_name> is the name of the container you want to enter. This will open an interactive terminal where you can execute commands just as you would in a local terminal. When executing commands inside the container, be careful not to accidentally delete or modify files within the container to avoid data loss.

If you only need to execute a single command, you can use the following command:

docker exec <container_name> <command>

Here, <command> is the command to be executed inside the container. This will run the command within the container and print the results to the terminal.

Multiple Interrelated and Dependent Containers?

In some cases, you may need to deploy multiple containers with a single command.

• For example: You might need to start the web server and database simultaneously.
• For example: You might need to deploy Prometheus and Grafana at the same time.

In such cases, using the docker commands mentioned above to start and stop containers one by one is cumbersome and inefficient. Instead, use Docker Compose to define how your data center should run!

For example, the configuration required for my common containers is as follows:

version: '3'
volumes:
  caddy-store:
    driver: local
  prometheus-store:
    driver: local
  grafana-store:
    driver: local

networks:
  internal:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 10.0.0.0/8

services:
  prometheus:
    image: bitnami/prometheus
    ports:
      - "9090:9090"
    volumes:
      - prometheus-store:/etc/prometheus/
    restart: unless-stopped
    networks:
      - internal

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana-store:/var/lib/grafana
    restart: unless-stopped
    networks:
      - internal

  caddy:
    image: caddy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - caddy-store:/etc/caddy
    restart: unless-stopped
    networks:
      - internal

In the code above, I defined three volumes and mounted them to three containers, respectively.

I defined the ports that the three containers listen on, and also defined that they need to share an internal network.

If you need to start the container cluster:

docker-compose up -d

To update the container cluster:

docker-compose stop
docker-compose rm -f
docker-compose pull   
docker-compose up -d

Of course, in many cases, we can directly use systemd to manage docker-compose:

anduin@hub:/etc/systemd/system$ cat ./docker-compose.service
[Unit]
Description=%i service with docker compose
PartOf=docker.service
After=docker.service

[Service]
Type=oneshot
RemainAfterExit=true
WorkingDirectory=/etc/docker/compose/%i
ExecStart=/usr/local/bin/docker-compose up -d --remove-orphans
ExecStop=/usr/local/bin/docker-compose down

[Install]
WantedBy=multi-user.target

Conclusion

The above outlines some key considerations and operational steps for deploying Docker containers within a VM. It is important to note that in a hybrid deployment environment, containers and VMs must be managed separately. Additionally, careful planning and management of persistent volumes and port mappings for the containers are required to ensure system stability and reliability. At the same time, future development directions should be considered, with a gradual transition toward a container-based deployment model.