Docker on three machines with swarm from scratch

2017-10-11 23:22:00

I found docker interesting technology something about 2 years ago, at this point I want to use it on larger scale then one machine but not necessary using cloud or kubernetes. I just want to be able to create 3 physical machines cluster ( swarm ) and deploy applications ( services ) on it.

1. Installation Of Docker

First I will install latest docker version just by using those commands:

curl -X GET https://get.docker.com/ > get-docker.sh
sudo sh get-docker.sh

then I will add my user to the docker by typing :

sudo usermod -aG docker USERNAME

2. Configuration Of Docker Deamon

Second I need to be able to run docker as deamon on public port and let it be 2375. To do it I will reconfigure docker daemon

sudo service docker stop
sudo vi /etc/docker/daemon.json

{
  "debug":true,
  "bip":"172.16.0.1/16",
  "hosts":["unix:///var/run/docker.sock", "tcp://0.0.0.0:2375"]
}

Also to get rid of -H fd:// in systemctl modify docker script

sudo systemctl edit docker

And paste:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd

and finally start docker using

sudo service docker start

Probably I could also use some proxy since dockerd is simply http server with json api, or at least som environment variable, but this way I can learn something more about systemd shitty configurations.

3. Installation Of docker-compose And docker-machine

Third I want to also use docker-compose to manage my multiple docker container application and since it is python package I will install it with :

pip install docker-compose

Last but not least I want to setup docker-machine to maintain my cluster. I will grab latest version from https://github.com/docker/machine/releases and just apply some console magic (I am installing version for raspberry pi 3):

wget https://github.com/docker/machine/releases/download/v0.12.2/docker-machine-Linux-armhf
sudo cp docker-machine-Linux-armhf /usr/local/bin/docker-machine
sudo chmod +x /usr/local/bin/docker-machine

So now since I got all lame docker prefixed shit in place including docker,dockerd,docker-machine,docker-compose I can add some machines to maintain and finally start to swarm my machines with containers.

4. SSH Keys

So I will be adding 3 machines and I want to access to them without password.

I will create simple configuration in /etc/hosts file for now:

192.168.1.101   pi01
192.168.1.102   pi02
192.168.1.103   pi03

First let's create key

ssh-keygen -t rsa -b 4096

I will add my key to my hosts using:

cat ~/.ssh/id_rsa.pub | ssh pi@pi01 'cat >> .ssh/authorized_keys'
cat ~/.ssh/id_rsa.pub | ssh pi@pi02 'cat >> .ssh/authorized_keys'
cat ~/.ssh/id_rsa.pub | ssh pi@pi03 'cat >> .ssh/authorized_keys'

5. Creation Of Docker Machine

then create one machine:

docker-machine create --driver=generic --generic-ssh-user=pi --generic-ssh-key=/home/pi/.ssh/id_rsa --generic-ip-address=pi01 --generic-engine-port=2375 pi01

the output for raspberry pi 3 should be something like this (first failure):

Detecting the provisioner...
Error creating machine: Error detecting OS: OS type not recognized

But it should be running so using docker-machine ls I will get (second failure):

NAME   ACTIVE   DRIVER    STATE     URL               SWARM   DOCKER    ERRORS
pi01   -        generic   Running   tcp://pi01:2375           Unknown   Unable to query docker version: Get https://pi01:2375/v1.15/version: http: server gave HTTP response to HTTPS client

Ok so I need to use https instead of http with my docker deamon to fix machine listings and also provision my machines manually.

6. Certificate Authority For HTTPS Traffic.

Let's start from locating cerfiticate authority (CA) and creating certificate for each machine. I am using docker-machine so files should be around here:

~/.docker/machine/certs/ca-key.pem
~/.docker/machine/certs/ca.pem

If I didn't have those files I can create them from command line:

openssl genrsa -aes256 -out ca-key.pem 4096
openssl req -new -x509 -days 365 -key ca-key.pem -sha256 -out ca.pem

Ok so now I will create certificate for pi01 machine with ip 192.168.1.101 and 127.0.0.1:

openssl genrsa -out key.pem 4096
openssl req -subj "/CN=pi01" -sha256 -new -key key.pem -out server.csr
echo subjectAltName = DNS:pi01,IP:192.168.1.101,IP:127.0.0.1 >> extfile.cnf
echo extendedKeyUsage = serverAuth >> extfile.cnf
openssl x509 -req -days 1000 -sha256 -in server.csr -CA ~/.docker/machine/certs/ca.pem -CAkey ~/.docker/machine/certs/ca-key.pem -CAcreateserial -out cert.pem -extfile extfile.cnf

7. Docker Deamon With HTTPS Trafic

After that I can modify configuration of my docker daemon:

sudo vi /etc/docker/daemon.json

with configuration:

{
  "debug":true,
  "tls":true,
  "tlsverify":true,
  "tlscacert":"/home/pi/.docker/certs/ca.pem",
  "tlscert":"/home/pi/.docker/certs/cert.pem",
  "tlskey":"/home/pi/.docker/certs/key.pem",
  "bip":"172.16.0.1/16",
  "hosts":["unix:///var/run/docker.sock", "tcp://0.0.0.0:2376"]
}

notice that I intentionally changed port from 2375 to 2376 so I need to change my docker-machine config file:

vi ~/.docker/machine/machines/pi01/config.json
# change section :
"Driver": {
    "EnginePort": 2376,

Now everything is all right (except active flag) with first machine:

docker-machine ls
NAME   ACTIVE   DRIVER    STATE     URL               SWARM   DOCKER        ERRORS
pi01   -        generic   Running   tcp://pi01:2376           v17.05.0-ce

8. Manual Provisioning Summary

Ok so since I cannot provision two other raspberry pi I will add certificates for each host and change systemctl manually. Again this could be done using environment variables

DOCKER_HOST="tcp://0.0.0.0:2376"

So for me it is (in short provisioning and certificates for pi02):

add machine

docker-machine create --driver=generic --generic-ssh-user=pi --generic-ssh-key=/home/pi/.ssh/id_rsa --generic-ip-address=pi02 --generic-engine-port=2376 pi02

add directory and create certs for ip 192.168.1.102 hostname pi02

cd ~/.docker/machine/machines/pi02
mkdir certs
cd certs
openssl genrsa -out key.pem 4096
openssl req -subj "/CN=pi02" -sha256 -new -key key.pem -out server.csr
echo subjectAltName = DNS:pi02,IP:192.168.1.102,IP:127.0.0.1 >> extfile.cnf
echo extendedKeyUsage = serverAuth >> extfile.cnf
openssl x509 -req -days 365 -sha256 -in server.csr -CA ~/.docker/machine/certs/ca.pem -CAkey ~/.docker/machine/certs/ca-key.pem -CAcreateserial -out cert.pem -extfile extfile.cnf
cp ~/.docker/machine/certs/ca.pem .

copy to remote machine

rsync -Paz --rsync-path="mkdir -p /home/pi/.docker/certs && rsync" * pi@pi02:~/.docker/certs

to remote machine

docker-machine ssh pi02

edit daemon.json

sudo vi /etc/docker/daemon.json

{
  "debug":true,
  "tls":true,
  "tlsverify":true,
  "tlscacert":"/home/pi/.docker/certs/ca.pem",
  "tlscert":"/home/pi/.docker/certs/cert.pem",
  "tlskey":"/home/pi/.docker/certs/key.pem",
  "bip":"172.16.0.1/16",
  "hosts":["unix:///var/run/docker.sock", "tcp://0.0.0.0:2376"]
}

edit systemctl

sudo systemctl edit docker

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd

restart docker service

sudo service docker restart

I won't repeat those steps for pi03. After running those commands I have something like this:

NAME   ACTIVE   DRIVER    STATE     URL               SWARM   DOCKER        ERRORS
pi01   -        generic   Running   tcp://pi01:2376           v17.05.0-ce   
pi02   -        generic   Running   tcp://pi02:2376           v17.05.0-ce   
pi03   -        generic   Running   tcp://pi03:2376           v17.05.0-ce

9. Configuration of docker-swarm

Now it's time for swarming. I will start with one host that will act as manager.

docker swarm init

I will add other hosts as guests by simply execute command printed to console on other hosts

docker swarm join --token tokenstring swarmip:port
This node joined a swarm as a worker.

And after that that to see my cluster I can run command:

docker node ls

So there it is.

ID      HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS
xxx *   pi01                Ready               Active              Leader
xxx     pi02                Ready               Active              
xxx     pi03                Ready               Active

if someone want to add some machines to the swarm later or simply want to add more managers to retrieve specific command just invoke:

docker swarm join-token worker
docker swarm join-token manager

Next I will try to write about how to build docker applications, use docker-compose and deploy it against swarm cluster.

Home

About