概述

最近，公司搬回来两台性能非常可观的服务器。在众人怂恿下，我带头用Citrix Xenserver 6.2这个开源版本的虚拟机帮两台服务器进行了裸虚拟化。暨此机会，可以把在开发机器虚拟机上部署的Docker container移植到虚拟机上，以便我开发机能够流畅运行。由于本人获取Docker镜像的途径是通过Docker Hub来下载的，加之工作环境是内网（air-gapped），所以以往都是通过docker save > xxx.tar这样形式以文件的形式对镜像进行传输。细思之下，想起以前的一个用例：一个公司在有96台X86服务器的情况下，在一台机器的Docker容器中编译Spark，并将镜像通过Registry分发至其余的机器。如果以Docker save/load的方式不可谓不繁琐。加之在内网中搭设Registry可以方便镜像管理，随即通过查看相关文档将想法付诸实践。本文中所有的Linux操作均在root用户权限下操作。

第一、安装Docker以及下载Registry镜像

阅读文档Deploy a registry server后知道，Docker将Registry作为一个Docker Image发布，并且以Container的形式运行。那么首先要做的，则是在机器上安装Docker CE（Community Edition）。由于本人所设定情景是在内网，所以上述文档中的在线安装方法apt-get install方法将失效，我们转由通过“Install from a package”的方法来进行安装，即通过下载地址选择你的host系统的安装文件版本来进行安装。由于本人使用的是Ubuntu 16.04，选择Download 文件进行下载。
在下载文件所在路径下输入命令

dpkg -i ./docker-ce_17.03.0~ce-0~ubuntu-xenial_amd64.deb

，这就样Docker就能很好地在host机上运行了。

等一下……

按照host系统的不同，可能出现不同docker-ce依赖包缺失的问题，比如我个人用的Ubuntu-16.04版本，则出现了

root@ubuntu-1:/opt# dpkg -i ./docker-ce_17.06.0~ce-0~ubuntu_amd64.deb
Selecting previously unselected package docker-ce.
(Reading database … 58701 files and directories currently installed.)
Preparing to unpack …/docker-ce_17.06.0~ce-0~ubuntu_amd64.deb …
Unpacking docker-ce (17.06.0~ce-0~ubuntu) …
dpkg: dependency problems prevent configuration of docker-ce:
docker-ce depends on libltdl7 (>= 2.4.6); however:
Package libltdl7 is not installed.

dpkg: error processing package docker-ce (–install):
dependency problems – leaving unconfigured
Processing triggers for man-db (2.7.5-1) …
Processing triggers for ureadahead (0.100.0-19) …
Processing triggers for systemd (229-4ubuntu4) …
Errors were encountered while processing:
docker-ce

的错误。正如错误信息所提示，的确缺少了 docker-ce依赖包libltdl7，只要按照错误提示一一安装上去，Docker 则应该可以顺利运行。本人曾经经历过应用迁移，原先以为只管把原编译好的文件往新系统拷贝了则了事了，后来下载source文件想要编译时则发觉系统连GCC也没有。事实上我是需要面对各种系统底层的软件依赖的，Docker 则能很好地避免这类问题，甚至可以套用一句话：“一次编译，处处运行”。只要都能够运行Docker进程，则无论host系统环境如何都能够将Container顺利运行起来。

之前论述了，内网部署是一个前提条件，所以只能依靠一个能够访问外网并能访问 Docker hub 的机器 PC1 下载Registry镜像至本地，并通过save/load方法将导出的tar镜像文件导入内网运行Docker进程的host PC2。
PC1：
docker pull registry//等待直至完成
docker save registry>registry.tar
将registry.tar通过U盘或者scp或者网络共享的什么方式，传送到内网机器PC2.
PC2：
docker load <registry.tar
docker images
如果列表显示有Registry镜像，则可以接着运行Registry Container了。

第二，为Registry签名证书，以及启动参数

接下来应该为Registry创建ssl自签名证书。
为什么要为Registry创建自签证书？
因为Docker进程对Registry通讯，默认是需要对Registry进行TLS认证的，所以可以通过对Registry所运行的机器的域名进行自签名。当然，也可以通过调整Docker进程对未认证的Registry进行通讯，详细可参考文档The Document。

正如上述文档内提及，生成自签名的证书的命令如下：
openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/xxx.com.key -x509 -days 365 -out certs/xxx.com.crt
接着会进入制证程序，会出现以下交互命令：

Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:xxx.com
Email Address []:

其中

Common Name(e.g. server FQDN or YOUR name)

一项是值得注意的，他代表了该证书所对应服务器的域名，其中xxx.com是一个自定义的域名。
生成了两个文件：xxx.com.crt/xxx.com.key,将这两个文件至于/certs目录下。
接下来将列出Docker Registry启动命令及参数
docker run -d -p 5000:5000 \
-v /certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:80 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/xxx.com.crt -e REGISTRY_HTTP_TLS_KEY=/certs/xxx.com.key\
--restart=always --name registry registry
其中：
-d:后台运行，退出了命令行之后此Docker进程继续在后台运行，大多数的Container都有这个命令；
-p 5000：5000: 将host机的5000端口映射到Docker container的5000端口，后者的端口一般是镜像的搭建者规定的；
-v /certs:/certs: 将host机文件系统中/certs文件夹挂载至Docker container文件系统中的/certs文件夹。这个方式经常用于让Docker container和host有共享目录，方便文件传输共享。比如我们可以将host系统的指定文件夹挂载到tomcat container,tomcat目录下的webapp目录下，则可以每次都将构建好的项目放在host指定的文件夹下，重启container即可更新项目。
-e :设置Docker container 中的系统变量，其中的系统变量应该按照Docker镜像说明来设置；上述REGISTRY_HTTP_TLS_CERTIFICATE和REGISTRY_HTTP_TLS_KEY分别指向/certs/xxx.com.crt和/certs/xxx.com.key,亦即上述生成并置于被挂载文件夹中的自签文件。
--restart=always:在重启之后启动该Docker container。当host重启之后，Docker进程随即启动，在启动命令中有--restart=always的container随即启动。
--name registry:命名该container为“Registry”，而不是由Docker自己分配名字。
至此，该Docker Registry已经能够正常运作了。

但是……
1、生成的证书，是针对域名的。而我们在内网部署何来的域名呢？如何解析呢？我个人所用的方法则是在Docker host机器上，或者在提供DNS服务的机器上，将域名指向运行Registry container的机器的IP上。本人使用的是ubuntu16.04，在/etc/hosts文件中，“127.0.0.1 localhost”一条下添加“(IP) xxx.com”，则每次提交或者拉取镜像时，都能够将域名解析至指定IP了。
2、在Docker Registry部署教程中，启动命令中是有一行命令:-v /mnt/registry:/var/lib/registry，这条命令是将host的指定目录挂载到container的/var/lib/registry 亦即Registry存储Docker镜像的地方。可以通过对/mnt/registry/中文件的删减来管理registry中存储的镜像。
3、如果在证书校验方面仍然出现问题，则有可能是某些Docker进程需要系统级别的认证，正如教程The Tutorial所说，在我使用的ubuntu下，将在Registry发布机器上生成的xxx.com.crt拷贝至/usr/local/share/ca-certificates/下，然后执行update-ca-certificates，则可解决此问题。

第三，推送与拉取

当我们的Registry已经部署好之后，就能在客户端进行拉取与推送了。这里以我有一个数据挖掘平台jupyterhub的镜像为例：jupyterhub的本名，就为jupyterhub。此时我们需要对jupyterhub重新命名：
docker tag jupyterhub xxx.com:5000/jupyterhub
docker push xxx.com:5000/jupyterhub
这样，则将jupyterhub镜像推送至xxx.com，亦即是经DNS解析为Registry的机器上了。
相似地，我们pull一个镜像
docker pull xxx.com:5000/jupyterhub
则会在本地得到一个名为xxx.com:5000/jupyterhub的jupyterhub的镜像。
这里仍然需要注意，本地的Docker进程可能仍然需要系统级别的TLS认证，按照第二步中导入证书到系统即可。

至此，该内网使用的Docker Registry服务已经能够正常使用了。

可能可以这样……

上文用openssl创建的自签证书，是专门为Docker Registry 创建的。可以使用Ngnix作为反向代理，为所有需要自签证书的服务提供证书验证服务。

English Version:

Summary

Recently, I received two high-performance servers. After discussed, I virtualizated the servers by using Citrix Xenserver 6.2, which is a open-sourced edition. By this chance, I want to immigrate the Docker containers deployed on the virtual machine operating on my development PC to the remote virtual machine. I used to obtain the Docker images by download from Docker hub and then deploy in a air-gapped environment, so the Docker save/load commands are frequently used. I heard a user case: There is a company owned 96 x86 servers. The engineers plan to deploy a Spark data processing engine on the servers. They used 2 hours to compile the Spark on Docker on one machine, then pushed the image to the registry and pulled by the others servers, after handy configured, a whole Spark system worked. My method is obviously inefficient and more important, construct a registry in a air-grapped environment is very convenient for user to deploy Docker container. So I followed the instruction from the Docker official site and build the registry. I want to notify that all the operation in the Linux narrate below are under the root privilege.

First, install the Docker and download the Registry image.

After reading the document Deploy a registry server, registry is released as a Docker image, and operating as a Docker container. So first to do, is install the Docker CE(Community Edition) on one server. As the background narrated above, so the normal method, install the Docker through “apt-get install”, is unavailable. Then I switch to the other method “Install from a package”. I used Ubuntu 16.04 as my host OS, according to the document, I can obtain the install file. After the install file downloaded, then install it:
dpkg -i ./docker-ce_17.03.0~ce-0~ubuntu-xenial_amd64.deb.
After installation, the Docker should be running on the host.

Wait for a minute……
Depend on the system you chosen, some different lib-dependency problems may appear. For example, there was a dependency-error appear in my Docker install process:

root@ubuntu-1:/opt# dpkg -i ./docker-ce_17.06.0~ce-0~ubuntu_amd64.deb
Selecting previously unselected package docker-ce.
(Reading database … 58701 files and directories currently installed.)
Preparing to unpack …/docker-ce_17.06.0~ce-0~ubuntu_amd64.deb …
Unpacking docker-ce (17.06.0~ce-0~ubuntu) …
dpkg: dependency problems prevent configuration of docker-ce:
docker-ce depends on libltdl7 (>= 2.4.6); however:
Package libltdl7 is not installed.

dpkg: error processing package docker-ce (–install):
dependency problems – leaving unconfigured
Processing triggers for man-db (2.7.5-1) …
Processing triggers for ureadahead (0.100.0-19) …
Processing triggers for systemd (229-4ubuntu4) …
Errors were encountered while processing:
docker-ce

As the error log illustrated, lack of the package libltdl7 cause such issue. In my opinion, install the missing package one by one will solve such type of problems and then Docker will operate properly. Once upon a time, I was assigned a application immigrate task. I thought it would work just copy the file to the new machine, but didn’t work. In fact when I try to compile the source file I found that there was no GCC installed on the OS. It is very common problem that is the software dependency problem, Docker will handle such problem properly like the concept “Compile once, run everywhere”.
As mentioned above, the system running in a air-gapped system. The registry image should be obtained by a machine which can access to Docker hub, and then transport the image by Docker save/load method to the air-gapped system. I label the Docker hub accessible machine as PC1, and the operating machine in the air-gapped environment as PC2
In the PC1:
docker pull registry//wait till end
docker save registry>registry.tar
transport the registry.tar to the PC2 by means like USB drive or scp or net sharing.
In the PC2:
docker load<registry.tar
docker images
If the registry showed in the list， we can go on to run a registry container.

Second, sign a certificate for registry, and start a registry container.

Why should I sign a certificate for the registry container?
Because the Docker system communicate with registry by TLS certificated by default. You can get a certificate from the machine which registry is running on. By the document, you can also let the registry and the client Docker running in a un-certificated situation.
Here is the command to generate the certificate:
openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/xxx.com.key -x509 -days 365 -out certs/xxx.com.crt

There will be some options can be fill as below:

Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:xxx.com
Email Address []:

Among those options, the

Common Name (e.g. server FQDN or YOUR name) []:xxx.com

should be cared more. The certificate is paired with it self-define domain.
Then you can place the xxx.com.crt and xxx.com.key to /certs directory.
Let me show the docker running command of the registry container.
docker run -d -p 5000:5000 \
-v /certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:80\

-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/xxx.com.crt\

-e REGISTRY_HTTP_TLS_KEY=/certs/xxx.com.key\
--restart=always --name registry registry

-d:run the Docker container in the background, most of the Docker containers run with it because the containers should run after the user exit the console.
-p 5000:5000: Mapping the host socket 5000 to the container’ 5000 socket, the later socket usually pointed by the image maker.
-v /certs:/certs: Mount the host /certs directory to the container /certs directory. Such option is widely used by sharing content between host system and container system. There is a example: We can mount a host directory to the tomcat container, mount to the web app directory in the tomcat directory. So that we can put the built app in the host directory and restart the container, the application will be up-dated.
-e: define the environment variables. Most of the environment variables were defined by images maker and should be configure according the menu. The REGISTRY_HTTP_TLS_CERTIFICATE and REGISTRY_HTTP_TLS_KEY assigned /certs/xxx.com.crt and /certs/xxx.com.key, the self-certificated files just generated by openssl.
--restart=always:always restart when the docker daemon restart.
--name registry: assign a name to the container instead of assign by Docker daemon.
Now, the Docker Registry should operating.

Something I want to mention:
1 The certificate files are paired with specified domain like xxx.com we used. But we deploy such application in a air-gapped environment. In such condition IP address is seen as address rather than domain. My solution is to place the domain-IP mapping in the local DNS file or the DNS server. For example, I place the mapping in the local DNS file, I using Ubuntu 16.04 so I edit the file /etc/hosts, place “(IP) xxx.com” under the row “127.0.0.1 localhost”. After the edit, any time you pull or push images to xxx.com, the DNS will help redirect to the IP you wrote down.
2 In the official registry deployment document, there is a additional option -v /mnt/registry:/var/lib/registry. The option mount the /mnt/registry to /var/lib/registry which is the store of images. So we can manage the images in the registry via manipulate the file in the /mnt/registry.
3 If there is anything still go wrong of the authentication, maybe the Docker daemon need OS-level authentication. As the document goes, update the OS certificate will work. I using Ubuntu, place the xxx.com.crt just generated to /usr/local/share/ca-certificates/, and execute the command “update-ca-certificates”.

Third, push and pull.

In the second step, I have finish the deployment of the registry, so the client Docker can push the images to or pull the images from the Registry. Now I use the jupyterhub(a data mining platform) image to illustrate how to push and pull. If we want to push a Docker to a self-instructed Registry, we should appoint where the image to push. Now we should attach a new tag to the jupyterhub image:
docker tag jupyterhub xxx.com:5000/jupyterhub
docker push xxx.com:5000/jupyterhub
So, the Docker image will be pushed to the registry.
Similarly,
docker pull xxx.com:5000/jupyterhub
will pull a jupyterhub images from remote registry to localhost.
There is a point still should be mentioned that may the Docker running local require OS-level TLS authentication, follow the second step to solve it on localhost.

At last, the air-grapped environment Docker Registry will operate properly.

One more thing……
The openssl certificate is generated for Docker Registry, using Nginx for reverse proxy may let the pair of certificate service multiple systems.

由 mulder 于 2019-01-25 0:08 更新