使用DockerFile构建Bare Metal镜像

Mutable和Immutable介绍

云原生的代表技术包括容器、服务网格、微服务、不可变基础设施和声明式API。容器技术的最大创造就是通过Dockerfile将应用打包为容器镜像，实现了不可变基础设施，标准化了应用模板。
在容器之前叫Mutable（可变的基础设施）在OS上部署应用，重启生效，可以随时进行修改。

容器技术就是Immutable的代表，引入容器镜像，通过Dockerfile将应用标准化打包为容器镜像，通过容器镜像启动容器，无法在容器中进行永久性修改，需要修改只能通过更新Dockerfile方式进行。
现如今Immutable理念也开始逐步从容器下沉到Bare Metal OS，通过Dockerfile构建Bare Metal镜像，实现Bare Metal OS Immutable。典型的开源项目技术Elemental项目

Elemental概述

Elemental 是一系列工具集合，主要是想通过 Kubernetes 实现集中式、完整的云原生操作系统构建和管理。集群节点操作系统是通过Elemental CLI通过容器映像构建和维护的，并使用Elemental CLI安装在新主机上。
Elemental Operator和Rancher System Agent使Rancher Manager 能够完全控制 Elemental 集群，从在节点上安装和管理操作系统到以集中方式配置新的 K3s 或 RKE2 集群。 Elemental项目组成

• elemental-toolkit - 包括一组操作系统实用程序，可通过容器启用操作系统管理。包括 dracut 模块、引导加载程序配置、cloud-init 自定义配置服务等。
• elemental-operator - 这连接到 Rancher Manager 并处理 machineRegistration 和 machineInventory CRD
• elemental-register - 这通过 machineRegistrations 注册机器并通过 elemental-cli 安装
• elemental-cli - 这会安装任何基于 elemental-toolkit 的衍生工具。实现OCI容器镜像构建为可在虚拟机、物理机、嵌入式设备运行的ISO镜像。
• rancher-system-agent - 在已安装的系统上运行并从 Rancher Manager 获取命令在系统上安装和运行rancher-agent，注册到Rancher中。

项目地址：https://github.com/rancher/elemental-toolkit

配置使用

在一台装有Docker的主机上进行提前准备项：

• 一台安装了Docker的主机
• Harbor镜像仓库
• EXSI或物理pc、服务器用于build后的ISO测试

使用Elemental-toolkit构建ISO流程

• 基础base镜像发行版：
  teal: SLE Micro for Rancher based one, shipping packages from Sle Micro 5.3.
  green: openSUSE based one, shipping packages from OpenSUSE Leap 15.4 repositories.
  blue: Fedora based one, shipping packages from Fedora 33 repositories
  orange: Ubuntu based one, shipping packages form Ubuntu 20.10 repositories
• 自定义镜像并制作OCI Image
• 在装有Docker的机器启动Elemental Build
  UEFI Boot，选择合适的实例类型
  Clout-init userdata 初始化
  Default user/pass: root/cos
• 升级自定义镜像
  elemental upgrade –no-verify –reboot -d niusmallnan/containeros\:dev

在安装了Docker的主机上创建/root/derivative目录。整体目录结构

/root/derivative/
├── Dockerfile
├── cloud-init.yaml
├── install.sh
├── installer.sh
├── k3s
├── k3s-airgap-images-amd64.tar.gz
├── manifest.yaml
├── nginx.yaml
├── overlay
│   └── iso
│       └── boot
│           └── grub2
│               └── grub.cfg
└── repositories.yaml

Demo架构

• 通过Elemental构建的OS中包含K3S
• 将需要部署的应用yaml放置到 /var/lib/rancher/k3s/server/manifests目录，K3S启动成功后会自动部署yaml启动应用。

下载K3S离线镜像包和CLI文件
https://github.com/k3s-io/k3s/releasesnginx.yaml文件用于k3s启动后加载此yaml文件，模拟演示是个应用

Dockerfile文件创建

ARG LUET_VERSION=0.32.0
FROM quay.io/luet/base:$LUET_VERSION AS luet
FROM registry.suse.com/suse/sle-micro-rancher/5.2
ARG ARCH=amd64
ENV ARCH=${ARCH}
# Copy the luet config file pointing to the upgrade repository
COPY repositories.yaml /etc/luet/luet.yaml
# Copy luet from the official images
COPY --from=luet /usr/bin/luet /usr/bin/luet
ENV LUET_NOLOCK=true
RUN luet install -y \
toolchain/yip \
toolchain/luet \
utils/installer \
system/cos-setup \
system/immutable-rootfs \
system/grub2-config \
system/base-dracut-modules
RUN  mkdir /var/lib/rancher/k3s/agent/images/ -p &&  mkdir /var/lib/rancher/k3s/server/manifests -p
COPY install.sh /system/oem/
COPY k3s /usr/local/bin
COPY nginx.yaml /system/oem/
COPY k3s-airgap-images-amd64.tar.gz /system/oem/
RUN  chmod a+x /usr/local/bin/k3s && chmod a+x /system/oem/install.sh
WORKDIR /system/oem
RUN  INSTALL_K3S_SKIP_START="true" INSTALL_K3S_SKIP_ENABLE="true" INSTALL_K3S_SKIP_DOWNLOAD="true" sh install.sh
## System layout
# Required by k3s etc.
RUN mkdir /usr/libexec && mkdir /usr/local/bin -p && touch /usr/libexec/.keep
# Copy custom files
# COPY files/ /
# Copy cloud-init default configuration
COPY cloud-init.yaml /system/oem/
# Generate initrd
RUN mkinitrd
# OS level configuration
RUN echo "VERSION=999" > /etc/os-release
RUN echo "GRUB_ENTRY_NAME=derivative" >> /etc/os-release
RUN echo "welcome to our derivative" >> /etc/issue.d/01-derivative

cloud-init文件创建，主要用于磁盘分区配置和登录用户名和密码配置
cloud-init.yaml

name: "Default settings"
stages:
initramfs:
# Setup default hostname
- name: "Branding"
hostname: "derivative"
# Setup an admin group with sudo access
- name: "Setup groups"
ensure_entities:
- entity: |
kind: "group"
group_name: "admin"
password: "x"
gid: 900            
# Setup network - openSUSE specific
- name: "Network setup"
files:
- path: /etc/sysconfig/network/ifcfg-eth0
content: |
BOOTPROTO='dhcp'
STARTMODE='onboot'                  
permissions: 0600
owner: 0
group: 0
# Setup a custom user
- name: "Setup users"
users:
# Replace the default user name here and settings
joe:
# Comment passwd for no password
passwd: "joe"
shell: /bin/bash
homedir: "/home/joe"
groups:
- "admin"
#authorized_keys:
# Replace here with your ssh keys
# joe: 
# - ssh-rsa ....
# Setup sudo
- name: "Setup sudo"
files:
- path: "/etc/sudoers"
owner: 0
group: 0
permsisions: 0600
content: |
Defaults always_set_home
Defaults secure_path="/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin:/usr/local/sbin"
Defaults env_reset
Defaults env_keep = "LANG LC_ADDRESS LC_CTYPE LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE LC_ATIME LC_ALL LANGUAGE LINGUAS XDG_SESSION_COOKIE"
Defaults !insults
root ALL=(ALL) ALL
%admin ALL=(ALL) NOPASSWD: ALL
@includedir /etc/sudoers.d            
commands:
- passwd -l root
# Setup persistency so k3s works properly
# See also: https://rancher.github.io/elemental-toolkit/docs/reference/immutable_rootfs/#configuration-with-an-environment-file
rootfs.after:
- name: "Immutable Layout configuration"
environment_file: /run/cos/cos-layout.env
environment:
VOLUMES: "LABEL=COS_OEM:/oem LABEL=COS_PERSISTENT:/var"
OVERLAY: "tmpfs:25%"
RW_PATHS: "/usr/local /etc /srv"
PERSISTENT_STATE_PATHS: >-
/etc/systemd
/etc/rancher
/etc/ssh
/etc/iscsi 
/etc/cni
/home
/opt
/root
/usr/libexec
/var/log
/var/lib/wicked
/var/lib/longhorn
/var/lib/cni
/usr/local/bin
PERSISTENT_STATE_TARGET: >-
/etc/systemd
/etc/rancher
/etc/ssh
/etc/iscsi
/etc/cni
/home
/opt
/root
/usr/libexec
/var/log
/var/lib/kubelet
/var/lib/wicked
/var/lib/longhorn
/var/lib/cni
/usr/local/bin
PERSISTENT_STATE_BIND: "true"
# Finally, let's start k3s when network is available, and download the SSH key from github for the joe user
network:
- name: "Deploy cos-system"
commands:
- elemental install /dev/sda 
- systemctl enable k3s && systemctl start  k3s
after-install:
- name: "install k3s"
commands:
- mount /dev/sda5 /var
- mkdir -p  /var/lib/rancher/k3s/agent/images/  && mkdir /var/lib/rancher/k3s/server/manifests -p
- cp /system/oem/k3s-airgap-images-amd64.tar.gz  /var/lib/rancher/k3s/agent/images/
- cp /system/oem/nginx.yaml /var/lib/rancher/k3s/server/manifests
- reboot

创建manifest.yaml文件定义OS启动引导所需要文件

iso:
rootfs:
- channel:system/cos
uefi:
- channel:live/grub2-efi-image
image:
- channel:live/grub2
- channel:live/grub2-efi-image
label: "COS_LIVE"
name: "cOS-0"
# Raw disk creation values start
raw_disk:
x86_64:
# which packages to install and the target to install them at
packages:
- name: channel:system/grub2-efi-image
target: efi
- name: channel:system/grub2-config
target: root
- name: channel:system/grub2-artifacts
target: root/grub2
- name: channel:recovery/cos-img
target: root/cOS
repositories:
- uri: quay.io/costoolkit/releases-teal
arch: "x86_64"

创建repositories.yaml文件

logging:
color: false
enable_emoji: false
general:
debug: false
spinner_charset: 9
repositories:
- name: "cos"
description: "cOS official"
type: "docker"
enable: true
cached: true
priority: 1
verify: false
urls:
- "quay.io/costoolkit/releases-green"

创建grub文件配置内核引导
在/root/derivative/overlay/iso/boot/目录创建grub2grub.cfg文件

search --no-floppy --file --set=root /boot/kernel
set default=0
set timeout=10
set timeout_style=menu
set linux=linux
set initrd=initrd
if [ "${grub_cpu}" = "x86_64" -o "${grub_cpu}" = "i386" -o "${grub_cpu}" = "arm64" ];then
if [ "${grub_platform}" = "efi" ]; then
if [ "${grub_cpu}" != "arm64" ]; then
set linux=linuxefi
set initrd=initrdefi
fi
fi
fi
if [ "${grub_platform}" = "efi" ]; then
echo "Please press 't' to show the boot menu on this console"
fi
set font=($root)/boot/${grub_cpu}/loader/grub2/fonts/unicode.pf2
if [ -f ${font} ];then
loadfont ${font}
fi
menuentry "cOS" --class os --unrestricted {
echo Loading kernel...
$linux ($root)/boot/kernel.xz cdroot root=live:CDLABEL=COS_LIVE rd.live.dir=/ rd.live.squashimg=rootfs.squashfs console=tty1 console=ttyS0 rd.cos.disable
echo Loading initrd...
$initrd ($root)/boot/rootfs.xz
}
if [ "${grub_platform}" = "efi" ]; then
hiddenentry "Text mode" --hotkey "t" {
set textmode=true
terminal_output console
}
fi

先构建镜像

docker build -t 172.16.1.208/library/example:v4.0 .

镜像要上传到镜像仓库才能build iso

docker push 172.16.1.208/library/example:v4.0

构建ISO

注：目前只支持公开的镜像仓库，不支持私有的镜像仓库
https://github.com/rancher/elemental-cli/issues/389构建完成，生成此cOS-0.iso镜像文件

将cOS-0.iso下载到ESXI或其他虚拟化平台也可以刻录U盘直接安装物理机。配置选4c4G 60G磁盘
加载ISO后自动分区，自动进行初始化，安装系统，完成后自动重启进入系统。密码ssh账号密码joe/joe
在安装后的系统查看已经部署好的K3S。 查看自动部署的应用
访问应用因为整个系统都限制了修改，所以在操作系统任何目录执行修改命令都无法修改。如

总结

通过Elemental实现了操作系统为不变基础设施，同时也可以将我们传统的OS带入云原生，通过Dockerfile去构建，通过CICD去统一发版维护，目前能想到的一个比较大的应用场景在于，一个是边缘场景，边缘设备操作系统批量部署安装。另外就是一些to b的客户将自己业务+容器编排和OS通过Elemental构建打包，直接到客户现场加载ISO就部署完了，开箱即用。另外OS也可以标准化，统一化管理。