Use Kubean in offline cases

Reminder

• For RHEL 8.4 series, the pre-installed fuse package may be uninstalled due to package dependency issues.

Preparation

1. Services requiring pre-deployment:

   • Document resource service minio
   • Image registry services: docker registry（2.7 below） or harbor

2. Necessary tools to be installed:

   • A tool for importing image files: skopeo, Required >=1.9.2; podman, Required >= 4.4.4
   • A tool for importing binary files: minio client

3. Deploy Kubean by Helmkubean

Download offline resources

The Github Releases page allows us to download the offline resources for the version we want.

Basic instructions for offline resources:

├── files.list                                  # List of file contents
├── files-${tag}.tar.gz                         # File compression package, including import scripts
├── images.list                                 # List of image contents
├── images-${tag}.tar.gz                        # File compression package, including import scripts
└── os-pkgs-${linux_distribution}-${tag}.tar.gz # Compressed package of each system, including import script

Importing offline resources into the proper service

Import binary resources

First unzip the files-${tag}.tar.gz file, which contains:

files/
├── import_files.sh       # This script is used to import binary files into the minio file service
└── offline-files.tar.gz  # Compressed package of binary 

Execute the following command to import the binary file into the minio service:

MINIO_USER=${username} MINIO_PASS=${password} ./import_files.sh ${minio_address}

• minio_address is the minio API Server address, typically on port 9000, e.g. http://1.2.3.4:9000

Import images

It's required to unzip the images-${tag}.tar.gz file, which contains:

images/
├── import_images.sh       # This script is used to import binary files into the minio file service, and can merge multi-platform images
└── offline-images.tar.gz  # Compressed package of image file

Execute the following command to import the image file into the Docker Registry or the Harbor image repository service:

# password-free mode
$ REGISTRY_SCHEME=http REGISTRY_ADDR=${registry_address} ./import_images.sh

# username-password mode
$ REGISTRY_SCHEME=https REGISTRY_ADDR=${registry_address} REGISTRY_USER=${username} REGISTRY_PASS=${password} ./import_images.sh

• REGISTRY_ADDR is the address of the mirror repository, e.g. 1.2.3.4:5000
• REGISTRY_USER and REGISTRY_PASS need to be set when username and password authentication exists for the mirror repository

Import OS packages

Note

• OS Packages for Centos / Redhat / Kylin / Ubuntu distributions are currently supported.
• The OS Package of UnionTech V20 series needs to be built manually, see README for the build method.

It's required to unzip the os-pkgs-${linux_distribution}-${tag}.tar.gz file, which contains:

os-pkgs
├── import_ospkgs.sh       # This script is used to import os packages to the minio file service
├── os-pkgs-amd64.tar.gz   # os packages for the amd64 architecture
├── os-pkgs-arm64.tar.gz   # os packages for the arm64 architecture
└── os-pkgs.sha256sum.txt  # The sha256sum checksum file for the os packages

Execute the following command to os packages into the minio file service:

MINIO_USER=${username} MINIO_PASS=${password} ./import_ospkgs.sh ${minio_address} os-pkgs-${arch}.tar.gz

Create offline sources

Create ISO image source

The following [Create local ISO image source] and [Create online ISO image source] only need to execute one of them.

Create a local ISO image source

OS Packages are primarily used to resolve docker-ce installation dependencies, but for offline deployments, other packages from the distribution may be used, and a local ISO image source will need to be created.

Note

It's required to download the ISO system distribution image for the host in advance, currently only the ISO image source for the Centos distribution is supported; This operation needs to be performed on each cluster that creates kubernetes nodes.

The script artifacts/gen_repo_conf.sh can be used to mount the ISO image and create the Repo configuration file by executing the following command:

# Basic format
./gen_repo_conf.sh --iso-mode ${linux_distribution} ${iso_image_file}

# Execute the script to create the ISO image source
./gen_repo_conf.sh --iso-mode centos CentOS-7-x86_64-Everything-2207-02.iso
# Check ISO image mounts
df -h | grep mnt
/dev/loop0               9.6G  9.6G     0 100% /mnt/centos-iso
# Check ISO image source configuration
cat /etc/yum.repos.d/Kubean-ISO.repo

[kubean-iso]
name=Kubean ISO Repo
baseurl=file:///mnt/centos-iso
enabled=1
gpgcheck=0
sslverify=0

Create an online ISO image source

To import the image source from the ISO into the minio server, use the script artifacts/import_iso.sh

MINIO_USER=${username} MINIO_PASS=${password} ./import_iso.sh ${minio_address} Centos-XXXX.ISO

Create the following file for the host /etc/yum.repos.d/centos-iso-online.repo to use the online ISO image source:

[kubean-iso-online]
name=Kubean ISO Repo Online
baseurl=${minio_address}/kubean/centos-iso/$releasever/os/$basearch
enabled=1
gpgcheck=0
sslverify=0

• It's required to replace ${minio_address} with the minio API Server address

For a SLES 15 SP7 Full ISO the source path is sles-iso, and the repo file belongs in /etc/zypp/repos.d/. Full SLE media keep each module in its own repository directory, so every module needs its own entry. The import copies Product-SLES, Module-Basesystem, Module-Server-Applications and Module-Python3, which is what Kubespray installs from:

[kubean-iso-online-basesystem]
name=Kubean ISO Repo Online Basesystem
baseurl=${minio_address}/kubean/sles-iso/$releasever/os/$basearch/Module-Basesystem
enabled=1
autorefresh=1
gpgcheck=0

[kubean-iso-online-python3]
name=Kubean ISO Repo Online Python3
baseurl=${minio_address}/kubean/sles-iso/$releasever/os/$basearch/Module-Python3
enabled=1
autorefresh=1
gpgcheck=0

• On SLES 15 SP7 $releasever resolves to 15.7, read from /etc/products.d/baseproduct.
• Module-Python3 provides python311, which the modern spray-job image requires. See Python versions and image selection.

Create extras software sources

Currently only supported on Centos distributions

When installing a K8S cluster, it also relies on extras, such as container-selinux, which are not always provided in the ISO image source. This is supplemented by the OS packages offline package, which requires us to create an extra repo configuration file for each node after importing minio.

The Extra Repo can also be created using the script artifacts/gen_repo_conf.sh, by executing the following command:

./gen_repo_conf.sh --url-mode ${linux_distribution} ${repo_base_url}

# Execute the script to create a URL source profile
./gen_repo_conf.sh --url-mode centos ${minio_address}/kubean/centos/\$releasever/os/\$basearch
# View URL source profile
cat /etc/yum.repos.d/Kubean-URL.repo

[kubean-extra]
name=Kubean Extra Repo
baseurl=http://10.20.30.40:9000/kubean/centos/$releasever/os/$basearch
enabled=1
gpgcheck=0
sslverify=0

Note

If the repo_base_url parameter has a $ symbol, it needs to be escaped \$.

It's required to replace ${minio_address} with the actual minio API Server address.

Create source profiles with ClusterOperation and playbook

Only Centos yum repo additions are currently supported

As the process of creating a source involves all the nodes in the cluster, manual scripting is relatively tedious, so a playbook solution is provided here:

apiVersion: kubean.io/v1alpha1
kind: ClusterOperation
metadata:
  name: cluster-ops-01
spec:
  cluster: sample
  image: ghcr.io/kubean-io/spray-job:latest
  actionType: playbook
  action: cluster.yml
  preHook:
    - actionType: playbook
      action: ping.yml
    - actionType: playbook
      action: enable-repo.yml  # Before deploying the cluster, run the enable-repo playbook
                               # to create a source configuration for each node with the specified url
      extraArgs: |
        -e "{repo_list: ['http://10.20.30.40:9000/kubean/centos/\$releasever/os/\$basearch']}"
    - actionType: playbook
      action: disable-firewalld.yml
  postHook:
    - actionType: playbook
      action: cluster-info.yml
    - actionType: playbook
      action: enable-repo.yml  # After deploying the cluster, restore the yum repo configuration for each node.
                               # (Note: This step can be added as appropriate.)
      extraArgs: |
        -e undo=true

Example for SLES 15 SP7 (zypper repo). Each module is a separate repository, so list them individually:

apiVersion: kubean.io/v1alpha1
kind: ClusterOperation
metadata:
  name: cluster-ops-01
spec:
  cluster: sample
  image: ghcr.io/kubean-io/spray-job:latest
  actionType: playbook
  action: cluster.yml
  preHook:
    - actionType: playbook
      action: ping.yml
    - actionType: playbook
      action: enable-repo.yml  # sles is the os-pkgs source, sles-iso is the imported ISO source
      extraArgs: |
        -e "{repo_list: ['http://10.20.30.40:9000/kubean/sles/\$releasever/os/\$basearch',
                         'http://10.20.30.40:9000/kubean/sles-iso/\$releasever/os/\$basearch/Module-Basesystem',
                         'http://10.20.30.40:9000/kubean/sles-iso/\$releasever/os/\$basearch/Module-Python3']}"
    - actionType: playbook
      action: disable-firewalld.yml
  postHook:
    - actionType: playbook
      action: cluster-info.yml
    - actionType: playbook
      action: enable-repo.yml
      extraArgs: |
        -e undo=true

Notes for SLES 15 SP7:

• $releasever resolves to 15.7, matching the layout of both sources.
• The node must already have the Ansible interpreter before ping.yml runs, so the interpreter cannot be installed by this playbook. With the modern spray-job image, install python311 first and set ansible_python_interpreter in hosts-conf. See Python versions and image selection.

• Until the pinned Kubespray revision recognises SLES, add the distribution in vars-conf. The variable replaces the whole list, so keep the existing entries:

  supported_os_distributions:
    - 'RedHat'
    - 'CentOS'
    - 'Fedora'
    - 'Ubuntu'
    - 'Debian'
    - 'Flatcar'
    - 'Suse'
    - 'SLES'
    - 'openSUSE Leap'
    - 'openSUSE MicroOS'
    - 'openSUSE Tumbleweed'
    - 'ClearLinux'
    - 'OracleLinux'
    - 'AlmaLinux'
    - 'Rocky'
    - 'Amazon'
    - 'Kylin Linux Advanced Server'
    - 'UnionTech'
    - 'UniontechOS'
    - 'openEuler'
  

• SLE 15 does not ship the ntp package. If you set ntp_enabled: true, also set ntp_package: chrony.

Configure cluster before deployment

Offline settings need to be referred to kubespray Located in kubespray/inventory/sample/group_vars/all/offline.yml configuration file:

---
## Global offline configuration
### Configure the address of the private container image repository service
registry_host: "{{ registry_address }}"

### Configuring the address of the binary file service
github_url: https://files.m.daocloud.io/github.com
dl_k8s_io_url: https://files.m.daocloud.io/dl.k8s.io
storage_googleapis_url: https://files.m.daocloud.io/storage.googleapis.com
get_helm_url: https://files.m.daocloud.io/get.helm.sh

### If you are using CentOS / RedHat / AlmaLinux / Fedora, you need to configure the yum source file service address:
yum_repo: "{{ minio_address }}"

### If using Debian, configure:
debian_repo: "{{ minio_address }}"

### If using Ubuntu, configure:
ubuntu_repo: "{{ minio_address }}"

### If containerd uses a non-secure HTTP authentication-free method, it needs to be configured:
containerd_registries_mirrors:
  - prefix: "{{ registry_address }}"
    mirrors:
      - host: "http://{{ registry_address }}"
        capabilities: ["pull", "resolve"]
        skip_verify: true

### Required if docker uses non-secure HTTP authentication-free methods:
docker_insecure_registries:
  - {{ registry_address }}

For offline deployment, additional parameters are required for some special operating systems:

OS	Additional parameters
RHEL Series	rhel_enable_repos: false
Oracle Linux Series	use_oracle_public_repo: false

We use examples/install/3.airgap as a template.

Adapt the offline configuration as above to your specific situation, especially if you need to replace <registry_address> and <minio_address>.

Finally add the configuration update to the examples/install/3.airgap/VarsConfCM.yml file.

We also need to change the cluster node IP and username password in examples/install/3.airgap/HostsConfCM.yml.

Finally, the ClusterOperation task is started with kubectl apply -f examples/install/3.airgap to install the k8s cluster.

Generate and use incremental offline packages

For details see Air gap patch usage.