diff --git a/modules/about-toolbox.adoc b/modules/about-toolbox.adoc index 7d5fa40524e..090fc1e3c68 100644 --- a/modules/about-toolbox.adoc +++ b/modules/about-toolbox.adoc @@ -11,8 +11,18 @@ ifndef::openshift-origin[] `toolbox` is a tool that starts a container on a {op-system-first} system. The tool is primarily used to start a container that includes the required binaries and plugins that are needed to run commands such as `sosreport`. The primary purpose for a `toolbox` container is to gather diagnostic information and to provide it to Red Hat Support. However, if additional diagnostic tools are required, you can add RPM packages or run an image that is an alternative to the standard support tools image. + +[IMPORTANT] +==== +Use `toolbox` for `sosreport` and similar node diagnostics. If you run `podman` yourself, pass `--network host` or `--network none`. The default Podman bridge uses `10.88.0.0/16` and can overlap production networks. See xref:#running-podman-on-cluster-nodes_{context}[Running Podman on a cluster node]. +==== endif::openshift-origin[] ifdef::openshift-origin[] `toolbox` is a tool that starts a container on a {op-system-first} system. The tool is primarily used to start a container that includes the required binaries and plugins that are needed to run your favorite debugging or admin tools. + +[IMPORTANT] +==== +Use `toolbox` for node diagnostics. If you run `podman` yourself, pass `--network host` or `--network none`. The default Podman bridge uses `10.88.0.0/16` and can overlap production networks. See xref:#running-podman-on-cluster-nodes_{context}[Running Podman on a cluster node]. +==== endif::openshift-origin[] diff --git a/modules/agent-install-networking.adoc b/modules/agent-install-networking.adoc index 50c22608c31..7d9dfb0cd6f 100644 --- a/modules/agent-install-networking.adoc +++ b/modules/agent-install-networking.adoc @@ -18,6 +18,7 @@ In addition to static IP addresses, you can apply any network configuration that ==== By default, Podman uses a subnet of `10.88.0.0/16` as a bridge network. Do not set the `network.machineNetwork.cidr` parameter to include this address range, otherwise a conflict causes the cluster installation to fail. +The same bridge is created after installation if you run `podman run` on a node without `--network host` or `--network none`. ==== [id="agent-install-networking-ports_{context}"] diff --git a/modules/rhcos-key-features.adoc b/modules/rhcos-key-features.adoc index 71e62a6b191..52915a2f049 100644 --- a/modules/rhcos-key-features.adoc +++ b/modules/rhcos-key-features.adoc @@ -20,6 +20,11 @@ The following list describes key features of the {op-system} operating system: CRI-O can use either the `crun` or `runC` container runtime to start and manage containers. `crun` is the default. For information about how to enable `runC`, see the documentation for creating a `ContainerRuntimeConfig` CR. * **Set of container tools**: For tasks such as building, copying, and otherwise managing containers, {op-system} replaces the Docker CLI tool with a compatible set of container tools. The podman CLI tool supports many container runtime features, such as running, starting, stopping, listing, and removing containers and container images. The `skopeo` CLI tool can copy, authenticate, and sign images. You can use the `crictl` CLI tool to work with containers and pods from the CRI-O container engine. While direct use of these tools in {op-system} is discouraged, you can use them for debugging purposes. ++ +[IMPORTANT] +==== +A plain `podman run` on a node creates the default bridge on `10.88.0.0/16` (`podman0` or `cni-podman0`). If that range is already used for nodes, NFS, or other production networks, the local route wins and traffic to those addresses stops. The bridge can remain after the container exits. For debug work, use `--network host` or `--network none`. See xref:../support/gathering-cluster-data.adoc#running-podman-on-cluster-nodes_gathering-cluster-data[Running Podman on a cluster node]. +==== * **rpm-ostree upgrades**: {op-system} features transactional upgrades using the `rpm-ostree` system. Updates are delivered by means of container images and are part of the {product-title} update process. When deployed, the container image is pulled, extracted, and written to disk, then the boot loader is modified to boot into the new version. The machine reboots into the update in a rolling manner to ensure cluster capacity is minimally impacted. diff --git a/modules/running-podman-on-cluster-nodes.adoc b/modules/running-podman-on-cluster-nodes.adoc new file mode 100644 index 00000000000..1467f6cd78e --- /dev/null +++ b/modules/running-podman-on-cluster-nodes.adoc @@ -0,0 +1,132 @@ +// Module included in the following assemblies: +// +// * support/gathering-cluster-data.adoc + +:_mod-docs-content-type: PROCEDURE +[id="running-podman-on-cluster-nodes_{context}"] += Running Podman on a cluster node + +[role="_abstract"] +If you run Podman on an {op-system} node, the default bridge uses `10.88.0.0/16`. That route can take precedence over the node's real network, including NFS, and it can stay in the routing table after the container exits. For debug work, attach the container to the host network, or point Podman at a subnet that nothing else on the site uses. + +The problem shows up most often during `oc debug node/` after `chroot /host`. A command such as `podman run ` with no `--network` option creates `podman0` or, on older nodes, `cni-podman0`. Longest-prefix match then prefers the local `/16` over the default gateway. Workloads that talk to addresses in that range start timing out, and the interface does not go away when the debug pod is deleted. + +.Prerequisites + +ifndef::openshift-rosa,openshift-rosa-hcp,openshift-dedicated[] +* You have access to the cluster as a user with the `cluster-admin` role. +endif::openshift-rosa,openshift-rosa-hcp,openshift-dedicated[] +ifdef::openshift-rosa,openshift-rosa-hcp,openshift-dedicated[] +* You have access to the cluster as a user with the `dedicated-admin` role. +endif::openshift-rosa,openshift-rosa-hcp,openshift-dedicated[] +* You have installed the OpenShift CLI (`oc`). +ifndef::openshift-rosa,openshift-rosa-hcp,openshift-dedicated[] +* You have a range that does not overlap the machine, cluster, service, or storage networks if you need bridged Podman. +endif::openshift-rosa,openshift-rosa-hcp,openshift-dedicated[] + +.Procedure + +. Start a debug session on the node: ++ +[source,terminal] +---- +$ oc debug node/ +---- + +. Switch to the host root file system: ++ +[source,terminal] +---- +# chroot /host +---- + +. For a one-off diagnostic container, do not use the default bridge. Either share the host network or give the container no network: ++ +[source,terminal] +---- +# podman run --network host --rm +---- ++ +[source,terminal] +---- +# podman run --network none --rm --volume /var/lib/etcd:/var/lib/etcd:Z +---- ++ +`--network none` is enough when the image only needs local disks, for example an `fio` test against etcd storage. Prefer `toolbox` when you are collecting `sosreport` data; that path is meant for node diagnostics. + +. If a leftover bridge is already present, delete it. Check the names first: ++ +[source,terminal] +---- +# ip -br link show +---- ++ +Then remove whichever interface exists: ++ +[source,terminal] +---- +# ip link delete podman0 +---- ++ +[source,terminal] +---- +# ip link delete cni-podman0 +---- ++ +Confirm that `10.88.0.0/16` is gone from the routing table: ++ +[source,terminal] +---- +# ip route | grep -E '10\.88\.|podman0|cni-podman0' +---- ++ +No output means the collision is cleared. A reboot also removes the route, but deleting the link avoids waiting on a node drain. + +ifndef::openshift-rosa,openshift-rosa-hcp,openshift-dedicated[] +. Optional: If you must run bridged Podman on the nodes, set `default_subnet` before the default network is created. Put the following in a file named `99-podman-default-subnet.conf`: ++ +[source,text] +---- +[network] +default_subnet = "" +---- ++ +Replace `` with a prefix that is free on that site. Do not reuse `10.88.0.0/16` if that range is already on your network. There is no universal "safe" private range; pick one that does not overlap machine, cluster, service, or storage networks. + +. Encode the file: ++ +[source,terminal] +---- +$ base64 -w0 99-podman-default-subnet.conf +---- + +. Create a `MachineConfig` object for each pool that you debug on. The following example targets worker nodes: ++ +[source,yaml] +---- +apiVersion: machineconfiguration.openshift.io/v1 +kind: MachineConfig +metadata: + labels: + machineconfiguration.openshift.io/role: worker + name: 99-worker-podman-default-subnet +spec: + config: + ignition: + version: 3.2.0 + storage: + files: + - contents: + source: data:text/plain;charset=utf-8;base64, + mode: 0644 + overwrite: true + path: /etc/containers/containers.conf.d/99-podman-default-subnet.conf +---- ++ +Create a second manifest with `machineconfiguration.openshift.io/role: master` if you run Podman on control plane nodes. Apply the manifests with `oc create -f`. `default_subnet` is read when Podman first creates the default network. If `podman0` or `cni-podman0` already exists, delete that interface as shown earlier so the new range can take effect. ++ +[IMPORTANT] +==== +Do not change the cluster-wide Podman default in the Machine Config Operator payloads. A drop-in on `/etc/containers/containers.conf.d/` is a site override. Replacing `10.88.0.0/16` for every cluster still collides wherever the new range is already in use. +==== +endif::openshift-rosa,openshift-rosa-hcp,openshift-dedicated[] diff --git a/modules/security-command-line-host-access.adoc b/modules/security-command-line-host-access.adoc index 01957241c49..fbc7c066bde 100644 --- a/modules/security-command-line-host-access.adoc +++ b/modules/security-command-line-host-access.adoc @@ -33,6 +33,11 @@ After connecting to the node, run the following command to get access to the roo ---- + This gives you root access within a debug pod on the node. For more information, see "Starting debug pods with root access". ++ +[IMPORTANT] +==== +If you run `podman` after `chroot /host`, pass `--network host` or `--network none`. A command such as `podman run ` with no `--network` option creates the default `10.88.0.0/16` bridge. That route can block NFS and other node traffic, and it can stay in the routing table after you leave the debug session. +==== Direct SSH:: Avoid using the root user. Instead, use the core user ID (or your own ID). To connect to the node by using SSH, run the following command: + diff --git a/support/gathering-cluster-data.adoc b/support/gathering-cluster-data.adoc index 3f5ad4e5981..37bc2112e9d 100644 --- a/support/gathering-cluster-data.adoc +++ b/support/gathering-cluster-data.adoc @@ -170,3 +170,14 @@ include::modules/support-installing-packages-to-a-toolbox-container.adoc[levelof // Starting an alternative image with toolbox include::modules/support-starting-an-alternative-image-with-toolbox.adoc[leveloffset=+2] + +// Running Podman on a cluster node +include::modules/running-podman-on-cluster-nodes.adoc[leveloffset=+1] + +[role="_additional-resources"] +.Additional resources + +* link:https://docs.podman.io/en/stable/markdown/podman-network.1.html#subnet-notes[Podman default network subnets] +ifndef::openshift-origin[] +* link:https://access.redhat.com/solutions/6961655[Red Hat knowledge base article about the Podman default subnet] +endif::openshift-origin[]