Docker Swarm Overlay Network Issue

Issue

The issue was indeed the wrong checksums on the outbound packets when the host OS was installed on the VMware ESX infrastructure. The VMware network interface was dropping the packets due to wrong checksums.

This issue was reported if the host OS is Red Hat/Centos or Debian/Ubuntu distribution.

Solution for Red Hat/Centos 7x

The solution was to disable checksum offloading. Using ethtool;

Bash

ethtool -K <interface> tx off

If you want this setting to be permanent/persistent, add below line to interface config file;

File located under /etc/sysconfig/network-scripts/ifcfg-xxxx
ETHTOOL_OPTS='-K ${DEVICE} tx off'

Solution for Red Hat/Centos 8.x/9x

Find interface which is used for docker swarm then edit configuration.

Bash

/etc/NetworkManager/system-connections/eth0.nmconnection

Add below lines

[ethtool]
feature-tx=false

Solution for Debian/Ubuntu

The solution was to disable checksum offloading. Using ethtool;

Bash

ethtool -K <interface> tx off

If you want this setting to be permanent/persistent, add below line to interface config file. File located under /etc/netplan/00-eth0-conf.yaml

Bash

Original Config File
network:
  version: 2
  ethernets:
    ens160:
      addresses: [ip-address]
      routes:
        - to: default
          via: gw
      nameservers:
        addresses: [dns1, dns2]
        search: [domain.com]

You must add below lines to interface configuration;

transmit-checksum-offload: false
tcp-segmentation-offload: false
tcp6-segmentation-offload: false

After the editing of the file, it looks like below example;

Bash

network:
  version: 2
  ethernets:
    ens160:
      transmit-checksum-offload: false
      tcp-segmentation-offload: false
      tcp6-segmentation-offload: false
      addresses: [ip-address]
      routes:
        - to: default
          via: gw
      nameservers:
        addresses: [dns1, dns2]
        search: [domain.com]