UDP Socket Invalid Argument exception #29992

Open

Description

Description

When running multiple containers on the same docker service and same network overlay, each sending and receiving UDP packets from/to every nodes I get Exception "OSError: [Errno 22] Invalid argument"

Steps to reproduce the issue:

  • Create a service consisting of a tracker running on an overlay network
  • Create a service with 150 nodes executing the python3 script running on the same overlay network
  • Request all ips from the tracker
  • Wait for nodes to start sending UDP packets
  • Describe the results you received:

    2017-01-09 09:01:53,170 - ERROR: Exception catched for address: 10.99.0.153
    Traceback (most recent call last):
      File "./server.py", line 56, in <module>
        sock.sendto(bytes('{}'.format(i), "utf-8"), (address, PORT))
    OSError: [Errno 22] Invalid argument
    

    Describe the results you expected:
    Every packet should be sent with no error

    Additional information you deem important (e.g. issue happens only occasionally):
    During the same run a node will one time be able to send the packet and the next time fail.

    We can see this in the following log: https://gist.github.com/jocelynthode/dc0ceea7658e6722cb9a5c2f3db5576a

    I am currently using Docker 1.12.2 although the problem also happens on Docker 1.12.5. This problem does not seem to appear on a inhouse OpenNebula cluster we use. We can only reproduce this problem on AWS (Either using Ubuntu server 16.04 or Debian stretch).

    Here is the UDP class I use to send messages : https://github.com/jocelynthode/EpTOTester/blob/aws-test/projects/EpTOTester/scripts/server.py

    Here is the tracker I use : https://github.com/jocelynthode/EpTOTester/blob/aws-test/projects/EpTOTester/tracker/tracker.py

    Output of docker version:

    Client:
     Version:      1.12.2
     API version:  1.24
     Go version:   go1.6.3
     Git commit:   bb80604
     Built:        Tue Oct 11 18:29:41 2016
     OS/Arch:      linux/amd64
    Server:
     Version:      1.12.2
     API version:  1.24
     Go version:   go1.6.3
     Git commit:   bb80604
     Built:        Tue Oct 11 18:29:41 2016
     OS/Arch:      linux/amd64
    

    Output of docker info:

    Containers: 0
     Running: 0
     Paused: 0
     Stopped: 0
    Images: 3
    Server Version: 1.12.2
    Storage Driver: aufs
     Root Dir: /var/lib/docker/aufs
     Backing Filesystem: extfs
     Dirs: 18
     Dirperm1 Supported: true
    Logging Driver: json-file
    Cgroup Driver: cgroupfs
    Plugins:
     Volume: local
     Network: host bridge overlay null
    Swarm: active
     NodeID: ek9vxftn7noow9gw3bbhb9ue9
     Is Manager: true
     ClusterID: c960n73tqhh9lr45tm0qywac1
     Managers: 1
     Nodes: 5
     Orchestration:
      Task History Retention Limit: 10
     Raft:
      Snapshot Interval: 10000
      Heartbeat Tick: 1
      Election Tick: 3
     Dispatcher:
      Heartbeat Period: 5 seconds
     CA Configuration:
      Expiry Duration: 3 months
     Node Address: 172.31.17.24
    Runtimes: runc
    Default Runtime: runc
    Security Options: apparmor seccomp
    Kernel Version: 4.4.0-57-generic
    Operating System: Ubuntu 16.04.1 LTS
    OSType: linux
    Architecture: x86_64
    CPUs: 4
    Total Memory: 7.303 GiB
    Name: ip-172-31-17-24
    ID: FE2S:3IMA:UZAP:HOEA:YAOR:3OR4:WILJ:EN3L:A5HR:GI5Q:SKTQ:LYL3
    Docker Root Dir: /var/lib/docker
    Debug Mode (client): false
    Debug Mode (server): false
    Registry: https://index.docker.io/v1/
    WARNING: No swap limit support
    Insecure Registries:
     127.0.0.0/8
    

    Additional environment details (AWS, VirtualBox, physical, etc.):

    I'm using 5 EC2 containers of the type c4.xlarge with Ubuntu server 16.04.

    I'm using Docker 1.12.2 but the problem is the same with Docker 1.12.5

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Assignees

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions