Description
When running multiple containers on the same docker service and same network overlay, each sending and receiving UDP packets from/to every nodes I get Exception "OSError: [Errno 22] Invalid argument"
Steps to reproduce the issue:
Create a service consisting of a tracker running on an overlay network
Create a service with 150 nodes executing the python3 script running on the same overlay network
Request all ips from the tracker
Wait for nodes to start sending UDP packets
Describe the results you received:
2017-01-09 09:01:53,170 - ERROR: Exception catched for address: 10.99.0.153
Traceback (most recent call last):
File "./server.py", line 56, in <module>
sock.sendto(bytes('{}'.format(i), "utf-8"), (address, PORT))
OSError: [Errno 22] Invalid argument
Describe the results you expected:
Every packet should be sent with no error
Additional information you deem important (e.g. issue happens only occasionally):
During the same run a node will one time be able to send the packet and the next time fail.
We can see this in the following log: https://gist.github.com/jocelynthode/dc0ceea7658e6722cb9a5c2f3db5576a
I am currently using Docker 1.12.2 although the problem also happens on Docker 1.12.5. This problem does not seem to appear on a inhouse OpenNebula cluster we use. We can only reproduce this problem on AWS (Either using Ubuntu server 16.04 or Debian stretch).
Here is the UDP class I use to send messages : https://github.com/jocelynthode/EpTOTester/blob/aws-test/projects/EpTOTester/scripts/server.py
Here is the tracker I use : https://github.com/jocelynthode/EpTOTester/blob/aws-test/projects/EpTOTester/tracker/tracker.py
Output of docker version
:
Client:
Version: 1.12.2
API version: 1.24
Go version: go1.6.3
Git commit: bb80604
Built: Tue Oct 11 18:29:41 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.2
API version: 1.24
Go version: go1.6.3
Git commit: bb80604
Built: Tue Oct 11 18:29:41 2016
OS/Arch: linux/amd64
Output of docker info
:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 3
Server Version: 1.12.2
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 18
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host bridge overlay null
Swarm: active
NodeID: ek9vxftn7noow9gw3bbhb9ue9
Is Manager: true
ClusterID: c960n73tqhh9lr45tm0qywac1
Managers: 1
Nodes: 5
Orchestration:
Task History Retention Limit: 10
Raft:
Snapshot Interval: 10000
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Node Address: 172.31.17.24
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-57-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.303 GiB
Name: ip-172-31-17-24
ID: FE2S:3IMA:UZAP:HOEA:YAOR:3OR4:WILJ:EN3L:A5HR:GI5Q:SKTQ:LYL3
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
Additional environment details (AWS, VirtualBox, physical, etc.):
I'm using 5 EC2 containers of the type c4.xlarge with Ubuntu server 16.04.
I'm using Docker 1.12.2 but the problem is the same with Docker 1.12.5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Metadata
Assignees
Labels
area/networking area/swarm version/1.12
Type
No type
Projects
No projects
Milestone
No milestone
Relationships
None yet
Development
No branches or pull requests
Issue actions