By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement . We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account
 main ------------------------------------------------------------------------------------------------------------------------                                                    │
│ main STAGE:  docker build                                                                                                                                                        │
│ main ------------------------------------------------------------------------------------------------------------------------                                                    │
│ main 2021/09/12 11:23:32 DEVTRON  -----> docker login -u xxx -p xxx  xxxxx                                                                    │
│ main WARNING! Using --password via the CLI is insecure. Use --password-stdin.                                                                                                    │
│ main WARNING! Your password will be stored unencrypted in /root/.docker/config.json.                                                                                             │
│ main Configure a credential helper to remove this warning. See                                                                                                                   │
│ main https://docs.docker.com/engine/reference/commandline/login/#credentials-store                                                                                               │
│ main Login Succeeded                                                                                                                                                             │
│ main 2021/09/12 11:23:33 DEVTRON  docker file location:                                                                                                                          │
│ main 2021/09/12 11:23:33  -----> docker build  -f Dockerfile --network host -t backend .                                                                                         │
│ main Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?                                                                           │
│ main 2021/09/12 11:23:33 exit status 1                                                                                                                                           │
│ main 2021/09/12 11:23:33 DEVTRON false exit status 1                                                                                                                             │
│ main 2021/09/12 11:23:33 DEVTRON  artifacts map[]                                                                                                                                │
│ main 2021/09/12 11:23:33 DEVTRON no artifact to upload                                                                                                                           │
│ main 2021/09/12 11:23:33 DEVTRON exit status 1 <nil>                                                                                                                             │
│ main stream closed

Is there any documentation how to ensure that build containers in the cluster are created in privileged mode so they can access docker engine? Thank you

@Born2Bake can you please help me to understand the usecase. In general in ci step users don't require to interact with docker daemon.
Also please share the Dockerfile

@Born2Bake can you please help me to understand the usecase. In general in ci step users don't require to interact with docker daemon.
Also please share the Dockerfile

 k get node -o wide
NAME           STATUS   ROLES                  AGE    VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE          KERNEL-VERSION   CONTAINER-RUNTIME
master-2       Ready    control-plane,master   2d4h   v1.20.1   172.16.1.131   <none>        Talos (v0.12.0)   5.10.58-talos    containerd://1.5.5
master-3       Ready    control-plane,master   2d4h   v1.20.1   172.16.1.132   <none>        Talos (v0.12.0)   5.10.58-talos    containerd://1.5.5
master-dev-1   Ready    control-plane,master   2d4h   v1.20.1   172.16.1.130   <none>        Talos (v0.12.0)   5.10.58-talos    containerd://1.5.5
worker-1       Ready    <none>                 2d4h   v1.20.1   172.16.1.133   <none>        Talos (v0.12.0)   5.10.58-talos    containerd://1.5.5
worker-2       Ready    <none>                 2d4h   v1.20.1   172.16.1.134   <none>        Talos (v0.12.0)   5.10.58-talos    containerd://1.5.5
worker-3       Ready    <none>                 2d4h   v1.20.1   172.16.1.135   <none>        Talos (v0.12.0)   5.10.58-talos    containerd://1.5.5
worker-4       Ready    <none>                 2d4h   v1.20.1   172.16.1.136   <none>        Talos (v0.12.0)   5.10.58-talos    containerd://1.5.5
worker-5       Ready    <none>                 2d4h   v1.20.1   172.16.1.137   <none>        Talos (v0.12.0)   5.10.58-talos    containerd://1.5.5
worker-6       Ready    <none>                 2d4h   v1.20.1   172.16.1.138   <none>        Talos (v0.12.0)   5.10.58-talos    containerd://1.5.5

Usecase: Bare-metal k8s cluster provisioned. Devtron installed via helm with default settings. Gitlab repo added, registry added. Pipeline created - build docker image using Dockerfile from the repository and then deploy it. It works for me if I am using gitlab-runners - they can build images using pods in kubernetes. However, when I try to build docker image via Devtron, I am getting that error after the build command. I ve tried many different executors for argo, and none of them work unfortunately.

docker daemon starting fails on hetzner vm wilth following logs

time="2021-11-20T13:33:57.656528738Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
time="2021-11-20T13:33:57.656645778Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
time="2021-11-20T13:33:57.657462489Z" level=info msg="libcontainerd: started new containerd process" pid=78
time="2021-11-20T13:33:57.657497089Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2021-11-20T13:33:57.657510199Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2021-11-20T13:33:57.657563029Z" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}]" module=grpc
time="2021-11-20T13:33:57.657579629Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2021-11-20T13:33:57.657629629Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420734950, CONNECTING" module=grpc
time="2021-11-20T13:33:57.672586864Z" level=info msg="starting containerd" revision=894b81a4b802e4eb2a91d1ce216b8817763c29fb version=v1.2.6 
time="2021-11-20T13:33:57.672785814Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1 
time="2021-11-20T13:33:57.672848524Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1 
time="2021-11-20T13:33:57.672936394Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" 
time="2021-11-20T13:33:57.672944954Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1 
time="2021-11-20T13:33:57.677685149Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe aufs failed: "ip: can't find device 'aufs'\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n": exit status 1" 
time="2021-11-20T13:33:57.677723609Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1 
time="2021-11-20T13:33:57.677848819Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1 
time="2021-11-20T13:33:57.678017559Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1 
time="2021-11-20T13:33:57.678167189Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter" 
time="2021-11-20T13:33:57.678180219Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1 
time="2021-11-20T13:33:57.678234699Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" 
time="2021-11-20T13:33:57.678244639Z" level=warning msg="could not use snapshotter aufs in metadata plugin" error="modprobe aufs failed: "ip: can't find device 'aufs'\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n": exit status 1" 
time="2021-11-20T13:33:57.678254199Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter" 
time="2021-11-20T13:33:57.766308744Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1 
time="2021-11-20T13:33:57.766352184Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1 
time="2021-11-20T13:33:57.766418074Z" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1 
time="2021-11-20T13:33:57.766466735Z" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1 
time="2021-11-20T13:33:57.766491205Z" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1 
time="2021-11-20T13:33:57.766514025Z" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1 
time="2021-11-20T13:33:57.766534675Z" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1 
time="2021-11-20T13:33:57.766548975Z" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1 
time="2021-11-20T13:33:57.766562565Z" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1 
time="2021-11-20T13:33:57.766576415Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1 
time="2021-11-20T13:33:57.766799905Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2 
time="2021-11-20T13:33:57.766925855Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1 
time="2021-11-20T13:33:57.767335685Z" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1 
time="2021-11-20T13:33:57.767361185Z" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1 
time="2021-11-20T13:33:57.767400005Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1 
time="2021-11-20T13:33:57.767414045Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1 
time="2021-11-20T13:33:57.767427085Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1 
time="2021-11-20T13:33:57.767439335Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1 
time="2021-11-20T13:33:57.767452455Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1 
time="2021-11-20T13:33:57.767465195Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1 
time="2021-11-20T13:33:57.767481945Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1 
time="2021-11-20T13:33:57.767494475Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1 
time="2021-11-20T13:33:57.767506935Z" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1 
time="2021-11-20T13:33:57.767804676Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1 
time="2021-11-20T13:33:57.767839297Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1 
time="2021-11-20T13:33:57.767853957Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1 
time="2021-11-20T13:33:57.767866897Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1 
time="2021-11-20T13:33:57.768088077Z" level=info msg=serving... address="/var/run/docker/containerd/containerd-debug.sock" 
time="2021-11-20T13:33:57.768166887Z" level=info msg=serving... address="/var/run/docker/containerd/containerd.sock" 
time="2021-11-20T13:33:57.768178787Z" level=info msg="containerd successfully booted in 0.096004s" 
time="2021-11-20T13:33:57.770262228Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420734950, READY" module=grpc
time="2021-11-20T13:33:57.774699012Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2021-11-20T13:33:57.774720382Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2021-11-20T13:33:57.774829453Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2021-11-20T13:33:57.774841053Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2021-11-20T13:33:57.775177974Z" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}]" module=grpc
time="2021-11-20T13:33:57.775192864Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2021-11-20T13:33:57.775222304Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420734ee0, CONNECTING" module=grpc
time="2021-11-20T13:33:57.775321054Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420734ee0, READY" module=grpc
time="2021-11-20T13:33:57.775360434Z" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}]" module=grpc
time="2021-11-20T13:33:57.775369244Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2021-11-20T13:33:57.775387324Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4207351c0, CONNECTING" module=grpc
time="2021-11-20T13:33:57.775466174Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4207351c0, READY" module=grpc
time="2021-11-20T13:33:57.895638290Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
time="2021-11-20T13:33:57.895933010Z" level=warning msg="Your kernel does not support cgroup memory limit"
time="2021-11-20T13:33:57.895967520Z" level=warning msg="Unable to find cpu cgroup in mounts"
time="2021-11-20T13:33:57.895980790Z" level=warning msg="Unable to find blkio cgroup in mounts"
time="2021-11-20T13:33:57.895989500Z" level=warning msg="Unable to find cpuset cgroup in mounts"
time="2021-11-20T13:33:57.896045240Z" level=warning msg="mountpoint for pids not found"
time="2021-11-20T13:33:57.896993651Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
time="2021-11-20T13:33:57.897136081Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
time="2021-11-20T13:33:57.897405352Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4207351c0, TRANSIENT_FAILURE" module=grpc
time="2021-11-20T13:33:57.897457462Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4207351c0, CONNECTING" module=grpc
Error starting daemon: Devices cgroup isn't mounted```  
          

We are still not sure if it might be related to #1130 cause docker build is working for me if I install Devtron using kubespray/ubuntu. However, it's still failing using Talos https://www.talos.dev/docs/v0.14/local-platforms/docker/#requirements && https://github.com/talos-systems/talos/releases/tag/v0.14.1 ; Also, it might be related to the https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#troubleshooting & https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4566

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

version: k3s, v1.23.6

I also encountered a similar problem here, is there a solution for this now?

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?无法连接到位于 unix:///var/run/docker.sock 的 Docker 守护程序。 docker 守护进程是否正在运行?

version: k3s, v1.23.6版本:k3s,v1.23.6

I also encountered a similar problem here, is there a solution for this now?我这里也遇到了类似的问题,请问现在有解决办法吗?

https://appuals.com/cannot-connect-to-the-docker-daemon-at-unix-var-run-docker-sock/

mark, this method can fix the error~

i find my problem。
in my /usr/local/bin/nohup.out log :
Status: invalid argument "--host=unix:///var/run/docker.sock" for "--insecure-registry" flag: invalid index name (--host=unix:///var/run/docker.sock)

code in ci-runner:

func StartDockerDaemon(dockerConnection, dockerRegistryUrl, dockerCert, defaultAddressPoolBaseCidr string, defaultAddressPoolSize int, ciRunnerDockerMtuValue int) 
        // my config in  OCI Registry 
        // dockerRegistryUrl = "registry.cn-hangzhou.aliyuncs.com"
	u, err := url.Parse(dockerRegistryUrl)
	if err != nil {
		log.Fatal(err)
        // ...
        // u.Host is null , fix : add `https://` to dockerRegistryUrl
        dockerdstart = fmt.Sprintf("dockerd  %s --insecure-registry %s --host=unix:///var/run/docker.sock %s --host=tcp://0.0.0.0:2375 > /usr/local/bin/nohup.out 2>&1 &", defaultAddressPoolFlag, u.Host, dockerMtuValueFlag)
      Bug:  Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
      #4389