Hello everyone!

I need some insight related to a consul-template issue. I have a loadbalancer (HAproxy) with a configuration rendered via consul-template. Everything works fine but there some issue visible in the logs I can’t wrap my head around.

Here is my Consul-template HAproxy template:

global chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin stats timeout 30s user haproxy group haproxy daemon ca-base /etc/ssl/certs crt-base /etc/ssl/private defaults log global mode http balance source option httplog option dontlognull option log-separate-errors option httpclose option abortonclose option http-server-close option redispatch timeout connect 5000 timeout client 50000 timeout server 50000 load-server-state-from-file global errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http listen stats bind *:8080 ssl crt /etc/haproxy/ssl/ stats enable stats refresh 10s stats uri / stats show-legends stats show-node stats admin if stats_admin_ips stats http-request deny if !stats_access_ips http-request set-log-level silent frontend external bind :::80 v4v6 bind :::443 v4v6 ssl crt /etc/haproxy/ssl/ alpn h2,http/1.1 redirect scheme https code 301 if !{ ssl_fc } option forwardfor http-request set-header X-Forwarded-Port %[dst_port] http-request set-header X-Forwarded-Proto http if !{ ssl_fc } http-request set-header X-Forwarded-Proto https if { ssl_fc } {{ $requiredTags := parseJSON `["live"]` }} {{- range services -}}{{ if containsAll $requiredTags .Tags }} # Service {{ .Name }} acl {{ .Name }} hdr(host) {{ .Name }}.example.net {{- end -}}{{- end }} {{- range services -}}{{ if containsAll $requiredTags .Tags }} # Backend to use for {{ .Name }} use_backend {{ .Name }} if {{ .Name }} {{- end -}}{{- end }} {{ range services }}{{ if containsAll $requiredTags .Tags }} {{ $service:=.Name }} backend {{ .Name }} {{- range service $service }} {{- if .Tags | contains "live" }} server-template {{ .Node }} 1-5 live.{{ .Name }}.service.consul:{{ .Port }} resolvers consul resolve-opts allow-dup-ip resolve-prefer ipv4 check ssl verify none {{- end -}} {{- end }}{{ end }}{{ end }} resolvers consul nameserver consul 127.0.0.1:8600 accepted_payload_size 8192 hold valid 5s

The consul-template process did build the configuration but here is what I can see in the daemon logs :

Sep 14 16:32:49 lb.example.net consul-template[129435]: 2022-09-14T16:32:49.557+0200 [WARN] (view) health.service(www|passing): Unexpected response code: 500 (rpc error: code = InvalidArgument desc = Key is required) (retry attempt 1 after "250ms")
Sep 14 16:34:25 lb.example.net consul-template[129435]: 2022-09-14T16:34:25.231+0200 [WARN] (view) health.service(www|passing): Unexpected response code: 500 (rpc error: code = InvalidArgument desc = Key is required) (retry attempt 1 after "250ms")
And these are the consul agent logs :
Sep 14 16:37:35 lb.example.net consul[19138]: 2022-09-14T16:37:35.132+0200 [ERROR] agent.rpcclient.health: subscribe call failed: err="rpc error: code = InvalidArgument desc = Key is required" failure_count=2160 key=www topic=ServiceHealth
Sep 14 16:37:35 lb.example.net consul[19138]: 2022-09-14T16:37:35.132+0200 [ERROR] agent.http: Request error: method=GET url=/v1/health/service/www?index=7028250&passing=1&stale=&wait=60000ms from=127.0.0.1:38948 error="rpc error: code = InvalidArgument desc = Key is required"
Do you guys have any idea why I’m seeing this? Did I miss something in my consul-template HAproxy template?
Thanks for your help.
              
Hey @drustan, thanks for asking.
It looks like this might have to do with the consul agent setting up a connection to the consul server via the streaming RPC API. Consul agent supports 2 methods for monitoring values in the consul servers. The first method is simple long polling which pulls the full data of the monitored value when an update occurs. The second is via a streaming RPC call that sends only the update-diff over a persistent stream. The second was added for performance on large clusters.
Some questions or things you could look at.
What version of consul?

Are the consul servers and consul agents the same version?

Are they both set up to use streaming?
Not 100% on that but it is my current best guess. Please ask any followup questions you might have and we can try to get to the bottom of this.
Thanks and good luck.
              
Thank you for your reply and the explanation about the existence of the two methods for monitoring values.
You asked me about the Consul version and I realized the agent was 1.13.1 and the servers were 1.12 so I upgraded and everything return to normal, it was that simple!
The error message wasn’t very clear but I sure should have checked that the versions matched first :-/
Have a great day!
              
Unfortunately the problem still there, but it comes out when upgrading in multi DC configuration from 1.10.12 to 1.14.7.
agent.rpcclient.health: subscribe call failed: err=“rpc error: code = InvalidArgument desc = Key is required” failure_count=20 key=kube-system-apiserver topic=ServiceHealth
This log caused by DNS query from one updated DC to another DC with previous version.

Example:

From server node 1.14.7 make query like

consul-metrics.service.11012_NOT_UPDATED_DC.consul - failing
From server node 1.10.12 make query like

consul-metrics.service.1147_UPDATED_DC.consul - resolving
Update all consul servers in all clusters solving the issue, but it’s impossible to update all at once. I made it in test environment, but i don’t have an idea how to update my personal production environment without problems.