out_http: setting Host header does not work · Issue #1428 · fluent/fluent-bit

Bug Report

When setting the Host header in the out_http plugin it simply does not work. The final value of the Host header that is set in the request is equal to the Host and Port values set in the configuration of the out_http plugin.

I tried to set another header with a dummy key-name with the same exact value and it works, the header is set. Maybe this specific header needs to be set in a special way, as it happens in the net/http golang package, that Request.Host needs to be set instead of adding it together with the rest of the headers.

To Reproduce

Configure the out_http plugin with a Host header set:

[OUTPUT]
    Name    http
    Match   “*”
    Host    10.142.215.105
    Port    80
    Format  json
    Header  Host example.com
Run FluentBit and watch how the request you get on the other side does not have the expected Host header:
  path: '/',
  headers: {
    host: '10.142.215.105:80',
    'content-length': '577',
    'content-type': 'application/json',
    'user-agent': 'Fluent-Bit’
  method: 'POST',
  body: ‘<JSONArray with the logs>',
  cookies: undefined,
  fresh: false,
  hostname: '10.142.215.105',
  ip: '::ffff:10.142.212.239',
  ips: [],
  protocol: 'http',
  query: {},
  subdomains: [],
  xhr: false
Expected behavior

Host header is set as expected.
Your Environment
Version used: 1.2
Environment name and version: k8s v1.12.6
Operating System and version: Amazon Linux 2 (using amazon-eks-ami)
          actually it works, but there is a special case where there is not a workaround.
In HTTP/1.1, the Host header is mandatory and is set by default by our HTTP client. So if you append a custom header it's not overriding the previous one in the request, e.g:
If you add the custom header "Host  example.com" to the config, the HTTP request will look as follows:
POST / HTTP/1.1
Host: 23.253.238.156:80
Content-Length: 1852
Content-Type: application/json
User-Agent: Fluent-Bit
Host:  example.com
So there are two Host headers in the request and the server or script parsing it, it's deciding which one to take in count.
Would you please elaborate what's the use case that you need to set a custom Host header ?. The Host header aims to indicate the desired virtual host in the server side, but... if you use the proper domain in the configuration where the Fluent Bit host config points to the domain, it should be ok and you won't need that custom header, e.g:
[OUTPUT]
    Name  http
    Host  example.com
    Port  80
          I didn't see the custom Host header in the tests I did, do you use any particular tool or container to help you debug these issues better? Maybe I am not using the appropriate one
I need to forward logs to a remote API Gateway behind a VPC Endpoint. So, I was provided with the VPC Endpoint name where I can route the requests and the custom Host header it needs to route the requests to the correct place.
I made it work by setting a rewrite rule in the CoreDNS of the cluster so I do not need to set a custom header in FluentBit. But it would be great to make it work in FluentBit, so I avoid adding complexity to CoreDNS.
Does it make sense to you to bring this feature into FluentBit? To be able to override the default Host header added to the request. It would be really helpful in these cases, where you cannot control the specific code that looks for the Host header. Since it is a service from a Cloud Provider which is looking for it, and it might not get the correct Host header.
Update
I found out in AWS docs that there is an alternative header that can be used instead of the Host header. It worked as expected and I could remove the custom configuration from CoreDNS and keep all the configuration in FluentBit.
Alternatively, you can access your private API via its API ID by using the x-apigw-api-id header
https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-private-api-test-invoke-url.html