Restart nginx when /etc/hosts changes
I have a problem where I have a role (vpnweb-frontend
) that has a nginx sites-enabled
template, like follows:
upstream be_vpnweb {
{% for host in groups['vpnweb']|sort %} server {{ host }}.vpnweb.{{ domain }}:{{ services['vpnweb'].public_endpoints[0].port }};
{% endfor %}
}
If this role runs after float-base
and things were rescheduled, the /etc/hosts
file gets populated, nginx would get reloaded before this role was evaluated, resulting in the template not being interpolated yet, and the backend no longer existing, and nginx will fail to start on the reload, killing nginx and stopping deployment. The deployment is unable to proceed to the point where the template would get evaluated to the right new value, which would keep nginx from being killed (because I am using the hostname-based service discovery syntax in the backend URL).
This only happens on rescheduling.
If I move to to the beginning, the inverse problem happens: The nginx configuration is changed through the interpolation of the template, to what the upstream host will be, and then nginx is restarted, but the /etc/hosts file hasn't been updated, and so no such host exists yet, resulting in nginx broken.
Why do I need to do this? Its because I have a non-default port I want to listen on (listen [::]:4430 ssl http2 ipv6only=off;
)
One solution would be to run the role after float-base
, but before other float things, but this would mean I'd have to write my own playbook to replace float/playbooks/all.yml
and make sure to adapt to it in the future.
Another less than great solution is to make a role that just forces a restart of nginx later.
If I use the service name in a nginx upstream {} clause (eg. <service>.your.domain
), then nginx should resolve all the backends at startup/reload time, which would solve this ordering rescheduling scenario problem, but it would require that nginx was reloaded whenever /etc/hosts
changes.
!261 was made to attempt to deal with this rescheduling problem, but it might be better/easier to just reload nginx when /etc/hosts
changes (although !261 may have other uses)