Distributed Monitoring with Proxies – Part Two

http://zabbix101.blogspot.com/2015/05/distributed-monitoring-with-proxies.html

In part one of this series we went through the process of installation, setup, and initial communication of a Zabbix Proxy.  Overall, it was a pretty easy process to get going, but simply HAVING a proxy doesn’t accomplish anything.  We need to configure the Zabbix server to use the proxy to do monitoring before it is of any use.

Assigning hosts to proxies

There are several ways to assign hosts to a proxy.

Single Host Configuration

First, let’s take a look at assigning a single host to a proxy.

  1. In the Zabbix web UI go to “Configuration”->”Hosts”.
  2. Click on the name of a host in the “Name” column that should be monitored by a proxy.
  3. Notice the “Monitored by proxy” drop-down at the bottom of the host configuration.  By default, it is set to “(no proxy)”.  In the drop-down select the name of the proxy.
  4. Click the “Update” button.

Mass-Update Host Configuration

Alright, here’s a way to configure multiple hosts at the same time to be monitored by the same proxy.

  1. In the Zabbix web UI go to “Configuration”->”Hosts”.
  2. Check the check-boxes next to all the hosts that should be monitored by a proxy.
  3. In the drop-down on the bottom-left corner of the host list, select the “Mass update” option and then click the “Go” button next to the drop-down.
  4. In the new page that comes up make sure the “Host” tab is on (this is the default).
  5. Select the “Monitored by proxy” check-box and then choose the appropriate proxy name in the drop-down box that appears.
  6. Click the “Update” button.

Multi-Host Configuration through Proxy Configuration

And…..surprise!  Here is yet another way to configure multiple hosts to be monitored by a proxy at the same time.

  1. Go to “Administration”->”Proxies”.
    1. In Zabbix 2.0 and Zabbix 2.2 this is actually “Administration”->”DM”.  Make sure that the drop-down in the upper-right corner of the page (far right side of the “CONFIGURATION OF PROXIES” bar) has “Proxies” selected.
  2. Click on the name of the proxy that should monitor other hosts.
  3. Notice that there are two “Hosts” boxes on this page.
    1. The “Proxy hosts” box contains the hosts that are being monitored by this proxy.
    2. The “Other hosts” box contains the hosts that are NOT being monitored by this proxy.
  4. Select the hosts in the “Other hosts” box that should be monitored by this proxy and then click the “<<” button.  That will move the hosts into the “Proxy hosts” box.
  5. Click the “Update” button.

Agent Configuration

Ta-da!  Zabbix is now configured to monitor the selected updated hosts through the proxy.  Oh wait….  Remember how for Agents that they have a parameter in the zabbix_agentd.conf file called “Server”?  Let’s take a look at that parameter.

Configuration Options

### Option: Server
#       List of comma delimited IP addresses (or hostnames) of Zabbix servers.
#       Incoming connections will be accepted only from the hosts listed here.
#       If IPv6 support is enabled then '127.0.0.1', '::127.0.0.1', '::ffff:127.0.0.1' are treated equally.
#
# Mandatory: no
# Default:
# Server=

Server=my.server.fqdn.or.ip

Notice the comment that says “List of comma delimited IP addresses (or hostnames) of Zabbix servers”? That comment really should say something like “List of comma delimited IP addresses (or hostnames) of Zabbix servers/proxies”. Notice I added “proxies” at the very end. If the host for this agent is configured to have any passive items, then the “Server” parameter MUST have the Proxy in it that is configured to monitor the agent!

Another parameter to keep in mind is the “ServerActive” parameter. Let’s take a look:

### Option: ServerActive
#       List of comma delimited IP:port (or hostname:port) pairs of Zabbix servers for active checks.
#       If port is not specified, default port is used.
#       IPv6 addresses must be enclosed in square brackets if port for that host is specified.
#       If port is not specified, square brackets for IPv6 addresses are optional.
#       If this parameter is not specified, active checks are disabled.
#       Example: ServerActive=127.0.0.1:20051,zabbix.domain,[::1]:30051,::1,[12fc::1]
#
# Mandatory: no
# Default:
# ServerActive=

ServerActive=my.server.fqdn.or.ip

Just like the “Server” parameter discussed earlier, the “ServerActive” parameter has a comment that says this: “List of comma delimited IP:port (or hostname:port) pairs of Zabbix servers for active checks”. It should say “pairs of Zabbix servers/proxies for active checks”. Notice that I added “proxies” there. If the host for this agent is configured to have any active items, then the “ServerActive” parameter MUST have the proxy listed here!  Active items will not work if this parameter does not point to the proxy that monitors the agent.

Big massive caveat: Unless this agent is monitored in multiple Zabbix environments (meaning, multiple Zabbix servers get data from this agent), you do NOT want to have multiple entries in the “ServerActive” parameter. You will only want the IP or FQDN for the proxy that is configured to monitor this agent. If you don’t understand what I’m talking about, just believe me and only put the IP or FQDN of the proxy monitoring this agent in the “ServerActive” parameter.

After having made any necessary configuration changes to the agent, go ahead and restart it.

Ready to rock and roll….or so we thought

Ahhh, now the Zabbix server is configured to receive/pull data from a proxy, hosts are configured to be monitored by a proxy, and the agent has been restarted.  Funny enough, we’re not done yet.  If the agent is configured to do active checks (meaning that the “ServerActive” parameter isn’t blank), then a line like this will show up (may be slightly different for Zabbix 2.0/2.2 agents):

2133:20150516:205905.015 no active checks on server [192.168.56.102:10051]: host [test host] not found

Wait, what?!  Note that the “no active checks on server” sentence can be translated to be “no active checks on proxy”.  Zabbix uses the words server/proxy interchangeably in some places.  Feel free to double-check, but the host was configured in the Zabbix UI to be monitored by the proxy and the agent config is pointing to the proxy.  Even after that the proxy claims that it knows nothing about the host.  To understand what’s going on, it’s necessary to learn about something called the “Configuration cache”.

If a passive proxy is being used, there will be an utter lack of item data from this particular agent.  That’s the only way to notice this particular problem.

The Configuration Cache

For performance reasons, both the Zabbix server and all proxies have something called a “configuration cache”.  This is an internal cache in which Zabbix stores all information about hosts/items/triggers/etc. that are configured.  When talking about the Zabbix server itself, it updates its configuration cache every 60 seconds by default.  This means that every minute it will read the Zabbix database and find all changes that have been made by someone in the UI and then store them internally.  Only after the configuration cache has updated itself will the Zabbix server become aware of any changes that have been made in the UI.

For Zabbix proxies, the idea is similar to the Zabbix server configuration cache, except with proxies, the default update interval is every 3600 seconds.  Yes, one hour.  Let’s discuss what this means a bit more in detail.

The Configuration Cache Waiting Game

Let’s use the example of the agent we recently configured to be a monitored by a proxy.  First, we updated the host in the Zabbix UI to indicate it should be monitored by a proxy.  Second, we updated the zabbix_agentd.conf file to allow the agent to communicate with the proxy.  Now here is what happened:

  • The change in the Zabbix UI updated the Zabbix database to show the host should be monitored by a given proxy.
  • Within one minute of that change (assuming the server default was never changed) the Zabbix server updated its configuration cache by re-reading the database.  It found the configuration change and updated its internal cache.
  • While that went on, the zabbix_agentd.conf file was updated and the agent was restarted.
  • In the case of an active agent (meaning the “ServerActive” parameter is not blank), the agent attempted to get its list of items from the Proxy, but the proxy had not yet updated its configuration cache.
  • The proxy responded to the agent that it had no information about the agent and therefore we saw the log entry mentioned earlier.

If we were willing to wait up to one hour, the proxy would update its configuration cache and then the agent would be able to successfully pull its item list from the Proxy.  I’m not that patient.

Changing Configuration Cache Intervals

There is a place to change the frequency at which the proxy updates its configuration cache.  The location of that configuration option depends on which type of proxy was created: active or passive.

If one stops to think about it for a moment, that location is easy to figure out.  Remember that active proxies initiate communication with the server, so on an active proxy the configuration option is in zabbix_proxy.conf.  On the other hand, passive proxies wait for the Zabbix server to do all the talking.  With a passive proxy, the configuration option is in zabbix_server.conf on the Zabbix server.

Active Proxy Configuration Cache Interval

Open up the zabbix_proxy.conf file in your favorite editor and look for the “ConfigFrequency” parameter.

### Option: ConfigFrequency
#       How often proxy retrieves configuration data from Zabbix Server in seconds.
#       For a proxy in the passive mode this parameter will be ignored.
#
# Mandatory: no
# Range: 1-3600*24*7
# Default:
# ConfigFrequency=3600

Uncomment the configuration option and set it to something more sane.  In the real world this should probably be something like five minutes or more (depending on the size of the Zabbix environment).  For the sake of testing, set the “ConfigFrequency” option to “60”.  This means that the Zabbix proxy will pull its configuration from the Zabbix server every minute and will then update its configuration cache.

Restart the Zabbix proxy after making this change.

Passive Proxy Configuration Cache Interval

Open up the zabbix_server.conf file in your favorite editor and look for the “ProxyConfigFrequency” parameter.

### Option: ProxyConfigFrequency
#       How often Zabbix Server sends configuration data to a Zabbix Proxy in seconds.
#       This parameter is used only for proxies in the passive mode.
#
# Mandatory: no
# Range: 1-3600*24*7
# Default:
# ProxyConfigFrequency=3600

Uncomment the configuration option and set it to something more sane.  In the real world this should probably be something like five minutes or more (depending on the size of the Zabbix environment).  For the sake of testing, set the “ProxyConfigFrequency” option to “60”.  This means that the Zabbix server will push the configuration information to the proxy every minute and the proxy will then update its configuration cache.

Restart the Zabbix server after making this change.

Ready to rock and roll, for real this time

Now that the proxy will update its configuration cache at a much faster interval, testing with a proxy will be much, much easier.  If the agent being monitored is using active items, go ahead and restart the Zabbix agent.  Assuming that it has been at least one minute since the active/passive proxy/server service restart, the agent will no longer show any message like the log entry we saw before.  In fact, the proxy should be happily monitoring active and passive items on this agent now without any problem.

Forcing Configuration Cache Updates

The Zabbix server and Active Zabbix proxies can be forced to update their configuration cache, regardless of the configured interval.  Here are the relevant commands for both a proxy and a server:

zabbix_server -R config_cache_reload
zabbix_proxy -R config_cache_reload

This can be used when changes need to take effect sooner rather than later.  Just be sure to update the server’s configuration cache BEFORE the proxy’s.  This is because the server gives the proxy its configuration based on what the server has in its configuration cache.

There is no way to force a configuration cache update for a passive proxy.

Tidying Up

At this point everything should be just fine with agents being monitored by a proxy.  There are just a few things to keep in mind after this exercise:

  • Configuration changes made in the Zabbix UI are NOT immediately seen.  The Zabbix server must first update its configuration cache before it knows about them.
  • By default, the Zabbix server updates its configuration cache every 60 seconds.
  • By default, Zabbix proxies update their configuration cache every 3600 seconds.
  • When an active proxy queries the Zabbix server for its configuration, that configuration is pulled from the Zabbix server’s configuration cache.  This means that proxies will only know about what the Zabbix server has read from its own database!
  • Due to caching, monitoring changes can take quite some time to take effect.  In fact, the maximum amount of time it takes is equal to: (Zabbix server update interval) + (Zabbix proxy update interval).  In the case of active agents, the value for the “RefreshActiveChecks” parameter in zabbix_agentd.conf also needs to be added in.