Dynamically discovering settings for a cluster?
Pretend I run a cluster, for hosting a site. Pretend that I have three-six web-nodes, and each one needs to know which database host to contact.
How do I control that?
Right now I have a /etc/settings.conf file, more or less, deployed by Slaughter. That works. Another common pattern is to use a hostname - for example pmaster.example.org.
However failover isn't considered here. If I wanted to update to point to a secondary database I'd need to either:
- Add code to retry the second host on failure.
- Worry about divergence if some hosts used DB1, then DB2, then DB1 came back online.
- Failover is easy. Fail-back is probably best avoided.
- Worry about DNS caches and TTL.
In short I'm imagining there are several situations where you want to abstract away the configuration in a cluster-wide manner. (A real solution is obviously floating per-service IPs. Via HAProxy, Keepalived, ucarp, etc. People do that quite often for database specifically, but not for redis-servers, etc.)
So I'm pondering what is essentially a multi-cast accessible key-value storage system.
Have a deamon on the VLAN which will respond to multicast questions like "get db", or "get cache", with a hostname/IP/result.
Suddenly your code would read:
- Send mcast question ("which db?").
- Get mcast reply ("db1").
- Connect to db1.
To me that seems like it should be genuinely useful. But I'm unsure if I'm trading one set of problems for another.
I can't find any examples of existing tools/deamons in this area, which either means I'm being novel, innovate, and interesting. Or I'm over thinking...