Background

We at CAPIOT are developing an ERP solution for one of the leading Indian e-commerce players. We chose the micro-services approach for this. Each module would be an independent node.js service that would define its own APIs. When a cross-service call is needed the requesting service would call the API for the requested service. We quickly realized the need for a discovery service or a registry for our micro-services.

During the initial phases of development when we only had 2 or 3 services a registry was not much of a concern. But we had to find a solution quickly.

In production, each module could have multiple instances running. We figured that if the registry could randomly return one of the active endpoints of a service, then we have achieved a very simple load-balancer too. The initial approach was to develop another node.js based micro-service. This was quickly ruled out as an approach. The registry service would be one of the most frequently accessed services and we didn’t want a scenario where we had to worry a lot about the stability of the registry itself.

We then looked at proven open-source technologies, and we found two – Apache ZooKeeper and Redis.

In this article i would explain how we developed a discovery service / registry for our micro-services environment using ZooKeeper.

Basic requirements for the registry

This registry should have these features for us to use.

1. Active list of end points. The registry at all times must have the endpoints of all my active services. Each of these service endpoints must be grouped under a key. This list should only contain the active endpoints and any service that is no longer unavailable must be removed from this list. For e.g. if i have a service called warehouse and I’m running 3 instances of warehouse service, then my registry should have all the three endpoints available under, say warehouse.
2. Return a random endpoint for a service. From the available list of endpoints, the registry should return a random endpoint for a service.
3. Lightweight and low memory footprint

Developing a ZooKeeper based registry

We designed the system around Apache ZooKeeper. ZooKeeper is positioned as a system used for distributed coordination and configuration management. So this very much fits our requirements.

ZooKeeper nodes, or Znodes, have a tree like structure and can hold data. So it’s easy for us to create a structure like myRegistry/endpoints/warehouse and store all warehouse endpoints under it . ZooKeeper also has ephemeral nodes. Ephemeral nodes are only active as long as the session that created it is active. So essentially we could have all or services connect to the required path and store their endpoints as ephemeral nodes. If the service goes down, the ephemeral node that it created also cease to exist.

Since our services are built on node.js, we used the  node-zookeeper-client and wrapped the functionalities to expose 3 methods as a library. We call this library puttu and the following are the methods exposed,

• connect(, )
• register(, , )
• get()

connect(, )

This is the first call that has to be made from the service. It accepts the ZooKeeper connection string and a base path. Base path is where the endpoints would be saved.

var puttu = require('./index.js')
puttu.connect('localhost:2181', '/myRegistry/endpoints')

Let’s check the ZooKeeper server by connecting to it using the zkCli client and check the structure that was created,

[zk: localhost:2181(CONNECTED) 3] ls /
[zookeeper, myRegistry]
[zk: localhost:2181(CONNECTED) 4] ls /myRegistry
[endpoints]
[zk: localhost:2181(CONNECTED) 5] ls /myRegistry/endpoints
[warehouse]
[zk: localhost:2181(CONNECTED) 6]

register(, , )

The path is where we want to save our configuration. In this example this would be warehouse. The second parameter, data is a JSON object that would have the following structure.

{
 "protocol": "https",
 "port": 10001,
 "api": "/warehouse/v1"
}

And finally interface, it is the interface on the server on which the service is listening on. If interface is provided then we try and find the IP address by calling the os module – os.networkInterfaces(). But this may or may not have the intended effect, so you can ignore interface and provide the IP address by setting a process environment variable PUTTU_IP

var puttu = require('./index.js');

puttu.connect('localhost:2181', '/myRegistry/endpoints');
var data = {
 protocol: 'https',
 port: 1001,
 api: '/warehouse/v1'
};

puttu.register('warehouse', data, "en0").then(
 () => console.log('Registered self'),
 e => console.log(e)
);

OR

var puttu = require('./index.js');

puttu.connect('localhost:2181', '/myRegistry/endpoints');

process.env.PUTTU_IP = "192.168.1.100";

var data = {
 protocol: 'https',
 port: 1001,
 api: '/warehouse/v1'
};

puttu.register('warehouse', data).then(
 () => console.log('Registered self'),
 e => console.log(e)
);

get()

Given a path, e.g. warehouse, this will return a random endpoint from the list of registered endpoints.

puttu.get('warehouse').then(
 d => console.log(d),
 e => console.log(e)
)

Testing

Using zkCli, we can test what gets stored under /myRegistry/endpoints/warehouse. I started 4 services using this library. So when i query ZooKeeper, i get the following data,

[zk: localhost:2181(CONNECTED) 9] ls /myRegistry/endpoints/warehouse
[warehouse0000000009, warehouse0000000012, warehouse0000000010, warehouse0000000011]
[zk: localhost:2181(CONNECTED) 10] get /myRegistry/endpoints/warehouse/warehouse0000000009
https://192.168.1.94:1001/warehouse/v1
cZxid = 0xab
ctime = Sat Dec 31 00:26:31 IST 2016
mZxid = 0xab
mtime = Sat Dec 31 00:26:31 IST 2016
pZxid = 0xab
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x1595109d72a0017
dataLength = 38
numChildren = 0
[zk: localhost:2181(CONNECTED) 11]

If the service that made the warehouse0000000009 entry is killed, the end point is also removed from ZooKeeper.

Every time i call get(“warehouse”), i also the endpoint from one of these entries, selected at random.

Links

1. puttu – https://github.com/capiotsoftware/puttu
2. node-zookeeper-client – https://github.com/alexguan/node-zookeeper-client
3. ZooKeeper znodes – https://zookeeper.apache.org/doc/r3.2.1/zookeeperProgrammers.html#sc_zkDataModel_znodes