Lab Day: Play with ZooKeeper

Objective

This activity is designed to provide students with hands-on experience in managing distributed systems using Apache ZooKeeper. By the end of this session, students will understand the fundamentals of ZooKeeper, its architecture, basic operations within a ZooKeeper ensemble, and how to use ZooKeeper to build distributed system primitives.

We recommend you forming groups of 3-4 students to do this lab together.

Prerequisites

  • Basic understanding of distributed systems and their challenges.
  • Java development environment set up on students’ computers (JDK 8 or above).
  • Apache ZooKeeper downloaded and installed. The latest version can be found on the official Apache ZooKeeper website.
  • ZooKeeper Python client library Kazoo downloaded and installed. The latest version can be found from wiki.

Part 1: Introduction to Apache ZooKeeper

What is Apache ZooKeeper?

Apache ZooKeeper is an open-source server which enables highly reliable distributed coordination. It is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications.

Key Features

  • Coordination and Synchronization: ZooKeeper helps to manage and coordinate a large cluster of machines.
  • Configuration Management: Stores and manages the configuration of the distributed system.
  • Naming Registry: Provides a namespace to uniquely identify nodes in a distributed system.

ZooKeeper Architecture

  • ZooKeeper Servers: In a production environment, ZooKeeper runs in a cluster known as an ensemble.
  • Znodes: Data nodes in ZooKeeper’s data hierarchy, resembling a file system tree.

Part 2: Setting Up Apache ZooKeeper

Installation

  1. Download the latest ZooKeeper release from official page and extract the ZooKeeper package to a directory of your choice. You can do it manually or run commandlines (Linux):
    1
    2
    wget https://dlcdn.apache.org/zookeeper/zookeeper-3.9.2/apache-zookeeper-3.9.2-bin.tar.gz
    tar -xvzf apache-zookeeper-3.9.2-bin.tar.gz
  2. Create a configuration file named zoo.cfg inside the conf directory with the following content: 
    1
    2
    3
    tickTime=2000
    dataDir=/path/to/zookeeper/data
    clientPort=2181
  3. Replace /path/to/zookeeper/data with the actual data directory path. For example, you can create a new directory under zookeeper called data, and put the full path `` to the configuration.

Starting ZooKeeper Server

  • Run the following command in the terminal:

    1
    2
    bin/zkServer.sh start
    bin/zkServer.sh status

    If you see success output which look like:

    1
    2
    3
    4
    ZooKeeper JMX enabled by default
    Using config: zookeeper/bin/../conf/zoo.cfg
    Client port found: 2181. Client address: localhost. Client SSL: false.
    Mode: standalone

    Congratulations! You’ve successfully run ZooKeeper on your computer. Now you can proceed and use client to play with it.

    If you did not install Java properly, you may encounter error information such as: 

    1
    Error: JAVA_HOME is not set and java could not be found in PATH.

    If so, install Java environment first by referring to official guide.

Part 3: Hands-On Lab

Task 1: Connecting to ZooKeeper

  1. Start the ZooKeeper CLI using the command: 
    1
    bin/zkCli.sh -server 127.0.0.1:2181
  2. You are now connected to your ZooKeeper server.

Task 2: Basic Operations

  • Create a Znode: 
    1
    create /myfirstznode "My first ZooKeeper data"
  • List Znodes: 
    1
    ls /
  • Get Znode Data: 
    1
    get /myfirstznode
  • Update Znode Data: 
    1
    set /myfirstznode "Updated data"
  • Delete Znode: 
    1
    delete /myfirstznode
    There are so many more you can do with the client, you can try the following features:
  • Ephemeral node
  • Sequential node
  • Watches
  • getChildren
  • …

Type h to get a list of commands in the client.

Task 3: Build Leader Election with ZooKeeper

In this task you will need to build a leader election service using ZooKeeper.
Your logic will be implemented in Python using zookeeper client interfaces.

  1. Download and install Kazoo. The simplest way is to use pip

    1
    pip install kazoo

  2. Try Kazoo in python

    1
    2
    3
    4
    python
    >>> import kazoo
    >>> kazoo.__version__
      '2.10.0'

  3. Write a python program to emulate the leader election. You can start new instances by running the program, or shutdown old instances by terminating the program anytime. Your codes should ensure there is always one and only one leader. Here is an incomplete sample program LE.py for you to start. To execute, just run python LE.py in multiple terminals:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    from kazoo.client import KazooClient
    from kazoo.client import KazooState
    import time

    def my_listener(state):	
    	if state == KazooState.LOST:
            	print("Connection Lost/Closed")
    	elif state == KazooState.SUSPENDED:
            	print("Connection Suspended")
    	else:
            	print("Welcome to the Zookeeper")


    zk = KazooClient(hosts='localhost:2181')
    zk.add_listener(my_listener)
    zk.start()

    #create a node
    zk.create("/leader", b"", ephemeral= True)
    #check if a node exists
    if zk.exists("/leader"):
        print("There is a leader!")

    # add your logic
    while True:
        time.sleep(5)
        print("do something")
    	
    zk.stop()

  4. Write the advanced version of leader election. Specifically, consider these questions:

    • Does your logic include any busy polling?
    • Does your logic avoid herd effect?
    • How to reduce the recovery time?
    • …

  5. Optional exercises for you

    • Distributed Queue: Implement queues that can be used by distributed systems to process tasks in a scalable manner.
    • Distributed Barrier: Implement barriers that allow nodes in a distributed system to wait at certain points until all nodes have reached the same state.
    • Unique ID Generation: Generate unique IDs (1-1024) across a distributed system, ensuring that IDs are not duplicated and expired ids can be recycled.
    • Reliable Messaging: Ensure messages are reliably passed between nodes, with mechanisms for persistence and retry.

Bounty Job: Battle Royale, details to be announced

We will announce the task in the middle of class.

Resources