A Simple Visual Inferencing App: from our laptop to edge, in minutes (part 2/2)

This series of posts includes 2 parts:

• Part 1, here, is about how to build, run a microservices-based Visual Inferencing app within our laptop;
• Part 2, which is this post, is about why we need Edge Computing and how we can roll out and manage such an app to many edge sites, at scale.

So, if you’ve gone through part 1, you’ve learned how to build a visual inferencing app like any other microservices-based one.

Let’s talk about edge computing in this post.

Note: All code within this post can be found in my GitHub repo at brightzheng100/vi-people-counting-example. Do check it out to dive into any levels that you want.

Why Edge Computing? And what are the major challenges?

But, what does Edge Computing mean exactly? And why we should embrace it?

Simply put, Edge Computing is a new way of how data is being handled, processed, and delivered from hundreds or even millions of devices around the world with much better user experience, as the proximity devices can offer services with much lower latency, better stability, higher resilience, and more highly localized customization for the users nearby.

How if the Singapore Government wants to roll out such “People Counting Workstation” to 1,000 places in a state country like Singapore?

How if the China Government wants to roll out such “People Counting Workstation” to 1,000 * 1,000 places in a much bigger country?

These are real Edge Computing use cases, in the real world.

But, what are the major challenges for doing so? Let’s think about these questions first:

• Can you imagine how much rollout effort required for the day 1, for 1,000 * 1,000 places/sites?
• Can you imagine how much maintenance effort required for the day 2 operations and beyond for the software lifecycle management, e.g. upgrade, patch, monitor?

This is where the Edge Computing solutions really shine, among tons of the advantages.

Let me walk you through how to embrace Edge Computing By Adopting IBM Edge Application Manager (IEAM).

Concepts

Before we start, it’d be better to understand some core concepts within IEAM.

• Management Hub. The IEAM Management Hub is the centralized management plane designed specifically for edge node management to minimize deployment risks and to manage the service software lifecycle on edge nodes fully autonomously. The administrators, including hub administrator, service administrator, manage the nodes, services, deployments and policies by interacting with the hub components. Software developers develop and publish edge services to the management hub.
• Hub Components. There are quite some core components deployed into the Management Hub, which include: 1) Agbot: Which is the short form of Agreement Bot. The Agbot instances are created centrally and are responsible for handshaking, negotiating, deploying workloads and machine learning models to IEAM. 2) Exchange: The Exchange provides the REST API that holds all the definitions (patterns, policies, services, and so on) used by all the other components in IEAM. 3) Management Console: The web UI used by IEAM administrators to view and manage the components of IEAM. 4) MMS: The Model Management System (MMS) facilitates the storage, delivery, and security of models, data, and other metadata packages needed by edge services. This enables edge nodes to easily send and receive models and metadata to and from the cloud.
• Edge Node: Any edge device, edge cluster, or edge gateway where edge computing takes place. 1) Edge Cluster: A computer in a remote operations facility that runs enterprise application workloads and shared services. An edge cluster can be used to connect to an edge device, connect to another edge cluster, or serve as an edge gateway for connecting to the cloud or to a larger network. Nowadays, we typically will build Edge Cluster by using Kubernetes, e.g. Red Hat OpenShift Container Platform, Rancher’s K3s, or any other Kubernetes distributions you prefer. 2) Edge Device: A piece of equipment, such as an assembly machine on a factory floor, an ATM, an intelligent camera, or an automobile, that has integrated compute capacity on which meaningful work can be performed and data that is collected or produced. In the real world, a VM / laptop installed with Docker can be an Edge Device too.
• Edge Agent: This is a containerized software that is installed as an agent on edge cluster or edge device to interact with IEAM Hub components for management purposes.
• Edge Service: An Edge Service is a business service that is designed specifically to be deployed on an edge cluster to perform business logic. For example, the ATM software system, or our visual inferencing app components that we’re going to use.

The evolving of architecture

Assumptions

Let’s assume that:

• The IBM Edge Application Server v4.1 has been installed on OpenShift Container Platform, which can be deployed on-prem, or on any cloud (e.g. IBM Cloud);
• Within the working laptop, we’ve logged into OCP by oc login;
• We’re going to set up the Edge Node based on a Ubuntu VM.

Important Note: If you want to simply taste the power of edge computing by using open source software (OSS), do try out Open Horizon for devepment and testing purposes as it’s simple and lightweight. I’ve already compiled an end-to-end process, here, by running everything within one single VM.
IBM Edge Application Manager (IEAM), which is an enterprise-grade IBM product built on top of OSS Open Horizon.

Get started

Set up a VM with a camera mounted

Let’s assume that a Ubuntu Bionic VM has been set up in VirtualBox (or a real edge site), which will be acting as our Edge Node.

Note: By the way, if you’re trying it out on your MacBook, there is a permission issue which will cause the VM crash once the app tries to connect the Facetime HD Camera. My current workaround is to start VirtualBox as root: sudo virtualbox. Check out the forum post here for the follow-up: https://forums.virtualbox.org/viewtopic.php?f=8&t=99299&p=481734#p481734

We need to stop the VM first and do some configuration so that we can attach the camera.

In VirtualBox UI, select this VM and click Settings:

1. In Display tab:

• Video Memory, change it to 64M;
• Make sure Graphics Controller is set as VMSVGA;
• Check Enable 3D acceleration

2. In Ports tab:

• Check Enable USB Controller, choose USB 2.0 (EHCI) Controller

And then start it up again.

# These are from my env, change them accordingly for yours
$ EDGE_NODE_NAME="ubuntu-edge-node" \
  EDGE_NODE_IP="192.168.56.101" \
  EDGE_NODE_USER="bright"
# Check the webcams and we can see the FaceTime HD Camera
$ vboxmanage list webcams
Video Input Devices: 1
.1 "FaceTime HD Camera (Built-in)"
0x8020000005ac8514
# Attach the default Facetime webcam to the VM
$ sudo vboxmanage controlvm "${EDGE_NODE_NAME}" webcam attach .1

Generate necessary files for Edge Node(s) by IEAM Hub Admin

A set of files will be needed to install the IEAM agent on the edge device(s) and/or edge cluster(s), and register them with IEAM Hub. This typically could be done by IEAM Hub Amin.

Let’s do this within our laptop.

# Clone the repo if you haven't done so
git clone https://github.com/brightzheng100/vi-people-counting-example
cd vi-people-counting-example
# Create a temporary folder
mkdir _edge-node-files-x86_64-Linux && cd _edge-node-files-x86_64-Linux
# Download the latest version of edgeNodeFiles.sh
curl -O https://raw.githubusercontent.com/open-horizon/anax/v4.1/agent-install/edgeNodeFiles.sh && chmod +x edgeNodeFiles.sh
# Copy the horizon-edge-packages-4.1.0.tar.gz, which can be downloaded from IBM PPA #CC6G0ML, to current folder.
# This is just my case and you may copy it from your download folder.
cp ~/workspaces/cloud-paks/IEAM/_download/horizon-edge-packages-4.1.0.tar.gz ./
# Run the edgeNodeFiles.sh script to gather the necessary files:
export CLUSTER_URL="https://$(oc get routes icp-console -o jsonpath='{.spec.host}' -n kube-system)"
export CLUSTER_USER="admin"
export CLUSTER_PW="<YOUR IEAM ADMIN PASSWORD>"
./edgeNodeFiles.sh x86_64-Linux -t -d bionic
# Check the files generated
ls -l
-rw-r--r--  1 brightzheng  staff   35920785 Aug  5 17:24 agentInstallFiles-x86_64-Linux.tar.gz
-rw-r--r--  1 brightzheng  staff  359830572 Aug  5 17:23 horizon-edge-packages-4.1.0.tar.gz
-rw-------  1 brightzheng  staff        179 Aug  5 17:24 key.txt
...
# And the content of the final .gz file
tar -tvf agentInstallFiles-x86_64-Linux.tar.gz
-rw-r--r--  0 brightzheng staff     319 Aug  5 17:24 agent-install.cfg
-rw-r--r--  0 brightzheng staff    2021 Aug  5 17:24 agent-install.crt
-rwxr-xr-x  0 brightzheng staff   69657 Aug  5 17:24 agent-install.sh
-rw-r--r--  0 brightzheng staff   43360 Jun  4 15:04 bluehorizon_2.26.12~ppa~ubuntu.bionic_all.deb
-rw-r--r--  0 brightzheng staff 16386958 Jun  4 15:04 horizon-cli_2.26.12~ppa~ubuntu.bionic_amd64.deb
-rw-r--r--  0 brightzheng staff 19470100 Jun  4 15:04 horizon_2.26.12~ppa~ubuntu.bionic_amd64.deb

Note:
1. The agentInstallFiles-XXX.tar.gz will be the zipped file that to be handed over to IEAM Node Admin;
2. A key has been generated also, as key.txt, which should be handed over to IEAM Node Admin as well — we can generate different keys for different IEAM Node Admins, which are managed centrally;
3. This procedure is to generate files for Edge Device. To generate files for Edge Cluster (e.g. K8s/OCP), instead of Edge Device, the procedure is slightly different. Please consult the doc here.

Register the VM as Edge Node by IEAM Node Admin

# Assuming we're still in the folder created by previous step
$ pwd
.../vi-people-counting-example/_edge-node-files-x86_64-Linux
# Copy over the files to our Edge Node
$ scp agentInstallFiles-x86_64-Linux.tar.gz key.txt ${EDGE_NODE_USER}@${EDGE_NODE_IP}:~/
# Let's ssh into the Edge Node
$ ssh ${EDGE_NODE_USER}@${EDGE_NODE_IP}

Now we’re in the Edge Node:

$ ls
agentInstallFiles-x86_64-Linux.tar.gz  key.txt
# untar the file
$ tar xf agentInstallFiles-x86_64-Linux.tar.gz
# Check the files
$ ls -l
-rw-r--r-- 1 bright bright      319 Aug  5 09:24 agent-install.cfg
-rw-r--r-- 1 bright bright     2021 Aug  5 09:24 agent-install.crt
-rwxr-xr-x 1 bright bright    69657 Aug  5 09:24 agent-install.sh
-rw-r--r-- 1 bright bright 35920785 Aug  5 09:27 agentInstallFiles-x86_64-Linux.tar.gz
-rw-r--r-- 1 bright bright    43360 Jun  4 07:04 bluehorizon_2.26.12~ppa~ubuntu.bionic_all.deb
-rw-r--r-- 1 bright bright 16386958 Jun  4 07:04 horizon-cli_2.26.12~ppa~ubuntu.bionic_amd64.deb
-rw-r--r-- 1 bright bright 19470100 Jun  4 07:04 horizon_2.26.12~ppa~ubuntu.bionic_amd64.deb
-rw------- 1 bright bright      179 Aug  5 09:27 key.txt
# Install software
$ sudo apt-get update && sudo apt-get install jq -y
$ sudo dpkg -i horizon-cli_2.26.12~ppa~ubuntu.bionic_amd64.deb
# Install Docker
$ sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
$ sudo apt-get update
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
$ sudo usermod -aG docker $(whoami) # remember to log out and in again to use `docker` instead of `sudo docker`
# Export the auth with the key
$ eval export $(cat agent-install.cfg) \
  export HZN_EXCHANGE_USER_AUTH="iamapikey:$( cat key.txt | jq -r '.apikey' )"
# Define the node policy, refer to /horizon/node.policy.json,
# which are some arbitrary attributes to be as properties and constraints
# In real world, you may standardize these with some sensible naming patterns
$ cat > node.policy.json <<EOF
{
  "properties": [
    { 
      "name": "openhorizon.allowPrivileged", "value": true
    },
    { 
      "name": "vendor", "value": "Ubuntu"
    },
    { 
      "name": "code", "value": "Bionic"
    },
    { 
      "name": "geo", "value": "SGP"
    },
    
    { 
      "name": "visual_inferencing_workstation", "value": "yes"
    },
    { 
      "name": "webcam_attached", "value": "yes"
    }
  ],
  "constraints": [
  ]
}
EOF
# Install and configure the Horizon agent and to register this VM as the edge device, with node policy
$ sudo mkdir -p /private/etc/horizon/
$ sudo ./agent-install.sh -i . -u $HZN_EXCHANGE_USER_AUTH -z /agentInstallFiles-x86_64-Linux.tar.gz -n node.policy.json
# Once IEAM agent has been installed and node has been registered, we can check it out
$ hzn node list
{
  "id": "ubuntu-edge-node",
  "organization": "mycluster",
  "pattern": "",
  "name": "ubuntu-edge-node",
  "nodeType": "device",
  "token_last_valid_time": "2020-08-04 17:53:34 +0800 +08",
  "token_valid": true,
  "ha": false,
  "configstate": {
    "state": "configured",
    "last_update_time": "2020-08-04 17:53:36 +0800 +08"
  },
  "configuration": {
    "exchange_api": "https://icp-console.XXX.cloud/edge-exchange/v1/",
    "exchange_version": "2.30.0",
    "required_minimum_exchange_version": "2.23.0",
    "preferred_exchange_version": "2.23.0",
    "mms_api": "https://icp-console.XXX.cloud/edge-css",
    "architecture": "amd64",
    "horizon_version": "2.26.12"
  }
}

You can repeat this to set up more Edge Nodes.

Please note that there are 3 major ways to setup Edge Nodes, at scale:

1. Manual agent installation and registration, like what I just did;
2. Bulk agent installation and registration;
3. Secure Device Onboard agent installation and registration

Publish Services to IEAM by IEAM Service Admin

Typically, we will publish services in a management workstation, or through CI/CD pipeline by IEAM Service Admin.

Let’s do it within our laptop.

# Assuming we're still in the folder created by previous step
pwd
.../vi-people-counting-example/_edge-node-files-x86_64-Linux
# Export the auth with the key
eval export $(cat agent-install.cfg)
export HZN_EXCHANGE_USER_AUTH="iamapikey:$( cat key.txt | jq -r '.apikey' )"
# cd to the top folder
cd $(git rev-parse --show-toplevel)
# Register the "bundled" services as a logical service
export DOCKERHUB_ID="quay.io/brightzheng100"
cat horizon/service.definition.json | \
  sed "s|__DOCKERHUB_ID__|${DOCKERHUB_ID}|g" | \
  hzn exchange service publish -f- -P -O

Once the service is published, we can check it out by:

# List to check what services now we have
hzn exchange service list
[
  "mycluster/people-counting-bundle-service_1.0.0_amd64"
]

Deploy Edge Services to desired Edge Node(s)

Typically, we will deploy services in a management workstation, or through CI/CD pipeline by IEAM Service Admin, based on deployment policy and/or pattern, to all desired Edge Nodes.

Let’s do it within our laptop.

# cd to the top folder
cd $(git rev-parse --show-toplevel)
# Register the "bundled" services as a logical service
eval $(hzn util configconv -f horizon/hzn.json)
hzn exchange deployment addpolicy -f horizon/deployment.policy.json ${HZN_ORG_ID}/policy-${SERVICE_NAME}_${SERVICE_VERSION}

The outcome?

The IEAM Agent in the Edge Node(s)

The systemd managed IEAM Agent, namely horizon, will be running in the Edge Node:

$ systemctl status horizon
● horizon.service - Service for Horizon control system (cf. https://bluehorizon.network)
   Loaded: loaded (/lib/systemd/system/horizon.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-08-05 09:36:47 UTC; 1h 7min ago
  Process: 730 ExecStartPre=/usr/horizon/sbin/horizon-prestart / (code=exited, status=0/SUCCESS)
 Main PID: 844 (anax)
    Tasks: 8 (limit: 2317)
   CGroup: /system.slice/horizon.service
           └─844 /usr/horizon/bin/anax -v 3 -logtostderr -config /etc/horizon/anax.json

The desired services will be automatically deployed to the matched Edge Node(s), as containers:

bright@ubuntu-edge-node:~$ docker ps
CONTAINER ID        IMAGE                                           COMMAND                  CREATED             STATUS              PORTS                    NAMES
79a487db70eb        quay.io/brightzheng100/detector-service_amd64   "python /darknet.py …"   5 minutes ago       Up 5 minutes        0.0.0.0:5252->80/tcp     8b8e79dba89e1cecbe4d2050471e95a43ad9c329a1a9971e1bbdd31d5f5a9e77-detector-service
cbbe4b52ac17        quay.io/brightzheng100/detector-mqtt_amd64      "./mqtt.sh"              5 minutes ago       Up 5 minutes        0.0.0.0:1883->1883/tcp   8b8e79dba89e1cecbe4d2050471e95a43ad9c329a1a9971e1bbdd31d5f5a9e77-detector-mqtt
cb683ef536ad        quay.io/brightzheng100/detector-monitor_amd64   "python /app/monitor…"   5 minutes ago       Up 5 minutes        0.0.0.0:5200->5200/tcp   8b8e79dba89e1cecbe4d2050471e95a43ad9c329a1a9971e1bbdd31d5f5a9e77-detector-monitor
c92bf02644b9        quay.io/brightzheng100/detector-cam_amd64       "./start.sh"             5 minutes ago       Up 5 minutes        0.0.0.0:8888->80/tcp     8b8e79dba89e1cecbe4d2050471e95a43ad9c329a1a9971e1bbdd31d5f5a9e77-detector-cam
e61a26797f29        quay.io/brightzheng100/detector-app_amd64       "python app.py"          5 minutes ago       Up 5 minutes                                 8b8e79dba89e1cecbe4d2050471e95a43ad9c329a1a9971e1bbdd31d5f5a9e77-detector-app

The IEAM Hub

We can visualize the managed Edge Nodes and services through IEAM Hub.

In this case, we now just have one Edge Device with services already deployed — in the real world the Edge Nodes, which can be Edge Device and or Edge Cluster, may be thousands or even millions.

Live Visual Inferencing for people counting

We now can access the Edge Node’s detector-monitor container through exposed port at 5200, in my case it's: http://192.168.56.101:5200

As expected, the camera will capture live photos for visual inferencing and count the people:

Well, here just myself so nothing fancy but, please imagine how it works in the real world!

Clean Up

Remove the services deployed in Edge Node(s) by removing the deployment policy

# cd to the top folder
cd $(git rev-parse --show-toplevel)
# remove the bundle service
eval $(hzn util configconv -f horizon/hzn.json)
hzn exchange deployment removepolicy ${HZN_ORG_ID}/policy-${SERVICE_NAME}_${SERVICE_VERSION} -f

Remove the services

To remove the services involved:

# cd to the top folder
cd $(git rev-parse --show-toplevel)
# the bundle service
eval $(hzn util configconv -f horizon/hzn.json)
hzn exchange service remove "${HZN_ORG_ID}/${SERVICE_NAME}_${SERVICE_VERSION}_amd64" -f

Unregister the Edge Node from IEAM

Log into the Edge Node:

hzn unregister -f

Or we can remove them from the IEAM Hub’s Exchange, from any management workstation:

# Assuming the key has been exported, like
# export HZN_EXCHANGE_USER_AUTH="iamapikey:$( cat key.txt | jq -r '.apikey' )"
hzn exchange node remove "mycluster/ubuntu-edge-node"

Conclusion

Building Visual Inferencing apps can be fun, but rolling out such apps to many edge sites and enabling them to be managed autonomously can be huge challenges.

By adopting edge computing technology, as what I’ve shown by using IBM Edge Application Manager (IEAM), these challenges can be easily mitigated.

Do check out the docs here to start evaluating this cool technology to solve more real-world problems!