This is not going to be the usual treatise on scalable architectures. Instead, I am going to stitch together many seemingly unrelated ideas to arrive at the point. At Epicode we build highly scalable telephony middleware that is used by Voice AI vendors and telephony application vendors around the world. Their developers use our API to build enterprise applications. This is an industry that really values low running costs, reliability, high availability and scalability. This is the story about how we ingrained these qualities in our middleware products. This is the second article in the Middleware Musings series, the first one can be read here.
First, let’s understand the concept of middleware. None of the applications these days are monoliths. There are many layers underneath that developers are often unaware of.
Let’s take any web application, and let’s delve into the layers:
- The UI that interacts with the human user is mostly written using HTML, JavaScript, CSS, etc. This requires the knowledge of how to interact with the user of that particular application.
- Those UI scripts are interpreted by the web browser, which could be Chrome, Edge, Safari, Firefox, etc. But the web browser knows nothing about the application. But they have to know how to run any UI scripts.
- The web browsers run on an OS, which could be Android, iOS, MacOS, Linux or Windows. So the OS layer should allow the web browser to communicate with the Internet by implementing the TCP/IP and UDP/IP layer. Computers on the local network or Internet are identified by an internet address, like 10.50.80.31 or a FQDN. And the applications on those machines are identified by TCP or UDP ports. The OS knows nothing about web browsers. It will treat any application trying to connect to the internet the same way.
- Next is the datalink + physical layer, which could be ethernet, Wi-Fi or Cellular (4G/5G) components. This component will provide a MAC address to the computer, so that it can join the network. The datalink layer doesn’t care which OS is running, or if there is an OS, as long as the device driver works. There are bare-metal applications that run directly on the hardware.
As you can see, at least 4 layers got involved even before a single byte goes out of your client device. Beyond the top layer, none of the lower layers need to understand your application. However, each layer does a very different set of functions, and they entirely focus on those functions. Similarly on the server side, there will be many layers before reaching the web server, and the web application running over it. The web browser and the web server are the middleware located right below the client and server application respectively. A middleware can be just one layer or a set of layers between UI application and OS.
As you go from the top layer that interacts with the user, towards the bottom, each subsequent layer gets more generic, more complex, and more and more thought must go into designing them. Now consider the longevity of these lower layers. Browsers have been around over 30 years, TCP/IP for 40 years and Ethernet for 50 years. Sure, they have been continuously improved over the years with better performance and features, but the concept and overall functionality have remained the same.
Nobody can vibe-code a middleware and rush to market with it. That can happen only with the top layer, which is the user application. Just like the lower layers, middleware has to be built over many years with multiple iterations before it becomes really useful.
A telephony middleware must provide all the expected APIs like dialing, call progress analysis, streaming, recording, transferring, bridging, etc., while ensuring that these functions are reliable, very fast, highly available, and scalable. The last two qualities also mean that the middleware should be distributed and load balanced, locally and geographically. After 3 iterations over many years, Epicode has developed a four layer middleware, consisting of a communication layer, state synchronization layer, cluster orchestration layer and the telephony layer. And we take security very seriously, you can take a deep dive into the encryption and authentication paradigms we implement in an earlier article.
Communication Layer
Distributed architecture involving real time traffic needs the ability to communicate instantaneously between various components. And the most appropriate model is publish-subscribe, where one can publish an event and forget about it. Any one or all of the subscribers may decide to act upon it.
State Synchronization
Telephony applications are stateful. During a call, it is associated with any one signaling and media server, and the state of the call changes many times before it ends. In a distributed architecture, every copy of the relevant applications have to be aware of the current state of the calls. Storing state in a persistent database will be too slow to be useful. Therefore, the state of calls and agents and bots are maintained in heap memory in a distributed fashion, in what we call a transient SQL database. This ensures that all select queries are done within the heap memory of the application, while the changes are replicated to all copies across the distributed installation.
Cluster Orchestration
This layer has the responsibility to seamlessly bring multiple applications to work together towards a common goal while providing redundancy and high availability. By tracking all the applications running in the cluster, it provides service discovery, feature licensing, application load balancing, broadcast and narrowcast messaging; thereby enabling the applications to behave as a swarm. The swarm application pattern is an architectural approach in which a group of decentralized, self-organizing agents or computing nodes work together to solve complex problems or maintain a desired state. The application load balancing is based on actual load on each instance, and doesn’t rely on round-robin.
Telephony APIs
Epicode’s telephony APIs are built upon the above layers, and therefore can be fully distributed, load balanced and highly available and horizontally scalable. In cloud environments, auto scaling and shrinking is possible with even stateful applications. You can assign 100s of SIP trunks to a bunch of dialers and they can load balance across all of them while complying with calls-per-second (CPS) specifications, capacity and status of each trunks. For example, if you have one trunk with 1000 channels and 50cps, second trunk with 100 channels and 10cps, and a third trunk with 10 channels with 1 cps, the dialer will match the CPS capacity of each trunk while dialing through them simultaneously. The in-built call progress analysis (CPA) can detect answering machines in 1700ms, apart from detecting various industry standard tone patterns, thus under use in many fortune 500 companies. There are many other API sets for predictive dialing, audio streaming, voice bot orchestration, SIPREC recorder, etc.
Bottom Line
Over the years, the contents of these layers have changed. Many in-house components have been replaced by vastly superior open source projects with permissive licenses to keep the cost low. The three bottom layers can be deployed in a Kubernetes cluster or as service in any Linux distro. Most of the telephony layer can too, but with some exceptions. This architecture can live on cloud, on-premise or in a hybrid setup. This particular architecture has scaled to even 10000 channels in one distributed installation spanning 3 cities.
In software design, there is more than one right way. But it takes many years and multiple iterations to figure out one way that works for you. One could always wish that we knew 10 years ago what we know now. But the enlightenment comes only after making the journey. And this was our right way.