How we eliminated video stream loss and achieved over 3,000 TB of recording per month

This case study will be useful for companies in the following sectors:

Objectives

• Ensure continuous, lossless video streaming and high I/O and recording speeds 
• Ensure high fault tolerance
• Process and store over 3,000 TB of data per month
• Reduce costs and simplify infrastructure scaling

vStack Products and Services

vStack HCP — a platform for building a fault-tolerant IT infrastructure with centralized management via a single dashboard

About the Company

A city video surveillance operator maintaining a camera infrastructure in a city with a population of over a million.

The system is used to record and store video streams from thousands of cameras 24/7.

The storage systems could not handle the constant load

Before the new solution was implemented, the infrastructure consisted of a mix of heterogeneous equipment: several servers and two data storage systems were in use. While this architecture technically met the requirements, limitations regularly arose in practice.

During peak periods, the storage system could not handle the data flow, resulting in video loss ranging from a few seconds to several minutes. For the city’s video surveillance operator, this meant increased risks for the client, as these failures directly impacted service quality and led to legitimate complaints.

Limitations and Requirements

Before searching for a new solution, the operator defined the following requirements:

• the system must be guaranteed to handle the data volume;
• it must provide a high level of fault tolerance;
• maintain stable performance under constant load. 

Particular emphasis was placed on the need to maintain high write speeds without performance drops, as well as ensuring rapid equipment service—specifically, the replacement of failed disks must be completed within a single workday and not impact system operations.

How the Infrastructure Was Built

Instead of a fragmented infrastructure, we deployed a vStack cluster, where all nodes are equal and operate as a single system

The disks of all servers were combined into a shared storage pool. The video stream is no longer dependent on specific hardware—data is distributed across the cluster.

The cluster was configured with N+2 redundancy. This means that the system continues to operate even if several components fail.

In exchange for the existing storage systems, the operator received both the vStack HCP software and new servers at the same cost.

We tested the solution under real-world conditions

Instead of traditional synthetic tests, we opted for an approach involving testing under real-world conditions. We deliberately avoided traditional metrics such as IOPS and latency, as they do not reflect the system’s behavior under continuous streaming conditions.

We deployed a vStack pilot stand at the customer’s site and directed a video stream from active cameras to it. This allowed us to test the system’s behavior under conditions of continuous recording and reading.

Results