OceanStor Pacific Series 8.1.3 Intelligent Video and Image Storage Best Practice with Milestone

Issue 01
Date 2023-01-20

 1/ Overview

Introduction

Video management systems are important parts of security protection systems. The planning and configuration of video storage are especially important. Improper planning and configuration often cause performance and availability issues. This document verifies the video storage capability based on the Milestone XProtect system service model, provides typical configurations for reference when OceanStor Pacific is used as the secondary storage, and provides configuration calculation formulas based on the number of cameras and bit rates. This document also verifies typical service scenario models and outputs configuration and verification reports.

Intended Audience

This document describes a video storage solution based on Milestone XProtect. It provides a reference architecture where Huawei OceanStor Pacific scale-out storage is used as the secondary storage to reduce risks during project expansion and delivery. This shows the values brought about by the Huawei video storage solution for video management platforms, including high reliability, efficiency, flexible expansion, and easy sharing.

Symbol Conventions

The symbols that may be found in this document are defined as follows.

Symbol	Description
	Indicates a hazard with a high level of risk which, if not avoided, will result in death or serious injury.
	Indicates a hazard with a medium level of risk which, if not avoided, could result in death or serious injury.
	Indicates a hazard with a low level of risk which, if not avoided, could result in minor or moderate injury.
	Indicates a potentially hazardous situation which, if not avoided, could result in equipment damage, data loss, performance deterioration, or unanticipated results. NOTICE is used to address practices not related to personal injury.
	Supplements the important information in the main text. NOTE is used to address information not related to personal injury, equipment damage, or environment deterioration.

Change History

Issue	Date	Description
01	2022-12-09	This issue is the first official release.

2/ Introduction to OceanStor Pacific Series Storage Products

Huawei OceanStor Pacific series is a data center-class storage product that provides intelligent and elastic large-scale scale-out capabilities. By using system software, OceanStor Pacific series organizes local storage resources of storage nodes to build scale-out storage pools and provide scale-out file, object, Hadoop Distributed File System (HDFS), and block services for upper-layer applications, as well as a variety of service functions and value-added features. It leverages unique elastic erasure coding (EC) technology to provide more available storage space while ensuring service performance. You can purchase and deploy storage services based on service requirements to help enterprises easily cope with ever-changing services and implement flexible, efficient, and on-demand data access.

3/ Solution Introduction

3.1 Solution Introduction

As a video management center, Milestone XProtect provides users with video management services and basic streaming media services to meet service requirements such as video access, video sharing, and video management.

Milestone XProtect provides tiered storage based on scenario requirements. The primary storage (Live DB) stores short-term video files and provides high read/write performance for flexible invoking, read/write, and modification of video data. After data is retained in the primary storage for a certain period of time, the streaming media service of Milestone XProtect provides video archiving and moves video data from the primary storage to the secondary storage (Archive DB) for long-term storage.

3.2 Solution Architecture

Based on the tiered storage architecture and archiving service requirements of Milestone XProtect, OceanStor Pacific 93 intelligent video and image storage series is recommended as the Milestone XProtect secondary storage. The storage product provides a highly elastic system architecture that allows you to add or remove nodes at any time based on service requirements and linearly adjust the performance and storage space resources for video analysis and storage.

The primary storage can be constructed if required by services. Generally, the iSCSI protocol is used to connect streaming media service nodes to primary storage devices based on the capabilities of Milestone XProtect.

Streaming media service nodes are interconnected with secondary storage devices through the CIFS or NFS protocol. Archiving plans are configured on Milestone XProtect streaming media service nodes based on service requirements. The system migrates video data from the primary storage to secondary storage as planned. Figure 3-2 shows the data flow.

3.3 Solution Components

Milestone XProtect consists of server, client, and storage components. Server components include management servers and streaming media servers (recording servers). Client components include management clients, user Client/Server (CS) clients, and web clients to support service O&M management and customer use. Storage components include primary storage devices and archive storage devices.

Figure 3-3 Solution Components

Server Components

• Management server

A management server is the core component in a Milestone XProtect video management system (VMS). It manages configuration information of all components in the system and user authorization information. These configurations are stored in an SQL database. The database can be installed independently or on the same physical server as the management server.

• Record server

A recording server processes video recording and related events of cameras, encoders, decoders, I/O modules, and metadata. Examples are listed as follows:

1. Recording videos, audios, and metadata
2. Offering live videos or playback
3. Obtaining the running status of connected devices
4. Generating alarms in the case of device faults

In an interconnection architecture, the recording server also interacts with other Milestone XProtect products.

• Log server

A log server stores all logs of the entire system, including system logs, authorization logs, and rule logs.

• Media database

It supports high-performance video and image retrieval and media services such as video and image archiving and video processing.

• SQL server

An SQL database stores configuration information of management servers, events and alarms of event servers, and logs of log servers.

• Mobile server

A mobile server connects mobile clients to web clients, encodes and decodes video streams on recording servers, converts stream formats, and distributes the video streams to mobile clients or web clients. This architecture ensures that recording servers are isolated from the Internet for data security.

Clients Components

• Management client

A management client provides an interface to manage the entire system. On the graphical user interface (GUI) of the management client, all system resources are displayed in a tree structure. After a node is selected, its attribute, which is configurable, is displayed on the right pane. When you expand the root node, the lower-level subnodes are displayed. You can also configure the subnodes.

• Smart client

A smart client allows real-time video browsing, playback, and query on the Milestone XProtect platform. It supports multi-screen playing, as well as window size and image quality adjustment.

Storage Components

As a standard file storage system, OceanStor Pacific is mounted to recording servers through CIFS and functions as the secondary storage of Milestone XProtect for video archiving.

4/ Planning and Configuration (Typical Configuration)

4.1 Network Design Principles

It is recommended that the overall network of the system be fewer than three layers of routes, and the front-end camera network be separated from the client network using NICs. In this way, the high load on the client network does not affect the performance of recording servers.

For details about the storage network configuration, see the configuration principles of OceanStor Pacific. Based on service requirements, the front-end service network of OceanStor Pacific must be 10GE or higher, and the back-end network must be 25GE.

4.2 Capacity Design Principles

Assume that the system contains N streaming media service nodes, each streaming media service node connects to X cameras at 4 Mbit/s, and the total video retention period is M days. The videos need to stay in the primary storage for P days, and the videos are archived every Q days. In this case:

Available capacity of the primary storage is not less than [(Q x 2 + P) x 24 x 3600 x 4 x X/8/1024 x 110% + X] x N/1024 (unit: TB).

If each streaming media server is configured with dedicated primary storage, the value is the total available capacity divided by N.

Available capacity of the secondary storage is not less than 4 x X/8/1024/1024 x 3600 x 24 x (M – P) (unit: TB).

4.3 Bandwidth Design Principles

Assume that there are X cameras streaming at 4 Mbit/s in the system, the bandwidth design of each component in the system must meet the following requirements:

• For details about the bandwidth design of streaming media service nodes, see the Milestone XProtect’s recommended configuration. It is recommended that the video access bandwidth of a single streaming media service node be less than or equal to 1 Gbit/s.
• Primary storage: The write/read bandwidth ratio must be greater than or equal to 1:1.5+n, where n indicates the ratio of viewed videos to recorded videos in the primary storage.
• Secondary storage: Based on basic service habits, if the ratio of video access to video read is 1:0.1, the write bandwidth of the system is not less than 4 x X x 1.5 Mbit/s and the read bandwidth is not less than 4 x X x 0.1 Mbit/s. To ensure the ratio of viewed videos to recorded videos in the secondary storage, the bandwidth pressure of streaming media services on each node in the scale-out storage system must be balanced.

Remarks: The bandwidth capability of the secondary storage (OceanStor Pacific) is recommended to meet the following requirement: 4 x X/Number of nodes/8 ≤ Write bandwidth capability of a single node. For details about the write bandwidth capability of a single node, see the product specification list.

4.4 Recommended Configuration

Hardware Selection of Milestone XProtect

For details about the hardware configuration of Milestone XProtect, see the calculation result of the Milestone official hardware configuration. The actual hardware configuration must be greater than or equal to the officially recommended configuration.

The software version of Milestone XProtect must be 2016R1 or later, and the Windows operating system must be Windows Server 2016 or later. The firewall must be disabled and virus scanning paths must be restricted according to the software operation guide.

To optimize the overall system performance, you are advised to mount the streaming media to different namespaces of the secondary storage to reduce the number of directories in each namespace. You are advised to modify parameters such as the file size (chunk_files), number of system processes (low_priority_archive_thread_pool_size), and media data read/write cache size (media_block_files) in configuration file (RecorderConfig.xml) of Milestone XProtect streaming media server based on service requirements and the performance of Milestone XProtect to improve the archiving or media playback efficiency.

Hardware Selection and Configuration of OceanStor Pacific

OceanStor Pacific 9350 is recommended for video-only storage (or when the image capacity does not exceed 10% of the total capacity). Adopt 10GE networking for front-end services and select the disk size based on specific capacity requirements.

OceanStor Pacific 9340 is recommended for mixed storage of videos and images. Adopt 10GE/25GE networking for front-end services and select the disk size based on specific capacity requirements.

It is recommended that OceanStor Pacific be interconnected with Milestone XProtect through the CIFS protocol.

Service Model Instance

Service model

Service scenario: There are 2448 cameras and the retention period of videos is 15 days. After video data is stored on the primary storage for six hours, the data is archived to the secondary storage every four hours. The data is retained on the secondary storage for 15 days.

For the minimum services in the lab, Milestone StableFPS can be used to simulate camera access at 4 Mbit/s. A single streaming media server (Milestone XProtect’s recording server) connects to 306 cameras. The bandwidth of the streaming media server is 1.2 Gbit/s.

Hardware configuration reference

In minimum networking scenarios, bare servers can be used to set up the environment for Milestone XProtect.

• Streaming media servers: x 8; Intel Xeon Gold 6130T, 16 GB RAM, Gigabit Network Connection, OS: 256 GB, SSD RAID10
• Management servers (including data services and log services): x 2, Intel Xeon Silver 4208, 16 GB RAM, Gigabit Network Connection, Size: 300 GB, HDD RAID1

Primary storage capability construction in minimum networking scenarios:

• In minimum networking scenarios, SSDs can be directly deployed on a streaming media server as the primary storage of a single streaming media node. In the lab, data redundancy of the primary storage is not considered. A single streaming media service node has 5 x 1.92 TB SSDs with RAID 10 to meet the basic test requirements of the primary storage in the lab.

Secondary storage capability construction in minimum networking scenarios:

• In video-only storage scenarios, two OceanStor Pacific 9350 chassis are used with four nodes in total. Each node is fully configured with sixty 10 TB disks to support 15-day video retention in the secondary storage.
• If the retention period needs to be increased, you can adjust the disk size or add nodes to meet the capacity requirement.

With the current number of nodes, the write bandwidth capability is no longer increased.

Performance objective reference

		Bandwidth (MB/s)	Bandwidth (MB/s)
Storage Cluster	Number of Recording Servers Connected to a Single Node	Single Recording Server	Single Storage Node	Total Data Volume on a Single Node
OceanStor Pacific 9350	2	153	306 to 459	378 TB

Remarks: The archiving duration must be shorter than the video write time. For example, the system archives 2-hour historical data within 1.5 hours.

5/ Configuration Example

5.1 Typical Networking

Figure 5-1 shows an example of the networking diagram of this best practice.

The networking is described as follows:

• Milestone XProtect cluster: Eight recording servers, one management server, and one AD domain server are deployed.
• Storage: Two OceanStor Pacific 9350 chassis with four nodes in total are deployed. Eight namespaces are created and shared to the eight recording servers.

5.2 Hardware and Software Configuration

Name	Description	Quantity	Operating System	Function
Storage	OceanStor Pacific 9350 Main storage: 60 x 10 TB NVMe SSD: 1 x 1.6 TB BBU: 16 GB Memory: 5 x 32 GB CPU: HiSilicon 1620 Network: 2 x 10GE optical ports; 2 x 25GE optical ports; 1 x GE electrical port	2 chassis (with 4 nodes)	Eulerosv2r10	Used as secondary storage to store video stream data.
Recording server	x86 server Data disks: 4 x 1.92 TB SSDs, forming RAID 5 System disks: 2 x 1.8 TB HDDs Memory: 8 x 32 GB CPU: 2 x Intel Xeon Gold 6130T Network: 1 x 10GE	8	Windows Server 2016	Used for recording and event processing for cameras, encoders, decoders, I/O modules, and metadata.
Management server	x86 server Data disk: 1 x 16 TB 7.2KRPM SATA System disks: 2 x 1.2 TB HDDs Memory: 6 x 32 GB CPU: 2 x Intel Xeon Gold 5120T Network: 1 x 10GE	1	Windows Server 2016	Core component of the Milestone XProtect VMS platform, responsible of managing configuration information of all components in the system and user authorization information. The SQL database is installed on the management server.
AD domain server	x86 server Data disks: 2 x 300 GB HDDs System disk: 1 x 3.84 TB HDD Memory: 7 x 16 GB CPU: 2 x Intel Xeon CPU E5-2620 v3 Network: 1 x 10GE	1	Windows Server 2016	Used for managing the AD domain service of the Milestone XProtect cluster.
Management switch	Huawei S5735 series	1	N/A	Used for connecting to customers’ management network for system management and maintenance.
Storage switch	Huawei CE6865 series	1	N/A	Used for service data communication between all nodes in a storage system. The RoCE protocol must be supported.
Service switch	Huawei S6720 series	1	N/A	Used for connecting customers’ application system.

Source	Software	Description
Milestone	XProtect Corporate 2022 R1	Milestone software version
Milestone	StableFPS	Simulates cameras to construct video sources.
Huawei	OceanStor Pacific 8.1.3	Advanced version for intelligent video and image

5.3 Interconnection Configuration

Configuration Process

Storage Configuration

According to the solution plan, eight namespaces and CIFS shares are created on OceanStor Pacific 9350. The configuration procedure is as follows: creating a converged storage pool, enabling the file service on nodes, creating namespaces, creating a service network, configuring an external DNS server, configuring a DNS server, and joining the storage service plane into an AD domain.

• Creating a storage pool

On DeviceManager, choose Resources > Resources > Storage Pool and click Create to create a storage pool. Figure 5-3 shows an example of configuring the storage pool. The EC policy is set to +2:1.

• Enabling the file service on nodes

On DeviceManager, choose Resources > Resources > Services > Basic Service and click Enable Service to enable the file service on nodes in the storage pool.

• Creating namespaces

On DeviceManager, choose Resources > Resources > Namespace and create eight namespaces to store video data from eight recording servers. When creating a namespace, set Security Style to Mixed. The hard quota and file quantity quota of each namespace can be configured based on the service volume. For details about how to calculate the service volume, see section 4.2 Capacity Design Principles.

• Creating a service network

On DeviceManager, choose Resources > Access > Service Network and click Create Subnet. After the subnet is created, click Create Access Zone. The following is an example for creating a subnet and an access zone, including enabling the common DNS service for the subnet, enabling IP address failover and the DNS service for the access zone, and setting the DNS policy to round robin. You can select a DNS policy based on the site requirements.

• Configuring an external DNS server

For details about how to configure an external DNS server, see Configuration > Basic Service Configuration Guide for HDFS > Configuring Basic Services > (Optional) Configuring an External DNS Server > Configuring an External DNS Server (Windows Operating System) in the OceanStor Pacific Series 8.1.3 Product Documentation (Huawei Engineer). In this best practice, OceanStor Pacific interconnects with the DNS server of the Milestone XProtect cluster in forwarding mode.

• Joining the storage plane into the AD domain

On DeviceManager, choose Resources > Access > Account and click an account name. The Domain Configuration page is displayed. In the AD area, configure the AD domain.

Milestone XProtect Configuration

For details about how to install and configure Milestone XProtect, contact Milestone technical support engineers.

Recommended Milestone XProtect configuration

To improve the video processing and archiving performance of Milestone XProtect, you can contact Milestone technical support engineers to modify the following configurations based on service requirements.

• Disabling motion detection

By default, the motion detection function is enabled for Milestone XProtect to detect moving objects in videos. Motion detection consumes computing resources of recording servers. If motion detection is not required, you can disable it.

You can log in to the management server and disable motion detection, as shown in Figure 5-9. By default, Milestone XProtect stores only recordings of moving clips. If motion detection is disabled, Milestone XProtect stops storing recordings. Therefore, you need to add rules to configure a recording plan, as shown in Figure 5-10.

• Modifying the recording file size

You are advised to modify the value of max_bytes_in_block_files in the configuration file of recording servers (the path is C:\ProgramData\Milestone\XProtect Recording Server\RecorderConfig, as shown in Figure 5-11). The value indicates the size of a single recording file. The default value is 16MB. You are advised to modify the value to 512MB to optimize the archiving efficiency, as shown in Figure 5-12. Recording servers need to be restarted after the modification so that the settings take effect.

• Modifying the value of maxframesinqueue

You are advised to modify the value of maxframesinqueue in the configuration file of recording servers in the path C:\ProgramData\Milestone\XProtect Recording Server\RecorderConfig. The default value is 50. You are advised to modify the value to 400. In case of busy storage, servers can buffer more frames to avoid camera overflow, as shown in Figure 5-13. Recording servers need to be restarted after the modification so that the settings take effect.

• Modifying the archive thread pool size

In the configuration file of recording servers, the default value of low_priority_archive_thread_pool_size is 1, and the default value of high_priority_archive_thread_pool_size is 4. The two parameters affect the maximum bandwidth that can be reached during Milestone XProtect archiving. When OceanStor Pacific is used as the secondary storage, the default values can meet the archive bandwidth requirements. You can check the archive bandwidth of Milestone XProtect. If the archive bandwidth is too low or too high, you can adjust the two parameters. The configuration file path is C:\ProgramData\Milestone\XProtect Recording Server\RecorderConfig. Recording servers need to be restarted after the modification so that the settings take effect.

• Modifying the value of read_buffer_size in media_block_files

In the configuration files of recording servers, the read_buffer_size in media_block_files is 4096. To optimize the performance of Milestone XProtect reading recording files from OceanStor Pacific secondary storage for playback, you are advised to modify the buffer size to 65536, as shown in Figure 5-15. The configuration file path is C:\ProgramData\Milestone\XProtect Recording Server\RecorderConfig. Recording servers need to be restarted after the modification so that the settings take effect.

• Checking the values in chunk_files

You are advised to check whether the values of read_buffer_size and write_buffer_size in chunk_files in the configuration file of recording servers are 65536. If not, modify the values to 65536 to increase the cache capability to ensure the recording and archiving performance. The configuration file path is C:\ProgramData\Milestone\XProtect Recording Server\RecorderConfig. Recording servers need to be restarted after the modification so that the settings take effect.

• Configuring disks

Log in to each recording server and modify the configuration of the disk for storing media data on the recording server. For example, if disk D is the media data disk, set the disk D to NTFS and 64 kilobytes, as shown in Figure 5-17.

• Configuring the firewall and Defender

By default, the firewall and Defender functions are enabled in the Windows operating system. In this case, the archiving performance of Milestone XProtect will be affected. If the archiving performance of Milestone XProtect is unstable, log in to each recording server and modify the firewall and Defender settings. For details about how to modify the settings, see Milestone official document Milestone XProtect Storage Architecture and Recommendations.

Interconnections Configuration

• CIFS share

On DeviceManager, choose Resources > Resources > Share > CIFS Share and click Create to create a CIFS share. When creating a CIFS share, you need to add the CIFS share access permission and set Type to AD domain user.

• Archive configuration

Log in to the Milestone XProtect management server, choose Management Client, click Servers > Recording Servers, and add secondary storage to each recording server, as shown in Figure 5-19. The secondary storage configuration is displayed, as shown in Figure 5-20. The following table describes the configuration items.

Item	Description	Test Configuration
Name	Secondary storage name	Archive 1
Path	Secondary storage path. The path format is \\domain name\sharename or \\storage service IP address\sharename.	The path is mounted using a domain name to implement load balancing.
Retention time	Video data retention period	15 days
Maximum size	Storage space of a recording server on the secondary storage. When the storage space is used up, the earliest historical data is deleted.	200 TB
Schedule	Archiving cycle. The archive interval must be shorter than the Live DB data retention period.	2 hours
Reduce frame rate	After this function is enabled, the recording frame rate decreases, saving storage space.	Disabled

In this practice, there are eight recording servers and four storage nodes. Therefore, two recording servers are mounted to each storage node based on the load balancing policy. If the load balancing feature is not enabled and IP addresses are used for mounting, you need to pay attention to the amount of data mounted to each storage node to avoid fat nodes.

5.4 Test and Verification

In this best practice, the performance test and reliability test are completed based on the following service model, and constraints for this scenario are provided.

Scenario configuration: 4-node OceanStor Pacific 9350, +2:1 EC
Video Code Stream	4 Mbit/s
Data Volume Written to a Single Recording Server	306 x 4 Mbit/s
Data Volume Written to a Single Storage Node	612 x 4 Mbit/s
Primary Storage Data Retention Period of Recording Servers	3 hours
Primary Storage Capacity of Recording Servers	5.76 TB
Archiving Period from Primary Storage to Secondary Storage	Every two hours
Total Video Data Retention Period (Total of Primary Storage Retention Period and Secondary Storage Retention Period)	15 days

Performance Test

The following table lists the performance test data when 612 4-Mbit/s cameras are connected to a single storage node and 62 4-Mbit/s cameras are played back.

	Bandwidth (MB/s)	Bandwidth (MB/s)	Bandwidth (MB/s)
Number of Recording Servers Corresponding to a Single Storage Node	Access Bandwidth of a Single Recording Server	Average Archive Bandwidth of a Single Storage Node	Average Read Bandwidth of a Single Storage Node
2	153	400	31

Reliability Test

An OceanStor Pacific fault will cause I/O suspension. After I/O suspension is rectified, Milestone XProtect continues the archiving task of the current period.

Module	Fault Mode	Duration of I/O Suspension
Node	Single-node fault	< 13s
Disk	Single-disk fault	< 7s
Network	NIC fault	< 10s

Components Capability Constraints

1. If the mounting IP address of the Milestone XProtect secondary storage is changed, Milestone XProtect will migrate data from the old secondary storage to the new secondary storage. The migration duration depends on the data volume.
2. Milestone XProtect does not detect storage capacity changes and only stores data based on the allocated capacity. Data will fail to be written to the primary storage if the actual primary storage capacity is less than the allocated capacity and is used up, and data has not been archived yet. Data will be overwritten and archived if the actual secondary storage capacity is less than the allocated capacity and is used up, and the retention period has not expired yet.
3. It is recommended that the primary storage space usage be less than or equal to 90%. Otherwise, the recording servers of Milestone XProtect will stop providing services.
4. Milestone XProtect configuration constraint 1: The value of low_priority_archive_thread_pool_size must be 1 and cannot be increased.

Milestone XProtect configuration constraint 2: The value of read_buffer_size in media_block_files must be increased (greater than 4096). The recommended value in the best practice is 65536.

6/ Scenario Constraints

Based on the best practice test results of interconnection between Milestone XProtect and OceanStor Pacific 9350, the pre-sales expansion solution review has the following constraints:

1. The playback ratio does not exceed 7%. It is recommended that the playback bandwidth of a single OceanStor Pacific 9350 node be less than or equal to 30 MB/s.
2. Milestone XProtect can cooperate to modify configuration parameters.

Milestone XProtect interconnects with OceanStor Pacific. The configuration constraints during delivery are as follows:

1. During the delivery, the number of connected Milestone XProtect recording servers must be properly allocated to ensure a balanced bandwidth load on each OceanStor Pacific node.
2. Milestone XProtect must be configured according to recommendations in this best practice.

7/ Reference

• XProtect Storage Architecture and Recommendations: https://content.milestonesys.com/media/?mediaId=E3DC3554-0E90-47B9-85C39AB5D670AA43
• OceanStor Pacific series product documentation