Open source to the left, self research to the right

category:Internet
 Open source to the left, self research to the right


What are the new trends of enterprise storage? What are the technological innovations of the new generation of qingstor products? When many manufacturers choose to do secondary development based on the open source storage platform CEPH, why does Qingyun qingcloud take the difficult road of self-study from scratch? Recently, intelligent evolution interviewed the research and development team of qingstor.

Wang Yu, head of research and development of qingstor storage product line, believes that there are two major trends in the development of enterprise storage market

First, distributed storage is still in a period of explosive growth. IDC report shows that, whether it is object storage, file storage or block storage, it will maintain a double-digit high-speed growth in the next few years;

Second, with the rapid decline of SSD flash media cost, full flash storage has gradually become the mainstream. IDC report shows that full flash storage has accounted for 80% of all enterprises using storage. And it will maintain an annual growth rate of 50% in the next few years.

Wang Yu, R & D director of qingstor storage product line

When the enterprises storage scenarios become more and more diverse and complex, no one storage solution can fully meet all storage scenarios. Some scenarios require extreme performance, some require large capacity, some require high IOPs, some require high throughput, and some require common interfaces.

Qingyun qingcloud divides storage scenarios into four categories: hot data, cold data, warm data and mixed scenarios. The four products of qingstor are developed for four scenarios: block storage for enterprise core business data; object storage for massive static data and unstructured data; file storage for general scenarios; and integrated storage for multiple storage scenarios.

Moreover, in terms of compatibility, qingstorneonsan can dock with mainstream virtualization or cloud computing platforms such as VMware, openstack, Hyper-V, etc. At the same time, in the face of the increasingly popular cloud native environment, qingstorneonsan can provide high-performance, reliability and enterprise level distributed storage capabilities for k8s and kubesphere container platforms.

According to the latest research report of IDC, the total amount of global data has reached 50zb (about 1 trillion GB), of which the unstructured data accounts for 80%. Object storage is the best choice for managing rapidly growing unstructured data. Qingstor object storage 3.0 is a storage service with both fish and bears paw. It has the ultra-high performance of massive data storage and data storage. Version 3.0 added more scenarios such as data migration, data protection, data backup, AI video monitoring, etc. In terms of performance, when the number of single bucket objects exceeds 10 billion, the storage performance of qingstor objects can remain stable, leading the industry.

File storage is the oldest and most widely used data storage method. The newly upgraded qingstor file storage has built a unified namespace through self-developed entire file system stack, which makes file storage free from the limitation of local file system. Each directory supports multiple access nodes to access at the same time. There are no restrictions on the capacity and the number of files, so that the processing capacity has horizontal expansion.

The integrated storage of qingstor meets the requirement that enterprises can obtain multiple storage types and storage interfaces in one purchase. It integrates the above three types of professional storage at the deployment level, but at the architecture level, it still maintains the architectural characteristics of each professional storage, as well as its complete functions and performance.

At present, many domestic storage manufacturers will choose CEPH based open source distributed storage platform as the bottom layer for secondary packaging and secondary development. Why does Qingyun qingcloud choose the very difficult road of 100% self research? According to Wang Yu, there are two main reasons:

First, the cloud has high requirements for security, isolation, and large-scale horizontal expansion capability in the core business of enterprises. It is difficult for open-source distributed storage products to fully meet these requirements. Through the strict test of CEPH performance and the investigation of a large number of enterprise customers, the research and development team of qingstor believes that the most fundamental logic of CEPH is to expect to meet all requirements through a set of solutions, which is not suitable for highly professional storage business.

CEPH provides a good platform for enterprises to develop distributed storage enterprise products quickly. However, if there is no accumulation in the underlying technology research and development, the later development will be more dangerous. This is why it is difficult for us to see CEPH in the core business scenarios of enterprises, such as high-performance databases. It is more used in emerging scenes, object storage interface, etc.

Second, self research can provide Chinese customers with high-quality products that can be controlled independently in the whole stack, thus eliminating the concerns of enterprises on data security. At present, from the software architecture, the underlying chip to the operating system, qingstor has fully realized the compatibility and mutual certification with domestic manufacturers, and built the localization ecology. At present, qingstor has completed the compatibility and adaptation test with Kunpeng, Feiteng, Zhongke haiguang, Inspur openpower and other domestic chips, realizing the compatible docking of main domestic operating systems such as unison UOS, Kirin and Euler.

It is self-study that enables Qingyun qingcloud to obtain some industry-leading new technologies in the storage field. For example, the technology of nvme over roce, referred to as nvmeof, was introduced in qingstorneonsan 3.0 to implement a new protocol for client access to storage. In the file storage, qingstor has used the most advanced rust language in the industry, which is very suitable for the underlying storage coding. It is becoming an industry trend to develop storage system with rust language. AWS server free service, core storage system of Dropbox and Facebook blockchain project Libra have supported rust language.

Strengthen the ability of scenario, do storage that can solve business problems

In addition to the overall improvement of technology, performance and function, the ability of scenario is also the heavyweight content of the annual upgrade of qingstor.

Feng Xiangdong, product manager of qingstor storage product line

On the other hand, qingstor makes storage products more landing scenarios, aiming to help enterprises directly solve business problems, not just enhance it capabilities. For example, in the AI video monitoring scenario, the qingstor object storage thoroughly breaks through the life cycle of video image data, and makes it easy for enterprises to manage the process from massive data storage to data analysis and processing to data archiving. Through SDK, intelligent terminal writes data directly to object storage, simplifies the end-to-end storage process, and smoothly connects with external data AI analysis engine.

Autopilot is a typical object storage business scenario, and the raw data collected by enterprises every day reaches hundreds of TB. In the case of an autopilot customer, because of its ultra-high performance, standard and open application interface, flexible deployment, simple operation and maintenance, qingstor object storage meets the needs of customers. At present, the qingstor object store has stored more than 10PB of data for this customer. In the early stage of business, it also solved the problem of massive data migration across clouds and regions. The choice of technical route is closely related to the enterprises values. Qingyun qingcloud chose the latter between developing products with low cost and providing the highest quality with self-study. After eight years of forging, 100% independent design and research and development, qingstor distributed storage for the cloud is not only the underlying support of Qingyuns own cloud products, but also has been verified by tens of thousands of enterprise customers. Source: editor in charge of mass news: Chen Tiqiang_ NB6485

Autopilot is a typical object storage business scenario, and the raw data collected by enterprises every day reaches hundreds of TB. In the case of an autopilot customer, because of its ultra-high performance, standard and open application interface, flexible deployment, simple operation and maintenance, qingstor object storage meets the needs of customers. At present, the qingstor object store has stored more than 10PB of data for this customer. In the early stage of business, it also solved the problem of massive data migration across clouds and regions.

After eight years of forging, 100% independent design and research and development, qingstor distributed storage for the cloud is not only the underlying support of Qingyuns own cloud products, but also has been verified by tens of thousands of enterprise customers.