Progress in quality evaluation and standardization of artificial intelligence medical devices
Post date: 2020-03-07
2019 is a year of steady development of the artificial intelligence medical device industry, and the pace of technological innovation and transformation continues to accelerate.
Industry to accelerate the artificial intelligence product updates, the strong demand to shorten the period of examination and approval, and the regulatory authorities and the public to ensure product is safe and effective, reliable requirements also more and more high, so countries all over the world are strengthening ethics, technical standards, and regulatory science of artificial intelligence research, in order to enhance the public service and regulatory decision-making ability.
Under the leadership of China's drug regulatory authorities and according to the domestic and international situation, China food and drug testing and research institute (hereinafter referred to as cas) has also carried out a lot of related work in the quality research and standardization of artificial intelligence medical devices.
Domestic and foreign industry development status
The types and USES of artificial intelligence medical devices in foreign countries are increasing, and their roles in clinical diagnosis and treatment paths, health interventions and services are becoming diversified.
According to public information, the us FDA has approved dozens of ai medical device products, including computer-aided diagnosis, aided detection, aided triage and other independent software.
On the other hand, new products represented by intelligent hardware and intelligent rapid reconstruction have emerged suddenly, and some products have been approved to market.
The clinical application of artificial intelligence develops to the process orientation, and the clinical work efficiency and service quality are optimized, such as CT dose optimization, X-ray intelligent positioning, image quality control and other USES.
In addition, artificial intelligence itself, as a technical means, is being integrated into the r&d, production, quality management and doctor education and training of medical devices, promoting the "AI+" upgrade in the fields of traditional active medical devices, medical robots, wearable devices, mobile health products, 3D printing and medical device combination products.
In order to meet the development demand of the industry, the trend of quality evaluation of ai medical devices abroad is flexible, economical and rapid.
Take the United States as an example. In recent years, the application of ai medical device products in the United States often adopts such means as algorithm verification based on third-party test sets, algorithm verification based on simulation data and body model, etc. to reduce the dependence on clinical trials, and the approval cycle of some products is shortened to a few months.
In 2019, FDA published a discussion paper on the product change regulatory framework for machine learning/artificial intelligence medical device software, focusing on the rapid evaluation path after product iteration, which provided useful enlightenment for the industry.
China's artificial intelligence medical device industry is also continuing the strong momentum of development, technological breakthroughs and products to accelerate the pace of landing.
Under the support of various favorable policies of the state, the production, the research and the application of a hundred flowers are in full bloom.
The research results of algorithm and clinical application in China have been listed on top journals such as NatureMedicine and Cell.
The analysis objects of domestic AI products have expanded from fundus photos of diabetic retinopathy (hereinafter referred to as sugar network) and CT images of pulmonary nodules to all aspects of medical images and medical signals.
Product form from a single software development into a variety of general, dedicated intelligent platform, and integration with hardware, knowledge map development.
While actively preparing for the registration and listing of their products, domestic enterprises are also accelerating the deployment of overseas markets and demonstrating China's strength in various important conferences and exhibitions at home and abroad.
Progress of standardization work
The establishment of the standardization technology focal point of ai medical devices is a milestone in the standardization of ai medical devices in China.
As a strategic emerging field, the national medical products administration attaches great importance to and supports the standardization of ai medical devices. In March 2019, the national medical products administration announced to the public the plan for the establishment of a centralized unit, solicited opinions from various parties, and formally approved the establishment in October 2019.
The expert group is composed of outstanding representatives from universities, scientific research institutions, production and management enterprises, clinical institutions, inspection, evaluation and supervision and other fields, including academicians of Chinese academy of sciences, Chinese academy of engineering and a large number of well-known experts and scholars at home and abroad.
The centralized unit for the standardization of ai medical devices is under the administration of the national medical products administration. It is engaged in the national standardization work in the field of ai medical devices, and is responsible for the preparation and revision of the general standards, special standards and other standards of ai medical devices.
As the secretariat of standardization centralization, China food and drug testing institute (cfda) provides the industry with comprehensive services and technical support, such as artificial intelligence product inspection and database quality evaluation.
, medicine, medical apparatus and instruments according to the national standardization reform review the macro situation of examination and approval system reform and the status quo of the domestic medical equipment industry artificial intelligence, expert group to follow "the four most strict" under centralized unit requirements, innovation needs to give attention to two or morethings, starting from the basic common problems is put forward, to develop the industry in urgent need of the standard specification, gradually establish and perfect the standard system of artificial intelligence medical apparatus and instruments.
Under the guidance of the state food and drug administration and the efforts of the expert group, China's first batch of artificial intelligence medical device industry standards are in the intense research and formulation stage, represented by the standard sequence of "quality requirements and evaluation of artificial intelligence medical device".
Part 1: general requirements for data sets (hereinafter referred to as dataset standards) and part 2: terms (hereinafter referred to as evaluation terminology standards) of this sequence have been approved in 2019, and it is expected that public opinions will be solicited in 2020.
The purpose of establishing the data set standard is to ensure the quality of the data set needed for artificial intelligence medical devices, promote the scientific and orderly development of data resources, and promote the healthy development of the artificial intelligence medical device industry.
At present, China is vigorously promoting the construction of data sets and supporting the research and development, supervision and other activities of artificial intelligence medical devices.
Since the quality of data sets has an important impact on the performance of artificial intelligence algorithms and the safety and effectiveness of products, it is necessary to standardize the development and quality management process of data sets in the initial stage, and carry out evaluation on the existing data sets to prevent quality risks.
Data set standards mainly emphasize quality control factors and define data set evaluation methods.
The application scope of this standard includes all kinds of data sets used in the whole life cycle of artificial intelligence medical devices, which plays an important role in supporting the production quality management, supplier audit and other links of artificial intelligence medical device enterprises as well as the subsequent formulation of quality system related standards.
The purpose of formulating the evaluation term standard is to provide a unified term and definition for the quality evaluation of artificial intelligence medical devices, clarify product classification, standardize the methods, concepts and indicators related to the safety and effective evaluation of products, and provide a basis for the quality evaluation.
At present, the general terminology standards at home and abroad are in the stage of revision, terminology is relatively scarce.
In addition to ISO/IECJTC1 / SC42, the ISO/IECCD22989 standard for artificialintelligence-conceptsandterminology of the international sub-technical committee on Artificialintelligence (ISO/IECJTC1 / SC42) and China's national standard, terms for Artificialintelligence in information technology, are being formulated.
The evaluation terms refer to the methods and definitions of safety, effectiveness, risk and quality management of artificial intelligence medical devices, and are intended for use by technicians engaged in scientific research, production, supervision, inspection and clinical application of artificial intelligence medical devices in the professional field.
In terms of international standardization, the IEEE working group on artificial intelligence medical device standards, led by the China institute of electrical and electronics engineers (IEEE), was officially launched in March 2019. Its members are from domestic and foreign industries, scientific research and regulatory fields, including representatives from the us FDA, Siemens, philips, tencent, yitu, baidu and others.
This standard was proposed by Chinese experts, and it is the international standard for artificial intelligence medical devices. Now the draft standard has been preliminarily completed.
On the basis of refining domestic and international consensus and strengthening international cooperation, we will work together with domestic industry standards to benefit global industries with Chinese wisdom.
Progress of product inspection and quality research
In 2019, the testing of domestic ai medical device products continued to advance steadily and faced more challenges.
In 2018 to issue the first domestic sugar after net product performance test report, issued by the court in this year in the check the first batch of domestic lung nodules product performance test report, implements the lesion detection, classification, region segmentation algorithm functions such as, size measurement and the quantitative evaluation of relevant indicators, declared by CT 3 d medical image AI phased achievements product quality evaluation, for quality evaluation of other AI radiation imaging products and related standard formulation provides the beneficial reference.
Up to now, dozens of enterprises have carried out algorithm performance verification by relying on the standard data set of ccie sugar network and pulmonary nodules, and some products have entered the clinical trial stage, laying a good foundation for the landing of products.
Compared with the batch concentrated products represented by sugar network and pulmonary nodules in 2018, new products in 2019 show a trend of diversification and differentiation, with different features in expected use, function, disease species and data modes. However, the standard data set for testing has not yet been established, which restricts the performance evaluation of the algorithm.
In order to solve such common bottleneck problems, the cas has cooperated with well-known domestic hospitals, clinical institutions and research teams to integrate the concept of quality control into the construction of data sets and jointly strengthen quality management.
At present, the team of cas is deeply engaged in ultrasound, ecg, pet-ct and other aspects, helping to build and guarantee the relevant data sets in China and accumulating practical experience for the preparation and revision of relevant standards.
In order to explore and enrich the means of product quality evaluation, the team of cas, based on the rich experience and historical accumulation of medical device testing, has carried out the following work from multiple perspectives.
First, in terms of test methodology, comprehensive use of various technical means, school team in the check based on the existing clinical data, carried out the fundus camera and fight modal tests such as CT and ecg, prospectively the challenge for the simulation of AI products in the real world, assess the robustness and generalization ability of products, increase the quality evaluation dimensions.
At present, some of the research results of adversarial testing have been published in domestic academic journals and made public to the public.
Some methods have been used in daily inspection, as a supplement to the existing inspection methods, which can help to improve the enterprises' understanding of product quality.
In the check in quality evaluation strategy, second, the school team work closely with clinical experts, from artificial intelligence, the concept of whole life cycle of product quality control, combined with the practice of product technical validation experience, put forward the clinical medical image AI products with quality control framework, uphold the personnel, equipment, data collection, evaluation method and experimental environment Angle to study the clinical deployment of artificial intelligence products how to carry out quality management and daily quality control, after parsing the human-machine comparison, more than expected use, adverse events recognition and response to the hot topic in the industry, such as clinical perspective to expand the connotation of product quality evaluation.
Third, in terms of new product evaluation, school team in the check for AI + wearable devices, clinical auxiliary decision system (CDSS), image process optimization and other hot direction further carry out the standard assessment and literature analysis, the special risk, evaluation of product strategy, taking on new product testing tasks at the same time, also to the industry to share the research results and quality awareness, have played an important role in the answer.
At present, the cas team has formed an artificial intelligence medical device testing system, including software and hardware testing, algorithm performance verification, data set quality control and evaluation, and countermeasure testing. The service objects are from enterprises, hospitals, scientific research institutions and other fields.
School team in the check on the basis of existing achievements, the subject of the successful declare multiple ministry of science and technology key research and development plan in 2019 and the province departmental level topic, is building a national medical artificial intelligence product whole life cycle testing platform, supporting the construction of a series of open medical standard data sets, committed to the overall increase in regulation, industry, use the link of technical support and service ability.
We see that more and more products have completed algorithm verification and are closer to landing.
The standardization technology focus unit for artificial intelligence medical devices was formally established. It will fully perform its duties as a "national team", give play to the advantages of interdisciplinary talents, mobilize social forces, and strongly promote the standardization process of artificial intelligence medical devices in China.
The first batch of artificial intelligence medical device industry standards has been set up, which will be jointly promoted with IEEE artificial intelligence medical device standards led by China and contribute to the healthy development of domestic and foreign industries.
With the arrival of 2020, the quality evaluation and standardization research of ai medical devices will face more opportunities and challenges.
It is believed that on the basis of collective efforts and sincere cooperation between enterprises, universities, research institutes and various sectors, China will continue to make breakthroughs, provide more guarantees for the high-quality development of the artificial intelligence medical device industry, and scientifically and steadily harness artificial intelligence to serve the people's health.
(author's institution: China food and drug testing institute)
Source: China food and drug network