Introduction

My name is Minoru Shimodera, and I have spent many years in a global pharmaceutical company leading post-marketing pharmacovigilance (PV) activities, post-marketing database study, and pharmacoepidemiology initiatives. As for my roles in supporting the industry’s effort, I have also been a member of working groups discussing MHLW/PMDA notifications and guidance regarding post-marketing database studies. In this series of column articles, I would like to share practical insights based on these experiences, particularly how to correctly interpret the notifications and administrative letters issued by the Japanese Ministry of Health, Labour and Welfare (MHLW). Simply put, my goal is to clarify the interpretation and practical implementation of PMDA-aligned post-marketing database study guidance.

We are now entering the eighth year following the introduction of post-marketing database studies. While many notifications and administrative communications have since been issued, we often find it difficult to know how to interpret these documents. In these columns, I explain notifications and administrative communications regarding Database Studies most frequently mentioned in inquiries from pharmaceutical companies.

Please note that these articles are based on the cases experienced by the author, and do not represent the consensus of the industry or experts. In this article, I focus on the MHLW notification regarding outcome validation and explain how companies can practically approach it.

Notification regarding outcome validation

Outline of the Notification

In July 2020, the Pharmaceuticals Evaluation Division and the Safety Division of the Pharmaceutical Safety and Environmental Health Bureau of the Ministry of Health, Labour and Welfare (MHLW) issued an administrative notification entitled Development of Basic Principles in Conducting a Validation of Outcome Definitions used in Post-Marketing Database Study. Outcome definitions are critical not only in post-marketing database studies but also in real-world data (RWD) research more broadly. This notification provides guidance on when validation of outcome definitions should be conducted, emphasizes the importance of scientifically justified validation, describes general methodologies for conducting outcome validation, and presents practical considerations for developing a validation protocol.

This article explains the notification in the following manner.

• Which post-marketing database studies require validation of outcome definitions, and when such validation may not be necessary
• Summary of outcome validation studies conducted at MDV
• Practical considerations for applying validation study results in post-marketing database studies
• Considerations for using validated outcome definitions in independent peer-reviewed publications

* Development of Basic Principles in Conducting a Validation of Outcome Definitions used in Post-marketing Database Surveys” (Administrative Communication, July 31, 2020)

Which post-marketing database studies require validation of outcome definitions, and when such validation may not be necessary

Do all post-marketing database studies require validated outcome definitions? The MHLW notification on validation of outcome definitions clarifies when such validation is scientifically required.

According to Section 2 of the notification, it applies to post-marketing database studies conducted for re-examination or re-evaluation applications, where validation of outcome definitions is performed for studies that serve as the primary evidence to inform concrete post-marketing safety measures. In other words, this applies to drug risk assessments for adverse events that are not yet described in package inserts or patient-oriented drug information and for which safety measures have not been implemented. If such studies identify a risk, additional safety measures would be implemented, and evaluation based on high-quality evidence would be required. This is particularly relevant for important potential risks in the risk management plan (RMP), where the use of validated outcome definitions is considered necessary.

Conversely, validation of outcome definitions is generally not required for important identified risks in the RMP for which adequate safety measures are already in place.
Similarly, validation may not be necessary for situations involving missing information, as these are typically addressed prior to risk evaluation, for example, when assessing whether relevant patient populations exist.

Through its experience conducting outcome validation studies for post-marketing database studies, MDV has encountered practical challenges. For instance, medical institutions commissioned to conduct such studies may lack prior experience in outcome validation, and once a study begins, recruitment of a sufficient number of cases can be difficult. Therefore, planning must consider both the scientific design and the operational feasibility of the validation study.

Summary of outcome validation studies conducted at MDV

At MDV, numerous post-marketing database studies are currently conducted or planned.
Among these, studies for which validated outcome definitions were required have been subjected to outcome validation studies in accordance with the MHLW notification. To date, three validated outcome definitions have been published in peer-reviewed papers, as summarized below.

Validation Study of Algorithms to Identify Malignant Tumors and Serious Infections in a Japanese Administrative Healthcare Database

The first study focused on outcome validation of case-identifying algorithms for malignant tumors and serious infections. Eli Lilly Japan K.K. conducted post-marketing database studies for Olumiant to examine the definitions of these safety outcomes. Broadly speaking, the results showed that combinations of diagnostic and therapeutic parameters achieved a certain level of positive predictive value while maintaining high sensitivity.

Validation Survey of Algorithms to Identify Malignant Tumors and Serious Infections in a Japanese Administrative Healthcare Database
Atsushi Nishikawa1, et al. Annals of Clinical Epidemiology 2022;4(1):20-21

Research paper summary and explanation here

Validation study of case-identifying algorithms for severe hypoglycemia using hospital administrative data in Japan

The second study was the validation study for severe hypoglycemia also conducted by Eli Lilly. Likewise, this was a validation study conducted to evaluate severe hypoglycemia in Database Studies for diabetes drugs. This study showed that, while placing more weight on sensitivity resulted in lower positive predictive value, modification of the outcome algorithm improved positive predictive value at the expense of sensitivity.

Validation survey of case-identifying algorithms for severe hypoglycemia using hospital administrative data in Japan
Satoshi Osaga, et al. PLoS ONE 18(8): e0289840

Research paper summary and explanation here

Malignant rumors and serious infections are listed as safety specifications in drug risk management plans for many antibody drugs, and hypoglycemia is almost always listed for diabetes drugs. By having validated outcome definitions regarding safety specifications for many drugs, we can significantly save time and money while ensuring evidence for studies.

What is noteworthy about these two studies is...