No matter how well an artificial intelligence model is designed, it is only as good as the data it is trained on. In medical AI studies, the vast majority of failures lie not in the model architecture but in data quality.
In this article, we cover the fundamental steps of turning raw hospital data into something suitable for a machine learning model.
Why Can't Raw Clinical Data Be Used Directly?
Data pulled from hospital information systems generally contains the following problems:
- Missing values: Certain measurements may not have been recorded for some patients
- Inconsistent coding: The same diagnosis may have been coded differently by different residents (ICD-10 mismatches)
- Outliers: Data entry errors or genuine clinical anomalies
- Categorical variable variety: The same information entered in different formats, such as "Yes/No", "Y/N", "1/0"
- Identity information: Personal data that cannot enter the model without being processed under KVKK
Step 1: Anonymization and KVKK Compliance
Every piece of data that enters the artificial intelligence model must be stripped of personal identifiers. This is not only an ethical obligation but a legal requirement.
The anonymization process covers the following:
- Removal of direct identifiers such as national ID number, full name, and date of birth
- Replacing hospital protocol numbers with random IDs
- Evaluating rare diagnoses or unique combinations in small cohorts in terms of re-identification risk
In the ethics committee application, how the data will be anonymized must be documented technically.
Step 2: Data Cleaning
The cleaning process varies according to the structure of the data, but the general steps are as follows:
Missing data analysis: In which variables, and at what rate, is data missing? Is the missingness random or systematic? The answer to these questions determines how missing data will be handled, whether by deletion or imputation.
Outlier detection: Outlying observations are detected using box-plot and Z-score methods. Not every outlier is an error; clinical expert opinion may be required.
Consistency check: A patient aged 200, a record with a negative hemoglobin value, these kinds of logical inconsistencies are scanned for systematically.
Step 3: Feature Engineering
Raw variables do not always enter the model directly. Transformations that require clinical knowledge are done at this stage:
- Splitting continuous variables into categories (e.g., BMI to normal / overweight / obese)
- Calculating clinically meaningful durations from date variables
- Composite scores derived from multiple variables (such as APACHE, SOFA)
Step 4: Train, Validation, Test Split
In medical AI studies, data splitting is especially critical. While k-fold cross-validation is preferred for small datasets, a standard 70-15-15 split can be applied for large datasets.
An important point: if your data comes from different hospitals, a separate cohort should be planned for external validation. This shows the model's real-world performance.
Step 5: Documentation
All data preparation steps must be documented for reproducibility. During the manuscript writing stage, these documents are used directly for the Methods section and protect you against reviewer questions.
The clinical data preparation process sits at the intersection of technical knowledge and clinical experience. A biomedical engineer, by uniting these two worlds, can ensure that the process is both technically sound and clinically meaningful.
To make your clinical data analyzable, request a free 30-minute consultation.
Where Do People Get Stuck Most in This Analysis?
- Your data in Excel is inconsistently coded, and you lose half of it during the cleaning process.
- You have done anonymization, but you are not sure whether KVKK compliance is technically documented.
- At the feature engineering stage, you cannot decide with clinical justification which variables should enter the model.