Textbook of Digital Health

Introduction

Healthcare generates large quantities of data: from clinical encounters, diagnostic tests, and administrative transactions; from research studies and disease registries; and, increasingly, from patient-generated sources such as wearable sensors and home monitoring devices. The analytical challenge lies in turning this raw data into information that is useful - for improving individual patient care, for managing populations, for research, and for policy. This is not straightforward. Health data is messy, incomplete, and often collected for purposes quite different from those to which analysts wish to put it.

This chapter examines the principal sources of health data, the analytical methods used to extract meaning from them, and the applications of these methods in clinical care and population health. It also considers the infrastructure required to support health analytics and the ethical questions that arise when algorithms begin to influence clinical decisions.

Health Data Sources and Types

A century ago, a patient's medical history existed primarily in the physician's memory, supplemented perhaps by handwritten notes in a ledger. Today, that same patient generates data continuously, from the clinical observations recorded during consultations to the heart rhythm captured by their smartwatch while they sleep. This explosion of health data represents both an opportunity and a challenge: we have more information than ever before, yet making sense of it requires understanding where different data types excel and where they fall short.

The most familiar source remains the electronic health record, which captures what happens when patients interact with the healthcare system. Diagnoses, medications, laboratory results, imaging reports, and clinical notes accumulate over years of care, creating longitudinal portraits of individual health (MIT Critical Data, 2016). Yet EHR data reflects clinical encounters, not health itself. A patient who avoids healthcare may appear healthy in their record while harbouring undiagnosed disease, while another with excellent health may generate extensive documentation through anxious over-utilisation. Documentation practices shaped by billing requirements, time pressures, and individual clinician habits introduce further distortions.

Claims data offers a complementary perspective by tracking healthcare utilisation across all providers and settings. When a patient sees a specialist their GP knows nothing about, or fills a prescription at a pharmacy in another city, claims data captures these encounters. This breadth comes at the cost of depth: claims record that services occurred without capturing the clinical reasoning or findings. Combining EHR and claims data begins to address limitations of each, though reconciling different coding systems and patient identifiers presents technical challenges.

Some questions demand more focused data collection than routine clinical care provides. Disease and procedure registries address this need by systematically capturing detailed information about specific conditions or interventions. Cancer registries track tumour characteristics, treatments, and outcomes with standardised protocols that enable meaningful comparisons across institutions and time periods. Joint replacement registries monitor implant performance over years, detecting problems that would be invisible in fragmented clinical records. This depth requires deliberate investment in data collection infrastructure, limiting registries to conditions and procedures where such investment is justified.

A significant shift has been the recognition that clinician-documented data captures only part of the picture. Patients experience their health continuously, not just during clinical encounters. Patient-reported outcome measures capture symptoms, functional status, and quality of life that clinicians cannot directly observe. A patient's laboratory values may be stable while their fatigue prevents them from working; their surgical wound may be healing while chronic pain diminishes their quality of life. Patient-reported data surfaces these realities, though collecting it systematically requires engaging patients as active participants in their own data generation.

Technology has extended data collection far beyond clinical settings. Wearable devices and home monitors capture physiological measurements continuously, including heart rhythms, activity patterns, sleep quality, and blood glucose levels. These devices generate volumes of data that dwarf what clinical encounters produce. A single day of continuous glucose monitoring yields more data points than years of quarterly laboratory tests. This granularity reveals patterns invisible to episodic measurement, such as the nocturnal hypoglycaemia undetected by daytime finger-prick tests or the atrial fibrillation episodes occurring only during sleep. Yet the sheer volume raises practical questions: who reviews this data, and what responsibility do clinicians bear for abnormalities in continuous streams they never explicitly ordered?

At the molecular level, plummeting sequencing costs have made genomic data increasingly routine. Tumour profiling guides cancer treatment selection; pharmacogenomic testing predicts medication metabolism; genetic risk scores inform screening decisions. Integrating genomic information with clinical data enables precision approaches tailoring treatment to individual biology rather than population averages.

One notable insight from health data analytics is that medical data alone explains surprisingly little about health outcomes. Social determinants (income, education, housing, food security, and social connection) are estimated to account for far more variation in health than healthcare delivery (Hood et al., 2016). Incorporating data about these factors reveals why two patients with identical clinical profiles may have vastly different outcomes, and why interventions targeting social needs may improve health more effectively than additional medical treatment.

No single data source tells the complete story. Each captures a different facet of health, shaped by who collects it, why they collect it, and what they choose to record. The analytical challenge lies in weaving different data types together into coherent understanding. The patient with well-controlled diabetes in their laboratory results, rising healthcare costs in their claims data, and declining functional status in their patient-reported outcomes may be struggling with food insecurity that no clinical data source captures directly.

Data Quality and Governance

IBM Watson Health faced a troubling problem. As documented by STAT News investigative reporting (Ross and Swetlitz, 2018), the artificial intelligence system, trained largely on synthetic and hypothetical patient scenarios compiled by oncologists at Memorial Sloan Kettering rather than on real patient records, was recommending treatments that oncologists at other institutions found inappropriate, sometimes dangerously so. The problem lay in the training data: rather than learning from large-scale real-world cases, Watson had absorbed a small number of curated scenarios that encoded a few specialists' treatment preferences as if they were generalisable evidence. This cautionary tale illustrates a core truth about health analytics: no amount of computational power can extract reliable insights from unreliable data.

The phrase "rubbish in, rubbish out" understates the problem. In health analytics, rubbish in often produces confident-sounding rubbish out: results that appear valid but lead to poor decisions. A predictive model trained on incomplete documentation might learn that certain diagnoses are rare simply because clinicians do not document them consistently. An algorithm analysing treatment outcomes might conclude that a medication is ineffective when patients who received it were simply sicker to begin with.

Data quality problems take many forms, each requiring different remedies. Missing data creates blind spots: important fields left empty, entire patient populations invisible in datasets, outcomes unknown for patients lost to follow-up. Inaccuracies arise when recorded values diverge from reality through transcription errors, outdated information, or misunderstandings. Inconsistencies emerge when different systems represent the same concept differently: one hospital coding a condition as "Type 2 diabetes mellitus" while another uses "adult-onset diabetes" or simply "DM2". These problems compound as data flows through analytical pipelines, each step potentially amplifying distortions introduced earlier.

Addressing data quality requires more than technical fixes; it demands organisational commitment. Data governance establishes the policies, processes, and responsibilities for managing data assets throughout their lifecycle. Who owns the data? Who may access it, and for what purposes? What standards must be met before data enters analytical systems? How is compliance monitored? Effective governance balances competing imperatives: making data accessible for legitimate analysis while protecting sensitive information, enabling innovation while ensuring safety.

At the operational level, master data management ensures consistent representation of core entities across systems. When the same patient appears in the emergency department system, the laboratory information system, and the outpatient EHR, their records must be linked accurately. Get this wrong, and analytics might treat one patient as three separate individuals, or worse, merge records from different patients into a single identity. The technical challenge of patient matching grows as data sources multiply, with names spelled differently, addresses changed, and identifiers varying across systems.

Ultimately, data quality is everyone's responsibility, but someone must be accountable. Data stewards serve this function, designated individuals who understand their data domains intimately and can help analytical teams navigate the gap between what the data appears to show and what it actually means. A steward might explain that the apparent spike in a certain diagnosis reflects a coding change rather than a genuine increase in disease, or that missing values in a particular field indicate normal results rather than absent testing. This contextual knowledge transforms raw data into usable intelligence.

Analytics Approaches

Consider a hospital administrator staring at a dashboard showing that readmission rates have increased by 15% over the past quarter. This single observation triggers a cascade of questions. What happened? Why did it happen? Which patients are most likely to be readmitted next month? And most importantly, what should we do about it? These questions represent a natural progression from observation through understanding to action, a progression that mirrors the evolution of analytical sophistication in healthcare.

The journey begins with descriptive analytics, the foundation upon which all other analysis builds. Before we can understand why something happened or predict what will happen next, we must first establish what has already occurred. Dashboards, reports, and visualisations transform raw data into comprehensible summaries: how many patients were admitted last month, what procedures were performed, how long patients waited in the emergency department. This may seem elementary, but many healthcare organisations struggle to answer even basic questions about their operations. A hospital that cannot reliably count its readmissions cannot hope to reduce them.

Yet description alone leaves important questions unanswered. Knowing that readmissions increased tells us nothing about why they increased or what to do about it. Diagnostic analytics moves from observation to explanation, drilling into data to identify contributing factors. Perhaps the increase correlates with reduced staffing levels during a flu outbreak. Perhaps certain discharge diagnoses show disproportionate increases. Perhaps patients discharged on Fridays, when community services are less accessible over the weekend, are more likely to return. Statistical methods can distinguish genuine associations from coincidental patterns, though they cannot prove causation. Understanding why something happened guides where to focus improvement efforts.

A major shift in health analytics has been the move from backward-looking to forward-looking analysis. Predictive analytics uses patterns in historical data to forecast what is likely to happen next. Rather than asking which patients were readmitted, we ask which patients currently in hospital are likely to be readmitted after discharge. Machine learning models can synthesise hundreds of variables (diagnoses, medications, laboratory trends, prior utilisation patterns, and social circumstances) into risk scores that identify high-risk patients while they are still in hospital, when intervention remains possible. A nurse who knows that a patient has a 40% probability of readmission can coordinate more intensive discharge planning; a care manager can prioritise follow-up calls to those most likely to benefit.

The next step is prescriptive analytics, which recommends what to do about predicted outcomes. If a patient is at high risk for readmission, which specific interventions would most effectively reduce that risk? Should resources be allocated to home visits, telephone follow-up, or transition care clinics? Prescriptive analytics attempts to optimise decisions by recommending actions tailored to individual circumstances. This represents the frontier of health analytics, powerful in principle but challenging in practice, requiring both accurate predictions and causal understanding of how interventions affect outcomes.

These approaches build upon each other rather than replacing one another. Organisations that rush to implement predictive models without mastering descriptive fundamentals often find themselves unable to explain or validate their predictions. The most sophisticated prescriptive algorithm is useless if the underlying data is unreliable or the recommendations cannot be integrated into clinical workflows. Analytics maturity develops progressively, each level providing the foundation for the next.

Table 5.1: Healthcare Analytics Approaches

Machine Learning in Healthcare

Machine learning has become central to health data analytics, enabling the discovery of complex patterns that traditional statistical approaches cannot detect (Rajkomar et al., 2019). Rather than programming explicit rules, machine learning algorithms learn from examples, identifying which combinations of variables predict outcomes by analysing thousands or millions of cases. The technical foundations of machine learning - supervised, unsupervised, deep, and reinforcement learning - and their clinical applications are explored in detail in Chapter 6: Artificial Intelligence in Healthcare. Here, we focus on how these capabilities apply specifically to health data analytics.

In the analytics context, machine learning's power lies in its ability to synthesise hundreds of variables simultaneously. A readmission prediction model, for example, can weigh clinical, utilisation, and social variables together, producing a single risk score available before discharge. Similarly, natural language processing can extract structured data from clinical notes, unlocking information that would otherwise remain inaccessible to analysis, such as diagnoses mentioned in free text, medications discussed but not prescribed, and symptoms reported but not coded. The significant challenges that accompany these capabilities - including bias, generalisability across institutions, and the difficulty of interpreting complex models - are examined in Chapter 6.

Population Health Analytics

A general practice with 10,000 patients cannot provide the same intensity of care to everyone. Some patients are healthy and need little beyond routine preventive services. Others live with multiple chronic conditions, precarious social circumstances, and frequent hospitalisations. If the practice treats everyone identically, it will provide too much care to some and far too little to others. Population health analytics addresses this challenge: understanding who needs what, so that limited resources can be directed where they will do the most good.

Risk stratification lies at the heart of this endeavour. By analysing clinical data, utilisation patterns, and social determinants, algorithms can estimate each patient's probability of adverse outcomes: hospitalisation, emergency visits, disease progression, and death. These predictions reveal the small proportion of patients who account for a disproportionate share of poor outcomes and healthcare costs. In most populations, approximately 5% of patients drive 50% of spending (Mitchell, 2019). Identifying these individuals before crises occur enables proactive intervention: care coordination, home visits, medication management, social service connections. Without such targeting, resources spread too thinly to achieve meaningful impact.

Population health analytics also surfaces opportunities hidden in aggregate statistics. A practice might appear to have excellent diabetes control overall, yet analysis stratified by demographic group could reveal that patients from certain ethnic backgrounds receive less intensive treatment and have worse outcomes. Care gap identification makes these patterns visible and actionable. Which patients are overdue for cancer screening? Who has hypertension documented but no blood pressure recorded in the past year? Where are patients falling through the cracks between primary care and specialist services? These questions, impossible to answer at scale without analytics, become routine when data flows through appropriate systems.

The same analytical capabilities that enable proactive individual care also support population-level surveillance. Long before COVID-19 demonstrated the value of disease surveillance, public health agencies were analysing emergency department visits, pharmacy sales, and laboratory results to detect emerging outbreaks. A spike in patients presenting with respiratory symptoms, an unusual pattern of antiviral prescriptions, a cluster of positive influenza tests: each signal might indicate an outbreak requiring response. Analytics can detect these patterns earlier than traditional reporting mechanisms, when intervention can still alter disease trajectories.

For healthcare organisations managing complex payment arrangements, population health analytics also illuminates financial sustainability. Which services generate the most value relative to their cost? Which patient populations are growing, and what will they need? Where do spending patterns suggest waste, and where might investments yield savings? These questions become increasingly urgent as healthcare payment shifts from rewarding volume to rewarding outcomes. Organisations that cannot answer them will struggle to thrive in value-based arrangements.

Importantly, population health analytics can make visible the inequities that aggregate measures conceal. When performance is reported as a single number, disparities disappear into averages. A hospital might report excellent overall outcomes while Black patients experience consistently worse results. A screening programme might achieve high uptake overall while systematically missing patients in deprived areas. Equity analytics stratifies results by race, ethnicity, geography, language, and socioeconomic status, forcing uncomfortable questions into view. This visibility is a necessary first step towards accountability and action, though analytics alone cannot address the systemic factors that create disparities in the first place.

Clinical Decision Support

Every experienced clinician has stories of near-misses: the allergic reaction almost caused by a prescription they nearly wrote, the diagnosis almost missed until a late-arriving laboratory result prompted reconsideration, the dosing error almost made for a patient whose renal function they had forgotten. Clinical decision support aims to make the computer an ally in preventing such errors, surfacing relevant information at moments when it can influence decisions.

The simplest form of decision support, and still the most widespread, uses rules to trigger alerts. If a clinician prescribes penicillin for a patient with documented penicillin allergy, an alert fires. If a laboratory result shows critically elevated potassium, the responsible clinician receives notification. If a prescribed dose exceeds safe limits for a patient's kidney function, the system warns before the order is signed. These rule-based systems encode expert knowledge into automated checks, catching errors that busy clinicians might miss.

Yet rule-based alerts carry an inherent tension between sensitivity and specificity that can lead to "alert fatigue" - a problem discussed in detail in Chapter 2: Electronic Health Records. Machine learning offers a potential path beyond this impasse: rather than alerting on every abnormal value, predictive models synthesise vital signs, laboratory trends, nursing assessments, and medication orders to identify patients whose overall pattern suggests impending crisis. The alert becomes meaningful precisely because it is selective.

The most ambitious decision support systems attempt to guide diagnosis and treatment selection. Imaging algorithms can highlight suspicious regions on radiographs, potentially catching findings that might otherwise be overlooked during busy reporting sessions. Diagnostic support systems can suggest conditions consistent with a patient's presenting symptoms and findings, prompting consideration of possibilities that might not have come to mind. Treatment recommendation engines can identify therapies supported by evidence for patients with particular characteristics, potentially reducing unwarranted variation in care.

Whether decision support improves outcomes depends less on the sophistication of the underlying algorithms than on how well they integrate into clinical workflows. A brilliant algorithm that delivers recommendations at the wrong moment, in the wrong format, or through the wrong channel will be ignored or resented. Effective decision support appears when and where clinicians need it, presents information clearly and concisely, offers genuinely actionable recommendations, and learns from feedback about what helps and what hinders. These human factors prove at least as challenging as the technical problems of prediction and recommendation.

Data Infrastructure

Even a well-designed algorithm is of little use without the infrastructure to feed it data and deliver its outputs. A hospital may hold years of clinical observations, laboratory results, and outcome data, but if that information remains siloed across dozens of systems - each with its own format and access controls - meaningful analysis becomes impractical. Data infrastructure receives less attention than the analytical methods it supports, but it is no less important.

The traditional approach centres on the data warehouse: a single repository consolidating information from operational systems across the organisation. Laboratory results from the laboratory information system, orders from the EHR, claims from the billing system, and demographics from the master patient index flow through extract, transform, and load (ETL) processes that standardise formats, resolve inconsistencies, and link records belonging to the same patient. The resulting warehouse presents a unified view optimised for analytical queries, questions that would be impossibly slow against operational databases designed for real-time transaction processing.

Building and maintaining data warehouses requires substantial expertise and ongoing investment. Every source system change threatens to break established data flows. Every new analytical need requires evaluation of whether existing structures can accommodate it. The rigid structure that makes warehouses reliable also makes them slow to adapt, a problem when the questions healthcare organisations need to answer evolve faster than traditional warehouse development cycles allow.

Data lakes represent a complementary approach, storing raw data in native formats without requiring predefined structure. Rather than transforming data before loading, lakes preserve original formats and defer structure until analysis time. This flexibility accommodates the diverse data types that characterise modern healthcare, including structured clinical data, clinical notes, imaging files, genomic sequences, and device streams. The trade-off is complexity at the point of use; analysts must understand source data formats and implement transformations themselves, rather than relying on pre-processed warehouse structures.

Cloud computing has changed the economics of health data infrastructure. Rather than purchasing and maintaining physical servers, organisations can provision virtually unlimited storage and processing capacity on demand, paying for what they use. Major cloud providers offer healthcare-specific services addressing regulatory requirements like HIPAA compliance, though the shared responsibility model requires careful attention to security configuration. The scalability of cloud platforms enables analyses that would be impractical with on-premises infrastructure, such as training machine learning models on millions of records or processing the continuous streams generated by connected devices.

None of this infrastructure delivers value without connections to the systems where data originates and decisions are made. Interoperability standards, particularly FHIR, enable data exchange across organisational boundaries and between vendor systems. Application programming interfaces (APIs) provide programmatic access for analytical applications, enabling algorithms to retrieve patient data and deliver recommendations in real time. As healthcare moves towards interconnected ecosystems rather than institutional silos, these integration capabilities become as important as the analytical capabilities they support.

Real-World Evidence

Randomised controlled trials remain the gold standard for establishing treatment efficacy, but they carry inherent limitations. Trials enrol selected populations, often younger, healthier, and more homogeneous than patients encountered in routine practice. They run for defined periods that may not capture long-term effects. They compare treatments under controlled conditions that differ from messy clinical reality. And they are expensive, slow, and limited in the questions they can address. Real-world evidence, insights derived from routine clinical data, offers a complementary perspective that no trial can provide.

Consider the problem of understanding medication safety after market approval. Pre-market trials, even large ones, enrol thousands of patients for months or a few years. Rare adverse events occurring in one per ten thousand patients, or problems emerging only after years of use, remain invisible in trial data. Yet once a medication reaches market, millions of patients begin taking it, and their experiences accumulate in EHRs, claims databases, and registries. Analysis of this real-world data can detect safety signals that trial data could never reveal: the unexpected cardiac risk emerging after years of use, the problematic interaction with another medication not tested in trials, the adverse effects disproportionately affecting patient populations underrepresented in trials.

The same logic applies to understanding effectiveness in practice. A blood pressure medication might lower blood pressure impressively in trial patients who take it consistently under careful monitoring. But what happens when the same medication is prescribed to elderly patients with multiple comorbidities, complex medication regimens, and variable adherence? Real-world data reveals what actually happens when treatments leave controlled environments and enter ordinary clinical care. Comparative effectiveness research uses this data to compare treatments as actually used in practice (OHDSI Collaborative, 2024).

Regulatory agencies have increasingly embraced real-world evidence. The FDA has developed frameworks for incorporating real-world data into regulatory decisions (FDA, 2018), recognising that such evidence can sometimes answer questions impractical to address through traditional trials. Expanded indications, label modifications, and safety updates can draw on real-world evidence when trial evidence is insufficient or impractical to generate.

Yet real-world evidence carries methodological challenges that must be acknowledged honestly. Patients receiving different treatments in routine practice differ in ways that influence outcomes. Sicker patients might receive more aggressive treatment, making the treatment appear harmful when in fact the underlying illness caused poor outcomes. Patients who tolerate a medication continue taking it while those who experience side effects switch to alternatives, making longer exposure appear falsely protective. Sophisticated statistical methods can adjust for measured confounders, but cannot eliminate bias from unmeasured factors. Real-world evidence complements rather than replaces trials, answering questions that trials cannot while remaining subject to limitations that trials avoid.

Table 5.2: RCTs vs Real-World Evidence

Ethical Considerations

In 2019, a study published in Science challenged assumptions about algorithmic fairness in healthcare (Obermeyer et al., 2019). A widely used commercial algorithm, deployed across major health systems to identify patients needing additional care, was systematically underestimating the needs of Black patients. At any given risk score, Black patients were substantially sicker than white patients with the same score. The algorithm was not explicitly using race; it was predicting healthcare costs, which happened to be lower for Black patients due to unequal access to care. By learning patterns from a healthcare system marked by racial inequality, the algorithm had encoded that inequality into its predictions, directing resources away from patients who needed them most.

This case illustrates a broader truth: algorithms do not exist outside the societies that create them. They learn from historical data that reflects historical practices, including historical biases. If training data underrepresents certain populations, as clinical trial data historically underrepresented women and minorities, algorithms trained on that data may perform poorly for those populations. If outcome measures themselves are biased, as cost-based measures disadvantage those who receive less care regardless of need, algorithms optimising those measures will perpetuate that bias. Preventing algorithmic discrimination requires technical solutions alongside critical questions about what we measure, whose data we collect, and whose interests our systems serve.

Privacy presents equally complex challenges. Health data analytics derives its power from combining information across sources, linking clinical data with claims data with social determinants with genomic information, to build comprehensive pictures of individual health. Yet this same power creates privacy risks. De-identification offers limited protection when re-identification becomes possible through data linkage. The combination of diagnosis, procedure, and demographic information that makes an individual identifiable in one dataset becomes identifying when linked with external information. Patients who consent to their data being used for one purpose may find it repurposed for analyses they never anticipated and might not approve.

A significant ethical challenge is transparency. When an algorithm recommends against a treatment, or flags a patient as high-risk, or suggests a diagnosis, on what basis does it make this recommendation? Deep learning models in particular operate as black boxes, transforming inputs into outputs through calculations no human can follow. Yet the patients and clinicians affected by these recommendations deserve to understand them well enough to evaluate whether they apply appropriately to specific situations. Explainability in machine learning remains an active research area, but current capabilities fall short of what genuine transparency would require.

These are not abstract concerns. Organisations deploying health data analytics need practical mechanisms to monitor for bias, protect privacy through both technical and governance measures, and maintain transparency about how analytical outputs inform clinical decisions. The benefits of data-driven approaches to healthcare are real, but they will be distributed fairly only if ethical considerations are built into analytical systems from the outset rather than treated as an afterthought.

Summary

Health data analytics draws on a widening range of data sources - clinical records, claims, registries, patient-reported measures, genomic data, and continuous streams from wearable sensors - to inform decisions at both the individual and population level. The analytical methods available range from straightforward descriptive reporting through to machine learning models capable of predicting clinical deterioration or identifying patients at risk of falling through gaps in care.

The value of these methods depends, however, on foundations that are easily overlooked: the quality and governance of the underlying data, the infrastructure that makes it accessible for analysis, and the ethical frameworks that govern its use. An algorithm trained on biased data, or deployed without attention to how it affects different patient groups, may cause harm rather than good.