TDM 30200 Project 11 - Survival Analysis

Censored Observations

As we mentioned, although this setup may appear similar to a regression problem, an important complication exists: many patients survive past the end of the study, so their exact event times are unknown.

Those patients are considered censored.

For censored individuals:

Censored data is still informative because each censored observation tells us the patient survived at least up to their recorded follow-up time.

Using the notation above, we still observe

$Y = \min(T, C)$

but now we also record a status indicator to tell us whether the event was actually seen.

$\delta$ = \begin{cases} 1 & \text{if } T \le C \quad \text{(event observed)} \\ 0 & \text{if } T > C \quad \text{(censored)} \end{cases}

So for each individual we have the pair $(Y, \delta)$.

Even though censored observations do not give us the exact event time, they still provide important information. Each censored patient tells us that the event did not occur before their last observed time.

Types of Censoring

There are several ways in which censoring can occur. The most common types are right censoring, left censoring, and interval censoring.

Right Censoring

Right censoring occurs when we know that the event time is at least as large as the observed time. In notation

$T \ge Y$

Examples

In both cases, we do not know the exact value of $T$, but we do know that $T$ is larger than the final observed time.

Left Censoring

Left censoring occurs when the event happens before the time that observation begins, but the exact event time is unknown.

In this case

$T \le Y$

Example

Suppose we survey a group of pregnant women at one visit late in pregnancy. Some of them have already given birth by the time of the survey. For those women, we only know that the birth occurred some time before the survey date, which means their pregnancy duration is less than the time since conception that corresponds to the survey.

Interval Censoring

Interval censoring occurs when we only know that the event happened between two observation times.

In this setting we know that

$T \in (L, U)$

where $L$ and $U$ are the last time the event was known not to have happened and the first time it was known to have happened.

Example

We do not know the exact day of the event, only that it occurred at some time between two visits.

Censored Data in Our Example Dataset

Figure interpretation:

In the figure above, each horizontal line represents one patient’s follow-up time starting at study entry (time = 0). An X marks when a death occurs (event observed), and an open circle marks a censored observation (patient still alive at last follow-up). The dashed vertical line shows a chosen reference (landmark) time at 365 days, which we use as a snapshot of follow-up rather than the true end of the study.

Notice that some patients die before this reference time (e.g., Patients 1 and 2), while others die after it (e.g., Patient 3 and 4). Patients who remain alive beyond the reference time (e.g., Patient 5) have survival times that extend past 365 days. From the perspective of a snapshot taken at day 365, patients who die after this point and patients who remain alive after this point are indistinguishable at that moment, because neither outcome has occurred yet. This demonstrates why survival analysis must explicitly account for censoring and time-to-event information rather than relying on simple yes/no outcome models.

Assumption About Censoring

In order for standard survival analysis methods to give valid results, we usually assume that

Informally, this means that whether or when a subject is censored should not depend on their unobserved survival time in a way that is not already explained by the variables in the model. When this assumption is reasonable, censored observations still contribute correct information about the survival experience of the population.

The survival function can be estimated using the Kaplan Meier estimator:

$\hat{S}(t) = \prod_{t_i \le t} \left(1 - \frac{d_i}{n_i}\right)$.

In this form

At each event time, survival is updated by multiplying the previous survival probability by a new fraction that accounts for how many deaths occurred out of the group still being observed.

Data Used for Illustration

We demonstrate this process using our lung cancer dataset. Each row of the data includes

Example rows from the dataset are shown below.

Identify Event Times

Kaplan Meier updates the survival probability only at time points where deaths occur. These are called event times.

From our dataset, the first two event times are:

This means

We now calculate survival probabilities at each of these two event times.

Step 1 Calculate Survival at Time 5

At the first event time:

Apply the Kaplan–Meier step formula:

$S(5) = 1 \times \left(1 - \frac{1}{228}\right)$

This gives

S(5) = 0.995614

Interpretation

After day 5, approximately 99.56 percent of patients are estimated to survive beyond that time.

Step 2 Calculate Survival at Time 11

Move to the second event time.

The survival probability is updated by multiplying the previous estimate by a new fraction:

$S(11) = S(5) \times \left(1 - \frac{3}{227}\right)$

This gives

$S(11) = 0.982456$

Interpretation

After day 11, approximately 98.25 percent of patients are estimated to survive beyond this time.

Summary Table

The survival probabilities calculated for our first two event times are summarized below.

Repeating this procedure at all event times builds the full Kaplan Meier survival curve.

Comparing Survival Curves: The Log-Rank Test

Once survival curves are estimated using the Kaplan–Meier method, we often want to compare survival between two or more groups (for example, male vs female patients). While visual inspection of Kaplan–Meier curves can suggest differences between groups, overlap or small separations can be difficult to interpret by eye alone. To formally evaluate whether any observed differences are statistically meaningful, we use the log-rank test.

The log-rank test compares:

These comparisons are summed across all event times to produce a test statistic and corresponding p-value.

Interpretation

The log-rank test therefore provides a formal statistical complement to Kaplan–Meier plots, moving beyond visual comparison to hypothesis testing.

The Cox Proportional Hazards Model

The Cox Proportional Hazards model describes how patient characteristics (covariates) affect the risk of death over time using the dataset survival_data.

The hazard function is:

$h(t) = h_0(t)\exp(\beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p)$,

where:

Each covariate receives one coefficient.

Example model using our dataset

Suppose our Cox model includes age, sex, and ph.ecog. The fitted model is written as:

$h(t) = h_0(t)\exp(\beta_1(\text{age}) + \beta_2(\text{sex}) + \beta_3(\text{ph.ecog}))$

Each coefficient represents the log hazard ratio of that variable while holding all others constant.

Log hazard coefficients

Example coefficient output:

From coefficients to hazard ratios

Coefficients are converted to hazard ratios using:

$HR = e^{\beta}$.

Why do we take the exponent of the log hazard coefficient?

Because the model uses an exponential function:

Hazard ratios

How to interpret hazard ratios

Applying the model to real patients

Using rows from survival_data:

Because Patient 0 is older and has worse ECOG performance, the Cox model predicts:

Patient 0 has a higher hazard of death than Patient 2.

Sample Cox output

Statistical significance

Example:

$HR(\text{ph.ecog}) = 2.46$, CI $(1.42,4.28)$

Because the CI does not include 1 and the p-value is < 0.05, ECOG performance status is a significant predictor of mortality.

Key takeaways

References

Klein, J. P., & Moeschberger, M. L. (2003). Survival Analysis: Techniques for Censored and Truncated Data. Springer.

James, G., Witten, D., Hastie, T., Tibshirani, R. (2023). Statistical Learning with Python. Springer — Survival Analysis section.