Model time to an event

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Background

When to use it

Outside medical research, the Kaplan–Meier estimator of survival may be used to measure how long people remain unemployed after a job loss, or the time-to-failure of machine parts.^[2] Reach for a survival model when the question is how long until something happens and some of the rows have not reached that moment yet. (Meridian's view)

Right censoring occurs when only a lower limit for the true event time is known, as for subjects still alive when they are lost to follow-up or when the study ends.^[1] Keep every row, and record for each one whether its event was observed: a duration without that flag cannot be told apart from a lifetime that is still running. (Meridian's view)

An event known to have happened before a subject entered the study, at an unknown time, is left-censored, and one known only to fall between two examinations is interval-censored.^[1] Truncation differs from left censoring: a left-censored subject is known to exist, while a truncated subject, with a lifetime below some threshold, may never be observed at all.^[1] Left-truncated data are common in actuarial work for life insurance and pensions.^[1] Censoring is called random, or non-informative, when each subject's censoring time is statistically independent of its failure time.^[3] Ask how each row came to be censored before trusting the result, because a subject who leaves for a reason tied to the event is not a neutral gap in the data. (Meridian's view)

The method

The survival function S(t) is the probability that a subject survives longer than time t.^[1]

The Kaplan–Meier (KM) estimator, also known as the product limit estimator, is a non-parametric statistic used to estimate the survival function from lifetime data.^[2] The KM estimator is named after Edward L. Kaplan and Paul Meier, whose manuscripts the journal editor John Tukey persuaded them to combine into one paper, published in 1958.^[2] At each time when at least one event happened, the KM estimate multiplies by one minus the number of events divided by the number known to have survived, neither having had the event nor been censored, up to that time.^[2] A KM curve can take right-censored data into account, and when no truncation or censoring occurs the KM curve is the complement of the empirical distribution function.^[2] A KM estimator needs at least two pieces of data for each subject: the status at last observation, event or right-censored, and the time to the event or to censoring.^[2] In Yggdrasil Decision Forests (YDF), a survival analysis task takes a numerical label for the age at which the event, or right-censoring, was observed, and a boolean event_observed column.^[4]

Proportional hazards models relate the time before an event to one or more covariates, and in them a unit increase in a covariate has a multiplicative effect on the hazard rate.^[5] In a proportional hazards model, the baseline hazard function describes how the risk of the event per time unit changes over time at baseline levels of the covariates.^[5] Sir David Cox developed the proportional hazards model in 1972.^[6] Sir David Cox observed that if the proportional hazards assumption holds, the effect parameters can be estimated without any consideration of the full hazard function, an approach called the Cox proportional hazards model, or Cox model.^[5] The Cox model's partial likelihood is the probability that subjects experienced their events in the order they actually occurred, given the times of occurrence and the subjects' covariates.^[5] Standard estimation of the Cox proportional hazards model does not directly estimate the baseline hazard rate, which inferences about probabilities of survival would need.^[5] Sir David Cox noted that the biological interpretation of the proportional hazards assumption can be quite tricky.^[5]

Compared with the alternatives

The log-rank test compares the survival times of two or more groups.^[1] Kaplan–Meier curves and log-rank tests are most useful when the predictor is categorical or takes a small number of values, and do not work easily with quantitative predictors such as age.^[1] For quantitative predictors, Cox proportional hazards regression is an alternative, and the log-rank test is a special case of a Cox proportional hazards analysis.^[1]

Survival models can be viewed as ordinary regression models in which the response variable is time, but censoring complicates computing their likelihood function.^[1] The Cox proportional hazards regression model is a linear model, similar to linear regression and logistic regression.^[1] Tree-structured survival models, including survival random forests, may give more accurate predictions than Cox models.^[1] Discrete-time survival models can be mapped to a binary classification problem, with the survival period split into intervals and a binary target recording whether the event takes place.^[1] Accelerated failure time models do not exhibit proportional hazards, and describe a life history that is accelerated or decelerated.^[5] If every row has an observed outcome and the question is only whether it happens, a classifier may be enough; once the question is when, and some rows are still waiting, a survival model is the honest choice. (Meridian's view)

Sources

1. [1]Survival analysissecondary source
2. [2]Kaplan–Meier estimatorsecondary source
3. [3]Censoring (statistics)secondary source
4. [4]yggdrasil-decision-forests/yggdrasil_decision_forests/model/abstract_model.proto at 0a54b57c95ff0cf42ecf0310ace0513a4f32cb65 · google/yggdrasil-decision-forestsdocumentation
5. [5]Proportional hazards modelsecondary source
6. [6]David Cox (statistician)secondary source