5. Structuring the Markov Model

Learning Objectives and Outline

Learning Objectives

• Understand differences between rates and probabilities, hazard rates, relative risks, and other relevant model inputs.
• Understand rate-to-probability conversion formulas and transition probability embedding.
• Explain how to embed a transition probability matrix with a defined timestep.
• Explore age-, time- and country/region-specific adjustments to a Markov model.

Outline

1. Rates, probabilities, and other model inputs.
2. Constructing the transition probability matrix.
3. Model dynamics

• Non-stationary rates: mortality and other time/age-varying parameters.
• Tunnel states and other disease-specific transition dynamics

4. Working backwards: adapting an existing model to your country/region.

Goal

What Never Happens

Problem

• Need transition probabilities but literature-based parameters are reported as:

1. Rates
2. Hazard ratios
3. Odds ratios
4. Relative risks
5. Transition probabilities (rare!)

How can we deal with this?

Solution

• Need “common ground” where we can combine and transform different model inputs.
• This “common ground” is often found in a transition rate matrix.

Solution

Solution

Solution

Solution

Transition Rate Matrix

• The central “hub” of a Markov model.
• Straightforward to convert rate matrix into a transition probability matrix.
• Can be used to change the cycle length.
• Facilitates modeling using alternative techniques:
  • Continuous time Markov
  • Discrete event simulation

Four-Step Process

1. Use data inputs/published literature to define a rate matrix \(\mathbf{R}\).
2. Make strategy-specific adjustments to \(\mathbf{R}\) as needed.
3. “Embed” the transition probability matrix using the rate matrix
4. Make further overall or strategy-specific adjustments to \(\mathbf{P}\) as needed.

What is the difference between a rate and a probability?

Rates vs. Probabilities

• Rate: number of occurrences of an event per unit of time.
• Probability: Liklihood that an event will occur for an in individual over a defined time period.
• Major difference is in denominator: rates take into account time at risk while probabilities do not.

Rates vs. Probabilities

Rates

• Number of events divided by the total time at risk experienced by all people followed.
• Ranges from 0 to \(\infty\).
• \(\frac{\# \text{events in time period}}{\text{Total time period experienced by all subjects followed}}\)

Rates

Probabilities

Rates vs. Probabilities: Example¹

• Suppose a study followed 100 people with congestive heart failure for 4 years.
• At the end of 4 years, 40 had died.
• The probability of death over 4 years is \(40/100=0.40\).

1. Source: Gidwani and Russell (2020)

Rates vs. Probabilities: Example¹

• A rate takes into acccount the time each person was at risk.
• The 60 who survived were at risk the entire 4 years and contributed \(60 \times 4 = 240\) years at risk.
• Once a person dies, he/she is no longer at risk.

1. Source: Gidwani and Russell (2020)

Rates vs. Probabilities: Example¹

• When a study does not report time at risk, the conventional assumption is that the events were spread evenly over the time period.
• Using this assumption, the average time at risk for the 40 who died was 2 years, adding 40 × 2 = 80 years at risk.

1. Source: Gidwani and Russell (2020)

Rates vs. Probabilities: Example¹

• Total time at risk for the cohort of 100 people is 320 person-years (240+80).
• Thus, the rate of death from CHF is 40/320 = 0.125 deaths per person-year.
• Let’s now construct a rate matrix with three states: CHF, Death from CHF, and Death from other causes.

1. Source: Gidwani and Russell (2020)

Rates vs. Probabilities: Example

	CHF	D_CHF	D_OTH
CHF	0	0	0
D_CHF	0	0	0
D_OTH	0	0	0

Rates vs. Probabilities: Example

• 0.125 per person year is the rate we’d enter into our rate matrix.
• We could also think of a separate death rate from background causes (e.g., 0.006 per person-year).¹

1. This often comes from separate vital statistics / life-table data sources.

Rates vs. Probabilities: Example

	CHF	D_CHF	D_OTH
CHF	0	0.125	0.006
D_CHF	0	0.000	0.000
D_OTH	0	0.000	0.000

Rates vs. Probabilities: Example

• The diagonal elements in a rate matrix are just the negative sum of the off-diagonal elements.
• In this example, the diagonal value for the 1st row would be -0.131 = -(0.125+0.006)
• We can leave all other rate matrix values at 0 because the rate of progression from death to other states is zero.

	CHF	Death_CHF	Death_Other
CHF	-0.131	0.125	0.006
Death_CHF	0	0.000	0.000
Death_Other	0	0.000	0.000

Summary: Rates vs. Probabilities

Statistic	Evaluates	Range	Applicable Domain
Rate	\(\frac{\# \text{events in time period}}{\text{Total time period experienced by all subjects followed}}\)	0 to \(\infty\)	Rate matrix
Probability/risk	\(\frac{\# \text{events in time period}}{\# \text{people followed for time period}}\)	0-1	Probability matrix

1. Place rates in a rate matrix \(\mathbf{R}\).

1. Place rates in a rate matrix \(\mathbf{R}\).

• Country/region-specific background mortality rate (0.006)
• Disease mortality rates from existing population-based epidemiological study. (e.g., 0.125 = 40 cases per 320 person-years)
• Diagonal value that is the negative sum of the off-diagonal values in each row (-0.131).

“Natural History” (i.e., “do nothing”) rate matrix:

	CHF	Death_CHF	Death_Other
CHF	-0.131	0.125	0.006
Death_CHF	0.000	0.000	0.000
Death_Other	0.000	0.000	0.000

1. Place rates in a rate matrix \(\mathbf{R}\).

• What we have just done is contruct a rate matrix for a “natural history” model of the disease (CHF).
• This is a version of the model in which we allow the disease process to play out naturally, with no further intervention.
• Sometimes called a “do nothing” strategy.

1. Place rates in a rate matrix \(\mathbf{R}\).

• The “natural history” model is useful because it can help us verify that the model matches what we see in the “real world.”
• The “natural history” model also can be used to calibrate transition rates to different countries/contexts.
• For example, we could recalibrate the model so the CHF mortality transition rate matches our country-specific data.

1. Place rates in a rate matrix \(\mathbf{R}\).

• Alternatively, suppose we “borrow” a model developed in another country/context.
• That model will be based on underlying rates specific to that context.
• The underlying rate matrix can be used to “swap in” transition rates that apply to our country/context.
  • Example: change the background mortality rate to match you country’s.

2. Make adjustments to \(\mathbf{R}\) as needed.

2. Make adjustments to \(\mathbf{R}\) as needed.

• Suppose that a new strategy can reduces the risk of CHF mortality by 20% (i.e., hazard ratio = 0.8).
• We can simply apply this hazard ratio directly to construct a rate matrix for strategy A.
• For the Strategy A rate matrix, the rate of CHF death is \(0.8 * 0.125 = 0.1\)
• Make sure that the diagonal element is adjusted to account for this change!

“Natural History” rate matrix

	CHF	Death_CHF	Death_Other
CHF	-0.131	0.125	0.006
Death_CHF	0.000	0.000	0.000
Death_Other	0.000	0.000	0.000

2. Make adjustments to \(\mathbf{R}\) as needed.

• Suppose that a new strategy can reduces the risk of CHF mortality by 20% (i.e., hazard ratio = 0.8).
• We can simply apply this hazard ratio directly to construct a rate matrix for strategy A.
• For the Strategy A rate matrix, the rate of CHF death is \(0.8 * 0.125 = 0.1\)
• Make sure that the diagonal element is adjusted to account for this change!

“Natural History” rate matrix

	CHF	Death_CHF	Death_Other
CHF	-0.131	0.125	0.006
Death_CHF	0.000	0.000	0.000
Death_Other	0.000	0.000	0.000

“Strategy A” rate matrix

	CHF	Death_CHF	Death_Other
CHF	-0.106	0.1	0.006
Death_CHF	0	0	0.000
Death_Other	0	0	0.000

2. Make adjustments to \(\mathbf{R}\) as needed.

• Another adjustment we could make at this stage is the time cycle length.
• Suppose our rate matrix is defined in terms of a one-year time cycle, but we want to convert to a monthly cycle.
• In that event, we’d simply divide each rate in the matrix by 12, and the resulting matrix would be for a monthly timestep.

“Natural History” rate matrix (one-year timestep)

	CHF	Death_CHF	Death_Other
CHF	-0.131	0.125	0.006
Death_CHF	0.000	0.000	0.000
Death_Other	0.000	0.000	0.000

2. Make adjustments to \(\mathbf{R}\) as needed.

• Another adjustment we could make at this stage is the time cycle length.
• Suppose our rate matrix is defined in terms of a one-year time cycle, but we want to convert to a monthly cycle.
• In that event, we’d simply divide each rate in the matrix by 12, and the resulting matrix would be for a monthly timestep.

“Natural History” rate matrix (one-year timestep)

	CHF	Death_CHF	Death_Other
CHF	-0.131	0.125	0.006
Death_CHF	0.000	0.000	0.000
Death_Other	0.000	0.000	0.000

“Natural History” rate matrix (one-month timestep)

	CHF	Death_CHF	Death_Other
CHF	-0.0109	0.0104	5e-04
Death_CHF	0	0	0
Death_Other	0	0	0

3. “Embed” the transition probability matrix using the rate matrix

3. “Embed” the transition probability matrix using the rate matrix

• Our next step is to convert the transition rate matrix into a transition probability matrix.
• Common practice is to use rate-to-probability conversion formulas

3. “Embed” the transition probability matrix using the rate matrix

• Our next step is to convert the transition rate matrix into a transition probability matrix.
• Common practice is to use rate-to-probability conversion formulas

\[ p = 1 - \exp(-rt) \] where \(r\) is the rate and \(t\) is the time-step.

3. “Embed” the transition probability matrix using the rate matrix

• This formula works fine when there is only one possible state an individual can transition to.
• The formula will not calculate the correct transition probability if there are two or more states someone can transition to.
• See the workshop blog posting for more.

3. “Embed” the transition probability matrix using the rate matrix

• We will cover two approaches for constructing a transition probability matrix.
• The first is technically incorrect, but is widely used and easier to implement because it ignores compound transitions (i.e., multiple transitions within a cycle).

3. “Embed” the transition probability matrix using the rate matrix

The two approaches

• 3a. Construct a transition probability matrix using rate-to-probability conversion formulas.
• 3b. Embed the transition probability matrix using the rate matrix exponential (i.e., \(\mathbf{P}=e^{\mathbf{R}}\)).

3a. Rate-to-probability conversion

The probability of transitioning from health state \(A\) to health state \(B\) is:

\[ p_{AB}= \frac{r_{AB}}{\sum_S r_{AS}}\big ( 1 - e^{-(\sum_S r_{AS}) t}\big ) \] where \(S\) captures all the health states (i.e., columns) in the transition rate matrix, and \(t\) is the time-step (e.g., \(t=1\) if 1 year, \(t=1/12\) if one month, etc.).

3a. Rate-to-probability conversion

Let’s build on our chronic heart failure example from earlier. Here is the transition rate matrix we constructed:

	CHF	D_CHF	D_OTH
CHF	-0.131	0.125	0.006
D_CHF	0.000	0.000	0.000
D_OTH	0.000	0.000	0.000

3a. Rate-to-probability conversion

Let’s now calculate the probability of transitioning from the CHF state to D_CHF (death from heart failure).

3a. Rate-to-probability conversion

Annual probability of transitioning from CHF to death from CHF:

\[ p_{\text{CHF,D_CHF}}= \frac{r_{\text{CHF,D_CHF}}}{r_{\text{CHF,D_CHF}} + r_{\text{CHF,D_OTH}}}\big ( 1 - e^{-(r_{\text{CHF,D_CHF}} + r_{\text{CHF,D_OTH}}) \times 1}\big ) \]

We can find each of these rates in our transition rate matrix…

3a. Rate-to-probability conversion

Annual probability of transitioning from CHF to death from CHF:

\[ p_{\text{CHF,D_CHF}}= \frac{\color{red}{0.125}}{\color{red}{0.125}+\color{green}{0.006}}\big ( 1 - e^{-(\color{red}{0.125}+\color{green}{0.006}) \times 1}\big ) = 0.1172 \]

	CHF	D_CHF	D_OTH
CHF	-0.131	0.125	0.006
D_CHF	0.000	0	0
D_OTH	0.000	0	0

3a. Rate-to-probability conversion

Let’s now calculate the probability of transitioning from the CHF state to D_OTH (death from other causes).

3a. Rate-to-probability conversion

Annual probability of transitioning from CHF to death from other causes:

\[ p_{\text{CHF,D_CHF}}= \frac{\color{green}{0.006}}{\color{red}{0.125}+\color{green}{0.006}}\big ( 1 - e^{-(\color{red}{0.125}+\color{green}{0.006}) \times 1}\big ) = 0.00562 \]

	CHF	D_CHF	D_OTH
CHF	-0.131	0.125	0.006
D_CHF	0.000	0	0
D_OTH	0.000	0	0

3a. Rate-to-probability conversion

• We now have the quantities needed to complete the first row of our transition rate matrix.

• Recall that the diagonal elements of the transition probability matrix are just 1 minus the other transition probabilities.

3a. Rate-to-probability conversion

Calculated transition probability matrix:

	CHF	D_CHF	D_OTH
CHF	0.8772	0.1172	0.0056
D_CHF	0.0000	1.0000	0.0000
D_OTH	0.0000	0.0000	1.0000

3b. Exponentiate the transition rate matrix

• The most technically correct approach for “embedding” a transition probability matrix is using the rate matrix exponential.

• This is a matrix analogue to the cellwise rate-to-probability matrix process we just went through.

3b. Exponentiate the transition rate matrix

\[ \mathbf{P} = e^{\mathbf{R}} \]

3b. Exponentiate the transition rate matrix

Pros:

• Embedding the transition probability matrix in this way ensures that the correct transition probabilities are calculated.

• Without going into too many details, this approach ensures that some compound transitions are not “hidden” in the Markov cycle.

• This ensures that our discrete time Markov model accurately represents the underlying continuous time process.

3b. Exponentiate the transition rate matrix

Cons:

• A major drawback is that this approach can create some “jumpover” states that are seemingly inconsistent with the underlying model (see blog for more).

• Accounting for “hidden” transitions and jumpover states requires augmenting the transition probability matrix (again, see blog for details).

3b. Exponentiate the transition rate matrix

Cons, cont’d:

• Excel does not easily do matrix exponentiation, however you can use an approximation (we’ll do this in a case study).

• Modern (free) statistical software can easily exponentiate a matrix:

3b. Exponentiate the transition rate matrix

Cons, cont’d:

• Excel does not easily do matrix exponentiation, however you can use an approximation (we’ll do this in a case study).

• Modern (free) statistical software can easily exponentiate a matrix:

library(expm)
m_P = expm(m_R)

3b. Exponentiate the transition rate matrix

For our chronic heart failure example, the exponentiated matrix yields a very similar answer to the first approach:

	CHF	D_CHF	D_OTH
CHF	0.8772	0.1172	0.0056
D_CHF	0.0000	1.0000	0.0000
D_OTH	0.0000	0.0000	1.0000

So Where Are We Now?

So Where Are We Now?

• By constructing our model using the “roots” of a transition rate matrix, we can incorporate disparate sources of information.

• Facilitates country/region-specific background mortality.

• Facilitates standardizing inputs measured at different time intervals.

So Where Are We Now?

• Not all literature-based parameters operate on transition rates.

• You will often find that the strategies you want to model have inputs based on odds ratios, relative risks, risk differences, etc.

CHF Example, Continued

• Recall from earlier that we constructed two transition rate matrices:

  1. “Natural History” rate matrix
  2. Strategy A (“New Drug”) rate matrix based on a hazard ratio (for CHF mortality) of 0.8.

CHF Example, Continued

These two rate matrices can be used to construct the following transition probability matrices:

m_P_NH =

	CHF	D_CHF	D_OTH
CHF	0.8772	0.1172	0.0056
D_CHF	0.0000	1.0000	0.0000
D_OTH	0.0000	0.0000	1.0000

m_P_A =

	CHF	D_CHF	D_OTH
CHF	0.8994	0.0949	0.0057
D_CHF	0.0000	1.0000	0.0000
D_OTH	0.0000	0.0000	1.0000

CHF Example, Continued

• Now suppose we wanted to model a second strategy (“B”) based on a randomized trial of another drug.
• That trial reports an odds ratio (OR) of CHF death of 0.75.
• The probability of CHF death in the placebo (control) arm in that trial was 0.15.¹

1. Note this is slightly higher than the corresponding probability in our natural history transition probability matrix (0.1172), reflecting the fact that randomized trial participants are not necessarily representative of the entire population!

CHF Example, Continued

• An odds ratio is based on the odds of an outcome happening, which is directly related to the probability.
• Odds = \(\frac{\text{Probability of Outcome}}{1 - \text{Probability of Outcome}}\)
• Odds Ratio = \(\frac{\text{Odds of outcome in exposed}}{\text{Odds of outcome in unexposed}}\)

CHF Example, Continued

Important

Not all literature-based parameters operate on the rate scale. Some operate on the probability scale!

4. Make adjustments to \(\mathbf{P}\) as needed.

4. Make adjustments to \(\mathbf{P}\) as needed.

• Constructing our final transition probability matrix—for natural history or more specifically for for a stratgey under consideration–may require further adjustment.
• We must be careful about the scale on which these parameters apply.
• Odds ratios, relative risks, and risk differences all operate on the probability scale.

4. Make adjustments to \(\mathbf{P}\) as needed.

Statistic	Evaluates	Range	Applicable Domain
Rate	\(\frac{\# \text{events in time period}}{\text{Total time period experienced by all subjects followed}}\)	0 to \(\infty\)	Rate matrix
Hazard Ratio	\(\frac{\text{Hazard rate of outcome in exposed}}{\text{Hazard rate of outcome in unexposed}}\)	0 to \(\infty\)	Rate matrix
Probability/risk	\(\frac{\# \text{events in time period}}{\# \text{people followed for time period}}\)	0-1	Probability matrix
Odds	\(\frac{\text{Probability of Outcome}}{1 - \text{Probability of Outcome}}\)	0 to \(\infty\)	Probability matrix
Odds Ratio	\(\frac{\text{Odds of outcome in exposed}}{\text{Odds of outcome in unexposed}}\)	0 to \(\infty\)	Probability matrix
Relative Risk	\(\frac{\text{Probability of outcome in exposed}}{\text{Probablity of outcome in unexposed}}\)	0 to \(\infty\)	Probability matrix
Risk Difference	\(\text{Probability of outcome in exposed}-\text{Probablity of outcome in unexposed}\)	-1 to 1	Probability matrix

4. Make adjustments to \(\mathbf{P_s}\) as needed.

CHF Example, Continued

Let’s turn back to our CHF example.

• Suppose we wanted to model a second strategy (“B”) based on a randomized trial of another drug.
• That trial reports an odds ratio (OR) of CHF death of 0.75.
• The probability of CHF death in the placebo (control) arm in that trial was 0.15.

CHF Example, Continued

• We can convert an odds ratio to a relative risk (i.e., ratio of probabilities) if we know the baseline (unexposed) probability of the outcome, \(p_0\).
• In this case we were able to find \(p_0=0.15\) in the underlying clinical trial.
• If we didn’t have this information, we might assume its the same (0.1172) as in our underlying natural history probability matrix¹

1. Though note this is an assumption, so we should incorporate a lot of uncertainty around it in our model!

CHF Example, Continued

• We can convert an odds ratio to a relative risk (RR).¹
• \(RR = \frac{\text{Probability of outcome in exposed}}{\text{Probablity of outcome in unexposed}} = \frac{p_1}{p_0} = \frac{OR}{(1-p_0+(p_0 \times OR))}\)
• A relative risk is the ratio of the probability of the outcome in the exposed group to the probability of the outcome in the unexposed group.

1. If the incidence of the outcome is fairly rare (\(p_0<0.10\)), you will often see people simply use the OR as a RR. Source for conversion formula: Zhang and Yu (1998)

CHF Example, Continued

• \(RR = \frac{p_1}{p_0}\)
• \(p_1 = RR \times p_0\)
• \(p_{D_{CHF}} = 0.75 \times .15 = 0.1125\)

CHF Example, Continued

m_P =

	CHF	D_CHF	D_OTH
CHF	0.8772	0.1172	0.0056
D_CHF	0.0000	1.0000	0.0000
D_OTH	0.0000	0.0000	1.0000

m_P_A =

	CHF	D_CHF	D_OTH
CHF	0.8994	0.0949	0.0057
D_CHF	0.0000	1.0000	0.0000
D_OTH	0.0000	0.0000	1.0000

m_P_B =

	CHF	D_CHF	D_OTH
CHF	0.8819	0.1125	0.0056
D_CHF	0.0000	1.0000	0.0000
D_OTH	0.0000	0.0000	1.0000

Figure 1: Transition probability matrices (NH = Natural History; A = Strategy A; B = Strategy B)

Full Process

Transition Dynamics

Transition Dynamics

• We have focused on “static” transition rate/probability matrices.
• Often, transitions vary as a function of time.

Transition Dynamics

• Nonstationary rates: background mortality rate rises with age.
• Nonstationary rates: transition to disease status may vary over different ages.
• Time-dependent event transitions: rate of adverse events after disease onset is higher in first cycle, lower in subsequent cycles.

Age-Varying Background Mortality

• Useful data source is life table data.
• May be available from vital statistics division in your country.
• Also available (by region) from the UN and other organizations.

Background Mortality

• You can fit a Gompertz survival model to each age’s life table data.¹
• The Gompertz model will estimate two parameters: a shape (a) and a rate (b) parameter.
• Gompertz parameters (by gender) are shown in the table.

1. See the workshop blog posting for R code!

Background Mortality

• You can fit a Gompertz survival model to each age’s life table data.
• The Gompertz model will estimate two parameters: a shape (a) and a rate (b) parameter.
• Gompertz parameters (by gender) are shown in the table on next slide.

Background Mortality

Age	a_shape	b_rate
0	0.08386	8e-05
1	0.08630	7e-05
2	0.08644	8e-05
3	0.08653	8e-05
4	0.08658	9e-05
5	0.08661	1e-04

Background Mortality

• Gompertz fit to mortality data for two selected ages (40- and 90-year old U.S. males)

(a) 40-Year Old Men

(b) 90-Year Old Men

Figure 2: Distribution of observed deaths (grey bars) and fitted Gompertz model (red) for two ages.

Background Mortality

• Suppose we have a model of males starting at age 40.
• Each age has its own estimated shape (a) and rate (b) parameter.
• We can find the values of \(a\) and \(b\) for our 40 year old male cohort:

Background Mortality

• Suppose we have a model of males starting at age 40.
• Each age has its own estimated shape (a) and rate (b) parameter.
• We can find the values of \(a\) and \(b\) for our 40 year old male cohort:

Age	a_shape	b_rate
40	0.0908323	0.0017132

Background Mortality

• These shape and rate values are now fixed for our model
• That is, we only need these two values, as they allow us to model the remaining life expectancy of our 40-year-old male cohort as it ages through our model!

Background Mortality

• As our cohort ages through the model (40, 41, 42, …), the background death rate at each subsequent age can be calculated using a simple formula.

• Given the \(a\) and \(b\) values for a 40-year-old males, we can calcualte the death rate at any subsequent age as \[r_{e} = \frac{b}{a} (e^{at_2} - e^{at_1})\]

Background Mortality

• The Gompertz parameters for 40-year old males (US life table data) are:
  • a = 0.0908
  • b = 0.0017

Background Mortality

• At t=0 (i.e., age 40), the mortality rate for the cohort over the next year is:

• \[r_{t=0} = 0.0017934 = \frac{0.0017132}{0.0908323} (e^{0.0908323 \cdot 1} - e^{0.0908323 \cdot (0))})\]

• Using a rate-to-probability conversion formula, this corresponds to a probability of death in the next year of \[p = 1-exp(-r) = 1-exp(-0.0017934) = 0.0017918\]

Verifying Accuracy

• Using this mortality rate, we can (randomly) draw death times (years from age 40) for 1 million 40 year old males.
• The average life expectancy is the mean of this value.
• In this example, the life expectancy is 38.3 years.

Background Mortality

• This life expectancy closely matches other published data

Background Mortality

• We can now repeat this exercise for every cycle in our model.
• This gives us a different background mortality rate for every cycle.

cycle	age	a_shape	b_rate	r_mort	p_death
0	40	0.0908	0.0017	0.0018	0.0018
1	41	0.0908	0.0017	0.0020	0.0020
2	42	0.0908	0.0017	0.0022	0.0021
3	43	0.0908	0.0017	0.0024	0.0024
4	44	0.0908	0.0017	0.0026	0.0026
5	45	0.0908	0.0017	0.0028	0.0028

Background Mortality

• We can now repeat this exercise for every cycle in our model.
• This gives us a different background mortality rate for every cycle.

cycle	age	a_shape	b_rate	r_mort	p_death
50	90	0.0908	0.0017	0.1683	0.1549
51	91	0.0908	0.0017	0.1843	0.1683
52	92	0.0908	0.0017	0.2018	0.1828
53	93	0.0908	0.0017	0.2210	0.1983
54	94	0.0908	0.0017	0.2420	0.2150
55	95	0.0908	0.0017	0.2650	0.2328

Nonstationary Rates

• You may have other parameters that vary by age.
  • Disease specific mortality rate:

Source: Russell et al. (2016)

Nonstationary Rates

• You may have other parameters that vary by age.
  • Disease specific mortality rate:
  • Age-specific intervention rates/probabilities

Source: Russell et al. (2016)

Nonstationary Rates

• It is straightforward to incorporate these dynamics into a Markov model.
• Essentially, you recalculate the transition probability matrix in each cycle.
• This works long as the underlying rates/probabilities change by age.

Nonstationary Rates

• This makes the Excel a bit more complicated, because you have to incorporate visual basic code to recalculate.
• We have constructed an optional case study that draws on age-specific background mortality rates – see the workshop website for more!

Other Transition Dynamics

• Suppose, instead, that event/transition rates, costs, quality of life vary over the course of a disease or other process in our model.
• Example: cost, quality of life weight, or risk of adverse event (e.g., death) is different in first year after disease onset, and changes thereafter.
• These types of dynamics are more challenging to implement – but you can often do it!

Other Transition Dynamics

• One option is to simply build a decision tree around events that occur at higher rates shortly after disease onset, surgery, drug initiation, etc.

• The decision tree can follow people until they reach a “steady state” — then the Markov model can pick up from there.

Other Transition Dynamics

• Another option is “tunnel states” — simply expand the health state to follow people over the first few cycles of onset.

• Tunnel states are “non-markovian,” so they need to be added once you have the transition probability matrix defined.

Tunnel States: Healthy, Sick, Dead

• Let’s take a simple healthy, sick, dead model.
• Illness onset rate = 0.0314
• Background mortality rate = 0.0094
• Illness increases the risk of death by a hazard ratio of 5.45.

Tunnel States: Healthy, Sick, Dead

Transition rate matrix: \(\mathbf{R}\)

	H	S	D
H	-0.0408	0.0314	0.0094
S	0.0000	-0.0513	0.0513
D	0.0000	0.0000	0.0000

Tunnel States: Healthy, Sick, Dead

Transition probability matrix: \(\exp(\mathbf{R})\)

	H	S	D
H	0.96	0.03	0.01
S	0.00	0.95	0.05
D	0.00	0.00	1.00

Tunnel States: Healthy, Sick, Dead

• Now suppose that the probability of death from disease varies by time since disease onset.

  • 0.08 in first year
  • 0.06 in second year
  • 0.04 in third year onwards

• We can add tunnel states to our transition probability matrix to accomodate this.

Tunnel States: Healthy, Sick, Dead

• Here is the new transition probability matrix.
• Notice how the tunnel states “force” people to transition either to the next disease state if they do not die of the disease in the cycle.

	H	S1	S2	S	D
H	0.96	0.03	0.00	0.00	0.01
S1	0.00	0.00	0.92	0.00	0.08
S2	0.00	0.00	0.00	0.94	0.06
S	0.00	0.00	0.00	0.96	0.04
D	0.00	0.00	0.00	0.00	1.00

Tunnel States: Healthy, Sick, Dead

• If we run out the Markov trace for five cycles, here are the results.

• You can find a nice discussion of how to incorporate tunnel states into an Excel model here.

cycle	H	S1	S2	S	S_total	D
0	1000	0	0	0	0	0
1	960	30	0	0	30	10
2	922	29	28	0	57	22
3	885	28	26	26	80	35
4	849	27	25	50	102	49
5	815	25	24	72	121	63

Working Backwards

Working Backwards

• Suppose someone has already constructed a detailed Markov model for the decision problem you are working on.
• However, this model was constructed for another country/region/context.
• Rather than start from “scratch” with your own model, you might be able to adapt it!

Working Backwards

Essentially, this means working backwards!

Working Backwards

Essentially, this means working backwards!

Working Backwards

• We won’t go into the (very mathematical) details, but we will show you how to do this!
• Essentially, we will “solve” for what is called the generator matrix for the (existing) transition probability matrix.
• This is done through a process known as “eigenvalue decomposition.”
• Note that this is not always possible, depending on the matrix!

Working Backwards

Let’s start with the example transition probability matrix from earlier:

m_P =

	H	S	D
H	0.96	0.03	0.01
S	0.00	0.95	0.05
D	0.00	0.00	1.00

Working Backwards

To obtain the rate matrix (m_R) from an existing transition probability matrix (m_P):

V  <- eigen(m_P)$vectors
iV <- solve(V)
Ap <- iV %*% m_P %*% V
lAp <- diag(log(diag(Ap)), nrow(Ap), ncol(Ap))
m_R  <- V %*% lAp %*% iV

Working Backwards

To obtain the rate matrix (m_R) from an existing transition probability matrix (m_P):

m_R =

	H	S	D
H	-0.040822	0.0314139	0.0094081
S	0.000000	-0.0512933	0.0512933
D	0.000000	0.0000000	0.0000000

Working Backwards

• We now have the rate matrix that is the “hub” or “root” of the underlying Markov model!
• From here, we could adapt rates using country-specific parameters.
• For example, we could adapt the rate of background mortality to match our country’s/region’s mortality rate.
• We could also add additional states, or make other changes as needed.

Working Backwards

• Once we make the changes we need, we just go back to our usual process.
• “Embed” the new transition probability matrix using rate-to-probability conversion formulas, or using exponentiation.

Now You Try It!

References

Gidwani, Risha, and Louise B. Russell. 2020. “Estimating Transition Probabilities from Published Evidence: A Tutorial for Decision Modelers.” PharmacoEconomics 38 (11): 1153–64. https://doi.org/10.1007/s40273-020-00937-z.

Russell, Louise B., Sri Ram Pentakota, Cristiana Maria Toscano, Ben Cosgriff, and Anushua Sinha. 2016. “What Pertussis Mortality Rates Make Maternal Acellular Pertussis Immunization Cost-Effective in Low-and Middle-Income Countries? A Decision Analysis.” Clinical Infectious Diseases 63 (suppl_4): S227–35.

Zhang, Jun, and Kai F. Yu. 1998. “What’s the Relative Risk?A Method of Correcting the Odds Ratio in Cohort Studies of Common Outcomes.” JAMA 280 (19): 1690–91. https://doi.org/10.1001/jama.280.19.1690.