EE6150: Stochastic Modeling and the Theory of Queues

Course information

Course Content

  • Poisson processes

  • Discrete-time Markov chains (DTMCs)

  • Continuous-time Markov chains (CTMCs)

  • Queueing models

  • Introduction to Markov reward processes (if time permits)

Grading

  • Mid-term: 25%

  • Final exam: 50%

  • Assignments: 25%

Textbooks

  • Vidyadhar Kulkarni, Modeling and analysis of stochastic systems.Third edition, Crc Press, 2016.

  • Geoffrey Grimmett and David Stirzaker, Probability and random processes, Oxford university press, 2020.

Additional References

  • Robert Gallager, Stochastic processes: theory for applications. Cambridge University Press, 2013.

Lecture Notes

Schedule

Lecture number Topics Covered Section reference
(Kulkarni's book)
Lecture 1 Exponential distribution 5.1
Lecture 2 Poisson processes: Definition
Shifted Poisson processes 		5.2
Lecture 3 Poisson processes: Finite dimensional distributions
Alternate characterization		 5.2
Lecture 4 Event times in a Poisson process 5.3
Lecture 5 Superposition and splitting of Poisson processes 5.4
Lecture 6 Non-homogeneous Poisson processes 5.5
Lecture 7 Problem solving session
Lecture 8 Discrete time Markov chain (DTMC): Definition, Examples
Chapman-Kolmogorov equations		 2.1, 2.2, 2.4
Lecture 9 DTMC: Occupancy times 2.5
Lecture 10 Computing matrix powers (diagonalizable case)
Application to DTMCs
First passage time - definition		 2.6, 3.1
Lecture 11 DTMC - First passage times
CDF, absorption probabilities		 3.2, 3.3
Lecture 12 DTMC - First passage time
Analysis of random walks
Expectation of the first passage time		 3.3, 3.4
Lecture 13 DTMC - Expectation of the first passage time 3.4
Lecture 14 DTMCs: Limiting behavior
A study using examples 		4.1
Lecture 15 DTMC - limiting behavior: Classification of states
 4.2
Lecture 16 Problem solving session
Lecture 17 Communicating classes, recurrence and transience 4.3
Lecture 18 Periodicity 4.3
Lecture 19 Determining transience/recurrence: infinite DTMCs 4.4
Lecture 20 Limiting behavior: transient DTMC 4.5
Lecture 21 Stationary distributions: Construction Grimmett-Stirzaker - Sec 6.4
Lecture 22 Stationary distributions: Existence Grimmett-Stirzaker - Sec 6.4
Lecture 23 Convergence to stationary distribution: Aperiodic case Grimmett-Stirzaker - Sec 6.4
Lecture 24 Convergence to stationary distribution: Cesaro sense Grimmett-Stirzaker - Sec 6.4
Lecture 25 Examples: limiting behavior 4.6
Lecture 26 DTMCs with costs 4.8
Lecture 27 Continuous time Markov chain (CTMC): Definition, alternate characterization 6.1
Lecture 28 CTMCs: Examples 6.2
Lecture 29 CTMC: Transient behavior, Chapman-Kolmogorov equations
Forward and backward equations		 6.4
Lecture 30 Examples: Applications of forward equations
Occupancy times 		6.4, 6.5
Lecture 31 Computation of P(t): Finite/infinite state space 6.6, 6.7
Lecture 32 Absorption probabilities, First-passage times 6.8
Lecture 33 CTMCs: Limiting behavior 6.9-6.11