Stochastic Processes I

Course Number: 
Hours - Lecture: 
Hours - Lab: 
Hours - Recitation: 
Hours - Total Credit: 
Typical Scheduling: 
Every fall semester

Discrete time Markov chains, Poisson processes and renewal processes. Transient and limiting behavior. Average cost and utility measures of systems. Algorithm for computing performance measures. Modeling of inventories, and flows in manufacturing and computer networks. (Also listed as ISyE 6761)


MATH 3215 or equivalent

Course Text: 

At the level of Kulkarni, Modeling and Analysis of Stochastic Systems

Topic Outline: 
  • Discrete Time Markov Chains (DTMC's)
    • Markov property
    • Transition diagrams
    • Chapman-Kolmogorov equations
    • Computation of transient probabilities
    • Classification of states
    • Recurrence and transient criteria
    • Limiting hehavior
    • Balance equation
    • Computation of stationary distribution
    • Utilities functions
    • Discounted costs and average costs
    • Applications: embedded Markov chains in queueing systems, slotted ALOHA, ...
    • Reversibility Note: conditional probabilities and conditional expectations may be reviewed as appropriate
  • Poisson Processes
    • One dimensional Poisson process
    • Event times in a Poisson process
    • Thinning and merging of Poisson processes
    • Non-homogeneous Poisson process
    • Compound Poisson process
    • Poisson process as a spatial process
  • Renewal Theory
    • The renewal equation
    • Regenerative processes
    • Poisson process as a renewal process
    • Blackwell and key renewal theorems
    • Applications to DTMC and queues