🃏Engineering Probability Unit 13 – Intro to Stochastic Processes

Key Concepts and Definitions

• Stochastic processes involve random variables that evolve over time or space
• Probability distributions describe the likelihood of different outcomes in a stochastic process
• State space represents the set of all possible values a random variable can take (discrete or continuous)
• Transition probabilities specify the likelihood of moving from one state to another
• Stationarity implies that the statistical properties of a process do not change over time
  • Strict stationarity requires all joint probability distributions to be invariant under time shifts
  • Wide-sense stationarity only requires the mean and autocovariance to be time-invariant
• Ergodicity means that the time average of a process converges to the ensemble average as the observation time increases
• Martingales are stochastic processes where the expected future value equals the current value given the past history

Types of Stochastic Processes

• Discrete-time processes have random variables defined at specific time points (integers)
• Continuous-time processes have random variables defined at any time point (real numbers)
• Markov processes exhibit the memoryless property, where the future state depends only on the current state
• Gaussian processes have random variables that follow a multivariate normal distribution
  • Brownian motion is a continuous-time Gaussian process with independent increments
• Renewal processes consist of a sequence of independent and identically distributed (i.i.d.) random variables representing inter-arrival times
• Point processes describe the occurrence of events in time or space (Poisson process, Hawkes process)
• Random walks are discrete-time processes where the next state is determined by adding a random variable to the current state
• Branching processes model population growth or extinction based on the number of offspring produced by each individual

Probability Theory Foundations

• Sample space is the set of all possible outcomes of a random experiment
• Events are subsets of the sample space, and their probabilities are assigned using a probability measure
• Random variables are functions that map outcomes from the sample space to real numbers
• Probability mass functions (PMFs) describe the probability distribution of discrete random variables
• Probability density functions (PDFs) describe the probability distribution of continuous random variables
• Cumulative distribution functions (CDFs) give the probability that a random variable is less than or equal to a specific value
• Joint probability distributions characterize the relationship between multiple random variables
• Conditional probability measures the probability of an event given that another event has occurred
• Independence means that the occurrence of one event does not affect the probability of another event
• Expectation is the average value of a random variable weighted by its probability distribution
• Variance and covariance measure the spread and linear relationship between random variables, respectively

Markov Chains

• Markov chains are discrete-time stochastic processes with the Markov property
• The Markov property states that the future state depends only on the current state, not on the past history
• Transition probability matrices describe the probabilities of moving from one state to another in a single step
• Chapman-Kolmogorov equations allow the computation of multi-step transition probabilities
• Stationary distributions represent the long-run behavior of a Markov chain
  • A stationary distribution $\pi$ satisfies $\pi P = \pi$, where $P$ is the transition probability matrix
• Irreducibility means that all states can be reached from any other state in a finite number of steps
• Periodicity refers to the return patterns of states in a Markov chain
• Ergodic Markov chains are both irreducible and aperiodic, guaranteeing the existence of a unique stationary distribution
• Absorbing states are states that, once entered, cannot be left
• Hitting times and first passage times measure the number of steps required to reach a specific state or set of states

Poisson Processes

• Poisson processes are continuous-time stochastic processes that model the occurrence of rare events
• The inter-arrival times between events in a Poisson process are independent and exponentially distributed
• The number of events in any interval follows a Poisson distribution with rate parameter $\lambda$
• The memoryless property of the exponential distribution implies that the waiting time until the next event is independent of the time since the last event
• Superposition of independent Poisson processes results in another Poisson process with a rate equal to the sum of the individual rates
• Thinning of a Poisson process involves randomly selecting events with a fixed probability, resulting in another Poisson process with a reduced rate
• Non-homogeneous Poisson processes have a time-varying rate function $\lambda(t)$
• Compound Poisson processes incorporate random jump sizes at each event occurrence
• Poisson processes are widely used to model arrival processes, failure rates, and rare events in various applications

Applications in Engineering

• Queueing theory utilizes stochastic processes to analyze waiting lines and service systems (call centers, manufacturing systems)
• Reliability engineering employs stochastic models to study the failure and repair of components and systems
  • Renewal processes can model the replacement of failed components
  • Poisson processes can represent the occurrence of failures or defects
• Stochastic control theory develops strategies to optimize the performance of systems subject to random disturbances (robotics, finance)
• Signal processing techniques, such as Kalman filtering, use stochastic models to estimate and predict the state of a system from noisy measurements
• Stochastic differential equations (SDEs) model the evolution of systems driven by random noise (Brownian motion)
• Markov decision processes (MDPs) provide a framework for sequential decision-making under uncertainty (reinforcement learning)
• Stochastic optimization methods, like simulated annealing and genetic algorithms, solve complex optimization problems with random elements
• Stochastic simulation techniques, such as Monte Carlo methods, estimate the behavior of complex systems by generating random samples

Analytical Techniques

• Moment-generating functions (MGFs) uniquely characterize probability distributions and facilitate the computation of moments
• Characteristic functions are Fourier transforms of probability distributions and provide an alternative way to study stochastic processes
• Laplace transforms are used to analyze continuous-time stochastic processes and solve differential equations
• Z-transforms are the discrete-time counterpart of Laplace transforms and are used to study discrete-time stochastic processes
• Renewal equations describe the behavior of renewal processes and can be solved using Laplace transforms or generating functions
• Kolmogorov equations (forward and backward) govern the evolution of transition probabilities in continuous-time Markov chains
• Fokker-Planck equations describe the time evolution of the probability density function for stochastic differential equations
• Martingale theory provides powerful tools for analyzing stochastic processes and deriving bounds on their behavior
• Stochastic calculus, including Itô and Stratonovich integrals, extends calculus to stochastic processes driven by Brownian motion
• Large deviation theory studies the asymptotic behavior of rare events and provides bounds on their probabilities

Practical Examples and Problem Solving

• Inventory management can be modeled using stochastic processes to optimize stock levels and minimize costs
  • Demand for products can be represented by a Poisson process
  • Markov chains can model the evolution of inventory levels over time
• Financial markets exhibit stochastic behavior, and models like the Black-Scholes equation are used for option pricing
• Weather patterns and natural phenomena, such as rainfall and earthquakes, can be described using stochastic processes
• Population dynamics in ecology can be modeled using branching processes and stochastic differential equations
• Machine learning algorithms, particularly in the context of Markov decision processes, rely on stochastic processes for sequential decision-making
• Stochastic processes are used in genetics to model the evolution of allele frequencies in populations over time
• Queuing systems, such as customer service centers and computer networks, can be analyzed using Markov chains and queueing theory
  • The number of customers in a queue can be modeled as a birth-death process
  • The M/M/1 queue assumes Poisson arrivals, exponential service times, and a single server
• Reliability of power grids and communication networks can be assessed using stochastic models for component failures and repairs
• Stochastic optimization techniques are employed in supply chain management to handle uncertainties in demand and lead times