2.2 Forward propagation and computation graphs

Forward propagation is the backbone of neural networks, pushing data from input to output through layers. It applies weights, biases, and activation functions to transform inputs into predictions, allowing networks to make inferences based on learned parameters.

Computation graphs visually represent the flow of data in neural networks. These graphs simplify complex architectures, aiding understanding and facilitating efficient implementation of backpropagation. The process involves initializing inputs, computing weighted sums, and applying activation functions layer by layer.

Understanding Forward Propagation

Forward propagation in neural networks

• Moves information through neural network from input to output layer by layer, left to right
• Applies weights, biases, and activation functions to transform input data into predictions
• Allows network to make inferences based on learned parameters
• Input layer receives initial data, hidden layers process and transform information, output layer produces final predictions

Computation graphs for data flow

• Visually represent mathematical operations in neural network with nodes (variables or operations) and edges (data flow)
• Input nodes represent data or parameters, operation nodes perform math (addition, multiplication), output nodes show results
• Simplify complex architectures, aid understanding of information flow, facilitate efficient backpropagation implementation

Output calculation with forward propagation

1. Initialize input layer with given data

2. For each subsequent layer:

   • Compute weighted sum: $z = Wx + b$
   • Apply activation function: $a = f(z)$

3. Repeat until reaching output layer

• Common activation functions: ReLU $f(x) = max(0, x)$, Sigmoid $f(x) = 1 / (1 + e^{-x})$, Tanh $f(x) = (e^x - e^{-x}) / (e^x + e^{-x})$
• Use matrix multiplication for weight-input products and element-wise operations for activation functions

Computational complexity of forward propagation

• Affected by number of layers, neurons per layer, and operation types (matrix multiplications, activation functions)
• Time complexity for fully connected network with L layers and n neurons per layer: O(L * n^2) for matrix multiplications, O(L * n) for activation functions
• Total time complexity: O(L * n^2)
• Space complexity considers storage for weights, biases, and intermediate activations
• Optimization techniques include sparse matrix operations and parallelization across layers or neurons