ReLU, Forward and Backward

The simplest layer in a network, and the shortest possible introduction to what a backward pass is.

Write relu_backward(x, grad_out) returning (out, grad_x):

• out is the forward pass: max(x, 0), elementwise
• grad_x is the gradient with respect to x

ReLU passes a gradient straight through wherever the input was positive and blocks it entirely everywhere else. So grad_x is grad_out at the positions where x > 0, and 0 at the rest. Both outputs have x's shape.

At exactly x = 0 the function has no derivative. Use 0, which is what every framework does.