Neural Network — How Artificial Neural Networks Work

A neural network is a mathematical model that turns inputs into outputs by passing numbers through layers of simple operations. Each layer takes the previous values, forms weighted sums, adds biases, applies a nonlinear activation function, and passes the result forward.

That description sounds abstract, but the core idea is small: a network learns many adjustable weights so that useful patterns in the input lead to useful predictions at the output.

What A Neural Network Is

For one neuron with inputs $x_1, x_2, \dots, x_n$, the basic computation is

$$z = w_1x_1 + w_2x_2 + \dots + w_nx_n + b$$

followed by an activation:

$$a = g(z)$$

Here $w_1, \dots, w_n$ are weights, $b$ is a bias, and $g$ is an activation function such as ReLU, sigmoid, or tanh.

A full feedforward neural network repeats that pattern across layers. In compact form, one layer is often written as

$$a^{(l)} = g\!\left(W^{(l)} a^{(l-1)} + b^{(l)}\right)$$

where $a^{(l-1)}$ is the previous layer's output.

The Intuition That Usually Makes It Click

Each neuron asks a weighted question about the input it sees. Large positive weights make some features matter more. Negative weights can push against a pattern. The bias shifts the threshold. The activation function then decides how strongly that neuron should respond.

Stacking layers lets the network build features in stages. Early layers detect simple patterns. Later layers combine those into more useful internal signals for the final task.

This is why neural networks are more than "many formulas at once." They are compositions of simple functions, and the composition is what gives them flexibility.

One Worked Example

Consider a tiny network with two inputs, one hidden layer, and one output. Let the input be

$$x = \begin{bmatrix} 2 \\ 1 \end{bmatrix}$$

Suppose the hidden layer has two neurons and uses ReLU, where

$$\operatorname{ReLU}(z) = \max(0, z)$$

Take these hidden-layer computations:

$$z_1 = 1 \cdot 2 + (-1) \cdot 1 + 0 = 1$$ $$h_1 = \operatorname{ReLU}(z_1) = 1$$ $$z_2 = 0.5 \cdot 2 + 0.5 \cdot 1 - 1 = 0.5$$ $$h_2 = \operatorname{ReLU}(z_2) = 0.5$$

Now send those hidden values to the output neuron:

$$s = 2h_1 - h_2 = 2(1) - 0.5 = 1.5$$

If the rule is "predict class 1 when $s > 0$," this input is classified as class 1.

The important point is not the specific numbers. It is the structure:

1. take inputs
2. form weighted sums
3. apply nonlinear activations
4. repeat
5. read the final score

That is a neural network doing a forward pass.

How A Neural Network Learns

Using a network is one problem. Training it is another.

In standard supervised learning, the network first makes a prediction. A loss function then measures how far that prediction is from the target. Gradient-based training computes how the loss changes with respect to each weight and bias, then updates them to reduce the loss.

In modern practice, this usually means backpropagation plus gradient descent or a related optimizer. This setup relies on a model and loss that are differentiable or at least piecewise differentiable enough for gradient methods to work.

The short version is:

$$\text{prediction} \to \text{loss} \to \text{gradients} \to \text{parameter update}$$

Over many examples, the weights shift toward patterns that help the task.

Common Mistakes

Thinking More Layers Automatically Mean Better Results

They do not. More layers increase capacity, but they also make optimization, data requirements, and overfitting control more demanding.

Forgetting Why Nonlinearity Matters

If every layer is only linear, the whole network is still just one linear map. The activation functions are what let deep networks represent more complex relationships.

Treating The Output As Guaranteed Certainty

A network output is only as useful as the model, data, and training setup behind it. A high score is not the same thing as a proof.

Ignoring The Input Representation

Networks do not learn from raw meaning. They learn from the numerical representation they receive. If the inputs are poor, inconsistent, or missing important structure, the network's performance will usually suffer.

When Neural Networks Are Used

Neural networks are used when the relationship between input and output is complicated enough that hand-written rules are brittle or incomplete. Common settings include image recognition, speech, language modeling, recommendation systems, and some forecasting tasks.

They are not automatically the best choice for every problem. On small, structured datasets, simpler models can be easier to train, easier to interpret, and sometimes just as effective.

A Good Mental Model

Think of a neural network as a layered function with many adjustable knobs. The forward pass turns one input into one output. Training changes the knobs so that future outputs become more useful for the task.

That is the cleanest way to hold both ideas at once: neural networks compute by composition, and they learn by adjusting parameters to reduce error.

Try Your Own Version

Keep the same tiny network, but change the input from $(2, 1)$ to $(0, 3)$. Recompute $z_1$, $z_2$, $h_1$, $h_2$, and the final score $s$. Then change one weight and see which part of the output moves. That small exercise makes the forward-pass idea much more concrete than memorizing definitions alone.