Regression Analysis — Simple, Multiple & Logistic

Regression analysis explains how an outcome changes as one or more predictors change. Use simple linear regression for one predictor and a numerical outcome, multiple linear regression for several predictors and a numerical outcome, and logistic regression for a binary outcome such as pass/fail.

That distinction solves the main search question quickly:

• Simple linear regression: one predictor, numerical outcome.
• Multiple linear regression: several predictors, numerical outcome.
• Logistic regression: binary outcome such as yes/no, pass/fail, or clicked/did not click.

After that, the real work is interpretation. A coefficient only means what you think it means if the model matches the outcome type and fits the data reasonably well.

What regression analysis does

Regression does not just draw a line through points. It builds a rule that links predictors to an expected outcome, so you can explain patterns or make predictions.

In linear regression, that rule is a straight-line model for the expected value of the outcome. In logistic regression, the model is built for probabilities, so predicted values stay between $0$ and $1$.

Simple linear regression: one predictor, numerical outcome

Simple linear regression uses one predictor $x$ and one numerical outcome $y$:

$$\hat{y} = b_0 + b_1x$$

Here $\hat{y}$ is the predicted outcome, $b_0$ is the intercept, and $b_1$ is the slope.

The slope $b_1$ tells you the predicted change in $y$ for a one-unit increase in $x$, if a straight-line pattern is a reasonable approximation over the range you care about.

Multiple linear regression: several predictors, one numerical outcome

Multiple linear regression keeps the same basic idea, but uses more than one predictor:

$$\hat{y} = b_0 + b_1x_1 + b_2x_2 + \cdots + b_px_p$$

This is useful when one predictor alone is too simple. Real outcomes often depend on several factors at the same time.

The key interpretation change is important: $b_1$ is the predicted change in $y$ for a one-unit increase in $x_1$, while the other included predictors are held fixed.

That "holding other predictors fixed" condition is what makes multiple regression different from a series of one-variable comparisons.

Logistic regression: binary outcomes and probabilities

Logistic regression is for a binary outcome, not a numerical one. If the outcome is things like admitted or not admitted, churned or stayed, or passed or failed, linear regression is usually the wrong tool.

Instead of modeling the outcome itself as a straight line, logistic regression models the log-odds of the outcome:

$$\log\left(\frac{p}{1-p}\right) = b_0 + b_1x_1 + b_2x_2 + \cdots + b_px_p$$

where $p = P(Y=1 \mid x_1, x_2, \ldots, x_p)$.

The left side is the log-odds, not the probability itself. That setup matters because probabilities must stay between $0$ and $1$: a plain straight-line model can predict impossible values like $1.2$ or $-0.1$, but logistic regression cannot.

Worked example: predicting a score vs predicting pass/fail

Suppose a teacher wants to study student performance.

If the outcome is exam score and the only predictor is study hours, a simple linear model might be

$$\hat{y} = 42 + 5x$$

If a student studies $6$ hours, the predicted score is

$$\hat{y} = 42 + 5(6) = 72$$

Here the slope says the predicted score increases by $5$ points for each extra study hour, if the linear model is a reasonable fit.

Now suppose the teacher also includes sleep hours and number of practice quizzes. A multiple regression model might be

$$\hat{y} = 20 + 4x_1 + 2x_2 + 1.5x_3$$

where $x_1$ is study hours, $x_2$ is sleep hours, and $x_3$ is practice quizzes completed.

The coefficient $4$ now has a more specific meaning: it is the predicted score change for one more study hour, holding sleep and practice quizzes fixed.

Now change the question. Instead of predicting a score, suppose the teacher wants the probability that a student passes. That makes the outcome binary, so logistic regression is the natural choice:

$$\log\left(\frac{p}{1-p}\right) = -6 + 0.8x_1 + 0.5x_2$$

If a student studies $6$ hours and sleeps $7$ hours, then

$$-6 + 0.8(6) + 0.5(7) = 2.3$$

so the predicted probability is

$$p = \frac{1}{1 + e^{-2.3}} \approx 0.91$$

This model predicts about a $91\%$ chance of passing. The exact numbers are just an example. The key idea is that when the outcome changes from a score to pass/fail, the regression family should change too.

Common mistakes in regression analysis

Using linear regression for a binary outcome

If the outcome is only $0$ or $1$, logistic regression is usually more appropriate because it is designed for probabilities. Linear regression can be used in some special settings as an approximation, but it can also produce poor probability predictions.

Treating regression as proof of causation

Regression can describe association and support prediction. It does not, by itself, prove that changing one variable causes the outcome to change.

Ignoring model conditions

A coefficient only means what you think it means if the chosen model is a reasonable fit. For linear regression, that often means checking whether a straight-line summary makes sense and whether the errors show a pattern the model missed.

Overreading multiple regression coefficients

In multiple regression, a coefficient is conditional on the other included predictors. If important variables are missing, or if predictors are strongly entangled with each other, interpretation becomes less stable.

Where regression analysis is used

Regression is used when you want to explain variation, estimate conditional relationships, or make predictions from data.

You will see it in business forecasting, medicine, social science, quality control, education, and machine learning. The exact form depends on the outcome: numerical outcomes often lead to linear models, while binary outcomes often lead to logistic models.

How to choose the right regression model

Ask these two questions first:

1. Is the outcome numerical or binary?
2. How many predictors do I want to include?

If the outcome is numerical, start with linear regression. If there is one predictor, it is simple linear regression. If there are several, it is multiple linear regression.

If the outcome is binary, start with logistic regression.

That does not guarantee the model is good, but it gets you into the right model family fast.

Try a similar problem

Take one small dataset and ask two different questions about it. First predict a numerical outcome, such as score. Then convert the outcome into a binary version, such as pass or fail. That side-by-side comparison is one of the fastest ways to make regression analysis click.