# The Simple Regression Model

13 Apr

The simple regression model can be used to study the relationship between two variables.For reasons we will see,the simple regression model has limitations as a general tool for empirical analysis. Nevertheless, it is sometimes appropriate as an empirical tool. Learning how to interpret the simple regression model is good practice for studying multiple regression.

Much of applied econometric analysis begins with the following premise:y and x are two variables,representing some population, and we are interested in “explaining y in terms of x”, or in “studying how y varies with change of x”.

In writing down a model that will “explain y in terms of x”, we must confront three issues.First, since there is never an exact relationship between two variables, how do we allow for other factors to affect y? Second, what is the functional relationship between y and x? And, third , how can we be sure we are capturing a ceteris paribus relationship between y and x(if that is a desired goal)?

We can resolve these ambiguities by writing down an equation relating y to x. A simple equation is

(equation 2.1)

This equation defines the simple linear regression model. It is also called the two-variable linear regression model or bivariate linear regression model because it relates the two variables x and y.

When related to this equation  the variable y and x have several different names used interchangeably,as follows.y is called the dependent variable, the explained variable, the response variable, the predicted variable, or the regressand. x is called the independent variable, the explanatory variable, the control variable,the predictor variable,the regressor.(The term covariate is also used for x).

The variable u, called the error term or disturbance in the relationship,represents factors other than x that affect y. A simple regression analysis effectively treats all factors affecting y other than x as being unobserved. If the other factors in u are held fixed, so that the change in u is zero,d(u)=0(d=delta),then x has a linear effect on y:

(equation 2.2)

Thus, the change in y is simply β1 multiply bu the change in x.This means that β1 is the slope parameter in the relationship between y and x holding the other factors in u fixed; it is of primary interest in applied economics.The intercept parameter β0 also has its use ,although it is rarely central to analysis.

The linearity of (2.1) implies that one-unit change in x has the same effect on y,regardless of the initial value of x. This is unrealistic for many economic applications.For example,in the wage-education example, we might want to allow for increasing returns: the next year of education has a larger effect on wages than did the previous year.

The most difficult issue to address is whether model (2.1) really allows us to draw ceteris paribus conclusions about how x affects y. We just saw in equation (2.2) that ( β1 does measure the effect of x on y, holding all other factors (in u) fixed. Is this the end of the causality issue? Unfortunately, no. How can we hope to learn in general about the ceteris paribus effect of x on y, holding other factors fixed, when we are ignoring all those other factors?

We are only able to get reliable estimators of β0 and β1 from a random sample of data when we make an assumption restricting how the unobservable u is related to the explanatory variable x. Without such a restriction, we will not be able to estimate the ceteris paribus effect, β1. Because u and x are random variables, we need a concept grounded in probability.

Before we state the key assumption about how x and u are related, there is one assump­tion about u that we can always make. As long as the intercept β0 is included in the equa­tion, nothing is lost by assuming that the average value of u in the population is zero.

Mathematically,

(equation 2.3)

Importantly, assume (2.3) says nothing about the relationship between u and x but sim­ply makes a statement about the distribution of the unobservables in the population..

We now turn to the crucial assumption regarding how u and x are related. A natural measure of the association between two random variables is the correlation coefficient.  If u and x are uncorrelated, then, as ran­dom variables, they are not linearly related. Assuming that u and x are uncorrelated goes a long way toward defining the sense in which u and x should be unrelated in equation (2.1). But it does not go far enough, because correlation measures only linear depen­dence between u and x. Correlation has a somewhat counterintuitive feature: it is possi­ble for u to be uncorrelated with x while being correlated with functions of x, such as x2.  This possibility is not acceptable for most regression purposes, as it causes problems for interpretating the model and for deriving statistical properties. A better assumption involves the expected value of u given x.

Because u and x are random variables, we can define the conditional distribution of u given any value of x. In particular, for any x, we can obtain the expected (or average) value of u for that slice of the population described by the value of x. The crucial assumption is that the average value of u does not depend on the value of x. We can write this as

(equation 2.4)

where the second equality follows from (2.3). The first equality in equation (2.4) is the new assumption, called the zero conditional mean assumption. It says that, for any given value of x, the average of the unobservables is the same and therefore must equal the average value of u in the entire population.

Assumption 2.4 gives β1 another interpretation that is often useful. Taking the expected value of (2.1) conditional on x and using E(u|x) = 0 gives

(equation 2.5)

Equation (2.5) shows that the population regression function(PRF), E(y|x), is a lin­ear function of x.

The linearity means that a one-unit increase in x changes the expected value of y by the amount ft. For any given value of x, the distribution of y is cen­tered about E(y|x).

When (2.4) is true, it is useful to break y into two components. The piece β0 + β1x is sometimes called the systematic part of y—that is, the part of y explained by x—and u is called the unsystematic part, or the part of y not explained by x.