Andrew Hu ECON 132 Project 2

Randomized Training Program

Introduction
This papers goal is to replicate results of an experiment on vocational training in Colombia. A group of unemployed youth were selected to be invited to attend training. Two-thirds were randomly chosen to be given the treatment while one-third were kept as control observations. In order to replicate the results, it is first necessary to examine whether any baseline characteristics (designed to reduce variance of coefficients) are correlated with allocation to treatment. Then, we will proceed to examine the effect of the treatment on earnings, salary, marriage, and other characteristics of these youths . It will be important to examine how randomizing by clusters (training centers) can be extrapolated to randomizing over all. Finally, we will conclude with some comments on issues with scaling this program on the basis of the results.

Baseline Characteristics and Allocation to Treatment

Below, we have displayed a table of the means of selected baseline characteristics between the treatment and non-treatment group pre-treatment and the associated t-values and p-values regarding their differences. Variable Name age_lb dmarried_lb empl_04 salary_04 profit_04 tenure_04 days_04 hours_04 contract_04 dformal_04 educ_lb Control Mean 21.23 0.2015 0.5005 99662.55 23247.91 3.201 12.04 24.88 0.0846 0.0845 9.9932 Treatment Mean 21.08 0.1978 0.5352 105722 22499 4.033 12.75 26.78 0.0838 0.0970 10.172 T-value -2.39 -1.25 2.19 1.22 -0.29 2.81 1.78 2.09 -0.08 1.37 3.13 P-value 0.017* 0.213 0.029* 0.222 0.769 0.005* 0.076 0.037* 0.934 0.171 0.002*

From this table, it is clear that not all baseline characteristics are balanced between the control and the treatment samples. Namely, there is a significant difference in the baseline means of control and treatment samples for age, employment, tenure, hours worked, and education. This means we will have to be careful in our selection of baseline characteristics. It is necessary to construct a joint F-test to see whether the baseline characteristics can jointly be considered to predict selection into treatment. Running the following regression: reg select age_lb dmarried_lb empl_04 salary_04 profit_04 tenure_04 days_04 hours_04 contract_04 dformal_04 educ_lb, robust provides a F-statistics of F(11, 3652) = 3.19, which occurs with probability 0.0003. This is significant so we reject the null that the baseline characters do not jointly predict selection into the

Andrew Hu ECON 132 Project 2 treatment. This does not bode well for our experiment so it is necessary to attempt some adjustments. If we stratify by gender, that is if we restrict our dataset to women, the regression reg select age_lb dmarried_lb empl_04 salary_04 profit_04 tenure_04 days_04 hours_04 contract_04 dformal_04 educ_lb if dwomen == 1, robust, gives an F-statistics of F(11, 2001) = 1.60 with p-value 0.0921 which is no longer significant. Thus, the experiment is jointly balanced in baseline characteristics with women. However, the rest of the analysis contains the entire dataset. Thus, baseline characteristics used in the following regressions will only contain those that were balanced in the initial t-tests. We use another joint F-test to test the null that these can be jointly considered 0 and that the experiment is well-balanced in regards to these characteristics. The regression: reg select dmarried_lb salary_04 profit_04 days_04 contract_04 dformal_04, robust gives an F-statistic of F(6, 3949) = 1.62 with p-value 0.1364 so we accept the null of these characteristics being jointly 0 and proceed to use them as baseline characteristics in the rest of our analysis.

Effect of Treatment
We examine the effect of treatment on various characteristics . This will be done by regressing post treatment measurements on selection into treatment and other baseline characteristics. For binary characteristics (such as marriage or employment), a logistic regression shall be used. Otherwise, standard regression will be used. Variable Name Coefficient on Z-value P-value Notes Select (SE of coeff) dmarried_s 0.0175 (0.105) 0.17 0.869 logit empl_06 0.178 (0.082) 2.16 0.031* logit salary_06 36802.64 (7254.06) 5.07 0.000* profit_06 -3003.30 (2861.94) -1.05 0.294 tenure_06 -2.108 (0.482) -4.37 0.000* days_06 0.854 (0.402) 2.12 0.034* hours_06 1.113 (0.945) 1.18 0.239 contract_06 0.458 (0.082) 5.61 0.000* logit dformal_06 0.371 (0.080) 4.67 0.000* logit educ_s 0.187 (0.058) 3.23 0.001* First consider employment by observing empl_06. As employment was a binary variable, it was necessary to run a logistic regression. It is impossible to determine the level of the effect of treatment on employment. Rather, we must interpret the coefficient of treatment at the margin. Suppose there was an individual with a 0.10 (10%) chance of being employed in the post treatment period. Then, the treatment effect increases this 0.10 chance of being employed by 0.1(1-0.1)0.178=0.1*0.9*0.178 = 0.016 or 1.6 percentage points. Now suppose there was an individual with a 0.50 chance of being employed in the post treatment period. Then, the treatment effect increases this by 0.5(1-0.5)0.178 = 0.0445 or 4.5 percentage points. Notice that because of the p-value of 0.031 < 0.05, this is significant and treatment had a significant effect on an individual being employed in the post treatment period.

Andrew Hu ECON 132 Project 2 Now let us consider the treatment effect on earnings. For this, we shall observe salary_06. This was run with a standard regression so we interpret the coefficient as such: in the post-treatment time period, those that participated in the treatment had salaries on average 36800 real Colombian pesos more than those that were in the control group. Note also that given a p-value of 0.000* for this coefficient, this extra salary given to those who participated in treatment is a significant estimate. We take a break from our analysis to note that this bodes well for our experiment. According to the analysis so far, the administered treatment had a significant effect on both an individuals probability of being employed and having a high salary. It is necessary not to be too ecstatic in our results though. First, remember there are other measures of employment and salary that could better reflect an individuals standard of living. Second, the cost and scalability of this treatment could prove a hindrance to longterm implementation. Third, we exhibited balance issues previously with this randomized experiment and thus, our results may contain composition bias. Now, we consider the effect of treatment on marriage. We do this by considering the regression for dmarried_s. Similar to employment, this was also a logistic regression and we cannot interpret the level of the effect of treatment; instead, the coefficient is interpreted at the margin. Suppose an individual had a 0.1 chance of being married in the post treatment period during the pre-treatment period. As a result of the treatment, this individuals chance of being married in the post treatment period went up 0.1*0.9*0.0175 = 0.001575 or 0.1 percentage points. Similarly, if an individual had a 0.5 chance of being married in the post treatment period, the treatment increased this by 0.25*0.0175 = 0.004375 or 0.4 percentage points. As intuition suggests, this does not seem like much. This is corroborated by the pvalue of 0.869 in this regression, which is not significant. This says that treatment did not have a significant effect on an individual being married in the post-treatment period. This is somewhat expected as I doubt this treatment involved facial reconstruction surgery and attitude therapy. All jokes aside, it is natural to see that a treatment that revolves around vocational training would have little effect on getting married if we fix all else. Now we consider the effect of treatment on formal employment or employment through formal channels, or the variable dformal_06. Being another logistic regression, this must be interpreted at the margin. Given an individual with a 0.1 chance of being employed in a formal channel in the posttreatment period during pre-treatment period, the treatment had an additional effect of 0.03339 or an extra 3.3 percentage points to an individuals chance of being employed through a formal channel in the post treatment period. Given an individual with a 0.5 chance of being employed, the treatment provides an additional effect of 9.25 percentage points. The p-value of this coefficient of 0.000* demonstrates that treatment has a significant effect of being employed through formal channels. This is expected as by being an individual in the treatment, that individual has been either put into or placed more prominently in the system, making employment through formal channels easier. We end on the note that dformal_06 and contract_06 have similar results as a contractual employment is highly correlated with formal employment. We also wish to give a more in-depth examination of the effects of the treatment on employment. We do so by examining the variable days_06. Being a standard regression, we interpret the coefficient on

Andrew Hu ECON 132 Project 2 selected as such: treated individuals worked on average an extra 0.854 days per month than individuals that were not treated in the post-treatment period. The p-value of this coefficient informs us that this value is significant and indeed, treatment did have a significant effect on the impact on days worked per month and by extension, employment. Note that we did no choose hours_06 as the variable to be examined here. This is because days is a measured on a month by month basis while hours was measured on a week by week basis. With a shorter measurement interval such as weeks, the data is much more susceptible to seasonal (not actual seasons) fluctuations then with a larger measurement interval such as months. This also could explain how the regression on days_06 demonstrated a significant effect of treatment while hours_06 did not. It is important to be mindful that results and analysis determined in this section and here on out were the result of analysis performed on a somewhat unbalanced experiment. As a result, there is possibly composition bias or selection bias present in our estimation of treatment effect on the treated. Explicitly, we are not estimating the effect of treatment on the treated. We are estimating that effect with some composition bias which could bias our interpretation of the estimates and significance.

Randomization within the Treatment Centers

Since training necessarily took place within a treatment center, randomization occurred at the treatment center level. However, if the probability induced by randomization is the same across treatment centers, this is equivalent to randomizing overall. We wish to formally test this hypothesis. Below are two regressions on the effect of treatment on employment with center fixed effects and without. Variable Name Coefficient on Select (SE of coeff) No FE 0.0175 (0.105) 0.178 (0.082) 36802.64 (7254.06) -3003.30 (2861.94) -2.108 (0.482) 0.854 (0.402) 1.113 (0.945) 0.458 (0.082) 0.371 (0.080) 0.187 (0.058) P-value No FE Coefficient on Select (SE of coeff) FE 0.055 (0.108) 0.156 (0.086) 33558.78 (7365) -2245.22 (2884) -2.161 (0.487) 0.645 (0.406) 0.834 (0.953) 0.440 (0.085) 0.343 (0.082) 0.202 (0.0560) P-Value - FE

dmarried_s empl_06 salary_06 profit_06 tenure_06 days_06 hours_06 contract_06 dformal_06 educ_s

0.869 0.031* 0.000* 0.294 0.000* 0.034* 0.239 0.000* 0.000* 0.001*

0.610 0.069 0.000* 0.307 0.000* 0.112 0.381 0.000* 0.000* 0.000*

Comparing these to the initial regression results when there was no accounting for center fixed effects, the results do not appear too different (means for the change in control and treatment are approximately equal). We see two cases where a characteristic went from being significant to not significant when adding fixed center effects. These are empl_06 and days_06. However, examining the actual means and standard errors reveals that this change was a rather marginal one when adding

Andrew Hu ECON 132 Project 2 marginal effects. Furthermore, the baseline versions of these variables (empl_04, days_04) demonstrated balance issues earlier. This means it is possible to conclude that by randomizing within centers, a similar result has been created had randomization occurred overall and center fixed effects do not drastically alter the analysis. It is necessary to consider whether we need to take into account clusters when computing standard errors. We know that vCLU = vOLS(1 + (M-1)xu) for any particular regressor (particularly select in this case), where vOLS is the variance if we disregard the problem. From the previous discussion, we know that treatment was randomized within the treatment center, or clusters. This means the intracluster correlation of treatment, or the regressor x, was 0. Thus, we get vCLU = vOLS and it is not necessary to take into account clustering when computing standard errors (given randomization within each cluster).

Better Sample Sizes

It is prudent to consider the question of how this experiment could be bettered if there were no cost to the treatment; namely, if it were possible to set the sample size to whatever is desired. Let us state these following parameters: We wish to detect a 3 percentage point increase in employment due to our treatment with a test of size 5% and power 80%. Since employment is a binary variable, we shall be conservative in our assumption and write 12 = 02 = P(1-P) = 0.5(1-0.5) = 0.25, maximizing the possible variance of employment (which is binary). By doing so, we will only overestimate the required sample size to achieve our goal, rather than underestimate. The project suggests a 2-1 split in treatment to control but the data suggests a 1-1 split in treatment and control so we set N1=N0=N. Then, ( ) . Thus, we wish to solve ( This reduces to ( ) ( ) ( ) ( ) ) ( )

( )

( )

Since the second term is approximately 0, this gives ( ) ( )

The total sample size is N1+N0= 2N = 8710. Now, we adjust parameters to a power of 90%.

Andrew Hu ECON 132 Project 2 ( ) ( )

The total sample size is N1+N0= 2N = 11664. Now, we return power to 80% but change the size to 1%. The critical value changes from 1.96 to 2.575. ( ) ( )

The total sample size is N1+N0= 2N = 12958. The progression of these necessary sample sizes make intuitive sense. As we wish to increase the power of our test (fixing all other parameters), we require a larger sample. Similarly, as we wish to increase the size of our test (fixing all other parameters), we also require a larger sample. We note that the sample size presented in the data is much smaller than the required numbers noted here. This could explain problems with the previous analysis OR that our assumption of 12 = 02 = P(1-P) = 0.5(1-0.5) = 0.25 was indeed too conservative.

Scalability of Program
It is clear from the results that treatment in this program led to at least equal if not higher employment numbers along formal channels, increased earnings, and increased productivity and access to jobs. This program, if it were cost-free, would be a very easy choice to scale up to encompass a larger population as it will have a non-negative. However, it is not cost-free. The experiment was demonstrated to be not balanced, suggesting some bias in the results. To clarify the benefits of this program and make a costbenefit analysis, it would be necessary to explore and understand the causes of this bias. We did observe that the selection to treatment was balanced for women so perhaps a paper detailing effects of the program for women could be scaled up to a treatment offered for women. It is also necessary to ensure that these gains from treatment last into the long-term as we do not have data regarding that. Given that these facts are true, this program is an excellent candidate to scale up on to a larger level. Having said that, there are caveats regarding our initial results. It is possible that the program encouraged a spurt of new job creation by increasing supply of productive workers, thus confounding our positive gains from treatment somewhat. Furthermore, the program could have also stolen some gains from nonparticipants; that is to say some of the benefit to participants could have come at the cost of non-participants. In both of these cases, a scaled up version of the program implemented in the long-term could have much less significant positive benefits to those treated or even introduce a

Andrew Hu ECON 132 Project 2 negative effect to the economy by shocking it. Thus, it is necessary to obtain a deeper understanding of the operations of the labor market and the economy in general before implementing a scaled up program.