Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
16,154 characters · 0 sections · 9 citation commands
Educational Effects in Mathematics: Conditional Average Treatment Effect depending on the Number of Treatments
\@startsection{section}{1}{\z@} {2ex}{1ex}{\reset@font}{Introduction} The Academic Support Center at Japan's Kogakuin University (hereafter, the Center) offers {\em face-to-face} (F2F) personal assistance, as well as remote personal assistance, along with basic classes and a variety of on-demand educational materials available on its Learning Management System. These support options are open to all students. The university operates on a four-quarter system (1Q to 4Q) and conducts a proficiency test for all students at the time of admission in addition to regular examinations for credit-certified subjects at the end of each quarter. (Appendix (ref))
We studied the educational effect of the Center, and found that the average deviation in the 1Q Differentiation (hereafter, Diff.) regular examinations for F2F assistance users was 46.8, or 3.5 points lower than the 50.3 average of the nonusers group. At first glance, it appeared that F2F assistance had no positive educational effect on a student's regular examination performance. However, the fact that the average proficiency test deviation for F2F users was lower than that for nonusers suggests the possibility of underestimation due to bias. To address this issue, we applied the framework of causal inference, setting the use of F2F as a treatment (Appendix (ref)).
To estimate a Conditional Average Treatment Effect (CATE) rubin, meta-learners have been introduced in a binary treatment setting kunzel. Extending the binary setting, multiple and multi-level treatments models have been proposed harada, hu, lech, lin,yutas1. Meta-learners for multi-valued treatments makes it possible to identify the impact of the number of possible treatments on CATE performance acha. A more extensive discussion of related studies is presented in the Appendix (ref).
The main contributions of the present study can be summarized as follows tnagai3. (i) Using T-learner, we estimate the CATE of F2F assistance on the 1Q Diff. regular examination. (ii) Extending T-learner, we propose a new CATE estimator that depends on the number of treatments, which is learned with variables that include the number of treatments. (iii) We predict CATE performance depending on the number of F2F sessions.
\@startsection{section}{1}{\z@} {2ex}{1ex}{\reset@font}{Causal Inference Setup } Because our decision tree Breiman results showed that the F2F branches were the most common, we chose F2F as our main focus (Appendix (ref)). In this study, we adopted the potential outcome framework ney, rubin2 and T-learner kunzel. The treatment corresponds to the use of F2F in 1Q. The potential outcome corresponds to the deviation value of 1Q Diff. regular examination. Regarding input variable ${\bf X}$ for the estimators, we considered two types. $X_1$ represents the proficiency test deviation value at the time of admission, which we deal with as a covariate. $X_2$ represents the number of F2F sessions attended by the student in 1Q, which, for us, corresponds to the number of treatments. The number of treatments was included as an input variable (Appendix (ref)).
The data from $N=$ 1,389 students who took 1Q Diff. examination were analyzed. Of this group, 91 students used F2F assistance, while the remaining 1,298 students did not. To estimate CATE, we used random forests forest, implemented by Python scikit-learn.
\@startsection{section}{1}{\z@} {2ex}{1ex}{\reset@font}{Results (Input Variable ${\bf X}, {\bf x} \in \mathbb{ R}^1$)} Initially, the input variable ${\bf X}$ was the one-dimensional vector ${\bf X}=(X_1)$. We estimated the CATE of F2F as a function of $X_1=x_1$ (Appendix (ref)). Fig. (ref) shows the estimated values of deviations of 1Q Diff. with $X_1=x_1$. The value of $\hat{\mu}_0(x_1)$ is the estimated deviation value of 1Q Diff. if F2F is not used ($\textcolor{orange}{\blacktriangle}$ in Fig. (ref)), and $\hat{\mu}_1(x_1)$ is the estimated value if F2F is used ($\textcolor{blue}{\blacklozenge}$). The CATE estimator $\hat{\tau} (x_1)$ ($\textcolor{red}{\bullet}$) is defined by the difference $( \hat{\mu}_1(x_1)-\hat{\mu}_0(x_1))$. For $x_1 \le 35 $, $\hat{\tau} (x_1)$ ranges from 4 to 5. This indicates that using F2F assistance improves the regular examination deviation value in the range of 4 to 5 points. $\hat{\tau} (x_1)$ becomes negative for $37 \le x_1 \le 48$ but becomes positive again for $x_1\ge 50$, showing a value of approximately 2.
Average Treatment Effect ($ATE$) was found to be 0.48 (Appendix (ref)). This means that using F2F assistance results in an average improvement of 0.48 in the 1Q Diff. deviation values. Thus, the effect of F2F was found to be a 0.48 deviation increase rather than the 3.5 deviation decrease described in Section (ref). Average Treatment Effect on the Treated ($ATT$) was 1.00, which is compared to the results in Section (ref).
\@startsection{section}{1}{\z@} {2ex}{1ex}{\reset@font}{Estimation and Results $({\bf X}, {\bf x} \in \mathbb{R}^2)$} Multiple regression analysis was used to estimate the potential effect of the number of F2F sessions on test performance (Appendix (ref)). Accordingly, we explicitly added the number of F2F sessions $X_2$ as an input variable. The input variable ${\bf X}$ is thus the 2-dimensional vector ${\bf X}=(X_1,X_2)$ (Appendix (ref)).
Extending the definition of CATE in T-learner, we propose a new CATE estimator $\varphi (x_1,x_2 )$ with proficiency test deviation value $X_1=x_1$ and the number of F2F sessions $X_2=x_2$, defined as follows:
Here, $k$ indicates the $k$-th out of $N$ students, and $S_{x_1}$ is the set of students whose $X_1=x_1$. $|S_{x_1}|$ is the number of students in $S_{x_1}$. $X_1^{k, obs}$ and $X_2^{k, obs}$ are the $k$-th student observations of $X_1^{k}$ and $X_2^{k}$.
Fig. (ref) shows the CATE estimator $\varphi (x_1,x_2 )$ in Eq. (ref) with the number of F2F sessions (see also 3D surface plot in Appendix (ref)). For proficiency test deviation value $x_1 \leq $ 35, $\varphi (x_1, x_2 )$ decreases from $x_2$=1 time ($\textcolor{orange}{\blacktriangle}$ in Fig. (ref)) to $x_2$=2 times ($\textcolor{blue}{\blacklozenge}$). However, $\varphi (x_1, x_2 )$ increases as $x_2$ increases from $x_2$=3 times ($\textcolor{red}{\bullet}$), 5 times ($\textcolor{green}{\times}$), 10 times ($\textcolor{cyan}{+}$), and 14 times ($\textcolor{purple}{-}$). That is to say, we have $\varphi (x_1,2) < \varphi (x_1,3) < \varphi (x_1,5) \cdots$. It is predicted that three or more F2F sessions contribute to a greater increase in the deviation value of 1Q Diff. as the number of F2F sessions increases. While for $x_1 \geq 50$, $\varphi (x_1,x_2 )$ is roughly 2 if $x_2 \leq 3$ times ($\textcolor{red}{\bullet}$), $\varphi (x_1, x_2)$ becomes approximately 1.4 after 5 sessions ($\textcolor{green}{\times}$). Although a single F2F session leads to a positive $\varphi(x_1, 1)$, it is not predicted that the deviation value of 1Q Diff. increases as the number of F2F sessions increases.
Let $ATT_2$ be the average treatment effect for users with two variables (Appendix (ref)). From the relation of $\hat{\mu}_0 (X_1^{k,obs},X_2^{k,obs})=\hat{\mu}_0 (X_1^{k,obs},0)$, $ATT_2$ should be equal to $ATT$. In fact, we obtained $ATT_2=1.00$, which is equal to $ATT$=1.00.
\@startsection{section}{1}{\z@} {2ex}{1ex}{\reset@font}{Conclusion} In this study, we investigated the CATE of the F2F personal assistance offered by Kogakuin University's Academic Support Center on the deviation value of the 1Q Diff. regular examination by using T-learner. We proposed a CATE estimator that depends on the number of F2F sessions engaged in by students. We optimized the estimator with two variables, including the number of F2F sessions. Using the CATE estimator, we predicted CATE performance.
\@startsection{section}{1}{\z@} {2ex}{1ex}{\reset@font}*{Acknowledgment} This research was supported by the Japan Society for the Promotion of Science KAKENHI Grant Number JP24K06289.