TY - JOUR
T1 - Erratum
T2 - GAMES: A Dynamic Model Development Workflow for Rigorous Characterization of Synthetic Genetic Systems (ACS Synth. Biol. (2022) 11: 2 (1009−1029) DOI: 10.1021/acssynbio.1c00528)
AU - Dray, Kate E.
AU - Muldoon, Joseph J.
AU - Mangan, Niall M.
AU - Bagheri, Neda
AU - Leonard, Joshua N.
N1 - Publisher Copyright:
© 2022 American Chemical Society.
PY - 2022/4/15
Y1 - 2022/4/15
N2 - After publication, we identified two minor mistakes in the code for the GAMES workflow. Each subtly affected our analysis of the example case study but had no effect on the GAMES workflow. Here we describe those mistakes and discuss how the resulting corrections modify the case study analysis. CORRECTION 1 There are two locations in the workflow where noise is added to each data point in the training data: to generate the parameter estimation method (PEM) evaluation data, and to calculate the threshold used for the parameter profile likelihood (PPL). In both locations, the value of the noise added to each data point is selected from a distribution centered at zero with a standard deviation equal to the standard error associated with the data point (assuming, for our case study, that the data point represents the mean value for three replicates). We set the standard deviation, σSD, for each data point to 0.05, so the standard error, σSE, is calculated with eq C1, where nreplicates is the number of replicates: {equation presented}. The original published code omitted the square root operation in eq C1, such that the distribution from which each added noise value was drawn was slightly smaller than it would have been with the correct distribution. We corrected the code in v1.0.2 and repeated the relevant PEM evaluation and PPL simulations. We noted minor changes in the results that impact the threshold of the PPL in the example study and affected the identifiability classification of one parameter for one model in the example study. This change impacted the interpretation of this one parameter and the quantitative values of the PPL threshold for all parameters. The other qualitative interpretations remain the same, and no changes were made to the GAMES workflow. With the correction, the PEM evaluation results are very similar to the previous results. The threshold used to define the PEM evaluation criterion remains the same (R2 = 0.99), and this threshold is satisfied for all models (Figure 4c model A, Figure S5a model B, Figure S6a model C, Figure S7a model C). With the correction, the main difference observed for the PPL results is that the calculated thresholds for each model are higher than in the original analysis. This change is consistent with our understanding of the PPL threshold, which is related to the extent of overfitting that is possible given a model, a training data set, and the associated measurement error. For models A (Figure 7b, Figure 8b, Figure S4a) and B (Figure 8b, Figure S5c), the increased threshold is the only substantial difference between the corrected results and previous results; all PPL shapes and parameter classifications remain the same. The corrected PPL shapes and parameter classifications for model C were also in agreement with the previous results (Figure 8b, Figure S6c), with the exception of the parameter m*, which now appears practically unidentifiable Ywhereas previously this parameter was deemed identifiable Yas the PPL reaches the threshold in the negative direction but not in the positive direction. However, this corrected result has no impact on downstream analysis because m* is still classified as identifiable for the final model (model D). The classification of m* as practically unidentifiable for model C is reasonable given that the increased PPL threshold necessitates that higher m* values be traversed when determining the PPL. As m* is a ratio between the parameters m and b, once m* reaches a sufficiently high value such thatm b, increasingm* further has no meaningful effect on the agreement between the training data and simulated data. This interpretation explains why m* does not reach the threshold in the positive direction for model C with the correction included here. The corrected results for model D are very similar to the previous results (Figure 8b, Figure S7c). All parameter classifications remain the same, and all parameters are identifiable. The qualitative shape of the PPL for m* is similar to the shape observed for m* in model C (with the correction), but in model D, the PPL crosses the threshold in both the negative and positive directions. This is reasonable because model D has fewer free parameters (four free parameters) than does model C (five free parameters), and therefore model D has a lower calculated PPL threshold, enabling the PPL for m* to cross the threshold in the positive direction. {figure presented}. CORRECTION 2 We also noted a minor mistake in the model D case study. For model D, an incorrect value for kbind (the fixed value of 1 rather than the reference value of 0.05) was used to define the reference parameter set and calculate the PPL threshold. This mistake was corrected before generating the simulation results reported here. Correcting this value led to some parameter sets having higher Χ2(Theta;fit) values than Χ2(Theta;ref) values (Figure S7c), because kbind cannot be fit to the reference parameter value for each noise realization. However, the resulting reduced model with kbind= 1 still yields very similar agreement between the training data and simulated data (Figure S7b), which shows that fixing kbind to 1 (and not to the reference value of 0.05, which would be unknown in a practical situation when the reference parameters do not exist) does not significantly affect the results. This phenomenon, in which some parameter sets have slightly higher Χ2(Theta;fit) values than Χ2(Theta;ref) values, was also observed in the original results for all models but to a lesser extent. In general, slightly negative values for Χ2(Theta;ref) . Χ2(Theta;fit) can be attributed to the optimization algorithm finding local minima (that have only slightly different Χ2 values than the global minimum) to define Χ2 (Theta;fit) for some noise realizations. CONCLUSIONS These corrections affected our interpretation of the example case study but had no effect on the GAMES workflow itself. The code used to define the case study example has been updated and annotated on GitHub: https://github.com/leonardlab/ GAMES. {figure presented}.
AB - After publication, we identified two minor mistakes in the code for the GAMES workflow. Each subtly affected our analysis of the example case study but had no effect on the GAMES workflow. Here we describe those mistakes and discuss how the resulting corrections modify the case study analysis. CORRECTION 1 There are two locations in the workflow where noise is added to each data point in the training data: to generate the parameter estimation method (PEM) evaluation data, and to calculate the threshold used for the parameter profile likelihood (PPL). In both locations, the value of the noise added to each data point is selected from a distribution centered at zero with a standard deviation equal to the standard error associated with the data point (assuming, for our case study, that the data point represents the mean value for three replicates). We set the standard deviation, σSD, for each data point to 0.05, so the standard error, σSE, is calculated with eq C1, where nreplicates is the number of replicates: {equation presented}. The original published code omitted the square root operation in eq C1, such that the distribution from which each added noise value was drawn was slightly smaller than it would have been with the correct distribution. We corrected the code in v1.0.2 and repeated the relevant PEM evaluation and PPL simulations. We noted minor changes in the results that impact the threshold of the PPL in the example study and affected the identifiability classification of one parameter for one model in the example study. This change impacted the interpretation of this one parameter and the quantitative values of the PPL threshold for all parameters. The other qualitative interpretations remain the same, and no changes were made to the GAMES workflow. With the correction, the PEM evaluation results are very similar to the previous results. The threshold used to define the PEM evaluation criterion remains the same (R2 = 0.99), and this threshold is satisfied for all models (Figure 4c model A, Figure S5a model B, Figure S6a model C, Figure S7a model C). With the correction, the main difference observed for the PPL results is that the calculated thresholds for each model are higher than in the original analysis. This change is consistent with our understanding of the PPL threshold, which is related to the extent of overfitting that is possible given a model, a training data set, and the associated measurement error. For models A (Figure 7b, Figure 8b, Figure S4a) and B (Figure 8b, Figure S5c), the increased threshold is the only substantial difference between the corrected results and previous results; all PPL shapes and parameter classifications remain the same. The corrected PPL shapes and parameter classifications for model C were also in agreement with the previous results (Figure 8b, Figure S6c), with the exception of the parameter m*, which now appears practically unidentifiable Ywhereas previously this parameter was deemed identifiable Yas the PPL reaches the threshold in the negative direction but not in the positive direction. However, this corrected result has no impact on downstream analysis because m* is still classified as identifiable for the final model (model D). The classification of m* as practically unidentifiable for model C is reasonable given that the increased PPL threshold necessitates that higher m* values be traversed when determining the PPL. As m* is a ratio between the parameters m and b, once m* reaches a sufficiently high value such thatm b, increasingm* further has no meaningful effect on the agreement between the training data and simulated data. This interpretation explains why m* does not reach the threshold in the positive direction for model C with the correction included here. The corrected results for model D are very similar to the previous results (Figure 8b, Figure S7c). All parameter classifications remain the same, and all parameters are identifiable. The qualitative shape of the PPL for m* is similar to the shape observed for m* in model C (with the correction), but in model D, the PPL crosses the threshold in both the negative and positive directions. This is reasonable because model D has fewer free parameters (four free parameters) than does model C (five free parameters), and therefore model D has a lower calculated PPL threshold, enabling the PPL for m* to cross the threshold in the positive direction. {figure presented}. CORRECTION 2 We also noted a minor mistake in the model D case study. For model D, an incorrect value for kbind (the fixed value of 1 rather than the reference value of 0.05) was used to define the reference parameter set and calculate the PPL threshold. This mistake was corrected before generating the simulation results reported here. Correcting this value led to some parameter sets having higher Χ2(Theta;fit) values than Χ2(Theta;ref) values (Figure S7c), because kbind cannot be fit to the reference parameter value for each noise realization. However, the resulting reduced model with kbind= 1 still yields very similar agreement between the training data and simulated data (Figure S7b), which shows that fixing kbind to 1 (and not to the reference value of 0.05, which would be unknown in a practical situation when the reference parameters do not exist) does not significantly affect the results. This phenomenon, in which some parameter sets have slightly higher Χ2(Theta;fit) values than Χ2(Theta;ref) values, was also observed in the original results for all models but to a lesser extent. In general, slightly negative values for Χ2(Theta;ref) . Χ2(Theta;fit) can be attributed to the optimization algorithm finding local minima (that have only slightly different Χ2 values than the global minimum) to define Χ2 (Theta;fit) for some noise realizations. CONCLUSIONS These corrections affected our interpretation of the example case study but had no effect on the GAMES workflow itself. The code used to define the case study example has been updated and annotated on GitHub: https://github.com/leonardlab/ GAMES. {figure presented}.
UR - http://www.scopus.com/inward/record.url?scp=85127415043&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85127415043&partnerID=8YFLogxK
U2 - 10.1021/acssynbio.2c00082
DO - 10.1021/acssynbio.2c00082
M3 - Comment/debate
C2 - 35271255
AN - SCOPUS:85127415043
SN - 2161-5063
VL - 11
SP - 1699
EP - 1704
JO - ACS synthetic biology
JF - ACS synthetic biology
IS - 4
ER -