The performence of CatPred model generate by reproduce_training.sh

[zunyun-Gong]

Thank you very much for sharing this excellent work and for providing such a detailed manual. I have some problems for reproduce details. I download your CatPred-DB from github and follow the instruction in reproduce_training.sh to reproduce CatPred for benchmark. But, I notice that the model performence is not as good as you reported in your paper. The command line is:
python train.py --protein_records_path kcat-random_trainvaltest.json --data_path removeX_random_kcat_data/catpred_db.kcat.random_trainval.csv --separate_test_path removeX_random_kcat_data/catpred_db.kcat.random_test.csv --separate_val_path removeX_random_kcat_data/catpred_db.kcat.random_val.csv --dataset_type regression --smiles_columns reactant_smiles --target_columns log10kcat_max --loss_function mve --seq_embed_dim 36 --seq_self_attn_nheads 6 --add_esm_feats --save_dir outdir --extra_metrics mae mse r2 --epochs 30 --batch_size 16 > capred_log
The evaluated result is Model 0 best validation rmse = 0.730834 on epoch 29 MoleculeModel( (softplus): Softplus(beta=1.0, threshold=20.0) (encoder): MPN( (encoder): ModuleList( (0): MPNEncoder( (dropout): Dropout(p=0.0, inplace=False) (act_func): ReLU() (W_i): Linear(in_features=147, out_features=300, bias=False) (W_h): Linear(in_features=300, out_features=300, bias=False) (W_o): Linear(in_features=433, out_features=300, bias=True) ) ) ) (seq_embedder): Embedding(21, 36, padding_idx=20) (rotary_embedder): RotaryEmbedding() (multihead_attn): MultiheadAttention( (out_proj): NonDynamicallyQuantizableLinear(in_features=36, out_features=36, bias=True) ) (attentive_pooler): AttentivePooling( (linear1): Linear(in_features=1316, out_features=1316, bias=True) (tanh): Tanh() (linear2): Linear(in_features=1316, out_features=1, bias=True) (softmax): Softmax(dim=1) ) (readout): Sequential( (0): Dropout(p=0.0, inplace=False) (1): Linear(in_features=1616, out_features=300, bias=True) (2): ReLU() (3): Dropout(p=0.0, inplace=False) (4): Linear(in_features=300, out_features=2, bias=True) ) ) Loading pretrained parameter "encoder.encoder.0.cached_zero_vector". Loading pretrained parameter "encoder.encoder.0.W_i.weight". Loading pretrained parameter "encoder.encoder.0.W_h.weight". Loading pretrained parameter "encoder.encoder.0.W_o.weight". Loading pretrained parameter "encoder.encoder.0.W_o.bias". Loading pretrained parameter "seq_embedder.weight". Loading pretrained parameter "rotary_embedder.freqs". Loading pretrained parameter "multihead_attn.in_proj_weight". Loading pretrained parameter "multihead_attn.in_proj_bias". Loading pretrained parameter "multihead_attn.out_proj.weight". Loading pretrained parameter "multihead_attn.out_proj.bias". Loading pretrained parameter "attentive_pooler.linear1.weight". Loading pretrained parameter "attentive_pooler.linear1.bias". Loading pretrained parameter "attentive_pooler.linear2.weight". Loading pretrained parameter "attentive_pooler.linear2.bias". Loading pretrained parameter "readout.1.weight". Loading pretrained parameter "readout.1.bias". Loading pretrained parameter "readout.4.weight". Loading pretrained parameter "readout.4.bias". Moving model to cuda Creating protein model Model 0 test rmse = 1.103250 Model 0 test mae = 0.755246 Model 0 test mse = 1.217161 Model 0 test r2 = 0.556943 Ensemble test rmse = 1.103250 Ensemble test mae = 0.755246 Ensemble test mse = 1.217161 Ensemble test r2 = 0.556943
The R2 is lower than you reported in your paper. Furthermore , the reproduced Unikp model also has this problem.
python external/UniKP/UniKP_Kcat_v2.py > unikp_log && python ./external/UniKP/parse_logs.py unikp_log unikp_log_parsed and I only change the input file just as above mentioned. The result is :
Test,R2_mean,R2_stderr,MAE_mean,MAE_stderr,p1mag_mean,p1mag_stderr
Heldout,0.5827,0.0006,0.7502,0.0005,74.96000000000001,0.16999999999999998
CLUSTER_99,0.3192,0.0017,1.0461,0.0015,61.029999999999994,0.41000000000000003
CLUSTER_80,0.2658,0.002,1.0613,0.0017,61.42999999999999,0.44
CLUSTER_60,0.2152,0.0022,1.1326,0.0021,57.8,0.49
CLUSTER_40,0.1321,0.0025,1.2235,0.0023,54.910000000000004,0.54
The performence of reproduced UniKP model in CLUSTER_** dataset are also lower than your mentioned in Supplementary Table 8.

About the input data, I only remove the entires contian "X" in Enzyme sequences. Could you provide more details about the model training and data processing? This would help me improve the model’s performance and use it correctly.

Thank you!

[Vedasheersh]

Thanks for raising this. We took another look at the reproduction details, and one important discrepancy on the CatPred side is that the command shown here appears to evaluate a single model rather than the intended 10-model ensemble. In your log, the reported Model 0 and Ensemble test metrics are identical, which means that only one model was effectively used. In the published CatPred workflow, we used an ensemble of 10 models, and the paper also notes that ensembling contributes to the final benchmark performance.

We also checked the official CatPred-DB archive linked in the README and counted the entries whose enzyme sequence contains X. For the kcat random split, removing X entries eliminates 104 rows out of 23,151 total (~0.45%). Broken down by file, this corresponds to 96 rows in kcat-random_trainval.csv, 8 rows in kcat-random_val.csv, and 8 rows in kcat-random_test.csv. That is a real data modification, but it is relatively small and by itself is unlikely to explain the full gap in performance.

For CatPred-kcat, the reproduced held-out R^2 you reported was 0.5569, whereas the published CatPred-DB benchmark value is about 0.6023. For UniKP, the held-out result is fairly close, but the gap grows on the lower-similarity out-of-distribution sets. Based on the values in Supplementary Table 8, the reported UniKP R^2 values are lower by about 0.0525 at 99%, 0.0828 at 80%, 0.0863 at 60%, and 0.1279 at 40% sequence identity cutoff.

So our current view is that the main CatPred discrepancy is most likely the mismatch in reproduction procedure, especially the missing 10-model ensemble, and not just the removal of X entries. If you are still interested in reproducing the benchmark closely, we would be happy to keep discussing it. In that case, the most helpful next step would be to rerun the CatPred benchmark using the repository reproduction script as-is, without modifying the dataset, and then compare those outputs directly.

[zunyun-Gong]

Thank you for your quick reply! As you suggested, after applying the 10-model ensemble, the performance became comparable to what your team reported, with the following results: Ensemble test rmse = 1.036709
Ensemble test mae = 0.702979
Ensemble test mse = 1.074766
Ensemble test r2 = 0.608776
It is a very talent design!

As for the UniKP baseline model, I followed the repository’s reproduction script as provided, yet the performance remains lower than expected. I am unsure whether I may have missed something, but I do not believe this affects the main conclusions of your paper.

Moreover, I noticed that your reactant_smiles include all reaction details. I wonder whether it is important to include all reaction substrates when reproducing the CatPred model.
In my case, when I reproduced CatPred on my dataset (all wild-type), the performance was somewhat weaker (R² = 0.53), and I did not observe improved generative ability compared to UniKP. I am not sure whether this might be due to differences in data processing or implementation, and would appreciate any guidance.

Thank you for your time and assistance!