top 30 models (across **all** models - 30240 parameter combinations) ```r param_idx fc_likert_corr_train fc_likert_corr_test cost_per_m_post_train cost_per_m_post_test max_models disagreement_threshold aggregation_method perf_metric model_selection_method cost_bin model_cost <int> <num> <num> <num> <num> <int> <num> <char> <char> <char> <int> <char> 1: 17628 -0.7137166 -0.7445546 133969.98868 495265.68646 6 0.40 combined perf_min random 7 0.00500143387096774,0.479745967741935 2: 15940 -0.7095018 -0.6821161 211015.95122 28362.46249 6 0.20 combined perf_geometric random 7 0.00500143387096774,0.479745967741935 3: 9135 -0.7061135 -0.6673081 73.40478 63.99031 4 0.10 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 4: 6195 -0.7034021 -0.6727010 73.40478 63.99031 3 0.10 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 5: 19312 -0.7018884 -0.7434469 29826.46078 181586.42033 7 0.25 combined corr_only random 7 0.00500143387096774,0.479745967741935 6: 19724 -0.7018091 -0.7417299 29826.46078 496082.57049 7 0.30 combined perf_mean random 7 0.00500143387096774,0.479745967741935 7: 10866 -0.7017509 -0.6830394 1066.57768 1036.03988 4 0.30 combined perf_mean best_to_worst 5 0.000425496774193549,0.000610483870967742 8: 10862 -0.7017393 -0.6830051 1066.57768 1036.03988 4 0.30 combined perf_geometric best_to_worst 5 0.000425496774193549,0.000610483870967742 9: 10873 -0.7016459 -0.6811477 1066.57768 1036.03988 4 0.30 combined corr_only worst_to_best 5 0.000425496774193549,0.000610483870967742 10: 10870 -0.7015151 -0.6823156 1066.57768 1036.03988 4 0.30 combined perf_min best_to_worst 5 0.000425496774193549,0.000610483870967742 11: 18040 -0.7001072 -0.7429203 498969.28141 53902.39085 7 0.10 combined perf_geometric random 7 0.00500143387096774,0.479745967741935 12: 13806 -0.6995211 -0.6876801 1066.57768 1036.03988 5 0.30 combined perf_mean best_to_worst 5 0.000425496774193549,0.000610483870967742 13: 13802 -0.6995181 -0.6876499 1066.57768 1036.03988 5 0.30 combined perf_geometric best_to_worst 5 0.000425496774193549,0.000610483870967742 14: 13810 -0.6994923 -0.6870330 1066.57768 1036.03988 5 0.30 combined perf_min best_to_worst 5 0.000425496774193549,0.000610483870967742 15: 13813 -0.6994749 -0.6864489 1066.57768 1036.03988 5 0.30 combined corr_only worst_to_best 5 0.000425496774193549,0.000610483870967742 16: 10488 -0.6994024 -0.7674427 29826.46078 493666.67056 4 0.25 combined perf_min random 7 0.00500143387096774,0.479745967741935 17: 9652 -0.6991175 -0.6616942 498549.65927 497630.51196 4 0.15 combined corr_only random 7 0.00500143387096774,0.479745967741935 18: 10908 -0.6984482 -0.7284636 29826.46078 16573.55943 4 0.30 combined perf_min random 7 0.00500143387096774,0.479745967741935 19: 16750 -0.6980950 -0.6893123 1066.57768 1036.03988 6 0.30 combined perf_min best_to_worst 5 0.000425496774193549,0.000610483870967742 20: 16746 -0.6980593 -0.6898958 1066.57768 1036.03988 6 0.30 combined perf_mean best_to_worst 5 0.000425496774193549,0.000610483870967742 21: 16742 -0.6980588 -0.6898689 1066.57768 1036.03988 6 0.30 combined perf_geometric best_to_worst 5 0.000425496774193549,0.000610483870967742 22: 16753 -0.6979782 -0.6889488 1066.57768 1036.03988 6 0.30 combined corr_only worst_to_best 5 0.000425496774193549,0.000610483870967742 23: 10904 -0.6973225 -0.6724522 30046.28649 22805.95348 4 0.30 combined perf_mean random 7 0.00500143387096774,0.479745967741935 24: 12075 -0.6970813 -0.6727185 73.40478 63.99031 5 0.10 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 25: 7926 -0.6963318 -0.6708180 1066.57768 1036.03988 3 0.30 combined perf_mean best_to_worst 5 0.000425496774193549,0.000610483870967742 26: 7922 -0.6963088 -0.6707834 1066.57768 1036.03988 3 0.30 combined perf_geometric best_to_worst 5 0.000425496774193549,0.000610483870967742 27: 7933 -0.6959143 -0.6673391 1066.57768 1036.03988 3 0.30 combined corr_only worst_to_best 5 0.000425496774193549,0.000610483870967742 28: 7930 -0.6958662 -0.6701029 1066.57768 1036.03988 3 0.30 combined perf_min best_to_worst 5 0.000425496774193549,0.000610483870967742 29: 8808 -0.6958639 -0.7337437 172326.76028 56218.31666 3 0.40 combined perf_min random 7 0.00500143387096774,0.479745967741935 30: 19690 -0.6957423 -0.6903813 1066.57768 1036.03988 7 0.30 combined perf_min best_to_worst 5 0.000425496774193549,0.000610483870967742 param_idx fc_likert_corr_train fc_likert_corr_test cost_per_m_post_train cost_per_m_post_test max_models disagreement_threshold aggregation_method perf_metric model_selection_method cost_bin model_cost ``` top 30 models (for cost_bin == 1 or 2 or 3, ie cheapest three bins) ```r param_idx fc_likert_corr_train fc_likert_corr_test cost_per_m_post_train cost_per_m_post_test max_models disagreement_threshold aggregation_method perf_metric model_selection_method cost_bin model_cost <int> <num> <num> <num> <num> <int> <num> <char> <char> <char> <int> <char> 1: 12892 -0.6710269 -0.4467646 19.16705 15.40985 5 0.20 combined corr_only random 1 3.18258064516129e-6,0.0000118854838709677 2: 7424 -0.6675753 -0.5243785 19.55709 13.38112 3 0.25 combined perf_mean random 1 3.18258064516129e-6,0.0000118854838709677 3: 18768 -0.6582458 -0.6900143 18.51659 23.61430 7 0.20 combined perf_min random 1 3.18258064516129e-6,0.0000118854838709677 4: 7433 -0.6567313 -0.6960304 19.28146 19.54367 3 0.25 combined corr_only worst_to_best 1 3.18258064516129e-6,0.0000118854838709677 5: 7853 -0.6567313 -0.6976672 19.28146 19.32496 3 0.30 combined corr_only worst_to_best 1 3.18258064516129e-6,0.0000118854838709677 6: 3086 -0.6544707 -0.6985797 17.06497 17.06497 2 0.10 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 7: 3506 -0.6544707 -0.6985797 17.06497 17.06497 2 0.15 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 8: 3926 -0.6544707 -0.6985797 17.06497 17.06497 2 0.20 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 9: 3924 -0.6544707 -0.5200404 17.06497 18.04901 2 0.20 weighted perf_mean random 1 3.18258064516129e-6,0.0000118854838709677 10: 4346 -0.6544707 -0.6985797 17.06497 17.06497 2 0.25 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 11: 4766 -0.6544707 -0.6985797 17.06497 17.06497 2 0.30 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 12: 5186 -0.6544707 -0.6985797 17.06497 17.06497 2 0.35 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 13: 5606 -0.6544707 -0.6985797 17.06497 17.06497 2 0.40 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 14: 6026 -0.6544707 -0.6985797 17.06497 17.06497 3 0.10 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 15: 6446 -0.6544707 -0.6985797 17.06497 17.06497 3 0.15 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 16: 6866 -0.6544707 -0.6985797 17.06497 17.06497 3 0.20 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 17: 7286 -0.6544707 -0.6985797 17.06497 17.06497 3 0.25 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 18: 7706 -0.6544707 -0.6985797 17.06497 17.06497 3 0.30 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 19: 8126 -0.6544707 -0.6985797 17.06497 17.06497 3 0.35 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 20: 8546 -0.6544707 -0.6985797 17.06497 17.06497 3 0.40 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 21: 8966 -0.6544707 -0.6985797 17.06497 17.06497 4 0.10 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 22: 9386 -0.6544707 -0.6985797 17.06497 17.06497 4 0.15 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 23: 9806 -0.6544707 -0.6985797 17.06497 17.06497 4 0.20 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 24: 10226 -0.6544707 -0.6985797 17.06497 17.06497 4 0.25 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 25: 10646 -0.6544707 -0.6985797 17.06497 17.06497 4 0.30 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 26: 11066 -0.6544707 -0.6985797 17.06497 17.06497 4 0.35 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 27: 11486 -0.6544707 -0.6985797 17.06497 17.06497 4 0.40 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 28: 11906 -0.6544707 -0.6985797 17.06497 17.06497 5 0.10 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 29: 12326 -0.6544707 -0.6985797 17.06497 17.06497 5 0.15 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 30: 12746 -0.6544707 -0.6985797 17.06497 17.06497 5 0.20 weighted perf_mean best_to_worst 1 3.18258064516129e-6,0.0000118854838709677 param_idx fc_likert_corr_train fc_likert_corr_test cost_per_m_post_train cost_per_m_post_test max_models disagreement_threshold aggregation_method perf_metric model_selection_method cost_bin model_cost param_idx fc_likert_corr_train fc_likert_corr_test cost_per_m_post_train cost_per_m_post_test max_models disagreement_threshold aggregation_method perf_metric model_selection_method cost_bin model_cost <int> <num> <num> <num> <num> <int> <num> <char> <char> <char> <int> <char> 1: 9135 -0.7061135 -0.6673081 73.40478 63.99031 4 0.10 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 2: 6195 -0.7034021 -0.6727010 73.40478 63.99031 3 0.10 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 3: 12075 -0.6970813 -0.6727185 73.40478 63.99031 5 0.10 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 4: 15015 -0.6862089 -0.6759401 73.40478 63.99031 6 0.10 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 5: 3255 -0.6824805 -0.6295397 50.28532 50.28532 2 0.10 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 6: 3253 -0.6824805 -0.6295397 50.28532 50.28532 2 0.10 combined corr_only worst_to_best 2 0.0000129181451612903,0.0000331427419354839 7: 3675 -0.6824805 -0.6295397 50.28532 50.28532 2 0.15 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 8: 3673 -0.6824805 -0.6295397 50.28532 50.28532 2 0.15 combined corr_only worst_to_best 2 0.0000129181451612903,0.0000331427419354839 9: 4095 -0.6824805 -0.6295397 50.28532 50.28532 2 0.20 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 10: 4093 -0.6824805 -0.6295397 50.28532 50.28532 2 0.20 combined corr_only worst_to_best 2 0.0000129181451612903,0.0000331427419354839 11: 4515 -0.6824805 -0.6295397 50.28532 50.28532 2 0.25 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 12: 4513 -0.6824805 -0.6295397 50.28532 50.28532 2 0.25 combined corr_only worst_to_best 2 0.0000129181451612903,0.0000331427419354839 13: 4935 -0.6824805 -0.6295397 50.28532 50.28532 2 0.30 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 14: 4933 -0.6824805 -0.6295397 50.28532 50.28532 2 0.30 combined corr_only worst_to_best 2 0.0000129181451612903,0.0000331427419354839 15: 5355 -0.6824805 -0.6295397 50.28532 50.28532 2 0.35 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 16: 5353 -0.6824805 -0.6295397 50.28532 50.28532 2 0.35 combined corr_only worst_to_best 2 0.0000129181451612903,0.0000331427419354839 17: 5775 -0.6824805 -0.6295397 50.28532 50.28532 2 0.40 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 18: 5773 -0.6824805 -0.6295397 50.28532 50.28532 2 0.40 combined corr_only worst_to_best 2 0.0000129181451612903,0.0000331427419354839 19: 17955 -0.6759055 -0.6778079 73.40478 63.99031 7 0.10 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 20: 9555 -0.6741190 -0.6712540 64.99771 60.62023 4 0.15 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 21: 21860 -0.6732029 -0.4732345 70.30306 79.09758 8 0.25 simple perf_geometric random 2 0.0000129181451612903,0.0000331427419354839 22: 6615 -0.6729495 -0.6760361 64.99771 60.62023 3 0.15 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 23: 12495 -0.6721171 -0.6752511 64.99771 60.62023 5 0.15 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 24: 10395 -0.6718317 -0.6740640 58.16696 55.90212 4 0.25 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 25: 9975 -0.6716156 -0.6770854 59.21784 56.80081 4 0.20 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 26: 10815 -0.6715620 -0.6737822 57.11607 55.00343 4 0.30 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 27: 25232 -0.6709462 -0.5214947 66.13933 48.78989 9 0.30 simple corr_only random 2 0.0000129181451612903,0.0000331427419354839 28: 7455 -0.6708909 -0.6786630 58.16696 55.90212 3 0.25 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 29: 13755 -0.6707887 -0.6757449 57.11607 55.00343 5 0.30 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 30: 7035 -0.6706463 -0.6814273 59.21784 56.80081 3 0.20 combined corr_only worst2_then_best 2 0.0000129181451612903,0.0000331427419354839 param_idx fc_likert_corr_train fc_likert_corr_test cost_per_m_post_train cost_per_m_post_test max_models disagreement_threshold aggregation_method perf_metric model_selection_method cost_bin model_cost param_idx fc_likert_corr_train fc_likert_corr_test cost_per_m_post_train cost_per_m_post_test max_models disagreement_threshold aggregation_method perf_metric model_selection_method cost_bin model_cost <int> <num> <num> <num> <num> <int> <num> <char> <char> <char> <int> <char> 1: 22168 -0.6013248 -0.6234257 107.3645 155.7587 8 0.25 combined perf_min random 3 0.0000332201612903226,0.0000909483870967743 2: 3275 -0.5843072 -0.6468260 144.4606 144.4606 2 0.10 combined corr_only worst2_then_best 3 0.0000332201612903226,0.0000909483870967743 3: 3273 -0.5843072 -0.6468260 144.4606 144.4606 2 0.10 combined corr_only worst_to_best 3 0.0000332201612903226,0.0000909483870967743 4: 3695 -0.5843072 -0.6468260 144.4606 144.4606 2 0.15 combined corr_only worst2_then_best 3 0.0000332201612903226,0.0000909483870967743 5: 3693 -0.5843072 -0.6468260 144.4606 144.4606 2 0.15 combined corr_only worst_to_best 3 0.0000332201612903226,0.0000909483870967743 6: 4115 -0.5843072 -0.6468260 144.4606 144.4606 2 0.20 combined corr_only worst2_then_best 3 0.0000332201612903226,0.0000909483870967743 7: 4113 -0.5843072 -0.6468260 144.4606 144.4606 2 0.20 combined corr_only worst_to_best 3 0.0000332201612903226,0.0000909483870967743 8: 4535 -0.5843072 -0.6468260 144.4606 144.4606 2 0.25 combined corr_only worst2_then_best 3 0.0000332201612903226,0.0000909483870967743 9: 4533 -0.5843072 -0.6468260 144.4606 144.4606 2 0.25 combined corr_only worst_to_best 3 0.0000332201612903226,0.0000909483870967743 10: 4955 -0.5843072 -0.6468260 144.4606 144.4606 2 0.30 combined corr_only worst2_then_best 3 0.0000332201612903226,0.0000909483870967743 11: 4953 -0.5843072 -0.6468260 144.4606 144.4606 2 0.30 combined corr_only worst_to_best 3 0.0000332201612903226,0.0000909483870967743 12: 5372 -0.5843072 -0.6223780 144.4606 124.1685 2 0.35 combined corr_only random 3 0.0000332201612903226,0.0000909483870967743 13: 5375 -0.5843072 -0.6468260 144.4606 144.4606 2 0.35 combined corr_only worst2_then_best 3 0.0000332201612903226,0.0000909483870967743 14: 5373 -0.5843072 -0.6468260 144.4606 144.4606 2 0.35 combined corr_only worst_to_best 3 0.0000332201612903226,0.0000909483870967743 15: 5795 -0.5843072 -0.6468260 144.4606 144.4606 2 0.40 combined corr_only worst2_then_best 3 0.0000332201612903226,0.0000909483870967743 16: 5793 -0.5843072 -0.6468260 144.4606 144.4606 2 0.40 combined corr_only worst_to_best 3 0.0000332201612903226,0.0000909483870967743 17: 6776 -0.5803098 -0.6761293 178.4637 180.6185 3 0.20 simple icc_only random 3 0.0000332201612903226,0.0000909483870967743 18: 5933 -0.5794375 -0.6814204 177.7174 177.7174 3 0.10 simple corr_only worst_to_best 3 0.0000332201612903226,0.0000909483870967743 19: 5936 -0.5794375 -0.6864036 177.7174 122.1440 3 0.10 simple icc_only random 3 0.0000332201612903226,0.0000909483870967743 20: 5922 -0.5794375 -0.6814204 177.7174 177.7174 3 0.10 simple perf_geometric best_to_worst 3 0.0000332201612903226,0.0000909483870967743 21: 5926 -0.5794375 -0.6814204 177.7174 177.7174 3 0.10 simple perf_mean best_to_worst 3 0.0000332201612903226,0.0000909483870967743 22: 5930 -0.5794375 -0.6814204 177.7174 177.7174 3 0.10 simple perf_min best_to_worst 3 0.0000332201612903226,0.0000909483870967743 23: 6353 -0.5794375 -0.6814204 177.7174 177.7174 3 0.15 simple corr_only worst_to_best 3 0.0000332201612903226,0.0000909483870967743 24: 6342 -0.5794375 -0.6814204 177.7174 177.7174 3 0.15 simple perf_geometric best_to_worst 3 0.0000332201612903226,0.0000909483870967743 25: 6346 -0.5794375 -0.6814204 177.7174 177.7174 3 0.15 simple perf_mean best_to_worst 3 0.0000332201612903226,0.0000909483870967743 26: 6350 -0.5794375 -0.6814204 177.7174 177.7174 3 0.15 simple perf_min best_to_worst 3 0.0000332201612903226,0.0000909483870967743 27: 6773 -0.5794375 -0.6814204 177.7174 177.7174 3 0.20 simple corr_only worst_to_best 3 0.0000332201612903226,0.0000909483870967743 28: 6762 -0.5794375 -0.6814204 177.7174 177.7174 3 0.20 simple perf_geometric best_to_worst 3 0.0000332201612903226,0.0000909483870967743 29: 6766 -0.5794375 -0.6814204 177.7174 177.7174 3 0.20 simple perf_mean best_to_worst 3 0.0000332201612903226,0.0000909483870967743 30: 6770 -0.5794375 -0.6814204 177.7174 177.7174 3 0.20 simple perf_min best_to_worst 3 0.0000332201612903226,0.0000909483870967743 param_idx fc_likert_corr_train fc_likert_corr_test cost_per_m_post_train cost_per_m_post_test max_models disagreement_threshold aggregation_method perf_metric model_selection_method cost_bin model_cost ```